publications - What should you do if you spotted a non-trivial error in a highly cited paper?

I have seen question on here regarding a citation error in a paper here: What should I do if I found a citation error in a published paper?, but the nature of this problem is very different.

Basically, I came across an error in a paper (call this error 2).

I have previously contacted the author before regarding a different problem (call this error 1, say) in this paper, and I got no response. (That was what I still believe to be an unproven proposition, and is actually also related to this problem.)

The reason I decided to ask here this time is the following:

1. 
   

   I am highly confident about error 2.

   
   



2. 
   

   Its correctness has a direct impact on my current research.

   
   


3. 
   

   The mistake cannot be easily fixed. (At least it appears to me, for the time being). I have spent months on a related problem before revisiting this paper and realised that I have tried this same technique in problem, which did not work in my problem, which led me to check why it worked in his paper - and it turned out, it did not.

   
   


4. 
   

   The result was probably the first/the only result of its kind in the literature and the paper is fairly important with 150+ citations.

   
   

What should I do here? I have previously contacted the author and I have no reason to believe he would respond this time. I want it to be correct/easily patched because of reason 2. This paper was published in 1997 so it has been a while....

EDIT: Precisely for reason 2, I would much prefer some way of getting the author to respond.

EDIT: I had a discussion with my supervisor. He immediately agreed with me it was a mistake. The thing is when we tried to use a similar technique for a different problem, we fell down the same trap about 10 times. At the moment, we have no fix and the mistake is serious enough that it takes away a lot of credit from the paper.

Answer

The main answer to your question is easy: talk to your advisor.

I find it a little curious that you have not done this already. As a graduate student, you read a very famous paper (in mathematics, more than 150 citations is a very large number) that was written a while ago, and at the first sign of trouble you contacted the author of the paper. That is already a little strange: although both are possible, the probability that you, a relatively young graduate student, have misunderstood something in the paper (or are applying standards that are different from those of the field you will be working in, or some other similar issue related to the fact that you probably do not yet have expert level knowledge and experience in this field) is higher than that the author has made a serious mistake. Writing to someone that you don't know at all and who is much more senior than yourself and pointing out a mistake is not without risk: the risk is that you will be wrong and they will dismiss you in the future as a less than serious person.

Your advisor is there exactly for such things: she is the person who is helping you transition from a neophyte to a journeyman to an expert, and she needs to see your mistakes and flawed reasoning in order to do this properly. Some outside expert really does not: they can wait to see the finished product that you become. Moreover, you are presumably reading the paper because your advisor wants you to, perhaps even because she directly told you to read it. Therefore a mistake in the paper is your advisor's problem as well as yours. Why are you holding that information back from her? You shouldn't.

Having said that, I feel like the OP is getting a fair amount of advice that would be more appropriate in other fields than mathematics. Especially:

In mathematics it is very difficult to write a paper whose sole or even primary purpose is to point out a mistake (even if it includes a correction) in someone else's work.

This is a cultural reality of the field; it is certainly not entirely positive. In many other academic fields, one can "score points" by pointing out others' mistakes, and in some fields this is one of the best ways to score points. Mathematics is not like this: if you can get such a paper published it will "take points away" from the author and give you a certain amount of notoriety, but if this is for instance your first published paper then many people will look at you strangely, almost as if they expect you to make further trouble. (Again, I'm not endorsing this cultural practice; I'm just describing it.)

Similarly, I would say that contacting the editor of the journal in question is maybe step 10 of a procedure that is mostly designed to terminate after one of the first 9 steps. You should do this only after you have exhausted every other possible avenue.

In mathematics -- very much unlike in some other fields -- it is prohibitively unlikely that an editor will publish any "commentary/correction/erratum" by you unless she believes that you are mathematically correct in the point you're making, and convincing someone with standing in the field like a journal editor that you are mathematically correct is largely what you're trying to do anyway.

Finally, while I barely know what a white paper is and to the extent that I do I'm not sure it's the appropriate terminology here -- in content I do agree with @eykanal's suggestion: it will certainly help your advisor if along with communicating the surprising news that Famous Paper X is wrong, you can supply a written version of your arguments. This can be hard to do: explaining why a difficult argument is subtly wrong is one of the sternest expository challenges in mathematics that I can think of. It takes a lot to go from "I don't understand the argument and find it rather unclear" to "I am convinced that it is wrong." By the way, in writing to an author it is a good strategy to err on the side of the former quotation.