publications - How to React to Incorrect Claims by Reviewers?

As this has happened to me several times for different papers now, I am not describing a specific case here, but the abstract aspects that were common to all situations.

Occasionally, in the "Tasks" or "Materials" section of the "Evaluation" or "User Study" chapter, papers of mine contain statements such as:

• "Figure ... shows one of the tasks from the user study. The correct answer was ... ."


• "For the user study, we had prepared eight tasks. They were given in the form of natural-language questions. Users were supposed to use our novel concept to input the appropriate parameters as described by the questions. Questions ranged from simple (such as '...?', see Figure ... for the solution) to complex (such as '...?', cf. solution in Figure ...)."

In these cases, the figure captions once again mention the figures depict questions and/or solutions from the user study.

Occasionally, these are met with reviewer comments such as:

• "The paper does not provide any example of the tasks presented during the user study. It would greatly improve the quality of the paper if the authors could at least provide one such example. Without this, I find it very hard to get a clear idea of how the study was conducted, and thereby, I also cannot tell how reliable the results presented in Section ... are."

I base my reasoning on the idea that this reviewer comment is, plain and simple, incorrect, as I did provide concrete examples of tasks. If I am somehow mistaken about this, please do let me know.

While such statements were certainly not the only reason for voting to reject, some of the reviewers who wrote something along these lines made it rather clear that they saw the alleged lack of any task examples as one of the most critical issues with the submission.

My question focuses on the situation that such a paper is getting revised - either, because it was accepted after all, and I am preparing a camera-ready version, or because it was rejected, and I am trying to improve it before submitting it elsewhere: What is the appropriate reaction with respect to such comments?

• Ignore the statements and do not change anything about the respective examples in my paper. This might be a viable way to go, but it leaves a bad taste to completely ignore some reviewer statements, even though they are clearly comprehensible and should be straightforward to implement.


• Re-word the sentences mentioning the examples. I might do that, but I am wary of making things worse, as I already consider the respective statements in my paper quite unambiguous as they are.


• Add more examples, and possibly at different locations. This could make it harder to miss the examples, but in CS conference papers, space is an extremely scarce resource.


• Ask the editor. Especially in cases where the paper was eventually still accepted, this seems quite extreme to me. I feel contacting the editor should be reserved to truly exceptional and severe issues, not "every-day worries" of all authors who are preparing their CR-versions.

Note that I am not asking how to deal with the review decision. Even though it is slightly annoying if a rejection is indeed based on incorrect reviewer statements, I am ok with resubmitting a revised version elsewhere. I am specifically asking how to revise the particular parts that were incorrectly criticized.

Some peculiarities of my field to note that might not be obvious:

• Only some conferences have a rebuttal phase. If they do, it is not guaranteed (or even reasonably likely) that especially reviewers who criticized a paper a lot will actually read the rebuttal and possibly even amend their review. Also, these rebuttals are usually limited in length (e.g. to 500 words), so you can usually mention only a selection of all issues raised by reviewers.


• CR-versions are rarely accompanied with explanatory letters to the editors (or, more precisely, the program committee) in my experience.
  
  
  
  • Submission usually happens by means of web-based submission systems which allow the upload of exactly one PDF file, which is sometimes automatically checked on formal factors, such as number of pages (so even tacking an extra page with explanatory notes in front of the paper will not work).
  
  
  • If an opportunity is provided to upload some additional files, this is usually meant for uploading sources and supplementary material such as videos that will be put on the proceedings CD. It is not clear whether the material will be sifted thoroughly enough so letters to the editors will be found in time and treated appropriately.
  
  
  
  


• My field is related to HCI.
  
  
  
  
  • Therefore, contributions are sometimes graphical in nature. Consequently, figures sometimes do not just provide information about the contribution (statistical graphs, exemplary architectures), figures are often a crucial part for showing the contribution. In other words, sometimes, the depiction is the contribution, or it shows how the novel concept presented in the paper does something.
  
  
  • User studies are used to test how well human users can get along with the novel concept. Therefore, tasks are often written as questions. Study participants then have to find the correct answers to these questions by using the novel concept being tested. This usually means that they have to interact/make inputs with/in the implemented prototype that shows the novel concept. This is meant to mimick how the concept would be used in real life, where users would use the respective novel HCI technique for retrieving information, as well.
  
  
  
  

Answer

I almost never ignore a reviewer comment. Even if the reviewer is simply wrong, the way in which they are wrong usually points out a way that some significant subpopulation of readers is likely to misread the paper.

Given this, when a reviewer seems to have simply overlooked a chunk of the paper, I will generally fiddle with the wording, the structure or the introductory material in order to make it pop out more clearly.

For example, with your "eight tasks" example, I might do any or all of the following to make the presence and visibility of the material more blatant:

• Make sure that the introduction has a "the structure of this paper" subsection that includes the word "example" in it.


• Do the same for the section including the tasks with a "structure of this section" paragraph at its beginning.


• Add a subsection header for "Examples of Tasks"


• Put the question examples into a bullet list, rather than prose

Remember, the reviewer may be incompetent and sloppy, but many of your readers will be also. You want to get your point across to as many of them as possible, regardless.