Reviewing a Paper

Hugh Davis

These are some notes I use in a lecture on Reviewing Academic Papers. They are offered in the hope that they might be a useful reminder of what we are trying to do.

Some things to consider when reading the paper.

• Is the abstract adequate?
• Does the introduction clarify the background and motivation?
• Does the manuscript clarify what has been done and why?
• Is the order of presentation satisfactory?
• Is the English satisfactory?
• What is the level of technical quality?
• What level of reader interest do you anticipate?
• Are the disclosed results accurate?
• Are the disclosed results significant?
• Does the manuscript describe original work?
• Does the paper have a significant tutorial/survey value?
• Should technical material be deleted or added?
• Are the references adequate?
• Have the authors kept to the guidelines for size and format?

The Report - Two Parts

1. To the committee

• your name.
• the paper's title, authors, and code number.
• a recommendation (accept, reject, etc.) and brief justification.
• describe your expertise in the subject area, especially if the paper is not in precisely your own line of research.No sneak attacks!

• say how confident you are of your views of the paper.
• remind the editor of any potential conflicts of interest you may have (naturally you mentioned these earlier, before agreeing to review the paper!)
• NO Sneak Attacks!

(In the context of Hypertext, the review form allows you to express most of the above pretty simply. If you really need to say something that you cannot say in your comments to authors, then email it to the chair. An example might be a suspected case of plagiarism or forged results.)

2. To the Author(s)

summary

• summarizes the paper succinctly and dispassionately. This is not the place to criticize, but rather to show that you understood the paper, and perhaps discuss how it fits into the big picture.

general comments

• gives the big critical picture, before sinking into the details. This is the place to take a breath, keep your perspective, and explain what the papers weaknesses are and whether they are serious, or intrinsic to our current state of knowledge, or whatever.
• Could you understand what the paper was saying from what they wrote?

constructive criticism

• not only of technical issues, but also organization and clarity.
• Are there things you wanted to see?
• Quality of results and argumentation?
• Quality of writing?
• Awareness of work in the field?

table of typos

• and grammatical errors, and minor textual problems. It's not the reviewer's job to copy edit the paper, so don't go out of your way to look for typos. And if the paper is a complete mess, just say so---but please be charitable, especially if English is not the author's native language.

To quote Ian Parberry: ``Desirable traits in a referee include objectivity, fairness, speed, professionalism, confidentiality, honesty, and courtesy'' and ``Before submitting a finished report, a wise referee asks ``Would I be embarrassed if this were to appear in print with my name on it?''

Primary Sources

Ralph E. Johnson, Kent Beck, Grady Booch, William Cook, Richard Gabriel, Rebecca Wirfs-Brock. 1993

How to Get a Paper Accepted at OOPSLA. http://www.acm.org/sigplan/oopsla/oopsla96/how93.html (last accessed Nov 3rd 2000)

Barak A. Pearlmutter. How to Review a Scientific Paper. http://www.cs.unm.edu/~bap/how-to-review.html (last accessed Nov 3rd 2000)

IEEE. Review Form. http://external.nj.nec.com/homepages/giles/reviewing/IEEE.TNN/new.review.form (last accessed Nov 3rd 2000)