Instructions for the HCI Reviewer

Thank you for agreeing to review a paper for HCI. You will help us judge whether the paper is publishable. Even more important, your constructive comments can help the author(s) improve their paper (often dramatically, in our experience) to make it a valuable contribution. Your role is vital.

Procedure

Send your review to the paper’s Review Editor, who asked you to be a reviewer, and please do it by the deadline given you by the Review Editor. Follow the format and use the criteria described below.

Format of the Review

The review should be in the body (not an attachment) of an email message, simply formatted, preferably plain text.

Begin the review with a Reviewer’s Summary, which gives your bottom-line assessment. Use the template at the bottom of this page. The rest of the review is free-form format.

To start, briefly state the nature and contributions of the paper. This establishes how you see the paper and provides a basis for your comments.

The bulk of the review is your comments on the paper. Make clear arguments as to why the paper is acceptable or not, and give constructive and specific suggestions as to how the paper can be improved. If you recommend that the paper be accepted, state what its major contributions are. If you recommend that the paper be rejected, state what its fatal flaws are and why they can't be dealt with by revising the paper. These comments should be as specific as possible and to the point. If you think the paper would be acceptable after revisions, then give a constructive critique of the paper's weaknesses and deficiencies and what it will take to remedy them. Also, suggest which parts of the paper are extraneous to its main thrust and can be cut or reduced. Your reviewing efforts can make a big difference.

Review Criteria

We want you to assess the quality of the paper’s contribution, which includes both whether it has something of substance to say and whether it presents it logically and clearly. The criteria for judging the quality of the content depend on what kind of paper it is. Human-computer interaction is an interdisciplinary field, and HCI is an eclectic journal, as reflected in its subtitle: A Journal of Theoretical, Empirical, and Methodological Issues of User Science and of System Design. While a paper can focus on a combination of these aspects, a paper will usually have a principal thrust. Your first job as a reviewer is to figure out what kind of paper it is (and not to worry about what it is not).

  1. If the paper is mainly theoretical, it should be judged on the insightfulness of its ideas and the rigor of its arguments. If it is mainly empirical, it should be judged on the validity of its results and the appropriateness of its techniques. If it is mainly a methodology paper, it should be judged on the usefulness of its proposed methodology. If it is a design paper, it should be judged on the quality, novelty, and appropriateness of its solution to a problem; the rationale for the design decisions; the implementation; and the assessment of its success and weaknesses.
  2. The paper should deal with important issues (i.e., having practical applications to or implications for human-computer interaction), and it must be technically sound in how it deals with these issues.
  3. The paper must be original in its insights, results, or approach; and it must not have been published previously.
  4. The paper must be scholarly and not insular, i.e., it must establish its relation to other work in the field.
  5. The paper should have a clear and logical presentation, so as to be readable by the whole interdisciplinary HCI audience and not just by specialists in some sub-field. Any interested HCI reader should be able to learn something from the paper.

A Few Other Issues

Each paper is sent to at least one "expert" reviewer, who is a technical expert on the paper's subject matter, and one "knowledgeable" or "general HCI” reviewer. Thus, you should assess the technical content of the paper to the level you can.

The default is for you, as reviewer, to remain anonymous. So don't identify yourself or your affiliation in your review. However, you may, if you wish, identify yourself to the author (e.g., the author may want to acknowledge your comments); do this by giving your name at the end of the review.

The paper you are reviewing is to be treated as the author's working paper, so please do not quote from it or distribute it.

If the paper is accepted, the HCI Production Editor will do a thorough copy-editing job. So, you don’t have to worry about low-level editing (e.g., punctuation, spelling, or wording), unless these have a significant impact on the content of the paper.

You may wish to make some comments to the Review Editor only. This is okay. Please mark these clearly, so the Review Editor does not send them to the author.

Template for the Reviewer's Summary
Copy the text below and delete all but one of the alternatives in each set.

On the subject matter of the paper, I am a [EXPERT | KNOWLEDGEABLE | GENERAL HCI] reviewer.

In its current version, the contents of the paper would be a [BREAKTHROUGH | MAJOR | SOLID | ORDINARY | MINOR | WEAK | NEGLIGIBLE] contribution to the HCI literature.

If the paper were revised, it could be a [BREAKTHROUGH | MAJOR | SOLID | ORDINARY | MINOR | WEAK | NEGLIGIBLE] contribution.

As for the presentation in the current version, the logical organization is [BEAUTIFUL | GOOD | ADEQUATE | POOR | TERRIBLE]; the writing style and clarity is [BEAUTIFUL | GOOD | ADEQUATE | POOR | TERRIBLE]; and the figures and tables are [BEAUTIFUL | GOOD | ADEQUATE | POOR | TERRIBLE].

I recommend that the paper be [ACCEPTED BASICALLY AS IS | ACCEPTED IF MINOR REVISIONS ARE MADE | ACCEPTED (PROBABLY) IF MAJOR REVISIONS ARE MADE | REJECTED, BECAUSE THERE IS NOT A SUBSTANTIAL BASIS FOR REVISING IT].