This is a new document serving as a repository of observations made during conference organization. It now has a few seeds, but will hopefully grow into more of a broad, full-fledge set of guidelines for those participating in conference organization.
A common problem in reviewing is that reviewers, although given a long time to write reviews, wait until just before the deadline. While they may allow enough time to give full attention to a single-pass review, this does not allow enough time to conflicts or questions their review raises to be addressed. Making the review process more of an ensemble effort, giving the process several phases, each with its own deadline helps to
The phases such a process could have include
Initial Reviews This is already a familiar phase. Here, each reviewer reads the paper and submits a review without communication with other reviews of the paper. In this manner, his or her opinions are not influenced by those of the other reviewers.
Meta-Review Gathering and Resolution of Conflicts It is helpful if each paper has one member of the committee that is a meta-reviewer. A paper's meta-review is responsible for gathering the reviews from all the reviewers of the paper and gathering from them one common understanding of the reviews for communication to the Program Committee. A key part of this process is the meta-reviewer recognizing where there are conflicts between the reviews and understanding what they mean. Reviewers that disagree should not be made to agree, but the meta-reviewer should understand the nature of the difference of opinion. Giving a reviewer access to the other reviews of the same paper after he or she has submitted his or her own helps start the inter-reviewer communication that moves this process along. Some Web-based review systems provide such access once the initial review has been fully submitted. The phase should at least find any conflicts that the author needs to address in the next phase, but ideal this time should be well used to discuss the paper in general.
Allowing Authors to Answer Specific Concerns Sometimes reviews find issues that block the review process that can only be resolved by the author. Examples include access to previous publications that reviews feel may have too much in common with the current. Authors could also be asked to explicitly clarify certain issues, such as what aspects of the paper are new work. Deadlines for this process need to be scheduled and communicated in a way that ensures authors have time to properly respond to concerns about their submission.
Having Reviewers of Each Paper Agree on Meta-Review By a certain date long enough before the Program Committee itself, all the issues about a paper should be resolved to the meta-review process can be considered over. This includes the author rebuttal process, and the reviewers' reactions to the author rebuttal. It also includes any discrepancies between the reviewers themselves being understood and documented by the meta-reviewer. Now that reviewer discrepancies and author comments have have been address, the meta-reviewers should be ready to deliver result of this meta-reviewing process: a written meta-review that the Program Committee can access with the detail reviews.
Discussion of Papers at the Program Committee Meeting All the work of the previous phases leading up to this one should leave the meta-reviewer and the committee as a whole ready to discuss and evaluate each paper's potential inclusion in the program. Since there is typically only one day to do this in, and the decisions are very important and relatively irrevocable, it is important to be as prepared as possible to make sure all necessary decisions can be made in this single day. This includes giving all Program Committee members good access to the materials for each paper: the meta-review, the reviews and the paper itself. Any material made available to the members before they travel to the meeting helps them to prepare to make the meeting proceed more quickly. It also helps to clarify what the specific goals are for the meeting as a whole and for each paper: what are accepted, what are rejected, what are recommended for other venues in the conference and what are candidates for the various prizes the conference offers.
Conferences often have different layers of participation. Some of these are refereed by the conference committee, having a submission, review and notification cycle. Typically, these are, in decreasing order of magnitude full papers, short papers, posters and demos. These often have different committees with different due dates. However, since submissions rejected for one layer are often well-suited for the next layer below, some conferences have mechanisms for proposing, or directly accepting, a rejected submission for the next layer under. Having these difference committees coordinate these passed-along acceptances helps to maximize participation in the conference.
Synchronizing Submission and Notification Dates One important aspect of this coordination is synchronizing due dates. If the due date for submitting a short paper is a reasonable amount of time (say, one week) after the full paper notifications go out, then those full paper submitters who were rejected but recommended for short papers can repackage their draft for a short paper submission. Similarly, if posters and are separate from short papers, having a similar buffer between short paper notification and poster submission allows authors to repackage their rejected short papers into posters and demos.
Access to Materials from Previous Submissions Communication between the committees for these layers helps. One potentially helpful communication is if a committee can have access to the reviews and original material for their submissions that are reworked rejections earlier submissions in the same conference. Higher-layer committees can also state explicitly to lower-layer committees what rejection notifications had recommendations for submission to their lower-layer venue. Such communication can help clarify:
Direct Acceptance to Lower Layer Venues One option for higher-layer rejects is to directly accept some to lower-layer venues without requiring resubmission. This is helpful if you did not receive enough submissions for a venue to fill up the number of available slots. While full and short papers often have a limited number of slots, since they typically have time periods in paper tracks with devoted audiences, poster often have no such constraint, other than the amount of physical space in the allocated room, which can often be inexpensively quite large. Thus, directly accepting full or short paper rejects as posters can help ensure that the number of posters is large, increasing conference attendance and the amount of ideas the conference communicates.
Contributors: Lloyd Rutledge