Jurgen J. Vinju Blog

Some of the main conferences in Computer Science and Software Engineering are (finally) moving their PC meetings online. This means that their program committees will not meet in person anymore somewhere on the planet to discuss the submissions to their conference anymore.

Arie van Deursen wrote about this move to online meetings and why this is a good plan here.

I’ve been PC member, PC co-chair, steering committee member and steering committee chair mostly of conferences who have never had anything but an online meeting. These conferences are generally younger than the ones being converted now, and therefore were never allowed to allocate as much resources from their respective communities as the older conferences have done. Also, being invented in more modern times, these conferences did not have the burden of tradition and the fear of change accompanied by this. I think its great that we are now all moving away from wasting our planet and missing our family quality time for having physical PC meetings.

But, having a great online PC “meeting” (as if you would call it that) is an art in itself due to a number of challenges and pitfalls. Based on my (subjective) experience I’ve listed some points here that may be beneficial to the newbies :-)

In particular, the concept of “groupthink” is useful when thinking about the online paper reviewing process. More on this concept and what it entails for us can be found below.

Not fearing to state the obvious, here comes the deal. An online meeting perhaps has the same purpose as an actual meeting and the same data is being processed, and some of its social parameters may be slightly comparable because of this. That does not change the fact that nearly everything about the way we interact during an online meeting is completely different.

PC members and the PC chair do not look eachother in the eye.
the actual discussion is written rather than vocal, and often more voluminous because of this.
the discussion is planned during a longer time span (a week, two weeks instead of two days).
the discussion is asynchronous between the members of each discussion.
and, most importantly, the big social group splits into many many sub-groups, one for each paper under discussion.

I’m not a social scientist, but you can imagine the impact these little points have on the quality of any discussion. Probably you’ve experienced what havoc email can do to the quality of social interaction as compared to a phone call. Well, this is just like that.

The quality of the reviewing and online discussion process is observed in two aspects:

the quality of the decision: are the right papers, accepted or rejected? In particular we need to avoid the accidental rejection of a paper due to minor issues, or the accidental acceptance of a paper which has serious flaws. Both pitfalls are more pronounced in online discussions due to the volume of the written material (paper + reviews + discussions = noise), and the effort it takes to produce and consume all this information. There are no vocal summaries anymore.
the quality of the feedback: do the authors receive the necessary information in a form that is both understandable and digestable? In an online meeting, a badly written review is not as visible and to fewer PC members than it is in a physical meeting.

Both these qualities are challenged by the above five differences in interaction between PC members and the PC chair. Both the PC chair and the PC members have a responsibility in taking up these challenges and making the best of it.

With this in mind, the rest of this blog contains things you can do to mitigate the drawbacks of online PC meetings.

Mitigating the causes of “groupthink”

A helpful construct in thinking about online PC meetings is groupthink. From Wikipedia:

Groupthink is a psychological phenomenon that occurs within a group of people in which the desire for harmony or conformity in the group results in an irrational or dysfunctional decision-making outcome. Group members try to minimize conflict and reach a consensus decision without critical evaluation of alternative viewpoints by actively suppressing dissenting viewpoints, and by isolating themselves from outside influences.

It is very important to realize that groupthink is not intended by any of the group’s members; it is a way we behave as human beings in a group without explicitly realizing we are doing this. There is no blame for groupthink to assign inside of the group. Those who manage (define, monitor, exploit) a group though, have to think how to avoid the causes of groupthink. They are responsible.

Among the prime causes for groupthink are the natural circumstances of an online PC meeting. Namely, groupthink is caused by:

High group cohesiveness; typically re-inforced by the community of the conference and the paper selection process which is based on preferences of the individual PC member.
Group insulation; i.e. online discussions are between PC members who have reviewed the paper only.
Lack of impartial leadership; online discussions are among peers without a designated leader of the discussion;
Homogeneity of members’ social backgrounds and ideology; a natural phenomenon in conferences
Lack of norms requiring methodological procedures; especially with ground-breaking research a paper can be difficult to assess in a methodological manner.
Highly stressful external threats; especially for PC members who do not have tenure yet, being part of a PC can be felt as an extended and ever-lasting evaluation process.
Recent failures; say no more. At every conference the quality of the decision making regarding papers on display is being discussed.
Excessive difficulties on the decision-making task; reviewing conferences papers is one of the hardest and most intellectual challenging (and also very gratifying!) tasks of our profession
Time pressures; online PC meetings are scheduled while the PC members are performing many other tasks and have many other deadlines.
Moral dilemmas; a general issue in the review process

So, groupthink is probably the worst enemy of the “online” PC chair. It makes us make the wrong decisions, and an online PC meeting has all the ingredients for it -more so than a physical PC meeting. Please read more about each cause on Wikipedia later.

Now these causes give rise to a number of simple (but not easy) things you can do as a PC chair to help avoid groupthink:

Put even more time in the paper assignment process to avoid high group cohesiveness; because an actual meeting can not iron out the accidental mistakes you’ve made there:
- balance expertise with diversity, yes reviewers should be knowledgable about the topic, but no they should not all have written papers about the same subject.
- balance personality traits: make sure different types of people are reviewing the paper (perhaps you could use Myers/Briggs as a yardstick or Jungs personality types; it does not really matter how you do it, just try to be more diverse in creating these small groups)
- allow for even more diversity in the PC member selection process, such that you have enough diversity to construct the subgroups from
- pro-actively involve more reviewers during the online discussion if you see:
  - decisions without factual and meaningful discussion
  - no discussion
Be present during the online PC meetings as a PC chair in every subgroup, this is to avoid lack of impartial leadership, and lack of norms, as well as providing a safe environment to make errors:
- kick-off every discussion on every paper with a balanced summary of the reviews. This helps focusing the discussion, or at least points the discussion towards first establishing what the discussion should be about. Since reviews are written and the discussion is also, different writing styles and rethorics can dominate any discussion. You have to make sure to level the playing field here and give everybody their voice. Also your objective and external presence will ensure some level of impartial leadership. Do not take any position in the discussion here. If you do, you’ve just contributed in the worst possible way to the groupthink phenomenon.
- repeat norms; offer (ad nauseam) aspects of the call for papers, community values and moral standpoints, also answer implicit and explicit questions about such things during the discussion. An email with these aspects to the PC members is not enough.
- ask specific questions; designed to target the pitfalls of unbalanced discussions. Offer a way out for PC members who’ve “dug in” too much during an earlier argument. Offer PC members the opportunity to backpeddle on their earlier writings by asking them open “stupid” questions.
- invite other members who have not reviewed the paper to join discussions, by association or to resolve an impasse, or to stir the waters of a seemingly “clear decision”, or for whatever other reason. If you can convince anybody to look at more papers somehow, it is always a win.
- give personal feedback directly and immediately to the author of a review (too short, too long, not understandable, indigestable, etc.) and provide a safe environment for the PC member to improve their reviews quickly. This also requires time, so make sure to motivate PC members to submit their reviews often and early.
Make personal contact, in real life or at least on the phone or voice over IP.
- Especially to welcome new PC members to the community, an email does not suffice to convey the importance of the job, or how welcome they are in the community. If some stranger requires an exceptional effort from you, would you give it if they ask you by sending a cheap email? No. Would you feel welcome to be a community member, never having heard somebody’s voice? No. So, pick up the phone and talk about the conference, or make sure to talk to prospective PC member during the previous instance.
- Call PC members during the discussion, and make transparent reports about this in the online meeting. You can organize a quick call between members of a subgroup, or you can gauge emotions of a particularly forceful PC member during a private call. Do not sit down and think “o this medium is so bad and I can not do anything about it”. You have cheap means to resolve issues personally and effectively: it is called a phonecall. Stevie Wonder has a nice song about it.
- Call PC members who are late in submitting their papers; listen to their personal circumstances, allow them to fix the issue, or allowing them to make a safe exit and be replaced in time. This all to avoid letting this wound in your procedure fester until it is rotten beyond repair. Remember, a physical meeting forces all members to either own up to their failure by letting you know they can’t come, or make sure to have done the work by showing up at the meeting. A prolonged online meeting fails in less visible, but sure as hell problematic ways, if the reviewers do not do their work in time.

Wrapping up

The groupthink construct helped me think about online PC meetings in the past. I hope it will help others. There is a lot more to say about improving the quality of the decision making process and the reviews in general. I feel the groupthink focus helps a bit in distinguishing online meetings from physical meetings. It provides starting points in mitigating the primary pitfalls of the online paper reviewing process.

Management summary:

An online PC meeting is not really a “meeting” in any sense of the word;
PC members and chairs: be aware of cheap consensus and the causes of groupthink, and actively work against groupthink;
PC chairs: take an impartial, open position, as a present leader in every discussion;
PC members and chairs: make more pro-active personal contact (via the phone or physically), while making sure to remain transparent to the other members of the community;