Repeatability & Workability Evaluation

The Goal

On a voluntary basis, authors of accepted SIGMOD 2010 papers can provide their code/binaries, experimental setups and data to be tested for

• repeatability of the experiments described in the accepted papers;
• workability in the sense of running different/more experiments with different/more parameters than shown in the respective papers;

The People

Instruction for Authors

The Procedure

Authors of accepted papers will be contacted by e-mail as soon as acceptance is determined and invited to submit

• (download URIs for) code (executables or sources)
• (download URIs for) data
• instructions on:
  • how the experiments described in the paper should be re-run;
  • how further experiments could be run, on inputs similar to those used in the paper, but different from those.
• the accepted paper

Past experience has demonstrated that repeatability and workability can greatly benefit from the availability of authors to interact with the repeatability reviewer, and help solve minor issues related to the installation, configuration, and usage of the code. Thus, the electronic tool will enable repeated interactions between the reviewers and the authors in the style of a message board. Thus, the 2010 process will involve such interaction, too.

The RWO will designate for each submission:

• A first reviewer, who will do all the verification work as far as he/she is able to (download and install the code, run it, write a repeatability/workability report). This should span at most 2/3rds of the reviewing period.
• A second reviewer, who will check the report of the first, help clarify any pending issues. The second reviewer is expected to interfere, if needed, in the last 3rd of the reviewing period.

During the evaluation, the first reviewer and the authors interact via the RWCMT. The second reviewer and the repeatability & workability co-chairs may also participate to the discussion. The recommendation is that the first reviewer is left alone with the authors during the first 2/3rds of the reviewing period, to avoid confusion. The RWCMT documents all interaction between the reviewers and the authors.

The identity of the reviewers is hidden during the evaluation process, but obviously will be revealed afterwards when the reviewers sign their report.

The Results

21 of the 80 accepted papers of SIGMOD 2010 participated in the repeatability & workability evaluation. The results of the 18 papers who's authors agreed on publication are available below.

RWE ID	Title (link to paper)	Authors	Reviews (link to details)		Results
			Primary	Secondary	Repeatability	Workability
07	Towards Proximity Pattern Mining in Large Graphs	Arijit Khan, Xifeng Yan, Kun-Lung Wu	Fabian Hüske	Zhenjie Zhang	Mostly confirmed	Mostly confirmed
08	Histograms Reloaded: The Merits of Bucket Diversity	Carl-Christian Kanne, Guido Moerkotte	Milena Ivanova	Zhenjie Zhang	Fully confirmed	Partly confirmed
10	GAIA: Graph Classification Using Evolutionary Computation	Ning Jin, Calvin Young, Wei Wang	Stratos Idreos	Zhenjie Zhang	Fully confirmed	(not checked)
11	Non-homogeneous Generalization in Privacy Preserving Data Publishing	Wai Kit Wong, Nikos Mamoulis, David Wai Lok Cheung	Jong Wook Kim	Frank Huber	Fully confirmed	Mostly confirmed
12	Bed-Tree: An All-Purpose Index Structure for String Similarity Search Based on Edit Distance	Zhenjie Zhang, Marios Hadjieleftheriou, Beng Chin Ooi, Divesh Srivastava	Dan Olteanu	Frank Huber	Mostly confirmed	Fully confirmed
13	Continuous Sampling for Online Aggregation Over Multiple Queries	Sai Wu, Beng Chin Ooi, Kian-Lee Tan	Stefan Manegold	Paolo Papotti	Mostly confirmed	(not checked)
15	Durable Top-k Search in Document Archives	Leong Hou U, Nikos Mamoulis, Klaus Berberich, Srikanta Bedathur	Frank Huber	Fabian Hüske	Fully confirmed	Fully confirmed
17	Efficient Parallel Set-Similarity Joins Using MapReduce	Rares Vernica, Michael Carey, Chen Li	Fabian Hüske	Wai Kit Wong	Fully confirmed	Partly confirmed
18	Sampling Dirty Data for Matching Attributes	Henning Koehler, Xiaofang Zhou, Shazia Sadiq, Yanfeng Shu, Kerry Taylor	Wai Kit Wong	Zhenjie Zhang	Fully confirmed	Fully confirmed
20	Connected Substructure Similarity Search	Haichuan Shang, Xuemin Lin, Ying Zhang, Jeffrey Yu, Wei Wang	Frank Huber	Ryan Johnson	Fully confirmed	Mostly confirmed
21	ParaTimer: A Progress Indicator for MapReduce DAGs	Kristi Morton, Magdalena Balazinska, Dan Grossman	Ryan Johnson	Wai Kit Wong	Fully confirmed	Fully confirmed
22	Positional Update Handling in Column Stores	Sandor Heman, Marcin Zukowski, Niels Nes, Lefteris Sidirourgos, Peter Boncz	Stefan Manegold	Dan Olteanu	Mainly confirmed	Partly confirmed
23	Low Overhead Concurrency Control in Partitioned DBMSs	Evan Jones, Daniel Abadi, Samuel Madden	Asterios Katsifodimos	Jong Wook Kim	Fully confirmed	Fully confirmed
24	Load-Balanced Query Dissemination in Privacy-Aware Online Communities	Emiran Curtmola, Alin Deutsch, K.K. Ramakrishnan, Divesh Srivastava	Paolo Papotti	Jong Wook Kim	Mostly confirmed	Mostly confirmed
25	Hierarchically Organized Skew-Tolerant Histograms for Geographic Data Objects	Yohan Roh, Jae Ho Kim, Yon Dohn Chung, Jin Hyun Son, Myoung Ho Kim	Wai Kit Wong	Asterios Katsifodimos	Fully confirmed	Fully confirmed
26	Active Knowledge : Dynamically Enriching RDF Knowledge Bases by Web Services	Nicoleta Preda, Fabian Suchanek, Gjergji Kasneci, Thomas Neumann, Wenjun Yuan	Dan Olteanu	Asterios Katsifodimos	Mostly confirmed	Partly confirmed
27	Automatic Contention Detection and Amelioration for Data-Intensive Operations	John Cieslewicz, Kenneth Ross, Kyoho Satsumi, Yang Ye	Ryan Johnson	Milena Ivanova	Fully confirmed	Fully confirmed
28	Data Conflict Resolution Using Trust Mappings	Wolfgang Gatterbauer, Dan Suciu	Paolo Papotti	Milena Ivanova	Partly confirmed	Partly confirmed