Maintained by Stefan Manegold RWE 2008 2009 2010 hosted by CWI

ACM SIGMOD 2010

Repeatability & Workability Evaluation

SIGMOD 2008 was the first database conference that proposed testing the code associated to conference submissions against the data sets used by the authors, to test the repeatability of the experiments presented in the submitted papers. A detailed report on this initiative has been published in ACM SIGMOD Record, 37(1):39-45, March 2008.

The experience has been continued in a slightly modified form in conjunction with the SIGMOD 2009 conference. A report on this effort has been published in ACM SIGMOD Record, 38(3):40-43, September 2009.

The repeatability and workability evaluation in conjunction with SIGMOD 2010 continues along the lines of the 2009 edition, with some improvements related to the procedure.

Contents

The Goal

On a voluntary basis, authors of accepted SIGMOD 2010 papers can provide their code/binaries, experimental setups and data to be tested for

by a repeatability & workability committee (RWC) under the responsibility of the repeatability & workability officers (RWO).

The People

The RWO are Ioana Manolescu and Stefan Manegold.
The RWC are

Instruction for Authors

Detailed instrcutions for authors as sent out by email are also available here.

The Procedure

Authors of accepted papers will be contacted by e-mail as soon as acceptance is determined and invited to submit

Submissions will be handled using a Repeatability/Workability Conference Management Tool (RWCMT, in short).

Past experience has demonstrated that repeatability and workability can greatly benefit from the availability of authors to interact with the repeatability reviewer, and help solve minor issues related to the installation, configuration, and usage of the code. Thus, the electronic tool will enable repeated interactions between the reviewers and the authors in the style of a message board. Thus, the 2010 process will involve such interaction, too.

The RWO will designate for each submission:

The first and second reviewer will interact until they are both satisfied with the terms of the report. They will both sign the report. If there is disagreement that the reviewers cannot work out, the RWO have the final say. They may propose alternative wording for the report, more tests, and/or endorse responsibility together with one reviewer, if the other cannot agree with the chosen wording (and thus is unwilling to sign it).

During the evaluation, the first reviewer and the authors interact via the RWCMT. The second reviewer and the repeatability & workability co-chairs may also participate to the discussion. The recommendation is that the first reviewer is left alone with the authors during the first 2/3rds of the reviewing period, to avoid confusion. The RWCMT documents all interaction between the reviewers and the authors.

The identity of the reviewers is hidden during the evaluation process, but obviously will be revealed afterwards when the reviewers sign their report.

The Results

21 of the 80 accepted papers of SIGMOD 2010 participated in the repeatability & workability evaluation. The results of the 18 papers who's authors agreed on publication are available below.

RWE
ID
Title (link to paper) Authors Reviews (link to details) Results
Primary Secondary Repeatability Workability
07Towards Proximity Pattern Mining in Large GraphsArijit Khan, Xifeng Yan, Kun-Lung Wu Fabian Hüske Zhenjie Zhang Mostly confirmed Mostly confirmed
08Histograms Reloaded: The Merits of Bucket DiversityCarl-Christian Kanne, Guido Moerkotte Milena Ivanova Zhenjie Zhang Fully confirmed Partly confirmed
10GAIA: Graph Classification Using Evolutionary ComputationNing Jin, Calvin Young, Wei Wang Stratos Idreos Zhenjie Zhang Fully confirmed (not checked)
11Non-homogeneous Generalization in Privacy Preserving Data PublishingWai Kit Wong, Nikos Mamoulis, David Wai Lok Cheung Jong Wook Kim Frank Huber Fully confirmed Mostly confirmed
12Bed-Tree: An All-Purpose Index Structure for String Similarity Search Based on Edit DistanceZhenjie Zhang, Marios Hadjieleftheriou, Beng Chin Ooi, Divesh Srivastava Dan Olteanu Frank Huber Mostly confirmed Fully confirmed
13Continuous Sampling for Online Aggregation Over Multiple QueriesSai Wu, Beng Chin Ooi, Kian-Lee Tan Stefan Manegold Paolo Papotti Mostly confirmed (not checked)
15Durable Top-k Search in Document ArchivesLeong Hou U, Nikos Mamoulis, Klaus Berberich, Srikanta Bedathur Frank Huber Fabian Hüske Fully confirmed Fully confirmed
17Efficient Parallel Set-Similarity Joins Using MapReduceRares Vernica, Michael Carey, Chen Li Fabian Hüske Wai Kit Wong Fully confirmed Partly confirmed
18Sampling Dirty Data for Matching AttributesHenning Koehler, Xiaofang Zhou, Shazia Sadiq, Yanfeng Shu, Kerry Taylor Wai Kit Wong Zhenjie Zhang Fully confirmed Fully confirmed
20Connected Substructure Similarity SearchHaichuan Shang, Xuemin Lin, Ying Zhang, Jeffrey Yu, Wei Wang Frank Huber Ryan Johnson Fully confirmed Mostly confirmed
21ParaTimer: A Progress Indicator for MapReduce DAGsKristi Morton, Magdalena Balazinska, Dan Grossman Ryan Johnson Wai Kit Wong Fully confirmed Fully confirmed
22Positional Update Handling in Column StoresSandor Heman, Marcin Zukowski, Niels Nes, Lefteris Sidirourgos, Peter Boncz Stefan Manegold Dan Olteanu Mainly confirmed Partly confirmed
23Low Overhead Concurrency Control in Partitioned DBMSsEvan Jones, Daniel Abadi, Samuel Madden Asterios Katsifodimos Jong Wook Kim Fully confirmed Fully confirmed
24Load-Balanced Query Dissemination in Privacy-Aware Online CommunitiesEmiran Curtmola, Alin Deutsch, K.K. Ramakrishnan, Divesh Srivastava Paolo Papotti Jong Wook Kim Mostly confirmed Mostly confirmed
25Hierarchically Organized Skew-Tolerant Histograms for Geographic Data ObjectsYohan Roh, Jae Ho Kim, Yon Dohn Chung, Jin Hyun Son, Myoung Ho Kim Wai Kit Wong Asterios Katsifodimos Fully confirmed Fully confirmed
26Active Knowledge : Dynamically Enriching RDF Knowledge Bases by Web ServicesNicoleta Preda, Fabian Suchanek, Gjergji Kasneci, Thomas Neumann, Wenjun Yuan Dan Olteanu Asterios Katsifodimos Mostly confirmed Partly confirmed
27Automatic Contention Detection and Amelioration for Data-Intensive OperationsJohn Cieslewicz, Kenneth Ross, Kyoho Satsumi, Yang Ye Ryan Johnson Milena Ivanova Fully confirmed Fully confirmed
28Data Conflict Resolution Using Trust MappingsWolfgang Gatterbauer, Dan Suciu Paolo Papotti Milena Ivanova Partly confirmed Partly confirmed


Maintained by Stefan Manegold RWE 2008 2009 2010 hosted by CWI