RealNest - Nested Data from Real-World Datasets

• 64 * 1024 row version: tables_65536.tar
• 10 * 64 * 1024 row version: tables_655360.tar

This readme is adapted from the RealNest repository. Visit the GitHub repository for the latest updates.

This website contains the static version of the RealNest dataset, a collection of nested data derived from real-world datasets. The dataset is designed to help computer science researchers benchmark and evaluate data systems and data formats supporting nested data types.

RealNest is provided as a script that downloads and generates the data, but for convenience and to facilitate standardized comparisons, we host on this website two static datasets in sizes of 64 * 1024 resp. 10 * 64 * 1024 rows. Each dataset is a .tar file containing a folder for each table. These sample datasets were downloaded and generated by our script in mid-May 2024.

Because we provide the script that downloads the original datasets and processes them into a common format, one can create the dataset from newer versions of the underlying data and also enlarge them with respect to the static datasets, since even the larger of the two statically downloadable datasets contains only a small part of each of the original data sources. This script is available at the RealNest GitHub repository

Note that the static datasets hosted here remain under the same licenses and terms of use as the original datasets they are generated from. If you are the owner of an original dataset, and object to the inclusion of your data in the RealNest static datasets hosted here, please contact Peter Boncz (boncz@cwi.nl), and we will take action.

Please note that below we attempt to properly attribute the individual datasets as required by their various open-source licenses and terms of usage.

Dataset Structure

The dataset contains a directory for each table with the following files:

• schema.json: The schema of the table. The schema is a JSON object with a single key, columns, containing a list of columns. Each column is a JSON object with 2 or 3 keys:
  • name - The name of the column as a string.
  • type - The type of the column as a string.
  • children - Optional, only exists for nested types (list, struct, map). Describes the child types of the nested type as a list of column objects. The list type always has a single child column with the name child. The map type always has two child columns with the names key and value.
• data.jsonl.gz: The data of the table in Gzip compressed JSON Lines format

The schema might contain a JSON type, which may happen for empty JSON objects in the data ({}) or when DuckDB's schema inference detects incompatible types. The columns of this type can be ignored since they are not typical for structured data, or they can be handled as VARCHAR columns, where the value is the JSON string.

Attribution

The data has been downloaded from various public sources and converted to a common format. We note that the real-world datasets from which RealNest is derived are released under varying open-source licenses and terms of usage.

The sources of the original datasets are:

1. Amazon Berkeley Objects (LICENSE)
   • J. Collins, S. Goel, K. Deng, A. Luthra, L. Xu, E. Gundogdu, X. Zhang, T. F. Yago Vicente, T. Dideriksen, H. Arora, M. Guillaumin, and J. Malik, "Abo: Dataset and benchmarks for real-world 3d object understanding," CVPR, 2022.
2. AWS Public Blockchain Data (LICENSE)
3. Data Lake as Code (ATTRIBUTIONS)
4. CORD-19 (LICENSE)
   • L. L. Wang, K. Lo, Y. Chandrasekhar, R. Reas, J. Yang, D. Eide, K. Funk, R. M. Kinney, Z. Liu, W. Merrill, P. Mooney, D. A. Murdick, D. Rishi, J. Sheehan, Z. Shen, B. Stilson, A. D. Wade, K. Wang, C. Wilhelm, B. Xie, D. A. Raymond, D. S. Weld, O. Etzioni, and S. Kohlmeier, "Cord-19: The covid-19 open research dataset," ArXiv, 2020.
5. Daylight Map Distribution of OpenStreetMap (Open Database License (ODbL))
6. GitHub Archive
7. CERN Open Data
   • CMS collaboration (2017). SingleMu primary dataset in AOD format from Run of 2012 ( /SingleMu/Run2012B-22Jan2013-v1/AOD). CERN Open Data Portal. DOI:10.7483/OPENDATA.CMS.IYVQ.1J0W
8. Overture Maps Foundation Open Map Data
   • Overture data is licensed under the Community Database License Agreement Permissive v2 (CDLA) unless derived from a source that requires publishing under a different license, such as data derived from OpenStreetMap, that constitutes a 'Derivative Database' (as defined under ODbL v1.0), which will be licensed under ODbL v1.0.
9. Twitter Stream Archive

By using the data from this website, you agree to the terms of use of the original data sources. The data source for each table is given below (Data source numbers refer to the list above):