Running relational queries on Hadoop-based systems is an attempt of bringing the known relational model into the massively parallel world. However, the performance of these systems varies widely. Your task is to perform a comparision between two systems in this space, Hive and Spark SQL (formerly "Shark"). The standard relational TPC-DS benchmark queries and data will be used.

• Run the TPC-DS benchmark queries using Hive
• Run the TPC-DS benchmark queries using Spark SQL
• Compare performance and visualize in plots
• Stretch goals: repeat with different dataset sizes, test different storage formats (e.g. Parquet)

This analysis will be using Hive and Spark SQL.

The data will be available on HDFS at /user/hannesm/lsde/tpcds .

• Shark: SQL and Rich Analytics at Scale
• Hive – A Petabyte Scale Data Warehouse Using Hadoop