Interest in topics is often said to only be influenced by mass media. We should investigate this. Wikipedia publishes hourly click figures for each Wikipedia article. Also, we have a list of all posts on the Volkskrant website in 2014. Your task is to correlate the reporting intensity of a topic with the interest in this topic on Wikipedia for the Dutch Wikipedia.

• Process the 2014 Wikipedia click data to generate daily click figures on articles in the Dutch Wikipedia
• Match newspaper articles to Wikipedia pages (e.g. using a list of pages/redirects)
• Test whether there is a correlation between a topic being reported on and clicks on the page of that topic on Wikipedia
• Visualize found correlations as timelines
• Stretch goals: IR preprocessing of newspaper articles, context based article matching, more sophisticated topic detection

This analysis will be using Apache Spark.

The data is available in text format. The data will be available on HDFS at /user/hannesm/lsde/news . Examples below:

Newspaper archive (date, url, title, text)

2014-01-05  "http://www.volkskrant.nl/binnenland/medewerkers-nrc-willen-flinke-bonus-egeria~a3849218/"  "Medewerkers NRC willen flinke bonus Egeria"    "De ondernemingsraad van NRC Media eist van eigenaar Egeria een stevige bonus voor de 350 medewerkers, nu de uitgever ... "
2014-01-05  "http://www.volkskrant.nl/buitenland/leiders-sharia4belgium-voor-jaren-de-cel-in~a3849141/" "Leiders Sharia4Belgium voor jaren de cel in"   "De rechtbank in Antwerpen heeft vandaag zware celstraffen opgelegd aan de leiders van de opgedoekte radicale moslimbeweging Sharia4Belgium. ..."
2014-01-05  "http://www.volkskrant.nl/buitenland/dag-van-de-waarheid-voor-europa~a3849112/" "Dag van de waarheid voor Europa"   "De spanning rond de Griekse schuldenlast loopt op in de EU en in Minsk wordt getracht tot een akkoord te komen over de crisis in de Oekraïne. ..."

Wikipedia click data (subdomain, page, clicks, size)

nl 1_World_Trade_Center 1 36727
nl 1_augustus 1 30309
nl 1_december 1 31844
nl 1_en_2_Samuel 2 50692
nl 1_februari 2 150123
nl 1_januari 6 272496
nl 1_juli 1 33273
nl 1_maart 1 113667

• Meme-tracking and the Dynamics of the News Cycle
• Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing