Use Spark to process and perform basic analysis on non-relational data, including its DataFrame and SQL interfaces.

INSTRUCTIONS TO CANDIDATES

ANSWER ALL QUESTIONS

Objectives: Use Spark to process and perform basic analysis on non-relational data, including its DataFrame and SQL interfaces.

Grading criteria: The tasks should all be completed, and questions should all be answered with Python code, SQL queries, shell commands, and markdown cells. The notebook itself should be completely reproducible (using AWS EC2 instance based on the provided AMI) from start to finish; another person should be able to use the code to obtain the same results as yours. Note that you will receive no more than partial credit if you do not add text/markdown cells explaining your thinking when appropriate.

Part 1 - Setup

Begin by setting up Spark and fetching the project data.

Note: you may want to use a larger EC2 instance type than normal. This project was prepared using a t2.xlarge instance. Just remember that the larger the instance, the higher the per-hour charge, so be sure to remember to shut your instance down when you're done, as always.

About the data

We will use JSON data from Twitter; we saw an example of this in class. It should parse cleanly, allowing you to focus on analysis.

This data was gathered using GWU Libraries' Social Feed Manager application during a recent game of the MLB World Series featuring the Los Angeles Dodgers and Houston Astros. This first file tells you a little bit about how it was gathered:

First make sure you are working from the right working directory

!pwd

This below file provides context and detail information on the files we are going to work with in this assignment

!wget https://s3.amazonaws.com/2017-dmfa/project-

3/9670f3399f774789b7c3e18975d25611-README.txt

!cat 9670f3399f774789b7c3e18975d25611-README.txt

The most important pieces in that metadata are:

• It tracked tweets that mentioned "dodgers" or "astros". Every item in this set should refer to one or the other, or both.

• This data was not deduplicated; we may see individual items more than once.

 

• Data was collected between October 29 and October 30. Game 5 of the Series was played during this time.

You should not need to know anything about baseball to complete this assignment.

Please note: sometimes social media data contains offensive material. This data set has not been filtered; if you do come across something inappropriate, please do your best to ignore it if you can.

Fetch the data

The following files are available:

• https://s3.amazonaws.com/2017-dmfa/project- 3/9670f3399f774789b7c3e18975d25611_003.json

• https://s3.amazonaws.com/2017-dmfa/project- 3/9670f3399f774789b7c3e18975d25611_004.json

• https://s3.amazonaws.com/2017-dmfa/project- 3/9670f3399f774789b7c3e18975d25611_005.json

• https://s3.amazonaws.com/2017-dmfa/project- 3/9670f3399f774789b7c3e18975d25611_006.json

Q1.1 - Upload the above files to your instance using wget. Verify the file sizes using the command line.

Each file should contain exactly 100,000 tweets.

Note: you are required to use all files. It will be easier to process more data if you use a larger EC2 instance type, as suggested above. Use the exact same set of files throughout the assignment.

 

Attachments:

Related Questions

. Introgramming & Unix Fall 2018, CRN 44882, Oakland University Homework Assignment 6 - Using Arrays and Functions in C

DescriptionIn this final assignment, the students will demonstrate their ability to apply two ma

. The standard path finding involves finding the (shortest) path from an origin to a destination, typically on a map. This is an

Path finding involves finding a path from A to B. Typically we want the path to have certain properties,such as being the shortest or to avoid going t

. Develop a program to emulate a purchase transaction at a retail store. This program will have two classes, a LineItem class and a Transaction class. The LineItem class will represent an individual

Develop a program to emulate a purchase transaction at a retail store. Thisprogram will have two classes, a LineItem class and a Transaction class. Th

. SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of Sea Ports. Here are the classes and their instance variables we wish to define:

1 Project 1 Introduction - the SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of

. Project 2 Introduction - the SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of Sea Ports. Here are the classes and their instance variables we wish to define:

1 Project 2 Introduction - the SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of