For this assignment the Cloudera Virtual Machine provided for this module must be used. All processing
INSTRUCTIONS TO CANDIDATES
ANSWER ALL QUESTIONS
Equipment and Facilities to be Used
For this assignment the Cloudera Virtual Machine provided for this module must be used. All processing
must be done via scripts and code, and these must be stored and included in submission. Terminal
commands must be stored in shell scripts, language specific code has to be stored in separate files (for
example, SQL code must be stored in SQL scripts and python code must be stored in Python scripts).
The solution has to be implemented using both SQL and Python.
Workload
For the successful completion of this assignment, a total of 80 hours should be budgeted.
Task
You will be given a dataset and a set of problem statements. Where possible (you will need to carefully
explain any reasons for not supplying both solutions), you are required implement the solution in both
SQL (using either Hive or Impala), and Spark (using pyspark or spark-shell).
General instructions
You will follow a typical data analysis process:
1. Load / ingest the data to be analysed
2. Prepare / clean the data
3. Analyse the data
4. Visualise results / generate report
For steps 1, 2 and 3 you will use the virtual machine (and the software installed on it) that has been
provided as part of this module. The data necessary for this assignment will be provided in a MySQL
dump format which you will need to copy onto the virtual machine and start working with it from there.
The virtual machine has a MySQL server running and you will need to load the data into the MySQL
server. From there you will be required to use Sqoop to get the data into Hadoop.
For the cleansing, preparation and analysis you will implement the solution twice (where possible). First
in SQL using either Hive or Impala and then in Spark using either pyspark or spark-shell.
For the visualisation of the results you are free to use any tool that fulfils the requirements, which can be
tools you have learned about such as Python’s matplotlib, SAS or Qlik, or any other free open source tool
you may find suitable.
5. As well as the official hashtags, each tweet may be labelled with other hashtags. Restricting the data
to ‘during-game’ tweets, list the top 10 most common non official hashtags over the whole dataset
with their frequencies.
6. Draw the graph of the progress of one of the games (the game you choose should have a complete set
of tweets for the entire duration of the game). It may be useful to summarize the tweet frequencies
in 1 minute intervals.
(it's on mysql only) creating an email marketing I believe it's easy.
Related Questions
. Introgramming & Unix Fall 2018, CRN 44882, Oakland University Homework Assignment 6 - Using Arrays and Functions in C
DescriptionIn this final assignment, the students will demonstrate their ability to apply two ma
. The standard path finding involves finding the (shortest) path from an origin to a destination, typically on a map. This is an
Path finding involves finding a path from A to B. Typically we want the path to have certain properties,such as being the shortest or to avoid going t
. Develop a program to emulate a purchase transaction at a retail store. This program will have two classes, a LineItem class and a Transaction class. The LineItem class will represent an individual
Develop a program to emulate a purchase transaction at a retail store. Thisprogram will have two classes, a LineItem class and a Transaction class. Th
. SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of Sea Ports. Here are the classes and their instance variables we wish to define:
1
Project 1
Introduction - the SeaPort Project series
For this set of projects for the course, we wish to simulate some of the aspects of a number of
. Project 2 Introduction - the SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of Sea Ports. Here are the classes and their instance variables we wish to define:
1
Project 2
Introduction - the SeaPort Project series
For this set of projects for the course, we wish to simulate some of the aspects of a number of
