logo Use CA10RAM to get 10%* Discount.
Order Nowlogo

For this assignment the Cloudera Virtual Machine provided for this module must be used. All processing

INSTRUCTIONS TO CANDIDATES
ANSWER ALL QUESTIONS
Equipment and Facilities to be Used
 
For this assignment the Cloudera Virtual Machine provided for this module must be used. All processing 
must be done via scripts and code, and these must be stored and included in submission. Terminal 
commands must be stored in shell scripts, language specific code has to be stored in separate files (for 
example, SQL code must be stored in SQL scripts and python code must be stored in Python scripts). 
The solution has to be implemented using both SQL and Python. 
Workload 
For the successful completion of this assignment, a total of 80 hours should be budgeted. 
Task 
You will be given a dataset and a set of problem statements. Where possible (you will need to carefully 
explain any reasons for not supplying both solutions), you are required implement the solution in both 
SQL (using either Hive or Impala), and Spark (using pyspark or spark-shell). 
General instructions 
You will follow a typical data analysis process: 
1. Load / ingest the data to be analysed 
2. Prepare / clean the data 
3. Analyse the data 
4. Visualise results / generate report 
For steps 1, 2 and 3 you will use the virtual machine (and the software installed on it) that has been 
provided as part of this module. The data necessary for this assignment will be provided in a MySQL 
dump format which you will need to copy onto the virtual machine and start working with it from there. 
The virtual machine has a MySQL server running and you will need to load the data into the MySQL 
server. From there you will be required to use Sqoop to get the data into Hadoop. 
For the cleansing, preparation and analysis you will implement the solution twice (where possible). First 
in SQL using either Hive or Impala and then in Spark using either pyspark or spark-shell. 
For the visualisation of the results you are free to use any tool that fulfils the requirements, which can be 
tools you have learned about such as Python’s matplotlib, SAS or Qlik, or any other free open source tool 
you may find suitable.

 

 
5. As well as the official hashtags, each tweet may be labelled with other hashtags. Restricting the data 
to ‘during-game’ tweets, list the top 10 most common non official hashtags over the whole dataset 
with their frequencies. 
 
6. Draw the graph of the progress of one of the games (the game you choose should have a complete set 
of tweets for the entire duration of the game). It may be useful to summarize the tweet frequencies 
in 1 minute intervals.
 
 
 

(it's on mysql only) creating an email marketing I believe it's easy.

Related Questions

. Introgramming & Unix Fall 2018, CRN 44882, Oakland University Homework Assignment 6 - Using Arrays and Functions in C

DescriptionIn this final assignment, the students will demonstrate their ability to apply two ma

. The standard path finding involves finding the (shortest) path from an origin to a destination, typically on a map. This is an

Path finding involves finding a path from A to B. Typically we want the path to have certain properties,such as being the shortest or to avoid going t

. Develop a program to emulate a purchase transaction at a retail store. This program will have two classes, a LineItem class and a Transaction class. The LineItem class will represent an individual

Develop a program to emulate a purchase transaction at a retail store. Thisprogram will have two classes, a LineItem class and a Transaction class. Th

. SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of Sea Ports. Here are the classes and their instance variables we wish to define:

1 Project 1 Introduction - the SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of

. Project 2 Introduction - the SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of Sea Ports. Here are the classes and their instance variables we wish to define:

1 Project 2 Introduction - the SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of

Ask This Question To Be Solved By Our ExpertsGet A+ Grade Solution Guaranteed

expert
Um e HaniScience

573 Answers

Hire Me
expert
Muhammad Ali HaiderFinance

635 Answers

Hire Me
expert
Husnain SaeedComputer science

935 Answers

Hire Me
expert
Atharva PatilComputer science

593 Answers

Hire Me

Get Free Quote!

407 Experts Online