logo Use CA10RAM to get 10%* Discount.
Order Nowlogo
(5/5)

Everything is herehttps://colab.research.google.com/drive/1gr3cP2BYJz7Wtt9mNEH5FO0VTGnwzzQz?usp=sha

INSTRUCTIONS TO CANDIDATES
ANSWER ALL QUESTIONS

 You are required to turn in this completed notebook and a Python's script that runs on the Hadoop cluster.

Please read the whole instruction

.You will need to complete Task2 from C to L. Where it says [TODO] and TO_BE_COMPLETED fill all the code snippet in each cell from C to L where it says TO_BE_COMPLETED so that it gives the expected output as shows after each cell from C to L. You might not see the exact number as it shown in the expected output but the format and structure has to be same. I have provided 2 data file to run all the codes and you can download those running cell block B. at the end of the Task 2 it will generate 9 output csv files. Plot those csv files using the code provided on the cell "Line Plot". Your 9 output files should generate 9 plots showing at the end of the notebook. Your plot should be similar the one is shown at the end.

Please use apache Spark RDD using Python.Most of the codes are provided you will need to complete where it says TO_BE_COMPLETED.This is a big data analysis problem and try to optimize your code such a way so that when i will take all your scripts and run it on a server using 10GB of csv file it should run in 2 minutes.

We still use the Safegraph data to better understand how NYC response to the COVID-19 pandemic. If you have any doubts about the data, please consult SafeGraph's documentation for Places Schema and Weekly Pattern.

Problem Description

To assess the food access problem in NYC before and during the COVID-19 pandemic, we would like to plot the visit patterns for all food stores (including restaurants, groceries, deli's, etc.) such as the one shown below.


 
 
[ ]
 

A visualization of Visit Pattern for all food stores in NYC

The solid lines show the median patterns and the transparent areas show the standard deviation across stores for 2019 and 2020.

 
 

 
 
 
 

However, we suspect that the visit patterns may vary across different type of stores. Our hypothesis is that we have changed our shopping behavior during the pandemic. For example, we visit Fast Food and Whole Saler restaurants more often comparing to full service restaurants and typical supermarkets. In particular, we are interested in the following store categories with their NAICS codes:

  • Big Box Grocers452210 and 452311
  • Convenience Stores445120
  • Drinking Places722410
  • Full-Service Restaurants722511
  • Limited-Service Restaurants722513
  • Pharmacies and Drug Stores446110 and 446191
  • Snack and Bakeries311811 and 722515
  • Specialty Food Stores445210445220445230445291445292, and 445299
  • Supermarkets (except Convenience Stores)445110

 
 
[ ]
 

The plot above was created by the linePlot() function (defined later), which takes a Panda's DataFrame consisting of 5 columns as follows.

 
 

 
 
 
 
  • year: column is used for showing the trend line category (orange or blue).
  • date: denotes the day of the year for each data point, for which we project to to year 2020. We chose 2020 as the base year because it is a leap year and would have all possile dates (i.e. month + day combination). The actual date for the data point would be month and day from date combined with the year in year.
  • median: is used to draw the solid line describing the median visit counts across all stores for that date.
  • low: the lower bound of the "confidence interval". In our plot, it is the median minus the standard deviation but will be kept at 0 or above.
  • high: the higher bound of the "confidence interval". In our plot, it is the median plus the standard deviation but will be kept at 0 or above.

NOTES

  • low and high value will be used to create the transparent area that we see in the plot.
  • lowmedianhigh should be computed not only for stores that had visits but also for all stores in Core Places that fit the category. As we learned previously, restaurants with no visits will not be reported in the Weekly Pattern data set.
(5/5)
Attachments:

Related Questions

. Introgramming & Unix Fall 2018, CRN 44882, Oakland University Homework Assignment 6 - Using Arrays and Functions in C

DescriptionIn this final assignment, the students will demonstrate their ability to apply two ma

. The standard path finding involves finding the (shortest) path from an origin to a destination, typically on a map. This is an

Path finding involves finding a path from A to B. Typically we want the path to have certain properties,such as being the shortest or to avoid going t

. Develop a program to emulate a purchase transaction at a retail store. This program will have two classes, a LineItem class and a Transaction class. The LineItem class will represent an individual

Develop a program to emulate a purchase transaction at a retail store. Thisprogram will have two classes, a LineItem class and a Transaction class. Th

. SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of Sea Ports. Here are the classes and their instance variables we wish to define:

1 Project 1 Introduction - the SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of

. Project 2 Introduction - the SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of Sea Ports. Here are the classes and their instance variables we wish to define:

1 Project 2 Introduction - the SeaPort Project series For this set of projects for the course, we wish to simulate some of the aspects of a number of

Ask This Question To Be Solved By Our ExpertsGet A+ Grade Solution Guaranteed

expert
Um e HaniScience

978 Answers

Hire Me
expert
Muhammad Ali HaiderFinance

538 Answers

Hire Me
expert
Husnain SaeedComputer science

812 Answers

Hire Me
expert
Atharva PatilComputer science

748 Answers

Hire Me
June
January
February
March
April
May
June
July
August
September
October
November
December
2025
1950
1951
1952
1953
1954
1955
1956
1957
1958
1959
1960
1961
1962
1963
1964
1965
1966
1967
1968
1969
1970
1971
1972
1973
1974
1975
1976
1977
1978
1979
1980
1981
1982
1983
1984
1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
2027
2028
2029
2030
2031
2032
2033
2034
2035
2036
2037
2038
2039
2040
2041
2042
2043
2044
2045
2046
2047
2048
2049
2050
SunMonTueWedThuFriSat
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
1
2
3
4
5
00:00
00:30
01:00
01:30
02:00
02:30
03:00
03:30
04:00
04:30
05:00
05:30
06:00
06:30
07:00
07:30
08:00
08:30
09:00
09:30
10:00
10:30
11:00
11:30
12:00
12:30
13:00
13:30
14:00
14:30
15:00
15:30
16:00
16:30
17:00
17:30
18:00
18:30
19:00
19:30
20:00
20:30
21:00
21:30
22:00
22:30
23:00
23:30