Import the csv data into a table in a database on your machine. You can use SQL, workbench or python to achieve this. Show the process you did to achieveDatabase

Description

You have been supplied with the datafile all_weather_data.csv which contains data from multiple weather stations. This data has been collected from Mesonet.

The locations that have been collected can be seen below.

The dataset needs to be analyzed using PCA, but it contains missing values and anomalies that must be removed before the analysis can be performed. It is your task to clean the dataset into a form that can be fed to PCA for analysis. After that, you will be doing a PCA analysis on the dataset.

Part 1: Data processing

  1. Import the csv data into a table in a database on your machine. You can use SQL, workbench or python to achieve this. Show the process you did to achieve

  1. Plot a timeseries plot of selected variables for station FGBT and FMMI. For each station plot heat_index_set_1d together with air_temp_set_1 in one plot and wind_speed_set_1 together with wind_gust_set_1 on another. All in all, provide 4 plot images (2 for each station). Below is an example for the station

  1. Create a query that gives the name of all stations that have at least one row of values for heat_index_set_1d

  1. Find columns that have more that have at least 90% values (not having more than 5% NULL values). The columns that have less then 10% missing values are to be used for analysis. You can choose to use SQL or Python. Show how you proceed and list the columns that have too many missing

  1. Create a view that shows only the columns that have more than 90% non-null values. This view should also remove all rows that have any null values. Provide the SQL code to create this

  1. The data we have in its current for is not normalized and therefore contains a lot of redundant data. Create a normalized model. Provide an image of the model and an explanation for which variables broke what normalization

Part 2: PCA analysis

Once you have your cleaned data file, carry out a PCA on the data matrix. All the weather measurements will be your observations, all the weather properties (temperature, dew point etc.) will be your  variables. The station IDs will be your labels.

Carry out the PCA using an existing PCA algorithm in Python, or write your own. With the hoggorm algorithm you can easily plot the scores and loadings https://hoggorm.readthedocs.io/en/latest/

With the scikit-learn you can plot the observations of the different station IDs with different colours.

https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html

You can see an example of how to apply the PCA algorithms in the python script in the lecture notes.

Make a figure showing a score plot and a loadings plot with PC1 and PC2.

Based on these plots, discuss shortly (2-5 lines) about the weather at different geographical locations.

Attachments:

Instructions Files

Database Experts

expert
Anuja Sharma
Database

30 Answers

View More Experts
Disclaimer

The ready solutions purchased from Library are already used solutions. Please do not submit them directly as it may lead to plagiarism. Once paid, the solution file download link will be sent to your provided email. Please either use them for learning purpose or re-write them in your own language. In case if you haven't get the email, do let us know via chat support.

Get Free Quote!

268 Experts Online