COVID-19-Spark-Big-Data-Analysis

COVID-19 analysis of total/mean number of vaccinations in Italy divided by regions using Apache Spark in Google Colab.

This repository includes three simple queries needed to analyze data for a research project. Moreover, it further includes their respective commands needed to initialize the HDFS, run the script, and save the output for Linux. The code is written utilizing Pyspark and few other libraries and it's conceived to get confident with RDDs and the MapReduce paradigm designed to compute parallel distributed data processing.

All datasets come from https://github.com/italia/covid19-opendata-vaccini/tree/master/dati and the needed ones are the following:

• https://github.com/italia/covid19-opendata-vaccini/blob/master/dati/punti-somministrazione-tipologia.csv
• https://github.com/italia/covid19-opendata-vaccini/blob/master/dati/somministrazioni-vaccini-latest.csv
• https://github.com/italia/covid19-opendata-vaccini/blob/master/dati/somministrazioni-vaccini-summary-latest.csv

Do not remove this line from the start of the script if using a Linux system -> # coding: utf-8

Helpful commands:

• data.collect()
• data.take(n° of rows)