README.md

Coursework for MIDS Scaling Up! Really Big Data

This is an index of coursework for the MIDS class "Scaling Up! Really Big Data". Please submit corrections if you find problems in the assignments. Submissions should be well-formed git pull requests.

Week 2: Cloud Computing 101

Labs

0. Salt States and Docker deployment of the ELK stack

Week 3: Openstack Introduction

Labs

0. Hadoop over OpenStack DevStack using Sahara

Week 4: Distributed Filesystems

Homework

This is a graded homework

0. Part 1- GPFS setup
1. Part 2- The Mumbler

Labs

There will be no in-class lab for this assignment

Week 5: Distributed Filesystems

Homework

0. Part 1- Hadoop v1 Setup
1. Part 2- Hadoop v2 Setup

Labs

(Complete the following in order)

0. Load Google 2-gram dataset into HDFS
1. Preprocess 2-gram data for Mumbler

Week 6: Apache Spark

Homework

0. Apache Spark Introduction

Labs

0. Machine Learning with Spark and MLLib

Week 7: Object Storage

Homework

0. Object Storage

Labs

(Complete the following in order)

0. Data Transfer Performance
1. Rsync Investigation

Week 8: NoSQL

Homework

0. NoSQL

Week 9: Spark Streaming

Homework

0. Streaming Tweet Processing

Labs

0. Spark Streaming and Cassandra

Week 10: Scaling Up

Homework

0. Orchestrate with Brooklyn

Labs

0. Brooklyn labs

Week 11: Spark ML Round 2

(Homework-free week!)

Labs

0. Streaming Analytics with AlchemyAPI

Week 12: Search

Homework

0. Elasticsearch