Skip to content

Latest commit

 

History

History
70 lines (45 loc) · 2.68 KB

chapter_2.md

File metadata and controls

70 lines (45 loc) · 2.68 KB
Raw

Chapter 2: Getting Started with Spark

Installing and Configuring Apache Spark

Before diving into the practical aspects of Spark, it's essential to have Spark installed on your machine. You can download Spark from the official Apache Spark website. Choose the version compatible with your system and follow the installation guide for your operating system.

After downloading and unzipping Spark, you might want to add Spark’s bin directory to your PATH to easily execute Spark’s command-line utilities. Additionally, ensure you have Java installed on your system, as it is required to run Spark.

Starting a Spark Session

A SparkSession is the entry point to Spark's DataFrame and SQL functionality. Here's how to start a Spark session in Scala and Python:

Scala Example

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .appName("Spark Session Example")
  .config("spark.master", "local")
  .getOrCreate()

// Importing SparkSession.implicits is a common practice for implicit conversions like converting RDDs to DataFrames
import spark.implicits._

Python (PySpark) Example

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("Spark Session Example") \
    .config("spark.master", "local") \
    .getOrCreate()

In these examples, .appName() sets the name of the application, which will appear in the Spark web UI. .config("spark.master", "local") tells Spark to run locally on your machine using all available cores.

Creating Spark DataFrames

DataFrames can be created from various data sources, including existing RDDs, structured data files, and external databases. Let's look at how to create a DataFrame from a collection of data.

Scala Example

// Create a DataFrame from a collection of tuples
val data = Seq(("James", 34), ("Anna", 20), ("John", 50))
val df = data.toDF("Name", "Age")

df.show()

Python (PySpark) Example

# Create a DataFrame from a list of tuples
data = [("James", 34), ("Anna", 20), ("John", 50)]
df = spark.createDataFrame(data, ["Name", "Age"])

df.show()

In these examples, we create a DataFrame from a collection of tuples, where each tuple represents a row in the DataFrame. The second argument to toDF (Scala) or createDataFrame (Python) specifies the column names.

With the Spark session setup and a basic understanding of creating DataFrames, you're well-equipped to start exploring and manipulating large datasets with Spark.

This chapter provides the essentials to get up and running with Spark, setting the foundation for more complex data processing and analysis tasks in subsequent chapters.