manutej avatar

apache-spark-data-processing

Complete guide for Apache Spark data processing including RDDs, DataFrames, Spark SQL, streaming, ML

by manutej|Open Source

Apache Spark Data Processing Skill

Master Apache Spark for distributed data processing, streaming analytics, and machine learning at scale.

Overview

Apache Spark is a unified analytics engine for large-scale data processing, offering high-level APIs in Java, Scala, Python, and R. This skill provides comprehensive guidance for building production-ready Spark applications across batch processing, real-time streaming, SQL analytics, and machine learning workflows.

Key Capabilities:

  • Process petabyte-scale datasets with distributed computing
  • Real-time stream processing with sub-second latency
  • Interactive SQL queries on structured and semi-structured data
  • Scalable machine learning with MLlib
  • Unified API for batch and streaming workloads

What You'll Learn

Core Data Processing

  • RDDs (Resilient Distributed Datasets): Low-level distributed data abstraction with fault tolerance
  • DataFrames & Datasets: Structured data processing with automatic query optimization
  • Transformations & Actions: Lazy evaluation patterns for efficient computation
  • Partitioning: Data distribution strategies for optimal parallelism

Spark SQL

  • DataFrame API: Declarative data manipulation with type safety
  • SQL Queries: Execute ANSI SQL on distributed datasets
  • Data Sources: Read/write Parquet, ORC, JSON, CSV, JDBC, Hive
  • Query Optimization: Catalyst optimizer and Tungsten execution engine
  • Window Functions: Advanced analytics with ranking, aggregations, and offsets

Streaming Processing

  • Structured Streaming: Unified batch and streaming API
  • Stream Sources: Kafka, files, sockets, and custom sources
  • Windowing: Tumbling, sliding, and session windows
  • Watermarking: Handle late-arriving data with configurable tolerance
  • Stateful Processing: Maintain state across micro-batches
  • Stream-Static Joins: Enrich streaming data with reference tables

Machine Learning (MLlib)

  • ML Pipelines: Chain transformations, feature engineering, and models
  • Classification & Regression: Logistic regression, random forests, gradient boosting
  • Clustering: K-means, Gaussian mixture models
  • Dimensionality Reduction: PCA, SVD
  • Feature Engineering: Encoders, scalers, assemblers
  • Model Selection: Cross-validation and hyperparameter tuning
  • Streaming ML: Train models on continuous data streams

Performance Optimization

  • Caching & Persistence: Memory and disk storage strategies
  • Broadcast Variables: Efficiently share large read-only data
  • Shuffle Optimization: Minimize data movement across network
  • Adaptive Query Execution (AQE): Runtime query optimization
  • Data Formats: Choose optimal formats (Parquet, ORC) for performance
  • Partition Tuning: Balance parallelism and overhead

Production Deployment

  • Cluster Managers: Standalone, YARN, Kubernetes, Mesos
  • Resource Allocation: Executor sizing and dynamic allocation
  • Monitoring: Spark UI, metrics, and logging
  • Fault Tolerance: Automatic recovery and checkpointing
  • Security: Authentication, authorization, encryption

Apache Spark Architecture

High-Level Components

┌─────────────────────────────────────────────────────┐
│                   Driver Program                    │
│  ┌────────────┐  ┌─────────────────────────────┐   │
│  │ SparkContext│  │   DAG Scheduler             │   │
│  │            │  │   Task Scheduler            │   │
│  └────────────┘  └─────────────────────────────┘   │
└─────────────────────┬───────────────────────────────┘
                      │ Cluster Manager
                      │ (Standalone/YARN/K8s/Mesos)
        ┌─────────────┼─────────────┐
        │             │             │
┌───────▼──────┐ ┌────▼──────┐ ┌───▼────────┐
│  Executor 1  │ │ Executor 2│ │ Executor N │
│ ┌──────────┐ │ │┌──────────┐│ │┌──────────┐│
│ │  Task 1  │ │ ││  Task 3  ││ ││  Task N  ││
│ ├──────────┤ │ │├──────────┤│ │├──────────┤│
│ │  Task 2  │ │ ││  Task 4  ││ ││  Task N+1││
│ └──────────┘ │ │└──────────┘│ │└──────────┘│
│   Cache      │ │   Cache    │ │   Cache    │
└──────────────┘ └────────────┘ └────────────┘

Components:

  • Driver: Coordinates execution, maintains application state
  • Executors: Distributed processes that execute tasks and store data
  • Cluster Manager: Allocates resources across applications
  • Tasks: Individual units of work sent to executors

Execution Flow

  1. Application Submission: Driver program creates SparkContext
  2. DAG Construction: Transformations build Directed Acyclic Graph
  3. Stage Division: DAG divided into stages at shuffle boundaries
  4. Task Scheduling: Tasks scheduled on executors based on data locality
  5. Execution: Executors run tasks, cache intermediate results
  6. Result Collection: Actions trigger computation and return results

Data Flow

Input Data → RDD/DataFrame → Transformations → Actions → Output
              (Partitioned)    (Lazy DAG)      (Trigger)

Lazy Evaluation:

  • Transformations (map, filter, join) build computation graph
  • Actions (collect, count, save) trigger actual execution
  • Optimizer analyzes entire DAG before execution
  • Minimizes data movement and computation

When to Use Apache Spark

Ideal Use Cases

Large-Scale Batch Processing:

  • ETL pipelines processing TB-PB datasets
  • Log aggregation and analysis
  • Data warehousing and data lake processing
  • Historical data analytics

Real-Time Stream Processing:

  • Real-time dashboards and metrics
  • Fraud detection and anomaly detection
  • IoT sensor data processing
  • Click stream analysis

Interactive Analytics:

  • Ad-hoc queries on large datasets
  • Business intelligence and reporting
  • Data exploration and discovery
  • SQL analytics on data lakes

Machine Learning:

  • Training models on massive datasets
  • Feature engineering at scale
  • Distributed hyperparameter tuning
  • Production ML pipelines

Unified Workloads:

  • Combining batch and streaming in single application
  • Lambda architecture implementations
  • Complex multi-stage data pipelines

Not Ideal For

Small Data (<100 GB):

  • Single-machine tools (pandas, R) are simpler and faster
  • Spark overhead not justified for small datasets

Ultra-Low Latency (<10ms):

  • Specialized stream processors (Flink, Storm) better for microsecond latency
  • Spark's micro-batch approach has 100ms+ latency floor

OLTP Workloads:

  • Transactional databases (PostgreSQL, MySQL) better for CRUD operations
  • Spark optimized for analytical, not transactional, workloads

Simple Transformations:

  • Traditional ETL tools may be simpler for basic operations
  • Spark's power needed for complex, distributed transformations

Quick Start

Installation

PySpark (Python):

# Install via pip
pip install pyspark

# Or with Conda
conda install -c conda-forge pyspark

Spark Standalone:

# Download from Apache Spark website
wget https://archive.apache.org/dist/spark/spark-3.5.0/spark-3.5.0-bin-hadoop3.tgz
tar -xzf spark-3.5.0-bin-hadoop3.tgz
export SPARK_HOME=/path/to/spark-3.5.0-bin-hadoop3
export PATH=$PATH:$SPARK_HOME/bin

Hello World Example

Word Count (Classic Big Data Example):

from pyspark.sql import SparkSession

# Create SparkSession
spark = SparkSession.builder \
    .appName("WordCount") \
    .master("local[*]") \
    .getOrCreate()

# Read text file
text_rdd = spark.sparkContext.textFile("input.txt")

# Word count transformation
word_counts = text_rdd \
    .flatMap(lambda line: line.split()) \
    .map(lambda word: (word, 1)) \
    .reduceByKey(lambda a, b: a + b)

# Collect results
results = word_counts.collect()
for word, count in results:
    print(f"{word}: {count}")

# Or save to file
word_counts.saveAsTextFile("output")

spark.stop()

DataFrame Example:

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, count

spark = SparkSession.builder.appName("DataFrameExample").getOrCreate()

# Create DataFrame
data = [
    ("Alice", "Engineering", 100000),
    ("Bob", "Sales", 80000),
    ("Charlie", "Engineering", 120000),
    ("Diana", "Sales", 90000)
]
df = spark.createDataFrame(data, ["name", "department", "salary"])

# Transformations
result = df.groupBy("department") \
    .agg(count("*").alias("count"),
         avg("salary").alias("avg_salary")) \
    .orderBy(col("avg_salary").desc())

# Show results
result.show()

spark.stop()

Local Development Setup

Configure Local Spark:

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("LocalDevelopment") \
    .master("local[4]")  # 4 local threads \
    .config("spark.driver.memory", "4g") \
    .config("spark.executor.memory", "4g") \
    .config("spark.sql.shuffle.partitions", 8)  # Reduce for local \
    .getOrCreate()

# Set log level to reduce verbosity
spark.sparkContext.setLogLevel("WARN")

Skill Structure

This skill is organized into three comprehensive files:

1. SKILL.md (This File)

  • Core concepts and architecture
  • Deep dives into RDDs, DataFrames, Spark SQL
  • Streaming processing guide
  • MLlib machine learning
  • Performance tuning strategies
  • Production deployment best practices
  • Troubleshooting and common patterns

2. EXAMPLES.md

  • 20+ production-ready code examples
  • Real-world scenarios and use cases
  • Performance optimization examples
  • Streaming analytics patterns
  • Machine learning workflows
  • All examples sourced from Context7's Apache Spark library

3. README.md (You Are Here)

  • Overview and quick start
  • Architecture diagrams
  • When to use Spark
  • Installation and setup
  • Skill navigation guide

Performance Characteristics

Execution Speed

In-Memory Processing:

  • 10-100x faster than Hadoop MapReduce for iterative algorithms
  • Sub-second query latency on cached data
  • Efficient for machine learning workloads with multiple passes

Disk-Based Processing:

  • 2-10x faster than MapReduce on disk-based workloads
  • Optimized shuffle and serialization
  • Efficient DAG execution

Scalability

Horizontal Scaling:

  • Linear scalability to 1000+ nodes
  • Process petabyte-scale datasets
  • Dynamic resource allocation

Vertical Scaling:

  • Leverage multi-core CPUs efficiently
  • Optimize memory usage with Tungsten
  • SIMD vectorization in execution engine

Latency

Batch Processing:

  • Seconds to hours depending on data size
  • Optimized for throughput over latency

Streaming:

  • 100ms to seconds micro-batch latency
  • Continuous processing mode for lower latency
  • Trade-off between throughput and latency

Data Processing Patterns

Lambda Architecture

Combine batch and streaming for comprehensive analytics:

Batch Layer (Historical)     Speed Layer (Real-time)
        ↓                            ↓
  Spark Batch Jobs           Spark Streaming
        ↓                            ↓
   Master Dataset             Real-time Views
        └──────────┬────────────────┘
                   ↓
              Serving Layer
              (Combined Views)

Kappa Architecture

Unified streaming-only architecture:

All Data → Kafka → Spark Streaming → Data Store
                         ↓
                   Reprocessing (same code)

Medallion Architecture (Databricks)

Structured data pipeline:

Bronze Layer (Raw)  → Silver Layer (Cleaned) → Gold Layer (Aggregated)
   Raw ingestion       Validation & cleaning     Business-level aggregates
   Parquet/Delta       Delta Lake format         Star/Snowflake schema

Integration Ecosystem

Data Sources

  • Cloud Storage: S3, Azure Blob, Google Cloud Storage
  • Databases: PostgreSQL, MySQL, Oracle, SQL Server (JDBC)
  • NoSQL: Cassandra, MongoDB, HBase
  • Data Warehouses: Snowflake, Redshift, BigQuery
  • Streaming: Kafka, Kinesis, Event Hubs
  • Files: Parquet, ORC, Avro, JSON, CSV, text

Data Formats

  • Parquet: Best for analytics (columnar, compressed)
  • ORC: Optimized for Hive (columnar, indexed)
  • Avro: Row-oriented, schema evolution
  • Delta Lake: ACID transactions, time travel
  • Iceberg: Open table format, schema evolution

Orchestration

  • Apache Airflow: Workflow orchestration
  • Databricks Jobs: Managed Spark jobs
  • AWS Glue: Serverless ETL
  • Azure Data Factory: Cloud ETL/ELT

Visualization

  • Tableau: Connect via JDBC/ODBC
  • Power BI: Spark connector
  • Superset: Open-source BI
  • Databricks Notebooks: Built-in visualization

Learning Path

Beginner (Week 1-2)

  1. Understand Spark architecture and core concepts
  2. Learn RDD basics and transformations
  3. Practice DataFrame operations
  4. Execute simple SQL queries
  5. Work with different data formats

Intermediate (Week 3-4)

  1. Master DataFrame API and SQL
  2. Implement streaming applications
  3. Basic performance tuning (caching, partitioning)
  4. Use MLlib for simple ML tasks
  5. Deploy to cluster (YARN/Kubernetes)

Advanced (Week 5-8)

  1. Advanced performance optimization
  2. Complex streaming patterns (stateful, windowing)
  3. Production MLlib pipelines
  4. Custom UDFs and data sources
  5. Tuning for large-scale production workloads

Expert (Ongoing)

  1. Contribute to Spark open source
  2. Develop custom Spark extensions
  3. Optimize query plans and execution
  4. Design large-scale architectures
  5. Train and mentor teams

Common Challenges and Solutions

Memory Management

Challenge: OutOfMemoryError in executors Solution: Increase executor memory, use appropriate storage levels, avoid collect() on large datasets

Data Skew

Challenge: Few tasks take much longer due to unbalanced partitions Solution: Use salting, repartition by skewed column, isolate and process skewed keys separately

Shuffle Performance

Challenge: Slow shuffle operations consuming resources Solution: Minimize shuffles (use reduceByKey vs groupByKey), broadcast small tables, tune shuffle partitions

Small Files Problem

Challenge: Many small files causing overhead Solution: Coalesce before writing, use appropriate partitioning, compact files periodically

Streaming Lag

Challenge: Processing falls behind data arrival rate Solution: Increase parallelism, tune watermarks, optimize transformations, scale cluster

Best Practices Summary

  1. Use DataFrames over RDDs - Better optimization and performance
  2. Cache Wisely - Only cache data reused multiple times
  3. Partition Appropriately - 2-4x CPU cores, partition by commonly filtered columns
  4. Use Parquet/ORC - Columnar formats for analytical workloads
  5. Broadcast Small Tables - Avoid shuffling large tables in joins
  6. Enable AQE - Leverage adaptive query execution
  7. Monitor with Spark UI - Identify bottlenecks early
  8. Test with Representative Data - Use production-scale samples
  9. Version Control Everything - Code, configs, schemas
  10. Implement Checkpointing - Ensure fault tolerance in streaming

Resources

Official Documentation

Community

Learning Resources

Tools

  • Databricks: Managed Spark platform
  • AWS EMR: Managed Spark on AWS
  • Azure Synapse: Managed Spark on Azure
  • Google Dataproc: Managed Spark on GCP

Next Steps

  1. Read SKILL.md - Deep dive into all Spark components
  2. Review EXAMPLES.md - Study 20+ production examples
  3. Set Up Local Environment - Install PySpark and run examples
  4. Build a Project - Apply skills to real dataset
  5. Deploy to Cluster - Move from local to distributed execution
  6. Optimize Performance - Profile and tune your application
  7. Contribute Back - Share learnings with community

Skill Version: 1.0.0 Last Updated: October 2025 Maintainer: Apache Spark Community Context7 Integration: /apache/spark (8000 tokens) License: Apache License 2.0

Download Skill