Data Stream Development with Apache Spark, Kafka, and Spring Boot
•The Course Overview
•Discovering the Data Streaming Pipeline Blueprint Architecture
•Analyzing Meetup RSVPs in Real-Time
•Running the Collection Tier (Part I – Collecting Data)
•Collecting Data Via the Stream Pattern and Spring WebSocketClient API
•Explaining the Message Queuing Tier Role
•Introducing Our Message Queuing Tier –Apache Kafka
•Running The Collection Tier (Part II – Sending Data)
•Dissecting the Data Access Tier
•Introducing Our Data Access Tier – MongoDB
•Exploring Spring Reactive
•Exposing the Data Access Tier in Browser
•Diving into the Analysis Tier
•Streaming Algorithms For Data Analysis
•Introducing Our Analysis Tier – Apache Spark
•Plug-in Spark Analysis Tier to Our Pipeline
•Brief Overview of Spark RDDs
•Spark Streaming
•DataFrames, Datasets and Spark SQL
•Spark Structured Streaming
•Machine Learning in 7 Steps
•MLlib (Spark ML)
•Spark ML and Structured Streaming
•Spark GraphX
•Fault Tolerance (HML)
•Kafka Connect
•Securing Communication between Tiers
•Test Your Knowledge
•Apache Spark: Tips, Tricks, & Techniques
•Using Spark Transformations to Defer Computations to a Later Time
•Avoiding Transformations
•Using reduce and reduceByKey to Calculate Results
•Performing Actions That Trigger Computations
•Reusing the Same RDD for Different Actions
•Delve into Spark RDDs Parent/Child Chain
•Using RDD in an Immutable Way
•Using DataFrame Operations to Transform It
•Immutability in the Highly Concurrent Environment
•Using Dataset API in an Immutable Way
•Detecting a Shuffle in a Processing
•Testing Operations That Cause Shuffle in Apache Spark
•Changing Design of Jobs with Wide Dependencies
•Using keyBy() Operations to Reduce Shuffle
•Using Custom Partitioner to Reduce Shuffle
•Saving Data in Plain Text
•Leveraging JSON as a Data Format
•Tabular Formats – CSV
•Using Avro with Spark
•Columnar Formats – Parquet
•Available Transformations on Key/Value Pairs
•Using aggregateByKey Instead of groupBy()
•Actions on Key/Value Pairs
•Available Partitioners on Key/Value Data
•Implementing Custom Partitioner
•Separating Logic from Spark Engine – Unit Testing
•Integration Testing Using SparkSession
•Mocking Data Sources Using Partial Functions
•Using ScalaCheck for Property-Based Testing
•Testing in Different Versions of Spark
•Creating Graph from Datasource
•Using Vertex API
•Using Edge API
•Calculate Degree of Vertex
•Calculate Page Rank
•Troubleshooting Apache Spark
•Eager Computations: Lazy Evaluation
•Caching Values: In-Memory Persistence
•Unexpected API Behavior: Picking the Proper RDD API
•Wide Dependencies: Using Narrow Dependencies
•Making Computations Parallel: Using Partitions
•Defining Robust Custom Functions: Understanding User-Defined Functions
•Logical Plans Hiding the Truth: Examining the Physical Plans
•Slow Interpreted Lambdas: Code Generation Spark Optimization
•Avoid Wrong Join Strategies: Using a Join Type Based on Data Volume
•Slow Joins: Choosing an Execution Plan for Join
•Distributed Joins Problem: DataFrame API
•TypeSafe Joins Problem: The Newest DataSet API
•Minimizing Object Creation: Reusing Existing Objects
•Iterating Transformations – The mapPartitions() Method
•Slow Spark Application Start: Reducing Setup Overhead
•Performing Unnecessary Recomputation: Reusing RDDs
•Repeating the Same Code in Stream Pipeline: Using Sources and Sinks
•Long Latency of Jobs: Understanding Batch Internals
•Fault Tolerance: Using Data Checkpointing
•Maintaining Batch and Streaming: Using Structured Streaming Pros
•