Pyspark - Zero To Superhero
Published 9/2025
Created by Ganesh Kudale
MP4 | Video: h264, 1280x720 | Audio: AAC, 44.1 KHz, 2 Ch
Level: Intermediate | Genre: eLearning | Language: English | Duration: 8 Lectures ( 1h 53m ) | Size: 711 MB
PySpark and Spark SQL
What you'll learn
Basics of PySpark
Reading PySpark Data Frame and various methods of creating PySpark Data Frames
Processing using PySpark Data Frames and Spark SQL - Deep Dive
Write Transformed Results from Data Frame to Expected Location
Requirements
Basics of Python Programming and Basics of SQL
Description
Course Description:This hands-on course is designed for aspiring and experienced data engineers who want to master PySpark-the powerful distributed computing framework built on Apache Spark. Led by Ganesh Kudale, a seasoned data engineer, the series walks learners through real-world scenarios, from foundational concepts to advanced transformations, with a strong focus on production-grade pipeline development.What You'll Learn
ySpark Essentials: RDDs, Data Frames, and Spark SQLData Ingestion & ETL: Reading from CSV, JSON, ParquetTransformations & Actions: Filtering, joins, aggregations, and window functionsWho Should Enrollata engineers working with big data platformsDevelopers transitioning from SQL to PySparkProfessionals building scalable pipelines in Big DataAnyone preparing for Spark-related interviewsCurriculum - Session 1 - Creating the raw data frame Session 2 - Defining the Schema in PySpark Session 3 - Reading the data frame from file stored at storage location Session 4 - Different ways of creating the data frame Session 5 - Transformations and Action in Apache Spark Session 6 - Data Frame Read Modes Session 7 - PySpark withColumn Transformation Session 8 - PySpark datatype conversions Session 9 - withColumn in PySpark VS spark SQL Session 10 - PySpark select transformation Session 11 - PySpark selectExpr Transformation Session 12 - Performance difference between select, selectExpr and withColumn transformations Session 13 - Renaming the column in PySpark data frame and using Spark SQL Session 14 - Performance efficient approach for renaming columns in PySpark data frame Session 15 - Filtering data in PySpark Session 16 - Efficient ways to filter the data in PySpark Session 17 - Sorting in PySpark Single Column Session 18 - Sorting in PySpark - Multiple Columns Session 19 - Sorting in Spark SQL Session 20 - Performance difference between sort and orderBy in PySpark Session 21 - Aggregations in PySpark Session 22 - Simple Aggregations in PySpark - Count, Average, Max, Min Session 23 - Introduction to Grouping aggregations in PySpark Session 24 - Grouping aggregations in PySpark - Continuation Session 25 - Grouping aggregations in PySpark - Continuation 1 Session 26 - Grouping Aggregations on Multiple Columns in PySpark Session 27 - Grouping Aggregations on Multiple Columns in PySpark Continuation Session 28 - Running multiple grouping aggregations together Session 29 - Windowing Aggregations in PySpark - Row_Number Session 30 - Windowing Aggregations in PySpark - Rank Session 31 - Windowing Aggregations in PySpark - Dense Rank Session 32 - Remove duplicates using PySpark window functions Session 33 - Top scorer students in each subject using PySpark window functions Session 34 - PySpark Window Function Lead Data Frame Session 35 - PySpark Window Function Lead Spark SQL Session 36 - PySpark Window Function - LAG Session 37 - CASE WHEN in PySpark - One when Condition Session 38 - CASE WHEN in PySpark - Multiple when Conditions and Multiple Conditions within when Session 39 - WHEN Otherwise in PySpark - One when Condition Session 40 - WHEN Otherwise in PySpark - Multiple when Conditions Session 41 - Working With dates in PySpark - Python List Session 42 - Working With dates in PySpark - Storage Location Session 43 - Adding created timestamp and created date to the newly added data in PySpark Session 44 - Joins in PySpark - Theory Session 45 - Inner Join in PySpark - Joining over one Column Session 46 - Inner Join in PySpark - Joining over one Column - NULL values in joining Columns Session 47 - Inner Join in PySpark - Joining over multiple Columns Session 48 - Left Outer Join in PySpark - Joining over one Column Session 49 - Left Outer Join in PySpark - Joining over one Column - NULL values in joining Columns Session 50 - Left Outer Join in PySpark - Joining over multiple Columns Session 51 - Right Outer Join in PySpark - Joining over one Column Session 52 - Right Outer Join in PySpark - Joining over one Column - NULL values in joining Columns Session 53 - Right Outer Join in PySpark - Joining over multiple Columns Session 54 - Full Outer Join in PySpark - Joining over one Column Session 55 - Full Outer Join in PySpark - Joining over one Column - NULL values in joining Columns Session 56 - Full Outer Join in PySpark - Joining over multiple Columns Session 57 - Left Semi Join in PySpark Session 58 - Left Anti Join in PySpark Session 59 - Reading Single Line JSON file as PySpark Data frame Session 60 - Reading Multi Line JSON file as PySpark Data frame Session 61 - Reading parquet file as PySpark data frame Session 62 - Data Frame writer API and data frame writer ModesWho this course is for
Beginner and Experienced PySpark and Spark SQL Developers
Code:
Bitte
Anmelden
oder
Registrieren
um Code Inhalt zu sehen!
