edX
Cleaning Data with PySpark
4.8(491)ADVANCEDEnglish
Korshub's take
Scale Your Data Cleaning
Moving a data prototype from a local laptop to a production environment often reveals the harsh reality of real-world datasets. This course is designed for those facing the challenges of missing fields, inconsistent formatting, and massive scale—specifically when dealing with millions or billions of rows.
What to Expect
- Production-Ready Skills: Transition from basic scripts to robust data processes.
- PySpark Mastery: Learn the specific terminology and methods required for Apache Spark.
- Optimization: Focus on building a platform that is not only performant but also maintainable and easy to understand.
Whether you are an experienced engineer or new to big data, this advanced guide provides the best practices needed to handle data at an enterprise scale.
About this course
Working with data is tricky - working with millions or even billions of rows is worse. Did you receive some data processing code written on a laptop with fairly pristine data? Chances are you’ve probably been put in charge of moving a basic data process from prototype to production. You may have worked with real world datasets, with missing fields, bizarre formatting, and orders of magnitude more data. Even if this is all new to you, this course helps you learn what’s needed to prepare data processes using Python with Apache Spark. You’ll learn terminology, methods, and some best practices to create a performant, maintainable, and understandable data processing platform.
Access
SubscriptionIncluded with a DataCamp subscription
Advertisement
