FeaturedUdemy
Building Data Pipelines with Airflow
4.7(12)171 studentsADVANCEDEnglish
Korshub's take
The Korshub Take
This advanced data engineering course explores modern workflow orchestration using Apache Airflow 3.2. Focusing on production readiness, you will construct DAGs via the modern TaskFlow API, schedule workflows around data readiness using Assets and Asset Partitions, and build a complete SQL ETL pipeline on DuckDB featuring native data quality checks without third-party dependencies.
Who It\'s For
Data engineers, backend developers, and data architects who want to master modern Apache Airflow practices and move pipelines from prototypes to production environments.
Key Takeaways
- TaskFlow API & DAGs: Author clear, pythonic DAGs using the TaskFlow API for clean data passing between tasks.
- Data-Driven Scheduling: Trigger workflows using Airflow Assets and new Asset Partitions based on data availability.
- Pipeline Reliability: Implement retries, automated callbacks, unit testing, and robust error handling for resilient execution.
- DuckDB ETL & Quality Checks: Build an end-to-end SQL pipeline on DuckDB with built-in data validation checks in Airflow.
About this course
Want to take your Airflow skills further? This course uses Airflow 3.2, so you'll learn the latest way of doing things. You'll author Dags with the TaskFlow API, schedule them based on data using Assets and the new Asset Partitions, and make them reliable with retries, callbacks, and tests.
In the final chapter, you'll build an end-to-end SQL ETL pipeline on DuckDB and add data quality checks directly in Airflow, without any third-party libraries, so the data your pipeline produces stays trustworthy. By the end, you'll know how to take a pipeline from a prototype to something you can actually run in production.
Advertisement