The Complete Hands-On Introduction to Apache Airflow 3

Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring workflows. If you have many ETLs  to manage, Airflow is a must-have.

In this course, you are going to learn everything you need to start using Apache Airflow 3 through theory and practical videos.

You will start with the basics such as:

• What is Apache Airflow?

• The core concepts of Airflow

• Different architectures to run Airflow

• What happens when a workflow runs

Then you will create your first data pipeline covering many Airflow features such as:

• Sensors, to wait for specific conditions

• Hooks, for interacting with a database

• Taskflow, for writing efficient, easy-to-read DAGs

• XCOMs, for sharing data

and much more.

At the end of the project, you will be equipped for creating your own workflows!

After the project, you will also discover the new Asset syntax that completely change your way of thinking about your tasks in Airflow 3.

• What is an Asset

• How to create dependencies between Assets

• How to materialize an Asset

and more.

You will discover the different executors for running Airflow at scale. More specifically, the CeleryExecutor which is extremely popular.

• How to configure Airflow for using the CeleryExecutor

• How to distribute your tasks on different Workers

• How to choose your Workers with Queues

and more.

You will explore advanced features to elevate your DAGs to a new level, and conclude by creating your own Airflow provider and a new decorator for executing SQL requests.

If you're working in a company with Airflow, you will love that part.

Enjoy