Apache Airflow: Building and Managing Data Pipelines Training Course

Apache Airflow serves as an open-source platform designed for the creation, scheduling, and oversight of workflows. Leveraging Airflow 2.x, enterprises can orchestrate data pipelines at a significant scale, seamlessly integrate with cloud providers, and uphold rigorous observability and security standards within production settings.

Delivered by an instructor in a live format (either online or on-site), this programme targets data and DevOps professionals ranging from beginner to intermediate levels who aim to utilise Apache Airflow to construct, automate, and expand end-to-end data pipelines.

Upon completing this training, participants will be equipped to:

Deploy and configure Apache Airflow across various environments.
Develop, schedule, and oversee DAGs leveraging the TaskFlow API.
Integrate Airflow with diverse data sources, cloud services, and machine learning workflows.
Administer ETL pipelines while guaranteeing data integrity.
Enhance the security and scalability of Airflow using executors, queues, and Kubernetes.
Implement observability frameworks and best practices for production-grade deployments.

Course Structure

Engaging lectures facilitated by open discussion.
Extensive exercises and practical drills.
Practical implementation within a live-lab setting.

Customisation Options

For tailored training requests regarding this course, please reach out to us to organise.

This course is available as onsite live training in Portugal or online live training.

Thank you for sending your enquiry! One of our team members will contact you shortly.

Thank you for sending your booking! One of our team members will contact you shortly.

Course Outline

Introduction to Apache Airflow

The concept of workflow orchestration
Principal features and advantages of Apache Airflow
Enhancements in Airflow 2.x and an overview of the ecosystem

Architecture and Fundamental Concepts

Scheduler, web server, and worker processes
DAGs, tasks, and operators
Executors and backends (Local, Celery, Kubernetes)

Installation and Configuration

Installing Airflow in both local and cloud environments
Configuring Airflow with various executors
Establishing metadata databases and connections

Navigating the Airflow UI and CLI

Exploring the Airflow web interface
Monitoring DAG executions, tasks, and logs
Utilising the Airflow CLI for administrative tasks

Authoring and Managing DAGs

Creating DAGs using the TaskFlow API
Employing operators, sensors, and hooks
Managing dependencies and scheduling intervals

Integrating Airflow with Data and Cloud Services

Connecting to databases, APIs, and message queues
Executing ETL pipelines via Airflow
Cloud integrations: AWS, GCP, and Azure operators

Monitoring and Observability

Task logs and real-time monitoring capabilities
Metrics collection using Prometheus and Grafana
Setting up alerting and notifications via email or Slack

Securing Apache Airflow

Role-based access control (RBAC)
Authentication through LDAP, OAuth, and SSO
Secrets management using Vault and cloud secret stores

Scaling Apache Airflow

Parallelism, concurrency, and task queue management
Utilising CeleryExecutor and KubernetesExecutor
Deploying Airflow on Kubernetes using Helm

Best Practices for Production Environments

Version control and CI/CD implementation for DAGs
Testing and debugging DAGs
Maintaining reliability and performance at scale

Troubleshooting and Optimisation

Debugging failed DAGs and tasks
Optimising DAG performance
Identifying common pitfalls and strategies to avoid them

Summary and Next Steps

Requirements

Proficiency in Python programming
Working knowledge of data engineering or DevOps principles
Conceptual understanding of ETL or workflow orchestration

Intended Audience

Data scientists
Data engineers
DevOps and infrastructure engineers
Software developers

21 Hours

Custom Corporate Training

Training solutions designed exclusively for businesses.

Customized Content: We adapt the syllabus and practical exercises to the real goals and needs of your project.
Flexible Schedule: Dates and times adapted to your team's agenda.
Format: Online (live), In-company (at your offices), or Hybrid.

Investment

Price per private group, online live training, starting from 3900 € + VAT*

(*The final price may vary depending on the technical specialization of the course, the level of customization, the method of delivery and the number of learners)

Need help picking the right course?
info@nobleprog.pt or +351 30 050 9666

Testimonials (7)

The instructor adapted the training to the participants’ level and responded to all questions. He was very communicative, and it was easy to interact with him. I really appreciated the format of the training, which included many practical exercises. Overall, it was a very engaging and well-organized session.

Jacek Chlopik - ZAKLAD UBEZPIECZEN SPOLECZNYCH