Skip to content
VirtualOnlinePast event

Apache Airflow in Production: DAG Design, Monitoring, and Field Lessons

Explore the topics

Event artwork for Apache Airflow in Production: DAG Design, Monitoring, and Field Lessons

This event has ended

This listing is kept as an archive. Explore upcoming events to find your next one.

See upcoming events

Supporting this event

Sponsors

PacktHumand Talent
About this event

What to expect

Learn how to build and operate Apache Airflow at enterprise scale in this online technical session with Pradeep Kalluri, Data Engineer at NatWest Bank and Apache Airflow open-source contributor. This session cuts through the basics to focus on real-world patterns for running Airflow reliably in production cloud environments.

What You'll Learn

This 90-minute session covers the critical aspects of running Airflow in production that go far beyond writing your first DAG:

  • DAG Design Patterns That Scale: Avoid common pitfalls in dependency management and task granularity that cause bottlenecks in large-scale deployments
  • Production Monitoring and Alerting: Learn what metrics actually matter, how to configure SLA alerts effectively, and proven techniques for debugging pipeline failures quickly
  • Executor Deep-Dives: Understand when to use Celery versus Kubernetes executors, with practical guidance on choosing the right executor for your specific workload characteristics
  • Open-Source Contribution Insights: Discover what contributing to the Airflow codebase reveals about how the platform really works under the hood
  • Cloud Deployment Best Practices: Get actionable patterns for running Airflow on AWS and Azure with high availability and resilience

Who Should Attend

This session is ideal for data engineers, platform engineers, and technical leads who are either operating Airflow in production or planning to deploy it at scale. Whether you're managing multi-team pipelines with hundreds of DAGs or just moving beyond proof-of-concept, you'll gain practical patterns you can apply immediately.

About the Speaker

Pradeep Kalluri brings 3+ years of hands-on experience building production data platforms at major enterprises including NatWest, Accenture, and Capgemini. He specializes in cloud-native architectures, real-time processing with Kafka and Spark, and data quality frameworks. As an active Apache Airflow contributor and published technical writer on Medium, Pradeep shares the real lessons learned from running data infrastructure at scale.

Event Format

This is an online event hosted via Microsoft Teams. The session runs from 18:30 to 19:55 BST, with the main presentation followed by a prize draw for Packt eBooks sponsored by Packt. The Teams link will be shared with registered attendees before the event.

Hosted by the Oxford Microsoft Data Platform Group, this community brings together data profe

Running an AI event?

List it here, free.

Meetups, workshops and conferences go in front of thousands of builders after a quick review.

List your event