Skip to content
VirtualOnlinePast event

Architecting an Agentic Data Pipeline: From Data Lake Discovery to Managed Orchestration

Event artwork for Architecting an Agentic Data Pipeline: From Data Lake Discovery to Managed Orchestration

This event has ended

This listing is kept as an archive. Explore upcoming events to find your next one.

See upcoming events
About this event

What to expect

In modern data engineering, the challenge isn't just building pipelines — it's architecting systems that can discover, transform, and orchestrate data at scale while maintaining governance and integrity. This online session from Global AI Jacksonville introduces a paradigm shift: using AI not as a code generator, but as a cognitive partner that reasons through complex architectural decisions alongside human engineers.

From Manual Scripts to Agentic Design

Traditional data engineering workflows involve manually writing transformation scripts, configuring orchestration tools like Airflow, and maintaining data quality checks across fragmented systems. This session presents an alternative approach where autonomous agents handle data lake discovery, propose logical abstractions for staging layers, and design physical models with built-in referential integrity — all while engineers focus on strategic design reviews and architectural intent.

You'll explore six critical stages of agentic pipeline development: deploying discovery agents to identify patterns in raw data lakes, establishing governance guardrails that maintain system consistency, building logical staging layers that separate sources from business logic, implementing physical dimension and fact tables, developing idempotent incremental update strategies, and orchestrating complex task dependencies with integrated observability.

Why This Matters for Data Teams

This isn't about replacing data engineers — it's about elevating their role from syntax writers to system architects. When AI agents handle repetitive implementation tasks, engineers can focus on solving higher-order problems: defining data grain strategies, establishing lineage requirements, and ensuring operational health is built into the design process rather than bolted on afterward. The session demonstrates practical applications using platforms like Databricks for cloud-based big data processing.

Who Should Attend

This session targets data engineers and architects ready to evolve beyond manual scripting, engineering leaders evaluating the ROI of autonomous agents in production environments, AI practitioners seeking real-world applications beyond chatbots, and technical decision-makers responsible for maintaining governance in AI-augmented organizations. Whether you're managing petabyte-scale data lakes or bui

Running an AI event?

List it here, free.

Meetups, workshops and conferences go in front of thousands of builders after a quick review.

List your event