Data Engineer Things Newsletter - Community Spotlight Edition (July 2026)
Lessons from Airflow, open source, and AI workflows on building data platforms that can adapt as systems, tools, and semantics drift.
Hi everyone,
In our next community spotlight, we are chatting with Vikram Koka, Chief Strategy Officer at Astronomer and Apache Airflow PMC (Project Management Committee) member.
Vikram has been part of Airflow’s modern evolution since 2019, contributing to major milestones including Scheduler High Availability in Airflow 2.0, data-driven scheduling, dynamic tasks, setup and teardown, and the program management of Airflow 3. Many of you may also recognize him from Data Engineering Open Forum 2026, where he joined the closing panel, Frontiers of Data: The Future of Data Engineering in an AI World, alongside leaders from Netflix and OpenAI.
In this interview, we discuss Vikram’s journey across three decades of data engineering, Airflow’s evolution, why data drift remains a constant challenge, and how AI is expanding the role of data engineers from moving data to understanding context and building reliable workflows.
Let’s dive in and walk through Vikram’s journey.
- Shubham & Eddy
🗓 DET Meetups in Toronto and Seattle
We have two meetup events coming up this month:
Seattle meetup at Downtown Seattle on Thu, July 16 (RSVP)
Toronto meetup at Downtown Toronto on Thu, July 16 (RSVP)
(🎤 Interested in speaking at our meetups or online webinars? Submit talk proposals here.)
Spotlight: Vikram Koka
“Before you solve a problem, you look at it and think, ‘Oh man, this is incredibly complicated. How are we ever going to pull this off?’ But the moment you crack it, you look back and say, ‘Huh, that actually wasn’t all that difficult after all.’”
For people who may not know your journey, how would you describe the path that brought you here?
My journey really kicked off during my master’s program at UC Santa Barbara, where I worked on the GNU Debugger (GDB), which is now distributed as part of Linux. That was my early introduction to open source quite a long time ago.
From there, I moved to Silicon Valley and got into manufacturing automation for semiconductors. My first project was in user interfaces, but my second project shifted into the data space where I was gathering data from manufacturing systems to provide process analysis. That organically evolved into building a full-fledged data warehouse pulling from manufacturing, supply chain, and financial systems to build dashboards and star schemas. I technically wrote my first data pipeline about three decades ago without ever knowing that “data pipeline” would become the actual term for it!
Since then, I’ve been through a variety of Silicon Valley companies. My last gig before Astronomer was in IoT, where I dealt with the massive explosion of data coming from embedded devices. We had to microbatch events over cellular networks purely for financial reasons, because sending data on every single event is just too expensive. That sparked my desire to get back heavily into data. When the opportunity at Astronomer presented itself in 2019, I saw the massive potential of Apache Airflow and the impending data explosion, and I leaped right in.
If you were to reflect back on your career, what did you feel was a decision that had the biggest impact on your trajectory?
Honestly, the biggest impact was my willingness to be non-linear.
We are often taught straight out of school to look at career trajectories in a very linear fashion. I was completely willing to throw that out the window. My primary focus has always been on what I enjoy most, what needs to change, and a willingness to jump across completely different roles and domains. Whether I was acting as an executive, an individual contributor, working in open source, or working in commercial software, the title didn’t matter. I just wanted to make a difference.
As a technologist, your core desire should be deeply understanding why someone wants a problem solved and having the empathy to make their life better through technology. Remaining open to non-linear shifts allowed me to focus on that.
What is the most proud you’ve been solving a problem? Is there a challenge that stands out?
I’m honestly not sure I can pick just one. I think this is true for most engineers: before you solve a problem, you look at it and think, “Oh man, this is incredibly complicated. How are we ever going to pull this off?” But the moment you crack it, you look back and say, “Huh, that actually wasn’t all that difficult after all.” I used to see this all the time with engineers when filing patent applications; once the problem is solved, they feel the solution is totally obvious.
Because of that, I don’t really dwell on past problems. I look forward. I have never been more excited about the scale of problems we can solve right now. We are standing on the shoulders of giants. With the current state of AI, computing infrastructure, and cloud evolution, we can tackle massive problems today that were completely impossible before. It’s always about the next problem for me.
At the time when you joined Apache Airflow, what did you see were the biggest limitations, and how has the project evolved?
I officially got involved in December 2019. At that time, Airflow had roughly 100,000 to 180,000 downloads a month, and the absolute biggest hurdles were reliability and enterprise scalability. Essentially, user code was running directly inside the scheduler, causing it to crash frequently because it wasn’t designed for high availability.
Recognizing this, I co-authored an Airflow Improvement Proposal (AIP) to build a highly available, active-active scheduler. We released it as part of Airflow 2 in December 2020. That completely catapulted the project’s growth because enterprises could finally deploy it reliably at massive scale. Today, Airflow sees tens of millions of downloads a month and is utilized by an estimated 80,000 to 90,000 enterprises globally.
Looking at Airflow 3, the project has taken another massive leap forward based on three pillars:
Run tasks anywhere: You are no longer restricted to a single cluster or data center; you can dynamically run tasks across local CPUs, remote GPUs, etc.
Run tasks in any language: While Airflow is natively Python-based, we have expanded support to execute tasks written in Java, Go, and soon TypeScript.
Run tasks at any time: Moving past strictly batch schedules to support API triggers and message queues, making it perfectly suited to handle heavy AI training and inference workloads.
What are some lessons from Airflow’s evolution that data engineers can apply when building their own internal platforms?
The number one lesson is: Don’t get locked in. Prioritize optionality.
The biggest consistency in data engineering from my very first pipeline 30 years ago to today is data drift. Data formats change constantly, and the semantic meaning of data shifts over time, which regularly throws software engineers for a loop. Because change is inevitable, your tools and frameworks must remain highly adaptable.
What made Airflow incredibly successful is that we intentionally chose to be completely unopinionated. We refused to pick winners among databases, file systems, or cloud providers. We joke internally that we are very opinionated about being unopinionated. For instance, with AI workloads today, people are heavily reliant on commercial vendor models like OpenAI or Anthropic. But we built Airflow with the optionality to easily support self-hosted, open-source models tomorrow because security, privacy, and architecture needs will inevitably change. Build your internal platforms with that exact same flexibility.
What do you think is the role of data engineering with AI these days?
The role of data engineers has dramatically expanded, and engineers need to embrace that. The technical elements like knowing how to construct a pipeline remain fundamentally important. However, the semantic understanding of data is now paramount. This means understanding what the data actually represents from a strict business context standpoint.
In the context of AI, the objective isn’t just to dump a massive haystack of data on an LLM and assume it will find the needle. If you do that, it will hallucinate heavily and cost you a fortune.
The true role of a data engineer today is to curate and feed the AI agent the smallest possible amount of the right, highly relevant data at the exact right time.
To do that successfully, you have to deeply understand the business imperatives and context. It is a non-trivial, holistic challenge, but a massive opportunity for the field.
What did you learn by automating your own work process?
💡Editorial Note: As a PMC member of Apache Airflow, Vikram tracks many Airflow Improvement Proposals (AIPs) across Confluence specs, GitHub PRs, and the repository file tree. Every week, he was spending hours cross-referencing these sources to understand what had shipped, what was still open, and where things actually stood. So he decided to build an AI skill to do that work for him.
👉 Read Vikram’s full write-up HERE.
With that context, here is Vikram’s answer:
When I set out to automate my own daily job of tracking all the proposals, Confluence specs, and GitHub PRs happening across the open-source Airflow community, it turned into an absolute voyage of discovery.
The biggest thing I learned is that if you just give an AI agent a prompt and let it autonomously call tools to gather data, the potential for drift and hallucination skyrockets. For instance, if you ask an agent to figure out if a project is “70% complete,” it really struggles. As humans, we know one PR might be massive while another is tiny, but a stochastic model tries to guess and make assumptions, which drives you nuts.
To solve it, I had to break the process down into a 12-step structured pipeline:
Use APIs for Data Collection: I stopped letting the agent collect the data. Instead, I used standard, deterministic APIs to pull the raw text from Confluence and GitHub.
Isolate the Agent for Reasoning: I fed that specific data to the agent and restricted its job strictly to reasoning over what was provided.
Layer standard code on top of AI: I introduced an evaluation model to check the reasoning, but I used a traditional, deterministic Python process to actually apply the final corrections.
By separating what an agent is actually good at (reasoning) from what deterministic code is good at (data collection and structured execution), I finally got a reliable, repeatable report.
But the most surprising lesson of all was that this pipeline approach was actually cheaper and faster. A pure agentic workflow builds up context with every sequential tool call, causing token usage to explode quadratically O(n2). By orchestrating it as a pipeline where steps are independent, token growth stays linear O(n).
Ultimately, it taught me a fundamental truth about modern orchestration: the best way to get predictable, deterministic outcomes out of AI agents is to wrap them inside a structured data pipeline.
When you’re hiring data engineers today, what are some qualities or signals that tell you someone is likely to have an outsized impact?
There are three key signals I look for:
Technical Competence: This is the baseline requirement. You need enough technical depth to apply solid system-level thinking to complex architectures.
Curiosity: I value intense curiosity because it drives an engineer to learn the semantic nature of data and understand its broader impact on the organization.
Business Correlation: The best engineers can step outside of the technical box and connect their code directly to business imperatives. They don’t just build a pipeline because they were told to; they ask why someone needs this to happen and map the technical tools directly to that business outcome.
What lessons from managing commercial products translate well to open-source projects, and what doesn’t apply?
What translates perfectly: You must always ask yourself, “Why does a user or customer actually care to use my product?” Whether it’s a paid commercial enterprise system or a free open-source project, the core human element is identical. You have to listen to your users, understand their pain points, and focus intensely on solving their specific problems.
What is completely different: The dynamics of alignment and community in open source. In a commercial setting, you have organizational authority. In a project like Airflow, you are dealing with a massive community, currently around 3,800 total contributors, with 300 to 500 actively writing code every single month. These people are contributing out of pure passion on their weekends, late nights, and days off.
Managing this is a delicate balancing act. You have to encourage them, value their time, and review their PRs quickly, or they will walk away. However, you also have to maintain strict alignment with the long-term architectural goals, code maintainability, and security implications of the project. Saying “no” to a passionate volunteer because their contribution doesn’t fit the long-term roadmap is infinitely harder than managing product alignment in a commercial business.
Who should we spotlight next in the Data Engineer Things Community Newsletter, and why?
I would say Laura Pruitt from Netflix or Paul Ellwood from Open AI.
I was very impressed by the conversation we had at the Data Engineering Open Forum earlier this year. From a business impact perspective and the quality of thinking about how data engineering is evolving, I think Laura would be a great conversation for the DET community.
We discussed Data Drift as a key consistent challenge in Data Engineering since I wrote my first data pipeline three decades ago.
Paul touched on AI drift as part of the Forum panel discussion. I think that would be a great topic to have Paul expand on.
💡 Editorial Note: The discussion Vikram references came from the DEOF 2026 closing panel, Frontiers of Data: The Future of Data Engineering in an AI World. The panel covered many of the same themes from this spotlight, including data drift, AI drift, semantic context, and the evolving role of data engineers.
Watch the full panel here:
Key Takeaways
Semantic understanding is now part of data engineering: Data engineers need to understand business context, not just move data, so they can provide AI systems with the right context.
Optionality is a core platform principle: Airflow’s success comes from staying flexible across databases, clouds, runtimes, and model providers.
Data drift remains the constant challenge: The interview connects long-standing data drift with newer challenges like semantic drift and AI drift.
High-impact data engineers connect tech to business: The strongest engineers combine technical depth, curiosity, and the ability to map their work to business outcomes.
Community poll
💬 How to stay connected
ℹ️ About Data Engineer Things
Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.