Hello Everyone,
Hope everyone is doing well!
One of my favourite things about the data engineering community is how much we learn from each other. Most of us may be working with different companies, different stacks and very different scales, but the problems often sound surprisingly familiar.
A pipeline that behaves perfectly until it reaches production. A platform nobody uses quite the way we expected. A technology everyone suddenly wants to adopt. Or that one issue which takes several hours to debug and turns out to have a painfully simple explanation. 🙂
That’s also what we try to capture with Data Pulse every month not just what is new, but what we think is genuinely worth a data engineer spending some time on.
For August, we found some interesting engineering stories from Airflow, Netflix, Uber and Spotify, covering everything from pipeline reliability and data platform architecture to AI infrastructure, governance and some clever thinking around how we access data in the lake.
Grab a coffee and have a read. Hopefully a few of these are useful in something you’re building right now.
– Sri
📚 Data Pulse
How Airflow is using AI to make data engineering more resilient, not more complex
📖 Topic: Data Orchestration
🧠 Level: Intermediate
Summary: Most data engineers have dealt with the familiar 2 AM pipeline failure: open the logs, figure out whether it is a network issue, bad data, expired credentials or an upstream schema change, and then decide whether retrying will actually help.
Vikram Koka, CSO at Astronomer and an Apache Airflow PMC member, walks through three capabilities that try to make some of this operational work smarter.
LLMSchemaCompareOperator uses an LLM to compare schemas semantically rather than relying only on exact type matches. A Task State Store allows a task to retain information across retries, which is useful when Airflow is orchestrating long-running jobs in systems such as Spark or Databricks. And LLMRetryPolicy can use the team’s own runbook to classify a failure and decide whether it should retry, fail immediately or escalate.
What I like about this approach is that the AI is being applied inside a very normal data engineering problem. We are not rebuilding the pipeline around an agent; we are using AI where rigid rules start becoming difficult to maintain.
💡 Why is this relevant for DEs?
It is a practical use of AI rather than AI for the sake of AI. Scheduling, dependencies and execution remain deterministic. The LLM is used only where interpretation can add value understanding schema differences or classifying an unfamiliar failure.
Retry logic deserves more thought than we usually give it.
retries=3is easy to configure, but an expired credential will still be expired after the third attempt. A rate limit, on the other hand, may genuinely benefit from retrying.Persistent task state solves a very real operational problem. If an external Spark or Databricks job has already started, retrying the Airflow task should ideally reconnect to it rather than submit another expensive job.
More broadly, this is a good example of AI augmenting data engineers rather than replacing the engineering around them. The team still defines the runbook and the expected behaviour; AI helps execute some of that knowledge consistently.
Simplifying Data and Product Integrations with a Data Abstraction Layer
📖 Topic: Data Platform Architecture & Data Modeling
🧠 Level: Intermediate
Summary:
Anyone who has worked with a mature data platform has probably heard some version of this:
“Use table_v2 for newer data, table_v1 for older dates, and remember that one of the metrics has a different name between the two.”
Uber built a Data Abstraction Layer (DAL) to stop pushing that complexity onto every consumer. Instead of applications querying physical tables directly, consumers request a logical table, the fields they need, a time range and filters. The DAL then figures out where that data physically lives, generates the required queries, runs them across the appropriate systems and assembles the final response.
One of the more interesting parts is the table resolution logic. Recent data might be available in a real-time OLAP table while historical data sits in a daily Hive table. Consumers still make one request; the DAL decides which datasets should serve each part of it.
Uber reports that this approach reduced the turnaround time for new advertiser reports from several weeks to under two days, and the DAL has since expanded beyond its original advertising use case.
💡 Why is this relevant for DEs?
Consumers becoming tightly coupled to physical tables is a common source of platform fragility. Once dozens of dashboards, services and pipelines know your exact table names and schemas, even a simple migration can become painful.
The logical-to-physical separation is the most useful idea here. Consumers describe what they need while the platform owns where and how that data is retrieved.
The table-resolution pattern is especially interesting. Many of us have written
UNIONqueries that combine recent real-time data with older batch data. Uber moves that complexity into platform metadata instead of repeating it in every consumer.This is also a good example of what useful platform abstraction looks like: not hiding everything, but hiding implementation details that consumers should never have needed to understand in the first place.
⭐️ From Sponsor: Take the Airflow AI Crash Course
Data engineering is evolving, and Orchestrate Everything on September 16 is bringing together industry leaders to explore what’s next:
Hear from the data teams at Lyft, Wix, Ramp, and more
Learn practical MLOps, AI inference, and context engineering patterns
Prep for the AI Orchestration certification and take the exam for free ($150 value)
👉🏼 Register HERE.
(This message is sponsored by Astronomer.)
In-House LLM Serving at Netflix
📖 Topic: AI and Data Engineering
🧠 Level: Intermediate
Summary: Once an organization starts running enough LLM workloads, model serving begins to look less like an ML experiment and more like another distributed platform that needs APIs, deployment strategies, caching, version management and observability.
Netflix describes how it incorporated LLM serving into its existing Model Scoring Service rather than creating an entirely separate AI platform. The team uses NVIDIA Triton as part of its serving stack and selected vLLM as the primary LLM serving engine after benchmarking alternatives. An OpenAI-compatible HTTP interface sits in front of the platform, which gives application teams a familiar API and makes moving workloads between hosted and internally served models less disruptive.
The post also gets into the things that tend to appear only after going to production: model startup time, shared model storage, Triton/vLLM version compatibility, red-black versus versioned deployments, merged operational metrics, and CPU bottlenecks introduced by constrained decoding even while GPU inference itself is efficiently batched.
💡 Why is this relevant for DEs?
The API abstraction is a pattern we already know from data platforms. Consumers talk to a stable interface while the platform team retains the freedom to change the implementation underneath. Today that implementation might be a hosted model; tomorrow it could be a self-hosted model running through vLLM.
Self-hosting an LLM is not primarily a “download model and attach GPUs” problem. Artifact distribution, cold starts, rollout strategy, capacity, monitoring and version compatibility quickly become the bigger engineering concerns.
The constrained-decoding example is particularly useful. Netflix found that per-request CPU-side work could become a bottleneck even when GPU inference was well optimized. It is a familiar distributed-systems lesson: after optimizing one part of the pipeline, the bottleneck simply moves somewhere else.
As DE teams increasingly own AI infrastructure alongside batch, streaming and analytics platforms, model serving is becoming another workload whose data movement, observability and reliability we need to understand.
AI governance is now an engineering problem
📖 Topic: Data and AI Governance
🧠 Level: Beginner
Summary: AI governance often sounds like something that belongs in policy documents, review boards and compliance meetings.
This article makes a much more practical argument: once AI starts influencing production decisions, governance becomes an engineering problem.
The examples are familiar even outside AI, a production change gets approved but nobody can reconstruct exactly why it was made; an automated tool makes several changes during an incident but there is no useful decision log afterward; or a multi-step workflow fails halfway through and nobody has designed how to reverse the steps that have already completed.
The author boils this down to three questions: Who owns the outcome? Can we reconstruct what happened? Can we safely undo it?
Those questions apply just as well to AI-assisted data pipelines as they do to software delivery.
💡 Why is this relevant for DEs?
Data engineers already work in highly governed environments. Access controls, lineage, approvals, auditability and data retention are normal parts of our platforms. AI introduces new decisions that need the same level of traceability.
Logging the final output is not always enough. If AI recommends a schema change, transformation or production action, being able to understand what was recommended, what context influenced it and who approved it can become important during an incident.
Rollback should be designed before automation is given more authority. A workflow that performs five actions successfully before failing at step six is very different from a transaction that simply rolls back everything.
The larger lesson is useful beyond AI: governance works best when it is part of the engineering workflow, not something engineers are expected to remember to do after the work is complete.
Indexing the Data Lake for Online Point Queries: Random Access Parquet (RAP)
📖 Topic: Data Lake Architecture & Parquet
🧠 Level: Intermediate
Summary:
We normally think of Parquet and data lakes as being great for analytical workloads, while low-latency point lookups belong in systems such as key-value stores or serving databases. Spotify’s Random Access Parquet (RAP) work challenges that assumption.
The idea is fairly simple: maintain an external index that tells the system where a particular key lives in the data lake. Once the relevant Parquet file and location are known, the reader can use Parquet metadata to narrow things down further and fetch only the small portion of the file it actually needs from object storage.
That changes the access pattern significantly. Instead of starting a distributed query and searching through a large dataset to find one record, the system already knows where to look before it touches the file.
What makes this particularly interesting is that the underlying dataset remains regular Parquet. Analytical workloads can continue using the same files, while selected access patterns can be optimized for much faster point lookups.
💡 Why is this relevant for DEs?
It challenges a common architecture decision. When an application needs low-latency access to data in the lake, our first instinct is often to copy that data into Redis, Cassandra, DynamoDB or another serving database. RAP shows that, for the right workload, indexing the existing data may be another option.
Access patterns should influence architecture. Analytical scans, range queries and point lookups have very different requirements. RAP is a good example of optimizing specifically for how the data will actually be accessed instead of expecting one query pattern to handle everything efficiently.
The Parquet fundamentals matter. File layout, row groups, metadata and minimizing unnecessary reads can have a significant impact on performance. RAP takes these ideas further by using an external index to get directly to the relevant data.
There is a cost angle too. If an application needs one record, launching a distributed query engine and reading significantly more data than required may be unnecessary overhead. Reducing the lookup to a small number of targeted object-store reads can make the serving path much more efficient.
The broader takeaway is simple: before introducing another datastore, first ask whether the data you already have can be accessed differently.
The Technology Adoption Curve, Twenty Years On
📖 Topic: Career Development
🧠 Level: Beginner
Summary: Data engineering has no shortage of new technologies to learn. The difficult skill is figuring out which ones actually deserve our attention.
Looking back at twenty years of technology adoption, InfoQ traces how ideas such as Agile, cloud, DevOps, Kubernetes, microservices and machine learning moved from early experimentation into mainstream engineering. Some became foundational. Others were over-applied before teams eventually found more sensible boundaries around when to use them.
The article places AI engineering and agentic systems near the early-adopter end of that same curve today. The useful takeaway is less about predicting which technology will win and more about developing the judgment to separate an interesting experiment from something your production platform genuinely needs.
💡 Why is this relevant for DEs?
We work in one of the most trend-heavy areas of engineering. Hadoop, Spark, data lakes, lakehouses, data mesh, streaming, semantic layers and now agents have all arrived with claims that they would reshape the stack. Understanding adoption maturity helps us avoid designing around hype.
Being an early adopter and being an early majority user require different thinking. An early adopter may accept operational pain in exchange for learning or competitive advantage. A production platform team usually needs a much stronger reason.
The underlying problem usually survives the technology. SOA became microservices; DevOps evolved into platform engineering; ML infrastructure is becoming AI infrastructure. Learning the underlying engineering problem tends to age better than memorizing whichever tool currently solves it.
Technology judgment becomes more important as you become senior. A senior DE or architect is increasingly expected not only to know what is new, but to explain whether the organization should adopt it now, experiment with it, or simply wait.
💎 Open Source Gem
RisingWave: Event Streaming for Agentic AI
What’s new / why now:
RisingWave has been around for a while as a streaming database, but the 3.0 release line is a meaningful expansion of what the project is trying to be. RisingWave now positions itself as an event-streaming platform for agentic AI, combining CDC and event ingestion, incremental SQL computation, low-latency serving and an open Iceberg-based storage path. RisingWave 3.0 also adds capabilities around vector search and deeper Iceberg integration, while the project exposes familiar PostgreSQL interfaces and tooling for AI/agent access.
This is interesting because the “real-time context for agents” problem is starting to look very similar to a problem data engineers already know well: continuously ingest changing data, maintain derived state, and serve the latest result with low latency.
💡 Why is this useful for DEs?
The programming model is SQL-first. Teams that do not want to build every streaming transformation as application code can maintain continuously updated results using materialized views and familiar relational concepts.
It potentially collapses a few layers of the streaming stack. CDC/event ingestion, incremental computation and serving can happen in one system. That does not mean every Kafka + Flink architecture should suddenly be replaced, but it is worth evaluating for new workloads where the operational overhead of multiple systems is hard to justify.
The Iceberg integration matters. Fresh operational state and durable analytical history do not have to become two completely disconnected architectures.
Agentic AI may be the current positioning, but the underlying engineering problem is broader: keeping derived data continuously fresh and immediately queryable. That has plenty of non-AI applications as well.
👉 GitHub: https://github.com/risingwavelabs/risingwave
💡 DE Tip of the Month
Row Counts Are Not Data Quality
A pipeline loads 10 million rows yesterday and 10 million rows today. Everything looks healthy.
Except today’s customer_id is null.
Row count checks are useful, but they mainly tell us that something arrived. They don’t tell us whether the data still makes sense.
For important datasets, combine volume checks with a few checks that actually understand the data:
Null percentage on critical columns : especially identifiers, dates and fields used in downstream joins.
Duplicate rates on expected keys : row counts may look perfectly normal even when the same records were loaded twice.
Min/max dates and timestamps : one of the quickest ways to detect stale data, bad filters or an upstream feed that stopped refreshing.
Distribution changes : a sudden shift in an important category or measure may reveal a problem that schema validation will never catch.
Referential integrity : if two datasets are expected to relate to each other, make sure that relationship still holds.
Source-to-target reconciliation : for business-critical measures, compare what entered the pipeline with what eventually came out.
You don’t need hundreds of checks.
Five checks that actually understand the dataset can often tell you more than fifty generic checks applied everywhere.
Another lesson from production: catch the problem as close to the source as possible. Discovering bad data after it has gone through three transformations, two aggregations and seven dashboards makes troubleshooting much harder.
A successful job run only tells us that the code executed.
Data quality asks a different question: Can someone safely use what the pipeline produced?
📊 Community Poll
Until the next one.
ℹ️ About Data Engineer Things
Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.