Data Engineer Things Newsletter - Data Pulse Edition (July 2026)
Netflix's multimodal AI for video search, Context layer behind Spotify’s data assistant, Harness engineering for coding agent users, Reddit data tech stack (120M+ DAU)
Hello Everyone,
This is Ananda again, writing from Philly. Every few years, data engineering gets a new “hard part.” First, it was scale. Then it was real-time. Now it's context and the harnesses we build around it, the scaffolding that lets an AI system, or an agent acting on its behalf, actually reason about a warehouse full of tables instead of just querying it blindly.
That raises a common question this edition keeps circling from different angles: what does it actually take to make a data system trustworthy enough for a person, a model, or an agent to rely on it?
For some teams, that means deliberately building context, curating the business knowledge an AI system needs rather than assuming raw schemas will speak for themselves. For others, it means treating schema change as a coordination problem that spans systems, not a single isolated edit, or reconciling outputs from models that interpret the same data in entirely different ways and on entirely different timelines. And for teams building on coding agents, it means designing the feedback loops that let an agent's output be trusted before it ships.
This edition spans a wide range, from large-scale data infrastructure choices to the less visible work of context, schema, and harnesses that make AI-assisted systems reliable.
– Ananda
📚 Data Pulse
The Context Layer Behind Spotify’s Data Assistant
📖 Topic: Data and Context Engineering
🧠 Level: Intermediate
Summary: Spotify built an AI data assistant (Vedder) to help teams query its 70,000+ dataset warehouse in plain English, recognizing that raw schemas alone can't capture the business context an LLM needs to write reliable SQL. The core innovation is the "cluster" model: domain-owned bundles of relevant tables, vetted question-SQL example pairs, and supplementary documentation, all curated by the data experts who actually understand that slice of the business.
💡 Why is this relevant for DEs?
Semantic/context layer as infrastructure: This reframes the "semantic layer" problem data engineers already wrestle with (documenting tables, defining metrics, capturing knowledge) as a first-class system component, structured, versioned, health-monitored context (datasets + vetted query pairs + docs) that directly feeds a production AI system.
Curation over automation validates governance investment: The finding that only 12.5% of auto-mined query history was usable as training examples is a strong data point for engineers arguing against "just scrape the logs" approaches to documentation/metadata. It shows human domain review remains the bottleneck worth investing in, not a step to skip.
Automated Schema Evolution: Pinterest’s Next-Gen DB Ingestion Framework
📖 Topic: Databases and Data Engineering
🧠 Level: Intermediate
Summary: Pinterest built an automated schema evolution framework for its CDC-based ingestion pipeline (Kafka, Flink, Spark, Iceberg), treating schema changes as a cross-system contract rather than an atomic operation, since a single upstream change requires coordinated updates to generated code, storage schemas, and bootstrap logic across every stage. The system deliberately restricts automation to additive, backward-compatible changes (new columns, numeric precision widening) and pushes riskier changes like type narrowing or primary key edits into manual migration paths, using both push-based (DDL-triggered) and pull-based (daily diff) detection to catch drift.
💡 Why is this relevant for DEs?
Reusable pattern for a universally painful problem: Schema drift breaking downstream pipelines is one of the most common CDC/ELT failure modes; the staged convergence model (schema → code → data) and the “additive-only” safety boundary are directly adaptable design patterns for anyone running Debezium/Kafka/Flink-style ingestion, not just Pinterest-scale teams.
Concrete answer to “when is eventual consistency okay?”: The SLA-based approach tolerating temporary nulls in Iceberg while Flink/Spark catch up is a well-reasoned tradeoff data engineers can cite when deciding whether their own pipelines need atomic schema changes or can accept a bounded convergence window, which has real cost/complexity implications.
Practical techniques for ambiguous DDL: The binlog-based audit trail for resolving ambiguous create table diffs (rename vs. drop-and-add) addresses a specific, recurring headache in schema-diffing tooling and is a useful reference for data engineers building or evaluating similar catalog/DDL-tracking systems.
How Netflix is Using Multimodal AI to Power Video Search
📖 Topic: Data Engineering and Multimodal AI
🧠 Level: Advanced
Summary: Netflix built a multi-modal search system to let editorial teams find specific moments across 2,000+ hours of raw footage per season, running an ensemble of specialized AI models (character recognition, scene classification, dialogue transcription) rather than one generalist model, since each excels at its specific task but produces incompatible output formats and misaligned time intervals. The core engineering challenge was fusion: a three-stage pipeline first persists raw model outputs to Cassandra with zero transformation, then an offline job normalizes everything into one-second temporal buckets and merges overlapping annotations into unified records via upsert operations, and finally indexes those fused buckets into Elasticsearch as nested documents for querying.
💡 Why is this relevant for DEs?
Data architecture for heterogeneous data sources: The decoupled three-stage pipeline (raw persistence → offline fusion → serving index) is a directly transferable architecture for any data engineer dealing with heterogeneous data sources on misaligned timelines or schemas, not just video, but IoT sensor streams, log correlation, or any scenario where multiple producers write disparate formats about the same entity.
Concrete tradeoff reasoning: The explicit choice of throughput-over-freshness (offline batch fusion vs. real-time) and the upsert-with-composite-key approach to idempotent, incremental enrichment are patterns data engineers can reuse directly when designing their own late-arriving-data or multi-model-output merge jobs, plus a clear worked example of why bucket/grain size (1-second buckets) is a real design decision with volume tradeoffs (7.2M buckets from one archive).
Data persistence justified by workload: Using Cassandra for high-throughput write-heavy ingestion and Elasticsearch for hybrid keyword+vector query serving each database used for what it’s actually good at is a clean case study for data engineers evaluating when to split storage layers by access pattern.
Reddit Data Tech Stack
📖 Topic: Data Infrastructure & Engineering
🧠 Level: Beginner
Summary: Reddit runs its massive data platform (120M+ daily active users, billions of comments, 500+ Kafka brokers processing tens of millions of messages/second) primarily on AWS, with GCP added via a commercial partnership that brought in BigQuery and Vertex AI. Kafka serves as the central event backbone, fanning data out to Flink (branded “Snooron,” built for real-time content safety rules) for stream processing and to Spark for batch reporting pipelines that feed Druid for real-time analytics; Debezium handles database change-data-capture into Kafka. Data is orchestrated with Airflow, stored raw in S3, and warehoused in BigQuery after migrating from Redshift.
💡 Why is this relevant for DEs?
Real-world reference architecture at extreme scale: shows a concrete, battle-tested pattern (Kafka → Flink/Spark → Druid/BigQuery, with Debezium for CDC) that DEs can benchmark their own designs against, especially for systems that need real-time processing alongside batch analytics.
Illustrates pragmatic multi-cloud trade-offs: Reddit’s AWS-for-core-infra and GCP-for-BigQuery/Vertex-AI split is a useful case study for DEs evaluating when cloud choices are technical vs. business-driven and how to run hybrid-cloud data platforms in practice.
Concrete tool-selection rationale: explains why specific tools were chosen for specific jobs (Flink for low-latency safety rules, Spark for batch reporting, Druid for fast aggregated queries, Airflow for orchestration), giving DEs a framework for matching tools to workload characteristics rather than defaulting to one-size-fits-all.
Harness engineering for coding agent users
📖 Topic: Agentic Engineering
🧠 Level: Intermediate
Summary: The author proposes a mental model for "harness engineering" everything around a coding agent besides the model itself built on two control types: feedforward "guides" that steer the agent before it acts, and feedback "sensors" that let it self-correct after, each of which can be computational (fast, deterministic tools like linters and tests) or inferential (slower, probabilistic LLM-based judgment).
💡 Why is this relevant for DEs?
Data pipelines already have sensors: Schema validations, data quality checks, and lineage tools map directly onto the feedback controls, meaning data engineers can bolt on harnesses to AI-assisted pipeline code fairly cheaply compared to teams without that tooling.
“Continuous drift” is a core data engineering problem: schema drift, stale data, degrading freshness SLAs, and upstream dependency changes are exactly the kinds of ambient, out-of-lifecycle issues that need dedicated sensors. The framework provides a language for building agent-monitored drift detection into data pipelines.
The “behavior harness” gap is evident for data transformations: verifying that a data transformation is semantically correct is hard when the tests themselves may be AI-generated, which matters a lot in data engineering, where silent logic errors in transformations can corrupt downstream data without ever throwing an error.
The Human Layer of Technical Work
📖 Topic: Career Development
🧠 Level: Beginner
Summary: “Human skills” like communication, empathy, and psychological safety matter as much as technical expertise for data professionals, especially as AI takes on more of the technical work itself, since AI can imitate connection but not genuine human presence. Human skills are as learnable and systematic as technical skills, and they can be developed through small, repeatable practices (pausing before reacting, asking a genuine question before presenting solutions) rather than treating them as innate traits. Most workplace friction technical professionals experience (defensive colleagues, meetings gone wrong, poorly received rollouts) has a human root cause, not a technical one.
💡 Why is this relevant for DEs?
Concrete tactics for cross-functional work: DEs constantly translate technical concepts (schemas, data quality issues, root-cause explanations) for non-technical stakeholders; the advice on simplifying without undermining a colleague and on reframing “fix the dashboard” requests as source-data issues is directly applicable to that daily friction.
Frames soft skills as a learnable, iterative system: appeals to a technical audience by treating communication and psychological safety like any other skill to practice and measure (small repeatable habits: pausing, asking questions, checking in after a tense exchange), making it easier for DEs to actually act on rather than dismiss as vague advice.
🗓 DET Bay Area Meetup on July 23
Join us at Microsoft's Mountain View office on Thursday, July 23rd, for an evening of technical talks, networking, and good food.
When: 5:00 PM to 8:30 PM on Thursday, July 23rd
Where: Microsoft’s Mountain View office
👉🏼 RSVP
(🎤 Interested in speaking at our meetups or online webinars? Submit talk proposals here.)
💎 Open Source Gem
OLake Go: Efficient, quick, and scalable data ingestion for real-time analytics.
OLake Go is a high-performance platform that replicates data from operational datastores (PostgreSQL, MySQL, MongoDB, Oracle, Kafka, DB2, MSSQL, S3) into open lakehouse storage, writing either Apache Iceberg tables (with catalog support for AWS Glue, Hive Metastore, and REST/JDBC catalogs like Nessie, Polaris, and Unity Catalog) or Parquet files on object storage such as S3, MinIO, and GCS. It supports Full Refresh, Incremental Sync, and Change Data Capture depending on the source, with core capabilities including parallelized chunking for faster large-dataset scans, stateful/resumable syncs that recover from interruptions without full resyncs, and per-job configurable connection limits.
💡 Why is this useful for DEs?
Faster, more reliable data pipelines: parallelized chunking speeds up large full-table scans, while stateful/resumable syncs mean a crash or network blip doesn’t force a costly resync from scratch, cutting both processing time and manual intervention.
Cleaner data with less pipeline engineering: automatic primary-key deduplication, schema evolution, and JSON normalization (flattening nested fields into columns) handle work DEs would otherwise hand-code, while two-phase commits prevent duplicate writes or inconsistent state on the Iceberg side.
Flexibility to fit existing lakehouse architecture: support for multiple Iceberg catalogs (Glue, Hive Metastore, REST catalogs like Nessie/Polaris/Unity) and both Iceberg and plain Parquet outputs means DEs can plug it into whatever stack they already run, rather than being locked into one catalog or storage layout.
👉 GitHub: https://github.com/datazip-inc/olake
💡 DE Tip of the Month
Read Replica vs. CDC: Choosing how to offload reads from your source database
Read replicas answer "how do I stop hammering my primary with more of the same kind of query?" CDC answers "how do I ask fundamentally different questions of this data, cheaply, with history?" They're not mutually exclusive; increasingly, teams run both from the same source, routing each consumer to whichever pattern fits its query shape.
📒 Rules of thumb
When to choose a Read Replica
Same-engine fit: Consumers need identical SQL dialects, indexes/transactional guarantees to the source (OLTP-shaped tools, point lookups) with near-zero freshness lag.
Low complexity, moderate scale: Minimal new tooling/schema to maintain beyond monitoring replication lag, cost-effective only while scale stays moderate, since replica compute tracks primary engine pricing.
When to choose CDC into a cheaper store
Analytical fit, decoupled cost: Workload is analytical (aggregations, joins, trend analysis) where OLTP engines underperform, CDC lets you pay scan-optimized storage/compute rates instead of OLTP-tuned pricing.
History and fan-out, less source pressure: CDC’s append-only change stream gives you history/time-travel and can feed multiple consumers (warehouse, search, cache, ML) from one pipeline, without adding replication load on the source as consumer count grows.
📊 Community Poll
Until the next one.
ℹ️ About Data Engineer Things
Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.
It’s a pleasure to be part of this list thank you for sharing @Colette Molteni and I article.🙏
Thank you for highlighting "The Human Layer of Technical Work" article I did in partnership with Pipeline to Insights.