Data Engineer Things Newsletter - Community Spotlight Edition (August 2026)
Near-Infinite Velocity: On Agents, Standards, and What Never Changes
Hello, fellow Data Engineering enthusiasts!
For this issue of Community Spotlight, we chatted with Bryan Brandow, who’s a leader, mentor, data enthusiast, problem solver, and currently a Data Engineer at OpenAI.
Bryan spoke at the most recent DET Bay Area Meetup on July 23. He delivered an insightful talk on Data Engineering in the AI Era, in which he explained how maintaining data standards is crucial for building durable and maintainable systems and advocated for "harness engineering," where team standards are baked directly into the system using a structured knowledge store.
This interview explores why the fundamentals of data engineering matter more than ever in the agent era. Bryan walks through harness engineering at OpenAI, and explains why near-infinite velocity makes good design decisions the new bottleneck.
Hope you enjoy reading this as much as we did putting it together!
- Eddy, Shubham & Sugandhi
Spotlight: Bryan Brandow
“Now we have near infinite velocity and the bottleneck is only being able to decide what we want to build. The price of building the wrong thing is higher than ever because a bad design decision can propagate so quickly.”
For those in the DET community that may not know you yet, could you briefly introduce yourself?
My name is Bryan Brandow and I’m a Data Engineer at OpenAI. I’ve spent my entire career in Data Engineering, perhaps most notably for 11 years at Facebook/Meta. I’ve worked through the entire stack from logging, core tables, aggregate layers, and dashboards. I spent a good portion of my career specializing in data visualization with tools such as MicroStrategy and Tableau. I have strong opinions (weakly held) about all things data, but enjoy the debate and discussion.
Your first data work involved hand-building pipelines with the tooling that came bundled with SQL Server, in an era with no cloud warehouse, no orchestration frameworks as we know them now, and no version-controlled transformation layer. You're now shipping agents that do that work for you. Across all of that, what has actually stayed the same about doing this job well?
Yeah, my early days are pretty funny to think about now! My first task at my very first job was loading CSVs that got FTP’d to us from third party data providers. Data quality and the handling of exceptions (skip or fail) is probably the most consistent thing that never changes. I remember having to manually fix bad characters in the CSV and retry the job until it worked. The biggest difference is just the scale of data. The first data warehouse I worked with was only about 10gb of data, and the main ETL jobs were only a few steps. As complexity grew, the need for better tooling and platforms increased with it.
💡 Editorial Note: FTP
FTP stands for File Transfer Protocol. It's an old but simple way to move files between two computers over a network. One computer acts as the server holding the files, and you connect to it from your computer to upload or download stuff. It was hugely popular for things like publishing websites (uploading your HTML files to a web host) and sharing large files before cloud storage existed.
One caveat: plain FTP sends everything unencrypted, including your password, so, today, people use secure versions like SFTP or FTPS instead.
You've moved between management and hands-on IC work more than once, going from leading a large org to individual contributor work on harder technical problems, and later from an engineering director role into building directly again. What draws you back to the work itself, and how do you advise people thinking about that move?
I’ve always just asked how I can best help the team, whatever that role may be. I’ve never asked to be a manager, but have asked to be an IC. I personally enjoy IC more because the engineer in me loves solving problems. The feeling of satisfaction I get from fixing or enabling something for someone is really amazing. I like having direct control of the outcomes and it enables me to form strong opinions on tooling or processes that I can then turn into changes that can help everyone.
I think it’s natural for people to think that becoming a manager is taking an important step up in the career ladder, and in a lot of companies, that’s probably true. But at companies like Facebook and OpenAI, this wasn’t the case. Anyone can have influence in a company where ideas win on merit and not authority. And frankly, those are the kind of companies I want to work at anyway.
I do think everyone should try it if they have the opportunity. The experiences definitely made me a better IC because of the development of soft and organizational skills that are necessary. It also strengthened my ability to multi-task and ramp up on context quickly since as a manager, you are constantly doing that as you run from meeting to meeting.
My advice to people is to recognize that management and IC are different roles with very different skillsets. It can be hard for a first time manager to let go of the details and work through others. And it can be hard to have difficult conversations and do performance management. Being a great manager has a lot of overlap with being a great therapist. Some people enjoy it and are really good at it, but it’s not for everyone.
As Pillar Lead, you helped set Facebook's overall data engineering vision, culture, and recruiting standards alongside the Growth and Ads leads. What did you learn about what makes a DE org healthy at scale, and what translates to smaller teams?
I’m a very big proponent of standards and their importance (see my July DET Meetup talk!). I saw first hand how one person’s personal preferences would directly lead to how entire orgs of data engineers would operate in the future. I could look at Instagram, Facebook, Messenger, Growth and track all of those lineages back to a single individual that made key design decisions back when the team was small. In some ways, those differences were in support of the product or functional requirements of those business groups. But the downside was that when we needed to work across those teams, data integration was unnecessarily challenging. It’s the reason I harp on the importance of standards, even when they seem trivial. They just have such a powerful multiplicative effect over the long run.
I think a healthy organization has strong communication and debate. It’s not that any of those standards are sacred, and even though I have my own strong opinions, I don’t believe there is a correct answer. Discussion and evolution over many iterations is key. An organization that can continuously improve together, not in silos, is in a really strong position.
💡Editorial Note:
In his talk at the July 2026 DET meetup, Bryan Brandow explains why foundational data engineering standards remain critical for system longevity in the age of AI.
Watch the full talk here:
Your writing has gone from technical deep dives read by humans to standards precise enough for an agent to follow. Does public technical writing still pay off for someone starting out, now that so much content is generated?
One thing that has always helped me out tremendously is to actively monitor support channels/forums and try to reply to everything. If I don’t know the answer, I figure it out. For one, I’ve always believed in “help others the way you want to be helped”, and I greatly appreciate it when someone goes through that much effort for me. But it’s also an incredibly efficient way to learn. You’re researching the exact problems your partners are facing and building a strong reputation and relationships.
After doing this for a while on the MicroStrategy forums around 2010, I decided to start a blog so I could have some artifact to link to people when the same problems would come up. I got really into it, posting multiple technical articles per week, and eventually just put up an email address for people to directly ask me questions. I learned so much from researching and refining topics and engaging in the comments. I met so many incredible people through the community at meetups and conferences.
I do think it is still incredibly valuable. I was never worried about whether or not anyone would read it, because I was doing it for myself. It was a great personal resource for me to search and go back to. And even if you get zero traffic, it’s a fantastic artifact on a resume. My eventual hiring manager at Facebook told me that my blog was the deciding factor for them to hire me because they could easily see long form, deep dive, technical explanations and articles that would never be feasible to assess in an interview.
Early in your career you built your own object migration approach in MicroStrategy and presented it at MicroStrategy World. That idea ended up making its way into the product. Walk us through how that unfolded and what you took away from it.
I’ve always had a knack for pushing vendor software past its limits by thinking creatively. I play a lot of video games, and when I get stuck in a game, I ask myself, “What are the variables? What is my goal and what’s stopping me?” and work backwards. That same approach works very often when working with vendor software. They add so many features to meet such a wide range of needs that you can often use those features via interactions they didn’t intend, to unlock more potential.
In this particular case, when migrating objects from Dev to Prod, you would have to copy them in a particular order and piece by piece. It was laborious and error prone. It would automatically copy dependencies and I noticed that sometimes I’d get lucky and have a large nested change that did most of the work. Then I thought, “the best migration is to copy one object that takes everything with it,” and it occurred to me that there was this type of prompt that let the user pick any arbitrary object. The feature was intended to support things like “pick a filter” or “pick a metric”, but instead, I just put everything I needed to migrate in it, like a “package”. Then I could just drag that one item and it would take everything with it. It also gave the migration an artifact that could be tracked and audited to ensure only what you intended to migrate got migrated. It ended up solving a real pain point for us and I got a lot of great feedback when I shared the blog post and presented it at their annual conference. A year or so later they introduced the same feature as a first class, standalone product.
Before joining OpenAI, you were leading teams integrating LLMs into existing data workflows, back when that was still uncharted territory. What did that early work teach you, that shaped how you approach agents today?
LLMs had just burst onto the scene, and my company was understandably cautious. We worked in the financial space, so had to be very careful around new technologies, but the benefits of LLMs were obvious from the start. To unblock the project and instil confidence with our security team, we created a chat wrapper around the OpenAI API and basically re-created an internal version of ChatGPT. This gave us the ability to monitor and filter prompts to make sure an employee wasn’t sending sensitive data to a third party.
Of course today, OpenAI has excellent support for sensitive data, including Enterprise ZDR (zero data retention) and administrative reports and tools. Shortly after I left that company (to join OpenAI!), they deprecated the homegrown system and adopted the ChatGPT Enterprise tools directly.
In that project, I did learn a lot about RAG (Retrieval Augmented Generation) and how we could improve answers by giving the models context. That stuff largely happens behind the scenes now via Plugins and as the models have significantly increased their context windows and capabilities. It showed me the difference in the quality of outputs that you get when providing the model access to company information to get more relevant results.
At the DET Bay Area meetup, you argued that the core tenets of good data engineering matter more than ever in the AI era, and walked the audience through harness engineering: AGENTS.md files, data modeling contracts, and a cleanup agent called TIE-D. For readers who missed it, what is the harness, and why start an agent on cleanup work rather than net-new pipelines?
A harness is the software layer around an AI model that gives it the context, tools, rules and execution loop needed to do useful work. The model itself can reason and generate outputs, but the harness turns that capability into an agent that can operate on a real system. So when I talked about OpenAI’s blog post, “Harness engineering: leveraging Codex in an agent-first world”, I’m referring to how you build up support for the model to operate in that layer in an efficient way. I focused on how to structure context and skills for the model, but it can also include the development of tools.
As for net-new vs cleanup, I think that was more of a “low hanging fruit” situation. Where things need to be cleaned up is pretty straightforward. We know what needs to be done, and the Agent can figure out how to do it. Cleanup work is also important because in addition to your AGENTS guidance, models are also going to pull from nearby examples. So legacy approaches or non-standard implementations can propagate like weeds. But when you’re creating something new, you need to be more involved in the decisions. I wouldn’t hand over the full architecture, schema, and design choices to an Agent (yet), but I would hand over the implementation. I think Codex is in a really good place right now such that I can engage on the designs and let it implement the details.
When I do want to run longer form net-new development, right now I’m leaning heavily on ExecPlans. Here are a few posts that talk about them: Using PLANS.md for multi-hour problem solving and Modernizing your Codebase with Codex. Think of it as a materialized version of Plan mode in Codex. We can talk through all of the design decisions and then I can entrust Codex to go to work on the full end to end implementation. It can build the tables, write and test the code and validate the data. I can check in along the way, review the PRs and after they merge, it can then handle the backfills. While it’s working, I can be designing and getting the next one ready and they can work in parallel. The amount of productivity increase is really incredible! It also leaves behind a durable lot of not just what it completed, but a documentation trail for why the decisions were made.
Most teams reading this don’t have OpenAI’s headcount or infrastructure. If a small data team wanted to point an agent at their own pipelines, starting with cleanup work like you described, what is the first thing they need to write down, and what can wait?
I think it starts with instructing the model on how it should build a pipeline. Most likely, you’ve already got some kind of framework or template that you use, so document that. We had a wiki page with instructions on how to use our internal ETL framework, so I started out by having Codex turn that into an `.md` file.
Then, as you use it, you iterate on what it got wrong. Anytime it doesn’t one-shot a pipeline and you have to correct it, add that instruction into the `.md` files. You can start out with just a single AGENTS.md and as it starts to get large, break it up into the Knowledge Store volumes I described in my talk. I think it’s best to discuss these standards with your team first and give everyone the opportunity to provide input. This will become the canonical way you build your data pipelines. Once you have that, then you can build an agent like TIE-D that can apply those standards broadly as they evolve over time.
Also consider if an ExecPlan would be a better fit than a running Agent. For example, I’ve found deprecating a table and all of its downstreams to be way more efficient for an ExecPlan than for TIE-D, because the ExecPlan can scope the entire amount of work, break it into phases, and burn it down, whereas TIE-D doesn’t have that high order planning routine and chips away at lots of smaller tasks with more loops and hence more resources.
💡 Editorial Note: TIE-D
TIE-D is an autonomous Data Engineering (DE) agent utilized at OpenAI to handle maintenance tasks and cleanup.
Key features of Tidy include:
Purpose: It automates menial but essential “cleanup” work that is often too time-consuming for engineers to do manually, preventing the accumulation of technical debt and “crust” in the system.
Evergreen Tasks: Tidy operates on the concept of “evergreen tasks,” which are high-level goals. Instead of being pointed at every individual issue, the agent actively searches for work, such as deprecating old tables, standardizing sensors, or removing redundant data quality (DQ) checks, and handles them independently.
You can learn more about how TIE-D functions as a tireless maintenance assistant by watching Bryan’s talk at the July 2026 DET Bay Area Meetup.
You've been growing OpenAI's Data Engineering and Analytics Engineering team. Given everything agents now handle, what do you screen for in candidates, and what makes someone stand out?
We focus a lot on design and architecture. I think the fundamentals of Data Warehousing will always be a critical skill. For a while, our industry favored software engineering skills more because that was the difference in being able to produce data or not. Frameworks were commonly celebrated because of the velocity that they enabled. But now we have near infinite velocity and the bottleneck is only being able to decide what we want to build. The price of building the wrong thing is higher than ever because a bad design decision can propagate so quickly. So I think you can never go wrong with mastering good data modeling foundations.
Key Takeaways
AI agents make writing pipeline code nearly instantaneous, shifting the core bottleneck to system design. Sound data warehousing and modeling fundamentals are more critical than ever, as poor design choices now propagate at agentic speed.
Turning LLMs into reliable data agents requires a “harness”, which is a structured execution environment provided with explicit rules and contexts. Translating tribal knowledge into documented standards prevents model errors and eliminates long-term integration debt across teams.
Start agents on cleanup and deprecation tasks to remove legacy patterns before they contaminate model context. For complex net-new features, engineers should direct high-level architecture while assigning end-to-end execution (coding, testing, backfilling) to structured execution plans.
Despite shifts from manual CSV uploads to automated agents, data quality, exception handling, and creative problem-solving remain unchanged. Master these foundations, document technical insights publicly, and treat the IC and management tracks as complementary skill sets for overall growth.
Community poll
💬 How to stay connected
ℹ️ About Data Engineer Things
Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.
Thanks for reading Data Engineer Things! Subscribe for free to receive new posts and support our work.