Field Notes
Notes on data-powered systems, AI workflows, architecture, automation, and the practical work of turning messy problems into useful software.
2026-07-31
Agentic development is turning data engineers into software producers—building locally, proving value with users, and distributing through platforms the enterprise already trusts.
Read field note
2026-06-07
Dashboards are wine lists: accurate, organized, and silent about what any of it means. What stakeholders always wanted was the sommelier — the story that turns data into a decision. AI can finally play that role: it reads datasets it has never seen, finds meaning across sources, and writes the narrative. And once you have table service, the legacy BI tool starts looking like what it always was — a narrow web app. From level-1 metrics to AI storytelling, and from dashboards to full-stack data apps.
Read field note
2026-05-07
Most data work doesn't fail because one runner was too slow — it fails because the track is too long. Agents change the shape of the race by shortening the distance between code, deployment, infrastructure, logs, databases, and the BI layer. The shift matters most in data engineering, where Fabric is finally making the messy middle agent-operable.
Read field note
2026-04-22
Data engineering used to mean keeping the plumbing running. Agentic coding environments are changing that — giving data teams a new control room that operates across code, cloud, data, and interfaces, and shifting the role from pipeline maintainer to general contractor.
Read field note
2026-03-29
We built the data warehouse like a museum — curated, organized, and full of artifacts from the live systems that actually run the business. That worked for analytics. But agents are not archivists. They are operators. And operators need to work in the live city, not a copy of it.
Read field note
2026-02-01
Data work used to be simple—isolated queries and scripts with built-in undo buttons made version control unnecessary. But AI agents have changed that, spreading changes across multiple files and layers faster than humans can track. This post explores why Git is shifting from a developer tool to a critical bridge between AI-driven automation and human oversight in modern data workflows.
Read field note
2026-01-11
SaaS succeeded by abstracting away infrastructure when building and running software was hard. That trade-off made sense for a long time. But as infrastructure becomes cheaper and AI lowers the cost of building custom solutions, the economics that justified renting platforms are starting to break down. This post explores why that shift is putting pressure on traditional SaaS models—and what it means for teams deciding what they should actually own versus outsource.
Read field note
2025-12-21
As data engineering matured, factories became the default. Assembly lines brought reliability when tools were weak, storage was expensive, and mistakes were costly. But the constraints changed, and the architecture didn’t. Today, we still build stacks by habit—pre-aggregating, orchestrating, and industrializing work before we understand it. In this post, I argue for an anti-stack mindset: start with workshops, use strong tools at query time, delay pipelines until they’re earned, and treat factories as an optimization—not a prerequisite. The goal of modern data engineering isn’t to keep the line running. It’s to deliver insights.
Read field note
2025-11-30
As organizations grow, they transform from solo musicians into full orchestras — and that's when the real challenge begins. Data doesn't stop being collected; it stops being aligned. Systems drift, meanings diverge, and interpretation fragments. Data engineering exists not to make the orchestra louder, but to keep it playing the same piece, in the same key, at the same tempo. In this post, I explore how coordination becomes essential at scale, why AI amplifies (but doesn't replace) this need, and how the discipline of data engineering becomes foundational leverage in the AI era.
Read field note
2025-10-29
Modern development has become a kitchen remodel — endless setup, plumbing, and wiring before anything gets cooked. Platforms like Replit flip that script, giving you a ready-to-go kitchen so you can focus on what matters: the meal. In this post, I explore how AI is becoming the logic layer — the brain of the kitchen — and why the real value now lies in the data that grounds it. Building for clarity means offloading logic to AI, anchoring it in a reliable database, and designing systems that serve ideas, not infrastructure.
Read field note
2025-09-23
CSV got us started, but it struggles at scale: bloated files, messy schemas, and painfully slow queries. Parquet flips the model—storing data in compact, labeled columns that query faster, compress better, and form the foundation of modern lakehouses. In this post, I use a Lego analogy to show why Parquet isn’t just a file format upgrade, it’s the workshop behind today’s analytics.
Read field note
2025-08-31
Modern data architecture can feel like alphabet soup: databases, warehouses, lakes, lakehouses, catalogs. With Microsoft Fabric making Delta Lake the default, it’s never been more important to understand how these systems fit together. In this post, I use a kitchen metaphor to break down the strengths and weaknesses of each—lunchboxes, buffets, pantries, and chefs—before looking at DuckLake, a new approach that puts metadata where it belongs: in a database. The goal isn’t hype, it’s clarity—so you can design an architecture that feeds your business real insight.
Read field note
2025-07-28
Most data projects don’t fail because of tools—they fail because of poor communication, overengineering, and ignoring real-world constraints. This post breaks down what data success *actually* looks like: delivering useful answers, on time, with minimal overhead. Learn how small, business-aware teams can ship faster, automate smarter, and scale simpler by focusing on impact, not architecture.
Read field note
2025-06-09
Pandas is a fantastic tool for small datasets and quick analysis but hits limits when scaling or persisting state. DuckDB fills that gap by combining SQL-native querying, persistent local storage, and high performance—allowing data engineers to build scalable, reliable pipelines on their laptop without spinning up clusters. This post explores the practical differences between Pandas and DuckDB, real-world use cases, and why DuckDB is the smarter tool for modern data workflows.
Read field note
2025-05-27
Traditional BI tools prioritize speed and ease but often sacrifice flexibility and customization. Today, the rise of AI, modular libraries, and instant cloud platforms like Replit empower data engineers to build highly customizable, interactive, and user-focused data experiences—without needing full-stack development expertise. This shift transforms BI from rigid, one-size-fits-all dashboards into composable, code-assisted data product kits that deliver tailored insights and enable narrative-driven storytelling. Discover how the future of BI is no longer a monolithic platform but a flexible toolkit that bridges data engineering and user experience seamlessly.
Read field note
2025-04-16
Constraints shape creativity in data engineering more than limitless resources ever could. From limited budgets and tight deadlines to technical and organizational boundaries, data teams constantly navigate tradeoffs that spark smarter, more pragmatic solutions. This post explores how real-world constraints drive innovations like Zero ETL, local-first engines like DuckDB, and integrated platforms like Microsoft Fabric. Instead of fighting limitations, successful engineers learn to embrace and design with constraints—delivering impactful, efficient data solutions that work within the messy realities of business.
Read field note
2025-03-31
Microsoft Fabric revolutionizes data workflows within the Microsoft ecosystem by unifying ingestion, transformation, modeling, and reporting into a seamless, serverless platform. It eliminates traditional Azure complexity and enables faster, more autonomous analytics with native Power BI integration, Lakehouses, notebooks, and real-time replication. While still evolving, Fabric dramatically improves productivity and collaboration by reducing tool fragmentation and infrastructure overhead—making it the most practical solution I’ve seen for getting data projects done in Microsoft environments.
Read field note
2025-02-05
Data modeling has been the backbone of structured analytics for decades, ensuring consistency, performance, and reliability. But with modern storage affordability, faster processing, and flexible BI tools, the necessity of rigid data models is evolving. This post explores when traditional modeling adds value—and when startups and agile teams can thrive by embracing more flexible, denormalized, or hybrid approaches. Learn how to balance structure and speed to deliver impactful insights without over-engineering your data pipeline.
Read field note
2025-01-17
Data engineering isn’t about building pipelines or managing infrastructure for its own sake. It’s about delivering clear, timely, and actionable insights that empower decision-makers. This post explores why stakeholders want insights—not raw data—and how data teams can focus on outcomes over technology. By understanding the organizational and technical challenges in turning data into useful knowledge, and fostering better collaboration and feedback, data engineers can truly move the needle.
Read field note
2024-12-23
Not every data request needs a fire alarm. This post cuts through the hype around “real-time” reporting—clarifying what it is, what it isn’t, and how to deliver fresh, actionable insights without burning down your team or your budget.
Read field note
2024-11-27
Traditional ETL pipelines are slow, brittle, and expensive—leaving data teams stuck serving yesterday’s leftovers. The Zero ETL movement flips the script by bringing data directly from source to analytics in real time, cutting out unnecessary prep and manual overhead. This post explains what Zero ETL is (and isn’t), why now is the moment for change, and how data teams can deliver fresher, faster, and more reliable insights using modern tools and automation. Discover how a farm-to-table approach is revolutionizing data engineering.
Read field note