Hello Folks,

Great to connect with you again for another edition! I’m writing this from Philadelphia, where the worst of winter is finally behind us and the days are slowly getting longer and warmer.

Much like the shifting seasons, data engineering itself is in the middle of a transformation. In 2026, it's no longer just about moving and storing data; it's about making data meaningful, trustworthy, and AI-ready. Whether you’re building pipelines, designing data platforms, or enabling AI systems, the common thread is clear that the future of data engineering is as much about context and reliability as it is about movement and scale.

If you’re looking for a place where these conversations are happening in person, check out the Data Engineering Open Forum in San Francisco on April 16th. The agenda is packed with sessions you'll actually want to stay for, and it's a great chance to connect with engineers navigating the same challenges you are.

- Ananda

📚 Data Pulse

Data Engineering After AI: ECL - Extract, Contextualize, Link

📖 Topic: Data & Context Engineering
🧠 Level: Beginner

Summary: As AI continues to automate data engineering tasks such as pipeline generation, transformation logic, and schema inference, the core role of data engineers is shifting from moving data to defining and managing its meaning. Traditional ETL architectures focused on data movement, and they often embedded business logic within pipelines, allowing interpretive context to drift as data passed through successive transformations. An alternative framework, ECL (Extract, Contextualize, Link), addresses this gap by emphasizing three stages: extracting reliable data from source systems, enriching it with contextual definitions, and linking entities across systems to preserve coherence.

💡 Why is this relevant for DEs?

• Shift in core responsibilities: Data engineers should focus more on designing architectures that preserve and govern the semantic meaning of data across systems.

• Need for semantic and governance infrastructure: With ECL, data engineers need to build and manage data contracts, lineage systems, and context stores that ensure data definitions remain consistent, versioned, and trustworthy as data moves through multiple transformation layers.

• Emergence of a new role: The discipline is evolving from pipeline engineering to “context architect,” where data engineers design the contextual frameworks that allow AI systems and downstream applications to interpret and use data reliably.

OpenAI: Scaling to 800 Million Users With Postgres - P99 latency

📖 Topic: Databases
🧠 Level: Intermediate

Summary: OpenAI scaled PostgreSQL to support over 800 million ChatGPT users using a single primary database with dozens of read replicas, skipping complex sharding entirely. By focusing on reducing primary writer load, optimizing queries and connections, and preventing cascading failures, they achieved low double-digit millisecond latency and 99.999% availability.

💡 Why is this relevant for DEs?

• Optimize before adding complexity: OpenAI showed that tuning PostgreSQL for query optimization, caching, and connection pooling can delay or eliminate the need for sharding and distributed systems.

• Design for your actual workload: ChatGPT is largely read-heavy, so read replicas and caching worked. Match your architecture to what your system actually does, not to generic best practices.

• Reliability is key: Tools like PgBouncer, rate limiting, and workload isolation show that modern data engineering is as much about keeping systems stable as it is about building pipelines.

Netflix: Scaling LLM Post-Training

📖 Topic: Data Engineering & AI
🧠 Level: Intermediate

Summary: Netflix built an LLM post-training framework to scale the adaptation of foundation models for production use cases such as recommendations, personalization, and search. Pre-trained models must be adapted to understand Netflix’s catalog and user behavior, but doing this at Netflix scale, with massive data pipelines and distributed GPU clusters, is a major engineering challenge. The framework, built on their ML platform Mako using PyTorch, Ray, and vLLM, enables techniques like fine-tuning, reinforcement learning, preference optimization, and knowledge distillation. The framework is organized around four pillars: data, model, compute, and workflow, providing a unified way to manage datasets, shard models, orchestrate GPUs, and run multi-stage training pipelines.

💡 Why is this relevant for DEs?

• Data pipelines are foundational to LLM training: Post-training workflows depend on well-prepared data curated datasets, proper tokenization, and efficient streaming to distributed training systems. Data engineers own the pipelines that handle all of this, from selecting and transforming domain-specific data.

• LLM systems require scalable data infrastructure: Netflix’s framework underscores the need for distributed workflows to coordinate across GPUs, storage, and orchestration layers. Data engineers are central to building the plumbing that moves, batches, and serves data reliably for both training and inference.

LinkedIn: Maintaining exabyte-scale Hadoop clusters

📖 Topic: Data Infrastructure
🧠 Level: Advanced

Summary: LinkedIn re-engineered the block placement strategy in Apache Hadoop’s HDFS to support exabyte-scale clusters storing about 5 exabytes of data and 10 billion objects while maintaining 99.99% availability. As clusters expanded to thousands of nodes, the default rack-based replication policy caused heavy overhead during maintenance because nodes had to replicate large volumes of data before going offline.

To address this, LinkedIn introduced upgrade domains, logical groupings of datanodes that distribute replicas across broader failure boundaries than racks. By updating the block placement policy without disrupting production traffic, LinkedIn eliminated large-scale replication during maintenance, reducing network congestion, speeding up upgrades, and allowing maintenance on up to 4.5% of datanodes per day while maintaining performance and reliability.

💡 Why is this relevant for DEs?

• Data reliability and availability at massive scale: Data engineers working with distributed storage systems such as Apache Hadoop must design architectures that ensure high data availability, redundancy, and fault tolerance, even when clusters contain thousands of nodes and exabytes of data.

• Operational scalability is a core data engineering challenge: As data platforms grow, routine operations such as hardware upgrades, patching, and cluster maintenance must be redesigned to avoid massive data movement or downtime, highlighting the need for an architecture that scales operationally as well as technically.

Elements of a Modern Data Strategy

📖 Topic: Data Strategy
🧠 Level: Intermediate

Summary: A modern data strategy is what separates organizations that talk about data from those that actually drive results with it. It aligns people, processes, and technology across five foundational pillars.

• Tying every data initiative directly to business outcomes through deep stakeholder engagement.

• Selecting a tech stack that scales and works as a cohesive ecosystem.

• Embedding governance for high-quality data and trust.

• Investing in talent with clear roles and continuous enablement.

• Laying out a prioritized roadmap that balances quick wins with long-term transformation.

Without this foundation, organizations end up with fragmented data, slow decision-making, shelfware investments, and an inability to capitalize on AI and automation when it matters most.

💡 Why is this relevant for DEs?

• The data stack, pipelines, storage architecture, and integration layers defined in a data strategy are largely designed and implemented by data engineers, making them central to turning strategy into working systems.

• Data engineers operationalize data governance, lineage, access controls, and reliable data pipelines, ensuring that data across the organization is trusted, consistent, and usable for analytics and AI.

⚒️ Data Engineering Open Forum 2026

The Data Engineering Open Forum (DEOF) is a community-driven conference featuring in-depth sessions that address the real challenges and innovations in data engineering today. The talks cover a wide variety of topics, including AI Agents, Multimodal data, Data lineage, Data Observability, Semantic layer, and more.

Out of a stacked lineup, the two sessions below caught my eye.

• From Manual to Magical: Building AI Agents for ETL Automation by Himanshi Manglunia, Senior Data Engineer @ AWS: This talk demonstrates a production-grade, autonomous ETL system in which AI agents (powered by Kiro and Claude) handle end-to-end pipeline changes.

• Inside OpenAI’s Internal AI Data Agent by Bonnie Xu, Staff Software Engineer @ OpenAI: This session provides a look under the hood of OpenAI’s internal data agent, a tool that lets employees turn questions into insights in minutes. Xu unpacks the agent’s core architecture, the multiple layers of context it uses to answer queries, and how the team keeps that context updated with almost zero manual intervention.

📋 Agenda and more details HERE.

👉 RSVP before March 22 to get an exclusive DET community discount (33% off).

💎 Open Source Gems

Cube Core: Semantic Layer

Cube Core is an open-source semantic layer that lets organizations define metrics, dimensions, and business logic once and reuse them across BI tools, embedded analytics, and AI agents through standard REST, GraphQL, and SQL APIs. It works with all major SQL data sources, Snowflake, Databricks, BigQuery, Postgres, and more, and includes a built-in caching engine for sub-second query performance.

💡 Why is this useful for DEs?

• With Cube Core, data engineers define metrics and business logic in one place, and every tool, BI dashboards, applications, and AI agents pull from the same definitions.

• Most organizations run multiple BI tools and data platforms. A semantic layer sits between them and provides a single, consistent way to query data, eliminating duplicate logic and unnecessary complexity.

Github: https://github.com/cube-js/cube

💡 DE Tip of the Month

Processing large time-series datasets in dbt

Microbatch incremental models in dbt efficiently process large time-series datasets by splitting transformations into small, time-bounded batches based on an event_time column.

• Atomic, idempotent batch execution: Each batch represents a self-contained unit of work that can run independently, be retried if it fails, and even execute in parallel for faster data processing.

• Simpler incremental logic: Unlike traditional incremental models that require custom SQL conditions, micro batch automatically determines which batches to run, simplifying model design.

• Flexible backfills and failure recovery: Engineers can easily reprocess historical data or retry failed batches by specifying time ranges (--event-time-start and --event-time-end) without rebuilding the entire dataset.

📊 Community Poll

Until next time, cheers!

Ananda, Sukanya & Volker

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hi everyone,

This series is designed for one outcome - clear lessons on how experienced builders approach data engineering problems, not just a founder story or a project overview, so that our DET community can have reusable mental model on how to think about scale, operability under real constraints.

To kick things off, we sat down with Weimo Liu, co-founder of PuppyGraph, whose career spans database research, TigerGraph, and Google’s F1 team. His perspective is a perfect stress test for the themes we care about: why graph ideas have been academically compelling for decades, why production adoption is still hard, and how “scale” in industry changes what success even means. In this interview, Weimo walks through the shift from chasing benchmark wins to optimizing for cost and operability—and shares a provocative approach to graphs: running graph queries directly on modern table formats like Apache Iceberg, instead of asking enterprises to migrate and reload everything.

Let’s dive in and walk through Weimo’s journey.

- Swetha Sekhar

🎟️ Conference: Data Engineering Open Forum 2026

We are hosting the 3rd Data Engineering Open Forum (DEOF) on April 16 in San Francisco! Here is what you will experience expect at the event:

• Sessions by speakers who are solving cutting edge problems in data engineering, for examples, Apache project creators and PMCs like Julien Le Dem, Boyang Jerry Peng, and Jack Ye.

• Intentionally-designed activities that you can sign up to engage in small-group networking (because we know it’s hard to make conversations at conference).

• Opportunities to connect with data engineering teams at top tech companies (like Netflix, Airbnb, and more) at their booths.

Our ultimate goal is that when you look back some day, you could confidently say “I’m so glad I went to DEOF”, because the people you met there or ideas you walked away with made a difference in your career.

👉 See the agenda HERE. RSVP before Early Bird price ends on March 11.

Spotlight: Weimo Liu

For those in the DET community who may not know you yet, could you briefly introduce yourself?

Hi everyone — I’m Weimo. It sounds like the “self-driving” car, Waymo. I’m the co-founder of PuppyGraph. Before starting PuppyGraph, I worked at TigerGraph and Google’s F1 team. TigerGraph is a graph database startup, and F1 is Google’s internal unified SQL query engine, serving billions of queries per day. Thanks so much for having me, it’s a real pleasure to be here.

You’ve been working in the database world since your PhD years. Tell us about your career journey — what initially drew you to databases, and what kept you in the space for so many years?

Back in college, I did very well in my data structures class, and my professor invited me to join his lab. That led me into database research, where I published papers on spatial databases.

When I applied for PhD programs, I actually looked up professors who had published heavily at SIGMOD and VLDB in recent years, and found my advisor Dr. Zhang.

After my PhD, I joined TigerGraph. The founding CTO was a close friend of Dr. Zhang, and I worked there for almost three years. Later, I joined Google — partly because it felt like a safe choice — and worked on the F1 team.

What keeps me in databases is that the field is very concrete. The problems are understandable, progress is measurable, and when something improves, you can usually prove it.

You once said that “half the database papers back then were about graphs.” What made graph workloads so captivating for researchers?

First, graphs are hard. The data is highly connected, and it’s very difficult to shard and distribute efficiently, which constantly creates room for new algorithms and system designs.

Second, motivation is very natural. Researchers don’t need to invent artificial use cases just to justify the work — real graph problems already exist everywhere, such as anti-fraud, network observability, social network analytics, and cybersecurity.

Third, graph theory in mathematics is extremely rich. Computer scientists can borrow powerful ideas from math and turn them into systems that are both interesting and practical.

After your PhD, you took your first industry role at TigerGraph. What surprised you most about the difference between academic database research and real-world database engineering?

In academia, performance is everything. You aim for benchmarks that are 10× faster than the state of the art.

In industry, speed alone doesn’t mean much. Cost, scalability, operability, these often matter far more. You can’t just spend unlimited resources to get a better benchmark number.

Also, no one really cares about “10× faster,” or even “100× faster.” If you walk around AWS re:Invent, every product claims to be 100× better than the state of the art, without even defining what that state of the art is. There’s no peer review, and nobody truly believes those numbers.

Later at Google, you worked on database systems at a much larger scale. What kinds of problems were you solving there, and how did that experience change how you thought about data systems?

F1 is a federated query engine that can query almost every data source and format inside Google. It serves billions of queries per day and handles most of Google’s OLAP workloads — from the fastest, most expensive systems to the slowest, cheapest ones.

In the early days, Google had many different OLAP systems across different organizations. Over time, more and more teams connected their data sources to F1. Eventually, most of them were either deprecated or absorbed into the F1 ecosystem.

That experience made me realize how important unified federation really is.

💡 Editorial Note: F1

F1 is Google’s globally distributed SQL database built on top of Spanner, giving full relational features (SQL, ACID, secondary indexes) at planetary scale. Spanner handles sharding, replication via Paxos, and external consistency using TrueTime (a globally synchronized clock), while F1 provides a stateless SQL layer that parses, optimizes, and executes distributed query plans. Data is organized with hierarchical, interleaved tables to co-locate related rows in the same key ranges, reducing cross-shard transactions. Writes use Spanner’s two-phase commit across replicas, and reads use consistent snapshots. In short, Spanner provides globally consistent distributed storage, and F1 turns it into a fully featured distributed SQL database.

👉 Read the full F1 paper HERE.
👉 Read the full Spanner paper HERE.

What prompted you to write a new graph query framework? What problem did you feel wasn’t being solved, or wasn’t being solved in the right way?

I had been thinking about this for a long time. At TigerGraph, many potential users showed strong interest in graph technology, but most of them couldn’t actually adopt it in production. For example, a large bank spent 18 months loading all of its data into the system — something that would be unrealistic for most enterprises. That told me something fundamental was wrong.

After joining Google’s F1 team, I realized these users weren’t really looking for a graph database, they were looking for the graph itself.

I didn’t start the project immediately because, unlike Google, the external data world wasn’t standardized. That changed when I read an a16z blog post announcing that the Apache Iceberg team had left Netflix to found Tabular. I realized the timing was finally right.

My co-founders and I actually reached out to the Apache Iceberg creators with a very simple demo: running graph queries directly on Iceberg, faster than most graph databases on the market. That surprised them too, they hadn’t optimized Iceberg for graph workloads at all. They supported us a lot, not just on engineering, but also on go-to-market.

During the development of the framework, what were some of the hardest technical problems you had to solve?

Graph systems are notoriously difficult to scale, which is one of the main reasons they have not been widely adopted by enterprises with very large datasets.

We define the operators in a graph query plan as NodeOperators and EdgeOperators, where the input and output of each operator is a collection of nodes or edges. PuppyGraph assumes that all graph queries, patterns, and algorithms can be expressed as combinations of these operators.

PuppyGraph implements a rule-based optimizer for logical query execution plans, and a hybrid rule-and-cost-based optimizer for physical execution plans. Because all inputs and outputs are collections, any individual operator can be massively parallel processed (MPP) and vectorized for evaluation.

How did you find your co-founders and early team members, and what qualities did you look for when building the team?

They’re all old friends. Our CTO was my college roommate, and our chief architect lived next door. They’re well-known competitive programming contestants, and honestly, much better programmers than I am.

Trust mattered the most. In the early days, you need to move fast and collaborate extremely smoothly. We were lucky to have already built that trust years ago, it really feels like a reunion of old teammates.

Has the graph problem space changed in response to the rise of AI? Where do you think the field is actually heading?

Yes, very much so. Initially, we didn’t think of this problem space as being closely related to AI. But more and more AI companies and AI teams started reaching out.

AI generates more data, and at the same time, it activates a lot of previously “cold” data. A human data analyst might be able to keep track of hundreds of tables, but AI systems can reason over thousands or even more simultaneously. Our view is that when you already have tables, you already have knowledge. You don’t need to build a separate knowledge graph — you can simply treat your existing data as a graph. That structure naturally provides context to LLMs.

Before we let you go… who should we interview next, and why?

Haha — I’d suggest Zhou Sun, the co-founder of Mooncake Labs. He’s sharp, opinionated, and deeply knowledgeable about databases. I think it would be a really engaging conversation.

Key Takeaways

• The central reframing: many teams don’t need “a graph database”; they need graph type queries over the data they already have, and open table formats like Apache Iceberg make that approach more feasible.

• AI shifts the bottleneck from “finding data” to “connecting data with meaningful relationships” When AI can reason across thousands of tables, the problem becomes building reliable structure/context across them.

• Weimo’s differentiator is translation: turning research-grade system design into something enterprises can run—where operability and time-to-value matter as much as performance.

Community poll

💬 How to stay connected

• LinkedIn

• PuppyGraph

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hello Folks,

Great to connect with you through this month’s newsletter! I hope the resolutions you set at the start of the year are still going strong.

I'm writing this from Coimbatore, India, where the cooler winter mornings are slowly transitioning to warmer and brighter days. There's something I love about this time of year here - clear skies, fresh energy in the air, and that sense of momentum building as the season shifts.

That momentum reminds me of data engineering. The big transformations rarely happen overnight, but every new tool we explore, every system we scale, every challenge we solve it all adds up over time. It’s that steady, consistent work that fuels real progress. And through it all, we're building something meaningful: data systems people can actually rely on.

Outside of work, I enjoy reading, travelling, and watching movies - small ways to recharge and stay curious. In many ways, this community offers the same: a space to learn from each other and continue growing together.

This edition is packed with ideas and resources from across the community. I hope you find something here that sparks a new idea or gives you a fresh perspective.

Happy reading, and thank you for being on this journey with us.

- Sri

📚 Data Pulse

Netflix: Simplify Data Movement using Data Bridge

📖 Topic: Data Ingestion
🧠 Level: Intermediate

Summary: Netflix introduces Data Bridge, a unified control plane that standardizes how data is moved across its vast ecosystem of data stores. Instead of teams building case-specific/custom pipelines for every new data movement use case, Data Bridge abstracts the implementation details of how it is moved from why and what data needs to be moved. Data engineers declare their intent once, and the platform handles routing, execution, and operational concerns behind the scenes.

💡 Why is this relevant for DEs?

• Platform over Pipelines: It helps data engineers move from building siloed data movement pipelines to designing a centralized orchestration layer. This enables reuse, governance, and delivery at scale, rather than reinventing the wheel for every use case.

• Self-Service: New teams and data stores can plug into the system without reinventing ingestion or replication patterns.

• Operational Consistency: It has built-in handling for retries, monitoring, and failure management, so it ensures consistent reliability across all data transfers.

• Reduced Fragmentation: It consolidates fragmented data movement systems into one standardized approach.

Advice to Start a Blog

📖 Topic: Brand Building
🧠 Level: All levels

Summary: David Robinson argues that aspiring data scientists (note: valid for DEs as well) should start blogging as a key strategy for breaking into the field. Rather than just completing courses, candidates should publicly share analyses, tutorials, and projects on topics they find interesting. Blogging serves three critical purposes: it provides hands-on practice with real-world data analysis and communication skills; it creates a portfolio that demonstrates capabilities to potential employers better than resumes alone; and it generates feedback from the community while helping build a professional network. Robinson emphasizes that posts don’t need to be perfect! Sharing any public work is valuable, and even simple explanations of concepts you’ve mastered can resonate with audiences.

💡 Why is this relevant for DEs?

• Portfolio & Personal Brand: A blog provides concrete examples of your work that make interviews and applications more compelling than resumes alone, while establishing you as a thought leader and building a professional network that creates unexpected job opportunities.

• Practice with Purpose: Blogging forces you to work with real-world messy data and communicate findings, developing the exact skills employers need while revealing knowledge gaps and hidden strengths through community feedback.

LinkedIn: Engineering the Job Ingestion System at Scale

📖 Topic: Data Platform
🧠 Level: Intermediate

Summary: LinkedIn shares how they built a large-scale job ingestion system capable of processing millions of job postings from diverse sources reliably and efficiently. Instead of relying on fragmented ingestion workflows, LinkedIn designed a unified, scalable ingestion architecture that standardizes parsing, validation, enrichment, and indexing of job data. The system focuses on handling high-volume, heterogeneous inputs while ensuring data quality, low latency, and operational stability across downstream search and recommendation systems.

💡 Why is this relevant for DEs?

• Scalable Ingestion Architecture: Instead of building ingestion logic per partner or per feed, LinkedIn invested in a standardized ingestion framework. This represents the move from ad hoc pipelines to platform engineering. The architecture supports horizontal scaling and distributed processing, allowing millions of records to be processed per day without degrading performance or increasing operational overhead.

• Data Quality at Ingestion Layer: Schema validation, normalization, and enrichment happen early in the pipeline, reducing downstream correction cycles and improving the reliability of search and recommendation systems.

• Handling Heterogeneous Sources: In real-world systems, inputs from different sources rarely follow a single format. A strong ingestion framework abstracts this variability and converts it into a unified internal schema. For DEs, this means designing flexible parses, schema evolution strategies, and metadata-driven mappings that can scale without rewriting pipelines for every new source.

• Operational Resilience: The architecture is designed to manage failures, retries, and backpressure effectively, ensuring stability even during traffic spikes.

Uber: Engineering for Trillion-Record-Scale Data Lake Operations

📖 Topic: Data Lake
🧠 Level: Beginner

Summary: Uber shares how they adopted Apache Hudi to solve large-scale data lake challenges such as late-arriving data, incremental processing, and inconsistent batch pipelines. Hudi adds transactional semantics, upserts, and time travel capabilities on top of cloud object storage, enabling Uber to treat their data lake more like a database while retaining scalability.

Instead of rebuilding datasets from scratch, Uber uses Hudi’s incremental ingestion and record-level updates to efficiently manage continuously evolving datasets across thousands of pipelines.

💡 Why is this relevant for DEs?

• Reliable Data Lakes (Lakehouse Foundations): Hudi introduces ACID transactions, schema evolution, and rollback support to the data lakes; it helps data engineers safely handle late data, reprocessing, and failures without breaking downstream pipelines.

• Scales with Streaming and Batch: Hudi supports both streaming ingestion and batch workloads, helping teams unify real-time and batch pipelines under a single data store.

• Incremental Pipelines: It allows pipelines to process just the new or updated data since the previous run, instead of scanning terabytes of data in each run.

• Indexing and Fast Upserts: It allows data engineers to perform fast row-level updates across hundreds of partitions and billions of records.

⚒️ Workshop: Getting Started with Protobuf APIs

If schema evolution keeps biting you in production (breaking changes, inconsistent validation, tooling sprawl), this Protobuf workshop is worth your time. You will learn:

• How to adopt Protobuf to prevent breaking changes and evolve schemas safely across teams and languages

• How to use Protobuf in event streaming and data pipelines for better data quality

• Best practices for designing real-world APIs

👉 Sign up for the workshop HERE.

(This message is sponsored by Buf)

💎 Open Source Gems

skills CLI for the open agent skills ecosystem

Skills are reusable capabilities for AI agents. They provide procedural knowledge that helps agents accomplish specific tasks more effectively. skills is an open-source CLI for installing and managing skill packages for agents.

Together with skills.sh, a directory and leaderboard for skill packages, it allows you to easily install and manage skills for all common AI tools like Claude Code, Codex, Cursor, OpenClaw, Gemini CLI, and many more.

💡 Why is this relevant for DEs?

Using AI to implement data pipelines, or any type of software development, has become the new norm. However, in Data Engineering in particular, we work with highly specialized tools, different versions, and varying best practices. This often leads to significant back and forth when using AI to speed up implementation. Installing skills can help, as they target specific use cases. Tools like Claude Code automatically apply a skill when they determine it fits the use case, which can significantly improve the quality of the output.

🧑‍💻 Skills for DEs

• data-quality-frameworks: Production patterns for implementing data quality with Great Expectations, dbt tests, and data contracts to ensure reliable data pipelines.

• brainstorming: Turn ideas into fully formed designs and specs.

• dbt-transformation-patterns: Production-ready patterns for dbt.

• authoring-dags: Creating and validating Airflow DAGs using best practices.

• data-storytelling: Transform raw data into compelling narratives.

And many more.

Github: https://github.com/vercel-labs/skills

💡 DE Tip of the Month

Treat data contracts as non-negotiable in modern pipelines:

As data feeds more dashboards, models, and AI systems, failures are shifting from job failures to correctness. Pipelines run successfully, but silent changes in schema, freshness, or semantics break dashboards, models, and business decisions.

Data contracts help prevent this. They define clear expectations between data producers and consumers so everyone knows what a dataset promises to deliver.

💡 Why it matters now more than ever

• AI agents, auto-generated SQL, and self-serve analytics are increasing the number of data consumers without deep context

• Faster development with a variety of tools like dbt, Spark, and Flink increases the risk of unintended schema changes.

• The cost of bad data is often higher than pipeline downtime.

💡 How to start

• Add contracts to your most critical datasets

• Enforce them with tests and freshness checks

• Make them visible in your metadata catalog

• Start small and expand gradually.

Teams that treat data contracts seriously spend less time firefighting and build stronger trust between data producers and consumers.

Let us know what you like the most in the newsletter. See you next time!

Until next time, cheers!

Sri, Sukanya & Volker

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hi everyone,

This interview highlights how diverse backgrounds and perspectives help people build better data tools.

I met Archie through a virtual data conference called MDS Fest where he introduced Evidence as an open-source reporting tool using only SQL and Markdown (source). What stood out to me was his non-traditional path in data moving from strategy consulting to building open-source data tools. Spending years translating raw data into business decisions gave Archie a deep perspective on how data actually gets used, and where data tools might often get in the way.

Since then, he’s been active in the open-source community shipping data tools, and he’s now leading growth at Evidence.

Hope you find the lessons from this conversation meaningful.

Let’s go!
— Eddy

Spotlight: Archie Wood (Head of Growth at Evidence)

For years, BI meant expensive, monolithic systems that stifled adaptability. But as data environments grew more complex, the demand for flexibility exploded.

This shift has driven the rise of composable BI. Instead of a “black box”, modern stacks use decoupled components to give teams total control over their metrics and visuals.

It was exactly this frustration with the “old way” that pushed Archie Wood (Head of Growth at Evidence) to move from analyzing data to building the tools behind it. His path is anything but traditional, and his story reveals a lot about where modern BI is heading.

📊 Sponsored Insight: The State of Airflow 2026 Report

The State of Apache Airflow 2026 Report, the largest data engineering survey ever, draws on insights from 5,800+ data engineers on how Airflow is actually being used today. In this report, you’ll learn:

• How the role of the data engineer is evolving

• How early adopters are leveraging Airflow 3 features

• How orchestration-first teams tend to ship AI to production faster

👉 Read the report HERE

(This message is sponsored by Astronomer.)

Could you walk us through your journey into the data engineering space?

Today I work as an open-source maintainer and lead growth for a BI tool but I got here by being deeply frustrated as a non-technical data user.

I maintain several open-source projects (DuckDB GSheets and gosql) on top of Evidence because I’ve lived the pain of data tools getting in the way of insights.

I’ve been working with data for about 8 years, though I came into data in a non-traditional way. I started in management consulting, where data was mainly used to make slides and working on Excel until it fell over at a couple hundred thousand rows. Tableau opened my eyes to what better BI tools could do, and I was hooked.

Later, I joined an e-commerce company in London called Patch Plants (UK houseplant retailer) as Strategy Manager and eventually Chief of Staff where I learned SQL and Python because I kept pestering the data team for requests. They eventually gave me developer access and let me self-serve. We used Snowflake, dbt, Looker, Metabase and a lot of Google Sheets. It was a wild couple of years. Between 2019 and 2021, Patch grew nearly 200% and delivered over a million plants to 300,000 customers. Being part of that growth while building out the BI function while navigating complex challenges like Brexit was an incredible learning experience.

In 2021, I moved to Canada and spent a bit of time working on my own data startup ideas, but nothing really stuck. Eventually, I met Adam and Sean, the founders of Evidence who were building exactly what I’d always wanted, a more flexible BI tool that gave you more control over visuals. It often felt like the BI software I used was slowing me down more than helping me and I remember thinking, “Someone has to fix this and I’d like to be part of that solution”. I traded SQL and spreadsheets for building code extensions that made it easier for data folks to tell data stories using software engineering principles.

💡 Key takeaway: Non-traditional paths into data are a superpower. Firsthand frustrations coupled with domain expertise become the fuel to build better data tools.

As an open source maintainer and Head of Growth at Evidence, how do you balance the engineering mindset with the community and adoption side of the work?

It comes in waves. Some months I’m deep in code, shipping features non-stop. Other months are more about storytelling and figuring out how to explain what we’ve built, who it’s for, and where it fits.

A big part of making that balance work is being clear about who we’re talking about when we say “users”, because we have two different audiences.

Our open-source product is something teams can self-host and deploy as a static site. The source code is fully public on GitHub, and the community around it includes contributors, maintainers, and people using it in production on their own infrastructure.

Evidence Studio is our commercial SaaS product and while it isn’t open source, we’re still working out whether we may open source parts of it over time.

When you’re balancing engineering, customer success, and an open-source community, empathy matters a lot. In open source especially, it’s easy to think “people are using it wrong,” but most of the time it means we designed something with too much friction. Evidence initially required installing Node.js and managing local dependencies, which worked fine for contributors but was a blocker for a lot of non-technical users. As a result, we’ve built a browser-based experience to make onboarding smoother and collaboration easier, while maintaining core functionality.

Keeping the project authentic to its open-source roots is a constant balance, and it’s not always straightforward. One way we’ve supported open-source development is by offering commercial support services for teams self-hosting the open-source version. Bigger enterprises often want the ability to pick up the phone and resolve issues quickly, and that support helps fund the work while keeping the open-source project healthy.

💡 Key takeaway: Balancing engineering and community comes down to empathy. Data engineers need to listen to users, understand their real needs, and remove friction wherever possible.

On a personal level, you’re balancing building tools, growing a community and parenting. What habits or systems have helped you sustain creativity and focus amid that mix?

I’m still early in parenting days so any “system” I have keeps changing. Kids force you to adapt constantly. My partner and I both work full time, and because she’s based in New York, I handle weekdays and she takes weekends. It’s intense, but it works.

I run my days in blocks: mornings and evenings are family time, the core workday is protected, and nights are for decompression or light work. Startups don’t respect schedules and if something breaks, you jump in but parenting has made me ruthless about focus. You quickly see how much time gets wasted otherwise.

💡 Key takeaway: It’s important to be intentional with time management and adaptability. Knowing when to pivot is crucial to staying productive and effective.

What’s something you’ve learned from the Evidence community that changed your thinking about how open-source BI should evolve?

BI tools are deceptively complex pieces of software. Setting one up can take weeks, and the person doing it usually has three other jobs. They might be a data engineer, director of data, or even a software developer doing BI on the side. By the time they’ve cleaned and modelled their data, users just want a tool that helps them explore and visualize it without the complex setup time.

That gap between lightweight data exploration (querying in SQL or using a notebook) and building polished dashboards is still massive. It shouldn’t be. Most people want something that feels as quick and flexible as a notebook or SQL client, but with the clarity and polish of BI.

That’s where open source really shines. It can move faster, stay flexible, and meet people where they already work. Projects like Evidence, Streamlit and Rill Data live in that middle ground between exploration and presentation. Where you can go from query to insight to something you’re proud to share, without waiting on a whole implementation cycle.

💡 Key takeaway: There’s a distinct product niche for BI tools that can offer out of the box data exploration and BI tools which provide well governed dashboards.

You mentioned other open-source projects (Streamlit, Rill) that sit in this middle ground. As a data engineer, how do these compare to Evidence, and when should one choose which tool?

Evidence is the most analytics-engineering friendly if your goal is trusted reporting as code. It’s SQL-first and Markdown-first, so dashboards behave like software artifacts: version controlled, reviewable in PRs, reproducible across environments, and easy to ship as a static site or data product. It’s especially strong when you want to pair charts with narrative and definitions (“what changed, why it changed, and what to do next”).

Streamlit is best thought of as a Python application framework, not a BI tool. It’s ideal when you need more than charts (custom workflows, business logic, forms) and UI needs to be bespoke. The tradeoff is governance isn’t built in: metric definitions, permissions, consistency, and performance patterns are on your team to implement. It’s great for prototyping, but can become “a collection of apps” without standards.

Rill is closer to a productized BI experience: fast dashboards, metrics-driven exploration, and workflows optimized for operational analytics and drill down analysis. It shines when you want a Looker-style experience without building an app. Compared to Evidence, it’s less about narrative reporting and more about interactive exploration with an opinionated structure for speed and consistency.

Rule of thumb:

• Choose Evidence for governed, version-controlled reporting with narrative.

• Choose Rill for fast metrics + interactive exploration with a BI-product feel.

• Choose Streamlit when you need a custom data application, not just dashboards.

Was there a specific GitHub issue or a discussion on a PR that provided a distinct technical learning moment you can share?

One small example that stuck with me was number formatting. A user opened a GitHub discussion pointing out that our default format:

€1,234.56 was incorrect!

In Germany, the same number should be displayed as:

1.234,56 €

That difference matters:

• Comma for decimals (,56)

• Period for thousands (1.234)

• Currency symbol at the end (€)

The user could technically “fix” it upstream in the database layer, but that would turn the number into a string and break Evidence features like charts and numeric props. This edge case required a code change to support locale-aware formatting properly.

It was a good reminder that small details can impact clarity and trust, and we only caught it because someone raised it publicly on GitHub.

💡 Key takeaway: Open source makes your blind spots visible. Real users surface real-world edge cases that make the product better for everyone.

What advice would you give to data engineers who are curious about building open source tools?

Start by contributing. If you use an open source tool and spot a bug or a feature gap, engage with the community on GitHub, Slack, or Discord. Small contributions are the best way to learn how open source really works: people building in public, improving tools they rely on, and collaborating across the world.

If you want to build your own open source tool or company, go in with equal parts excitement and realism. It’s deeply rewarding but hard to sustain. You’ll meet amazing contributors and a few demanding users, and you’ll eventually face the hardest question which is how to monetize without breaking what makes it open.

💡 Key takeaway: Start small and engage the open source community on public channels.

Community poll

💬 How to stay connected

• https://www.linkedin.com/in/archiesarrewood/

• https://github.com/archiewood

• https://evidence.dev/

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

First of all, Happy New Year, everyone!

Hope you all had a happy and restorative holiday season. My own break was a beautiful blur of firsts: spending the first Christmas and New Year with my new daughter. There is something profoundly grounding about watching a child experience the magic of a Christmas market for the first time. Between the smell of roasted nuts and the twinkling lights, I found myself doing a different kind of “data collection”capturing every giggle and wide-eyed stare.

That calm period gave me some much-needed space to reflect on 2025. In the data world, 2025 was the year we stopped talking about AI alone and started weaving it into our actual pipelines. We moved from Copilots that write SQL to AI-assisted metadata management. Personally, it taught me that while the tools change faster than a toddler’s mood, the fundamentals: reliability, cost-efficiency, and trust remain our North Star.

Let’s make 2026 the year we build less “tech debt” and more “data wealth.”

With that, we start our first newsletter of the year with a bonus section. A few of our DET editors share their New Year Resolutions with you, our amazing community. Hope you enjoy this edition and perhaps draw some inspiration for yours, if you’re in the process of creating your own.

- Chozhan

📚 Data Pulse

Templatizing Spark Declarative Pipelines with DLT-META

📖 Topic: Data Pipelines
🧠 Level: Beginner

Summary: Databricks introduces DLT-META, an open-source framework designed to solve pipeline sprawl. It allows engineers to use a metadata-driven approach to generate Delta Live Table (DLT) pipelines. Instead of writing unique code for every table, you define the pipeline logic in a JSON/YAML template, and the framework dynamically generates the Spark jobs.

💡 Why is this relevant for DEs?

• Operational Scalability: It enables us to move from “coding” individual pipelines to “architecting” frameworks, allowing a single engineer to manage thousands of tables.

• Reduced Tech Debt: By templatizing the logic, you ensure consistent data quality checks (expectations) and governance across all data assets.

• Faster Onboarding: New data sources can be integrated by simply updating a metadata file rather than deploying new code modules.

• DRY (Don’t Repeat Yourself): It enforces a “standard library” of transformations, making maintenance and debugging significantly simpler across the lakehouse.

Meta: Video Invisible Watermarking at Scale

📖 Topic: Data Engineering
🧠 Level: Intermediate

Summary: Meta uses Video Invisible Watermarking, a system designed to embed traceable metadata into video content without affecting visual quality. This enables Meta in detecting AI-generated videos, verifying who posted a video first, and identifying the source and tools used to create a video. This is a massive-scale data engineering challenge given the amount of videos being circulated across Meta platforms every minute. It requires processing billions of video uploads in real-time, ensuring the watermark survives compression, cropping, and screen recording. In this post, Meta shares how they overcame the challenges of scaling invisible watermarking, including how they built a CPU-based solution that offers comparable performance to GPUs, but with better operational efficiency.

💡 Why is this relevant for DEs?

• Complex Data Modeling: It showcases how to handle unstructured data as a first-class citizen in a pipeline.

• High-Throughput Processing: DEs can learn about the infrastructure required to run computationally expensive ML models (for watermarking) on every single write operation.

• Data Provenance and Trust: As GenAI content floods the web, building “Trust Infrastructure” like watermarking will become a core responsibility for data platforms.

• Multimodal Engineering: It bridges the gap between traditional signal processing and modern distributed data pipelines at an exabyte scale.

Lyft’s Feature Store: Architecture, Optimization, and Evolution

📖 Topic: Data Architecture
🧠 Level: Advanced

Lyft's revamped Feature Store is a mission-critical "platform of platforms" designed to centralize and scale machine learning feature management across the organization's entire rideshare stack. The implementation utilizes a "Features-as-Code" approach where engineers define features via Spark SQL for business logic and JSON for metadata. This configuration is automatically converted into production-ready Airflow DAGs that compute and publish data to both a Hive-based offline store for training and a high-performance online serving layer. For low-latency retrieval, the system employs a hybrid architecture featuring DynamoDB backed by a ValKey write-through cache, alongside OpenSearch specifically for vector embedding support. This robust setup now handles over a trillion annual operations, serving as the foundational infrastructure for everything from real-time pricing to fraud detection.

💡 Why is this relevant for DEs?

• Democratizing feature engineering: By allowing anyone with SQL and JSON knowledge to deploy production-grade pipelines.

• Separating storage and computation: To ensure high-throughput batch writes never interfere with sub-millisecond read performance.

• Solving the small file problem: Through automated background compaction and clustering within the streaming lakehouse layer.

• Enforcing organization-wide Data Contracts: Guaranteeing feature freshness, ownership, and schema stability for downstream consumers.

• Reducing operational overhead: By migrating complex orchestration from in-house tools to managed services like Astronomer (Airflow).

• Future-proofing infrastructure for AI: With native vector indexing in OpenSearch to support emerging LLM and agentic workflows.

• Standardizing multimodal data management: Unifying raw metadata, binary blobs, and embeddings into a single searchable ecosystem.

💡Blog: Building AI Agents for Data Engineering Ops

Debugging production data pipelines is a critical part of a data engineer’s job but it can be particularly difficult. The team at Corelayer explored how AI agents can assist with investigations: detecting anomalies, correlating logs and DAG runs, forming hypotheses, and producing evidence-backed root cause analyses with humans firmly in the loop. In this article, you will learn the failure modes, operational complexity, and the design principles (evals, guardrails, transparency) that matter when applying AI to real-world data engineering workflows.

👉🏼 Read the full article HERE.

(This message is sponsored by Corelayer.)

✨ New Year Resolutions from DET Editors

Volker

In our first Community Spotlight newsletter, the vote showed that the biggest blockers to write or speak about work and experiences are “stuck in drafts forever” and “too tired after coding all day.” I feel both. I tend to overthink everything I share, so ideas die in perfectionism. But data engineering moves fast, and the value of insights fades quickly. This year, I want to be braver: share earlier, share imperfectly, and don’t be afraid of having opinions.

Swetha

In 2026, I want to develop consistent technical writing in data engineering, using it to support others in the field and also make it as a learning tool. Alongside this, I want to also expand the mentoring guidance to provide aspiring data engineers, give them industry trends, interview expectations and help them achieve their goals, I have always loved contributing to the community and I look forward to taking the deliberate effort to do that this year.

Eddy

In 2026, my resolution is to create more conversations about data engineering. A lot of my growth as a data/software engineer came through meetups and having conversations with dev/product/sales/marketing folks across the tech spectrum. I’ve been fortunate to grow in a place with a vibrant data community, and that environment played a huge role in my professional and technical development. In the future, I hope to carry that same mindset forward: investing in conversations, building community, and learning in public.

Ananda

I plan to contribute to open source, as it gives me significant leverage to create impact across the data engineering community. I intend to increase my knowledge in large-scale data engineering and infrastructure optimization. A more ambitious idea is to learn a new programming language, probably Rust, to expand my skill set. I’ll share what I learnt with the community through mentoring, blogging, and speaking engagements, taking small, consistent steps that compound into decent progress.

Chozhan

Starting late 2025, we’ve been inching toward Autonomous Data Engineering, where pipelines don’t just alert us when they break, but suggest the fix or even auto heal. With that as an opportunity, I approach 2026 with the mindset of reclaiming the mental space for creative architecture & strategy that drives more business value by (re)designing every system to automate the mundane & repetitive. Also, I learned a lot in the past years from being part of communities and this year, I’d like to give back more by sharing my experiences, especially failures & lessons learned using different mediums.

So, dear community, we are curious to hear what your 2026 resolutions are. Let us know via the poll at the end 👇 or in the comments or the community chat 💬.

💎 Open Source Gems

LanceDB: The Multimodal Vector Database for the AI Era

LanceDB is an open-source, serverless-native vector database designed to simplify the management and retrieval of unstructured data for AI applications. Built on the high-performance Lance columnar data format, it offers up to 100x faster random access performance compared to traditional formats like Parquet. It is uniquely useful because it allows engineers to store raw data (images, videos), metadata, and vector embeddings together in a single, unified table. This "SQLite-like" approach means it can run embedded directly in your application or scale to massive cloud-native lakehouses without the overhead of managing a complex database cluster.

💡 Why is this relevant for DEs?

• Unifies raw media and embeddings in a single table, eliminating complex sync logic between disparate data stores.

• Eliminates Parquet’s random access bottleneck, delivering the sub-millisecond retrieval required for real-time RAG.

• Scales from local to petabyte-scale cloud storage using an embedded, zero-infrastructure serverless architecture.

GitHub: https://github.com/lancedb/lancedb

💡 DE Tip of the Month

Build Context-Aware Metadata & Smart Lineage

Most lineage systems only show which tables depend on which, but context-aware metadata explains why changes happened and what triggered them. By capturing deployment info, config changes, and upstream data shifts, teams can perform faster root-cause analysis and smarter testing. This turns metadata into an operational control plane rather than passive documentation.

💡 Key ideas & how to apply:

• Capture deployment metadata alongside data changes
  Example: tagging runs in Airflow / Dagster with context.

  run_tags = {
      "change_type": "schema_update",
      "jira_ticket": "DATA-2411",
      "release": "v2026.01",
      "affected_columns": "order_status, delivery_eta"
  }

  Store these tags in your metadata system (OpenLineage, DataHub, Marquez).

• Track schema evolution events and backfills explicitly (change_type = backfill)

Use this metadata to auto-trigger downstream tests & validations and feed lineage to impact analysis before major releases. When something breaks, you already know, what changed, who changed it, what downstream systems are affected, so the incidents resolve faster and safer.

Let us know what you like the most in the newsletter. See you next time!

Until next time, cheers!

Chozhan, Shubham & Sugandhi

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hi everyone,

I’m excited to launch our new Community Spotlight series! Alongside our classic Data Pulse edition, this format focuses entirely on deep-dive interviews. We have one rule for these: they must offer actionable learnings.

So, grab your favorite warm drink and meet our first guest: Ben Rogojan, aka the Seattle Data Guy! From leaving Facebook to building an independent empire, Ben breaks down how he grew a following of 100k+ and why communication often beats coding.

One thing Ben mentions really stood out to me: “You can get a lot of scale through content, but it’s hard to beat in-person relationships.”

I recently returned from AWS re:Invent in Las Vegas, and I can absolutely confirm that. It was wonderful to meet so many people there, but now it’s time to slow down 🎄.

Let’s dive in!

- Volker

Spotlight: Ben Rogojan (Seattle Data Guy)

Please introduce yourself briefly to the Data Engineer Things community and share how you’ve built your personal brand to reach over 100,000 followers.

Hello! My name is Ben Rogojan, I’ve been working in the data world for over a decade now. I started as an analyst and then found data engineering when I was looking for a role that matched the skills I enjoyed over title. While I was working full-time I had a few people ask me to help on some side projects so I started the Seattle Data Guy brand as a consulting company but I very quickly realized I needed to find a way to get myself known. So I started writing content I wish I had more of when I started.

If your journey from corporate data engineer to successful consultant and content creator was a Git repository, what would be the commit message for where you are right now? And what was the most significant “merge conflict” you had to resolve along the way?

I think the commit message would say something like “breaking out and trying new things” or “pattern interrupt phase”. I’ve been spending a lot of time trying to figure out how to shake up my thinking and habits. I’ve been consulting for a few years and it can be tempting to fall into the same habits. So I have been looking for places where I can challenge my habits.

For merge conflicts, I think one that sticks out was when I was making the decision to leave Facebook. I had spent so much time and effort getting a job there that it felt wrong. I had a consulting business that was growing but it was hard to rationalize the decision due to all the prior effort.

You made the transition from working at Facebook to running your own data consulting business. What were the most critical steps in that journey, and what would you do differently if you were starting over today?

I had been consulting off and on through most of my career. The first project I did was actually helping a client move from Access to SQL Server and now its a lot of SQL Server to cloud migration projects.

The most critical step for any consultant is figuring out how you will land clients. Some people are good at marketing, others sales motions, still others are great at networking in person. I think in terms of what I would have done differently is I would have put more effort into meeting more people in person and building relationships.

I believe that’s even more true now than it was in the past. You can get a lot of scale through content, but it’s hard to beat in person relationships.

Your LinkedIn profile states you’re “Tool-Agnostic, Outcome-Obsessed”. How did you develop this clear value statement, and how has it helped you attract the right clients for your consulting business?

I think it can be tempting to prescribe a prior solution to every client. Also, I’ve come across many clients whose data stacks have gotten decently chaotic due to the fact that a consultant decided to add their preferred stack on top of what already exists.

Instead, I aim to come into all my projects by understanding the companies business needs first, their technical talent, budget, and so forth. From there I look for the tools that meet the companies’ needs. In some cases buying a solution can prove far faster and easier on a company with limited data resources and in others the company’s goal is to make data a major part of their offering and the overhead of adding more data per customer would be too expensive for an out of the box solution.

Your newsletter and YouTube channel have both reached over 100,000 subscribers. What content strategy has been most effective for growing your audience, and how do you balance creating content with client work?

I’ve tried multiple times to create and follow a content strategy and calendar. But I always tend to drift.

So my approach is to find 2-4 themes I enjoy at the moment based on the problems I am either experiencing with recent clients, or discussing with data leaders and create content around that. I find that there are trends in problems so if you keep experiencing a similar problem, it’s likely there are plenty more people dealing with it.

If you had to start over today with zero followers across all platforms, which single platform would you focus on first, and what specific content format would you prioritize to build your audience most efficiently as a beginner?

I think writing is always the easiest place to start. Start sharing your ideas on a platform like Substack (as long as it stays friendly to email) or if you aren’t wanting to write long articles then consider a platform like LinkedIn. I don’t think there is a best place. Early on, I believe you’re focused on finding your voice. So having a large audience isn’t the goal. You’re trying to learn, figure out what type of content you enjoy, etc. Overall, good content gets noticed.

Many professionals struggle to grow their audience while maintaining focus on their core business. What’s your framework for deciding which content to create, and how do you balance creating content that attracts followers versus content that converts followers into clients?

Early on I was mostly focused on creating content that I enjoyed and felt like needed to be covered. Since then I’ve started to create a mix of both content for new data engineers as well as for possible clients.

I think this is an area I could generally improve. I don’t spend a lot of time on a content plan with clear ratios of content x and y. Instead, I write what I want to write about.

For data professionals considering consulting, what has been your most effective approach for finding and securing new clients, and how has this evolved as your brand has grown?

I’ve spoken with dozens of consultants from both technical and non-technical backgrounds and the most common way most of them land clients is through their network. I know that can seem hard as if you’re just starting out as a data engineer or analyst as you’ve likely got a small network. But there are plenty of ways to grow it.

Working, going to events, looking for chances to give whether it be sharing content, working on open source projects, etc.

I’d also add that you don’t need to find large projects first. There are plenty of projects out there where people need help automating an Excel report or some other task that you might think is small but you’ll learn a lot from (I did plenty of those projects).

What are the two most impactful actions data professionals can take today to start building their personal brand, even if they’re currently employed full-time?

We are all building our brand everyday. That doesn’t have to mean posting on LinkedIn, YouTube or Twitter.

So I’d say first, build a reputation at your job as the person that gets things done well. Be willing to take on hard problems. That’s how I started consulting as well, at a job a director who was a consultant and turning around a team learned that I was technical through the grape vine and he reached out asking if I wanted to help on a project.

Second, if you do want to do content, create content that you wished you’d had when you started.

How important are communication skills versus technical skills for data professionals today, and what are the three most impactful lessons you’ve learned?

Learning how to communicate seems to be a forever lesson for me. I am always speaking with people who are better and conveying ideas, getting buy-in, teaching and other skills that require different forms of communication.

Technical skills are always important. I really enjoy Alex Ewerlöf’s diagram of skills for making sure that’s not skipped. In terms of lessons:

1. Think about who you are communicating and tailor the message for them - It’s temping to get frustrated when you explain something you think should be simple to understand but the other party doesn’t seem to get it. I view this as my failure to understand my audience instead of their failure to understand.

2. Images work - Even an image that isn’t perfect is more likely to keep your reader engaged. It also makes it far easier to explain concepts like your internal network map of your various servers so that new employees can quickly get up to speed or if you’re trying to get buy-in for a dashboard having a mock-up makes the end state all the more real. Don’t just write when you can draw or diagram.

3. Cut out fluff - I tend to lean on the fluffy side of writing. I ramble., sometimes add sections purely for self indulgence. But when I reread it, I realize that what I’ve written adds very little to the overall piece. So cut it out.

What’s your vision for the future of independent data professionals, and what emerging opportunities should they be positioning themselves for?

I think data problems will continue to grow for a few reasons.

I believe businesses will continue to demand more and more from their data.

For some businesses that will mean more granular or complex data like images and unstructured data.

For others it’ll be integrating data sets that have never been connected.

But still for others it’ll just be answering key questions about the business. I think many people would be surprised how many businesses and organizations are early in their data journey or perhaps needing to revamp it. I like to say that many companies are in different data decades. So there will continue to be plenty of work for the next few years.

What’s one message you’d like to share with the Data Engineer Things community?

You don’t have to learn or be everything at once. Wherever your journey is now, enjoy it. You never know when it’ll shift. I didn’t know that the last day I was going to enter Facebooks offices was in March 2020. I kept assuming I’d be able to go back. Then it was gone.

And one day I am sure I’ll have my last consulting client.

I’ll put out my last YouTube video.

So enjoy being in whatever stage you’re at.

If you’re learning, learn! Dive deep, don’t worry about what other people are doing.

If you’re executing, execute to the best of your ability.

If you’re raising a family, do that. Be in that moment, because it’ll end and you’ll miss it.

Connect with Ben

• LinkedIn Ben

• LinkedIn Seattle Data Guy

• Substack

• YouTube

Community poll

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hi Everyone,

Welcome to the final edition of the 2025 newsletter, a year that pushed data engineering through one of its most transformative leaps. From the rise of agentic data engineering, and multimodal distributed engines, to real-time graph architectures reshaping event processing at massive scale, one thing became clear: data engineering is evolving faster than ever, fueled by the breakneck pace of AI.

Amid all this change, one constant stood out: data engineers remain vital for organizations. They build the reliable, scalable foundations that data systems depend on, ensuring data is clean, governed, and available in real time while optimizing cost and performance as workloads grow.

Today, data engineers sit at the very center of AI readiness, designing data layers that support multimodal workloads, enforcing data contracts to keep teams aligned, and building real-time infrastructure that powers everything from fraud detection to personalized experiences.

We close the year with a powerful reminder: as our systems scale, so must our influence. Becoming a force multiplier is no longer optional for data engineers; it’s essential for driving lasting impact. 2026 will be an even more exciting and challenging year for data engineering.

Thank you for being part of this journey.

- Ananda

📚 Data Pulse

Thinking Like a Data Engineer

📖 Topic: Data Engineer
🧠 Level: Beginner

Summary: Ananth Packkildurai’s article reminds us that becoming a data engineer isn’t really about mastering technology and tools; it is about systems thinking. With the help of a few mentors, he realized that curiosity, the ability to model the world around you, the willingness to iterate, and the confidence to trust yourself matter far more than any framework or technology.

💡 Why is this relevant for DEs?
This article is valuable for data engineers because it reframes the discipline beyond tools, cloud platforms, or pipelines, and focuses on the thinking patterns that truly drive long-term success. In an industry where tools and frameworks change every year, curiosity, systems thinking, iterative design, and self-belief are timeless skills that help engineers navigate complexity and uncertainty. For anyone entering or growing in the data engineering field, these lessons offer a grounding perspective on what actually makes a data engineer effective, resilient, and innovative.

Python User-Defined Table Functions (UDTFs) in Unity Catalog

📖 Topic: Data Engineering
🧠 Level: Intermediate

Summary: Databricks introduces user-defined table functions (UDTFs) in Unity Catalog for PySpark. This feature allows users to write stateful, table-generating logic in Python and register it as a governed object in Unity Catalog (UC). Unlike a scalar UDF (which returns one value per row), a UDTF (User-Defined Table Function) returns zero, one, or many rows for each input row, making it powerful for exploding data or complex transformations. You define it as a Python class with an eval method and call it directly from SQL using the TABLE() keyword. UDTFs are useful for complex tasks like ML inference or pattern detection, but are generally not useful for simple, one-to-one transformations best handled by built-in Spark functions.

💡 Why is this relevant for DEs?
With this new UDTFs, Data Engineers can now implement complex Python logic once, register it centrally under Unity Catalog’s security model, and instantly make it available for use across all workspaces, SQL Endpoints, and pipelines.

Furthermore, it provides automated lineage capture, reduces code duplication, and enables heavy setup logic (like loading a model) to run only once per partition via the Python class structure, significantly boosting performance for stateful processing.

Netflix: Real-Time Distributed Graph (Data Ingestion and Processing)

📖 Topic: Data Architecture
🧠 Level: Advanced

Netflix has evolved from pure video streaming into a multi-vertical platform offering ads, live events, and mobile games, which requires connecting member behavior across many devices and services. Their microservices architecture created siloed data, making it challenging to understand cross-app interactions. To solve this, Netflix built Real-Time Distributed Graphs (RDG) that link events such as watching, logging in, or playing games into a unified, relationship-driven graph. Events flow from devices → API Gateway → Kafka → Apache Flink, where they are filtered, enriched, deduplicated, and transformed into graph nodes and edges before being stored through their Data Mesh at >5 million records per second. Distributed Flink jobs (one per Kafka topic) allow scalable, low-latency processing. The RDG stores a property-graph model of nodes (members, titles, devices) and edges (watching, logging in, playing).

💡 Why is this relevant for DEs?
This system illustrates the real-world challenges DEs face when dealing with large-scale, cross-device behavioral data. Netflix’s RDG shows how traditional warehouses or microservice-siloed datasets fail for real-time identity stitching, behavioral tracking, and relationship-centric analytics. The architecture highlights core DE competencies: event-driven design with Kafka, low-latency stream processing with Flink, schema governance with Avro + registry, backfill strategies with Iceberg, and graph modeling. For data engineers, the RDG is a model for designing high-throughput, low-latency, scalable systems that unify fragmented data to power personalization, fraud detection, recommendations, and cross-domain insights.

Become a better force multiplier in 4 steps

📖 Topic: Career Development
🧠 Level: Intermediate

Force multiplication means creating a large, sustained impact without being directly involved in every task. Individual Contributors (ICs) can achieve this by sharing knowledge instead of repeating answers, delegating with guidance rather than control, removing friction points that slow others down, and stepping back to connect broader patterns across teams. At its heart, force multiplication is about shifting from doing everything yourself to helping others move faster and grow. It’s the impact you create when you document what you know, coach teammates, remove friction, and help everyone stay aligned. In the end, it’s not just about taking on more work; it’s about empowering people so things keep moving forward even when you’re not in the room.

💡 Why is this relevant for DEs?
For data engineers, force multiplication is essential because modern data systems scale far beyond what any single engineer can build or maintain alone. High-impact DEs create durable value by documenting pipelines and playbooks, mentoring teammates on standards, designing reusable data frameworks, improving infrastructure bottlenecks, and aligning cross-functional teams around unified data models or platform capabilities. Whether you’re building ingestion frameworks, managing governance, introducing orchestration standards, or guiding architectural decisions, your ability to enable ML engineers, software teams, and fellow DEs, multiplies organizational velocity. Force multiplication turns a DE from a “pipeline builder” into a strategic technical leader whose influence shapes reliability, scalability, and long-term data culture.

🎤 Online Summit: What’s Ahead in 2026 for Data & Analytics

The summit focuses on preparing organizations for next-generation trends in data, analytics, and AI by showcasing evolving technologies, modern architectural best practices, and emerging AI/ML trends.

• “Streaming Intelligence: Unlocking AI’s Potential with Real-Time Context” by Jake Bengtson, VP of AI, Striim.

• “Putting GenAI to Work with Business Data: Driving Value and Managing Risk” by Kristy Hollingshead, Associate Director of Data Science, Further.

👉🏼 Sign up for the online summit HERE (free registration).

🔖 Featured Read

SIMD: The real superpower behind super fast databases

Author: Shubham Tomar

Modern analytical databases like ClickHouse, DuckDB, BigQuery, and Redshift feel incredibly fast because they’re smart at every step, how they store data, how they fetch it, and especially how they process it. Things like columnar storage, compression, and pruning help reduce the amount of data they touch, but the real boost comes from what happens inside the CPU when it starts crunching the numbers. This is where SIMD (Single Instruction, Multiple Data) becomes the hidden superpower, instead of performing operations one value at a time, SIMD allows CPUs to process multiple values simultaneously using wide vector registers (SSE, AVX, AVX-512). Because registers operate at near-zero latency compared to RAM, vectorized operations massively boost throughput for large-scale analytical workloads.

These databases utilize SIMD using vectorized execution engines, where operations are performed on batches of values rather than one row at a time. Early systems like MonetDB/X100 introduced this idea, and modern engines such as DuckDB, ClickHouse, and other columnar databases have fully embraced it. By running thousands of values through SIMD in a single instruction, these systems make far better use of the CPU’s cache, avoid unnecessary branching, and often achieve speedups of 10x or more.

For data engineers, this means queries return faster without endless tuning, pipelines scale smoothly as data grows, and compute costs stay under control because the engine does far more work per CPU cycle. Understanding how SIMD and vectorized execution work helps data engineers make more intelligent choices when selecting analytical databases during the design phase. When you know which engines fully utilize SIMD under the hood, you can pick systems that deliver faster queries, lower compute costs, and better scalability right from day one instead of discovering performance issues later and trying to fix them with tuning, caching, or hardware upgrades.

Key Features and Learnings

• SIMD enables true parallelism: one CPU instruction can operate on multiple values at once.

• CPU Registers are the real performance hotspot: operations on registers take 1 cycle vs. 100–200 cycles for RAM access.

• Massive speedups for scans & aggregates: SUM, COUNT, and predicates run across entire vectors in one instruction.

👉🏼 Read the full article HERE.

💎 Open Source Gems

Flotilla - Distributed engine - Multimodal data workloads
Flotilla is Daft’s next-generation distributed execution engine built for multimodal data workloads that involve massive PDFs, images, audio, video, embeddings, and GPU-heavy operations. Flotilla introduces a node-level, streaming, concurrent processing model powered by its Swordfish engine. Benchmarks show Flotilla achieving 2–7× faster performance than Ray Data and 4–18× faster than Spark, all while using smaller, cheaper clusters and avoiding OOM issues through bounded-memory streaming execution.

Why is this useful for data engineers?
Flotilla simplifies the challenge of building scalable multimodal pipelines that previously required complex configuration tuning, cluster resizing, and careful partition management. You can now write workflows such as PDF ingestion, image processing, video object detection, transcription, and embedding declaratively using DataFrame APIs, and Flotilla executes them efficiently across CPUs and GPUs in a distributed manner. Flotilla is faster, more reliable, and far easier to operate than traditional distributed engines, unlocking productivity gains for data engineers working on AI, LLM retrieval systems, and unstructured-data-heavy applications.

🧑‍💻 Daft on GitHub.

💡 DE Tip of the Month

Handling rapidly changing dimensions in the data warehouse:

Fast-changing dimensions should not be handled with a single SCD Type 2 approach, as highly volatile attributes can cause dimension bloat and slow performance. The best practice is to split attributes by their nature, keep slowly changing attributes in the main dimension, track the history with Type 2, and move frequently changing attributes into a new mini-dimension.

• Identify fast vs. slowly changing attributes: Attributes that seldom change should remain in the main dimension table.

• Create a junk dimension: For the fast-changing attributes, create a junk dimension table

• Create a mini-dimension table: Bridge table to link the main dimension and the junk dimension

• Update fact tables: Update the fact tables to include both the primary dimension key and the mini-dimension key

Let us know what you like the most in the newsletter. See you next time!

Cheers,

Ananda, Chozhan, Srivignesh

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hey folks,

October brought me to the Pacific Northwest for the Airflow Summit 2025, and I’m still processing everything I learned and everyone I met.

I caught up with Ben (Seattle Data Guy) over coffee, and we did what data engineers do best: talked about the good old days when Hadoop was king and SSIS was the ETL workhorse 😅.

For me, the conference was a reminder that tools like Airflow aren’t just open source software. They are collective intelligence. When OpenAI shares how they use Airflow, when SAP shows how they’re building RAG pipelines, when someone releases an open benchmark framework, we all get better. Open source is king!

Ben said something in his talk that I keep thinking about: Companies aren’t in the same decade when it comes to data infrastructure. Some are still wrangling Hadoop, others are orchestrating LLM pipelines. But batch is still king, dashboards are still slow, and aligning with the business is still hard. The tools change, but the fundamentals don’t.

There’s something grounding about that. Much like the changing seasons, data engineering is about balance, keeping systems reliable while adapting to change. And when we do that work in the open, sharing our lessons and mistakes, we all move forward together.

Hope you find something here that sparks your curiosity.

Happy reading, and thanks for being here.
- Volker

📚 Data Pulse

4 Senior Data Engineers Answer 10 Top Reddit Questions

📖 Topic: Data Engineering
🧠 Level: Beginner

Summary: Four data engineers: Ben Rogojan, Julien Hurault, Mehdi Ouazza, and Simon Späti, answer the 10 most upvoted questions from r/dataengineering (174K members), covering topics from interview preparation and data quality to cloud costs and career wisdom.

💡 Why is this relevant for DEs?
This article addresses the real-world challenges data engineers face daily, offering battle-tested wisdom from DEs with decades of combined experience. Here are our top 5 lessons from the post:

• Fundamentals matter more than chasing the newest tools.

• Select your data infrastructure based on real-world factors like budget, timeline, and team skills rather than following trends.

• It requires strong leadership, a clear strategy and transparent communication to acknowledge the complexity of seemingly simple changes like a schema change.

• Challenge demands for real-time to avoid complexity, a must often turns into don’t need it (YAGNI)!

• Focus on stakeholder needs and business objectives while designing systems that can gracefully handle and recover from technical issues.

The 2025 AI + Data Engineering Roadmap

📖 Topic: Career
🧠 Level: Intermediate

Summary: Zach Wilson’s 2025 data engineering roadmap outlines the essential skills needed to break into the field, emphasizing that traditional SQL and Python knowledge must now be complemented with AI integration capabilities. The guide covers six critical areas: SQL fundamentals, Python with AI integration, distributed compute systems, data modeling and quality, portfolio projects, and personal branding.

💡 Why is this relevant for DEs?
This article is highly relevant because it reflects the rapidly evolving landscape where data engineering and AI are converging. Modern data engineers can no longer rely solely on traditional ETL skills, they must understand how to build pipelines that support AI workloads, including embedding generation, vector search, and RAG implementations. The roadmap provides actionable guidance on which technologies to prioritize (Spark, Iceberg, vector databases) and emphasizes practical demonstration through portfolio projects that showcase both data engineering and AI integration skills.

The Evolution of LinkedIn’s Generative AI Tech Stack

📖 Topic: Generative AI
🧠 Level: Advanced

Summary: LinkedIn evolved its GenAI infrastructure from fragmented, team-specific implementations to a unified platform capable of supporting sophisticated multi-agent systems. The transformation involved shifting from Java to Python as the primary language, adopting LangChain as the application framework, and building centralized systems for prompt management, skill registries, memory, and model inference. By 2025, LinkedIn leveraged its existing Messaging infrastructure as the orchestration backbone for AI agents, enabling complex workflows like the Hiring Assistant while maintaining strict privacy, security, and human-in-the-loop controls.

💡 Why is this relevant for DEs?
This article shows three architectural principles data engineers can apply when building AI systems. Reuse before rebuild: LinkedIn used existing Messaging infrastructure for orchestration rather than creating new systems. Standardize foundations first: centralized prompt management and skill registries prevented fragmentation as teams scaled. Reduce friction: adopting Python for both development and production eliminated costly translation steps. Successful AI platforms prioritize developer velocity and shared abstractions over chasing the newest frameworks.

Building a Unified Lakehouse for Large-Scale Recommendation Systems with Apache Paimon at TikTok

📖 Topic: Lakehouse Architecture
🧠 Level: Advanced

Summary: TikTok built a unified Lakehouse using Apache Paimon to support Large-scale Recommendation Models (LRMs) that rely on user behavior sequences, replacing fragmented pipelines with a four-layer architecture (DIM, DWD, DWS, ADS) that unifies stream and batch processing. The solution processes 600TB of data in 5 hours using Flink CDC for real-time features and Spark for point-in-time joins, while reducing state size from 1PB to 300GB through user-grouped samples.

💡 Why is this relevant for DEs?
Schema standardization across teams eliminates 70%+ of redundant pipeline development, while unified storage with transactional guarantees solves the classic Lambda architecture consistency problems that plague many organizations. The bucket partitioning by user ID strategy is particularly valuable for any user-centric analytics, enabling efficient sort-merge joins without expensive shuffling operations. Most importantly, TikTok’s approach of treating data as shared assets rather than team-owned silos demonstrates how proper lakehouse design can transform data engineering from reactive pipeline maintenance to proactive platform building, reducing time-to-insight from days to hours while supporting both real-time serving and batch analytics from a single source of truth.

🎥 Watch the related talk!

Flink Forward Asia 2025 talk by Shuiqiang Chen, Big Data Engineer at TikTok

Developer Marketing in the Age of AI

📖 Topic: Personal Brand
🧠 Level: Beginner

Summary: AI has transformed developer marketing by expanding the audience to include AI agents, compressing development timelines from days to hours, and enabling non-traditional developers to ship production code. While core principles remain (show don’t tell, be helpful, remove friction) the key differentiator has shifted from technical capability to brand taste and trust in an era of abundant AI-generated demos.

💡 Why is this relevant for DEs?
As data engineers build their personal brands and engage with the community, this article highlights critical shifts in how technical professionals should present their work and expertise. In an era where AI can generate demos and code quickly, a data engineer’s personal brand becomes a trust signal: showing taste, depth of thinking, and clear opinions about data architecture matters more than just technical capability. Data engineers should focus on creating content that serves both humans and LLMs (documentation, tutorials, architecture explanations), engage authentically in real-time communities (Discord, Slack, Twitter), and demonstrate genuine helpfulness rather than self-promotion.

🎤 Online Conference: Signals Summit 2025

Signals Summit 2025 brings together data leaders to explore how to design resilient, reliable, and AI-ready systems. Here are a few sessions that caught our eye:

• “The Promise of MCP-Powered Workflows: A New Operating System for Data” by Maxime Beauchemin, creator of Apache Airflow.

• “Beyond the Modern Data Stack: Why Architecture Is the Real Source of Trust” by Joe Reis, co-author of Fundamentals of Data Engineering.

• “Data Contracts, Metadata, and Lineage: What Actually Works?” by Hannah Rounds and Callum O’Connor.

👉🏼 Sign up for the online summit HERE (free registration).

(This message is sponsored by Sifflet.)

🤝🏼 DET Mentorship Program

The new DET Mentorship Program is here! The revamped mentorship program is designed to meet you where you are in your professional journey by offering different types of mentorship experiences. During the pilot phase, you can book:

• Career Development: Beginner + Early Career

• Career Development: Mid-Career

• Resume Review

• Interview Preparation

👉🏼 To get started and find a mentor (or apply to be a mentor), read the program guide.

✏️ DET Writers Workshop Series

The DET Medium Editorial team is hosting a 4-week virtual technical writers series. This workshop is designed to help intermediate to advanced writers sharpen their technical writing skills.

🗓️ Dates: Every Monday between Nov 24 and Dec 15, 2025

🕕 Time: 6:00 PM - 7:15 PM Eastern

👉🏼 Sign up for the workshop HERE.

🔖 Featured Read

Why Data Product Management Is Nothing Like Software Product Management

Author: Clay Gambetti

Data product management isn’t just software PM with different content, it’s a fundamentally different discipline.

In software PM, conceptual technical understanding suffices. In data PM, it doesn’t. When an engineer asks about acceptable data latency, they’re asking a product question that requires technical depth to answer. Say “as fast as possible” and you drive expensive architectural decisions. Say “daily is fine” and you can use simpler infrastructure. You can’t make that call without understanding both user needs and technical implications.

Software products have uncertainty around user needs. Data products add fundamental uncertainty about the data itself. You often don’t know if data quality will be adequate until real data meets real users. This affects how you scope work, communicate with stakeholders, plan roadmaps, and define success.

The Five Core Competencies

1. Technical fluency to evaluate tradeoffs and understand when constraints are real vs. negotiable

2. Adapted user research that validates data feasibility early and understands downstream impact

3. Translation skills between business needs, technical requirements, and business risk

4. Cross-discipline stakeholder management for leaders, engineers, analysts, and compliance

5. Data product sense to anticipate quality issues and recognize when simpler is better

Managing Tensions, Not Optimizing for One Dimension

Data product management is about managing tensions: business urgency vs. technical reality, comprehensive solutions vs. focused products, short-term wins vs. long-term investment.

👉🏼 Read the full article HERE.

💎 Open Source Gems

just
just is a command runner that saves and executes project-specific commands from a file called ⁠justfile, offering a simpler and more user-friendly alternative to ⁠make with features like cross-platform support, command-line arguments for recipes, and syntax inspired by makefiles but without the build system complexity.

Why is this useful for data engineers?
It replaces scattered bash scripts and hard-to-remember commands with a single, self-documenting ⁠justfile that standardizes how your team runs dbt models, data quality checks, pipeline deployments, and other data workflows, making onboarding faster and ensuring everyone executes critical data operations consistently.

🧑‍💻 just is just on GitHub.

💡 DE Tip of the Month

Build idempotent pipelines! Running the same pipeline multiple times with the same inputs should produce the same result.

Why it matters: When you need to backfill historical data or retry a failed task, non-idempotent pipelines become your worst enemy. Non-idempotent pipelines create duplicates, corrupt historical loads, and turn simple reruns into multi-day debugging marathons.

1. Replace INSERT with UPSERT

2. Avoid using datetime.now() in your logic, use execution context instead

3. Read from specific partitions - Always process data for a specific time window

4. Use execution date - Let your orchestrator tell you what data to process

How to work with execution context:

• Airflow: Airflow template variables

• Dagster: Partitioning assets

• Prefect: Runtime context

Let us know what you like the most in the newsletter. See you next time!

Cheers,

Chozhan, Srivignesh and Volker

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hey folks,

I’m writing to you from Cincinnati, where fall is settling in and the leaves are starting to show their colors. The crisp air, football weekends, and streets lined with shades of red and gold make this one of my favorite times of the year. There’s something about the change of seasons that feels a lot like data engineering—constant shifts, small adjustments, and moments of beauty when everything comes together.

Much like the rhythm of fall, data engineering is about balance—keeping systems reliable while adapting to change. Whether it’s new tools, evolving patterns, or fresh ideas from the community, there’s always something to learn and something to improve.

This edition brings a mix of ideas and resources from across our community. Hope you find something here that sparks your curiosity as you sip on your favorite fall drink. 🍂

Happy reading, and thanks for being here.

- Sukanya

📰 Data Pulse

• Stream Processing: Open-source project Apache Iggy is a persistent message streaming platform written in Rust, supporting QUIC, TCP (custom binary specification), and HTTP (regular REST API) transport protocols, capable of processing millions of messages per second at ultra-low latency.

  

• Trends Report: InfoQ released its annual AI, ML & Data Engineering Trends Report. Physical AI and multi-modal models will revolutionize data processing pipelines by enabling real-time analysis of diverse data types in unified workflows. MCP standardization means easier integration between different AI tools and data systems, while commoditized RAG enables rapid deployment of knowledge-based applications. The evolution toward agentic AI directly impacts Data Engineering by automating complex ETL processes, data quality monitoring, and infrastructure management.

• AI: Introducing Claude Sonnet 4.5! Anthropic released Claude Sonnet 4.5, achieving 77.2% on SWE-bench. Key updates include checkpoints in Claude Code, VS Code extension, and the new Claude Agent SDK. For DEs, the 30+ hour task focus enables end-to-end pipeline development without losing context, while the Agent SDK allows building sophisticated data processing agents for ETL workflows, monitoring, and error recovery.

• AI: Open-source project Genkit is an open-source framework for building full-stack AI-powered applications, built and used in production by Google’s Firebase. It provides SDKs for multiple programming languages with varying levels of stability, including JavaScript/TypeScript, Go and Python.

import { genkit } from ‘genkit’;
import { googleAI } from ‘@genkit-ai/google-genai’;

const ai = genkit({ plugins: [googleAI()] });

const { text } = await ai.generate({
    model: googleAI.model(’gemini-2.5-flash’),
    prompt: ‘What is Data Engineer Things?’
});

• Conferences: Big Data LDN (London) 2025 took place from September 24–25 at Olympia London, bringing together thousands of data, analytics, and AI specialists to discuss building data-driven businesses. Watch out for the 2025 playlist on the official YouTube channel.

🔖 Featured Read

Stop That Platform Hype for Good

Author: Bernd Wessely

Every vendor claims their platform is the answer to enterprise IT chaos—but are platforms really simplifying our lives, or just adding another layer of complexity?

This article cuts through the hype, explains why most platforms fail to deliver on their promise, and challenges the idea that “platform engineering” is the silver bullet.

Highlights from the article:

• Platform ≠ Productivity: Platforms often become another layer of indirection instead of solving core problems.

• One-size-fits-none: What works for one company’s scale, culture, and team may be a terrible fit for another.

• Illusion of simplification: Instead of making life easier, platforms can hide complexity until it explodes later.

• Focus on fundamentals: Teams thrive when they improve automation, monitoring, and developer experience—not when they chase buzzwords.

• Reality check: Engineering isn’t about buying platforms; it’s about building reliable systems that solve your problems.

👉🏼 Read the full article HERE.

📚 Articles of the Month

• Avoid these AI coding mistakes: PostHog engineers break down what actually works (and doesn’t) when coding with AI in production environments. Key insight: in large codebases (think 8,984 files), you need serious guardrails. AI excels at autocomplete, fixing tests, and research, but struggles with unfamiliar languages and writing quality tests from scratch. Success requires treating it like a skill you develop through experimentation, understanding its limitations, and staying hands-on because you’re ultimately responsible for what ships.

  

• Scaling Muse: How Netflix Powers Data-Driven Creative Insights at Trillion-Row Scale: Learn how Netflix scaled Muse to trillion-row datasets, combining Druid, Iceberg, and probabilistic (HyperLogLog) sketches to balance speed with accuracy. A masterclass in high-performance data engineering.

• Going direct is all about empowering teams to communicate openly across departments and levels without following org chart hierarchies, enabling faster decision-making, increased collaboration, and reduced bottlenecks. For Data Engineers, direct communication is crucial for resolving data pipeline issues quickly during incidents, enabling cross-team data integration projects, and allowing autonomous tactical decisions while escalating only high-risk architectural changes - helping data teams move faster in today’s leaner organizations.

• Note To My Younger Self: In this reflective piece, a Google engineering leader shares wisdom he wishes he had known earlier, from avoiding the chase for validation to embracing uncertainty. A valuable read for data engineers thinking long-term about their careers.

(✍️ Interested in publishing articles on DET on Medium? Read submission guidelines here.)

🎨 Community Spotlight

Please introduce yourself briefly to the DET community and share how your journey has shaped your unique perspective on data systems.

Hi Everyone! I’m Anandaganesh Balakrishnan, a Data Engineering leader with over 18 years of experience designing and modernizing enterprise data infrastructure across the banking, trading, real estate investment, and utilities sectors. I’ve led major transformation projects at ING Bank, Susquehanna International Group (SIG), and Roc360, focusing on building data systems that are faster, more scalable, and efficient. Working across Asia, Europe, and North America has been a enriching journey that’s taught me to adapt to different cultures, collaborate with diverse teams, and see how great ideas can lead into large-scale innovation.

What was the most challenging “merge conflict” you had to resolve when transitioning from traditional systems to cloud-native architectures?

Architecting modern, scalable cloud-native data platforms across organizations meant redoing the data flow and the underlying technology, of course, but it also meant getting teams, processes, and everyone’s way of thinking on board with new tools. Getting past those disagreements taught me that updating things isn’t just about the tech itself; it’s about guiding changes in how people work, the steps they follow, and the tools they use.

What’s your framework for cutting through architectural complexity when modernizing enterprise data platforms, and what’s one common mistake you see organizations make during these transformations?

When modernizing enterprise data platforms, the key to managing complexity is keeping strategy (business/data) and execution in sync. It starts with having a clear vision of what the business wants to achieve, whether that’s faster insights, lower costs, or stronger compliance, and making sure every architectural choice supports those goals. On the ground, this means building modular, flexible systems that can adapt quickly today while scaling for tomorrow. A common pitfall I see is teams rushing into tool choices or cloud migrations without that alignment, which often leads to fragmented systems and unnecessary technical debt. Real transformation happens when strategy guides every architectural decision and day-to-day actions work together to bring that vision to life.

As someone who mentors others and has led strategic programs, what’s the most important skill data professionals need to develop to transition from IC to enterprise architect, and how should they build credibility with C-level executives?

The most important skill for data professionals looking to grow into enterprise architects is strategic thinking, seeing how every technical decision connects to the bigger business picture is important. It is about understanding how data drives revenue, manages risk, and supports the company’s overall objectives. To earn credibility with C-level leaders, speak the language of business outcomes rather than outputs, show how your architectural choices drive measurable value, save costs, or create a competitive edge. When you align technology efforts with what really matters to the business, you help bridge the gap between engineering and leadership. That alignment builds trust, strengthens collaboration, and shows that technology is a true partner in driving the company’s success.

Your career has involved positions as a project leader, database developer, data engineer, technical editor, and more. With your broad perspective on the data job landscape, what are the key roles for data professionals in the future, and what emerging technologies should data professionals be positioning themselves for?

Looking ahead, the most impactful roles in data will center on data architecture, agentic data stacks, platform engineering, and AI-ready infrastructure. As organizations move toward intelligent, automated systems, data professionals will need to design sustainable data platforms, infrastructures, and architectures that enable self-optimizing, adaptive data pipelines, which I refer to as agentic data engineering.

On your LinkedIn profile, you mention your favorite books: Atomic Habits, The Black Swan, and Money - Master the Game. For data engineers, what are the most impactful lessons you’ve learned from these books? Feel free to share one lesson from each.

Atomic Habits instilled in me the discipline to stay focused and execute projects with consistency and efficiency. The Black Swan, my favorite among them, deepened my understanding of how rare, unpredictable events, like today’s AI disruption, can reshape entire industries, reinforcing the need to unlearn and relearn continuously. Money - Master the Game taught me that sharing knowledge and empowering others not only creates collective progress but also leads to personal and professional growth.

What advice would you give to career starters who want to get into the world of data engineering? What should they focus on first?

If you’re beginning as a data engineer, get the basics down first before you mess around with fancy tools. Get good with Python, SQL, and distributed computing, these skills are super important. Also, learn how data helps businesses make decisions and makes money for the company.

💡 DE Tip of the Month

Before adding new orchestration or scheduling tools, explore the capabilities of what you already use. For example, Databricks Jobs, Airflow, and Prefect all support retries, alerting, and conditional workflows out of the box, often enough for most pipelines.

If you outgrow these basics, you can consider specialized workflow enhancements:

• Dagster: Strong focus on data-aware orchestration and observability

• Argo Workflows: Kubernetes-native workflows for large-scale automation

• Mage: Lightweight and friendly alternative for quick pipeline building

Begin with what’s built-in; you’ll be surprised how much complexity it saves.

Let us know what you like the most in the newsletter. See you next time!

Cheers,

Sukanya and Ananda

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hey folks,

Good to connect with you all! Our newsletter intros have rotated between teammates in recent editions, and this time I get to do the writing. I’m excited to welcome you to this issue.

I’m writing from the Bay Area as summer winds down. Having lived in Arizona, Texas, and now California, I’ve experienced heat in all its forms. From dry to humid to the famous Bay Area microclimates. And much like climate, data engineering is all about navigating different conditions and making sure systems stay reliable.

That’s why the value of data engineering lies not just in moving and shaping data, but in enabling people to make better decisions and build better products. Without it, analytics and AI rest on shaky foundations. With it, organizations gain speed, resilience, and trust in the data they use every day.

This newsletter is written by and for the data engineering community. Each edition brings together perspectives, tools, and lessons that reflect the collective experience of practitioners in the field. We hope it sparks ideas and helps strengthen the connections across our growing community.

Happy reading, and thanks for being part of the community.

- Shubham

📰 Data Pulse

• Data Engineering: Sail released 0.3.2 with native Delta Lake support. Sail is an open-source Rust-based drop-in replacement for Apache Spark. The 0.3.2 release marks a significant milestone with native Delta Lake integration, enabling direct read/write operations on existing lakehouse datasets across S3, Azure, GCS, and Cloudflare R2. Built entirely in Rust, Sail eliminates JVM overhead and garbage collection pauses, delivering 4x faster execution at just 6% of the cost compared to traditional Spark deployments. The Delta Lake support, built against low-level APIs for optimal performance, makes Sail worth a look!

• AI: MLflow 3.3.0 is now available! This release introduces several major features and improvements, especially for open-source AI observability and evaluation, including Agno Tracing integration. Agno is an open-source framework for building multi-agent systems with memory, knowledge, and reasoning, along with tracing capabilities.

• AI: Unsloth now supports training the new gpt-oss model from OpenAI! Unsloth, the open-source LLM fine-tuning framework written in Rust, now supports OpenAI's newly released gpt-oss models (20B and 120B parameters), enabling fine-tuning on just 14GB of VRAM for the 20B variant through custom MXFP4 quantization techniques. The official Colab notebook is a great way for Data Engineers to get started.

• Data Engineering: Aiven rolls out Iceberg Topics for Apache Kafka: Zero ETL, Zero Copy. Iceberg Topics turn any Kafka topic into an Apache Iceberg table with zero ETL and zero data copies, making streaming data instantly queryable with SQL. By removing connectors and avoiding duplication, this open-source approach cuts cost, simplifies pipelines, and gives teams both real-time and analytical views of the same data.

• AI: Creating AI agent solutions for warehouse data access and security. Meta is tackling growing data warehouse access complexity with a new agentic architecture that uses intelligent data-user and data-owner AI agents. These agents streamline access requests, enforce security, and guide users through data discovery, exploration, and permission workflows, while auditing and feedback mechanisms ensure guardrails remain in place.

🗓 DET Meetups in Seattle, Warsaw, NYC, and Bay Area

We have three meetup events coming up in September and one more in October:

• Seattle meetup at Databricks on Thu, Sept 18 (RSVP)

• Warsaw meetup at Netflix on Thu, Sept 25 (RSVP)

• NYC meetup at Capital One on Thu, Sept 25 (RSVP)

• Bay Area meetup at Altimate AI on Wed, Oct 1 (RSVP)

(🎤 Interested in speaking at our meetups or online webinars? Submit talk proposals here.)

🏅 Get Apache Airflow Certification for Free

Join the Beyond Analytics virtual conference on Sept 16 for a free workshop on Apache Airflow 3 fundamentals. This workshop will help you prepare for the official Airflow certification exam and answer any questions you may have. Plus, you will get a discount code for a free certification ($150 value).

👉🏼 Sign up for the workshop HERE.

(This message is sponsored by Astronomer.)

🔖 Featured Read

Ever Wonder What Actually Happens When You Hit “Send” on ChatGPT?

Author: Deepanshu Tyagi

We all know ChatGPT feels fast and reliable, but what really happens in those few seconds after you press send? This article reveals the hidden data engineering that powers the experience and shows why the real magic lies in the infrastructure behind the model.

Scale at millions: ChatGPT manages millions of simultaneous conversations with low latency and high reliability.

Streaming backbone: A customized PyFlink + Kafka setup processes events in near real time, orchestrated across regions with Kubernetes.

Kafka Forwarder: Middleware that hides Kafka’s complexity and allows engineers to focus on AI instead of distributed systems.

Zero-downtime upgrades: Multi-cluster architecture and smart traffic shifting let OpenAI swap infrastructure while keeping the system live.

Event-driven AI: Each interaction becomes part of a data flywheel that continuously improves future responses.

Want to dive deeper? The article references these talks:

• Building Stream Processing Platform at OpenAI

• Taming the Kafka Chaos: How OpenAI Simplifies Kafka Consumption

• Changing engines mid-flight: Kafka migrations at OpenAI

👉🏼 Read the full article HERE.

📚 Articles of the Month

• The Equality Delete Problem in Apache Iceberg: Equality deletes in Apache Iceberg make streaming CDC ingestion tricky, slowing queries and limiting compatibility. This article explains the problem and shows how RisingWave tackles it with smarter delete strategies.

• Kafka to Iceberg - Exploring the Options: Thinking of streaming Kafka data into Apache Iceberg? This article breaks down three practical approaches - Flink SQL, Kafka Connect and Confluent Tableflow.

• How do Iceberg, Delta Lake, and Hudi ensure atomicity? Consistency in data lakes depends on atomicity. See how Iceberg, Delta Lake, and Hudi safeguard reliability by guaranteeing all-or-nothing writes.

• How experienced engineers get unstuck in coding interviews: With many companies sticking to algorithmic interviews despite AI tools, this post reveals a systematic approach to getting unstuck during whiteboard coding. Data Engineers will recognize the parallels between boundary thinking for algorithm optimization and the same mental models needed for query optimization and distributed system design.

• From Facts & Metrics to Media Machine Learning: Evolving the Data Engineering Function at Netflix: This article explores how Netflix is expanding its data engineering capabilities with a new Media ML Data Engineering specialization. It highlights the role of their Media Data Lake in powering machine learning workflows across video, audio, image, and text assets.

• Soft Skills Matter Now More Than Ever – Harvard Business Review: In today’s AI-driven workplace, technical expertise alone isn’t enough. This article highlights research showing that collaboration, adaptability, and critical thinking are now the skills that set professionals apart and future-proof careers.

(✍️ Interested in publishing articles on DET on Medium? Read submission guidelines here.)

💡 DE Tip of the Month

Getting started with data quality checks doesn't need to mean introducing new frameworks to your data infrastructure. You can accomplish a lot with built-in open source functions from tools like Airflow.

However, if you need more advanced capabilities, here are the most popular dedicated data quality projects to consider:

• Soda Core: SQL-based data testing with YAML configuration

• Great Expectations: Python-based data validation with extensive documentation

• Deequ: Scala library built on Apache Spark for large-scale data validation

Start simple with your existing tools before adding complexity.

Let us know what you like the most in the newsletter. See you next time!

Cheers,

Shubham and Volker

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hey All!

Great to connect with you through this newsletter, made possible by the incredible team working diligently behind the scenes. This month, I have taken the lead in introducing excellent content that will be helpful in your data engineering journey.

I am from Philadelphia, Home of the Liberty Bell and the boldest ideas. I have worked in India and the Netherlands before making a move to the US 12 years ago to do my Master’s. I have primarily worked in fintech industries and am currently working in utilities. In my free time, I read, play tennis, travel, and watch movies.

Data engineering is vast, and hundreds of tools and features are added each quarter. The fundamentals remain the same, and data engineering innovations have pushed the boundaries in how businesses efficiently utilize quality data. Knowing data engineering design patterns will help data engineers, irrespective of their level, to build high-quality data systems.

In this newsletter, I have included content based on where data engineering is heading, caveats in implementing changes, and modeling data products.

Read, gain insights, and make an impact!

- Ananda

📰 Data Pulse

• AI: Ollama released their new app for macOS and Windows. The new app offers a user-friendly UI to download and chat with AI models, with support for file processing, multimodal capabilities, and adjustable context length.

• AI: Agentic Data Engineering: Ascend.io – Agentic data engineering leverages AI agents with context metadata, logic, dependencies, and runtime behavior to build, manage, and optimize data pipelines efficiently and at scale. Ascend’s automation engine uses rich metadata to orchestrate pipelines by handling dependencies, initiating tasks, and enabling custom event-driven workflows.

  

• AI: Lakeside AI enables organizations to deliver AI-ready data without needing a full migration to a lakehouse architecture. Through federated access, it helps to explore data in existing systems and brings only the most relevant datasets to the lakehouse when needed.

• Data Analytics: SQLRooms is a React-based open-source framework that enables the building of data-centric applications using DuckDB. It allows quick, local analytics by directly interacting with file formats such as Parquet, Avro, and CSV through SQL. Ideal for dashboards, data exploration, and prototyping. A high-performance alternative to a database-backed app.

🗓 DET NYC Meetup on Sept 25

Attention, New Yorkers! You asked for it, and it's finally here: we are officially launching the DET NYC Meetup! Join the first NYC meetup on Sept 25th for an evening of learning, networking, and fun. Big thank you to Capital One for providing the venue and refreshments for the event.

• When: 6:00 PM to 8:00 PM on Thursday, Sept 25th

• Where: Capital One's NYC office

• 👉🏼 RSVP

To stay tuned for all future NYC events, make sure to join the meetup group (clicking the “Subscribe” button on the top right).

We are also looking for volunteers to help organize NYC meetups. Contact Sanchit Burkule if you want to be part of the team.

(🎤 Interested in speaking at our meetups or online webinars? Submit talk proposals here.)

🎬 Webinar: Improving Airflow Data Pipeline Reliability

At Astronomer, a team of just five data engineers manages 27,000 daily tasks, powering 18+ data products. In this webinar (11 am ET, Aug 21st), Maggie Stark, Staff Data Engineer at Astronomer, will share how her team reduced DAG failure rates by 81%:

• Using Airflow Asset scheduling to prevent upstream dependency issues

• Orchestrating cross-DAG dependencies with a Control DAG

• Implementing centralized observability to monitor SLAs and debug faster

👉🏼 Sign up for the webinar HERE.

(This message is sponsored by Astronomer.)

🔖 Featured Read

Data Modeling for Data Products: A Practical Guide

Author: Mahdi Karabiben

Rethinking Data Modeling for the Age of Data Products
As data teams embrace product thinking and business-driven use cases, this article explores how to shift from rigid, monolithic models to adaptive, use-case-focused designs that evolve with the business. A data product is a curated, reliable dataset designed to serve a specific business purpose or use case.

Approach to modeling data products:

• Understand the “why” (core business needs and key business metrics)

• Build the conceptual model (Business domains and how they relate to each other) and get the stakeholder buy-in.

• Define the logical model at the domain level.

• Adopt distributed ownership (Universal conceptual model with domain-specific modeling autonomy).

• Data products and models must evolve continuously through incremental development.

• Build a metric tree to connect business goals to underlying data components.

• Create a semantic layer that maps business metrics to data, ensuring consistency, governance, and simplified access for users.

• Use entity-centric modeling with enriched, denormalized tables to simplify queries and boost performance.

👉🏼 Read the full article HERE.

📚 Articles of the Month

• DoorDash's realtime processing platform with Iceberg: Learn how DoorDash changed their architecture (Kafka → Flink → S3 → Snowpie → Snowflake) to include Iceberg (Kafka → Flink → Iceberg + S3) for their real-time processing platform (30 million messages per second) and saved 25-49% in storage costs, enabled concurrent writes, utilized hidden partitioning and increased the operational ability using query engines like Trino.

• Database Sharding: Key Strategies: Sharding is a horizontal scaling technique used to distribute the load of a single database server across multiple machines, thereby spreading the data and query load evenly. This article explores the core concepts of database sharding, including its importance, how it functions, and the trade-offs involved.

• Mussel - Airbnb’s Key-Value Store for Derived Data: An excellent article on how Airbnb uses Mussel to store derived data efficiently, achieving 99.9% availability, average read QPS (Queries per second) > 800k and write QPS > 35k, and average P95 read latency less than 8ms.

• Speed Without Understanding - One of the Biggest Risks in Data Engineering: Rushing to implement changes without a complete understanding of the effect of changes is a huge risk in data engineering. Even small changes require thorough investigation and communication across teams, as this helps pinpoint unintended consequences.

(✍️ Interested in publishing articles on DET on Medium? Read submission guidelines here.)

✨ Share Your DET Story

We’re gathering testimonials from community members (like you!) to showcase the impact of the DET community and inspire others to join us. If the community has helped you learn, grow, or connect in any way, we’d love to hear your story! Fill out this short form to share your DET story.

🎨 Community Spotlight

Please introduce yourself briefly to the Data Engineer Things community!

Hi! I’m , I am a data executive with over 20 years of experience. Today I’m an independent advisor. I help companies succeed with their data, and help individuals grow their careers.

Throughout my career I’ve built data and analytics teams in various companies, ranging from scale-ups to bigger companies such as PayPal and Meta.

If your career were a Git repository, what would be the commit message for where you are right now?

Shachar was starting to get comfortable so we had to change something. This change may or may not work, we’ll find out..

I left what most people would consider the dream job (DE Director at Meta), with a salary I could have never dreamt of. I left A LOT of money on the table for something completely unknown. It was scary, nerve-wracking, and most people would say it was a crazy thing to do or even a mistake.

Today, two years later, I’m not looking back. Best decision of my life.

In one of your recent posts, you discuss the "career danger zone" for tech managers who lose technical skills while managing small teams. How can data professionals avoid falling into this trap?

In short – the “career danger zone” is the situation where you have a manager of a small team and not hands-on. Managers basically make themselves irrelevant this way, and when they look for another job they fail the technical interviews and realise they are not recruitable anymore.

The solution is simple:

1. Understand and recognise the danger zone

2. Unless you are a Director leading a 50+ people organisation – STAY HANDS-ON. Don’t lose your technical skills. You will become irrelevant.

What's the single best piece of career advice you've received that data engineers and aspiring data leaders should hear today?

Nobody cares what you do, and nobody cares how you do it. What they care about is how it’s impacting the business. If you understand the business and use your skills to drive business outcomes – you will win this game. Anything else is just a waste of everybody’s time.

Based on your experience across multiple industries, what skills do you believe will be most valuable for data professionals in the next 3-5 years?

1. Curiosity – asking questions and being in the details

2. Changeability – ability to take feedback, adapt, and change

3. Clarity – ability to understand and explain complex topics, and ask the right questions

4. Collaboration – ability to build relationships, influence, build communities, and get things done as a group

It’s pretty much guaranteed that through the next 3-5 years:

• Our roles will change

• Technologies will change

• Our workplace and work culture will change

I strongly believe that people with these 4 skills will survive anything that comes their way.

What's one message you'd like to share with the Data Engineer Things community about creating impact through data leadership?

Data is the glue that connects every function in the organisation. Understand the business, learn how to lead with data, and your growth will be unlimited.

➡️ Learn More

If you want to learn more about building a successful career in Data Engineering, have a look at the following fireside chat with Shachar and Xinran.

💡 DE Tip of the Month

Data federation is the most effective way to gain seamless access to multiple data sources. The Federation doesn’t transfer data, but accesses it virtually from the source location. For quick prototyping or linking cloud and on-premises systems, this is useful. This solution works well when you want to limit data movement, create a single-window view, and desire timely information. However, this shouldn’t be used for high-volume or complex transformations.

Let us know what you like the most in the newsletter. See you next time!

Cheers,

Ananda, Sukanya, and Xinran

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

Hey there,

Nice to meet you! You might wonder why Xinran isn't writing the intro this time - that's because we now have a motivated team behind the newsletter, and this month it's my pleasure to introduce you to it.

I'm writing to you from sunny California, where I've recently settled after a life-changing move from Germany to the US. Change has been the defining theme of my year - from navigating the relocation to watching my children experience their first American summer. Like in Data Engineering, sometimes the most challenging transformations yield the most rewarding results!

Speaking of change, you might notice our newsletter looks different this month. We've reimagined the format to deliver even more value to you, our readers. You might also see the theme of change in this month's featured read, where a Git-for-Data approach to managing change in data systems is discussed.

In this context, let me start the newsletter with a quote from a respected fellow Data Engineer:

Enjoy reading!

- Volker

📰 Data Pulse

• AI: Anthropic published a new resource: Powered by Claude. A curated list of projects using Claude in production, a great source for AI inspiration.

• AI: Google released Gemini CLI: your open-source AI agent.

• Data Modeling: dbt Fusion engine public beta is now available on BigQuery.

• Event streaming: Going beyond micro-batch. The new real-time mode in Apache Spark Structured Streaming provides p99 latencies less than 300 milliseconds for both stateless and stateful streaming processing.

• Multimodal data formats: The rise of AI led to unprecedented needs for handling multimodal data at scale. Check out these up-and-coming open-source frameworks like Meta’s Nimble, Lance, and Daft.

• Apache Iceberg v3: New features include deletion vectors, row lineage, semi-structured data, geospatial types, and interoperability with Delta Lake.

• Full Apache Iceberg support in Unity Catalog: Read and write Managed Iceberg tables and use Unity Catalog to access and govern Iceberg tables in external catalogs

• Declarative Pipelines: The declarative API for building robust data pipelines is now open-source and available for Apache Spark.

• Lakebase: Serverless, Postgres-compatible OLTP database for the lakehouse.

• Free Edition: Free Edition offers the same suite of tools that were previously limited to paying customers, allowing everyone to experiment and learn all the latest in data and AI technologies.

🗓 DET Seattle Meetup

Join the next Seattle Meetup for a deep dive into Apache Hudi and Lance—two open-source frameworks for building scalable lakehouse architectures.

• When: 5:00 PM to 8:00 PM on Thursday, July 24

• Where: Docusign Tower, 999 3rd Ave #1000, Seattle, WA 98104

• Talk #1: Redefining Open Lakehouse Architecture with Apache Hudi 1.0, by Dipankar Mazumdar

• Talk #2: Multimodal AI Lakehouse with Lance & LanceDB, by Jack Ye

• 👉🏼 RSVP

(🎤 Interested in speaking at our meetups or online webinars? Submit talk proposals here.)

🔖 Featured Read

In Git We Trust: Git for Data over Data Lakes

Author: Ciro Greco

Just as every software team adopted Git by 2010, every data team will adopt Git-for-Data by 2030.

Data engineering is facing a crisis of reliability. While software developers enjoy the safety of Git's branches, commits, and rollbacks, data teams still work with loosely managed files with no built-in versioning. This creates three critical problems:

• Reproducibility nightmares

• Dangerous experimentation

• Manual everything

The Solution: Git Concepts for Data

Building on open table formats like Apache Iceberg and Delta Lake, the article envisions data workflows with:

• Branches to isolate experiments on real production data

• Commits to track multi-table changes as atomic units

• Merges to publish only after passing data quality checks

• Rollbacks to instantly undo bad pipeline runs

As AI becomes core to every product and data powers user-facing applications, we need software engineering-grade reliability for data operations. The convergence of AI adoption, real-time data applications, and Python as the data lingua franca is driving this inevitable shift.

The article showcases two use cases for Git-for-Data using Bauplan, a Pythonic data platform for transformation and AI workloads with built-in Git-for-Data capabilities.

👉🏼 Read the full article HERE.

📚 Articles of the Month

• What is Apache Hive?: Lakehouses might be trending, but many companies still rely heavily on Apache Hive. Here's an excellent article that clearly explains Hive fundamentals.

• Roblox’s Path to 2 Trillion Analytics Events a Day: Learn how Roblox built a real-time schema-based analytics pipeline, enabling them to handle over 2 trillion events daily with reduced latency and costs.

• The 6 Most Common Customer Questions on Data Projects: A playbook on communication, expectation-setting, and collaborative processes between engineering teams and customers.

• Embedding-Based Retrieval for Airbnb Search: Discover how Airbnb tackled the challenge of searching millions of listings by building a two-tower neural network that maps both queries and homes into numerical vectors. Their innovative approach to training data construction and serving infrastructure delivered booking improvements comparable to their largest ML ranking advancements in years.

(✍️ Interested in publishing articles on DET on Medium? Read submission guidelines here.)

🎨 Community Spotlight

Tell us about yourself and how you started to engage with the DET community.

I'm Volker, a Principal Data Engineer who has been working with data in the gaming industry for nearly 15 years. I started publishing articles with DET about a year ago, and now I'm happy to be contributing to the newsletter!

What's one Data Engineering resource (book, course, tool) you'd recommend that most engineers might not know about?

The Pragmatic Programmer by David Thomas and Andrew Hunt. It might seem a bit out of the box since it focuses on general software development rather than data engineering, but the lessons it teaches can be easily adopted. Its core concepts have become guiding principles throughout my career in tech.

If your data engineering career were a Git repository, what would be the commit message for where you are right now?

refactor(career): add documentation so others can learn from it

Tip: Use Semantic Commit Messages in your data engineering code changes to make your work more understandable for your team!

What's the best piece of career advice you've received that other Data Engineers might benefit from?

Ask the Spice Girls question: "So tell me what you want, what you really, really want."

This question forces you to uncover the motivation, the WHY, behind a stakeholder's request. Before you start working on any project, I recommend clarifying three things for yourself and your team: WHAT are we building? HOW are we building it? WHY are we building it? Understanding the why helps you design simpler, more valuable solutions and increases your visibility because you understand the business value behind the technical requirements.

💡 DE Tip of the Month

Always set alerts on pipeline failures! Silent failures quickly become holiday spoilers. I learned this the hard way by spending my 4th of July debugging pipelines. Don't let this happen to you. Monitor early, alert often. For example, set up alerts for missing or delayed data to catch jobs that fail quietly without throwing an error.

Let us know how you like the new newsletter format. See you next time!

Cheers,

Volker, Shubham, and Xinran

ℹ️ About Data Engineer Things

Data Engineer Things (DET) is a global community built by data engineers for data engineers. Subscribe to the newsletter and follow us on LinkedIn to gain access to exclusive learning resources and networking opportunities, including articles, webinars, meetups, conferences, mentorship, and much more.

I’m writing to you from San Francisco, while attending the Data + AI Summit this week! I had the honor of attending the Open Lakehouse Mini Summit on Monday, a mini-conference for open-source contributors. I enjoyed learning about the Apache Spark 4.x roadmap, including real-time streaming mode, variant type, DataSource API v2, and declarative pipelines, and having open discussions on yet-to-be-solved problems like handling multimodal data and BLOB at scale.

🗓 DET Webinar on Numaflow on June 18

DET Webinar is back! In the upcoming session, you will learn about Numaflow, an open-source framework from Intuit that rethinks real-time event processing for a broader audience of engineers and developers. Whether you're working on real-time processing, designing event-driven applications, or powering ML workflows, Numaflow offers a simple, flexible way to connect streaming sources, transform events, and move data in real time. Join us to see how modern stream processing is becoming more accessible and why it matters more than ever.

• Speakers: Sri Harsha Yayi (Staff Product Manager, Intuit) and Vigith Maurice (Principal Software Engineer, Intuit)

• When: 10 am - 10:45 am on Wed, June 18th (PT)

• Where: Google Meet

• 👉🏼 RSVP

🎬 Free eBook: Practical Guide to Apache Airflow 3

Apache Airflow is one of the most popular open-source frameworks for data orchestration. Whether you are a new or experienced Airflow user, this book will be your guide for getting started with Airflow 3. You'll learn how to:

• Set up a local development environment and write your first pipeline.

• Use new Airflow 3.0 features, including DAG versioning, backfill, asset-oriented syntax, and the new UI.

• Prepare your DAGs for a smooth upgrade from Airflow 2 to 3.

👉🏼 Read the book HERE.

(This message is sponsored by Astronomer.)

🙋🏻‍♀️ Volunteers Needed

We are looking for passionate and dedicated volunteers to join the DET team and shape the future of our community! As a volunteer, you will help us build the new mentorship program, design a one-stop community site, curate content for the newsletter, or host community events. This will be a fantastic opportunity to make meaningful connections, learning technical and soft skills, and enrich your professional portfolio!

👉🏼 Apply to be a volunteer HERE.

☕️ Seattle Coffee Social on June 13

Meet me and Saransh Arora (DET Seattle Lead) at Downtown Seattle for a casual social time over coffee:

• Where: Mr West Cafe Bar Downtown, 720 Olive Wy, Seattle, WA 98101

• When: 5 pm - 6 pm, Friday, June 13

• 👉🏼 RSVP

📚 Articles of the Week

• How We Implemented a Custom B-tree to Handle 10TB of Time-Series Data by Coders Stop

• Why Are There So Many Databases? by Cai Parry-Jones

• Data Quality With Airflow SQL Check Operators: A Step-by-Step Guide by Lorena Gongang (You might think you need some advanced framework for Data Quality, but sometimes Airflow could be all you need. This is a comprehensive follow-along guide for any engineer who wants to implement DQ in their pipelines.)

• Prompting Strategies by Google Cloud (aka. How to clearly communicate your requirements to GenAI to get exactly what you need.)

Have a great week and see you next time!

Cheers,

Xinran Waibel

Head of the Data Engineer Things community

(💡 Don’t forget to follow us on LinkedIn and join the DET Slack community!)

How have you been? I had the most amazing April! I visited my family in China and took a week-long trip to Chengdu, Sichuan, a city famous for numbing peppers and pandas. It was the most amazing culinary adventure I’ve ever had. Then I flew to Los Gatos to host the 2nd Data Engineering Open Forum, and I met many of you in person at the event. I’m hoping to bring the open forum to more cities and exciting things are in the works. Stay tuned!

🗓 Bay Area Meetup at eBay HQ on May 28

Join the next DET Bay Area Meetup for an evening of learning and networking!

• When: 6 pm - 8 pm on Wednesday, May 28

• Where: eBay's Global Headquarters, 2025 Hamilton Ave, San Jose, CA 95125

• Talk #1: Graph Data Engineering for Complex Low-Latency Analytics & AI at Scale, by Sahana Chattopadhyay Debnath (Senior Data Engineer at eBay)

• Talk #2: Inside the next-gen Presto C++ engine, by Aditi Pandit (Principal Engineer at IBM)

• 👉🏼 RSVP

🎬 The OpenXData Conference on May 21

The OpenXData conference on Wednesday, May 21, is a free virtual event on open data architectures, covering topics like data lakehouses, stream processing, query engines, and data for AI/ML. At the event, you will find talks by great speakers from Netflix, dbt Labs, Databricks, Microsoft, Google, Meta, Peloton, and many more:

• Powering Amazon Unit Economics at Scale Using Apache Hudi, by Jason Liu (Senior Software Engineer at Amazon)

• A Flexible, Efficient Lakehouse Architecture for Streaming Ingestion, by Rajwardhan Singh (Engineering Manager at Zoom)

• Data Mesh and Governance at Twilio, by Aakash Pradeep (Principal Software Engineer at Twilio)

👉🏼 Check out the full agenda and sign up HERE.

(This message is sponsored by the MLOps Community.)

🗣️ Speaking at Future Community Events

Within the community, we want to foster a safe environment for both learning and sharing, and we encourage our members (like you) to share their knowledge with everyone. It could be any data engineering topics you are passionate about: how you solved a real-world problem with data engineering, techniques of leveraging open-source frameworks, how to build a data team, communication skills, etc.

👉🏼 Want to speak at one of our in-person meetups or online webinars? Submit your talk proposals HERE.

📚 Articles of the Week

• Why Most Data Engineering Resumes Get Rejected in 10 Seconds (And How to Fix Yours) by Coders Stop. (This article resonated with me. It has detailed examples of dos and don’ts and practical tips. If you are working on your resume, do check it out.)

• Airflow 3 and Airflow AI SDK in Action — Analyzing League of Legends by Volker Janz

• Behind the Scenes: Building a Robust Ads Event Processing Pipeline by Netflix Technology Blog

• Turn workplace conflict into collaboration by David Dye and Karin Hurt. (In case you haven’t noticed, I’ve built a habit of recommending at least one article about soft skills in each newsletter. This time, it’s about conflict resolution.)

Have a great week and see you next time!

Cheers,

Xinran Waibel

Head of the Data Engineer Things community

(💡 Don’t forget to follow us on LinkedIn and join the DET Slack community !)

I hope 2025 is treating you well. You haven’t heard from me for a while because I have been focusing on my own growth. Once again, I’m venturing outside of my comfort zone, and it’s still so uncomfortable and scary no matter how many times I’ve done it! I’ve been learning and reflecting a lot on delegation, ownership, and candor, and I hope to share my takeaways with you in a blog later. But until then, if you are also going through this, stay strong and keep climbing!

🗓 Upcoming DET Meetups

Spring is here! If you are in London or Seattle, it's time to get out there and join our meetups in March and April.

London Meetup on March 20:

• 👉🏼 RSVP

• Talk #1: Scaling similarity: running algorithmically complex computation at scale in Spark for ELT by Robert Vadai

• Talk #2: Building a Data Platform in AWS, dbt, Iceberg Glue and Athena by Sanchit Agarwal

• Talk #3: Data Platforms without Boundaries by Hugo Lu

Seattle Meetup on April 2:

• 👉🏼 RSVP

• Talk #1: Data Governance with Unity Catalog

• More talks are coming soon.

• Don’t forget to say hi to Saransh Arora - he is a data engineer at AWS and the new lead for our Seattle meetup.

Bay Area Meetup

👉🏼 We are looking for a co-organizer (to lead event organization and logistics) and a sponsor (to provide venue and food) for our Bay Area meetup. Email dataengineerthings@gmail.com if you are interested in learning more.

(🗣️ Would you like to speak at one of our meetups? Submit your talk proposals here.)

👋🏼 Meet the DET Medium Editorial Team

Please join me in welcoming the newest members of our Mediun editorial team: Toma Khalifa, Sreyashi Das, Lakshmi Kristam, Parthiban Manavalan, and Nancy Amandi!

In Yaakov’s own words: “As our team expands, we’ll be able to take on larger projects, such as hosting additional writing workshops, trainings, and mentorships. I’m enthusiastic for what the future holds for us.”

👉🏼 We are looking for more editors for our Medium publication. You’ll build connections, network with industry leaders, deepen your knowledge of data engineering, and grow professionally. Apply to be an editor by filling out THIS form.

📚 Articles of the Week

• I Interviewed 200+ Data Engineers. Here’s What Separates the Best from the Rest! by Shashwath Shenoy

• A non-beginner Data Engineering Roadmap — 2025 Edition by Ernani Castro

• Winning the Data Job Interviews (video) by Shachar Meir

• Introducing Impressions at Netflix by Tulika Bhatt

• Build a productive code review culture by Ara Ramanathan

Have a great week and see you next time!

Cheers,

Xinran Waibel

Head of the Data Engineer Things community

(👉🏼 Don’t forget to join the DET Slack community and follow us on LinkedIn!)

Time flies by and we are near the end of 2024! My family started a holiday tradition several years ago: writing down and sharing a list of things we are grateful for. Here are a few things on my gratitude list. I’m grateful for the DET community members who generously offer their time and knowledge to help others grow. I’m also thankful for the genuine constructive feedback I received from my colleagues and for being able to grow from that feedback. Lastly, I’m grateful that I can be myself at work and with the community - because I’m at my best when I don’t need to pretend, when I can be curious all the way, and when I can say no.

While a growth mindset requires us to be aware of where we lack, it’s just as important to recognize what we have, such as strengths, achievements, and connections. I encourage you to create your own list and celebrate it!

🎉 A New Chapter for DET Medium Publication

Introducing the new Editor-in-Chief for our publication on Medium: Yaakov Bressler!

With over 7 years of experience in data engineering, Yaakov is currently a Lead Data Engineer at Capital One. Not only is he an active writer himself, but he also has been an editor for DET since the beginning of our publication. In the last year of working with Yaakov, I've witnessed firsthand how responsible, reflective, and selfless he is, so I have confidence in the positive impact he will bring to the publication.

Interested in learning and growing through writing? Here is how you can get started:

• Write for DET on Medium

• Join our editorial team

• New to technical writing? Sign up for the writer's workshop

🎬 DEML Summit 2024 Session Recordings

Recordings from DEML Summit 2024 are now available on YouTube, including:

• Data Management for LLMs by Abi Aryan, Author and Founder at Abide A.I.

• Event Tracking Redefined: A Data Engineer's Guide to Creating Actionable Insights by Saurabh Mishra, Staff Software Engineer at ThredUP Inc.

• Maestro Netflix’s Data ML Workflow Orchestrator by Jun He, Staff Software Engineer at Netflix.

• The Resilience of SQL by Rui Machado, VP of Data Engineering and Architecture at H&M Group.

Thank you to our sponsor (Databricks), co-host (SeattleDataGuy), and volunteers (Mert Bozkir, Matthew Hoang, Sreyashi Das, Rho Lall, and Aishwarya Prasad Venkatesh) for making this conference happen!

🪴 2025 Looking Ahead

We are always exploring new ideas to make the community better. I’d like to share a few things coming up in early 2025:

• I will be speaking at the Data Day Texas on Jan 25. If you plan to attend as well, drop me a note on Slack.

• We are now part of O'Reilly’s community partner program - which means our members will get access to free O'Reilly learning resources, e.g., free books!

• “My involvement in the DET community helped me land the new job” - I’ve heard this from several folks! I’m brainstorming a new program for community members to participate in to build their professional brands.

Any suggestions from you?

Have a great week and see you next time (probably in 2025)! ❄️

Cheers,

Xinran Waibel

(👉🏼 Don’t forget to join the DET Slack community and follow us on LinkedIn!)

The Data Engineering and Machine Learning (DEML) Summit 2024 is happening tomorrow and Friday (Oct 3rd - Oct 4th)!

Before the conference, I would like to share a few sessions I plan to attend myself. Hopefully, you will find some of these topics interesting too.

🎧 Xinran’s DEML Summit 2024 Playlist

1. Opportunity to Advance Data Engineering in the Environmental Sector

• Speaker: Prashank Mishra, Analytics & Data Engineer at Seirify Data

• Time: 8:00 am Thu, Oct 3rd (PT)

• Why I’m attending: I’ve heard someone joking that most data professionals today are working on getting more people to click on ads. Regardless of how true this joke is, I definitely noticed a lack of visibility in data engineering work in other fields, such as the environmental sector. I’m curious to learn about how Prashank solves challenges in the water utility space through data engineering.

2. Data Management for LLMs

• Speaker: Abi Aryan, Author and Founder at Abide A.I.

• Time: 9:00 am Thu, Oct 3rd (PT)

• Why I’m attending: As more teams are adopting LLMs into their products, so will they need support from data engineers and ML engineers for managing backend data, and the data requirements will look different from those of traditional ML models. I’d like to hear Abi’s perspectives on how data engineering and ML engineering will shift in the LLM world.

3. Event Tracking Redefined: A Data Engineer's Guide to Creating Actionable Insights

• Speaker: Saurabh Mishra, Staff Software Engineer at ThredUP Inc

• Time: 2:00 pm Thu, Oct 3rd (PT)

• Why I’m attending: The best data quality strategy focuses on prevention: in other words, establishing programmatic data contracts to stop data regressions from releases. How do you implement a technical solution and people process that can scale for constant product innovation? That’s a question I think about everyday. I want to hear about how Saurabh and his team are tackling this problem.

4. 10 Most Neglected Data Engineering Tasks

• Speaker: Veronika Durgin, VP of Data at Saks

• Time: 10:00 am Fri, Oct 4th (PT)

• Why I’m attending: As a data engineering team, you often have to juggle between business priorities and less visible work that offers long-term value. Securing time and resources for the latter can be hard. I’m curious to hear Veronika’s strategy addressing those common yet neglected tasks.

4. Unified Data + AI Governance with Unity Catalog

• Speaker: Victoria Bukta, Member Of Technical Staff at Databricks

• Time: 2:00 pm Fri, Oct 4th (PT)

• Why I’m attending: Likely you have already heard about the biggest data news of the year: Databricks acquiring Tabular. How will the existing open-source catalog frameworks evolve to support table format unification and AI? I’m hoping to get insights from Victoria’s talk on Unity Catalog.

🗓 Full Agenda and Registration

Check out the complete conference agenda here and explore other sessions. Sign up for DEML Summit 2024 for free at demlsummit.com/signup.

(Special thanks to Databricks for sponsoring the conference!)

Have a great week and see you next time! 🦆

Cheers,

Xinran Waibel (LinkedIn)

The DET community will be part of Big Data LDN this year!

• On Sept 18, I will be co-hosting a session on career development in data engineering in the X-Axis Keynote Theatre. If you are a data professional or aspirational date engineer hoping to level up in your career, this session is for you.

• At the end of the conference day, there will be a DET London Meetup. Please join us to network with the local data community. Looking forward to meeting you in person there. (Special thanks to Airbyte, Hetz Ventures, and Orchestra for co-hosting the meetup.)

👉🏼 Sign up for Big Data LDN and the meetup today (free registration).

🎟 Register for Data Engineering & Machine Learning Summit 2024

The DEML Summit is a month away! It is a free-to-attend virtual conference comprised of talks by data practitioners from various industries. Our lineup includes:

• “Event Tracking Redefined: A Data Engineer's Guide to Creating Actionable Insights” by Saurabh Mishra, Staff Software Engineer at thredUP

• “Idea to Insight: Changing Data Culture through Innovation” by Gu Xie, Head of Data Engineering at Group 1001

• “10 Most Neglected Data Engineering Tasks” by Veronika Durgin, VP of Data at Saks

• … and many more!

👉🏼 Register for the conference HERE (free registration).

(Special thanks to Databricks for sponsoring the conference.)

📚 Articles of the Week

• Netflix Maestro and Apache Airflow — Competitors or Companions in Workflow Orchestration? by Volker Janz

• PySpark Interview Questions for Data Engineers by Vishal Barvaliya

• Pinot for Low-Latency Offline Table Analytics by Uber Engineering

• Did Google Just Kill Streamlit? by Om Kamath

(Want to publish blogs with DET? Read our submission guidelines here.)

One last thing before you go… I want to give a special shout-out to the Data Engineering for AI/ML virtual conference on Sept 12, organized by the MLOps community. You will hear from data leaders from DuckDB, Salesforce, NVIDIA, Lidl, and many more. Don’t miss out on this great opportunity to network and learn.

👉🏼 Sign up for the conference HERE (free registration).

Have a great week and see you next time! 🦆

Cheers,

Xinran Waibel (LinkedIn)

I’m thrilled to announce that DET will be co-hosting the Data Engineering and Machine Learning (DEML) Summit with SeattleDataGuy on Oct 3rd and 4th!

The DEML Summit is a free-to-attend online conference, made of talks by those who truly get their hands dirty in real-world data — the practitioners. It will be our 3rd time hosting and we had over 3500 attendees last year.

If you are interested in attending, speaking, or sponsoring the conference, read on!

🌟 Speaker Lineup Sneak Peak & Registration

The agenda is still a work in progress but here are a few speakers we have invited and I’m super excited about:

• Jun He (Staff Software Engineer at Netflix): Jun will share more about Maestro - Netflix’s data orchestrator that recently became available to the public.

• Himateja Madala (Engineering Leader at Disney Streaming): Himateja will talk about the role of data lineage and metadata management.

• Shachar Meir (Data Advisor, Ex-Director of Data Engineering at Meta): Shachar has prepared a session with tips to help you stand out in data interviews.

The conference will provide a wealth of learning and networking opportunities that you wouldn’t want to miss. Grab your ticket today and stay tuned for more session announcements!

👉🏼 Sign up for the virtual conference HERE (free registration).

(Special thanks to Databricks for sponsoring the conference.)

📝 Call for Proposals (CFP)

We are actively looking for sessions for the following conference tracks:

• Data Engineering & Data Infrastructure

• Data Science & Machine Learning

• Tech Demo (tutorials that demonstrate how to implement techniques or incorporate frameworks)

👉🏼 Please submit your talk proposal HERE by Monday, Aug 12th. Speakers of accepted talks will be notified in early September.

Sponsoring DEML

Contact us at demlsummit-info@googlegroups.com if you are interested in sponsoring the conference.

Have a great week and see you next time! 🦆

Cheers,

Xinran Waibel (LinkedIn)

Creating more in-person learning and networking opportunities for the community has been my focus this year. The DET team hosted two more successful meetup events in the last month: one in Seattle and another in London!

The next meetup in the Bay Area and Seattle will happen in late June. Stay tuned.

🌟 DET Online Webinar on May 23

Career and personal growth isn't linear. It happens in pivotal "moments" that transform your perspective and actions forever. The same applies to Data Engineering. In this online webinar, Shachar Meir, Data Advisor and Ex-Director of Data Engineering at Meta, will share some of the most impactful "growth moments" from his career in data. These are real-world stories with significant lessons that offer valuable insights and learnings.

👉🏼 Sign up HERE (limited to 100 attendees).

🌟 DET Mentors at Snowflake Data Cloud Summit on June 6

Looking for guidance on growing your career in data engineering? Join us on Snowflake Dev Day to get mentorship from experienced DET Mentors:

• Date & Time: 3 PM - 4 PM on Thu, June 6

• Location: Moscone Center, 747 Howard St., San Francisco, CA 94103. Meet us at the Connect() lounge area.

• Signup: HERE (Free registration)

Below are the mentors you will get to meet and connect with in person:

• Hao Xu (Lead Software Engineer at JPMorgan & Chase)

• Jai Balani (Senior Data Engineer at Netflix)

• Sekhar Sahu (Staff Software Engineer at Zeta Global)

• Sharath Vandanapu (Senior Manager, Data Engineering at Confluent)

• Sreyashi Das (Senior Data Engineer at Netflix)

• Yaakov Bressler (Senior Data Engineer at Headspace)

🎟 Community Discount for the AIQCon

The MLOps community is hosting the AI Quality Conference on June 25 in San Francisco. In this event, you'll engage with industry leaders and builders in creating the gold standard of AI Quality. These topics of AI quality will be covered: accuracy, transparency, generalization, bias mitigation, efficiency, cost, and more.

Apply code testinprod at checkout for 20% off. (Thank you, Demetrios!)

📚 Articles of the Week

• How to Think about Internal Data Products as a Data Engineer by Hugo Lu

• Minds and Machines — AI for Mental Health Support, Fine-Tuning LLMs with LoRA in Practice by Volker Janz

• Data Gateway — A Platform for Growing and Protecting the Data Tier by Netflix Technology Blog

• One Big Table vs. Dimensional Modeling on Databricks SQL by Databricks (Most concepts discussed are applicable outside of the Databricks platform)

• How to turn an engineering incident into an opportunity by Cory Watson

(Want to publish blogs with DET? Read our submission guidelines here.)

Have a great week and see you next time! 🦆

Cheers,

Xinran Waibel (LinkedIn)