Quicklinks

Description

Context Diagram

Pattern Template

Press Release Template

AgileData App / Platform Example

AgileData Podcast Episode

AgileData Podcast Mind Map

AgileData Podcast Transcript

Agile Data Engineering Pattern

An AgileData Engineering Pattern is a repeatable, proven approach for solving a common data engineering challenge in a simple, consistent, and scalable way, designed to reduce rework, speed up delivery, and embed quality by default.

Pattern Description

The Data Match pattern provides an automated, granular comparison capability to efficiently identify and report discrepancies between two datasets, moving from row counts to specific data values.

This 'data diff' solution transforms hours of manual data reconciliation into minutes by optimising comparisons for cloud analytics databases like BigQuery, serving as a support feature for on-demand exception handling rather than a continuous trust rule.

Pattern Context Diagram

Pattern Template

Pattern Name

Data Match

The Problem It Solves

You know that moment when you're trying to figure out why your data numbers don't add up between two systems or tables? Or you're trying to check if everything from your source has made it to your target?

Often, you're faced with hours, or even days, of painstaking manual reconciliation, writing complex SQL queries, or dealing with inefficient brute-force comparisons that cost a fortune in compute resources.

This pattern solves the problem of quickly and efficiently identifying discrepancies between two datasets, saving immense time and frustration.

When to Use It

Use Data Match primarily as an exception thing or a support feature. It's most useful:

• When something goes wrong and you suspect data misalignment between a source and a target.

• For reconciling data after a migration or a complex transformation, especially when trying to pinpoint missing records.

• When you need to quickly compare two tables or datasets to find differences without writing custom SQL.

• When manual reconciliation is proving horrendous due to large volumes or complex logic.

• It's not designed as a core trust rule for every data movement, but rather for on-demand verification.

How It Works

This pattern turns a complex data reconciliation task into a few simple clicks.

Trigger: A user needs to verify data consistency between two datasets because a discrepancy is suspected, or an audit is required.

Inputs:

• A "table on the left" (source data) and a "table on the right" (target data). This could include data uploaded from an Excel spreadsheet as a "new tile".

• Specific "things in each table they want to double check", such as primary keys or particular columns.

• Access to a data catalog where all relevant "tiles" (data assets) are loaded.

Steps:

1. The user selects the first dataset (e.g., "tile A") and the second dataset (e.g., "tile B") from an interface.

2. The user specifies the columns or keys within each dataset that need to be compared.

3. The user initiates the comparison, often with a simple "hit go" or a "1 2 3 4 five click exercise".

4. Under the covers, the system performs an increasingly granular match:

   • It starts by comparing row counts.

   • Then, it compares keys between the two tables.

   • Finally, it compares specific data values (e.g., "date of births"). This layering of rules optimises the comparison and avoids costly brute-force operations.

5. The system optimises the underlying queries for the specific database environment (e.g., BigQuery), leveraging features like column storage and partition pruning for efficiency.

Outputs:

• A report detailing "all the things in the left that aren't in the right or vice versa".

• Specific identification of discrepant records, such as a list of "customer IDs that haven't flowed".

Why It Works

Data Match works because it automates and optimises a typically complex and manual process. It replaces hours of writing and running custom SQL with an intuitive, guided workflow, essentially providing a "data diff" capability as a service.

The pattern's effectiveness comes from its layered approach to comparison, moving from high-level checks (like row counts) to granular value comparisons, which makes it highly efficient and cost-effective, particularly for large datasets in cloud analytics databases.

It's like having an automated detective that quickly sifts through vast amounts of data to highlight the exact discrepancies, allowing analysts to focus on why the data is different, rather than how to find the differences.

Real-World Example

Consider a scenario where a data engineering team is trying to reconcile customer data that has been processed through new business rules with an existing Excel spreadsheet used by the business. Despite their efforts, they constantly find themselves "one customer out" after processing 100,000 customers, and each discrepancy is for a different, often obscure, reason. Manually finding that single missing customer is a "horrendous" and time-consuming task.

With Data Match, the team can quickly upload the Excel data as a new "tile," then use Data Match to compare it directly with their processed customer data. The tool rapidly highlights the exact single record that is out, turning "many hours of frustration" into "minutes" of investigation. This allows the team to spend their time understanding the root cause of the discrepancy with the business, rather than painstakingly searching for it.

Another example involves a data migration project where 100,000 customer records were sent via an API to a new vendor system, but only 80,000 appeared in the new system. Manually debugging this took "hours if not days". If Data Match had been available, they could have "back flushed" the final data loaded by the vendor as a "tile" and then compared it with the data they sent. This would have immediately identified the 20,000 records that didn't make it, saving significant time and effort in proving where the discrepancy occurred (e.g., showing the vendor that changes were made on their side, despite an agreement not to).

Anti-Patterns or Gotchas

• Brute-Force Comparisons on Large Datasets: Trying to match everything between two very large tables without any optimisation or "layering of rules" will be extremely costly in terms of compute, credits, or tokens.

• Using Non-Optimised Tools: Relying on generic open-source libraries that are not specifically optimised for your cloud analytics database (e.g., a tool skewed towards row storage databases like Postgres when you're using a column-oriented database like BigQuery) will lead to inefficient queries and high costs, failing to leverage the database's performance benefits.

• Overuse as a Primary Trust Mechanism: Data Match is an "exception thing," not a core "trust rule" to be run for every data movement. Over-relying on it for continuous validation can be inefficient and indicates a potential gap in proactive data quality monitoring.

Tips for Adoption

• Implement for On-Demand Use: Position Data Match as a powerful, on-demand "support feature" for when anomalies occur or specific reconciliations are needed, rather than an always-on data quality check.

• Optimise for Your Platform: If developing an internal version, ensure it's specifically tailored and optimised for your primary data platform (e.g., BigQuery) to maximise efficiency and minimise costs.

• Integrate with Data Catalogues: Make it easy for users to pick and compare any "tile" (data asset) loaded in your data catalogue, reducing the overhead of manual configuration.

• Focus on Post-Detection Analysis: Emphasise that Data Match quickly identifies what is different, enabling data professionals to then spend their valuable time on why the data differs and how to fix it.

Related Patterns

• Data Diff: This is the general term for the concept that Data Match embodies.

• Tracing Values: This related feature helps users specifically look for the flow of individual data points once discrepancies are identified by Data Match.
  

Press Release Template

Capability Name

Data Match

Headline

AgileData Launches Data Match to Slash Data Reconciliation Time from Hours to Minutes for Data Teams

Introduction

AgileData is thrilled to announce the availability of Data Match, a powerful new capability designed to simplify and accelerate the process of identifying discrepancies between two datasets. This feature empowers data analysts, engineers, and business users to quickly verify data consistency and pinpoint missing or mismatched records with unprecedented ease, ensuring greater confidence in their data.

Problem

"As a data professional, I've spent countless hours, sometimes even days, painstakingly trying to figure out why my numbers don't match between two systems or after a data migration. It's a horrendous, manual process of writing complex SQL or sifting through spreadsheets, often just to find that one elusive missing record. I just want to know what's different, quickly, so I can fix it."

Solution

Data Match transforms this laborious task into a quick, intuitive process. Users simply select two datasets (or "tiles"), specify the keys or columns they wish to compare, and with a few clicks, the system performs an optimised, granular comparison. It efficiently checks everything from row counts to specific data values, then generates a clear report highlighting all discrepancies. This eliminates the need for manual SQL queries and immediately pinpoints the exact records that are out of sync, saving hours of frustration and compute costs.

Data Platform Product Manager

"With Data Match, we're not just offering a new feature; we're fundamentally improving trust and auditability within our data ecosystem. It provides our users with an on-demand, highly efficient tool to quickly validate data alignment, ensuring that discrepancies are identified swiftly, reinforcing confidence in our data pipelines and overall data quality."

Data Platform User

"Honestly, Data Match is a game-changer. What used to take me 'hours, if not days,' to manually reconcile data or prove a discrepancy, now literally takes 'minutes' with just a few clicks. I don't have to remember complex queries; I just hit 'go' and get my answers, letting me focus on solving the why, not just finding the what."

Get Started

Ready to transform your data reconciliation process from hours to minutes? Data Match is available now within the AgileData platform. Connect with your AgileData team today to learn more about how to leverage this powerful capability, or visit agiledata.io for further details on adopting new patterns to craft your Agile Data way of working.

AgileData App / Platform Example

AgileData Podcast Episode

https://podcast.agiledata.io/e/data-match-agiledata-engineering-pattern-7-episode-75/

Listen to Podcast Episode

Subscribe: Apple Podcast | Spotify | Google Podcast | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser | Deezer | Podcast Addict |

AgileData Podcast Episode MindMap

AgileData Podcast Episode Transcript

Shane: Welcome to the Agile Data Podcast. I'm Shane Gibson. And I'm Nigel Vining. Hey, Nigel. Another data engineering bytes. Today we are gonna talk about a feature that we term data match. So tell me what it is and why I care

Nigel: Data match. Came out of the age old question, how do I know something in my source is in my target?

Or as we like to say, what's in the left isn't in the right or vice versa. This is generally called a data diff in a lot of places. Generally, it's a Pattern of doing an increasingly granular match of something on the left, which is generally a table of. Data and something on the right. So we start off and we say, is the number of records the same?

Yes, it is. Great. Cool. Are all the keys the same between these two tables? Yep, they are. Cool. Now is the date of births on the table on the left the same as the date of births on the table. And we effectively, we go from the very wide row count down to something very specific. Do the actual values match from left to right?

Now that all sounds pretty straightforward and it technically is, but under the covers there's a whole lot of SQL and engineering patterns that are happening to basically run all those queries. So that's not something we would expect a analyst to generally do 'cause it's a bit of a faf. So we came up with this feature called Data Match, where we effectively lead a user.

Pick a table on the left. Pick a table on the right. Pick the thing in each table. They want to double check and hit, go under the covers. Then we optimize those comparisons and then we produce a report straight back to the user saying these are all the things in the left that aren't in the right, or vice versa.

So we've made it a 1, 2, 3, 4, 5 click exercise and you can reconcile anything in your environment and it's.

Shane: I think this one from memory was an interesting problem. So we had a customer that we were doing the data work for. They had a series of business logic or rules that was in a fairly horrendous Excel spreadsheet.

So we used our way of working and we extracted the, the. Concept of those rules and we modeled the data properly and we applied those rules. And whenever we were trying to reconcile the numbers we got with the numbers in the spreadsheet, we were always one thing out. So let's just say it was a reconciling customers, they would have a hundred thousand and one customers.

We would have a hundred thousand customers. So we'd manually go through, find that one customer, work out that it was a timing problem or make sure we ran it at the same time. There would be one customer out and we'd go and check it. And then there was a bit of logic that they had, they didn't tell us about.

So we had to add that rule and somehow we just got into this loop where. We always won customer out and it was always for a different reason, but the cost of doing that manual reconciliation was horrendous. Data match allowed me to go, I can run that really quick. We grab the Excel data, I'd upload it, just dump it in like we do, get a new tile, compare it to the numbers we were producing consistently, and it would then highlight the one record that was out, you know.

Very short amount of time, and then I could spend all the time trying to work with them about why they had this record that we didn't. Or vice versa. So yeah, it just again, took something that was many hours of frustration and made up minutes, which was great. The idea of layering those rules though, that's important because otherwise you're just gonna brute force two very large tables.

Match everything and that is gonna cost you a shit ton of compute, a shit ton of credits, a shit ton of tokens depending on how your cloud analytics database vendor is charging you for that compute.

Nigel: Yeah, so we poked a couple of reasonably well known open source libraries when we first started, 'cause we're like, we're not gonna reinvent the wheel.

This seems to be a fairly solved thing. Surely there's just gonna be a package we can pull down, point it to two tables and hit go. And that's, and we will run it. Technically there are, and we did start with one and it did work. Where we tend to run into, where we ran into rolls was it needed quite a lot of configuration, so we effectively had to come up with a whole wrap to pass it, enough configuration to make it work.

And that was fine. That was more just a bit of app development to give it what it needs. But then some of the problem was, it was, as is usually the case, it had been developed to run. On a particular database, I think it was Postgres from memory, which is quite common, or was MySQL. So it was heavily skewed towards a row storage database and how row storage databases work.

And so it was optimized. So the queries. So the queries when we came to run them on BigQuery, they ran and that was fine, but we didn't really get any. The benefits of BigQuery being a column and database and partition pruning and the like. So we played with it and played with it, and it got closer and closer.

In the end, we thought actually it'd be just quicker to write a template that would run a BigQuery and we'll effectively do the same thing, but we'll make a template and make it. Specific BigQuery. And we did, and that's effectively where we got to so we can optimize what we give to BigQuery. So it's very efficient and it runs very quickly and it doesn't really cost us anything.

'cause we know where the performance and cost savings are with BigQuery and that's how we got to our Pattern. We effectively just took an open source, one found the strengths and weaknesses. Rolled a variant of it, uh, for us, for BigQuery.

Shane: And I think the other thing is we only run this when we need to. So it's not baked in as a core trust rule for every movement of data through every layer for every tile.

Is it?

Nigel: No. This is effectively an exception thing. This is when something goes wrong. This is somewhere where we can quickly go click and say, ah, there's 10 customers that aren't. Aren't in this table where we'd expect them to be. So it's a really quick way without having to regress to and go and customize something because it already has all the tiles and a catalog loaded.

You can just go pick tile A, pick tile B, compare them, show me the differences, uh, and go away. So it takes the first layer of context and the overhead, sorry, of thinking about it gives you your answer in a report. Then you can go and do, as you said, do the analysis. 'cause now I've got a list of customer IDs that haven't flowed.

I can grab one of those customer IDs and actually go specifically look for it. And that's a really quite simple proposition because that nicely flows onto some of the other features we've built around looking, tracing values.

Shane: I remember when we did that data migration use case, remember, where we grabbed data from a legacy source system and then pushed it through us, and then made it available as a API so that the new vendor could migrate the old data into the new platform.

And we had that gentleman's agreement, which was we do all the logic to match the new business rules for the new system. So effectively they'd hit the API for the data, grab the data, load it straight into their system, and there'll be no transformations between those steps. So that we always knew when we needed to change the way the data looked, it sat with us.

And when we did that test run and all of a sudden, let's say customer again, we passed. A hundred thousand customers out and only 80,000 turned up in their system. And we spent all that time manually trying to figure out why. And actually the answer was they had done some changes on their side between getting the data and loading it through their APIs, even though they said they wouldn't.

If I had just been able to take the final result that they'd loaded from their system and back flushed it in as a tile and then said, compare, that would've told me exactly which records didn't make it. And then yes, I would still have to talk to 'em about how come they didn't make it. But again, that would've saved hours if not days of proving we send a hundred thousand, you loaded 80.

We know. Therefore, it's somewhere between those steps and it's nothing to do with. Everything to the left of us would've saved us time. If we had to build it back then.

Nigel: Yeah, it's, that's why, I guess it's in the app, it's what I would call a support feature. It's something we don't use very often, but if we need to, it's there to quickly do something and we don't have to remember how do I data diff, what queries do I need to run?

Grab out some queries, change the table names and the key names in them to run them. Again, it's click, click, here's my report. You know, it's a small overhead, but. When you're trying to do a whole lot of things. Yeah. That you're grateful for it.

Shane: Yep. Hours to minutes. That's what I care about. Yep.

Nigel: Excellent.

Shane: Alright. I hope everybody has a simply magical day.

Quicklinks

Description

Context Diagram

Pattern Template

Press Release Template

AgileData App / Platform Example

AgileData Podcast Episode

AgileData Podcast Mind Map

AgileData Podcast Transcript

Agile Data Engineering Pattern

An AgileData Engineering Pattern is a repeatable, proven approach for solving a common data engineering challenge in a simple, consistent, and scalable way, designed to reduce rework, speed up delivery, and embed quality by default.

Pattern Description

The Activity Event Tile pattern tracks a sequence of events (e.g., subscription signups, subscription payments) for a core business concept by storing minimal event data and using a hydration layer to automatically generate pre-calculated metrics as views.

This allows business users and analysts to easily understand business performance and event sequencing without complex SQL, providing immediate, consistent insights into activity movements and financial impacts

Pattern Context Diagram

Pattern Template

Pattern Name

AgileData Activity Event Tile

The Problem It Solves

You know that moment when you're trying to analyse a sequence of related business events, like customer subscriptions changing over time, and you need to answer questions about trends, movements, or financial impacts?.

Typically, this involves writing "really horrible SQL" with complex "windowing functions" to connect these events and calculate metrics like 'time between' or 'dollar changes'.

The problem is, these manual calculations are often "not repeatable or reusable in other reports," leading to inefficiency and inconsistent insights. Furthermore, dealing with activities that happen out of expected sequence can be a significant challenge without this pattern

When to Use It

Use this pattern when:

• Track a sequence of events related to a single, core business concept, such as subscriptions.

• Derive time-based metrics and comparisons (e.g., changes between months, opening/closing balances, first/last occurrences).

• Easily understand the financial impact or count of specific activities over time.

• Simplify complex reporting that traditionally requires extensive SQL "windowing functions" or "slamming events together".

• Handle and identify activities that occur out of expected sequence (e.g., a subscription going from "cancelled" to "paid") without hardcoding sequencing logic.

• Provide repeatable and reusable calculations for your "last mile tool" or reporting layer

How It Works

This pattern provides a structured and automated way to manage and report on a sequence of activities.

Trigger:

• A relevant activity occurs for a core business concept (e.g., a subscription is created, paid, renewed, or cancelled).

Inputs:

• A business key that identifies the core concept (e.g., the subscription ID).

• A clear description of the activity itself (e.g., "subscription created," "subscription paid").

• A financial value associated with the activity (e.g., the payment amount) or a simple '1' if it's a non-financial occurrence.

• The business date when the activity occurred

Steps:

1. For each activity, a minimal record is created containing only the business key, activity description, value, and date.

2. These records are appended to a dedicated "activity event table". This table is intentionally "tiny".

3. The "magic happens in the hydration layer".

4. A "consume tile" is automatically generated. This tile includes a list of standard metrics that are calculated "on the fly" as views. These metrics automatically provide insights such as:

   1. Is this the first or last event seen for that concept?

   2. How many occurrences have been seen?

   3. What is the revenue impact or value change compared to previous activities?

   4. What are the opening and closing balances over time?

   5. What was the previous activity for every record?

Outputs:

• A highly efficient, append-only activity event table that can hold "squillions of records" with minimal storage.

• A "consume tile" that acts as a ready-to-use Data Asset, providing pre-calculated, real-time metrics and views.

• The ability to directly plug this data into "last mile tools" for immediate and intuitive reporting on business health metrics, such as subscription performance.

Why It Works

It's like having a dedicated, intelligent accountant for every ongoing business item, automatically calculating its dynamic financial and state changes. This pattern works because it:

• Automates complex calculations: It handles the "horrible SQL" and "windowing functions" that are typically required to analyse event sequences and calculate movements or changes, making these complex tasks simple for end-users.

• Provides real-time, non-stale metrics: By generating calculations as views "on the fly," the metrics always reflect the latest reality from the activity feed, preventing stale data issues.

• Enables easy trend analysis: It allows businesses to effortlessly answer questions about "movement between months" or "total value of cancellations" by simply dragging pre-calculated columns onto a report.

• Efficiently stores data: The core activity event table is "tiny" and contains only essential keys, values, and dates, making it very performant and cheap to store vast amounts of event data.

• Optimises query performance: Clustering the table based on activity types and dates ensures that queries for calculations are "fast and cheap".

• Simplifies sequencing and anomaly detection: By tracking every activity and its predecessor, it inherently handles activities happening out of normal order, making it "really easy to pull out" anomalies like a subscription going from "cancelled to paid". It also simplifies creating "funnel reports" by providing current and last seen activities

Real-World Example

Consider a Software as a Service (SaaS) company that needs to monitor key business metrics, often referred to as "pirate metrics," which track user acquisition, activation, retention, revenue, and referral.

Before adopting the Activity Event Tile pattern, an analyst might manually track events like "subscription created," "subscription paid," "subscription renewed," and "subscription cancelled" in a standard event table. To determine things like the net change in subscriptions month-over-month, the value of cancelled subscriptions in a given period, or the time elapsed between different subscription states, they would have to write highly intricate and time-consuming SQL queries involving multiple joins and complex window functions. These queries would often be specific to one report and difficult to reuse.

With the Activity Event Tile pattern, the company's data platform automatically records each subscription activity with its business key, type, value, and date. The pattern then generates a consumable data tile that exposes columns such as "subscriptions created this month versus last month," "total value of cancellations this month," or even the "previous activity" for each subscription.

Now, the analyst can simply drag and drop these pre-calculated columns into their reporting tool (their "last mile tool"), and the backend automatically performs the complex calculations, providing immediate and reliable insights into the health of their subscription business

Anti-Patterns or Gotchas

• Bundling Unrelated Concepts: A major pitfall is trying to include multiple, distinct core concepts (e.g., users, customers, and subscriptions) within the same activity event tile. This makes reporting "skewed" because these are separate activities, even if related, and keying them together becomes "problematic." This pattern works best when focused on a single, binding core concept.

• The original "activity schema patterns" that attempted to manage addiotnal related Core Business Concepts this with extra columns or JSON stringing doesn’t solve this problem at scale.

• Hardcoding Activity Sequencing: Don't attempt to hardcode the expected sequence of activities. This pattern naturally reveals when activities happen out of normal order, providing valuable insights rather than breaking the system.

• Pre-storing All Metric Combinations: Avoid the temptation to pre-store every possible combination of every metric. The pattern relies on calculations as views, which ensures metrics are always real-time and avoids persisting "stale value[s]"

Tips for Adoption

• Understand its Specialised Use: Recognise that this is "a slightly different Pattern" compared to standard event tiles, designed specifically for tracking streams of activity for a core concept.

• Keep the Activity Table Minimal: Ensure the underlying activity event table stores only the bare essentials: the business key, activity type, associated value, and the event date. This keeps it "tiny" and highly performant.

• Leverage Clustering: Cluster your activity event table on fields like activity types and dates. This significantly improves query speed and cost-effectiveness for the automated calculations.

• Focus on a Single Core Concept: When designing an activity event tile, ensure it revolves around one central business concept (e.g., subscriptions). Resist the urge to combine unrelated concepts into the same tile.

• Trust the Hydration Layer: Rely on the pattern's "hydration layer" to automatically generate the complex, real-time metrics and views, rather than trying to pre-calculate everything manually

Related Patterns

• Standard Event Tiles: This pattern is a variation of the AgileData Event Tile. Event Tiles typically record "a row of keys into the event table like customer, orders, product" at a specific "point in time" for reporting. The Activity Event Tile extends this to handle sequences of events and time-series analysis more effectively.
  

Press Release Template

Capability Name

Activity Event Tile Data Modeling

Headline

New Activity Event Tile Delivers Easy-to-Understand Business Insights from Complex Sequences of Events for Analysts and Business Users

Introduction

Today, we're excited to announce the launch of our new Activity Event Tile capability. This powerful feature is designed to transform how businesses track and analyse sequences of events related to a core concept, such as customer subscriptions or product usage. It provides automated metrics and views, making it incredibly simple for business analysts and users to gain deep, consistent insights into dynamic business performance without needing to write complex code

Problem

"As a business analyst, I used to dread trying to track changes in our subscriptions and derive meaningful insights. I had to write really horrible SQL with complex windowing functions just to connect events and calculate things like 'time between' activities or 'dollar changes'. The worst part was, those calculations were often not repeatable or reusable in other reports, which meant I was constantly reinventing the wheel and couldn't rely on consistent data."

Solution

The Activity Event Tile solves this by creating a highly efficient, append-only table that records minimal data for each activity – just the business key, a clear description of the activity, an associated financial value (or a simple '1' if non-financial), and the business date. The true power lies in its "hydration layer", which automatically generates a "consume tile" with a comprehensive list of standard metrics calculated "on the fly" as views.

This means users can now easily answer questions like "how many new subscriptions did I get this month compared to last year?" or "what's the total value of cancellations this month?" by simply dragging these pre-calculated columns onto their reports. It also inherently handles activities that happen out of expected sequence, providing insights into anomalies without hardcoding sequencing logic. This capability eliminates the need for complex manual SQL, providing real-time, non-stale insights directly in your reporting tools.

Data Platform Product Manager

"With the Activity Event Tile, we've transformed complex, bespoke reporting into a repeatable, auditable, and highly performant capability. Our data platform now consistently delivers quick business insights from multiple events, dramatically improving reusability and building greater trust in our data assets across the organisation."

Data Platform User

"It's incredible! Now, I can just drag and drop pre-calculated metrics like 'total value of cancellations this month' or 'movement between months' directly onto my reports. I get instant, reliable answers to my key business questions, without wrestling with complicated SQL or worrying about stale data – it's completely changed how I track our business health!"

Get Started

The Activity Event Tile capability is available today. Simply look for the new "consume tile" in your last mile tool, which exposes all the pre-calculated metrics. For more detailed information or to discuss specific use cases, please contact your data platform product manager.

AgileData App / Platform Example

AgileData Podcast Episode

https://podcast.agiledata.io/e/activity-event-tile-agiledata-engineering-pattern-6-episode-72/

Listen to the Podcast Episode

AgileData Podcast Episode MindMap

AgileData Podcast Episode Transcript

Shane: Welcome to the Agile Data Podcast. I'm Shane Gibson.

Nigel: And I'm Nigel Vining.

Shane: Hey, Nigel. Onto the next Agile Data Engineering Patterns episode. This time we're gonna talk about activity events. For anybody listening, if you haven't listened to the previous episode of Data Engineering Patterns, where we talked about our event tiles, I suggest you go and listen to that first, because we're gonna assume you listened and understood that before we move on to this one.

So activity events, it's a slightly different Pattern that we use for our event tiles compared to our standard Pattern. So module, why don't you explain to me what it is and why I care.

Nigel: Sure. So the previous one we talked about events where we basically. Insert a row of keys into our event table, like customer orders, product.

And that's cool because those three things are lined up at a point in time and we report on those. The activity events are a little bit different because effectively it's like a stream of things happening that we care about. Subscription is a really good example of where. Because we've got this, basically a whole lot of events come in where subscriptions are created, subscriptions are paid, subscriptions are renewed, subscriptions are canceled, and generally the type of.

Reporting that businesses do around subscriptions is, how many new subscriptions did I get this month compared to last year? And my number of subscriptions going up or going down, is the amount of revenue from my subscription static increasing, decreasing? So there's all these really useful questions that the activity schema can answer.

So the difference with our rule template for activities, they're really simple. All we capture is what the activity is, the business key. Subscription in this case. Basically a nice clear description. Like I said, subscription created, subscription paid, subscription canceled. We effectively describe what it is.

We also assign the. Financial value to each activity. Some activities don't lend themselves to a value. It may literally be a one that something happened compared to a dollar amount where a subscription payment is received, and that's where we would have that amount. So we can track those over time. And of course, the effectively the business data of when this activity.

It's really quite simple. We just create those and we just keep appending them into that table. The magic happens in the hydration layer, so where you take that list of keys attached to activities and we hydrate them into a. Tile and at the same time we include a list of standard metrics into that view that we calculate on the fly.

'cause we're looking across time periods of data, so we automatically have things like, is this the first subscription created event we've seen? Is this the last subscription created event we've seen? How many occurrences of subscription created have we seen? Are they repeated? What is the revenue impact of this?

Subscription paid activity, is the value going up or is the value going down when we effectively put it against the previous payment for that subscription? And what is that value change? What is the opening balance if we add up all the financial impact? Values. What's the opening balance of those over time?

What's the closing balance of those? Over time, although very simplistic, each of these has a relatively straightforward window and function, but we apply all those. Automatically on our activity data, so you don't care about it, you just throw your activities in and a consumer tile will turn up and all those metrics are there.

You can plug it straight to your last mile tool, and you are immediately reporting by the health of your business based on its

Shane: subscriptions. The key for this one is we were building out some reporting for software as a service company that needed to do all the pirate metrics and all that kind of stuff.

And what I was forced to do is create a bunch of events. So I'd say that a subscription was created, a subscription was paid, description was renewed, description was canceled using our standard event table, but then to be able to work out the between time or the between dollars for those events, I had to write really horrible sequel to slam those events together.

And then doing all those windowing functions. So what I ended up doing was slamming them all together and then doing all the window functions and calculations of the last. Which means that none of those calculations were repeatable or reusable in other reports. So by introducing this new Pattern, I can effectively now go pick that data up and I can say, what's the movement between months really easily?

How many subscriptions were traded this month versus last month? How many subscriptions were created three months ago and then canceled this month? What's the total value of cancellations this month? They're all really simple because I just drag, those calculations or those columns onto my report and.

The back end does all the heavy work. And so that's one of the things that is quite important. Those calculations are effectively views. So we're not pre storing every combination of every metric. It's each column is effectively being generated or calculated on the fly when I grab it into that last mile tool, correct?

Nigel: That is correct. Yep. 'cause we always. Want those metrics to reflect the reality in real time based on our last feed of subscriptions, because it's moving all the time and we're not moving historically, but we might be comparing to historic. So, but doing as a view, it really doesn't cost us a lot of extra and it makes sure we never persist a stale value.

Shane: The other thing is we are effectively storing. The idea of an event happening twice. 'cause we will still have the event in a standard event tile and then we are storing a another version of it effectively in the activity event tile. But the data is minimal because we're holding a very small number of columns that are effectively just holding keys or time or dollar values, correct.

Nigel: Yeah, the activity event table is tiny. It is literally the business key, what the activity was, the value associated with it, and when it happened, literally four or five columns, minimal. It's nothing. It's when we hydrate it and we go wide with the attributes and the metrics, but the actual activity table, we can put squillions of records in there and that's all good.

We cluster that one. Based on the activity types and the dates around that, because generally all of our calculations and metrics are all based around when something happened and what it was. So by clustering on those fields, it means that. The queries are actually very cheap 'cause we say we know where all the subscription created records are on the table and effectively all the dates are ordered within that.

So any of the calculations automatically based on cluster data. So it's fast and cheap.

Shane: The other thing that allows us to do is deal with those horrible, any sequencing problems. Subscriptions, created subscriptions, paid subscriptions, canceled subscriptions, renewed, right? It's reopened. The person's come back after six months and turn that subscription back on, and so we don't actually have to hard code the sequencing of the activities.

Because those activities come from the data. But what it means is we can see when activities happen out of the normal order we would've expected because they're turning up as something that happened that we didn't expect. And that's really important. But we get that for free because effectively an activity happens and activities inserted, and now I can see a state change for that subscription when it happened and what the dollar value, the impact of it was.

And that saves a lot of complex SQL querying. Last mile. You have to answer those type of questions.

Nigel: Yeah, because we keep track of what the previous activity is for every record, so we know if something's happened. We also give you the previous one for free because then you can say that they went from, as you said, canceled to paid.

That's an anomaly, and that's really easy to pull out because you shouldn't expect to see that.

Shane: But we. Show you that it's happened, which means if I want to create a funnel report, it's really simple because I have the last seen activity and the current activity and that's data. And therefore most of the reporting tools will be a pretty funnel report with very little need for me to do a whole lot of sequel gymnastics.

Absolutely. That's exactly what's happening. So what we've done though is we've iterated a version of activity schema modeling. Yes. So we've taken the bits that work for us, the idea of having a core concept that's. Finding as the core content for this activity event tiles, in this case subscription. The idea of having activity types and having a row every time we see a, an activity type happen for that subscription.

The idea of holding a date and a financial metric or value, which we need to report off the idea of. Precalculating or holding all those complex calculations of betweens and increases and decreases and last seen. But what we haven't done is gone to the next step, which is have the ability to hold multiple other concepts.

And that's because when we looked at the activity schema Pattern, it didn't deal with that particularly well. It was effectively an anti-patent that they pushed. And originally what it said was you had two extra columns and that kind of didn't work. Then they started Jason stringing all those other concepts in there.

So you put them in and unpassed them. And again, that didn't work for us. So what we took is the things that worked for us to solve that one particular problem. It's given us a whole lot of value for a bunch of other use cases, but we haven't picked up that last part of the Pattern because in our view.

Doesn't actually solve the problem that we need to solve for that use case.

Nigel: Yeah, bundling users, customers, subscriptions into the same activity doesn't make a lot of sense because effectively the reporting becomes a little bit skewed from it. 'cause they don't really go together. There are three activities that are happening side by side.

They're related, but to key them together is problematic.

Shane: Yeah. So we'll probably solve that another day, but not today. Alright, and on that note, I hope everybody has a simply magical day.

Quicklinks

Description

Context Diagram

Pattern Template

Press Release Template

AgileData App / Platform Example

AgileData Podcast Episode

AgileData Podcast Mind Map

AgileData Podcast Transcript

Agile Data Engineering Pattern

An AgileData Engineering Pattern is a repeatable, proven approach for solving a common data engineering challenge in a simple, consistent, and scalable way, designed to reduce rework, speed up delivery, and embed quality by default.

Pattern Description

The Event Tile Data Modeling pattern captures business processes and transactions within AgileData's three-layered data architecture, modeling "who does what" at a specific point in time.

It achieves this by storing only a small number of core business keys in insert-only event tables.

These event records are then automatically hydrated into a "consume tile" by joining with all relevant detail attributes that were current at the exact moment the event occurred, ensuring an accurate, time-staged narrative.

This provides data consumers with a rich, wide, and historically accurate view of events, simplifying the querying of complex historical changes and state transitions.

Pattern Context Diagram

Pattern Template

Pattern Name

AgileData Event Tile

The Problem It Solves

You know that moment when you're dealing with complex business processes, like an order progressing through various statuses (entered, updated, approved, paid), and you need to understand exactly what happened at each point in time?

Or when a customer's details change, and you need to see the state of an event at the exact moment it occurred?

If you only use simple Concepts and Details (C&D) for modelling, tracking these complex, temporal changes becomes a real headache, making it hard to query and maintain a consistent view of historical events.

The Event Tile pattern solves this by providing a clear, point-in-time view of business processes, ensuring accurate historical context for every transaction

When to Use It

Use this pattern when:

• You need to model a business process or transaction that happens, such as a customer ordering a product or an invoice status changing.

• There are complex changes within the event itself, or if the attributes related to the event's participants (like customer addresses) change over time, and you need to capture the state at the moment of the event.

• You require auditability and accountability of changes, providing a clear history of what happened, when, and why.

• Transparency of progress and impediments for data changes is valuable, similar to how transparency aids team coordination.

• You're working within a three-layered data architecture consisting of history, designed, and consume layers, and the event represents a core object type in the designed layer.

• You need a single, consistent pattern for loading various types of event data, regardless of their native source format

How It Works

This pattern works by creating lean, insert-only records that represent a specific business occurrence, then richly hydrating them for consumption.

Trigger:

• A business process or transaction occurs, or a significant change happens to a designated "driving concept" within that process. For example, a new order is placed, or an existing order's quantity or status changes.

Inputs:

• A set of business keys that uniquely identify the event and its associated participants at a specific point in time (e.g., customer ID, order ID, product ID).

• A driving concept identified for each event tile to dictate when new event records are inserted.

• Concepts: These are the "things" involved in the event, such as Customers, Orders, or Products, which hold unique identifiers.

• Details: These are the attributes related to the Concepts (e.g., customer name, product description, order quantity).

Steps:

1. A business process or transaction is identified for modelling.

2. An Event Tile is created, designed as a simple table that is a collection of only business keys. These tables typically contain no more than seven or eight keys, often closer to three to five, representing a "who does what" structure (e.g., customer orders product).

3. Event tables are insert-only; new records are only added if the unique situation described by the keys and effective date has not previously occurred.

4. Each event record includes an effective date (or business effective date), capturing the precise moment the event happened.

5. A driving concept (e.g., the 'order' in a 'customer orders product' event) is chosen. Any change to this driving concept (like an order update or quantity change) triggers the insertion of a new event record.

6. Records in the event tile are never end-dated. Instead, windowing functions are used during query time to determine which event was active at a specific point in time and what details relate to it.

7. The event tile is then automatically hydrated into a Consume Tile. This involves joining the event's keys to the relevant Concepts and their Details, ensuring that the attributes picked up are those that were current at the event's effective date.

8. For native source events (like Google Analytics data that arrives as an event), they can be unbundled into their component parts (concepts and details) and then re-hydrated back into an Event Tile. This standardises the loading pattern. Alternatively, if unique elements within the native event (e.g., page titles from GA4) need to be reported on, it can be more efficient to treat that native event as a Concept directly and split out the relevant data

Outputs:

• lean, history-preserving Event Tile table consisting solely of business keys and an effective date, representing a specific business occurrence.

• A Consume Tile that provides a "rich, wide narrative" of the event, encompassing all relevant attributes (from associated concepts and details) as they stood at the event's time. This provides a fully contextualised record for analysis

Why It Works

It's like having a perfect, unalterable ledger of every significant business action.

Instead of storing all attributes directly with the event (which would lead to data duplication and difficulty in managing change), the Event Tile stores only the relationship of keys at a point in time. This simplicity allows for insert-only operations, which inherently preserves history and makes auditing straightforward.

When a view of the event is needed, the details are rehydrated from their respective Concepts, always ensuring the attributes are accurate for the moment the event occurred.

This approach allows for a single, hardened pattern for loading events regardless of their original source, making data engineering workflows far more efficient, easier to troubleshoot, and simpler to understand.

It provides a consistent, reliable mechanism for handling complex data changes and ensuring that historical analysis always reflects the true state of affairs at the time of an event

Real-World Example

Consider a retail scenario where a "Customer Orders Product" event occurs. The Event Tile for this would simply contain keys like CustomerID, OrderID, ProductID, and the OrderDate (as the effective date). The OrderID might be chosen as the driving concept.

Later, the customer updates their address, or the order quantity changes. Because the OrderID is the driving concept, if the order quantity changes, a new event record is inserted into the Event Tile, capturing the new state. No existing records are updated.

When a user queries for the "Customer Orders Product" event data, the system automatically hydrates the Consume Tile.

This involves joining the event record with the Customer Concept's details (e.g., customer name, address at the time of the order) and the Product Concept's details (e.g., product name, price at the time of the order), and the Order Concept's details (e.g. order quantity, value at the time of the order). This way, even if the customer's address changed last week, a query about an order placed two months ago will show the address the customer had two months ago.

This pattern is also crucial for modelling complex state changes, such as an invoice moving through entered, updated, approved, and paid statuses. Each status change would trigger a new event record, providing a complete, granular history of the invoice's lifecycle.

Anti-Patterns or Gotchas

• Storing detail or numeric attributes directly on the event tile: This is a major pitfall. The Event Tile should only contain keys and an effective date. Storing details or numeric values on the event tile negates its core benefits, leading to data duplication and complexity when attributes change.

• Applying it to overly simple cases: For straightforward data, where attributes are self-contained and don't require complex historical context, a simple Concept and Detail (C&D) model might suffice and be more efficient. Don't force Event Tiles where they aren't needed.

• Ignoring the overhead of joins during consumption: While the Event Tile itself is lean, the hydration into Consume Tiles involves joins. If not managed well (e.g., with efficient partitioning and clustering in BigQuery), this can incur significant computational cost.

• Failing to define a clear "driving concept": Without a consistent rule for when a new event record should be inserted (i.e., what constitutes a "change" to the event), the data can become inconsistent or incomplete.

• Directly treating native source events as-is when internal unique reporting is needed: If unique concepts embedded within a native event (like unique page titles in Google Analytics data) need to be reported on, it's often more efficient to unbundle and model them as distinct concepts rather than querying across all hydrated event

Tips for Adoption

• Establish a clear three-layered architecture: This pattern thrives within a "history, designed, consume" data architecture, with Event Tiles residing in the designed layer.

• Strictly define object types: Adhere to the principle that your designed layer only contains three types of objects: Concepts, Details, and Events.

• Enforce insert-only behaviour: Ensure that Event Tiles are append-only. New records are inserted, never updated or deleted, to maintain a full history.

• Utilise effective dating and windowing functions: Implement effective dates on Event Tiles and use windowing functions for querying to correctly pull associated details that were current at the time of the event.

• Carefully select the driving concept: Clearly identify which concept's changes will trigger a new event record insertion to ensure the granularity of your event history is appropriate.

• Optimise for joins: When hydrating, ensure your underlying data platform is configured for efficient joining (e.g., correct partitioning and clustering in BigQuery) to mitigate computational costs.

• Standardise native event ingestion: Develop a consistent process for unbundling and re-modelling native source events (like Google Analytics) to fit the Event Tile pattern, or identify when it's more beneficial to model parts of them as standalone concepts

Related Patterns

• Concepts: The fundamental "things" or entities in your data model that Event Tiles link together (e.g., Customer, Order, Product).

• Details: The attributes associated with Concepts that are joined to Event Tiles during hydration to form a Consume Tile (e.g., Customer Name, Product Price).

• Consume Tile: The wide, denormalised table that results from hydrating an Event Tile with its associated Concepts and Details at a specific point in time.

• Activity Event: A discussed variation of the Event Tile pattern that provides another way of doing event modelling, intended for a future deep dive.

• Data Vault Modelling (Link Tables): While Event Tiles share similarities with Data Vault's Link tables (representing relationships between business keys), they deviate by not directly connecting to "details" in the same way, creating a variation of the pattern.

• Dimensional Modelling (Fact Tables): Event Tiles behave somewhat like fact tables as they capture business events, but they do not store numeric attributes directly. Instead, these attributes are stored in related Concepts and hydrated at the Consume layer
  

Press Release Template

Capability Name

Event Tile Data Modeling

Headline

New Event Tile Pattern Delivers Accurate, Time-Staged Event Narratives for Data Consumers

Introduction

Today, the AgileData team is excited to announce the adoption and refined application of our Event Tile data modeling pattern. As a core component of our three-layered data architecture, Event Tiles provide a revolutionary way to capture business processes and transactions. By modelling "who does what" at a specific point in time, this capability ensures that data consumers receive a rich, wide, and historically accurate view of events, making complex data easy to understand and use

Problem

"As a data consumer, I always struggled with understanding complex changes in our data over time. It was a real problem to query historical events accurately, especially when attributes like a customer's address changed. I also faced massive problems trying to decide what the grain of an event was, or what triggered a new event or a change to an existing one."

Solution

The Event Tile pattern addresses these challenges by specifically modeling business processes and transactions as simple collections of business keys—typically between three to five, and never more than eight. These event tables are insert-only, meaning new records are added only when a unique situation occurs. A "driving concept," such as an order, dictates when a new event record is inserted, capturing every change of state and ensuring a complete historical record.

When an event is captured, it is automatically hydrated into a "consume tile". This process involves joining the event's keys with all relevant detail attributes that were current at the exact moment the event occurred. This results in a comprehensive, time-staged narrative that provides an accurate snapshot of the business process at any given point in time, solving the complexity of querying historical changes. This approach also provides a consistent and efficient pattern for loading events, regardless of their source

Data Platform Product Manager

"With our refined Event Tile pattern, we've transformed how we manage historical data, directly solving the long-standing issue of accurately tracking complex event changes. This capability significantly enhances the auditability and integrity of our data assets, leading to a far more efficient and robust data engineering pipeline for our team."

Data Platform User

"I absolutely love that when I query an event, I now get a complete and accurate picture of precisely what happened at that exact moment in time, including all the relevant details. It makes understanding complex business processes so much clearer, and I no longer have to worry about data inconsistencies or attributes changing unexpectedly."

Get Started

The Event Tile data modeling pattern is a fundamental part of AgileData's architectural design and is automatically applied to new data product development, ensuring robust and accurate data delivery. For more details on how this capability underpins your data solutions, please speak with your AgileData Platform Product Manager.

AgileData App / Platform Example

Event Tile

Data Preview

Fields (Schema)

Related Tiles

Consume Tile

Data Preview

Fields (Schema)

AgileData Podcast Episode

https://podcast.agiledata.io/e/data-engineering-patterns-for-the-agiledata-event-tile-agiledata-engineering-pattern-5-episode-71/

Listen to the Podcast Episode

AgileData Podcast Episode MindMap

AgileData Podcast Episode Transcript

Shane: Welcome to the Agile Data Podcast. I'm Shane Gibson.

And I'm Nigel Vining.

Hey, Nigel, another episode in our data engineering Pattern series, and today we're going to talk about event modeling. I think what we'll do is I'll start off talking a little bit about the concept of event modeling, and then from there we can deep dive into the engineering practices and patterns that you use to make.

So we've talked before about we hold a three layered architecture. Data comes into history. We then move it into a designed layer where we measure it all up, make it fit for purpose, and then we have a consumer layer where everybody comes in and uses that data. Within that designed layer, we have three major object types.

We have, a concept, which really is a thing we see. It holds effectively a key for a thing that we know exists. Customers, employees, suppliers, orders, payments. We another tile type object type, which is detail, which holds the attributes about those things. So customer name, customer date of birth, supplier type, employee address. And so those details are bound next to the concepts. And then the third type we have is an event which basically says we saw a business process or a transaction happen. And within those event tiles, we have two types that we deploy. The first one is our standard event. This is where we see a business process happen a customer orders a product, a customer pays for an order, somebody ships the product or the order to the customer.

The customer returns a product to an order. And a little while ago, we introduced a second patent that we used conceptually called an activity event. And we can talk about that next time. But for now, let's deep dive into this idea that we've actually got a business process or a business event and we model it as an event.

Nigel: Yeah. So effectively what we do to make this magic happen is an event tile is basically just a collection of keys that something happened at a point in time, and these are all the business keys that are associated with it. So as you said at this date. This customer id, this order id, this product id, we effectively create that row in our event table of customer orders product, and then the next step is we automatically hydrate that event into a consumed tile.

So we effectively take that event and we join it to the details that are associated with it at that point in time. So the consumer's quite wide because effectively it's got all the customer attributes at that point in time. All the order attributes in a nice wide table, and that's all done at each point in time.

So then when something else happens with that order, or it's changed or it's status changed, we effectively insert another event record, which generates another consumer record out the other side to reflect the change to some aspect of that. And it also picks up all the same product attributes and customer attributes at that point in time.

So it's quite a simplistic thing because the tile is really quite boring. It's just a whole lot of business keys. But then when we hydrate it into a consume, it becomes a really rich, wide narrative for these events that are happening.

Shane: And so effectively that tile just holds a small number of columns.

And unique IDs or keys sitting in there. So we've found that when we model it, it's typically no more than seven or eight keys in that table.

Nigel: Yep. Seven or eight is getting up there. I haven't seen many past four, but Yep. I think seven is pretty much acknowledged as generally as wide as you go for those keys.

I haven't seen any business processes that tend to need more than that,

Shane: and because we use that format from Lawrence Co Beam of who does what. We end up with event tables that are always more than two keys. So we don't relationally model our event tables as two-way links. We're effectively modeling that core business process using that language.

So we always have, like you said, three to four to five concepts. Part of an event.

Nigel: Yeah. And it is generally , some variant of a, who does what? Who being a customer, for example, or a user that does being what they've done. They've ordered something. They've created something. And then the third part of the care is usually the what?

The thing. They've acted upon a product.

Shane:And one of the key things for me is because the orbit. The thing that typically holds a value, like quantity dollar is still a concept and it's part of that event the like a fact. Table from dimensional modeling, except we don't store the numeric attributes in that event.

We get those attributes when we build the consume because those attributes are held by the order concept . And if the event is customer orders product, then we know that the order quantity order value is sitting there and we rehydrate it. But it's different to a fact from dimensional modeling because we don't hold any attributes at all on that event tile, do we?

There's no detail on the event tile if I ever go and query it. It's just a bunch of keys.

Nigel: Yes. The detail comes from effectively the. Concept keys, concepts are associated with attributes. So yes, we don't store any attributes on that event table because we don't need to because they're sitting in our detail tables.

When we hydrate that we get the applicable. Detail attributes at the point in time.

Shane: The other thing is it's a form of a link table from DataBot modeling, but we don't follow the DataBot 2.0 standard where there is detail directly connected to that link table event table. In our term, it's not a link table because we don't follow the rules.

So we take the part of the patent we liked, and we created , a variation or our own patent for it. The other thing is those event tables are insert only. So we only ever insert new records in there. We never do any updates.

Nigel: That is correct. We check if the unique situation, that we're about to put in there is has already happened.

And if it hasn't, we then insert that, , a new row of keys effectively.

Shane: And then how do we deal with effective dating or dating those events?

Nigel: So what we do is the event table has an effective tape. So we say at this point in time, or this business effective tape, this customer had a relationship with this order and this product.

So we've effectively, we've created our slice, and then when we join the attributes to that road to hydrate it, we basically pick the attribute that was current at the point in time of that event. So we say what customer record? Was the one that was applicable because it may have changed since the customer may have gone on and done something else.

So they've updated their address, they've updated something else, or their status or whatever. So what we do is when we hydrate that row and the consume, we hydrate it at the point in time the event was created. So it is always accurate for when it occurred.

Shane: And then there's also this concept of a driving concept we call it.

Do you wanna talk me through that and how that works from an engineering point of view?

Nigel: So we picked the first concept in the event as our driving concept, and that's the one we look to. Changes occur. We insert a new event row, so customer orders product, the driving concept in that case is likely to be the order, because the order is created, the order is updated, it's fulfilled, dispatched, so we want all those changes of state of the order.

So that's the one that we drive a new record. So when their order is updated. We insert new record and we pick up the applicable customer and product at the time. If that order's not updated, we don't create a new event row because we are not driving off the customer or the product. So if the customer's updated, it doesn't matter because this is an order event.

Shane: just to clarify that when we talk about order being updated, what we're talking about is a new order Id turning up, so if a new order number turns up. Of an.

Quantity changed. We will then go and drive an insert of an event that related to that order changed.

Nigel: For that instance, we would, because the order is the driver, so that order has changed. So we want a new record to basically capture that the order has changed. So when we hydrate that record, you can see that the order quantity has changed on it.

Okay. So we're

Shane: effectively doing an insert of every. Of that event, driving off the driving concept. Yep. And then we don't end date any records, so now we are using a windowing function to see which event was active at a certain point of time. Yes. And therefore, what details relate to it? That is exactly right.

So what happens when we go and get data from somewhere like Google Analytics, which is another type of event, and it's not really customer orders product, it might be individual views webpage. Do we treat that differently

Nigel: at all? We've done both versions of this. We went through a phase of treating Native Source events as events, and then we went through a phase of basically saying an event in that situation can be treated as a concept because every row from GA four is effectively a concept because it's a marker that someone.

Something and it was captured. And then the second part of it is the detail is what that something is. So GA four is a little bit unique in that it arrives as an event. So we can treat it as a concept or we can model it as event by splitting keys out. They both get the end result. One's just a little bit more technical 'cause you're unbundling the event just to turn

Shane: it back into an event.

If we think about that, so we take what is an event? Record, and then we unbundle it to its component parts, a bunch of concepts and details and the idea that there was a relationship or an event happened, and then we rehydrate it back to itself because then we have a single Pattern for how we load events regardless of what type of data comes in.

And that makes us far more efficient in terms of hardening our code and troubleshooting and understanding the stuff.

Nigel: There is use cases where we would split an event that's coming in when we want to report on the individual concepts that are bundled in it. So if there's a requirement to report on all the unique, say for example, page titles coming through, that's where splitting it out into a page title or page.

Concept makes a lot of sense because then you effectively have this master tile, which has all your titles in it. So for reporting, you can go straight to that. You don't have to go through a convoluted Pattern just to get a unique title. As

Shane: one of the examples there is we have customers that hold customer attributes on their Google Tag Manager GA four Data.

So when it comes through, and we know we're gonna have to report on how many of those unique things there are. It's much cheaper for us to break those out to be concepts so we can report on them because they are things we wanna count and manage versus having to query across all the hydrated events to then find that attribute.

And on that, if you think about it, we're creating some event tiles. And then we're having to join to the concepts and then join to the details. There's lots of joins and we run on BigQuery. So isn't that an anti-patent to join all that data? Doesn't that cost us a fortune?

Nigel: That is a really good point.

Joins do come with an overhead and a cost and a computational cost. We mitigate some of that by making sure the amount of partitions we are presenting to join and. We are to effectively bring down that overhead, but join we a bit overhead.

Doing the things that keep it efficient.

Shane: What's the anti-patent for event tiles, do you reckon? When wouldn't we use this Pattern? We did try using Season Ds only because effectively I could create a concept of an order and then I could put the customer and the product and the detail for it. Yes, but we found as soon as we got complex changes within the event, like the customer's address changed.

Then it became a real problem to try and query it and event solved it because it gave us a view of everything about that event at the time the event happened. I can't think of an anti Pattern when we wouldn't do it. I think if we weren't modeling exactly the way we are modeling, we probably wouldn't use event times if we didn't have the simplicity of concepts, details, and events as being the only three types of objects that we model in our design layer.

We may have done something different.

Nigel: Yeah, season Ds definitely have their place and because it's really easy to take ized data, to wrap it in a concept and the detail and then rehydrate it as a consume. And for some data sets, that's absolutely fine 'cause we pass it straight through. And that's all it is because the concept is really simple.

It might be simple attributes about, , a user. So we've got a user ID as the concept, and then the user detail is the user's name and login and stuff. So it's a really simple thing and it's self-contained. We get a file of updates for users when they last logged in, and that's a good example of a c and d, and we just basically rehydrate that as a user tile.

We've got a unique list of users in the concept. And that's fine. And we might just run that as it is. And then as you said, we may have another consume, which is something else, and then we would create a rule to put them together at the consume layer, because sometimes that is just straightforward.

Shane: And I think that's the simple case.

And then we get the complex case, which is things like status change, invoice was entered, invoice was updated, invoice was approved, invoice was paid. And so that. Used to always cause me massive problems trying to decide what the grain of the event was, what was the trigger to say that a new event turned up or a change to that event happened, which probably leads into this idea of activity events that added as a another way of doing the event modeling for a whole bunch of reasons.

But I think we'll keep that one for another day. So I hope everybody has a simply magical day.

Quicklinks

Description

Context Diagram

Pattern Template

Press Release Template

AgileData App / Platform Example

AgileData Podcast Episode

AgileData Podcast Mind Map

AgileData Podcast Transcript

Agile Data Engineering Pattern

An AgileData Engineering Pattern is a repeatable, proven approach for solving a common data engineering challenge in a simple, consistent, and scalable way, designed to reduce rework, speed up delivery, and embed quality by default.

Pattern Description

The Automated Load Patterns based on Source Data Profiles pattern automatically profiles incoming data to determine its optimal loading pattern.

It classifies data as either event data (immutable, append-only) or change data (evolving records) by analysing characteristics like unique key volume and column names over time.

This automated classification then dictates the load strategy: an efficient partition replace for event data or a cost-effective upsert (end-dating historical records) for change data.

This capability removes the manual burden and accelerates data onboarding, providing a trustworthy and efficient process even when the data type is initially unknown.

Pattern Context Diagram

Pattern Template

Pattern Name

Automated Load Patterns based on Source Data Profiles

The Problem It Solves

You know that moment when you receive new data from a customer or system, and you're not sure if it contains historical changes, is a stream of unique events, or a mix?

Manually figuring out how to load this data efficiently and correctly can be a real headache. It slows down data onboarding, requires a lot of manual thought, and if you pick the wrong approach, you could end up with duplicated data or incorrect historical views

This pattern solves that by automating the decision of which data load method to use, significantly speeding up onboarding and reducing the cognitive burden on data engineers.

When to Use It

Use this pattern when:

• You're receiving new data with an unknown or uncertain structure – you're not sure if it's event data (immutable, append-only) or change data (records that might be updated over time).

• You need to onboard data quickly, especially when prototyping new loads for customers.

• Cost-efficiency and speed of data loading are crucial, as different data types benefit from different, optimised load methods.

• You require auditing and accountability of changes, as the pattern helps manage historical data accurately by end-dating previous records.

• You want an automated initial classification with a safety net for errors.

How It Works

This pattern functions like a smart data intake system, analysing incoming data to determine the best loading strategy.

Trigger:

• New data arrives in your system, often a file dropped into a secure bucket.

Inputs:

• The incoming data itself

• Optionally, user-defined change rules that help classify the data.

Steps:

1. Data Classification: The system first classifies the incoming data into one of two primary types: event data (immutable, e.g., GA4 user views, sensor readings, which typically have event timestamps and event names) or change data (records that can change, e.g., customer status, order details, often linked by a business key).

2. Source Data Profiling: The system profiles the incoming data, typically looking back over a recent period (e.g., 90 days). This involves:

   1. Analysing column names for identifiers common to event data.

   2. Determining the "shape" of the data, such as how many unique keys appear per day. For example, if 200,000 unique customer IDs appear daily, it's likely event data; if only a few hundred change, it's more likely change data.

   3. Applying thresholds based on known characteristics of event and change data.

3. Based on the profiling, the data is "lightly tagged" as either event or change data.

4. Load Pattern Determination: The system then uses this tag to select the appropriate load pattern:

   1. For 'Concepts' data, which involves inserting new unique keys, an Insert pattern is always used, regardless of event or change classification.

   2. For 'Details' data classified as Change Data, an Upsert pattern is employed. If a record is new, it's inserted. If an existing record's "row hash" has changed, the existing row is marked as 'ended' (end-dated), and a new row with the updated information is inserted. This is chosen for its cost-effectiveness due to partitioning and clustered keys.

   3. For 'Details' data classified as Event Data, a simple Partition Replace pattern is used. Since event data is typically received only once as a stream, the system deletes the last couple of days of relevant partitions and then re-inserts those events. This is a very cheap and fast method as it avoids looking through the entire table for existing records

5. Automated Trust Rules: Post-load, "automated trust rules" monitor the loaded data. If the system's initial guess about the data type was incorrect (e.g., 1% of the time), these rules will trigger alerts, such as "unique key warnings" or "effective date" inconsistencies, indicating a potential misclassification that needs review.

Outputs:

• Data that is correctly and efficiently loaded into the data platform using the most suitable pattern.

• Early identification of potential data quality issues or misclassifications through automated alerts.

Why It Works

This pattern works because it's like having an intelligent gatekeeper for your data pipeline. Instead of a human needing to manually inspect and configure every new data source, the system takes on the burden of "cognition". It leverages the inherent characteristics of data (timestamps, keys, volume patterns) to make an informed, automated decision.

This leads to faster data onboarding, as teams can prototype new loads without knowing the data's exact nature upfront. The selection of specific load patterns (like cost-effective partition replace for events or end-dating for changes) ensures optimal performance and reduced processing costs.

Finally, the integrated "automated trust rules" act as a safety net, alerting you if the initial automated guess was off, turning potential errors into actionable insights and maintaining data trustworthiness.

It builds a system that hints at the right approach and then tells you if it's going wrong, allowing for quick corrections

Real-World Example

Imagine a new client starts sending you daily data dumps into a secure cloud storage bucket. One day, they send a file named web_analytics_clicks.csv.

With the Automated Load Pattern Selection pattern, when web_analytics_clicks.csv arrives, the system automatically profiles it.

It observes that each record has a unique click_timestamp and that hundreds of thousands of unique records (like "GA4 user viewed page" events) appear daily.

Based on this profile and volume, it correctly tags it as event data. The system then automatically selects and executes a Partition Replace load pattern, efficiently deleting and re-inserting the latest days of click data.

A few weeks later, the same client sends customer_master_updates.csv.

The system profiles this file.

It identifies a customer_id as a business key and notes that while the volume of new customer IDs is low (e.g., 100-1000 per day), existing customer records occasionally show changes in fields like customer_address.

This pattern is recognised as change data, prompting the system to automatically select an Upsert load pattern. This means if a customer's address changes, the old record is effectively "end-dated" by the system, and a new record with the updated address is inserted, maintaining a full history of changes cost-effectively.

If, by chance, the system misclassified a very low-volume event stream as change data, "automated trust rules" would trigger an alert for "unique key warnings" because of unexpected duplicates, prompting a human review

Anti-Patterns or Gotchas

• Drastically Different Data Profiles: The pattern can be tripped up if the incoming data's characteristics dramatically change or are unusual for its type. For example, if an event stream suddenly has a very small volume (e.g., 5-10 events a day instead of thousands), the system might misclassify it as change data, leading to key errors and duplicate records on subsequent loads.

• Ignoring Automated Trust Rule Alerts: If the system guesses incorrectly (which happens in about 1% of cases), it will generate alerts (e.g., "unique key warnings," duplicates, or issues with effective dates). Ignoring these alerts will lead to untrustworthy data and broken processes, defeating the purpose of the pattern.

• Over-reliance without Understanding: Simply letting the system do its thing without understanding the underlying data types or the logic of the chosen patterns can lead to confusion when errors do occur, requiring manual intervention to correct configurations.

Tips for Adoption

• Trust the Automation (Initially): For new data, "let the system do that quick profile for us". This allows for much quicker prototyping and onboarding.

• Heed the Trust Rules: Pay close attention to the "automated trust rules". These are your early warning system for misclassifications or data quality issues.

• Understand Profiling Thresholds: Be aware that the system uses "thresholds based on what we know event data tends to look like". For very unusual data volumes, a manual check might be necessary, or the system might flag it for review.

• Consider AI Enhancements: As the data landscape evolves, explore using AI (e.g., LLMs) to provide better "hints" or "judge type patterns" to decide when profiling is unclear or volume is insufficient.

• Combine Patterns: The strength of this approach comes from "racking and stacking those patterns together to automate that work" – integrating profiling with specific load patterns and automated validation.

Related Patterns

• Automated Trust Rules: This pattern is highly complementary, as it validates the accuracy of the automated load pattern selection and alerts on issues.

• Schema Detection: Can work in conjunction, especially when initial profiling is inconclusive, to infer data structure and further inform load pattern decisions.

• Judge Type Pattern: A potential advanced pattern to resolve ambiguities when the automated profiling struggles to definitively classify data (e.g., very low volume events).

• Specific Load Patterns Enabled: This pattern selects from established data loading patterns such as:

  • Insert Pattern: For append-only data like 'Concepts'.

  • Upsert Pattern: For managing changes in 'Details' data by end-dating previous records and inserting new ones.

  • Partition Replace: A highly efficient method for loading 'Details' event data by replacing specific data partitions.

  • End-Dating: A key component of the Upsert pattern for managing historical versions of records
    

Press Release Template

Capability Name

Automated Data Loading with Smart Profiling

Headline

New Automated Data Loading Feature Accelerates Data Onboarding and Boosts Trustworthiness for Data Teams and Users

Introduction

Today, the Data Platform team is excited to announce the launch of our new Automated Data Loading with Smart Profiling capability. This revolutionary feature automatically determines the optimal way to load incoming data, removing manual guesswork and significantly speeding up the process of getting new data into your analytics systems. It's designed for anyone who needs to bring data into the platform, ensuring efficiency and accuracy from the first moment.

Problem

"As a data engineer, when new data arrives – especially from external sources – I always have to spend time figuring out if it's event data, change data, or something else. This manual classification slows down onboarding new data sources and carries the risk of picking the wrong load method, leading to data quality issues like duplicates or incorrect historical views. I wish the system could just figure it out for me!".

Solution

Our new Automated Data Loading capability intelligently profiles incoming data to automatically determine its 'shape' and optimal load pattern. When new data arrives, the system first classifies it as either event data (immutable streams like user views or sensor readings) or change data (records that update over time, like customer details or order statuses).

The system achieves this by:

• Profiling the source data over a recent period (e.g., 90 days), analysing factors like the number of unique keys appearing per day and common column names associated with event data. For instance, if millions of unique records appear daily, it's likely event data; if only a few hundred changes occur, it's probably change data.

• Lightly tagging the data based on this profile.

• Automatically selecting the most efficient load pattern:

  • For 'Concept' data (new unique keys), a simple Insert pattern is always used.

  • For 'Detail' data classified as Change Data, an Upsert pattern is applied. This means if a record is new, it's inserted; if an existing record has changed, the previous row is automatically 'end-dated', and a new row with the updated information is inserted, maintaining full history cost-effectively.

  • For 'Detail' data classified as Event Data, a Partition Replace pattern is used. Since event data is typically append-only, the system efficiently deletes and re-inserts only the most recent partitions of data, avoiding expensive full-table lookups.

This automated approach removes the manual burden and potential for errors. Furthermore, automated trust rules monitor the loaded data, alerting users to potential misclassifications (e.g., unexpected duplicates or effective date inconsistencies) if the system's initial guess was incorrect, ensuring data trustworthiness and providing a safety net.

Data Platform Product Manager

"With Automated Data Loading with Smart Profiling, we're removing the cognitive burden from our data engineers, allowing them to onboard new customer data significantly faster and with greater confidence. This capability enhances the trustworthiness and auditability of our data assets by ensuring the correct load patterns are applied, and crucially, it actively flags any data quality issues that arise from misclassification, turning potential problems into actionable insights."

Data Platform User

"I absolutely love that I can just drop new data files into our secure bucket, and the system automatically figures out the best way to load them. It’s incredibly fast, and I don't have to worry about complex configurations or whether I'm managing changes correctly. It just works, making my job so much easier and giving me trusted data right away!”

Get Started

You can start benefiting from Automated Data Loading with Smart Profiling immediately. Simply provide your data files as usual, and the system's intelligent profiling will do the rest, automatically applying the most efficient and correct load pattern. For more information on how this capability works or to understand the trust rules, please contact your data platform product manager.

AgileData App / Platform Example

Concept / Detail / Event Rules Logic

Concept = Insert

Detail = Upsert

Event = Insert

AgileData Podcast Episode

https://podcast.agiledata.io/e/automated-load-patterns-based-on-source-data-profiles-agiledata-engineering-pattern-4-episode-70/

Listen to the Podcast Episode

AgileData Podcast Episode MindMap

AgileData Podcast Episode Transcript

Shane: Welcome to the Agile Data Podcast. I'm Shane Gibson.

Nigel: And I'm Nigel Vining.

Shane: Hey, Nigel. Today, another data engineering Pattern. This time we're gonna talk about automated load patterns based on source data profiling. So take me away with what it is and why I care.

Nigel: This one's quite an important one, and it underpins our product in lots of ways. So essentially we have, we classify the two types of data that turn up in our projects as event data. Think GA4 user viewed page or sensor type data which captured a reading of something or change data info about a customer.

Customer changing status, customer changing address, orders, order is created and order is processed. We do a little bit of magic when you data turns up and we look at the data to determine the shape. If we think it's event data or change data, event data is generally quite clear cut because it will have event timestamps.

You'll have an event name. It'll have attributes that basically give us pretty good clue that it's event data. But what we also do is we look basically back over the last 90 days. Of that data and we profile it to see what shape of it. How many records are turning up each day based on the timestamps or the business key that we know.

So if we have business keys and we see changes of the data in the business key, we're fairly confident that it's changed data. So once we know. Whether the data is event or change data, we lightly tag it and then what we do is when we go to load it, this is where we determine which load pattern where we use.

So you know, we use concepts, details, events, concepts we always use in. Insert pattern and that's fine. Whether it's event data or change data, it doesn't matter. All we're doing is inserting a new key when we see it. When we get to detail about the data, this is where we change. So for change data we run in upsert pattern.

So if it's change data we haven't seen before, we insert it. If it's change data, . We already have, but it's changed. The row hash has changed. Then we update the existing row to mark it as effectively ended and we insert a new row. Whereas if it's event data, event data is generally only given to us once.

Cause it's just a long stream of something happened, something happened, something happened. So what we use for that is we don't need to go looking back through the whole table to see if we've got it. We use a simple partition replace and we basically just delete the last couple of days of petitions and then we insert those events again.

So we're always basically putting the new events and at the end of the table and we don't go looking for them. Cause this is a really cheap and fast pattern. So that's effectively what we're doing with this pattern.

Shane: So we're looking to say as the data we're being given immutable it's an event that happened. We'll never be given that exact same event again. That event never changes. So that's typically, gA fours where we use it a lot. Somebody viewed a webpage. And then if it is, then we do an insert Pattern ,

we just basically insert those records because we know, we don't have to worry about whether they have changed or not. 'cause they're all immutable and new. Otherwise it's a different Pattern where we know the data might change. Probably comes from some form of relational schema in the source system.

So we know that, the customer name might change or the customer address might change or the order quantity may change. And we therefore have to manage those changes and rack and stack them. So that's, so therefore we do an upset Pattern , where effectively, end date, the previous record and insert the new record

Nigel: correct, yes,

Shane: That end dating is quite an old Pattern , it's a bit of a ghost of data past. Why don't we do that fancy one where we don't end data and then just leave the insert dates and then go through into a whole lot of windowing functions whenever we need to determine what is the latest record.

Nigel: a lot of that's round cost we partition by those dates so we can quickly look at the current data. If we weren't to do that, it means we would be looking across. Potentially all the rows in the table every time for each business key to find out where that record started and ended.

So there's a lot more overhead to do it dynamically, whereas to do a quick look up because it's a partition and a clustered key, we can do it and endate the record very cheaply.

Shane: Cool. So it's a cost decision that we've used to pick that Pattern.

Nigel: That's exactly right. We could definitely do it with a lot less Complexity just to load all the records and then at the end do a window as you said, look for the latest record and mark it as current, but that's more processing. We have to read more data to do that pattern.

Shane: Talk me through the way the profiling works, how does it actually determine whether it thinks it's an event or change data?

Nigel: So the profiling is based on the change rule that the user creates, so the user will create rules, say in this case a. concept rule to load a concept tile. Once we have that rule, we can effectively run that rule and bucket the results that come back to work out how many unique keys are appearing.

Per day and we also check the column names as well looking for identifiers that we know that event data typically have, but profiling generally tells us because if we are getting lots of unique keys turning up every day, it's generally indicative of event data because you don't create say 200 ,000 customer IDs every day.

You may create. You know, 100 or 1000, but not large volume. So if we see 200 ,000 keys turning up every day, those are probably all unique event keys and we're pretty safe with that. So we use some thresholds in there based on what we know event data tends to look like.

Shane: And why do we do it? Like why don't you just force me to tell the system what kind of data it is when I start to collect it.

Nigel: that's a good question. We're effectively. Removing a little bit of cognition from your load and also what happens is it means we can onboard data a lot quicker because typically when we prototype a new load for a customer, we just keep giving a file. We may not know if it's event or change data. We just load that file.

And effectively we lead the system do that quick profile for us so you can happily go and create rules on that in 99 percent of the time it's going to go cool. It's event data and a load like this you say. Great, it's changed data below like this. The 1 percent of the time that we get our guests wrong, it'll still load, but effectively we would generally find that the second load throws a few errors and warnings because we're not loading the data quite right.

Shane: And that's where the trust rules come in, those automated trust rules will give us an alert to say this has been flagged as event data, but we are starting to see that the customer key is constantly getting change records coming through. That's an anti-patent for the patent that you've been selected.

Therefore, you probably need to go and change the config in context and rebuild some stuff so that it's safe, it’s trustworthy.

Nigel: that's exactly what happens. Effectively, the first thing we get is usually a unique key warnings because we're always checking for unique records. If we're not loading the data quite right, it'll start to create duplicates very quickly. The other thing we check for is the effectivity date. Of those records, because if we're getting lots of records turning up with the same effective date and we're trying to update based on that, it's also an indication that we possibly using the wrong pattern because we don't have an effective date for each business key.

So it's more likely to be event data.

Shane: And then Antipas where wouldn't you use this? Or what's the gotchas? You gotta watch out for?

Nigel: The anti pattern is where we get data turning up. That's very different to what we've seen and what we're based our profile on. Like an example would be event data, but very small number of events. Say you know five or 10 events a day, whereas typically events are in the magnitude of thousands, tens of thousands, hundreds of thousands.

You wouldn't expect to see. Four or five events being created each day, so that would trip us up. We would go. Oh, that's changed data. Is that volume so low and it would load right for a day and then we would on day two would start to get key errors because it would start to duplicate the data.

Shane: I like the way it effectively gives us a hint, that picks it for us. And then if something goes wrong, it tells us and we can fix it. I like. That because often we have customers that just send us their data, so they're not comfortable with us reaching in and collecting it, connecting internally to their systems or their cloud systems.

So we give them a secure bucket and they effectively just dump some data in there and we've never seen it before. So this helps us do that first bit of design around that data. I think with the new AI wave, there'll be some interesting ways we can enhance it. In theory as we see more and more systems, we could actually use that as a hint to one of the LMS to actually say, here's all the patterns of data we've seen before.

And then we could actually tag certain systems like google Analytics to say it is event data so that when we get, a customer that's only got three events happening through Google Analytics on a daily basis, it's gonna drop back to say, volume's too small. Let's go look at some of the other patterns and use a judge type Pattern to decide which one wins and also flags it to say not enough volume to actually determine the profile properly.

Fill back to schema detection. you probably want to go and look at this one and just check it or wait to see if you get any alerts to say the data's not so trustworthy. It's all about racking the, stacking those patents together to automate that work. Excellent. I hope everybody has a simply magical day.

Quicklinks

Description

Context Diagram

Pattern Template

Press Release Template

AgileData App / Platform Example

AgileData Podcast Episode

AgileData Podcast Mind Map

AgileData Podcast Transcript

Agile Data Engineering Pattern

An AgileData Engineering Pattern is a repeatable, proven approach for solving a common data engineering challenge in a simple, consistent, and scalable way, designed to reduce rework, speed up delivery, and embed quality by default.

Pattern Description

The Trust Rules pattern provides automated data validation to ensure all incoming data is fit for purpose and trustworthy.

It bakes in essential checks such as unique business keys and business effective dates, which run automatically upon data load or table refresh.

Users can also define custom validation rules for specific columns.

Results are collected, persisted, and surfaced via applications or alerts, with the system optimising validation for cost and speed through smart partitioning and clustered columns.

Pattern Context Diagram

Pattern Template

Pattern Name

Trust Rules - Automated Data Validation

The Problem It Solves

You know that moment when you're working with data, and you're just not sure if you can actually rely on it? Is it accurate? Is it complete? Are there duplicate entries mucking things up downstream? Trust Rules effectively solve the problem of ensuring data is "fit for purpose" and addressing the "bane of engineers lives" when it comes to data validation. When teams don't automatically validate data, crucial issues like duplicate business keys or missing effective dates can go unnoticed, leading to untrustworthy data and downstream problems. This pattern ensures that a core set of necessary data quality checks are performed automatically, removing the burden from individual engineers to constantly remember and implement them

When to Use It

Use Trust Rules as a default, not an exception, and have them "baked into the product from day one"

Use this pattern when:

• When new data arrives or a table is refreshed.

• When you need to "check its fit for purpose" and ensure data is trustworthy.

• To ensure fundamental data integrity, such as checking for unique business keys or the presence of a business effective date for every row.

• When specific, user-defined validations are needed for particular columns (e.g., email validity, phone number format).

• To reduce the effort and cognitive load on data engineers by automating common, mandatory checks.

It's especially helpful for establishing a baseline of data quality that is always applied, regardless of context

How It Works

This pattern establishes a robust, automated data validation system.

Trigger:

The pattern is initiated when:

• A task is identified that requires data validation, typically when new data is loaded, turns up, or a table is refreshed.

• This process is often initiated by a PubSub message indicating that a table refresh is complete

Inputs:

• Pre-defined automated trust rules (e.g., unique keys, effective dates).

• Pre-defined user applied trust rules (e.g., not null, is number, is email, is date).

• User-specified trust rules (e.g., email domain validation, value between, regex, masking, formulas).

Steps:

1. A table is loaded or refreshed with new data.

2. Upon completion (triggered by PubSub), a series of data validation checks are run against the new data.

3. Automated (baked-in) checks are executed: These are mandatory, cannot be turned off, and include fundamental validations like checking for unique business keys, a business effective date for every row, and non-nulls in strategic places.

4. User-specified checks are executed: Users can define additional, custom validation rules for specific columns they care about (e.g., validating email formats or phone numbers using regex). The logic for these rules can often be leveraged from open-source libraries.

5. The results of all checks are collected and persisted into a data layer, treated like any other data.

6. These results are then surfaced to users through various channels such as an application interface, reports, or alerts (e.g., via Slack), with the severity of the failure dictating the notification method.

7. A history of all validation rules run against each table (down to a column level), including successes and failures, is maintained

Outputs:

• Validated data that is confirmed as "fit for purpose".

• A clear, traceable history of data quality for every table and column.

• Early identification of data quality issues, allowing for immediate action and reducing downstream problems.

• Increased trust in the data for all users and stakeholders

Why It Works

This pattern works by automating the foundational data quality checks that data engineers would otherwise have to manually implement for every new dataset. It removes the burden of remembering and applying these core rules, allowing the "machine to do it for me". By baking these rules in "by default and not as an exception after the fact," it ensures consistent quality from day one. The system tells the user if they've done "something dumb," providing immediate feedback and fostering correct behaviour without constant manual oversight.

Furthermore, the pattern works by optimising the validation process for cost and speed. By moving from external, expensive, or less controllable tools to an internally managed system, and by leveraging database features like partition pruning and clustered columns, checks can be performed more efficiently and cheaply. For instance, with a column-oriented storage engine like BigQuery, only the relevant column for a specific time window is scanned, significantly reducing compute and query costs. It turns code into a shared asset, not a fragile mess of files (this is for Git, but the underlying principle of making data a shared, trustworthy asset applies).

Real-World Example

Imagine a data engineer onboarding a new "history tile" of data. With Trust Rules, they don't have to manually write checks for every fundamental requirement. The system automatically performs checks to ensure there's a unique business key for every row and that the effective dates are correct, without the engineer having to worry about these basic validations.

Separately, if a user specifically cares about an "email" column within that data, they can define a user-specified trust rule to ensure that "all the values loaded into here are from a specific email domain" This check runs automatically when new data is loaded, and if any emails are invalid, the system persists the result and can send an alert, perhaps to Slack, notifying the team of the data quality issue.

Anti-Patterns or Gotchas

• "Data quality on everything everywhere, every time": Applying every possible data quality test to the entire table, regardless of necessity, leads to excessive cost and performance issues. Some data becomes immutable, meaning earlier checks don't need re-running over the full historical dataset. Smart partitioning and focused checks on the latest data window are crucial.

• The "Noise Problem": Creating too many trust rules, especially for columns or conditions that don't genuinely impact data trustworthiness or stakeholder value. This can lead to a deluge of alerts that are ignored, causing truly important issues to be missed. If "nobody cared about that rule," then it shouldn't generate noise.

• Using external, expensive, or overly technical data quality products: While good for initial exploration, relying on external services that are difficult to maintain, run, or lack granular control (e.g., over partition pruning) can become a "technical" and "expensive" anti-pattern. Bringing the pattern "back into the core of our context plane and our execution patterns" allowed for greater control and cost-efficiency

Tips for Adoption

• Bake it in by default: Ensure that fundamental, non-negotiable data quality checks (like unique keys, non-nulls in critical fields) are automated and unchangeable from the outset.

• Provide user-defined flexibility: Allow users to easily specify additional validation rules for data columns they specifically care about, empowering them to ensure quality for their particular use cases.

• Persist and surface results: Store the results of every rule execution as data and make it easily accessible through reports, applications, or alerts, adjusting the notification method based on the severity of the failure.

• Optimize for cost and speed: Design the system to leverage underlying database capabilities (like partitioning and clustering) to scan only necessary data windows, reducing query costs and execution time.

• Prioritise value over quantity: Only apply trust rules where they truly add value and address a genuine concern about data trustworthiness, rather than implementing "data quality on everything everywhere, every time". This helps reduce noise and ensures that teams focus on critical issues.

Related Patterns

• PubSub: Used as the mechanism to trigger trust rule execution after a table refresh is complete.

• Load Patterns: Trust rule results data can be used to assess if data is arriving at the expected time, indicating missing expectation levels from the organisation.

• Context Plane and Execution Patterns: The trust rules themselves are defined within a "context plane" and are part of the broader "execution patterns".

• Data / Reporting Layer: The results of trust rule executions are persisted in a data layer and surfaced through a reporting layer
  

Press Release Template

Capability Name

Trust Rules

Headline

New Trust Rules Feature Ensures Data is Always Reliable and Ready for Use for Data Engineers and Information Consumers

Introduction

We're thrilled to announce the launch of Trust Rules, a powerful new capability designed to automatically validate your data. This feature provides immediate confidence in data quality for data engineers and business users alike, ensuring all data is "fit for purpose" and trustworthy from the moment it lands. It's about ensuring data is reliable by default, not by exception signed for anyone who needs to effortlessly stack data from multiple similar tables, such as those from multiple publishers in a data clean room, without having to write intricate or "horrible SQL". It delivers a "magical" experience by making data integration both easy and efficient

Problem

"As a data engineer, I used to have to constantly remember to build standard data validation checks for every new dataset, like making sure keys were unique or that effective dates were always present. It was a real pain, and if I missed something, it could cause big problems all the way downstream. The 'noise problem' from too many irrelevant alerts also made it hard to spot what truly mattered."

Solution

Trust Rules automates the foundational data validation process, making it easy and largely automated.

Whenever new data is loaded or a table is refreshed, the system automatically performs essential, baked-in checks, such as verifying unique business keys and the presence of a business effective date for every row.

Furthermore, users can easily define their own custom rules for specific columns, like ensuring all emails are valid or checking phone number formats, without needing to write complex code.

The results of these checks are collected, persisted, and then clearly surfaced through an application, reports, or alerts (e.g., via Slack), with the severity of the failure determining the notification method.

This not only ensures data integrity but also optimises the validation process for cost and speed by leveraging smart partitioning and clustered columns

Data Platform Product Manager

"With Trust Rules, we're taking away the 'bane of engineers' lives' by automating core data validation, making our data platform significantly more auditable and profoundly boosting trust in the data. This foundational capability means our teams can focus on delivering high-value work, knowing the data quality is inherently managed and consistently applied."

Data Platform User

"What I love about Trust Rules is that I don't have to constantly worry if the data is accurate; the system automatically tells me if I've done something 'dumb' before it becomes a problem. Now, when I load data, I just know the essential checks are handled, and I can easily add my own custom validations for what I care about—it's truly a game-changer for my confidence in the data”

Get Started

Trust Rules are baked into the platform by default for all core data, running automatically when new data arrives or a table is refreshed. To leverage user-defined trust rules for your specific data quality needs or to learn more about the automated validations, please consult the AgileData platform documentation or speak with your Data Platform Product Manager.

AgileData App / Platform Example

Notifications Dashboard

Trust Rule results for a Tile

Apply Predefined Trust Rule

Create Custom Trust Rule

AgileData Podcast Episode

https://podcast.agiledata.io/e/trust-rules-to-validate-your-data-agiledata-engineering-pattern-3-episode-69/

Listen to the Podcast Episode

AgileData Podcast Mind Map

AgileData Podcast Episode Transcript

Shane: Welcome to the Agile Data Podcast. I'm Shane Gibson. And I'm Nigel Vining. Hey, Nigel, another episode of Agile Data Engineering Patterns. Today we are going to talk about trust rules at a high level this time because this one has patterns within patterns within patterns. So let's start from the beginning.

Nigel: Cool. So trust rules, one of those bane of engineers lives, but it was something that we believed and passionately we believed it should be easy, largely automated and baked into the product from day one. Trust rules was something that we did by default and not as an exception after the fact. Trust rules effectively.

They're data validation rules, as I talked about on our previous bytes. Um, when a table is loaded, when new data turns up, we refresh a table. Then as I said, we use pub sub to say, Hey, I'm done. What do you want me to do now? So one of those things that we do off the end of a table refresh because new data's turned up, is basically to turn around and say, cool, new data's gone to the table.

We need to run data validation on that data. To check its fit for purpose. So what we then do is we turn around and we basically run a series of checks. Largely they're automated things like, are the business keys unique? Do we have a business effective date for every row? Those run every time, regardless.

You can't turn them off. You can't change them. They are baked in. 'cause if we have duplicate rows, then we've got a problem all the way downstream. The second level of checks are user specified. Ones a user may say. Hey, I really care about this email column. I need to make sure all the values loaded into here are valid emails.

So the user will effectively say, Hey, run an email validation check when this table is loaded. Once we've run those checks, we basically grab the results. We persist them. Uh. Reporting report to notify users if any of them fail and how many of them fail and we keep track of them so we can see the history for every table down to a column level, what validation rules have been run against it, and all the successes and passes.

That's in a nutshell.

Shane: So effectively we define the the trust rules which sit in our context claim. We then execute them when they need to be executed. We store the results of every execution as if it's data, and then we surface that data, those results, we call them in any way we want to. Maybe in the app, maybe a report, maybe an alert coming out via Slack, depending on the severity of that failure.

It's interesting, the name trust rules, you often hear of them called data quality, data validation. Why did we call it trust? Actually, these are the things that help people trust the data they're seeing. So that's what we wanna do. We wanna make sure it's trustworthy. And that's why we picked trust rules as a name.

And then the other thing is if you're a data engineer, there's a bunch of automated tests you always have to build, like you said, are the keys unique? Do we have duplicate values? All those kind of things. And as part of the trust rules patent, we just automated them as the user of the system. I don't have to care.

If I do something dumb, the system tells me I've done something dumb and it stops me. I don't actually have to keep remembering that there's a core set of things that I need to do because the machine does it for me, and I think that's part of it, is that I don't need to do the effort. I don't need to remember to do it.

It just. Gives me a, a lovely message when I'm doing the wrong thing and I change what I do. So I think that was actually one of the biggest wins for us when we implemented this patent.

Nigel: Yeah, exactly. The built in rules, by default, it's fantastic because we can onboard new data, you can chuck it in, you can tick a few boxes and get that pipeline basically productionized.

Within an hour or two. And then you don't have to worry about, you don't have to go, oh, I need to make sure that's unique. Oh, I need to make sure that's not now. Or I need to make sure that's a proper sequence. Because effectively we say, Shane has just onboarded a history tile. These are the things that a history tile has to have checked to be safe.

And we do those checks, and it doesn't actually matter what data's going into that tile because we still need to make sure we've got a. Unique business key for those rows. We need to make sure that the effect of those rows is correct and we're checking for nails in some strategic places. So all of those things, there is no context, none have to worry about.

It's not afterthought, it does it automatically. So you just load your data. Use it, the checks being taken care of for you.

Shane: And then there's some things I do care about. So then I use the user to find trust rules and I can create, is it an email address? Is it a phone number? All those re rejects field to make sure it's got one.

I need all those kind of, not custom, but those commonly applied ones. But I don't want them applied every time. And one of the keys that we found was. That the logic for those user defined trust rules, effectively the rejects or the masking or the formula to say is this thing, this thing. It is readily available out there.

There's a large number of open source products where basically you can grab that library of data quality rules, and you can apply it yourself. You don't actually have to reinvent the wheel for that one, but from memory, in the beginning, we did actually use a Google Cloud service. To define and act, build and execute all the data quality checks.

And then a while ago, I think it was a year or two ago, we actually bought it back into the core of our context plane and our execution patterns. Why was that?

Nigel: Yeah, that's right. We started out using an open source product that Google wrapped. It was basically a data quality. Engine defined by the YAML files, which described all the tests, which were basically the sequel that would be run, and that was pointed at tables and it executed those tests and it produced the results and stuff.

And that sort of got us up and running and gave us a flavor of how these things work. But it became apparent that it was quite technical to maintain it. And run it. And we wanted to pull basically trust rules back into the app and make it as simple as ticking a box rather than have to configure the separate product outside.

The second thing was it was quite expensive, relatively a little bit more expensive. With the way the validations were run. We had a little bit less control over. Partition pruning and using clustered columns. So effectively we took the Pattern, the open source Pattern, and we basically wrote our own version of it.

We followed some of it, we wrote our own, and effectively we sped up the tests. We can do a lot more tests and less time 'cause there was less building on the fly happening. And they were tended to be cheaper because we were doing a lot. Smarter petitioning pruning because we knew the watermark of the data we just loaded so we effectively could pass.

Hey, we've just loaded 14 days worth of data. You don't need to check prior to that 14 days 'cause we've already done that. So we did a whole lot of smart stuff to just speed up those tests and make them cheaper. And that's effectively what we've had now for, oh, it's probably coming up two years. It's quite a while.

Shane: I think that's probably the core anti-patent that we see, isn't it, is a data quality on everything everywhere, every time. So that idea that you have to scan the full table for every data quality test is an anti-patent because there are certain tests where the data becomes immutable. That test will never change.

So you can run it, store it. And then there are other tests where, for example, you want to check how many distinct values there are in a particular column. Then you are going to have to table scan or do something smart and to reduce the cost of running that.

Nigel: Yeah, and that's what I mean by we know those because we have architected the tables themselves.

We know which tables are partitioned and those are the ones we will partition scan for the data. We know which ones are clustered, so they're the ones typically where we're gonna look at. Every key, like a concept table of customer IDs, because we would automatically cluster the customer ID to do a is unique check on that clustered column is basically a freebie because under the covers, the database knows that column is unique.

We don't have to do a lot of work to do that. So on a concept, we only care that the key is.

We are scanning a window of data in multiple partitions because we're looking at actual attributes, like you said, email addresses, but we don't need to look at the email addresses from last month because we've already checked those. We only need to look at the email addresses that have turned up today, yesterday in the current window, and we can do that very cheaply.

'cause basically we are saying just grab the email address column for. Two days, three days, and check them. And it's a tiny shard of data and it works really

Shane: well. I think that's one of the benefits of using a column or storage engine engine under the covers like BigQuery because we just grab one column outta the entire table and records for a small period of time.

And therefore the compute cost, the query cost is, is tiny. And the other one is when you do the everything everywhere, every time is the noise problem. And when I first started off, because it was so easy for me to create trust rules, I would add them everywhere and never look at them, and then I'd start getting massive amount of alerts because actually a column I really didn't care about that much, or we didn't need to validate for any trust reasons, would just keep giving me constant problems.

It wasn't that the data was untrustworthy, it was that the data didn't quite match that rule, but nobody cared about that rule. Nobody was gonna go fix it in the source system. And so that noise to signal ratio is always a problem. And we can go through lots of examples in more detail around how we do trust rules, but.

One of the trust rules we'll talk about in the future is the idea of the load patterns. So using the load patterns for tables to determine when we may be missing the expectation level from the organization of when data should turn up. So again, you can use this trust rule results data to solve many problems, but the key is I only put it on where it has value.

Otherwise, you spend money where you don't need to. And you've gotta go manage thousands of messages and you're just gonna ignore them and miss the ones you really care about.

Nigel: Exactly. Yep. Keep it simple.

Shane: Yeah. Reduce that complexity. All right. That's another agile data engineering Pattern in the can. I hope everybody has a simply magical day.

Quicklinks

Description

Context Diagram

Pattern Template

Press Release Template

AgileData App / Platform Example

AgileData Podcast Episode

AgileData Podcast Mind Map

AgileData Podcast Transcript

Agile Data Engineering Pattern

An AgileData Engineering Pattern is a repeatable, proven approach for solving a common data engineering challenge in a simple, consistent, and scalable way, designed to reduce rework, speed up delivery, and embed quality by default.

Pattern Description

The Automated Table Unioning pattern automatically combines two or more tables by intelligently looking up column names and data types to generate safe SQL under the covers. It supports disparate data sources, such as those from multiple publishers in a data clean room, and creates a view or incrementally loads the unified data into a physical table while tracking load watermarks to prevent duplicates.

Pattern Context Diagram

Pattern Template

Pattern Name

Automated Table Unioning (or Union Tables Pattern)

The Problem It Solves

You know that moment when you need to combine data from multiple sources into one cohesive view, but the thought of writing complex SQL for unioning gives you a headache? Especially when dealing with tables that have hundreds of columns and potentially differing column names or data types, manually handling the complexity is a huge mission. This pattern solves the problem of needing to "stack together" multiple tables automatically, without requiring users to write "horrible SQL". It streamlines the process of consolidating data from various origins into a single, unified view that end customers can easily consume.

When to Use It

Use this pattern when:

• You need to combine two or more tables to create a single, unified dataset.

• The complexity of managing column names and data types during the unioning process is too high for manual SQL.

• You are receiving data from multiple disparate sources (e.g., different publishers in a data clean room) that needs to be consolidated for a single customer view.

• The data coming from these different sources is "roughly similar" in its underlying structure or intent.

• You want to avoid writing extensive, repetitive, and potentially error-prone SQL for union operations.

• Input tables have a large number of columns (e.g., over a hundred) making manual column mapping or selection impractical.

How It Works

Trigger:

The pattern is initiated when:

• The need to combine multiple tables from various sources into a single, comprehensive table or view.

Inputs:

• Two or more tables that contain related or similar data.

• These tables might originate from different "tenancies" or "projects".

Steps:

1. User Selection: The user selects a "driving table" and then specifies other tables to union with it.

2. Metadata Lookup: Under the covers, the system performs a comprehensive dictionary lookup of the data types and column names across all selected tables. This lookup can extend to tables in other projects if shared tables are used, ensuring the most up-to-date metadata is acquired.

3. Safe SQL Generation: The system then intelligently works out the "safe SQL" required to union the tables, ensuring that column names and data types are correctly matched and aligned.

4. Robust Column Matching: The logic for matching columns is robust, handling inconsistencies like mixed case or reversed column names, making it "a whole lot smarter" than a simplistic string match.

5. SQL Execution & Output: The generated SQL is produced and executed, loading data into downstream "consumption tables".

6. Physicalisation & Incremental Loading: While the first version created a view, the pattern was iterated to "physicalisation," where it creates a new physical table. This table is then incrementally loaded daily, keeping track of "load watermarks" for each source table and safely appending new data without inserting duplicates.

7. Column Exclusion Choice: The user maintains the choice to exclude specific columns that they do not wish to bring through to the final unioned table.

Outputs:

• A single, combined, and incrementally loaded table that stacks data from multiple tables, ready for consumption.

Why It Works

This pattern works like a smart, automated data assembler. Instead of you manually figuring out how to fit different pieces (tables) together, it automatically assesses their shapes (column names) and materials (data types), then precisely connects them. It's akin to having a super-efficient robot manage all the complex plumbing behind the scenes, ensuring data lines up perfectly without any human intervention in the intricate SQL details.

It "just works," providing a "magical" experience by automating a traditionally time-consuming and error-prone task. By automatically handling schema alignment and ensuring safe, duplicate-free incremental loads, it significantly reduces manual effort and improves data consistency, ultimately saving "hours" of development time. The iteration to physicalisation also makes it highly cost-effective for large volumes of data.

Real-World Example

Imagine running a data clean room where data arrives from five different publishers, each in their own private data tenancy. Each publisher's data might be in a separate table, and after cleaning and anonymising sensitive information, you need to combine all these 5 or even 10 tables into a single, unified table for your end customer to report off. Instead of writing a massive, complex SQL query to union these tables—especially if each table has 100 or more columns—the Automated Table Unioning pattern allows you to simply say, "pick this table and union it," and the system handles all the underlying complexity automatically. The result is one easily consumable table for the customer, without the need for manual SQL coding or tedious column mapping

Anti-Patterns or Gotchas

• Unioning Unrelated Data: The primary anti-pattern is using this feature to combine tables that have no inherent relationship or similar data intent. While the pattern will technically run and produce valid SQL, if you "grab three random tables and union the hell out of it," the resulting table will likely be "90% null cells" because the columns won't match, and it won't meet any meaningful reporting or analysis purpose. It will do "the job, but probably not the job that you wanted to do".

• Lack of Human Oversight: Relying purely on automation without a human in the loop to understand the underlying data relationships and intent can lead to meaningless outputs.

Tips for Adoption

• Build When Needed: This pattern was developed out of a direct need when existing methods were too difficult or required automation. Adopt it strategically when a clear problem of complex unioning arises.

• Iterate and Evolve: Be prepared to iterate on the pattern based on performance, cost, and changes in other related patterns. For example, the pattern evolved from creating views to physicalising tables for cost efficiency, and its column matching logic was enhanced to accommodate changes in automated field naming conventions or shared data practices

• Understand Data Relationship: Ensure the data you are about to union is genuinely "similar" and has a logical relationship to meet your reporting intent.

• Leverage Exclusion Choices: Utilise the ability to selectively exclude columns you don't need, which helps manage the width of the final table, especially with inputs having many columns.

Related Patterns

• Data Ops Platform: The Automated Table Unioning pattern is an overarching feature within the broader data operations platform.

• Shared Tiles: The pattern had to be iterated to work with "shared tiles," where data is safely shared across different tenancies or projects.

• Automated Field Naming Convention: Changes to this convention necessitated updates to the unioning pattern's column matching logic.

• Append Pattern: The incremental loading aspect of the union pattern leverages an "append pattern" to safely add new data without inserting duplicates.
  

Press Release Template

Capability Name

Automated Table Unioning

Headline

New Automated Table Unioning Feature Simplifies Data Consolidation for Users, Eliminating Complex SQL

Introduction

We're thrilled to announce the launch of our new Automated Table Unioning capability, a powerful feature that allows you to automatically stack data from two or more tables into a single, unified table. This is designed for anyone who needs to effortlessly stack data from multiple similar tables, such as those from multiple publishers in a data clean room, without having to write intricate or "horrible SQL". It delivers a "magical" experience by making data integration both easy and efficient

Problem

"As a data analyst, I constantly needed to bring together data from different sources that could have over a hundred columns and inconsistent naming conventions. I hated writing pages of really horrible SQL just to union tables, especially when dealing with data from multiple publishers in our data clean room that needed to be stacked together into one single table. It was a huge mission that wasted hours of my time trying to manually map columns

Solution

The Automated Table Unioning feature streamlines this process by allowing you to simply select a "driving table" and then specify other tables to union with it.

Under the covers, the system performs a comprehensive dictionary lookup of data types and column names across all selected tables, automatically generating the "safe SQL" required for the union.

It robustly handles inconsistencies like mixed case or reversed column names, making the matching "a whole lot smarter".

You also have the choice to exclude specific columns you don't wish to bring through to the final unioned table.

This capability also incrementally loads the final output into a physical table, keeping track of "load watermarks" for each source table and safely appending new data without inserting duplicates, ensuring both cost-effectiveness and up-to-date information detected, generally taking care of 99% of common issues without manual intervention.

Data Platform Product Manager

"This capability is a game-changer for our data platform, significantly enhancing maintainability and trust by automating complex data integration tasks that previously required substantial manual effort. It has proven to be incredibly cost-effective at scale and highly adaptable to evolving data patterns and shared data practices across tenancies."

Data Platform User

"This feature is truly magical! I can just pick my tables and say 'union this,' and it just works, saving me hours of tedious SQL writing and column mapping. I get one clean, combined table exactly how I need it, without the headache, allowing me to focus on analysis rather than data preparation”

Get Started

The Automated Table Unioning capability is available today within the AgileData platform. Simply select your desired tables and apply the union rule step to effortlessly stack your data. Talk to your data platform product manager for more details and to access this powerful feature for your data.

AgileData App / Platform Example

Context Set via Change Rule UI

AgileData Sudo Code Generated

GIVEN new or updated records in the consume.adt_products input tile

UNION the consume.dbt_products tile

UNION the consume.xyz_products tile

AND a business effective date of TIMESTAMP(effective_date)

THEN populate the all_products consume tile

USING the replace pattern

AgileData Podcast Episode

https://agiledata.podbean.com/e/orchestrating-dynamic-data-flows-agiledata-engineering-pattern-1-episode-67-1751583778/

Listen to the Podcast Episode

AgileData Podcast Mind Map

AgileData Podcast Episode Transcript

Shane: Welcome to the Agile Data Podcast. I'm Shane Gibson.

Nigel: And I'm Nigel Vining.

Shane: Hey, Nigel. We're onto our second agile data engineering Pattern so today we're gonna talk about union two or more tables together automatically. Or automatically. Take me away with what it is and why I cared.

Nigel: Cool, thanks Shane. This one was topical this week because Snowflake did an announcement around, basically this very feature, unioning tables together and taking care of the complexity of the covers, which I thought was funny because it's something that crossed our paths going back about four years now, we needed the ability to union two tables together, or more, and have the complexity of this taken care of under the covers. Now, the complexity is that when you union two tables together, you need to take into account the column names, and the data types. To basically line all the data up correctly, and usually this is handled in the SQL, but if you have a end user like Shane, working out that SQL's a step too far, it's quite a mission sometimes, considering that some of the tables may have, you know, 100 columns in them and have some different data types.

So we come up with this pattern called basically union tables. and it's a rule. So let's Shane pick a driving table and then effectively pick another table and say union this. Pick another table if he wants and say union this. Under the covers we do a basically a big dictionary lookup of the data types and the columns in those tables.

And then we work out what the safe SQL is to union them together. Basically, make sure the columns match up and make sure the data types match up. And then we produce the SQL and it runs and it does all the normal stuff that loads out downstream consumption tiles. So that's it in a nutshell.

Shane: And, I just, I was just looking at this we actually built it 14th of June, 2022. We released it. So the interesting thing for me is we built that feature way back then because we needed it. It comes back to this, one of the overarching patterns of the data ops platform is build it when you need it.

When something is too hard or needs to be automated, then we build it, we use it. If we don't need it, we don't build it until we need it. So for me, what this allowed me to do was grab multiple tables, tiles in our language and to slam them together. And then I could come up with one view or magically off that and just use it.

And the reason we needed to do that was we were running data clean room. So we'd have data coming in from multiple publishers. Each one going into their own private tenancy. And then once we cleaned all the data and hidden the stuff that we needed to hide from our customer we then needed to slam all those tables together.

So we'd bring in, 2, 5, 10 tables, one from each publisher, and I needed to combine them to be a single table. And I didn't want to write a whole lot of really horrible sequel to do that. So this Pattern effectively allowed me to say, pick this table, and union it.

And then all the complexity is done under the covers.

Nigel: Yeah, exactly right. In the end it turned out to be quite a versatile pattern for that Because yes, we have data arriving from different rooms. As you said. And then effectively you just want to put it all together because the end customer just wants one view across all that data.

So I've already got a consistent format. So. Basically, as long as all the data comes up from the cleanrooms, roughly similar, the union pattern takes care of it and safely puts them together and incrementally loads that table every day because the second part of the pattern, the covers, is it's keeping track of the Effectively the load watermarks for each of those tables in the consume and when new data turns up basically a big append pattern but done in a safe way where we check we never inserting duplicates.

Yeah, it just works. It's it's actually a quite a magical one.

Shane: I was just reading our release notes. We actually iterated it to do that. So the first version of that patent that we did, we actually created a view. Data was stored in the original tables, and we created a view across the top as we got to a large volume of data.

Where BigQuery still ran it really fast, but the cost went up. Then we moved to physicalization, right? We changed it. That actually creates a table at the end and you incrementally load that. So again, we iterated the Pattern as we found something that didn't quite work. Off version one.

Nigel: Yeah, yeah, that's that's exactly right, the cost went up and so we the engineering a little bit and the cost came way down now and now that's quite a cheap pad to run because we only incrementally add a few partitions into that table each day, which is really effective. Okay.

Shane: the second thing we did is as we changed some of our other patterns, we had to iterate that union Pattern. So for example, when we started using shared tiles, so instead of data moving across tenancies, we shared data between tenancies in a safe way. , You had to iterate that Pattern,

And then the other one was we had a automated field naming convention for our consumed tiles. And we decided that actually the readability of those columns, those field names, was not best.

So we flipped it. And actually changed the way we dynamically generated those field names. So again, when we did that, you had to add another set of logic to the union rule so that it could actually pick up both field formatting

Nigel: yes, so there are two things that went on. As you see, the first one was we were unioning in tables from other projects, so we needed to extend the metadata lookup wider into the project that that table was coming from. So we had the most up -to -date list of Column names and data types, so we would pull them straight off the source project and then the second one is.

We changed some of our naming for the tiles going in so which made our the first version was very simplistic match was basically if the end of the string matches then it was the same column name so we made that a lot more robust we rejected that and we basically made sure that the name the names could be a little bit inconsistent with mixed case we Reverse the column names and it would still find the correct match because effectively we made a whole lot smarter.

And that's the version now we've been running probably about 18 months nearly, I believe. And that's still got legs, so we'll just keep running that as is.

Shane: And the other thing you mentioned is some of the input tables we use to union have over a hundred columns because in our consume layer. We effectively use one big table as much as we can. So I'm literally grabbing five tables, each of a hundred columns, saying to a union, it's working out where those columns are the same rack and stacks that data.

And then when the columns are different, it treats that as a separate column. Then I can choose which columns I don't wanna bring through. So I may say don't do these ones because effectively I don't want, a table with 300 columns when I'm racking and stacking it. But again, I get that choice when I use that rule.

So that's really powerful for me. I don't have to actually go through and do a whole lot of colon mapping. The columns are the same. It picks it up. , But I can actually then exclude the ones that I'm like, yeah, I don't need that this time. This one probably saves me hours actually, because I just don't need to care.

\It just does what I want.

So that's the Pattern that we use. When wouldn't you use this Automated Unioning Pattern.?

I think for me one of the ones would be, I know that the data that I'm about to union is similar. Because I know where it comes from. I know what we've done to it. I think if you grab two random tables that had no relationship of the data and try to union them without any. Human being involved to look at it.

So just grab three random tables and union the hell out of it. It'll run, it'll create a consumed table that's actually the union and all of it. But I'm not sure it'll meet the intent of what I would actually need to report off. So I think for me, that's probably an anti-patent, that it just does it without a human being involved in that loop.

Nigel: Yeah, it definitely allows you to do exactly that scenario, put three things that aren't related together, it'll definitely union them, it'll run, it'll produce valid SQL, 90 percent of that produced table will be, No, because they don't match.

Shane: think that's the only one I can think of. The table versus view that was performance. It's not really an, it's just something we need to do to that. And then the name of the columns for helping with the automated matching, they're just sub patterns of the patent itself.

So that's the main one is the ability to use it to slam and rack and stack tables that actually aren't related at all and have had no shared data. It'll run it'll do the job, but probably not the job that you wanted to do. Alright, I think we got that one done. I hope everybody has a simply magical day.

Quicklinks

Description

Context Diagram

Pattern Template

Press Release Template

AgileData App / Platform Example

AgileData Podcast Episode

AgileData Podcast Transcript

Agile Data Engineering Pattern

An AgileData Engineering Pattern is a repeatable, proven approach for solving a common data engineering challenge in a simple, consistent, and scalable way, designed to reduce rework, speed up delivery, and embed quality by default.

Pattern Description

The Dynamic Data Flow Orchestration pattern dynamically generates and self-heals data flow manifests (DAGs) at runtime from a central context repository, enabling adaptive, cost-effective, and low-latency data processing primarily for micro-batching workloads

Pattern Context Diagram

«TBD»

Pattern Template

Pattern Name

Dynamic Data Flow Orchestration

The Problem It Solves

You know that moment when you've updated a data rule or schema, only to find your data pipelines have broken or are out of sync, forcing hours of manual fixes? Or perhaps you're struggling with inconsistent DAG deployments, especially when multiple engineers are making changes, leading to "drift" in what should be running? This pattern addresses the recurring challenge of maintaining data flows that are constantly up-to-date, reliable, and cost-effective, without constant manual intervention or the "sinking problem" between rules and physical DAGs.

When to Use It

Use this pattern when:

• You need data pipelines to self-heal and dynamically adapt to changes in rules, schemas, or dependencies.

• You require your data lineage and manifests to be always up-to-date at runtime.

• You are primarily working with microbatching workloads, where refresh latency is typically 15 minutes or more.

• Your pipelines are often event-driven, starting when new data or files arrive.

• You want to achieve low latency and low operational costs by avoiding persistent orchestration servers.

• Maintaining auditability and preventing invalid changes in your data logic is critical.

How It Works

Trigger:

The pattern is initiated when:

• The first piece of data arrives, triggering a refresh of a data table.

• A scheduled execution occurs (e.g., a daily run at 7:00 AM).

• A new row of data or file appears in a dependent upstream table, if the process is configured for "autosync".

Inputs:

• Data transformation and loading rules, stored within a context database.

• Configuration details, including attributes like "autosync" or "manual sync".

Steps:

1. Define Rules in Context: Data rules and logic for transformations are created and stored in a central "context database".

2. Initial Trigger and Lookup: When data arrives or a schedule triggers, a refresh of the relevant data table is initiated. Once loaded, the system pauses and performs a lookup in the context database to identify all downstream objects dependent on this table.

3. Dynamic Manifest Generation: A manifest (or Directed Acyclic Graph - DAG) is built on-the-fly, outlining all necessary steps and their dependencies. This ensures the manifest always reflects the freshest logic.

4. Parallel Job Execution: All identified jobs in the manifest are "seeded off" simultaneously. Each job is tagged, and they run in parallel.

5. Pub/Sub Communication: As each job completes, it publishes an "I'm done" message via Pub/Sub. The system monitors these messages to track progress.

6. Iterative Orchestration: Once a "driving table" (a key dependency for the next stage) is loaded, another config lookup occurs, generating a new manifest to continue the flow. This repeats until all steps in the manifest are completed.

7. Change Validation and Self-Healing: Any changes to the context layer undergo validation before being deployed to production. If an upstream change (like adding a column) is detected, the system can automatically recreate tables, roll back watermarks, reload data, and revalidate rules downstream, handling 99% of common issues without manual intervention. Alerts are raised for changes that cannot be automatically resolved.

Outputs:

• Always up-to-date and reliable data flows that adapt dynamically to changes.

• Significantly reduced manual intervention for pipeline failures.

• Low latency and cost-effective data processing due to serverless execution and no persistent orchestration server.

• Data catalog and lineage graphs that are inherently accurate and current because they are driven directly from the context layer.

Why It Works

This pattern works because it embraces a context-driven approach. Instead of static, manually managed DAGs that can get out of sync with changing rules, all data logic is stored in a central repository. Data flows are then dynamically generated at runtime from this context, much like a recipe that's always updated with the freshest ingredients, ensuring what's running is always the latest version.

The use of Pub/Sub messaging and serverless infrastructure (like BigQuery and Cloud Run) enables a "fire and forget" execution model. This means there’s no constant, costly orchestration server running; jobs are spun up only when needed, then destroyed, leading to "very little latency" and "low costs".

Built-in validation and "stage gates" ensure that only valid changes are pushed to the production context, acting as a quality control filter that prevents broken pipelines before they even run. Furthermore, its self-healing nature means that common changes, like adding a column, are automatically handled, dramatically reducing manual firefighting and boosting trust in the data. This pattern effectively shifts the burden of dependency management from the data engineer to the automated system.

Real-World Example

Imagine a data engineer adds a new column to a table or renames an existing one. In older systems, this might require manual updates to numerous DAGs, leading to errors and delays. With Dynamic Data Flow Orchestration, the engineer simply updates the rule in the context database. The next time data arrives, or a scheduled execution occurs, the system detects this change via its validation checks. It then automatically recreates the affected table, rolls back its watermark, reloads the data, and revalidates all downstream rules. The data flows seamlessly adapt to the new structure, often handling 99% of such changes without any manual intervention, ensuring the data remains fresh and pipelines keep running smoothly.

Anti-Patterns or Gotchas

• Streaming Data: This pattern is primarily designed for microbatching (data refresh latency of 15 minutes or more). For true real-time, row-by-row streaming, this pattern's natural breakdown into nodes and links could create bottlenecks. A different approach, like submitting the entire end-to-end transformation as a single code stream, or a "two-speed pipeline" (similar to a Lambda Architecture), might be required for lower latency streaming needs.

• Skipping Context Definition: The system relies entirely on the context layer. Trying to create transformation code, tables, or schedules without first defining them in the context will not work, as it forces all work to be based on the latest, validated version.

Tips for Adoption

• Prioritise Context Definition: Ensure all data rules, logic, and dependencies are meticulously captured in the context database, as this is the single source of truth for dynamic flow generation and lineage.

• Leverage Validation: Implement robust validation mechanisms for any changes to the context layer. This ensures that invalid configurations don't reach production, preventing pipeline breaks. This includes integrating practices like peer review, Git review, PR pull requests, and data quality tests before context updates are active.

• Embrace Serverless: Utilise cloud services like Google Cloud's BigQuery, Pub/Sub, Cloud Functions, and Cloud Run to fully benefit from the low-cost, fire-and-forget execution model.

• Focus on Supporting Patterns: While the core orchestration pattern is robust, ensure that supporting patterns (like automated rebuilding, deployment, and destroy models for safe self-healing) are also "bulletproof" to avoid issues.

• Start with Microbatches: Begin by applying this pattern to workloads that fit the microbatching use case (15 minutes + latency) to build familiarity and confidence.

• Scalability Mindset: The pattern offers headroom for scaling (horizontal/vertical scaling of BigQuery instances, chunking data flows) if performance or cost issues arise, so be prepared to iterate.

• Trust the Automation: Allow the system to self-heal. It can handle 99% of common changes automatically, reducing the need for manual intervention.

Related Patterns

• Context Driven Development: This pattern is a direct application of the broader "context driven" principle, where all logic is dynamically generated from a central repository.

• Pub/Sub Messaging: Utilised as the core communication mechanism between processing steps, enabling the fire-and-forget execution and decoupled architecture.

• Layered Data Architecture: Operates within a typical layered data architecture (e.g., History, Design, Consume layers) common in modern data platforms.

• Lambda Architecture: A hybrid approach (streaming + batching) that might be adopted when the anti-pattern of real-time streaming is encountered, augmenting this pattern.

• Data Validation and Quality Gates: Integral to ensuring that changes to the context don't break downstream processes, encompassing peer review, Git PRs, and data quality tests.

• Deploy and Destroy Infrastructure: The underlying cloud infrastructure model (e.g., Cloud Functions, Cloud Run) that allows resources to be provisioned only for the duration of a job, contributing to cost efficiency.
  

Press Release Template

Capability Name

Dynamic Data Flow Orchestration

Headline

New Dynamic Data Flow Orchestration Ensures Self-Healing, Cost-Effective, and Always Up-to-Date Data Pipelines for Data Teams

Introduction

The Data Platform team is thrilled to announce the launch of our new Dynamic Data Flow Orchestration capability. This revolutionary approach automates the management of data pipelines, dynamically adapting to changes in data rules and dependencies. It’s designed for data engineers and platform users, ensuring data flows are always current, reliable, and efficiently processed.

Problem

“As a data engineer, I used to dread making changes to data pipelines. Things would constantly get out of sync, and I’d spend hours manually fixing failures just because a dependency changed or a new column was added. It was hard to trust that what was running was the freshest version of our data flows.”

Solution

Our new Dynamic Data Flow Orchestration leverages a context database to store all data rules and logic. When new data arrives, or a scheduled trigger occurs, the system dynamically generates a manifest on-the-fly, identifying all downstream dependencies and launching jobs in parallel. This means data flows self-heal and adapt automatically to changes like new rules or renamed tables, eliminating manual syncing and reducing failures.

By using Pub/Sub messaging and serverless infrastructure like BigQuery and Cloud Run, pipelines run with very little latency and significantly lower costs, as there’s no constant orchestration server running. Validation mechanisms ensure only valid changes update the production context, preventing breaks. The system will even automatically rebuild and refresh tables downstream if upstream changes are detected, generally taking care of 99% of common issues without manual intervention.

Data Platform Product Manager

“With Dynamic Data Flow Orchestration, we’ve fundamentally improved the maintainability and reliability of our data pipelines. The self-healing nature and always-up-to-date lineage significantly boost trust in our data, allowing our engineers to focus on value creation rather than constant firefighting.”

Data Platform User

“I love that I don’t have to worry if the data I’m using is fresh or if a pipeline has broken behind the scenes. This system just works! I can add a new rule or even rename a table, and the data flows seamlessly adapt without me having to lift a finger.”

Get Started

This capability is active across our data platform today, ensuring reliable and fresh data for all users. To learn more about how Dynamic Data Flow Orchestration ensures always-fresh and robust data, please contact your data platform product manager or visit agiledata.io for documentation.

AgileData Podcast Episode

https://podcast.agiledata.io/e/orchestrating-dynamic-data-flows-agiledata-engineering-pattern-1-episode-67/

Listen to Podcast Epsiode

AgileData Podcast Episode Transcript

Shane: Welcome to the Agile Data Podcast. I'm Shane Gibson.

Nigel: And I'm Nigel Vining.

Shane: Hey, Nigel. Today we're trying a new thing. We are going to start a series where we effectively describe one Pattern in each episode that we use, and I'm thinking of it as pages in our Pattern library. The first one we wanna talk about is the way that we orchestrate the data flows in our way of working.

So why don't you start off. By giving me an overview of the patents that we use to engineer that particular task.

Nigel: So we do a little bit different after many years background, using the likes of Composer and Airflow to deploy dags and create run manifests we thought would be a little bit different and.

Use a context driven, generate the manifest on the fly approach. So effectively what we do is we create our data rules for transforming loading data. We put them into our context database, and then what happens is the first piece of data that turns up effectively triggers a refresh of that. Data table, and then this is where it changes from the usual.

So as soon as that table's loaded, we then pause and we do another lookup of the config, and we basically say, what are the downstream objects that hang off this table? Now that's loaded. From that, we build a manifest on the fly. So we effectively say, great, we need to load this table. Depends on this one.

So we will load it first. So we build that manifest and then we seed off all those jobs. We basically set 'em off all at the same time. We attach a tag to each of them and then we wait for them to finish. They all run in parallel, and as each one finishes, it checks that manifest and goes, am I the last?

Nope. Am I the last? Nope. Until we effectively get the driving table, which would start the next flow. So once that table comes back and says, yep, loaded, we do another config lookup and we start again, we create another manifest and we keep repeating this until we've effectively got nothing left in the manifest.

The. Beauty of this approach is in our early days we did a lot of rebuilds and changing context. Shane would introduce new contexts while jobs were already running. He'd delete tables or he'd recreate 'em on the different names. So the whole time what was happening is we were getting lots of failures until we went to this Pattern of context and manifest on demand, where effectively the whole time it was always checking and saying, oh, new piece of configs turned up.

Got another table to load and I would effectively slot it into the mix and it was all quite seamless and painless for me. So that's our Pattern a nutshell.

Shane: And the reason I like it is because if we think about each of our blobs of transformation code as been a rule, that's the term we use. I. I can go and add a new rule.

So I can say there's a table sitting in history, think raw bronze. I need to do something nasty to the data to bring it through into our design layer. And I can go and add a rule, I can add a small piece of logic that takes that data and moves it into our design layer. And as soon as I do that, the next time there's a scheduled execution or an execution that's triggered for a load, the manifest, the dag directed graph for that flow will dynamically rebuild itself.

And so it just self-heal. And then as you said, when I break my own rules for naming conventions, I can go and rename things. And again, that the data flows self-heal themselves. So effectively I can do the work I wanna do, and I don't need to care about what the directed graph for those data flows are. I don't need to care about the dependencies because the context engine.

This Pattern we use for orchestration of these data flows is doing it for me, right? I just don't care. And importantly, it's doing it for you 'cause you don't care either, right? Is effectively we can make any change we want and the machine just takes care of it. So I think for me, that was the biggest value.

But think about the alternatives. So one of the core patterns we use, which you can talk about in another podcast, is this idea of context driven. So everything we do, every piece of logic is stored in a repository, and everything that needs to execute is hydrated or generated dynamically from that repository.

So what that means is when we make a change, we can dynamically regenerate those data flows versus the old way of doing it, which is we'd hold that context in one place and we'd have a physical instantiation of it as a set of dags, and now I have to sync it. After, like, I wanna change the rules. Okay, now I need to change the dag.

And those things used to always get outta sync. We're kinda like, that's not the way it's meant to work. Oh shit. There's this piece of code stored as a stored process somewhere. As part of that dag. Well, the dags were a stored process and that sinking problem used to be a real problem in the old patterns.

Nigel: Yeah. And even today, I occasionally. Struggle with consistent DAG deployment. If there's multiple people, engineers, developers, making changes in that space, you can effectively get a little bit of drift of what should be running and what's associated with other things. 'cause there's a little bit of change going on as commits are happening, whereas this way we effectively leave it to runtime and it's at runtime.

As an object loads, we look at what's next. So it's always the freshest piece of lineage or manifest that you can have for any object. It's always up to date because it's always doing a query back to our context layer to say, Hey, what's next? And then it starts next, and that's cool until that's finished.

So I like it for that because it takes care of itself.

Shane: Also all of the maps that we generate, so our data catalog, our lineage graphs, everything else is driven off that context versus the previous generation where we would've treated all the code and all the schemas and all that structure and all those schedules as exhaust, we would've sucked it into the catalog to try and render it.

We flipped that model, which is you define the, the things. In context, and then everything else is generated off it. It's always up to date because I cannot do a piece of work without the context being entered. I cannot create a transformation code. I cannot create the idea of a table. I cannot create the idea of a schedule.

I cannot orchestrate dependencies across tables without first creating that piece of context. So again, that. Forces us to make sure that anything I'm looking at is the latest version because there is no choice.

Nigel: Yeah. Yep. And I guess the natural question that others would ask in the profession, what about when a change is made that break something?

And this is an interesting one because effectively when any changes are made, we validate them and we don't allow. Updates to the context layer unless it's a valid update. So every change that's made won't update the latest production context until it's valid, so we effectively stop anything getting into the context layer that's gonna cause a problem.

It stays at a draft. Effectively until it's validated. So when the manifest is generated itself, it's always looking for the active record, and it's fine for this to be draft floating around. It won't touch it until it's valid, and then it flips over to replace the current production record.

Shane: And then there's a bunch of attributes for that context.

So when I define a table or a piece of code, I can define that it's what we call auto sync. That, that it will execute, that manifest that data flow. Whenever a a row turns up in, in one of its dependencies, Salesforce sends us some records. It has that history tile or table. So raw bronze. Then if it's set to auto sync, it will then trigger.

The hydration of that manifest and run it. But if I flip it and say that for that piece of coat or that table, it's manual sink, custom sink, and I say it's gonna only run at 7:00 AM on a Monday, then that's when it triggers the hydration of that manifest and running that coat. So there's a bunch of attributes that are actually really important when you start moving all your data flow stuff back to a context layer.

Nigel: Yeah, absolutely, and that's a good point. You touch on effectively allowing that orchestration to start based on a record turning up or a file turning up or a schedule that's been attached to it that says, run it daily at 7:00 AM which is quite a common Pattern for pipelines, but 99% of our pipelines tend to start when something arrives.

File some data, which effectively triggers the pipeline to start and pull that data all the way through the layers.

Shane: And then the other Pattern that we use a lot in there is this idea of pub sub. As you said, it's a fire and forget. It figures out all the steps. It fires them off into effectively a queue.

It waits for the queue to run. So we, again, as you always do with patterns, you're using other patterns to make this Pattern really efficient and effective.

Nigel: Yeah, so the driver, as I said at the start of, I'm done, are you done? Are we ready to run something? Those are all pup sub messages. So we have a standard pup message, which basically says.

I'm done. And that's attached to everything in the execution layer. So anytime a table's refreshed, we get a, I'm done message off the end of that. And then we basically look for those messages as we, uh, walk through our layers. So that's our, basically, I guess our wrapper Pattern, which holds it all together.

Has something happened. Yep. Cool. Trigger something else. And we just use messages.

Shane: And then what's our average runtime? It's always dependent on we, we use BigQuery under the covers and Google Pub sub, and it depends on how much data on that, from what you are seeing of all the different customers we've got and the volumes of data.

What's an actual execution for a standard data flow way to go from, you know, arriving data and history all the way to data being consumable in that final layer.

Nigel: It is generally quite fast. We have very little latency because there's no overhead and like an airflow example, there's no overhead of starting up, I guess execution pods, Kubernetes pods to run a task to shut down, to start up and run another task.

Because we are effectively just using a message. We can tell BigQuery to run something. And, and parallel as much as possible. So our, our execution times are generally very fast 'cause there's zero overhead. We're basically just telling BigQuery, here's a list of things I want you to do and letting it do it.

Shane: And our costs are low because we're not running a container, Kubernetes and EC2, anything like that permanently to run the orchestration of the scheduling engine. 'cause we're using pub, sub and cloud functions. But moving to cloud run, it's doing all that magic for us. So it turns itself on the code's deployed, it's run, it's then destroyed, and we only get charged for that period of time that it's actually running, rather than have a 24 by seven service sitting there to just schedule and orchestrate our code.

Nigel: Yeah, that's exactly right. So we have no concept of an orchestration server running the background. Uh, we are effectively creating the BigQuery jobs on the fly, let it run them, and waiting for that I've finished message to come at the end of it. So there is never a service that is physically waiting for a job to run and costing us money.

Basically BigQuery is running jobs, and when it finishes, we get notified and we automatically start the next job. So it makes it very economic to run and we can run hundreds of jobs, thousands of messages, but it basically costs us nothing because there's nothing running to orchestrate that. Those are just messages bouncing around, starting jobs, waiting for jobs to finish.

Very effective.

Shane: But there is an anti-patent for this because we only use this for what we call micro batching. We think about anything above 15 minutes of refresh latency. This works for anything below that, we start hitting a problem and it's not a problem of Pattern around this orchestration of data flows.

It's a problem of the dependencies of everything else we've built. So if we think that we have a layered. Data, architecture, history, design, consume. We use tables under the cover in BigQuery, so there's a dependency on who's writing to them. And we know we can orchestrate this stuff very quickly, but if we move to a streaming model.

If we wanted to pick up one row and push it all the way through this, we would actually end up slightly rearchitecting the way we deploy the code to pub sub. We wouldn't actually fire off 10 tasks and wait for them to daisy chain and orchestrate themselves and have dependencies. What we'd probably do is look at the entire piece of code for the end-to-end orchestration and then submit that to run as a single code stream in BigQuery.

End to end is one task, wouldn't we? Because the patent we've got naturally breaks the task down into nodes and links. It says, run these, and when they're done, run these. And then if we keep firing it with volumes of streaming data, we will actually end up having a bottleneck, won't we? So that's an anti-patent.

We would use the same context approach. We would still hydrate and fire, but. We would slightly change the way the Pattern is deployed at runtime to handle that streaming volume.

Nigel: So in that case, we would let data stream constantly into the landing area, which is what we do in some projects. And then effectively, I guess our compromise around that one is then we would micro batch at 15 minute intervals from that stream data and move it through in micro batches currently.

Otherwise, yes, we would architect to stream. Through the layers, and we would probably run a two speed pipeline. In that case, we would micro batch it, say. Maybe 15 minute intervals. But then what we would also do is we would stream around the outside data directly to the consumer layer for the required attributes that need to be consumed in real time.

And then we would catch up the rest of the layers of our slower data. It through the normal pipeline, and then we get the bests

Shane: effectively adopting a lamb architecture, right? Yes. But the, the majority of our patents would just survive for this orchestration of data flows. But we know that we actually have to change or tailor or augment some of them when we hit that anti-patent for the way that we've currently running it and we're currently designing it.

So. Apart from a little bit of tweaking for streaming, is there any reason you wouldn't use this Pattern? Any use cases where you think this dynamic hydration of the data flows and this creating a manifest and this pushing it off to fire and forget and run? Is there any reason that you wouldn't use this Pattern?

Nigel: No. We've seen it run. Day in, day out for coming up five years now. The original architecture and code that was deployed back in the first year is largely exactly the same. We have not changed the core Pattern, so it's run happily across multiple customer use cases for five years. I think that's a fairly good testament to, it's a Pattern that works.

Shane: I think the other thing we can do is we can, if we needed to, we can scale horizontally or scale vertically. If we ever do get to a situation where we need to reduce the latency time of the end-to-end process, we can scale up the BigQuery instances and make them execute faster. We could chunk the. The data flows down into more domain orientated buckets and then figure out how to paralyze the runs of them.

So again, we've still got lots of headroom of how we can iterate this Pattern if we ever strike a cost or a performance or a complexity issue. So we know we're always gonna be iterating it, but like you said, it's been fairly good and bulletproof. I think the other thing is you've spent quite a lot of time on what we call the blast radius.

As you started building out that core Pattern as code and as I started doing the things that I would expect to do as a non-data engineer to data, I should be able to do this. And you suck your teeth going, oh my God, you've built those safety nets around that Pattern. So if you talk about the fact that I can create a piece of context, but it's not actually treated as part of that.

Production data flow is not executable when it runs, until it meets all these checks. We're effectively bringing in the peer review process, the Git review, the PR pool requests, the data quality tests, all of that's happening before any of those things allow the data flow to actually execute as a new version.

Nigel: Yeah, and so the other thing that's. Wrapping, this is in the context layer. We've got a couple of flags that reply to the context. So if something has changed in the context layer and successfully deployed into production state, we effectively put a flag on it that says, this config has changed. It's gonna require changes of downstream objects to flow this change.

For example, you've added a new column into a table. Now you expect that column to flow through and hydrate and all the layers downstream of it. So effectively we tag that object and say, Hey, this has changed. You're gonna need to recreate it and you're gonna need to check all the things. Downstream of it to make sure that they're gonna be okay.

So when we come to run that manifest, we run it as usual, but effectively we hit an object and say, oh, there's a flag on this that says it's changed. So we identify the change. We usually recreate the table and we roll the watermark back. We reload that table. That's cool. We get to the next one and we say.

Great. Something upstream of me has changed. I need to make sure how that's gonna work. So what we actually do is we do a, another validation of that context to make sure it's a hundred percent gonna work with that change. And at that point, if we determine, oops, this is gonna have to be rebuilt as well, we do that and we keep doing this all the way the end of the manifest, but we also stop the manifest.

Some reason we've actually caused something that's not gonna keep working and we stop and we raise an alert and say. Hey, this context, uh, is now gonna be invalidated because something, two or three layers upstream has changed enough that I can't fix it for you automatically. You're gonna have to look at change and then basically say what you need to do.

So that was quite a breakthrough 'cause we used to spend a lot of time. Manually trying to fix pipelines that stopped for small things like comma being added or removed. Whereas 99% of the time we can handle that automatically just by doing a full rebuild and refresh the table and then revalidating the rules.

And generally everything takes care of itself. 'cause we wanted it to be self-healing. 'cause there's nothing worse than pipelines that fall over just because you've added a column. Whereas something like that's easy, a column can just be added, populated, and away you go. So that was quite a breakthrough, basically putting those extra bags in the context.

Shane: Yeah. And effectively they're like stage gates. It's basically every time it's validating, am I still to run the next step? Yep. And then in the beginning, like you said, we had lots of times where it stopped. I would do something and then it would say, oh, actually you're not meant to be able to do that. But as each one of those happened, we'd go in and we'd either fix the way I created that context so I could no longer create it in an.

Invalid way, or we'd automate the rebuilding, the deploy and destroy model that it was trying to rebuild so it could rebuild itself safely. So again, we didn't change the core orchestration Pattern, we changed all the other things around it to make sure that patent was bulletproof and ran. And we didn't get call out at two o'clock in the morning because our data flow had stopped, or at seven o'clock to have a look at it and have a whole lot of catch up work to rebuild it.

So I think those supporting patterns were. As important as this core orchestration of the data flows by them being dynamic and using Manifest publisher.

Nigel: Yeah, that's exactly right. It makes it all quite robust and as I said, generally the pipelines run very accurately without a lot of tension.

Shane: Excellent.

Alright, well I think that one's done. So orchestrating data flows in a dynamic way using manifest in pub sub, right. Another one next week. But for now, I hope everybody has a simply magical day.