Agile Data N’ Info

Subscribe: Apple Podcast | Spotify | Google Podcast | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser | Deezer | Podcast Addict |

You can get in touch with Patrik via LinkedIn or over at https://daana.dev

Tired of vague data requests and endless requirement meetings? The Information Product Canvas helps you get clarity in 30 minutes or less?

Google NotebookLM Mindmap

Google NoteBookLM Briefing

Executive Summary

The Focal framework is a mature, metadata-driven approach to data warehousing and integration that prioritizes non-destructive architecture and business understanding over manual coding. Developed and refined since 1998, Focal employs a specific form of ensemble modeling that decomposes data into three core components: keys, descriptions, and relationships.

The framework’s primary strength lies in its “documentation-first” methodology, where a massive metadata layer (consisting of approximately 80 tables) generates the physical implementation. This ensures that documentation and code remain perfectly synchronized while making the physical layer “cattle”—disposable and easily regenerated. In the context of identity resolution, Focal provides a robust technical pattern using an “Identifier Table” to map multiple, changing business identifiers to a single, immutable identity. As the industry shifts toward AI and Large Language Models (LLMs), the framework’s highly patterned, metadata-rich structure positions it as an efficient alternative to traditional, manual coding practices.

--------------------------------------------------------------------------------

1. The Focal Framework: Philosophy and Automation

The Focal framework is distinguished from other methodologies by its tight coupling of data modeling patterns and a technical automation engine. It is designed to create a “non-destructive architecture” that allows for rapid changes without affecting existing structures.

The Documentation-First Approach

A central tenet of the Focal framework is that the value of data warehousing lies in the understanding of the business, not the code itself.

Metadata over Code: The framework captures metadata models, mappings, and rules into a repository. This metadata then generates the necessary code and holds comprehensive lineage (business, technical, and operational).
Disposable Code: In this paradigm, metadata is treated as a “pet” (highly valued and preserved), while code is treated as “cattle” (disposable and easily replaced).
Synchronization: Because the documentation (metadata) creates the code, the two are never out of sync, solving a common problem in long-term data warehouse maintenance.

Structural Complexity and Learning Curve

Despite its efficiency, Focal has historically faced a low adoption rate outside of specific European regions due to its abstract nature. It is often described as an “engineering dream” that requires a significant shift in mindset from traditional relational modeling.

--------------------------------------------------------------------------------

2. Focal Modeling Mechanics

Focal is a form of ensemble modeling, similar to Data Vault and Anchor Modeling, but with distinct differences in how it handles attributes and normalization.

The Three Pillars of Focal

The model is built on three specific structures to ensure agility and minimize the impact of change:

Key Structure: Manages and controls the unique identifier.
Description Table: Holds the descriptions of data using “typed tables.”
Relationship Structure: Manages associations between keys without the use of traditional foreign keys.

Atomic Context and Named-Value Pairs

Focal utilizes an abstract method for storing attributes:

Typed Tables: Rather than standard business columns, tables use generic columns (e.g., start_timestamp, end_timestamp, value_string, unit_of_measure).
Atomic Context: Data is grouped to answer one “atomic question” per row. For example, a monetary value and its currency are stored on the same row because one cannot be understood without the other.
Human-Centric Querying: The model strives to ensure that all data necessary to answer a basic human question is available on a single physical row, effectively reaching a state similar to fifth normal form.

Comparison of Ensemble Modeling Techniques

--------------------------------------------------------------------------------

3. Identity Resolution and Key Integration

Identity resolution in Focal addresses the challenge of recognizing the same entity across disparate systems where identifiers may overlap or change.

Identity vs. Identifier

The framework makes a critical distinction between two concepts:

Identity: A stable, immutable surrogate key that represents a unique instance of data throughout its entire lifecycle in the analytical system.
Identifier: A business-derived value (e.g., account number, email) used to find an identity. Identifiers are unique within their own data space but may change over time.

The Identifier Table Pattern

Focal uses a technical “Identifier Table” to manage the relationship between multiple identifiers and a single identity.

Key Collision Avoidance: By using source-system prefixes or concatenated strings, the framework prevents different entities with the same business ID from merging.
Multi-Identifier Mapping: The table allows multiple identifiers (e.g., a Social Security Number and a Customer ID) to point to the same surrogate key.
Key Propagation: When a new system introduces a new identifier (e.g., a local ID) alongside a known one (e.g., SSN), the framework automatically propagates the existing identity key to the new identifier.

--------------------------------------------------------------------------------

4. Performance and Cloud Optimization

As data architecture has moved to cloud platforms like Snowflake and BigQuery, the Focal framework has evolved to manage costs associated with columnar storage and large-scale lookups.

Single vs. Multi-Identifier Modes

Cost Efficiency: Performing lookups for surrogate keys on billions of rows is expensive in the cloud. Focal now allows entities to start in “single identifier” mode (using natural or hash keys).
Automated Migration: If an entity eventually requires multiple identifiers, the framework uses metadata-driven scripts to automatically migrate the data from a single key to a surrogate key structure across all related tables and relationships.

Storage and Indexing

Relational Databases: Focal tables are “deep” rather than “wide,” making them highly efficient for indexing due to their consistent, repeatable patterns.
Columnar Databases: The framework’s reliance on unique values and patterns works well with the compression algorithms of modern cloud data warehouses.

--------------------------------------------------------------------------------

5. The Impact of Artificial Intelligence

The emergence of AI and LLMs is viewed as a turning point for metadata-driven frameworks like Focal.

Devaluation of Manual Coding: As AI becomes capable of generating code, the value of manual programming skills decreases. The primary value shifts to the ability to design and understand complex systems and architectures.
Machine-Readable Architecture: Because Focal is entirely pattern-based and metadata-driven with no exceptions, it is exceptionally easy for AI to interpret. An AI can read the metadata layer to understand the data layer and answer business questions without the “cheating” or “bespoke” code often found in manual implementations.

--------------------------------------------------------------------------------

6. Significant Insights and Quotes

“The value of data warehousing... was the understanding of data. The value was not in the code. The code had a secondary value in the architecture.”

“Your coding skills are not that valuable anymore. Your ability to design and understand systems, how a good design and a good architecture is, becomes much more important than being very skilled at programming.”

“The metadata is our pet; the code is cattle... we end up protecting or caring about that metadata... because the code becomes disposable.”

“Focal... can create an extremely non-destructive architecture that is very easy to change and add data... without affecting anything else in your architecture.”

Tired of vague data requests and endless requirement meetings? The Information Product Canvas helps you get clarity in 30 minutes or less?

Transcript

[00:00:00] Shane: Welcome to the Agile Data Podcast. I’m Shane Gibson.

[00:00:04] Patrik: This is Patrik Lager.

[00:00:06] Shane: Hey, Patrick. Thank you for coming on the show. Looking forward to this one today. Today we’re gonna be talking about focal data modeling and identity resolution. But before we rip into that, why don’t you give the audience a bit of background about yourself.

[00:00:19] Patrik: Yeah, I’ve been within the data area for the last 25 years working with data warehousing mostly, but more or less just integration is a lot of work. I worked in SOA projects and other kinds of integration areas as well. I’ve been around and seen a lot of things on the way. I also have been one of the main developer of something that is called the focal framework and the Focal modeling Method. So I’ve been working with that since 2004. Or less. The focal came along 1998 by a man who invented it more or less, but then he and I, plus a lot of other people, worked [00:01:00] a lot with developing it and moving it forward, so to speak. So that’s me.

[00:01:04] Shane: focals been around for a long time, but it’s not well known. In some European countries is relatively well known, but in the rest of the world it’s not. Why do you think that is?

[00:01:16] Patrik: The focal framework has a Achilles heel, if you want to call it. It’s an extremely abstract way of using data, but what it makes, happens, it’s, it can create an extremely non-destructive architecture that is very easy to change and add data and change data without affecting anything else in your architecture. Which makes it very agile and very efficient when building things. But as I said it’s an engineering dream, so to speak, has been a lot of smart people working with it. So it’s not easy to take in to learn. And that has been, its Achilles heel, so to speak.

[00:01:57] Shane: And when you talk about focal being a [00:02:00] framework, can you just describe what you mean by that? Because a lot of times we see data modeling patterns that are described as frameworks or methodologies and really what they are as a good way of structuring data and not really much around it. Or sometimes we see methodologies that are agnostic around the data modeling patterns that you can use.

[00:02:19] Whereas from what I understand, focal kind of binds those two things together closely. The framework and the modeling patterns are coexistent is that right?

[00:02:28] Patrik: Yeah. The, the focal framework as it all started with the focal modeling technique, but then it, it became a focal framework was more for documenting. Really what we ran into in all of these projects we work was the documentation always got outta line from what we implemented.

[00:02:47] It doesn’t connect it after a while. So the idea was that we would capture all the metadata models mappings rules and stuff like that into a framework that then would generate the [00:03:00] code for us and generate the things and hold lineage and everything in it. So that became the framework, so to speak.

[00:03:07] You could call it a automation engine, but it’s much larger because it also works as a. A vocabulary and description models holds all kinds of lineage, business lineage technical lineage, operational lineage. It’s all over the place, so to speak. The metadata model is pretty huge, is like 80 tables containing all kinds of metadata around what you’re doing.

[00:03:30] So that’s the framework, so to speak.

[00:03:33] Shane: so if I play that back it’s a technical framework on how you can implement and automate the focal patterns. So it doesn’t take care of people and process, it doesn’t to talk about standups or the way teams are structured. So it’s more in that how you build a technical framework to implement and execute focal without having to handcraft code on a regular basis to make it work.

[00:03:55] Is that right?

[00:03:56] Patrik: Yeah, exactly. The thing was that we, at one point [00:04:00] we came to the conclusion that. The value of data warehousing or the value is the understanding of data. The value was not in the code. The code had was a secondary value in the architecture. So we wanted to make the focus on understanding what you’re doing, understanding data, understanding the business, so that was the focus, and then could use that knowledge to generate the code instead of writing it, and then also always be in sync.

[00:04:29] The documentation would always be in sync with the code because the documentation creates the code.

[00:04:35] Shane: And there’s been a few of us that have been working on this idea of metadata driven data tooling or, active metadata or config driven. Now the new buzzword is context driven. I always saw three patterns in the world. There were the, let me just write a blob code. The blob code executes, it’s it’s on its own.

[00:04:57] It has no relationship with anything else. It’s pretty much gonna do with the [00:05:00] heavy dating work from the left hand side, the raw data history all the way through to consume. And then the next one is tooling that allows you to write blobs a code, but then produces the documentation after the fact.

[00:05:13] WhereScape was probably old enough to remember that that was one where, a lot of the value was when you generated your code, it created the documentation for you. And if I look at people using DBT, a lot of it was around that documentation, that lineage that you got outta it.

[00:05:26] And then there’s a smaller group like us where we said, create the documentation first. And then let the documentation effectively generate the code, So we define the metadata, the context, and then that code will get hydrated based on that core. And we end up protecting or caring about that metadata that config because the code becomes disposable.

[00:05:48] The metadata is our pet the code is cattle and DevOps language. And we see tooling come outta that metadata driven Pattern over the last 30 years, [00:06:00] but it never really gets traction. Why do you think that is? Why do, and we still keep doing it because we know it’s much faster for us.

[00:06:06] But why do you think that this middle ground where people can write code and then document it after the fact always seems to be more popular than defining the metadata, the logic, and letting it hydrate the code for us.

[00:06:19] Patrik: The love of the code. That’s the thing. everyone who is a developer loves their code. They are proud of it. They want to do it. They want to show how good they are at it. And then they come a tooling, a framework that you do not need to be good at writing code. You need to be good to understanding the business and describe the business and describe models and create things like that. And it’s hard to get organizations to accept that because you get often a a lot of fight back from the development side that says you can’t write as good code as me. And I’m better at that. And then always it comes. One of the biggest thing is often that they say, but can you do this? And [00:07:00] you say, no, you can’t do that. Okay, then it’s useless. Even if you can do 90% of the organization because you can’t do the 10%, it’s useless. So they. Drive that line up in again to their own management side. And then it’s, it becomes hard to go, to come into companies doing that. They probably feel threatened as well.

[00:07:20] The interesting point is now with ai, there’s no way out anymore. This is what’s gonna happen. Your coding skills are not that valuable anymore. Your ability to design and understand systems, how a good design and a good architecture is becomes much more important than being very skilled at programming.

[00:07:42] Yeah.

[00:07:42] Shane: Also writing code by hand takes time.

[00:07:45] It’s busy work. So you can be busy writing that code and you’re not having to engage with the stakeholder side to understand the value, deliver that value incrementally. It, it’s not unusual for a data team to be able to do busy [00:08:00] work for a month and actually deliver nothing to a stakeholder.

[00:08:03] We, we somehow have this expectation, and I’m with you, I think the LMS and Gen AI are gonna reduce their expectation timeframe that a month, doing busy work is is not gonna happen. Vibe code and LLM or use a metadata config driven framework pick your choice. But I’d rather leverage an architecture that somebody else has created for me that I know works than let the LLM guess at an architecture that hasn’t been proven before.

[00:08:29] Okay. So framework is the technical implementation of fo. Focal started out in 1998, I think you said, as a data modeling paradigm. Do you wanna talk me through the focal modeling just for people that are more familiar with other modeling techniques and how it’s the same and how it’s different?

[00:08:46] Patrik: Yeah, the focal has been from the first point really method data driven. But it came up with the thought of three parts within the model where you controlled the key and that was its own structure. [00:09:00] And then you had a table that represented the description of data, and then you have their own structures for relationships.

[00:09:09] So you never had foreign keys between tables. And this was not my idea, this was the one who invented it. That this is how it should work. Because he had seen that complexity growth and that changes always happened. And when changes affects the model and when it affects the code, it becomes much slower to change.

[00:09:32] And a data warehouse after five or 10 years becomes very complex. So he was trying to find a way to do, to lessen that effect. And the thing is, one of the things that is making it abstract from a point of view is that the tables that are holding the descriptions are what we call typed tables. You don’t have normal business attributes, columns in the tables. They have generic columns like [00:10:00] start timestamp, end timestamp value string, the numeric string unit to measure. And then you have a key or a identifier of the information that lies on the row. So it’s what we call an tight key or an atomic context key.

[00:10:18] But what really says is it’s like a name value pair, if you know what that is. But it’s the inventor, he said it’s modeled name, value, pair, because it’s not one value and one key. It is one key. And you can have up to five different attributes if those attributes belongs together. So this is one of the, one of the problems with Focal, everyone you talk about it is it has a lot of underlying philosophical ideas and theories that doesn’t match how we do normal data models. So it’s always an abstract step to take in your mind, but why do you do that and how does it work? But from a, from, if [00:11:00] you want to simplify it very much, you would say that the tables that hold the description are named value pairs. You could say. So you have a ID or a key or what you normally that name and that says what is in the value string. And then you can add new attributes. They are just new names and they hold their own values. So you never change the table, you just add new types of data into that. So that is one of the biggest differences with normal data modeling really. So it’s it’s built for agility. More or less change, do non-destructive change.

[00:11:37] You can add attributes, remove attributes, and you never change the code. You don’t change the tables, you just do the change, so to speak.

[00:11:45] Shane: So let me play that one back. So you talked about first object is a key. Second object is a table which describes the values effectively and the key, and there’s some unique technical architecture around that, or modeling architecture compared to most [00:12:00] patterns. And then the third object is relationships,

[00:12:02] the keys that have a relationship at a point in time. my understanding is that is a form of ensemble modeling, You are decomposing the key from the descriptors of the key and you are decomposing the relationships and holding those separately. So that’s true, right? Focal is a form of ensemble modeling.

[00:12:20] Patrik: Yes that it is.

[00:12:22] Shane: Excellent. And then there’s two other ones that I know of, There’s data vault as one form and anchor modeling as another. So what I might do is just actually we’ve never had anybody on to talk about anchor. I need to do that but let me try and just play back what you told me around focal and how I think it’s different to data vault and anchor modeling.

[00:12:40] And I’ll get the anchor one wrong ‘cause like I said, I haven’t had anybody on to explain it properly. So the first thing is both anchor and data vault hold the key in a separate object like focal does. So that’s the same, correct.

[00:12:53] Patrik: Yes,

[00:12:53] Shane: Okay. And then in data vault we’ll hold one or many satellites that hold the descriptors or the [00:13:00] attributes against those keys.

[00:13:02] But we will hold a table that has, or a view that has many columns. So if I have person’s name and person’s age or date of birth, I will hold those as columns in the satellite and anchor. My understanding is I would hold each of those attributes as a separate table.

[00:13:20] Patrik: yes, that’s

[00:13:22] Shane: And what you are saying is, in focal, what we do is we use a named peer key.

[00:13:26] So we’re making those columns rows. However, there’s an extra little bit of complexity or value in there that you can hold multiple values against one pair. So I could decide to have name and date of birth. And what sounds like a Jason string is it’s a way of. Typing an object and having a subset of values against it.

[00:13:50] Is that right?

[00:13:51] Patrik: No the underlying theory, you can do that, but that’s not why it was designed for it. It is designed for what we call an atomic context. And the atomic [00:14:00] context is, so the key but we hold is we call it the atomic context. Key is a group of attributes, one, two, many, but five more or less is what we have in the tables, and it is supposed to answer one atomic question. So if you put a shoe number and first name, you don’t have an atomic question for that, right? Because you want to ask what’s your first name? What’s your shoe number? That’s two different questions. So they would be two different atomic context. But on the other hand, if I would say, what is the loan amount I have on my loan that I, my sign, my original loan amount, I might say it is 1000 euro.

[00:14:43] Because you can’t answer the question of a monetary value without its currency. So you need to have the value and the currency on the same row. Therefore, we have, as an example, we have a unit measure that we call unit to measure column that you say then, so you use the [00:15:00] Val Nu column and you say That’s 1000.

[00:15:03] And then in the unit measure it says Euro. So then you answer the atomic question. And as another example, more complex one is like a balance. That you have a balance. And to understand the balance, which is a point in time data, is only true at a certain point in time. You need the balance date. When was this balance value true? And what was the unit to measure of the balance? So obviously sudden you have three attributes within the same atomic context to be able to answer the atomic query. so this is more like from a human perspective think how does human answer ask questions? you can ask who are you?

[00:15:39] And you will get a lot of things. But if you ask an atomic question, the idea is that all the data should be on the same row. To answer that in the physical table.

[00:15:48] Shane: So let’s go back to that example then. So if I’m holding balance on my bank account, then I need to know what the point in time, the date time of that balance is. And I probably also [00:16:00] need to know what currency it is. So what you are saying is you hold those attributes together because actually they need to be together to answer that business question or to provide the context of one of those numbers.

[00:16:13] And therefore, rather than holding it outside of the physical structure in some form of, physical data model where I know I’ve gotta go join that with that, or query that row and grab these three columns. You’re actually saying by bounding them together in a row, the answer’s already there, You have to grab them all because we went to the effort of putting them there because that’s the information that you need to actually answer a question around what is my balance.

[00:16:40] Patrik: Yeah, so you can almost see it like a normalization of data. Because if you would have a, if you had a satellite in data vault, you could have , these three columns that we talked about, the date the value, and the currency. And then you could have what kind of classification balance is this?

[00:16:56] What is, so you just added up. But as we do [00:17:00] that, we break it out. So these, this answer one question, and these columns also these questions and break it out in rows then and these on, in, in in anchor. They go all the way. They do not care about how data from a human perspective is represented.

[00:17:17] They just break it out totally mechanically. One attribute is one table. So you have the value in one table and the currency, another table, and the date in another table. And then they generate views that joins everything together. so it’s three different approaches. Sometimes when I talk to last Runback, who is the inventor of Anchor, we talk about it, he says, anchor is in six normal form. And he, and not from a truly formalized perspective, but he says it feels like focal is fifth normal form somewhere around there. And then we have data was very small, third normal form.

[00:17:53] So it’s, you could see it like that. It’s more normalization is.

[00:17:57] Shane: And it would be interesting when you start putting [00:18:00] LLMs across it, because with data vault we’re gonna have to provide the context of which columns, which attributes in the satellite actually have value given the business question and anchor, we’re gonna have to say which tables right? Effectively need to be put together to answer that question.

[00:18:16] So that context is held outside the data. But in focal, what I think I’m hearing is actually the data has context as well. So if I give it, that name peer key and it can see that there is a balance, a currency and a date time. It’s gonna be pretty good at saying those three things are important to ask the business question of what was my balance, in New Zealand dollars.

[00:18:38] Okay. I can see doing it. One of the things that people often say against any of the ensemble modelings is, oh, storage is too expensive, it’s too hard. And so I can see that the estate currency, we’re gonna store currency again and again, right Where it’s bound to the context of one of the other attributes.

[00:18:57] In my view, storage is cheap now. And [00:19:00] actually with coer databases actually querying across tables as more expensive than querying within a table. But what have you found that’s one of the pushbacks with focal is data storage and duplication is bad. ‘cause it was 30 years ago when we had mainframes.

[00:19:15] Patrik: No never run into the issue of of storage. Really. And more, when we move really far back in the early 2000 when we had the normal relational databases the easy thing with the focal was that it was so easy to do, set up good indexes because the tables looks exactly the same all the time.

[00:19:34] And the, you could say the, the logic for True Delta always looks the same and always how you query it looks the same. It’s always the same. So, it was very easy to set up good indexes. So you even then the tables became very deep. It still worked very well because the indexes was so well defined and the Pattern was reusable all the time.

[00:19:57] So it just worked, so to [00:20:00] speak. What I’ve seen on the other side is that when we moved up into the cloud area where indexes is not so important anymore or important, but they aren’t, doesn’t exist, so to speak, true indexes but they are column narrative, right? Often snowflake and if you look at BigQuery, so on and so forth it’s really efficient because it doesn’t need to store everything, right?

[00:20:24] It’s just store one unique value. It is a really good what do you call it when you press data together? yeah. so focal works very well with those formulas for doing that. So it becomes very small in storage instead because it doesn’t need to store each and every value in the engine, so to speak.

[00:20:46] Shane: And then from what you’ve said the focal framework and the modeling Pattern has basically described the types that sit in those name peer keys. So there’s a dictionary or a shopping list of types [00:21:00] that, that you need because they’ve been proven over the last, what is it, 20, 30 years.

[00:21:05] So you’ve probably gone through and figured out all the types that are needed, pretty much in, in most examples. And you get those kind of free when you are using the methodology or the patterns.

[00:21:14] Patrik: Yeah. You always have to keep in mind that the focal today is the data layer, so to speak, where the data is size is very crucial. See, machine driven is not for the human eye. It’s been very specialized to be fully met data oriented. So there you can’t really look at look at the table in, in the focal data model and understand what kind of data is there, you need to add them.

[00:21:41] Then metadata layer or knowledge layer upon it, that is directly related then. But but everything describes what is there, how it is stored, and everything is up in that layer. So that layer is the, the heart of the solution, so to speak. The physical layer is more just [00:22:00] how do we make this reusable?

[00:22:01] How do we not care about what the human think, care about good engineering sort of principles? And that has been very effective. And now when AI comes along, it’s all of a sudden it’s shit. the machine understands it without a problem, but a human thought was hard for a machine.

[00:22:18] It seems like the AI goes like just, okay that’s not hard.

[00:22:22] Shane: Because it’s patent based, right?

[00:22:24] Patrik: Data, everything is described. So he just reads the metadata layer and then he understands what’s in the data layer. And then you can just ask him questions

[00:22:33] Shane: and there’s no exceptions

[00:22:34] because it’s meta data driven. . There’s no, oh Bob came and did that one and he cheated and Yeah. That’s an exception that you’ve gotta deal with. .

[00:22:41] Patrik: No. So that’s, yeah, it’s fully patterned. So it’s, it is, as you say, there is no exceptions that it has to take into consideration

[00:22:48] Shane: So one of the negatives that you often hear around modeling patterns like data vault is consuming, the data is hard.

[00:22:56] There’s a lot more tables, a lot more joins, even though it’s pattern [00:23:00] based, people that try and query it directly have a little bit more work. And what we always say is your consume layer should be automatically generated,

[00:23:07] it’s a pattern based design pattern. So creating the consumption patterns can be automated and you can use one big table with. Dimensional modeling or whatever, anything that flavor that you prefer to query the data. But those patterns for querying, it should be automated.

[00:23:23] I’m gonna assume that for focal, it’s the same that you wouldn’t expect a human with Tableau to go into the core focal data model and try and figure out how to query it.

[00:23:32] Patrik: No, it’s certainly not. No. It’s the focal framework itself creates, automatically creates views more or less like anchor does. So it you as a human never reads the physical tables. You go through the views, which will pivot the data into this is the customer view.

[00:23:50] This is a customer table. It looks like a normal table. When you look at it with all the column names and stuff like that underneath, it’s a totally different thing. But for a human, you as [00:24:00] go in and. Access the views and it looks like a normal table. So that’s how we have abstracted the complexity.

[00:24:07] So you don’t work against the physical the machine does that. You just read the views and you don’t have to care how complex or how strange or abstract it’s underneath really. So that’s more or less how we work with it. You don’t go in and hand code a dimension from the physical tables?

[00:24:25] No.

[00:24:25] Shane: And that’s part of the framework. When you talk about your metadata repository, having 80 tables, that’s where you’re gonna define these keys. These name peer keys, the relationships, and then it’s gonna generate the physical structure under the covers using the focal modeling technique. And then it’s gonna also generate the views that you need to make that data useful

[00:24:47] and usable when you query it.

[00:24:49] Yeah.

[00:24:49] Patrik: Yeah.

[00:24:50] Shane: And that’s the value of the framework over and above the modeling pattern, correct.

[00:24:55] Patrik: Yeah. I think I, I agree that most of them, both from anchor and data [00:25:00] vault and focal and ensemble models, it is harder to write queries against it. You can’t really dodge that’s how it is, but there is a reason for it, and that’s what people forget. So you can’t say it’s just as easy as a dimensional model.

[00:25:16] No, it isn’t. But there is a reason why the ensemble models looks and works like they do, which from an architectural perspective and longevity of the system and changeability and stuff like that. these layering ideas or have layers, is that you have a layer which is specialized on a. Specific area and use the modeling technique that support what you need to do in that layer. Have another layer that makes it easier to do other things, but don’t suboptimize the layer by trying to do everything in one layer. you have to have sort of a system design mind to understand when and where to use things.

[00:25:56] Shane: Yeah, that idea of a layered data architecture and use the modeling techniques [00:26:00] that have the most value for the problem you’re trying to solve. And then automate them and standardize them so they’re part of your architecture. Not little things bolted on the end that, people forget that for this one we somehow use something that was slightly different.

[00:26:13] Last question around that though. The focal framework itself, it’s open source, correct.

[00:26:17] Patrik: at this point there is a focal CLI that we are about to release it. It’s a lot of testings that goes round. The focal framework itself is not in GitHub at this point because more or less, it’s just so hard to understand it. So we need to abstract it, give you tools to use it, so to speak, so you don’t have to be a focal expert to use it, because that just takes too much time, more or less

[00:26:45] Shane: So the other thing we were gonna talk about is how Focal helps us with identity resolution. So this idea that, you know, one of the largest problems always is we see the same person in multiple different places and [00:27:00] systems and we want to stitch them together or the same organization.

[00:27:03] And that is a big problem that most people ignore, right? They focus on the simple problems when they bring out new patterns, frameworks, methodologies. They do the nice, customer orders, product simple ones. So let’s deep dive into a identity resolution and how Focal helps us with that.

[00:27:21] Patrik: Okay. Let’s let me set some concepts first around it, on why we’re doing it just for the audience. So when I’m talking about entity resolution or key integrations, I also used to call it is that we have two things. Two concepts, and that’s one that you have something is called an identity, and that is something that is stable, immutable key that represents an instance of data. And that key should be it doesn’t have to be, but it should be unique throughout the whole analytical system. And it’s [00:28:00] identifies a thing on event unique over its whole life cycle. So it’s immutable. So that’s an identity when I call it identity. The other one is to call an identifier, which is a way of identifying a thing in event it has to be unique within its data space. Might not identify the instance as its full lifecycle, but we should strive to find one. So these are two different concepts. The identity often is called surrogate keys is normally represented as a surrogate key in your system. And the identifier is what people sometimes call business identifier or something like that. But it’s good to understand that you should keep those two apart. That there are two different things and that they are there for, to handling things differently. The other thing is what are we trying to do and what do we not want to happen when we do the sort of integration?

[00:28:53] Key integration is that one thing we don’t want to happen is key collision. The idea is that we have two [00:29:00] different systems both holds accounts and they have an account number series on them and we send them into our account focal or hub or whatever. And we use the account number as the identification because it’s the business identifier, but for some reason these numbers are overlapping.

[00:29:17] So we can start getting key collision. That means that. Two different accounts that are two different instances in the real world, start writing their data on each other and changing data back and forth because they are not truly the same account. That’s a key condition area that we want to avoid. The other thing is that we want to achieve key integration. And and that is the thing that we want to, if they are the same thing in two different system, we want them to integrate on the identifier or the key, so to speak. And the, this is the hard part when working with this is to understand. [00:30:00] How unique should I do something and make something and still make it integrate? So let’s say you could have this account thing and you go okay, but I set source system ID in front of it, and then the account numbers will be unique. Yeah. And then you have two different customer systems and the same customer lives in both system, but you just go, wow.

[00:30:21] But we set the same system ID in front of the customer ID, and then it’s unique, but then we all of a sudden don’t get the key integration that we want to happen because we have made it too unique for how it works in the it. So this work, as you said, this work with integration and entity resolution is. It’s really hard work. It’s not something that just comes along freely is you have to put time and effort to understand the IT landscape, the business processes. How does data move? Where is the unique instances of things and where do they live and how do they move in the IT landscape [00:31:00] to be able to understand how should I build my identifier, because this is what we talk about, so we need to create an identifier. Some people talk about business identifiers. for me, that doesn’t ring correctly because I talk about I normally want to say integration identifier because for me, business identifier is something that the business can use to identify things, but it doesn’t have to be unique for their work, in their daily work.

[00:31:28] So you could have something that is true in one point in time, that it’s unique, but it can change. And it’s still, the business uses it because it works from an operational perspective. But from a analytical system where we historize things and keep track over time how things changes, we need identifiers that stay true throughout the life cycle as much as possible. So there is always that kind of work we need to do to understand how [00:32:00] do we create the identifiers so they will truly integrate and not end up with a key coalition. So it’s hard work, more or less

[00:32:10] Shane: There’s, there was a lot in there. So let me unpack it and try and map it to some patterns that I understand, and I’m probably gonna get them wrong. So I’ll pause after each one to make sure that I’ve got it. So the idea of key collision, so let’s use let’s use accounts that, in system one, we have an account, which the account sequencing numbering is 1, 2, 3, 4,

[00:32:32] so it’s a number. And then in system two, we have a separate series where it’s A, B, C, and based on that, if both of those are held true forever, we know there is no key collision, we can trust that as a unique. Account record, and we’d never collide. But if system one is 1, 2, 3, 4, and system two is 1, 2, 3, 4, and they’re not integrated anywhere else we know that potentially we’ll see that 1, 2, 3, 4 come through from each system, but for a different [00:33:00] account, account for Bob versus account for Shane or, account for savings versus an account for investment.

[00:33:07] And so our natural way of dealing with that is to put source identifiers at the front. Yeah. So system 1, 1, 2, 3, 4, system 2, 1, 2, 3, 4, and now we know we will never get a collision. So that’s the first Pattern that we typically do is that correct?

[00:33:22] Patrik: Yes.

[00:33:22] Shane: And that source and account combination, is that what you call an identifier?

[00:33:30] Patrik: Yeah, it could be, but it could be multiple. You could use multiple columns to create an identifier. So it’s depending on your physical realization of how you want to do it, you could keep the columns apart, but in the focal we, at least we concatenate all the different columns we need to create a unique string of data.

[00:33:48] Shane: Yeah, but that’s the identifier because we can identify that was true at that time. It’s a fact. We can see the data there. We didn’t make it up. We didn’t infer it. It’s a fact. Okay. And then we [00:34:00] know that if we want to say, this account system one is the same as this account system two.

[00:34:06] And now we don’t have an identifier that is exactly the same identifier. We have to infer that they are the same. And it’s like a surrogate key in my head. We’re trying to find this unique identity. I’m just trying to get my language right, the unique identity to say that actually is the one thing that looks differently elsewhere and it’s the same.

[00:34:26] Is that right? So we’re creating this unique identity ID that then everything’s bound to if we can match it. Is that right?

[00:34:34] Patrik: Yeah, I mean it’s the same for data vault people that you have the business key and then you have the Sid, right? It’s the same principle. You have an identification identifier, which is related to a surrogate key. So that’s just to keep them apart, so to speak. That’s what they have two different sort of workings in the system.

[00:34:54] Shane: And so obviously creating that identity ID, because that becomes this golden [00:35:00] id, It’s this immutable thing that we bind everything to when we find something that’s related in the future. And we can see changes over time, so if my account number goes from 1, 2, 3, 4 to 5, 6, 7, 8, 9 a year later and we bind that back to the identity Id, you still know it was my account.

[00:35:18] it’s identifier changed and sometimes happens, especially when we’re using email, When we use email addresses the identifier or name, we know that those identifier IDs will change over time and they cause us a major problem. So what we’re trying to do is find that, that golden identity ID for the thing the event, the person, people place or those kind of concepts who does what’s all those kind of concepts that we wanna manage.

[00:35:42] Is that right?

[00:35:43] Patrik: Yeah. Yeah.

[00:35:44] Shane: Okay. Lots of people talk about that and then they leave it to you to figure out how to do it. So take me through the next bit.

[00:35:52] Patrik: Yeah. So since we are talking about patterns let’s talk about how, first, how data vault does it and Anchor does it just to [00:36:00] make it see, and then we go down to how focal does it. So in the data vault, we already touched on it, right? You, we have the hub, and in the hub we have a business key and we have a c and a surrogate key.

[00:36:11] Now I know a lot of people have discontinued use of surrogate keys in, in the data vault area because they’re using hash keys. And there is a reason for that, and I agree that data vault should do that because they do not have the ability to relate multiple identifiers to one identity because it’s hardwired in the hub. So if it is a one-to-one relationship between the one identifier and it’s key. There is no really architectural functionality for the surrogate key. You can just as well just hash the identifier and then you have a key that always works through. Instead, when the reality hits us in the face, we all of a sudden understand that, oh, this one can be identified multiple different ways. So you could have [00:37:00] a customer ID and you have a social security number, and both are used to be identifying you in different systems and in data vault. if you use the customer ID and it has its surity or you just hash it, that becomes one instance of the customer. You do the social security number, it becomes another instance of the customer. Or then they use a same as link that says This ID is the same as this. ID from an instance perspective. So that’s how they Do it in their solution. In anchor, they point at an attribute table. So you have, in the anchor, you only have the surrogate key, and then they point at the table and one of the attribute tables, and you say, this one is also used as an identifier.

[00:37:44] So their code uses that for lookups. Okay, I, I have my identifier, I check there and then I can pick up from the anchor, I can pick up the surrogate key, I get the surrogate key, and then you have the pickup and they have the ability that they can choose, okay we want [00:38:00] another attribute as well as part we could be able to identify with social security number as well.

[00:38:06] And then they can point in the table that also social security number. The issue is that they have to generate the code or change the code because it has no generic solution around it. So it’s. The code is changing. It’s is a destructive change in the architecture. When you need to add a new way of identifying things, then you get a destructive change in your code.

[00:38:28] So you need to change your code. In Foco, we have what we call an idea for a table, which is to identify a table, which sole purpose is to integrate, to be used, to create integration and hold the relationship between the identifier and the key that are created. And how this works is no magic in the focal framework.

[00:38:53] As of today, there is no fuzzy logic solution or things like that. It’s only [00:39:00] purpose and only functionality is that you can, connect multiple identifiers to one key if you have them in the same table. More or less. So let’s say you have a table where you have your customer ID in the table, and you also have the social security number in the table, probably from the source, right? You can then point out and say, the customer Id identifies the customer, and the social security number identifies the customer. How do we know it’s true? They are on the same row in the table, right? So we know by how the built that they belong together. There’s no magic really.

[00:39:36] It’s just we can just point out multiple things and those different columns that we use to identify becomes their own identifier stream. Now, in the identifier table, there is, I’m not gonna talk about the whole, because there’s a lot of technical things in it as well, but from a conceptual point of view. You have one column that is [00:40:00] called identifier, and you have one column that is called key. And the thing is that each row represents the relationship between an identifier and a key. Now in the code, if you point out social security number and you point out the customer ID as identifying the identifier code will then put both of those into the same SQL, and then he will check for each of those. He will do multiple lookups, one for each. We will look up with the string for the customer ready, we’ll check against the identifier table. Do I have a key for this, yes or no? Okay. And then he checks for the social secured number. Do I have a key or not? In my identifier table, if there is no key, new key is created. Both of these becomes pivotal to its own roles. So the social security number becomes related to customer [00:41:00] key one. the identity, the customer id, key one, and the social security number will also be aligned to the customer key one, so to speak, this surrogate key. So now they are aligned. The, one of the interesting point is, let’s say we also have something else I don’t come up with a good example, but some kind of other identification of a customer, maybe in the same table, but could also be in another table. So in another table, in another system, we have the, let’s say the social security number, and they have a local identification of the customer as well in another color.

[00:41:36] And they want, they need that because they have other systems that. Works with that identifier in the ILE landscape and not the social security number. so the thing is that when they map that system and say it’s a social security number and this local ID that identifies a customer from this source, when that goes in and checks the data in the identifier, he will check the [00:42:00] social security number and he will pick up a key number one, right? And then he will go with a local and he will find no key. But since he already found the key, he will propagate that key to the local ID and that will become a new row in the identifier table. The row that represents the social security numbers, relationship to the key is not multiplied, it’s just exists only one instance independently.

[00:42:26] How many system that uses it. So it just becomes one instance, one row per. Unique string, so to speak. So all of a sudden then you have three different identifiers related to the same key. Now the fascinating thing about this is that since the table is just identifier and key, and when you instantiate this Pattern, you do it for the concept.

[00:42:51] So you have customer identifier and then the table will call it customer identifier. And in the column there will be a customer, [00:43:00] IDFR column and customer key. And do you do it for product? Same thing. Product, ident. so from a patent base there is identifier, key idea, four key, that’s it. And then you just add what kind of entity you want to use this identification for. And the code is exactly the same for independently or which entity. You can also say independently of what industry you do it for, it doesn’t change. It is the same. So the table and the code works the same way independently or who you want to do identification on. So this Pattern is for me, it’s not really a focal Pattern. It is part of the focal solution. But from my point of view, you can use this for any modeling technique really. If you want to be able to create a surrogate key, you just create this look up table, so to speak. And you can have it for dimension modeling. You can have it for data vault if you dare to break the hub thinking. [00:44:00] And you can do it, you could really do it for anchor as well or yeah, whatever modeling method you want, as long as you, you like some sort of, I need to be able to have multiple ways of identifying things. But I don’t want duplicates in my data. I just want one surrogate key for it. It’s totally reusable for anyone to use.

[00:44:19] Really?

[00:44:20] Shane: So let me play it back to make sure I got it right and then we’ll jump into a couple questions. So what we’re doing is we’re holding a table, let’s call it table. And in that table there is a golden unique identity, something we are generating that we, that never changes to say we, we have an instance of a thing, and then we are binding that to identifiers we see.

[00:44:41] So in your example, I might bind it to an email address and a social security number and a passport number and a driver’s license number because each one of those are identifiers of a person.

[00:44:53] So what we end up with is we end up, in that scenario, we would end up with the golden key that we’ve [00:45:00] generated four times four rows, and then each of the related identifiers, social security number, email address, license, passport number, and the benefit of focal is.

[00:45:11] The framework does that automatically for you, so if you go and say, that social security number is the same as related to that driver’s license number because you have a table that tells you that, then the focal framework is gonna go and do an upsert into that golden table.

[00:45:28] If it hasn’t seen it before or say we’ve seen it before, I know that relationship exists, we’re okay. Am I right so far?

[00:45:34] Patrik: Yes.

[00:45:35] Shane: And so the benefit of that is if I look at something like data vault where we have same as links. I’ve gotta code fire the code. I’ve gotta build my old framework to go, how do I do these relationships?

[00:45:46] And frankly, I’m not so sure, but probably the same. And then, so that’s the first part of the problem, how do we codify or patternise the identifications of these keys without having to manually do it? And then the second one is [00:46:00] how do we then use that in all our queries to be able to let me query any data with any of those identifiers that is valid at the point in time.

[00:46:09] And that’s the bit that you are saying in the focal framework. You still you haven’t patternised or solved that automatically yet that still requires some human to do that query side of using that data, or is that something you have solved?

[00:46:23] Patrik: It’s depending on what, because, ‘cause from a focal perspective we more or less never use the identifiers for anything else than creating the relationship to the key, so to speak, if there is multiple ways of doing it, multiple identifiers. So I will talk about that a little bit later. But so the identify table is a technical table. It’s not about using as a business representation of things. , You have your key, that is a unique instance or something if you want to search or do a query on the social security number. Then the social security number also is stored in the descriptive data [00:47:00] as its own. It’s its own representation. So you search there for a social security number, and we have said that there is a, there’s a separation of concern here. We should have, the identifiers should be able to live and develop however they will need to be to achieve integration. While the business identifiers should not be contaminated by that. So the business identifiers is held on their own in its free form, it’s clear. So if we do a concatenation in the identifier table, it’s not done in the, on the business identifier. Social security number. Social security number if we need to say. Sweden social secure number, and Russia social for some reason. Whatever the number the idea or the, in the scripted table, it’s just the social secure number clean, so to speak, as a business from a business perspective, which means you could get up duplicates when you search for that. You could get two rows, one Russian and one Swedish guy. [00:48:00] But yeah, that’s how it works. Then you just have to choose what, whatever you want to have it, what do you want to see? So that’s a separation of concern in the architecture the identifier table is a technical table should not be accessed by people, so to speak, because the key itself is really what hold binds it all together

[00:48:19] Shane: And then again, because it’s pattern based, you’d assume that LMS would understand the role that technical identifier table plays because it’ll see it being used the same way every time, and therefore it should use it more effectively than we’ve got bespoke tables or other things that are used in different ways.

[00:48:38] And then I think you said it doesn’t solve the whole fuzzy matching problem, but again, there’s patterns and code bases out there that allow you to do it. So all you’d be doing is injecting the fuzzy matching logic. To determine what rows you need to insert or check for in the identifier table.

[00:48:57] Right.

[00:48:57] Patrik: No, there is no, you can do it, [00:49:00] but the code as it is today is just you have it on the same row. They mean the same thing, two different columns. Yeah. So there is no magic. I usually say that there is no magic. It’s just.

[00:49:11] Shane: Okay. we talked about three object types. We talked about a thing that holds the keys, a table that holds the descriptions or the attributes as name, peer values, and then something that holds the relationships of the keys, this identifier table that holds this unique identifier and the system identifiers.

[00:49:29] Is that a different object type or is that reusing one of the, just to get, I got a funny feeling. I know where you’re gonna go with it, but Yeah, it, is it a different object type or is it just reusing one of those? A couple of those three.

[00:49:41] Patrik: No, it’s an, it is a different object

[00:49:43] type In the focal. We have the focal, there is a focal table that holds the surrogate key Exactly. As a anchor key anchored us. It’s only a surrogate. As I said, the identified table is a technical table to achieve integration on the [00:50:00] key, so to speak. So it’s its own type.

[00:50:02] But normally when I talk to Hans, we talk about, he says, yeah. The concept is the key, even if it’s two tables.

[00:50:10] Shane: so like we have with Data Vault where we have, HALs and SALs and these other things that we need to do from a technical point of view that break the, there’s only three simple type of tables you need. I’m assuming within focal, there are other instances of that where there are other technical tables or objects that we need for certain things, or is this the only one?

[00:50:31] Patrik: this is the only one

[00:50:33] I mean from a physical data modeling perspective, but then you have the whole metadata layer, but that’s another animal, so to speak.

[00:50:40] Shane: then we get to the problem of point in time reporting. And so if I changed my email address as one of my identifiers, at a certain point in time, it’s gonna solve that problem anyway because I’m now going to automatically get another row in that identifier table to say, against this golden [00:51:00] key, I’ve got another identifier.

[00:51:03] And then in my queries I’ll be able to use that to say, was it this or was it that? Okay. Yeah. I’m just trying to think around the whole, changing of of identifiers and then how that goes back.

[00:51:14] Patrik: I mean there is always that There is, yeah, there is no magic. So if you don’t have a stable identifier and you change your email address, you only have email addresses an identifier, and then you change the email address and you don’t have a column that says this was the old email address or something. It will create a new key because he doesn’t know about the previous email address. but if you have a social security number and an email address, the social security number will be stable and you change the email address. It’ll add the new email address to the same key.

[00:51:48] Shane: But if I had a table that held a change of email addresses an event, and so that table held previous email and new email,

[00:51:57] then it should pick it up against that golden key that [00:52:00] the email address had changed. And so there’s just another identifier against that key. Correct?

[00:52:04] Patrik: Yes. Then it would do that

[00:52:06] the identifier table doesn’t care which column you use. So it’s just a generic column of identifier. You just have a string of values. That one is one you check. There is of course the ability to going up to the metadata layer and see. What attributes in the source was used to concatenate this string. So it exists up there, but in the physical from a code perspective, it’s just, here’s a string. Does that string exist? When it checks, when it do a looks up, it doesn’t care if it’s what column you use to create that string.

[00:52:39] Shane: And again, it goes back to that idea that the metadata or the context is held outside of the physical data, so if you’re using an LLM you need to make sure that both are exposed because if you give a physical focal data model with its , physical data to an LLM, it’s gonna struggle to understand the context of how that works.

[00:52:57] Patrik: Yes. Very much yeah.

[00:52:59] Shane: Okay, that [00:53:00] makes sense.

[00:53:00] Patrik: I can just do another thing around this because when focal started and for the up to, yeah, 2014 or something like that, and 15, the, or even later. But anyway the identifier table was not optional. It was always, you always have a surrogate key. That was how it worked. But when we moved into the cloud area, we the cost of doing a lookup when you have a lot of rows costs every time. So if you have an entity that does not have multiple identifiers, why should you use a lookup? Why do then you can just as well hash the thing or use the natural key, right? So in the focal framework today the multi identifier functionality is the optional. You always start with a single identifier point of view. So you set up your entities and if you don’t have any , multiple identifier at this point, you just run it as like data vault [00:54:00] or something. It’s just natural or hash keys. But the thing is then that when you come to a point in your development that you all of a sudden know, okay, we need both social security number and email address to make this work. Then you can, in the focal framework, then just point at the entity and say, now this entity has multi identifier entity. And then that gives you the ability to map multiple identifiers to that as you do your mapping and say, this identifies and this identifies. And then the framework will also migrate your data for that entity. So it will. Propagate the new surrogate key that is generated. Why when you deploy, before you load it deploys and take all the wall data, integrate it to the surrogate key that you have created and populates all the tables that are involved. this is also when you think about it from a ensemble modeling perspective, is you have an entity and it has three relationships, right? And you point at that entity and says, this one is now [00:55:00] multi identifier. So it has to run on a surrogate key. It also has to migrate the data in all the relationships. And once again, since everything is metadata driven and pattern based, once we figured out the script on how to migrate from the normal key or natural key to surrogate key, it works for every entity. It was just a one go. You needed to figure out how do you do this? How do we move what? What kind of things do we need to do?

[00:55:27] And once that is done, it was repeatable. So it became repeatable. So you don’t have to think about it either. Oh, how should I migrate then? No, it’s just point that it say this is multi identifier now, and then it would migrate itself.

[00:55:40] Shane: And this is one of the value of these frameworks that are put on top of these modeling patterns is that they have been built out over so many years, decades, they’ve seen lots of edge cases, and the framework then gets updated with those edge cases and then. That code gets tested. That [00:56:00] pattern gets tested at time and time again by many people and many organizations and many industries.

[00:56:05] And it pretty much becomes bulletproof versus a senior engineer writing it for the first time or, go and find a blo of code from somebody, and then obviously you’ve got all the get repos, you go and harvest, and now the new one is your vibe code on the LLM.

[00:56:18] But none of that is proven versus you go to a framework that’s been tested over decades and you know that those, each cases where they’ve been dealt with have been proven. One of the questions I always have is, when do I ever trust an organization who tells me they don’t have a secondary identifier?

[00:56:35] And my answer is never, because they’ll tell you they’ve only got one. And then, three to six months later another one will turn up in the data and they’ll be like oh yeah, sorry, I forgot about that. And so for me, yeah, I probably would just go with multi identifier in the beginning and wear the cost of the surrogate lookup

[00:56:52] Patrik: at the same time, you don’t need to when you have the migration script, so you just, you run it as single until you need [00:57:00] multi. Just automatically migrates your data over to multi identifier,

[00:57:03] and then it’s just for that, and then it’s just for that

[00:57:06] entity also.

[00:57:07] Shane: if I’ve got billions of rows.

[00:57:09] Patrik: This is something we have seen when we worked in BigQuery and Snowflake. This is really important from a cost perspective, really. Because when it’s huge data sets, it costs a lot to do those lookups, even if they’re really efficient. So still costs data especially since there are no true indexes and stuff like that on these platforms. So you, even if you do a very good setup for. For the order of data and petitioning of data and what, can do in these areas. It still costs money. so that was the decision why we said, we know that a lot of entities will get multiple identify over time, but until they do not need it, we can run it on single identifier code, so to speak.

[00:57:56] Shane: Yeah. And it’s that idea of optimizing where it makes sense.

[00:57:59] You know, [00:58:00] 1, 1 1 of the things we do is, we will for smaller volumes of data, when we see an event, we will just do an upsert where we see larger volumes of data and we know some things are gonna happen. We might do a petitioning replace, but the system takes care of it,

[00:58:15] because it. You optimize it for, okay, if it looks like this, do that. Because we know over time that’s gonna save us money. And if it looks like that, do this and we can switch between if we need to. But it’s that baking in optimize at the right time at the right place. And don’t be afraid to change it later if you can change it safely.

[00:58:32] Yeah. Whereas in the past we would never touch anything like that. You say to somebody, oh, can you go into your dimensional data warehouse and just rekey all the surrogates for me? Yeah. That’s gonna be a dangerous change, right? Because yeah, it hasn’t been designed to allow you to do that.

[00:58:48] Patrik: Oh, exactly.

[00:58:50] That’s

[00:58:51] Shane: excellent. All. If people want to get hold of you, if they wanna understand more about focal, if they want to learn more about the modeling [00:59:00] patterns and the framework, where do they go?

[00:59:03] Patrik: I would say you can either go to LinkedIn to my profile, or you can go to Daana dev, which is the project where we’re working with with a focal framework creating CI around it so we can, you can easily test it.

[00:59:20] Shane: And give me an example of what you mean by building a CLI out for this. What are you looking to achieve?

[00:59:25] Patrik: But it’s more like. Doing an abstractional layer of of the focal framework since it is complex in its sort of design to make it easy for people to use. So the Daana CLI works with YAML files where you set up your models and your mappings and then that get processed into the framework and then the framework creates the code and stuff like that. So it’s more like just ease of use. So instead of trying to figure out how you’re gonna use the focal framework, there is already a command line interface where you can [01:00:00] do all the things you want to do with it, so to speak.

[01:00:04] Shane: The metadata tables and the physical tables, they’re technology agnostic, right? So you can run those on any of the data platforms or data technologies.

[01:00:14] Patrik: Yeah, and not, there’s always these small quirks in SQL that the different platforms uses. the interesting point is a focal framework have had Oracle sgl, L Server snowflake, BigQuery Postgres. But we are now in the middle of our sort of re visiting the architecture. So now we have pulled us down to Postgres and BigQuery. Where we start to build up all the different platforms again. But it’s still certain parts are platform specific. But since it is metadata driven, the it’s not that hard. Once you get it to work, it’s just shift.

[01:00:58] And now with ai, [01:01:00] LMS also, they can easily translate your patterns. So you can go now I want this to work in Snowflake and just show them all the patterns, and then they just transform it over. You have to test it, of course, because it’s lms, you don’t know what they’re gonna do. But it goes very quickly to, to move to a new platform at this point.

[01:01:21] Shane: But if somebody was starting out and they, wanted to reduce the uncertainty of things they need to learn, then you’re saying Postgres and BigQuery is probably the safest way to start out, to learn on it

[01:01:30] Patrik: now you can go in for the homepage, that’s Daana dev and sign up for be a better be tester and try it out. And then we have building up a community around it because people was to be able to contribute and stuff like that.

[01:01:47] So yeah, we see where it goes for me it’s really fun because all of a sudden the focal framework becomes accessible for people that, that doesn’t want to put the energy or effort trying to [01:02:00] understand underneath the hood. And that’s really fascinating, I would say. So it’s really fun.

[01:02:06] Shane: And even if we look at something like data Vault, which is probably a lot more well known across the world even then it’s still, you’ve gotta learn some stuff. So anything we can do to remove that complexity and make it for people to, simple for them to use the patterns.

[01:02:19] And that’s a good thing in my view, right? But it takes time and effort to make something that’s complex under the covers simple to use and execute and get that value of automation and. Optimization out of it.

[01:02:31] Patrik: Yeah. Yeah, exactly.

[01:02:33] Shane: Excellent. Alright, thank you for that. It’s been a great chat. I now know a lot more around focal than I did a while ago.

[01:02:39] So thank you for that and I hope everybody has a simply magical day.

[01:02:44] Patrik: Same. Have a nice day everyone.

«oo»

Stakeholder - “Thats not what I wanted!”
Data Team - “But thats what you asked for!”

Struggling to gather data requirements and constantly hearing the conversation above?

Want to learn how to capture data and information requirements in a repeatable way so stakeholders love them and data teams can build from them, by using the Information Product Canvas.

Have I got the book for you!

Start your journey to a new Agile Data Way of Working.

https://youtu.be/GjSD1pahPRE?si=1yg8gNsCb_4PvJWN

«oo»

Data Foundation and the Critical Role of Data Governance

Shagility — Mon, 20 Apr 2026 09:41:20 GMT

In this episode, host Mustafa Qizilbash and agile data coach Shane Gibson unpack the meaning of a data foundation, exploring why defining measures of success, rethinking data governance, and prioritising business context are the real keys to delivering actionable organisational value

Listen
View MindMap
Read AI Summary
Read Transcript

Listen

Listen / watch over on Youtube:

Tired of vague data requests and endless requirement meetings? The Information Product Canvas helps you get clarity in 30 minutes or less?

Google NotebookLM Mindmap

Google NoteBookLM Briefing

Executive Summary

This briefing document synthesizes key insights from a strategic discussion regarding the definition, implementation, and governance of data foundations. The analysis shifts the focus from purely technological solutions to a holistic framework encompassing management, organizational culture, and business context.

Critical Takeaways:

Definition of Success: A data foundation must be predicated on measurable business outcomes—specifically “up and down arrows” such as reduced time-to-decision—rather than the implementation of tools.
Governance vs. Management: While debate exists on whether governance should be an integrated part of management or a separate “legislative” body, there is a consensus that current data governance often lacks the power to enforce rules effectively.
The Three Ps: Effective foundations rely on Principles (cultural preferences), Policies (immutable rules), and Patterns (reusable templates).
Context-First Approach: Metadata should no longer be treated as an “exhaust” of system activity. Instead, “Context” (Business, Structural, Operational, and Agent) should be defined upfront to hydrate and drive data systems.
Organizational Culture: Organizations must identify as either Pioneers (experiment-led) or Town Builders (process-led) to align their data strategy with their operational reality.

--------------------------------------------------------------------------------

1. Defining the Data Foundation

A data foundation is often misconstrued as a collection of technology stacks (cloud databases, ETL tools, etc.). In practice, it is a multifaceted subdomain of data management that integrates architecture, platforms, ways of working, and governance.

The Problem of Silos

The separation of data governance groups from data teams creates significant operational friction. Management is the overarching discipline; if data is not being governed, it is not being managed. A foundation must bridge the gap between “artisan” manual work and “automated” factory processes.

--------------------------------------------------------------------------------

2. Measurement and Strategic Alignment

Before investing in technology, organizations must answer: What is the definition of success?

Success Metrics: Success should be measured by tangible indicators (up/down arrows). For example:
- Down Arrow: Reducing the time taken to make a decision.
- Up Arrow: Increasing the number of business questions answered.
Business Questions vs. Decisions: Framing requirements as “decisions” often leads to vague answers. Stakeholders provide higher-quality requirements when asked for “business questions” (e.g., “How many customers do we have in this region?”).
The Strategy Story: Data strategy is simply the story of how data will execute the broader business strategy. Without a clear business roadmap, a data strategy becomes an empty “PowerPoint deck” without execution value.

--------------------------------------------------------------------------------

3. Structural Frameworks: Value Streams and Factories

Data management functions through two concurrent value streams that must be balanced to avoid “building a plane while flying it.”

The Stakeholder Value Stream (Product Thinking)

This follows the progression from problem identification to value realization:

Problem Statement & Ideation: Defining the issue.
Discovery & Prioritization: Identifying the most viable solution.
Design, Build, & Test: Creating the product.
Deployment & Maintenance: Ensuring long-term value.

The Data Factory (Engineering Process)

This focuses on the internal mechanics of the data team:

Identifying data layers (Staging, Raw, etc.).
Establishing hand-off points between team members.
Automating testing and deployment to reduce “blockers” in the factory line.

--------------------------------------------------------------------------------

4. The Governance Framework: Principles, Policies, and Patterns

A foundational blueprint should be structured around three tiers of guidance:

The Power Gap: A major critique of modern data governance is its lack of empowerment. Unlike financial governance (overseen by a CFO with auditing power), data governance often lacks the authority to “shut down systems” or “fire people” for policy violations, rendering many policies as mere “wishes.”

--------------------------------------------------------------------------------

5. Cultural Archetypes: Pioneers vs. Town Builders

The implementation of a data foundation depends on the organization’s cultural starting point:

Pioneers: These organizations prioritize experimentation. They go out, settle new “land” (use cases), and see what works. Governance in this model is iterative, hardening into policy only after successful patterns are discovered.
Town Builders: These organizations prefer to build the infrastructure (the “town”) first. They establish the roads, schools, and zoning (rules) before inviting residents (users). This requires heavy upfront design but provides immediate structure.

Failure occurs when an organization’s stated culture (e.g., “we are pioneers”) does not match its processes (e.g., “we have heavy, slow approval gates”).

--------------------------------------------------------------------------------

6. The Shift to “Context-First” Data Environments

Modern foundations are moving away from treating metadata as a byproduct (exhaust) of technical processes. Instead, they embrace a Context-First model where context hydrates the system.

Four Tiers of Context

Business Context: Definitions of entities like “Customer” or “Product” that remain stable regardless of technology.
Structural Context: The layout of tables, fields, and system architectures.
Operational Context: Tracking who accessed what data, query performance, and decision logs.
Agent Context: The prompts and skills provided to AI/LLMs to guide their behavior.

Data Contracts

A data contract is a formal agreement between data producers (e.g., software engineers) and consumers (e.g., data engineers) regarding schema, frequency, and quality. Governance teams should mandate these contracts to ensure that data moving from operational systems to platforms is audited and reliable.

--------------------------------------------------------------------------------

7. The Role of Modeling and AI

As the industry moves from the “Decision Era” to the “Action Era” (automated actions driven by AI), the importance of conceptual modeling increases.

Conceptual Modeling as Foundation: The conceptual model is a representation of business behavior, not data storage. It should be defined without looking at source systems.
AI and Metadata: AI (LLMs) can “vibe code” applications and replicate front-end structures, but they struggle with back-end logic and physical modeling if they lack the underlying business context.
Physical Modeling as “Cattle”: Physical modeling techniques (Data Vault, Star Schema, Big Table) should be treated as disposable. If the conceptual and logical context is strong, the physical implementation should be able to change (”be hydrated”) as technology evolves.

Tired of vague data requests and endless requirement meetings? The Information Product Canvas helps you get clarity in 30 minutes or less?

Transcript

Mustafa: Hello, Shane. Hi. How are you?

Shane: Hey. I’m good. I’m good. Good to see you again. Thanks for having me on again.

Mustafa: Pleasure. Pleasure. Thanks for filling up the slot. Actually, one of the guy pulled out. He had some personal issues. Which is fair enough. All of us are senior guys. And we do book it, like, couple of quarters ahead and commitment can come in. Thanks for jumping in and making this flock fill up.

Shane: No problem. Hey, I’m looking forward to it. It’s kind of a subject that’s close to my heart anyway. So, as always, I’m sure to have an opinion.

Mustafa: Yeah. Because last, last week on our panel discussion, we had this session. What is the impact of not having a data foundation? So we all discuss this four or five panelists, and people really lied. This. And people always forget to click like on YouTube. But they always send messages one to one that they really like it. What is the importance of data foundation? But one obvious question start coming in. What is data foundation? Because for different people data foundation are different. Like data governance is data foundation. But data governance is huge. Then data modeling also comes into data Foundation. There are tons of number of data modeling techniques going in. So people are asking what it is. So you said you want to jump in, you want to solve this puzzle. Floor is all yours. Shane, let’s start it. What’s in your mind?

Shane: Yeah, I’m not sure I’ve solved it, but I’ve been working on it for a few decades, and I’ve got a set of what I call patterns and pattern templates. So. And kind of. I think about it like this. So my view, and it comes back to semantics, it’s all part of data management. So our foundational patterns, you know, patterns and governance, our patterns in engineering, our patterns and platform, our patterns and way of working, they all kind of, for me, come under that, that turn data management. And the reason I say that is I don’t like this idea that there’s a separate data governance group. And a separate data team. That silo behavior causes us massive problems. And so for me, it’s, it’s all within the same way of working. We may organizationally have our data team structures slightly different where, you know, people are focused and for scaling reasons and separate pods or groups. But I think governance is part of management. Yeah. And if we’re not governing our data, then we’re not managing it. So that, so that’s the first thing. If I kind of look at it from that lens, and then I look at. Kind of subdomains within data management, you know, we have ones which are around architecture and platform. And so when people say foundation. What I typically hear them talking about is technology foundations. You know, I need cloud analytics database. So I need an ETL tool. And what often happens in an organization is I’ll, I’ll get asked to come in and do some work with them either on a greenfields environment where they’re starting from scratch again. Or they haven’t had a data management capability or platform before. Or a brownfields one where they come in and ask me to do an architecture review. And as an ex technologist, my natural reaction is to get into the technology, right? Oh, that’s what they’ve asked me to do. Come in and look at our architecture, look at our platform, look at our tooling. And what I’ve learned over the decades of doing this is we need to step back and we need to look at some other things as well. We need to look at our way of working. We need to look at what our strategy, our team designers. And so over the years, I’ve kind of been building out this data blueprint template and set of patterns that I use. And what I’ve learned is the first foundational question that I always ask now, regardless of it’s greenfields or it’s brownfields or why I’m being bought in is what is the definition of success? Yeah. So if we’re going to spend this money, right, in terms of tooling and technology and people and change, and even if we’re using open source, let’s not. It’s not tell ourselves it’s free, right? Yes, there may not be a license cost upfront. But the five data people are going to cost you a million dollars in salary for a year if you’re in the US. Right. So there is a cost there. So let’s say we spent that year. We delivered everything we promised. Yeah, we made the changes that we needed to make. What’s the measures of success? Like what’s actually going to happen? And for me, I want to keep them really simple. For me, they’re just up arrows or down arrows. And I’m looking for things like, you know, when I often hear, we want self service. I’m like, well, that’s not a measure of success, right? That’s just a tool. It’s a pattern. So what does that give you? Oh, actually, we get quicker time to decision. Ah, okay. Right. So now what you’re saying is the time taken to make decisions in the organization based on data reduces right down arrow. Good. Or we, we don’t need to talk to the data team. Yeah, cool. That’s, that’s a process thing. What’s the measure of success? Oh, again, we can make decisions quicker. Okay, so we’re still back on that down arrow. Or how do we want to measure we’re successful with that? Okay, well, the number of requests going to the data team is reduced. Yeah. Or maybe we say, actually, it’s not self service is not the tool we’re going to use to deliver that measure of success. Maybe the data team are just going to get better and faster answering questions. Yeah. So the time to decision still goes down. But the tool we use, the thing we measure is how quickly the data team respond and answer that question. Yeah. And that should go up, right? We should be faster and faster. Yeah. Less time or more questions answered. And so that’s the first foundational piece I always think an organization should do. If we make this investment, what actually changes in our organization? What are the measure of success? What are the indicators, the arrows that are even going to go up or down that we can then prove that that investment was worth it? And I see lots of people not doing that foundational piece. What have you seen?

Mustafa: No, absolutely. Absolutely. As you, as I mentioned the other day, right, I have come up with this canvas, which is, of course, influenced by your canvas. Where you had gone. Last time we had one year back, we had this session. Right? So you’re absolutely right. I always ask, first of all, everybody knows what is value, but value is very subjective for me. Everyone want to create a value. There’s no doubt about it. Right? You ask anyone there. Yeah, we have a value. But how this value come up? So you need to have questions, answers based on the answer. You need to have actions. And based on the action, what you will actually have in your hand tangible measurable. That’s. I also look into it. So that’s also my first step. I always start with the canvas. Where I say. Upfront. Okay. Let me put in a different way. Right? I know because as you mentioned, wherever, whenever you go and talk to people, people are normally talking about technology. Normally. They have new generative AI, gente AI. We want to, we want to do something with it. Right. We have a car. There’s a new car in the market you want to buy. The question is why you want to buy a car? Right. So you need to have those things in your hand. So, yes, for me, that Foundation is also you put up your questions and answer the ones you are already be doing. Normally people want to explore new things. I tell them, do what you already are doing and how you can improve it. If you do not know the measure of improving it. Forget about the new things. I start from there once they know. Okay. Okay. We have been doing this with X technology or maybe why technology. And it has been giving us this value, but we need to go next level. Okay, let’s discuss. But what, what you’re achieving. So that’s how I also started.

Shane: Yeah, I, I use a frame in that similar but slightly different maybe. So let me talk through those. And it comes from the struggle that most data teams have. Differentiating between the products they’re building. If they’re a product team or the dashboards they’re building, if they’re not. But the things you’re building, right? The things they’re building using data and information that solve business problems, you know, help drive decisions, actions, outcomes and value. And so they like boxes of cereal right on the, on the shelf. You know, cocoa pops, honey puffs, right? They’re things that people are going to intangibly buy in your organization and use because it has value to their lives. And so we’ve got these products and then we’ve got the foundations of how we work. It’s the factory that builds those products. And data is a funny thing, right? We kind of cross people that have a factory. We want to build everything as a factory, fully automated, but for some reason, the data is always so shitty or the requirements are also new. They actually, we can’t just cookie cut a factory that works as a bunch of machines. So it’s this blend of artisan and automation. And people tend to confuse the foundational work around that, that factory and that way of working with the product we produce. And then what they’ll do is they’ll tend to focus on one or the other. You know, let’s do a founder. I don’t have a time to hear this. There’s still a foundational build. Right? It’s going to spend a 12 months. It’s always 24 to build in the foundation, change the technology, build in a lake. Right. And the problem with that is there’s no value to the organization. Yeah. Like, if we were building a physical factory, then maybe you could go and say, you know, there’s some steel, there’s a roof, there’s some machines. We’re getting there. But the problem with data is we, we don’t show how we’re building out that, that Factory way of working. And so 18 months later, we built our foundational, we’ve built our technology foundation in our factory. But we’ve delivered no value to the organization, and that causes a major problem. What’s the alternative? Well, the alternative is we just build ad hoc products. Yeah, we go in and we just build a product and deliver value. Right. Great. But then we get no automation. They changed our way of working. No ability to scale. And as we add more and more of those products, the business as usual over here to maintaining and just bleeds all the effort out of the team. So what we want to do is we want to balance those two things. Yeah. We want to build out the foundations at the same time we’re delivering value. And that’s hard because if you think of it, what we’re doing is we’re building the plane as we’re flying it. Yeah. And tell me any other industry when we do that, right? It’s kind of, you know, you don’t go, you know, if you’re a surgeon, you don’t go into an operating theater that’s emptying and go, oh, yeah, I’m going to do open bypass heart surgery, but I’m also going to build the machine that pumps the blood around. Right. And design the, the scalpel.

Mustafa: Yeah.

Shane: And so we’ve got to realize that we have to work on both of those things together. And an easy way to think about it is people always talk about people, process culture and Technology. Yeah. So the reason I’m looking left is I’m looking at my blueprint, right? And just going through all the bits that I make sure I fill out because that’s the foundational building blocks or patterns that I use. And so if we think about the people processing technology. The thing we don’t focus on after we’ve got our measures of success is the team design. Right. The version of the operating model. And that’s the next thing I always do is a foundational piece. As I go, what are the personas in our organization? Do we have this idea of data stewards or not? Yeah. And what do they actually do? Do we have a data engineer on how it looks engineer? Like, what’s the language? If we’ve already got a team that we already have. And how do they operate together, what’s their team design? You know, you can get things from team topologies. Are we working in small squads? Are we working in bigger teams? Do they do end to end? Do they do handoffs? And for me, that’s the next foundational piece of work, because by documenting the team design. And that flow of work, we get a foundational piece of visual hints on how the way of working is going to happen. And we can do that relatively quickly and early, right? We don’t have to over bake it. And once we have that, then we go into the next stage, right, which is what other foundational pieces do we need in place? But to come back ways of working and building products that have value, they’re two things that have to happen together. Otherwise we give ourselves in trouble with our stakeholders. Quite rightly, because we’re not earning value.

Mustafa: Yeah, absolutely. I see. Then what, what you are going toward is data governance, because nowadays I’m building a kind of data governance strategy for someone. Right? They ask you to build them this thing. I said, okay, let me build it for you. So their first question was, what framework you will bring in? What tangible thing we will have in our hand? I said, no, you can’t start from the what first. You need to understand what is why then how you’re going to do it and then the technology comes in. They start asking you which tool you will use. I said, stop for tattoo. Don’t talk about tools. First go towards governance. Because, because nowadays we have been discussing this CDO rule, Chief data officer. Right? So, like, what this role will do is. They will create a data strategy with the business. They will create a data governance strategy with themselves and they, of course, did us two words. And then there are data management strategy comes in. These three things comes right. Of course, first is business strategy. Then data strategy, then governance strategy and then data management strategy. Right. So what you are going toward is yes, as I previously just mentioned that I normally tell them, hey, if you want to do something, a data product, let’s take an example, how you are doing things now. People normally forget about it. People think that let’s take an example of generative AI. A tool came in. Hey, let’s bring the tool and let tool give the tool the data and let tool tell us what use cases they can bring it out. Right. That was the reason failure of generative AI. So I told them, no, tell me, even if you are doing manual, get me with your data analyst. What you are doing manually or semi manually or maybe something else. Let’s see that. Which is the actions can be manual or semi manual XYZ and then automated. And from that practices, then personas will come out. Stewards will come out. Processes will come out. So processes has to be predefined processes. It cannot be bring a tool and tool will bring a process. You need to have a process minus technology. What do you think?

Shane: So I think the, there’s a whole lot of bundle in there, right? So let me kind of unpack it.

Mustafa: Yeah.

Shane: So I think one of the points you raised and at some point I kind of missed is one of the first things I’ll do when I’m working with an organization on their foundations, as I will observe. Yeah. Because I want to understand what the current state is because it gives me a whole lot of hints. If it’s a green fields, sorry, if it’s a brownfield environment, right, I’ll observe the data team, obviously of the way they work. I’ll do some workshops with them to understand that. I’ll talk to the stakeholders to see where their real pains are. If it’s green fields, I still do the same thing because even though there’s not a data team and there’s not a data platform, there’s a whole lot of data work, right? It’s just been done in excel and outside.

Mustafa: Somebody else is doing it. Yes.

Shane: Yeah. Yeah. Somebody’s doing the data work, right? Normally. And I will create some very light artifacts around current state.

Mustafa: Yeah.

Shane: Because that helps us understand what those measures of success are. Right. What we have to change, and if we change things, what success if we make those changes looks like. Second pointers, and you kind of mentioned it is. The data strategy is just a story on how we’re going to execute the business strategy. So if somebody comes in and says, do a data strategy, it’s like, well, what’s the business strategy? You know, and if I get a slide that makes no sense to me.

Mustafa: Yeah.

Shane: The conversation in is, well, I don’t understand how we’re going to tell a story around how data is going to achieve that because that doesn’t, I don’t understand the strategy. You know, like what’s measure success for the organization, make more money, reduce costs, reduce risk? Yeah, great. But actually what we’re going to do, right? Like what actions in the next two years are we going to increase market share? Are we going to sell more products? Are we going to reduce cycle time for a patient coming into a hospital? Right? Like what actually is our strategy and our roadmap to achieve it? And from there we can say, right, how does data support it? What I see happen a lot. And again, one of the things I do when I come into an organization do the blueprint, the first thing I ask them is, could you please give me the four other data strategy documents that you’ve had from forever data consulting companies? Because, you know, they’ll be really useful. And they say why? And then my standard joke is I just want to know whether triangles, circles, squares or matrixes of the flavor of the month right now. But that’s not really the reason. The reason is there’s always good information in there. You know, there’s spent some time with some good people to create this document that’s a strategy. And that is valuable. It shortcuts how much work we have to do going forward. We still need to revalidate and make sure it’s not out of date. But the reason they have four other ones is because they got a PowerPoint deck. With some pretty pictures that tells them a strategy and there is no execution plan. There is no roadmap. There is no measures. I’m going to come back to it. No measure success. There’s nothing telling them whether this strategy is working or not and whether they should change it. So based on all that. The other thing I talk about is the blueprints we do should be light. Yeah. So we’ve got this again big design up front problem. We kind of want to see a bit of a blueprint, which is a vision of the future and how we’re going to get there. Measure success team design, architecture, ways of working processes. But we don’t want to spend a year doing that because a year of that work has minimal value. So we’ve got to be able to do these light artifacts, right, and then iterate them as we go forward. And then I’ll come back to the last one, right? Data stewards. So my question there I always have for an organization is, okay, if our team design has a role or a persona called data steward and we know what they’re going to do, right, what, what their job is, what their persona is, what their role is, what their skills are. What changes. And what will happen, right, is I will often hear we get better data quality. And then I’ll go back to the team design and I go, really? So what we got, right, is we got this data governance group, and this is typically a large enterprise. Right? Data governance group over here. Data team over here. Data management across the top. And you’re telling me that the steward is going to change the quality of the data. Because, hell, the ETL developer going in there and writing code in dbt, right? Yeah, they got no accountability on quality. It’s just wrong. Right. And so we’ve got to tie back to that foundational piece of if that metric, if that measure of success of increased data quality, whatever that means improves, like which group of people are responsible in the team design and the way of working for making that change. And that’s how I think about it, right? Like almost like Lego pieces. We kind of. Plugging them in. And the problem is the Lego pieces are always correlated. You know, your team design, your measures exist. They change the way you work. They change the way the factory works. They change the technologies you need. You know, my close-up one the last couple years put in the data catalog. Well, yep. The invisible context for an organization has value. But needs to start talking about, well, what’s the team design for the catalog? Right? What’s the measure of success? Is it number of data assets logged? I’d say that was a weak measure. So you can see how they all kind of, the foundational pieces all kind of interrelate. And you’ve got to be really good at choosing small versions of each of them and then iterating them and see how they then work or don’t work with all the other foundational pieces. And then you kind. Of doing systems thinking, right, you’re tuning the overall system of the organization over time. Does that make sense?

Mustafa: It makes sense. So for me, this is the first part. Right? Because then, so you, what you’re referring is when we start a journey. What we need to do, we need to understand the success factor, as you mentioned, the image. I call it decisions. Nowadays I’ve started talking like four hours. We had operationalization error where this called ERP, sap oltb applications came in. The second era was analysis era. Which was, which brought in data warehousing, task schema, snowflake. Then third era was analytical error. Analytical analyze data mining protection forecast data science stuff. Now the current era, what I define is a decision error. Everything has to start. With a decision. What you want to do. Starting point, what decisions will be impacted with this initiative? And from that everything. Your tickle back, you can, you can bring it back. Okay, I’m going to take a decision. Like buy go on the cloud from on premises to cloud. That’s the one decision. Now what you need, I will find it out for you. So that’s, that’s the starting point foundational that the starting point of a journey. Now let’s go to the next step. Next step will be your, I would like to bring your canvas into it because once we go into artifacts. Do you think, I think you will say that and I want to emphasize on it. And I’m following that as well after my canvas. Where I finalize a product or a requirement. Next, my canvas is your canvas, by the way, which I take to the canvas to the, to the customer and start asking questions. And as in your book, you mentioned the moment you customers start thinking about something else. Stop over there. Whatever is on top of their mind, that is what your initial requirement should be. Tell us about that, that foundation stuff.

Shane: Yeah. So. So it’s interesting when you said the word decision, I naturally took to a business decision using data because I’m a data person, right? So I’m like, which decision do you want to take? And then I naturally went, well, actually, we’re in the era of decision right now, but. We’re accelerating so fast. We’re going to be in the, in the wave of era of action. Next. Right. We’re going to stop talking about decisions and which action needs to be automated. So that’s where I went. Right? And then you qualified it with, which a decision around a change we have to make. And that, for me, is, is the key, right? Is this idea that we’re changing something, changing the way we work? We’re changing the way the team’s designed. We’re changing the technology we use. And so I sat more and more talking about change. As kind of the thing we’re focused on because decisions and actions, as I said, always takes me back to the data world of the product we’re building, not, not our foundational way of working. And then, of course, we get into change management, which has been around for a long, long time. Business process reengineering. And we bring us back patterns that have had value and we kind of lost, but we, we won’t go there. So the way I think about the next step is I think about value streams and factories. And so a value stream is the end-to-end process from when a stakeholder knows they have a problem. And they put their hand up. I have a problem. I need some help and it needs some data to when we deliver something of value that solves that problem. And that is the process that involves things like problem statements, ideation on how we might solve it, discovery or requirements, prioritization. The left hand side of what I call my continuum. Which is, and I’m a great fan of product thinking in that space. Right. So not IT thinking, not projects, not, you know, rescues, none of those things that we, we get from the IT domain. I’m a great fan of pulling out patterns from the product domain on the left hand side of that. So identify the problem, idea how we can solve it, discover which solution is going to be the most viable prioritize which one we’re going to work on next. And then the right hand side of my value stream is around design, build, test and deploy and value. Right. And then maintain. So we kind of lost designing in our, in our data domain a while ago with the modern data stack. Sure as with AI now, we’re going to be bringing it back in because those designs are actually the context or the knowledge that llms need to do the work without us. So, you know, that design, build, deploy, maintain value phase. And so I want, I want the blueprint to have really simple, what we call nodes and links. Like, just think of them as stickies and lines. Like who’s defining the problem space? And if I have a problem as a stakeholder, how do I engage the data team? Or do I not? Do I have a team of ba’s back to the organizational design team design? How do they get into the ideation phase? Right. How do we know how we might solve it? How much time are we going to spend on each one of those all the way through? And then I couple that with the factory process. And that is as the data team. Okay, I understand now which problem I’m going to work on. What’s my process? You know, do I naturally just go and grab the data first? Yeah. Or do I grab, see what data is available and mock up something and give some feedback? Or do I go grab all the data or a thin slice? Yeah. What, once my data layered architecture? You know, which, which layers am I having, which ones are rules I have to comply with, what tooling do I have available? How do we hand off the work between the team members and that factory? How do we test it? Yeah. How do we deploy it? So that all the way from I’m going to touch a bit of data because I know what the problem I’ve got to solve is to I’ve deployed something that somebody can use and it has some value. And so I’m a great fan of a thing called tedex talk called how to make toast. And it’s just a visualization way of doing systems thinking. And so as part of that foundation, I’ll get the data teams and the organization to, to work through those two value streams, right? Those two flows of steps, nodes and links. And so when we do that, what that also allows us now is to visualize where we’re about to make change. Because we can start basically voting and say, well, part of the value stream is broken, you know, so perfect example. If I look at somebody’s prioritization queue, their backlog and they’ve got 554 jira tasks in the, in their backlog queue. And some of them have been sitting there for a year. I would say we have a problem on the left hand side. Yeah. Around ideation, discovery or prioritization. And then we probably also have a problem on the right hand side around our factory that it’s too slow to move the, the work, but we want to focus on the lift. Right. We have a whole lot of work there that can’t be a priority because we haven’t built it. If we look at the way the teamwork and the data factory, you know, we can see blockers, right? We can see that, you know, this piece of work seems to sit in, in the factory for too long. You know, if you think about it as a, as a product, like a actual factory building cereal, making cereal, you know, we can watch the machines and we can see bring in some lean thinking and see how long they take. And again, we can go back to measurement. You know, the time between these two things takes too long. Like, what can we do to reduce the time? Is that more humans, more automation? Doing less? They’re all choices we can make. So I come back to your point around change, right? We want to visualize the system in those two different ways, and then we want to make conscious decisions on what we’re going to experience and changing. And that’s the last point. They’re experiments. We have a hypothesis, right? If I change this thing. Then good things will happen. And sometimes bad things happen, right? My favorite one. Is bring three more people into the team because we’re a bit, bit behind on building our products. Okay, let’s think that through. Right? We’re going to bring somebody in and we’re going to put them on the ground with the team. And let’s assume we don’t have a lot of documentation around the way we work or the way our systems work. So now they’ve got to learn our way of working. They’ve got to learn our technology systems. They’ve got to learn our domain. And how’s that going to work? Well, the team doing the work right now are going to spend half their time bringing those people up to speed. Now, in theory, they’ll, they’ll help us accelerate in the future. But right now that change is going to cost us three months of working slower.

Mustafa: Yeah.

Shane: Is that okay? Right. Because, you know, that’s the reality of that hypothesis. Yeah. So I’m with you, right? And your foundational space. Think about changes, right? Changes should result in things being better. You know, faster.

Mustafa: Yeah.

Shane: Quicker, more fun, easier, those kind of words. Does that make sense?

Mustafa: Yeah. Yeah. It makes sense. It absolutely makes sense. Before you go to governance and decomp management again, chicken thing, right? I have those questions as well. But let’s bring from this data product concept as well, just for our audience as well. And just to head out your, your thoughts around it, because as we mentioned, the previous adults were about project errors. We build a project. We start a project. We build everything front end to end centralized. And we, we wish that we can do something out of it. Right. That was projects. And then this product era comes in. Which you still feel is there. For me. I want to did a product is perfect. You can have a data product or you can have a data as a product, whatever you want to call it. But decision I want to bring in decision, right? Because why sometime I conflict with data product or software engineering practices is because you’re again creating a product, product may or may not work. I want to attack just, just to share my thought, right? You can comment on it. I want to attach a decision with a data product. I don’t want to start a data product without a decision. Because you can create a product, you can have a super good dashboard or headboard or slicing, enticing, and you end up having more decisions benefit on top of it. So what’s your thought on it? That die die or data product have to map. With a decision rather than analysis or analytics?

Shane: Yeah. So a little bit of semantics, right? Just for people listening so that when I use different words, they know kind of where I’m at. So in my language, an information product, there’s a boundary of a whole lot of things, data code, everything, visualization in a little box that solves the business problem. Yeah. So that’s, that’s what I call an information product. When I use the word data product or something like that, I’m talking about data essay. I’m talking about a table, right, that data asset identifying data products shouldn’t, but they tend to become just tables, right? Their data assets. So that’s, that’s my language. Right. And, and, you know, we’ve all had arguments around data product, data product thinking, data as a product. Right. And, and so I always think you just got to anchor your definition so that when you’re talking, people are clear what you’re saying. Right. It’s kind of the English to French problem.

Mustafa: Yeah.

Shane: But I’m with you. Right. When I talk about information product, I talk about the problem to be solved. The decision to be made. The action that will happen that needs to be happened based on that decision. The outcome that will happen if I take that action and it’s successful and the value that will be realized if that outcome is, is positive. And so that’s, that’s my thinking frame, right? And that comes from the product world. And then I make in the, in the canvas, which is a requirements gathering template. I change the language from decision to business question. Because I’ve found that if I say to a stakeholder what decision do you need to make? I don’t tend to get good answers back. Yeah. I tend to get. Fluffy words that I don’t understand and I can’t figure out how we can give them information to support that decision. Not always, right, but nine times out of ten. If I ask them what business question they want to answer with data and they start with who, how long, how many, how much, where, why? I will get back five to ten good business questions. And, you know, there’s things like, you know, how many customers do we have, which general are they coming from and how much money do we make out of them? Yeah, they’re good business questions. I can then ask them, okay, if I answer those business questions, what action will you take? Yeah. Oh, well, I’ll identify the customers with low margin and which general they’re coming from. And we’ll either, the action will take us either remove that channel or put investment in to make the people coming through that channel more profitable. Right. So what I’ve gone from as a business question to an action and then the outcome and the value. And these are decision in between, right? You’re making a decision to shut that channel down or you’re making a decision to invest in those customers through that channel to increase margin. But it’s kind of inferred, right? Now, what’s interesting in the AI world. Is with LLMs, does that change. The model, the pattern? Right. And I haven’t finished that work. Because what would happen if I told the llm that this action I wanted to make, is that enough now? And these are questions I haven’t answered, right? Because we’re into the, we’re moving into the action era, right? We’re moving away from the decision era and we’re moving into the action wave, I think is where the world will go. But yeah, so that’s where I’m at, right, is I talk about business questions that you need to answer and then what action will you take if you answer it? And then I get really strong requirements that a data team can, can build based off.

Mustafa: Awesome. Awesome. We are on the same page till now. Now let’s go to the next stage.

Shane: Yeah.

Mustafa: Because normally what we have just defined in last three, we move so quickly here. It’s already 35 minutes. So fun. Right? This is what, what normally I call it when I went into your canvas when you had a discussion. What you’re doing, it’s by right. It should be an obvious thing for me. It’s not any journey. All these things should be obvious. But people have forgotten about it. Because they were time pressure. There are so many pressures and they want to do it things very quickly. That’s why they forgot about these things. But your canvas says, right? You have to start with why and then how and what all those finish and stuff. Now let’s go. Okay. We as a customer, customer says, okay, we agree with you. That we have defined these basics. Now what? Now what is next stage of data foundation? Because the moment you have a data strategy, which we just discussed why thing, everything. Now there’s a governance and in parallel there is management data management. Two things underneath. Right? That is where I feel is an egg and chicken. Because if you create governance, which include principles. Procedures, principles, policies, procedures, metrics, tools and responsibility, all those things. Right? That you define and you are telling your workers, do it. But sometimes what happens doing ground reality is much different as compared to what we define. Now you go the other way. You say, okay, let’s build something and learn from there and then create a governance practice policy, blah, blah, blah. I normally give an example. You, you, you have a new colony. We have a open land. People came and start building houses. In the middle. You have a road can go through. You don’t have any direct any hurdles over there. But once kids comes out. They can run across the roads. Now you have a, you need to have a policy or you need to have something or maybe a divider across both sides so kids cannot go through. Now kids close up. They start playing. They play with cricket. We, we both know cricket with hardball and this stuff that can damage the cars who are going fast. Now you create bigger walls. So now these are the best practices or experience coming in which goes into a governance documentation. That if you build a new load, you need to follow this, right? So it’s an thing that is where normally wherever I’m talking, they are confused. What is foundation? Because if you build up front, we might not need it. But if you bring later, maybe it’s already broken. And we go under compliance. Way to start governance and management practices, various stuff.

Shane: So I remember my language governance as part of management. I’ll just keep reinforcing that. So this is really interesting, right? And this is something I’m working on right now.

Mustafa: On.

Shane: And I kind of think I have a set of patterns now, but I’m just testing them. And it’s one I’ve struggled with for decades, right? Because it comes back to this idea of building the plane while you’re flying it, right? Like how much of the plane needs to exist. And experimented for years around this and never really got traction. And so the framing of the pattern that I have in my head right now is this idea of pioneers and town builders. And you need to decide which of those two patterns is your organizational culture. And that is a decision that you need to make upfront. Because that frames everything else. Now everything else is just a lego block, right? It’s a pattern. It’s a patent template. I’m doing lots of work on those at the moment. For a whole reason. I mean, we could talk about in a minute around why I think templates actually help solve the problem that we had, which is we lost a lot of our domain expertise when, when we went to democratization, when we bought more people into our domain to do the work that we used to do as experts. We lost the education, we lost the training. And that was a bad thing, but it was also a good thing. So I think we have to teach people to do the work we used to do. And ways that had value. Maybe not the ways we used to do them, but in the same pattern they had value. Like data design, right? We should always do a light data design because it is a valuable thing, but we don’t need to spend six months doing enterprise data model anymore. So I think these patent templates are actually the unlock to teach people to do what we’ve got taught over decades to do. But I’ll go back to pioneers and town builders. So are we going to go out and experiment. Heavily. Find things at work? Buy things that don’t, and then start codifying those into our town. And build the town after we’ve pioneered the settlements. Is that our culture? Or do we want to actually build the town? And then get people to come and live in it? Right. Is that our culture? Now, when we build the town again, we can’t spend two years building the town before we let them come, right? So we’ve still got to do it and they’re iterative and a way that has agility. So let’s say that that’s a culture decision. And it’s an important one. And then what I can go back to is I can take that decision that organization has done and I can look at our team design, our measure success, our value stream and our data factory processes that we think we’re going to run with and go, well, you told me that you’re going to be pioneers and yet I see heavy team design, heavy value stream processes, right? So you actually, that’s not your culture. Like one of those two things is out of sync and we need to fix it. And so then, okay, let’s say that we pioneer all we town builders. The key thing is we have to think about principles, policies and patterns. Those are the three P’s that I use. And so again to anchor my language. A principle is like a culture thing. It’s like an agile minute based to us. We prefer this over that. We prefer small teams of three over big teams of 10. Yeah, we prefer delivering value earlier, even though it’s riskier. Than waiting three months and delivering value later than as higher quality. Or we prefer delivering value later with high quality over early with high risk. They’re just principal decisions that we want to make. And they set the culture of the team and the way the organization works with data. Policies for me are immutable rules. They are things we will get fired for. Yeah. So you should never store a customer’s name date of birth and passport number and clear text. Right. That is a policy. But we have to enforce that policy. So if I ever see a customer name date of birth and passport number in an excel spreadsheet. Then we’ve broken that policy. And the person who’s done that. Should be. Censured or fired. Right. And what happens though is when we talk about data foundations and especially data governance, we treat policies as wishes. And I’ve done it, right? I used to write these principle and policy documents that were massive and they were kind of rules, but they either got fired. You know, I work for a bank and they had all these policies. And yet I saw them posting a customer’s name. Their account number and their last transaction in a teams chat because that was the, yep, because that was the only way that team could solve the problem for that customer at the right time. And so they were breaking the policy. Now, the reason they broke the policy was because no other system was in place that let them solve the problems and they had to solve it. So they solved it with the tools they had. But what we should be saying is that team’s breaking the policy. What do we need to do to change what that team’s doing? So they actually don’t break the rules? Yeah. So policy is very small targeted set of rules. Like you said, the police state, you know, police things, you know, you, you can’t drive a car at 200ks an hour past a school road, you know, during school time or you will be locked up in prison, right? Quite rightly. Um, so that’s there. And then patterns are all the other things that are useful. Yeah. They typically what I see people write policies as if you do this, it’s valuable. If you do this, it’s useful. You should kind of do that. And that’s what I look at those three P’s principles, things we should subscribe to this versus that policies rules that I’ll get fired if I actually break them. And so I’ll come back to that. If we say that that’s the rule, then we even need to automate detection of when we break the policy. Or have auditors in our organization that will go out and audit that those policies aren’t being broken. If you don’t enforce a policy, then actually it’s not a rule, right? It’s a wish. And then patterns, what we should do is have people that share their patterns or people that define their patents. So let’s take that back to. The idea of pioneers and town builders. So if I’ve got a pioneer organization, yeah, then I’m expecting to set some principles upfront. You know, some disperses there. I’m expecting those pioneers to iterate around the patterns. And then I’m expecting to discover the policies that are important. Yeah, because they’re going to do some things that are wrong. And then I’m going to go, yeah, no, actually, we need to harden that. Right. And so we’d then bring that back into a policy. If I’m a town builder, I would expect the principals policies and patents to be built by the town building team, the data team as they work. Still iteratively, not two years up big build up front. But I’d be expecting them to do that work. So then it comes back to team design. Right? Because what I’m then saying is data governance is just a skill and a set of roles within the people doing the data management work. It’s not a separate team. But you can make it work. If you’re a town building organization, you can have a separate governance team, you know, like a team of architects or a team of town planners from the council that make sure you’re actually buildings of high quality that aren’t going to fall down in the next rate. Right. But you’ve got to decide which of those you are. And then bring in everything else to align with that way of working. Does that make sense?

Mustafa: Yeah, that makes sense. But I want to, I want to throw a challenge over here in my point of view as well. Regarding governance and, and data management. Right. As you correctly mentioned that. When we are doing stuff, then, then governance, as you mentioned, is part of the management. Right. But, and you, you gave an example of pioneer or Greenfield or wrong food kind of stuff. Right. But for me, in data world, right, because everyone wants data. Decisions on data or value out of data. And for me, they might not be any organization. It will be hard to say that anyone who do not have data, everyone have data and everyone have one or the other way data management practices already there. Because we have come from four hours operationalization analysis, analytics. And now decision you’re saying actions as well. But I bring action under the umbrella of decision. Right. Because without any action decision cannot happen. Right. So now the thing is. I want to bring this anger into it. That that era where you do things and learn and create practices. And policy, as you correctly mentioned. I always principle. Policies and procedures or patterns, these things. Right. I think these are these templates are already there. I think that is where I am already also telling customers that either you are, your domain specific, your banking or telco or some one of the, one of the domain you are holding. And there are a lot of patterns and templates already there. So I. Am on the other side of the table with apologies to you. Is I don’t want to mix governance with management. Because if you merge governance with management. Governance change based on what you’re doing. I feel we have come out of that era where governance practices are already matured. I feel you can bring up the template with by the, by the smes subject matter expert, as you mentioned. Last time subject matter expert was the one who doing something. Now we brought other domain people into the data world so they can do things for us. And that is there, I think they’re all chaos started happening. Right. So I feel data governance need is kind of a law. You’re creating a town. The data governance is law and legislations, whatever you want to call it. And managing is doing the construction. Now there are best practices when you’re building a town. You already know which kind of housing Society is going to have, which kind of knowling you are going to have. You need to have a police station. You need to have how many hospitals, how many pharmacies you want to have. So I, I feel this team need to be little small team. I’m not saying a bigger team, small team on governance need to be there. And there has to be domain expert and they have to be data savvy. Well, I can, you can say I’m a biased person for a data because I am from data. I always emphasize that in data governance, data people need to be there. Because once data people there, they already know what they are building and they can create principles, policies and procedures and patterns based on their past experiences. We don’t need to reinvent the wheel. I think we are already 10 minutes left already. Do you have a hard stop?

Shane: No, no, I’m good so we can carry on if it’s okay with you.

Mustafa: Yeah. Do stop me because I’m just throwing my ideas because I’m building a data model strategy for someone. So I was telling them, bring up the best practices, right people, because skill set is very important. Nowadays, because of AI, AI is a black box. It creates something. I have an example I gave into last panel discussion. I give you to you now as well for your thought process. Right. That if you don’t have this, then everybody will do whatever they want to do based on their experiences. And you will have multiple flavors of data governance. If I go with your mindset that data management need to have a data governance, do you see this risk multiple flavor of governances.

Shane: Yeah, yeah. But.

Mustafa: Oh, you don’t mind that.

Shane: No, I do mind it, but I think it’s the reality at the moment. So again, I’m going to slightly disagree with you but agree with you at the same time.

Mustafa: Absolutely. And on the, I want to hear it out for my. I can.

Shane: Okay. So let’s, I’m going to put a stake in the ground and then I’ll talk around and come back to it that I don’t believe we have data governance. I don’t believe I’ve ever seen it. Right. And let me justify that. So if I’m in an enterprise organization. We have financial governance. Right? Like we have people who are accountable, CFO to make sure that our financial records are accurate. And we don’t overspend. And then they have a bunch of principles, policies and patterns that help them do that job. They have a thing called a profit and loss. It is a pattern. That every accountant applies like a balance sheet, like a trial balance, like a cash flow statement. That is part of the practice. As part of their way of working, you get qualified as an accountant, you know what those patents are. You apply them. You are ordered by them. You are held to account. They are governing artifacts. And the data domain, we have check. We have data modeling, but you can have any flavor you want and you don’t even have to follow it. We never audit any of our data models. Right. You can say that you’re storing data encrypted, but does anybody ever audit you? You can say that you can’t do this. But then people go and do it. And that’s part of the problem. And if we take our town plan, you know, the idea of building a city. We have governance because there is normally a council, a regulatory body that says if you’re going to get, if we agree that you’re allowed to build that town, here’s the rules. You know, you have to have a school. You have to have buildings that aren’t more than three stories high. And that governance body has power. To punish you if you don’t follow the rules. In my experience, it’s very rare for a data governance group to have power in an organization. Now in some organizations it does, but I find that in my experience a very rare occurrence. And so I then say back to my stake in the ground. I don’t think most data governance teams are governance teams. They’re not auditors. They’re not the CFO. They’re not the risk committee for a bank that will, you know, shut you down if you break the reserve bank rules. You try to do the right thing, but they’re not empowered to govern. And that for me is the problem.

Mustafa: See. That’s why I’m saying that skill set comes very obvious over here. When I say, of course you are right. Data governance comes under the domain of governance. Right. And there are financial governments. There are many other governance in the organization. Can you believe that in a financial governance strategy is built by non finance guide? You can’t think about it. The same way I feel for data governance. It has to be built by data. Guys. What you’re saying is absolutely right because I have been working in right in organization where I was not part of data governance. We never heard about. We never follow their policies. Reason being we already have way to justify. Right. Because they are not data people. That is my obvious first prerequisite is if someone is building a data governance team. Data people need to be there and senior data people because it’s about law. You have to implement a law. Law can only be implemented if people understand the law. Otherwise people will bypass the laws. That’s not my point of view.

Shane: Yeah, I’m going to disagree with you again. And the reason, and I haven’t thought about this, right? And so I’m kind of speaking as you’re making me think. So that’s great. Thank you. I go back to as data professionals, we have no practice. We have no rules. We have no certification that we’re held accountable for. Therefore, we can’t govern ourselves. What we are is we’re Practitioners. Thinking of the hoop, you know, I think I’ve seen CFOs, chief financial officers being the best data governance people because they come from a practice of governance. But like you said, they’re not data professionals, so they don’t, they can kind of know what to govern but not how to, how to make sure it is governed. Right. And so I go back to this user car analogy. You know, the rule is I can’t drive more than 100ks.

Mustafa: Yeah.

Shane: That’s the rule, right? That’s the governance in this area. Well, actually, as a car manufacturer, why don’t I limit the car to never going over 100ks? That doesn’t happen. Right. And it’s the same with data. Like, yeah, I can’t do this, but everybody does. So I think that’s the problem. And then we come to llms. They, they’re, they follow the rules as much, you know, they’re non-deterministic so. But everybody’s trying to make them deterministic. So let’s say we finally get that right and it’ll be really interesting. We ever do. The second problem we’ve got to solve is, is the rules aren’t clear. Right? They’re not described in a way that if the elderly was deterministic, it actually knows how to follow it. And that is a problem with hemp for 30 years. If I drop into an organization and I say to them, what are the rules? I get a conversation. I don’t get a set of rules. And that’s part of the problem that teams have struggled with. And that’s why I come back to this idea of, of patent templates. And what I mean by that, and the canvas was just one of them. Is if we have a thing that we fell out or do as we’re doing the work, and that tells us whether we are meeting the governance rules or not. We kind of make a whole lot of problems go away. Because I’ve got a, the first problem is I have to understand what I’m filling out, right? Like, I’m, I’m filling something out. I have to understand what that process is. So if I have to fill out a conceptual model. You know, I’ve just Concepts and relationships, you know, customer orders product from store. I have to understand how to fill out that concept model, that template, but I actually have to understand what I call a concept is, like, how do I Define what a customer and a product and a supplier and a store is? How do I Define that relationship? But once I’ve done that, now I’m by default, I’m going to start to get a template that’s got some data in it that can then test the policy of we will only Define customer ones. Yeah, probably a principle or a patent, not a policy. Right. Because you only get fired for doing, doing it wrong. But that’s kind of where I’m at the moment. Right. And so what I think from a data governance lens, if we, if the data governance team can’t believe the true stakeholders that are accountable for that government’s been applied, then they’ve got to be enablers, right? They’ve got to have ways of giving patent templates and ways of working and patterns to the data teams so that those rules are complied with. And that’s kind of where my head’s at the moment, right, is, is giving teams things that if they follow those things, the rules are automatically applied for them or flagged when it’s not. Yeah. But really think I’m still in the mid of thinking of that one. Right. So that’s why there’s not a, not 100 Clarity on, on that one.

Mustafa: Absolutely right. Because we, I am trying to tag this data Foundation where governance and management. Because let us bring another flavor for your food for your thought. Right. Matter data. Let’s bring matter data because for, for me, the data foundation which we are talking about, as we mentioned earlier in this session is your product can best practice as all we discuss decisions and blah, blah, blah. Those should be an obvious. Now those have surfaced because AI came up and start showing that what we are doing wrong. But those were supposed to be their practices supposed to be there. Now. Metadata. Because metadata is also for me is key because when AI comes in. Okay, let me say this example again which I gave in my panel discussion last week. What happened? I was not left coding like 15 years back. But this wipe coding everybody keep telling Mustafa go and try, try, try. Never ahead of time. Now I’m a jobless like few months. You know that. So I went into coding. I opened a tool called cursor. It’s a white coding tool. I started creating, replicating. LinkedIn. So it’s an awesome tool. I gave it a LinkedIn URL. It replicated 50 to 60% of LinkedIn. Those things did not replicate it, which was not. Possible to open without a username as it’s replicated. It treated the whole data model underneath. And it was working perfectly fine. Now the question is AI can create a product. Now what I did is. I picked up that data model. Put it into a different schema. And ask the same wipe coding to read that schema. And create. Linkedin like portal. It could not do it. Because when it creates that schema, it went from application front end and created backend. But when I gave it a back end, it could not create anything for me. It was a rubbish thing. Like which gave me an idea that, hey, because metadata of those tables are not there because when it created from linked in meta data was there. Conceptual, it went back. It created business model information model. It created logical model. It created conception model. Well, there are many kind of data modeling technique backend. It created that. Then it created a physical model. And it works. Now the second example which I had, it could not. Then what I did is I asked it create a business model on top of it. Informational model fcoim conceptual logical. I created all those and mapped with the physical model. And then I asked it, go and create a portal. 90. So for me, if AI. Asking you to start creating a product, it will create. But if you build a foundation of data product, AI might not be able to. Because metadata is not there. So now correlating our topic data foundation. If we, are we saying that. Creating modeling modeling, as I mentioned, business modeling information modeling, all those things should be a separate practice. And that is the foundation which we are talking about. And we, we can argue there are so many data modeling three and a half star schema data world focal enter hope. So many are there. Right? If we are talking about, are we talking about only up to the logical foundation as a logical modeling? Because after logical modeling, physical comes in. And that is where AI is unable to do it. Are we saying that we should emphasize or spend time building foundation of data modeling up to logical and which will be, by the way, part of the greater governance? If not data management, data management comes when you go towards physical modeling. Are we saying that. Okay, you, you can comment on it. You can consider for me physical modeling can will always change because new tools and Technologies will come up until unless we have a proper up to the logical model. Then these things can be reused. Are we saying these are the data foundation we are looking for or not?

Shane: Okay. So that’s a big question. So let’s, let’s go back to the core preface that every part of our foundation is a lego block. Yeah.

Mustafa: Okay.

Shane: So there’s small parts that we put together to work the way we want our team and our organization to work. And we don’t want to spend three years building out all the lego blocks. Because three years of great fun, but no value to the organization. And we don’t want to just do ad hoc where there are no lego blocks that we can reuse later. Right. So we want to be in that dodgy middle ground and we’ve got a whole lot of decisions like which Lego blocks are most important. Which ones do we do first? Which one do we use second? What order do we need to do them? Which one’s a small Lego blocks? We can spin a little amount of time, and that’s okay. Which ones do we actually have to spend a little bit more time because it just involves them?

Mustafa: Yeah.

Shane: So, so let’s use that as our framing and then let’s go back to the blueprint. Right. So we, we’ve understood our current state. We’ve gone and figured our measures of success. If we spend this money in time, this is how the organization is going to change. We’ve done our, our team design. We know kind of how the team is going to be structured and where they live in the organization and the skills and roles they have. We’ve done our value stream. Yeah, we know kind of the process from when somebody has a problem, all the steps we’re going to take. We think it’ll take till we deliver value. We have our data Factory kind of sketched out, we kind of think this is the way the data will flow and some of those decisions. What I, and we got some data principles. Yeah. Some kind of disv is that. We might have a couple of policies because we already know there’s some bad things we shouldn’t do. We probably don’t have any patterns yet. Then what we do is we then look at, for me, I look at kind of capabilities. Like actually, given all that, what do we actually have to put in place on day one? To, to make some stuff work, you know, get some products out the door that are governed, you know, that are managed. They managed. Right. And so I, I kind of then that’s where we start getting into tooling capabilities, technology now, right? Like, I need something that does this. I need something that takes the data out of those systems and brings it into the platform. Because we’ve made an architectural decision that we’re going to bring all the data into one place. Yeah, it might be a lake, it might be a data warehouse. That pattern, right? We haven’t made that decision yet. And we’re not done that in the past. I’ve always had metadata is important. But we’ve always treated metadata as an exhaust. I’ve always gone and said, well, we don’t click the data. We can’t build anything really important. We can’t store the data. Yeah. Again, if we can’t transform the data. Yeah. And if I can’t visualize or deliver something kind of thing. And so I foundationally always came back to those moving parts as being the first moving parts of this Lego blocks that have to be built out. And then I’d bring in some development practices. I’d say, well, you know, we want to be able to check our code in. We want to be able to deploy and test. And I bring in some other Lego blocks. And for whatever reason, metadata. Kind of always came last, right? It was kind of treated as exhaust. We did all these things. They’re not now need a catalog. Right. But over a week while I flipped my thinking and, and I kind of said, actually, we need to capture that stuff first and then hydrate everything else. Now I’m just going to go bound us to some language again. Right. So 10 years ago, I talk about metadata. I’ve talked about being metadata driven. Yeah. Tools that are metadata driven to find the metadata. The tool does everything else. And then I kind of moved on to config. Yeah, it’s config driven because we’re kind of configuring how the system should work and it should hydrate everything else for us. And then last year, there’s a big, big semantic war in the data world, which was basically semantic layer or context. Yeah. And I’m a, I’m on team context, and I think we’ve won that argument. Although our library sciences friends are now bringing an ontology taxonomies, our and RDF and we’re back into another semantic war. So I’m just going to use the word context. Right. And the way I position it now as patterns in my head is we have business context. Yeah. So definition of a customer is business context. Our domains are business context. We have a whole lot of context that is relating to the way we run our organization. We have structural context, you know, the way our tables are structured, the way our fields are structured, the way our systems, the, the things that I would have called metadata 20 years ago. We have operational context. The queries that have been run against our data, who accessed what data when, who exported what data when, what decisions, there’s a whole decision logging kind of thing coming through now. So there’s all this operational stuff that’s really useful from a context point of view. And then we have agent context. The prompts that we give the agents, the skills that we give them if we’re using something like Claude. The contents that we give it to help reinforce some of the models. Right. This context or things that, that drive the way the agents behave. And that’s how I think about it now. Right. And, and foundationally, I would start capturing parts of that as I go. But I, again, I’d flip the model. The way I think about it is we want to capture the context and then have that context drive everything else we do. So give you an example. If we say we want to base on all that, we decided that policies, that we’re taking in more of a town plan approach and we want to put in place some policies, right, that are complied with as we built that town. Or even if we’re doing pioneers, we say this, you know, one or two policies applying is can’t break. Yeah, they can build a town wherever they want, but they can’t build it on the land over there because it’s owned by a different organization. Right. So, yeah, don’t just know you’ll be fired if you, if you do that. Everything else is on. How do we capture those policies as context? And then how do we hydrate everything from it? Yeah. So how do we say, you know, you can’t store customer name. Yeah, it’s a policy, that’s context. And then how do we hydrate a system that actually checks it’s never been stored. And automates that formula as a test. Yeah. And that’s how I think about it. Right. So, yes, I agree with you metadata first, but truly metadata first. Right. We write the context first and then the system builds itself as much as it can off that context. Now, if we don’t have technology that can do that, that’s fine. We define the context first and then a human rights code that enforces that, uses that context. I think that’s where we’re going to end up in the new world and the new data world. Is everything that’s going to be defining context first and then hydrating all the other moving parts that we used to do as humans over time.

Mustafa: You’re absolutely right. Just to, just to bring in same example which you’re mentioning. Right? That. Policies has to map with the, if you have a proper metadata down there, then you can attach your policies with the meta data. Like you give an example of mobile number or ID card or a name or account number of the bank. Right? You cannot. Make those visible. That’s a policy. Now that policy need to be implemented somewhere by data management folk, which is engineers. Right? Data Engineers. Now giving this policy to them. They are not good enough to understand the business. For me, this policy need to tag with the meta data because in metadata, I normally segregate business metadata, technical metadata and operational metadata. I feel the foundation is business metadata. Because once you have defined business metadata, that won’t change. In an organization until process doesn’t change. Business meta data is about processes, how everything works, what is the definitions of blah, blah, blah, blah, all those things. And then policy attached to that particular business model. And now when management folks comes in. This policy get attached into the development activities as well. That wherever one example, wherever ID card is there, it has to be masked. So people don’t have to think about a developer, don’t have to think about it because it will go through a, maybe a lookup table or maybe something which automatically do it rather developer has to manually do it. So that is where just to bring my thought process back that data governance has to be little bit maybe coming era, you think about it again. In coming at a data governance has to be from the prospective of business matter has to be separate. Because if you give it to management team, they won’t care developers are who are the developers python developer or SQL developer. Their developers, we don’t expect them, as you correctly mentioned early in this session is we brought other domain people into data domain and start doing developers because they don’t care and we should not put this responsibility on them.

Shane: Yeah.

Mustafa: To find out what they need to mask, what is the policies. We make it. Again a principle. Principle is follow all the policies. That can one principle.

Shane: Yeah. But, but the least type of scenario, right? So, and this is where I go back to finance, because I think they’re well ahead of us. And, and some of this. You know, they, they have this idea. I’m sorry. I go back to governance. I see governance teams either taking one or two stances. They’re auditors or their coaches. If their auditors, what they say is, well, here’s the policies, and I’m going to come in and actually audit you. To make sure you’re following them. If the coaches, they’re saying, here’s the policy and here’s patents or tooling or things you can use that make sure that policy is enforced and they don’t have to worry. If I’m a finance auditor and I know that you’re using a financial system and you have a p m l. That’s a pattern based governance because I know that there’s no way the p l can not balance, you know, because it’s these things. And within the data world. I still, like, we don’t have the idea of microservices for our tests, right? Global ones. We have local ones, but we don’t have. A microservice that says, hey, that data, you know, anytime we see something that has ID in it, an ID value, it’s encrypted. By default. Right. The team doesn’t have to care. And actually, we used to have that in databases. And when we went to file systems like hadoopa now iceberg and we kind of lost a whole lot of the free that we used to get in made our life easier. So auditors or coaches have systems that enforce what’s right. And then if we’re auditors, if I look at people that do auditing, they actually have a plan on how they’re going to check that control, that policy. Right. What are we doing data governance? We say, here’s a policy. And I go, how are you going to go check that that’s been enforced? It’s the data team’s problem. Right. And again, we haven’t solved that. I think the last thing for me is if I go back to this idea of context. Being defined first and hydrating everything. Those capabilities don’t exist yet. And they don’t even, it’s not even that they don’t exist in, in the data teams. We need them to exist in an organization level. So, you know, if I say the policy is ID can’t be stored in clear text, then that policy has to be automatically applied in my data platform and my software as a service and my operational systems and my Excel spreadsheets and my SharePoint and my text file sitting on my local hard drive. Like, it’s a policy, right? It’s not just a policy for the data team. And that idea of defining context and having it hydrate all our systems, I haven’t seen that yet. We get it one day, right? But I think it, it’s a big technical ask to figure out how the hell you, you do that as well. Right? But any step we can do, which takes us. Away more and more away from this idea that metadata is exhaust context is exhaust light. We kind of look at what the system did and said that was the context. And we flip the model to define the context and get the systems to follow that context. As we do the work, I think that’s a good change in our foundational. So coming back, as I talked about, you know, the foundational technology pieces that I’d always looked to as data collection, data storage, transformation, visualization, and then a whole lot of other stuff around it. I’m now saying actually defining context early. Is the first foundational piece before I do any of those. So defining what sort of systems we have as context, where the data lives in them and then how that hydrate the data collection of that data into my platform. That’s what I do. Yeah. It’s a change of thinking and it’s also a complete change of working.

Mustafa: No, absolutely.

Shane: Yeah.

Mustafa: Just to close this session, right, you are absolutely right. That for me, at least people, most of the people think that data governance is more towards only data people. I join data governance business as well. Data governance need to be implemented as much as in data side as much in business side as well. It cannot be one sided and this aside we all know data store word is also become is one of the critical rule in data governance. That they have to work hand in hand with data governance team so they can make sure that business is also following the same thing. As you mentioned, it’s not about putting policies on data folks. It’s putting policy, even IT card if it is shown into your email, it should not show. Even it is in your excel, it should not show, which is on your desktop. So yes, there are a lot of technical. Depth or technical implementation need to happen. With the governance. So just to repeat myself, it had to business side. It has to be the it and data side as well. Following all those governance thing. So I think we have covered most of the thing. I want you to go to a data modeling as well, but we have already passed our dedicated time because data people are asking that when you just to throw in last question, right, people are asking, when you talk about data foundation, are you talking about data modeling? Physical data modeling, right? Even before logical like let’s go to one step back conceptual data modeling. Because logical data modeling is something which. Will. Normalize the tables and entities up to the level it can be implemented. Right. People asking me. That data foundation are you saying that we should finalize our decision that it will be star schema. It’s a data vault. Which kind of data modeling we need to implement. Is that the foundation? Just the last question.

Shane: So I think we have a foundational lego block of our data architecture layers. You know, how many layers staging raw, whatever we’re going to call them. Are they enforced or are they optional? How does the data move through a system? That is a foundational piece we do early because it kind of sets the scene. So we’re going to do later. And it’s, it. S harder to change later than it is to define upfront. I’m co-writing a book on conceptual modeling right now with your copola. So I’m very, very opinionated on this. But if I go back to this idea of context and hydration, my view is. A conceptual model is one of the foundational pieces that I would now bring in very, very early into a positive date as of part of data governance. Then logical modeling. Has some value, you know, where do the attributes fit? And then physical modeling. It’s important. But what I would do is treat it as a candle. Right. So I would say if I use the devops terms, my conceptual model is my pit. And then I should be able to hydrate a physical model. From that conceptual and logical model whenever I felt like it. Yeah. Data Vault, dimensional one big table activity schema. Right. I, I should treat those modeling techniques as disposable now because my conceptual and my logical models are my business context. And those physical ones are my structural context. So that’s how I think about it now. But most people don’t. Right. So then.

Mustafa: On this one. Absolutely.

Shane: Yeah. If you don’t think about that way, then you, you need a patent or a policy. On your physical modeling technique because you’re, you’re losing all the value of doing the context stuff up early. So, yeah, pick one. Yeah. And just be clear as an append. Like I should use data vault or it’s a policy. I’ll get fired if I don’t use data vault. Right. Which of those are, which are those two? Are you telling me? And then I take your point about governance. Right. So I was just kind of thinking about as you’re talking about it, and I kind of give you an example to close it out. There’s a lot of work around in the, in the data domain right now around data contracts. And again, let me, let me anchor my, my terminology. Right. A data contract is an agreement. Between somebody on the left and somebody on the right on how they’re going to share data. Yeah. So it discusses schema frequency, quality. It is a, an agreement. Right. And then we tested that agreement that’s been there. Typically, it’s an agreement between the software engineers and the data engineers because we’re moving data from a software system to the data platform. But I’ve seen lots of people now using it internally within the data platform between ETL jobs. Yeah, it’s a, it’s a contract and agreement that this table will look like this and this table will look like this. And therefore I can write the code in between. If we think about that in terms of true data governance. The data governance team should be mandating that data contracts are put in place whenever data moves from a software system to a data platform. And they should be auditing that actually happens. But we don’t. I never see that. Right. Yeah. It’s always a data team.

Mustafa: Don’t do it.

Shane: Thing, right? It’s a tool for them. But actually, just think about that, right? From a governor’s point of view, you know, when we go and have, we go and buy something from a supplier outside our organization, most organizations, every procurement team that governs there’s a contract in place. Right. We know what we’re getting and what we’re paying and isn’t illegal agreement. Data contract is exactly that, but for data, and therefore, and the way you think about it, the data governance team should be responsible for making sure contracts are in place and either auditing that they are or giving patents to make sure they happen by default. So, yeah, I’m like that. So, yeah, contracts, data modeling, they’re all foundational pieces. For me, I’d focus on conceptual and logical and treat high physical data modeling as just a execution problem. It’s cattle. I should be able to change my mind.

Mustafa: Yeah.

Shane: And the system should just absorb it. Right. I, I don’t think we’re there yet. Right. But I think that’s where we’re going.

Mustafa: I wanted to close this, but I have to throw one more question. As you mentioned, you’re writing a question writing a book on conceptual modeling. Because again, for me, the anchor is conceptual and somehow little bit logical as well. But conceptual is something which is, which is whole business up to conceptual and then goes to the management side. Right. When you talk about conceptual, are you saying that conceptual model of course need to have a business modeling information modeling all those techniques, taxonomies and all those things. Are you in your book by the way when your book is coming?

Shane: I don’t have a date. Juha has been much better at riding his parts of it that we need than I am. So I am dragging the chain and I’m trying to get that chain pulled up a lot faster. So, yeah, no, no idea when we’ll be done done on that one.

Mustafa: So now when you say conceptual modeling, are you considering all business modeling information modeling? All those things up to the conceptual modeling, are you saying that as a foundation?

Shane: So.

Mustafa: Last question.

Shane: So the book’s called modeling business Concepts. And it’s called that specifically for, for a bunch of reasons. And the, the. The goal of the book is to teach the steps that you need to create a concept model. Right. And so we’re kind of just working all the way through, like, what does that actually mean? And so we’ve identified 11 steps, right? So the first step is Define the domain that you’re going to work in. Like, don’t model the whole organization, try and figure out a domain. The next step is find a subject matter expert. Yeah. Find somebody in organization that knows how the organization works, and they’re going to give you the information you need to define those Concepts. And then we go through. And. It has the idea of where we Define the concepts and we Define the relationships, and then we identify where those relationships and Concepts are an event, a core business event. You know, customer orders product, the order is, is something that was really important to us. And then we have a visualization that we’re working on, like, how do we draw that map, that concept map, what does it look like? And what we’re doing is we kind of both aligning that map to what a conceptual data model map looks like with a few tweaks. Because that’s how we think about it, right? That’s how our expertise and experience has already done it. What’s interesting is I often use an event Matrix, kind of like a bus Matrix, but a version of it that kind of got from Lawrence core and has been stuff. And so I’m always looking at it going, yeah, we’re building this concept map, but actually maybe an event Matrix is another way of articulating that. And I think what we end up with just the core we’re teaching is the concept map, but we’ll use those event maps and those other visualizations as a different lens. The thing I struggle with is what is a business model look like? Yeah. I, I, because I always think business map, concept map, logical, physical, right? Like, that’s in my head. But I’ve struggled to find a business. I mean, the business canvas is kind of a business map, but nobody really treats it that way. And so, yeah, I’m very intrigued. If you can find somebody that goes, well, the thing we do before a concept map is the business map. What does it actually look like? That’d be, that’d be cool piece of content because I struggle to find a pattern or a patent template that makes sense to me. So I’m not sure I answered your question on that one because we’re still writing it, but it’s kind of a rep.

Mustafa: Because then you say conceptual, you are looking towards default conceptual because conceptual doesn’t come from the air. It has to come from business, as we said.

Shane: Lica. Right. Yes. Yes. So, so.

Mustafa: So. It has to come from business.

Shane: Yes. So when we talk about a concept map, it is a representation of the way the business behaves, not a representation of where the data is stored. Yes. So we should be able to do a concept map without ever looking at a source system. Either looking at a database.

Mustafa: Yes. Yeah. And just to answer you to find out, by the way, in my latest book, I am data media. I have added 37 data modeling techniques. So over there I go step by step. Maybe I can share with you. You already bought my previous book. Right?

Shane: Probably. I’ve got a whole bunch of books sitting left to me in paper, and I’ve got a whole folder of books I bought in PDF, and it’d be fair to say I have.

Mustafa: Yeah. So whoever. Yeah. So whoever have bought my previous book, I’m giving them free my latest book. So let me pass you my PDF of the latest book. Actually over there you can go into it, have a look. 37 techniques. So wherever conceptual before you can go and see all those techniques, what they do. I spend a lot of time because someone asked me how many typings are there. We have worked only five, six techniques. Normally we work three and a half. Star schema snowflake. Data vault. Logical physical. This is work. But actually when I really went into it from above, I discovered there are 37 data modeling technique either what most of them I didn’t know.

Shane: So, so you know how, you know how you said that you kind of went vibe, started vibe coding and you picked LinkedIn. Yeah. So it’s kind of, what I find is whenever a wave comes, right, the best way for me to understand the impact of a wave is to learn by doing. And so, of course, I always talk, used to talk about current code, don’t code, won’t code, right? Like, I, I tried to code in my early days. I was the worst coder there ever was. But I thought, well, okay, we got this new world. I’m going to go try and vibe code something. And so the, you know, the obvious candidate for me was I vibe coded the information product canvas. Yeah. So I vibe coded a small app, which means you can do it online. And then I published it as an open source thing. So you could download it and use it and hack and do whatever you want. And then because I’ve been doing a lot of work on this idea of context and context playing, I was like, oh, well, maybe we’d extend this out. So I vibe coded the business model canvas and lean campus. And then, of course, I couldn’t stop myself. And I started vibe coding the business event matrix. The data layered architecture checklist concept model, data dictionary, a glossary. But I purposely kept them all separate little apps, right? Because I’m testing this theory that these little micro apps, if you use them in your day job have value and store and capture context without you really realizing you’re doing it. So if you’ve got 37 different patterns. For the way we can diagram or canvas things, yeah, I’d love to work with you on it that we kind of just create little bespoke apps for these and publish them as open source. Because I think seeing by doing, like being able to fill out one of those things is a great learning mechanism for people to go, oh, you talk about a concept model and you’re talking about Concepts and relationships, and that’s it, right? No boxes and lines and a bit of car melody. I kind of get it now. Like, I got more questions, but I kind of played with something and it’s visual and I kind of get, I think I kind of get what you’re talking about. I think that’s a great way of rounding out the learning in our domain that we’ve, we’ve missed or lost. So, yeah, if you’re up for it once, you know, flip me your book. I’ll, I’ll have a look at it and then I’ll come back to you and go, it’s vibe code together. You have to move to claud, not, not cursor. But, yeah, and maybe we’ll do them and publish a couple and see what happens.

Mustafa: Same thing cloud is and cursor same thing both are lm behind it cursor is people like me who have lost.

Shane: Yeah.

Mustafa: Development like 15 20 years back. I can’t do coding in vs code. So if you are using clot, you need to have vs code as well. So I don’t know to go into there.

Shane: No, no, no, no. I use the claw code gooey. I I can’t, I can’t go into that terminal thing or VS code. No, no, I just, I, I go into something that looks like chat GPT and I go, like, I did one last night. I went and I said, I want a small app for metric trees. And then I went to bed. And then this morning I got a really bad app for metric trees. And now I’m yelling at it, telling it what it needs to fix. So, yeah, I mean, there’s. Interesting thing, right? That in theory, we can create this context layer for these apps, like the decisions and, and those kind of things that are being used when the app gets written back to your idea of building LinkedIn and, and hold the context of LinkedIn, not the physical data structure. And if we do that, then in theory, we can have Claude or codex or cursor or any of these things. Read that context of that app and then update it. But I’ve tried it. And, and again, I tried it a while ago. So what happens with the new stuff is you have to try everything every week because something bloody changes and it gets better. But, but what happened was the, the coding lms argued. Like codex would go, oh, Claude obviously wrote their, and it would refactor the code without making anything of extra value. And I kind of liken it to humans. I don’t know if you’ve ever seen it, but you walk into a brownfield site and, and the engineers that build all the ETL are gone. And the new people came in and, you know, they typically go, oh, that’s not the way you write that code. And they’ll refactor all the code. So it writes the way they wrote, normally write it, but it hasn’t changed anything. It doesn’t run faster. It doesn’t run safer and it hasn’t edited any new products. So, and so I find that llms are funnily enough doing that as well. So, yeah, we could give it a go. You know, I’ll give you a point cursor at it and see what it does.

Mustafa: I can share it with you. I created a training as well. It’s not very polished, but I’ll share with you. I have to polish it, but you can have a look all 37 techniques are in there I gave some examples. And I’m planning to get agent for each. I think I created agent for first four five techniques. It really, it was awesome because I talked, it narrated into a text. That text was in first prompt for the first modeling. That modeling created an output that one agent that output was an input for a next agent which can do the second modeling. So it was so awesome. And when I gave it to the cursor. Because all metadata was there. It was able to create the application very good, but it was such a big effort. Because someone has to create 37 agents. I didn’t have a time.

Shane: Yeah.

Mustafa: I learned in four or five that it’ll work. So maybe I’ll share with you.

Shane: Interesting thing on that is that lots of people are working now on skills for things like Claude, right, where all the expertise sits in it. And they’re really, really valuable. But there’s no feedback loop. And because we know that the lms are still probabilistic, not deterministic, which means that you ask it the same question 10 times, one of the times you will get a different answer. I’m still fundamentally believe we need a combination of the skills that do the work and visual techniques so somebody can understand the work that’s been done. So. And your scenario, a skill that I can ask it and it will create that model. And then a visualization of the model in a way that I can understand where that model is correct or not. And that, that becomes the problem. Like, how do you know it’s correct? Like, do you build another skill that tests it, what we call a judge mechanism? Like first skill writes it, second skill validates it. Yeah, we, we can’t, we’re still not really to move to a world where those things are right and therefore people without the expertise can just trust what it’s giving them every time. But, hey.

Mustafa: Meditation after the human. Why I feel it can be my own perspective. That why I was successful because I knew what I want. AI didn’t new because when I gave it a physical mode, it didn’t know. A physical model customer is just. Six or seven character world. It doesn’t know what is customer until I gave it a metadata that what is customer and customer can be different for different domains. For HR customer employee for HR is different. For a loan officer is a different. So I have to give a context. That is how it works. Never mind. I won’t take you a lot of time. We already one hour 35 minute. I will always chain. I think it was a great session. It become more of a data governance session. But I think data foundation has a big play of data governance as a quick play in data foundation. Because as we are discussing, you need to have right people at the right place. Because you don’t have a time to test and run and create something governance. I feel which still contradict with you is I feel that data governance team need to be separate. At least in this era. Because AI is doing their own shit until as right people do not know what they are doing. We are lost. That’s my prospect. Ive.

Shane: So I, if we say that data governance now behave like Auditors or risk managers or the CFO, then I agree with you. Yeah, that actually, it’s, it’s outside the people doing the work and they are governing that work is done right.

Mustafa: Yeah, that’s what I mean.

Shane: That’s a different pattern to what we see when we see data governance right now, in my view. Right now, we see some people trying to do their best, but they’re not empowered. Yeah. They can’t fire people. They can’t stop things happening. They can’t delete data.

Mustafa: Right.

Shane: They can’t shut down systems. They don’t have the power to govern. Like a council does if you’re building a house, like a CFO has if you’re paying for money, like a procurement team does if you’re buying something. So, yeah, I’m with you. We need to elevate governance to be governance and not management.

Mustafa: Yes.

Shane: And then, then I agree with you. Right.

Mustafa: Before you convince me. Okay, thanks. It was such a talking to you. I think we can have more sessions if you share. Take care. Have fun.

Shane: Thank you very much for having me. We’ll talk soon.

Mustafa: Yeah. Have a good week. End.

«oo»

Stakeholder - “Thats not what I wanted!”
Data Team - “But thats what you asked for!”

Struggling to gather data requirements and constantly hearing the conversation above?

Want to learn how to capture data and information requirements in a repeatable way so stakeholders love them and data teams can build from them, by using the Information Product Canvas.

Have I got the book for you!

Start your journey to a new Agile Data Way of Working.

Information Product Canvas

«oo»

I am now an uber Claude powered vibe coding app developer! (well not really)

Shagility — Thu, 09 Apr 2026 12:01:27 GMT

When we started AgileData seven years ago we made an initial decision that has served us well for the last seven years, until now.

We would never build the last mile tool, the BI front end.

The reasoning behind this principles was creating a BI tool or BI platform had too many table stakes from day one to have a chance of succedding.

Too many expected features, too much competition, too much time, effort and moolah to build something that could compete with the established players.

So we used third party tools. Looker Studio, Power BI, whatever the customer had already paid for. We focused on the hard bit underneath, the data platform, the data, the business context, the patterns, the scaffolding.

All bound with the core principles of being able to scale at will, and rin on the smell of a oily rag ( well based on the cost of oil before the latest middle eastern “adventure”)

That was the right call and has lasted us well for the last seven years.

Then vibe coding turned up and we revisited our thinking.

TL;DR
We are betting the BI platforms of the previous waves become legacy and “one shot BI apps” become the norm.

Bold statement right?

Let me give you some context (after all without Context all we have is data)

Why we think “One Shot BI Apps” are the new black

Up until now if an organisation wanted a bespoke Information Product / BI App that did exactly what a consumer needed, they had to pay an expert development team to build it and then maintain it.

That took time, effort and money.

So instead they compromised, they bought a BI platform, a general purpose tool that kind of did what everyone needed but never quite did what anyone actually wanted.

Stakeholder don’t actually want a dashboard with 24 areas and 47 filters they have to squint at to find the one number they care about. That is just what they have had to live with until now.

They want an app that answers their specific question or even better tells them the next best action. They want an app that works the way they think and the way they work, with nothing else getting in the way.

Data teams doen’t want to spend three iterations building a Power BI report that gets used twice and then someone asks for “just one more chart”. They want to deliver something that actually gets used, that drives proven organisational outcomes and value and most of all and makes somebody’s life better.

That sure as shit ain’t the dashboards of old.

Stakeholders and Data Teams want something that does one thing well, enables one specific job to be done, and gets out of the way.

The BI platform was always a compromise. It existed because bespoke was too slow and too expensive.

And that where vibe coding using tools like Claude Code have changed the promise.

They promise:

Low cost :: Making bespoke cheap as chips.
Speed to market :: Minutes and hours, not weeks and months.
Democratisation :: Anybody who can describe what they need can build it.
Cost :: Less than your annual BI platform licence, and your big server/services infrastructure costs by a long way.

With all these promises why would they keep compromising?

We have seen this movie before

We have seen promises like this before, promises that are early in the hype wave, and then eventually get grounded in reality

We have seen how every time we democratise access to something in the data domain, we eventually end up with sprawl.

The OLAP wave democratised access to data. What did we get? A mess of cubes nobody could find or trust.
The Tableau wave democratised access to dashboards. What did we get? Thousands of dashboards, most of them showing slightly different numbers for the same thing, or a slightly different number of data columns.
The dbt/modern data stack wave democratised access to data transformation. What did we get? 5,000 dbt models with no actual data model, anybody?

Now we are about to democratise access to building BI apps. And the sprawl problem is going to make the previous waves look like a gentle ripple.

I have written thoughts about some of these anti-patterns in the new “AI” wave before:

But what if you already had the scaffolding?

Luckily we have spent the last seven years building out a data platform that is both opinionated and has all the scaffolding that should (in theory) help manage this inevitable sprawl.

So we started by creating a simple templating pattern in our platform. A way to quickly build and deploy a “one shot BI app” using your favourite LLM tool. Claude, ChatGPT, whatever you prefer.

Then we let it loose with one of our talented early adopters Network patterns, to see what he would actually do with it.

We were amazed with what he started creating.

Next we used it to build a prototype for a customer who wanted us to upgrade the capabilities they had been using with us for the last few years.

They loved it.

So once more into the breach, my friends.

“Can’t code, won’t code, don’t code” tries vibe coding

One of my frequent sayings is

I “can’t code, won’t code, don’t code”.

But I wondered, with the latest tools like Claude Code and the latest models like Opus 4.6, is that still true?

I find I learn best by doing.

So I thought I would experiment with creating an app using Claude Code to see how the process would work for somebody like me. Somebody who understands the patterns and requirements inside out, but has spent decades successfully avoiding writing code.

Now I could have vibe coded a “one shot BI app” for a customer, or built something with public dummy data to experiment and learn.

But the Information Product Canvas was the obvious first cab off the rank.

I have been iterating that pattern template for over a decade. I know the 12 areas, how they relate to each other, what the user experience should feel like, and what the anti-patterns look like.

If Claude Code got something wrong, I would (hopefully) know immediately.

Plus I had “build an IPC app” gathering dust on my backlog for years. I never wanted to spend the moolah to pay somebody to build it.

Vibe coding supposedly changed the economics.

The result

One of my other common sayings (apart from #Whoot!) is

Sharing is Caring.

I published the resulting Information Product Canvas app as open source.

You can read about what it does and how to get it running over on the Information Product Canvas companion site:

The Standalone Information Product Canvas App

Do you prefer the feel and joy of reading a physical book…

a month ago · Shagility

And grab the code from GitHub:

https://github.com/AgileDataGuides/information-product-canvas

But of course I didn’t stop at one app

For those that know me know when I find something interesting, something that looks like a useful pattern, but I cant describe that pattern with clarity results in something I can’t leave well enough alone.

The Information Product Canvas app scratched one itch.

But it also made me realise how quickly I could experiment with ideas and templates that had been stuck on the backlog for years. Pattern Templates like the:

Business Event Matrix;
Concept Models;
Business Glossary;
Layered Data Architecture Checklist;
Data Dictionary;
Data Contracts;
Data Asset Catalog;

Ideas I couldn’t justify spending the time or money on before, but could now explore by whispering sweet nothings to Claude Code in the background for a few hours, while still doing other more important work.

So I have kept building. More standalone apps, all connected by a shared backend.

Some of them are getting close to useful. Some of them I still hate.

I will publish each of them as open source standalone apps as they get to a stage where I don’t hate them.

If you have been following my Context Plane experiments you will start to see how these pieces fit together.

https://agiledata.info/t/context-plane

If you want to help build them with me, just sing out and let me know, the more the merrier.

The bit I haven’t been sharing

As part of my Sharing is Caring mantra I realised I haven’t been sharing my journey as I learn the process and pros and cons of vibe coding something that needs to be actually used, as a person in the data domain who can’t code.

So as I keep working, off and on, on my vibe coding process and building out apps, I will post articles on what I have experienced and my thoughts around it.

What worked. What went sideways. What surprised me. What I learned about building software through conversation rather than writing code.

If you are a data person curious about vibe coding, or a builder wondering what it is like when someone who “can’t code, won’t code, don’t code” picks up Claude Code and starts whispering sweet nothings to it, hopefully the series will be useful.

If not, maybe turn off substack notifications for this site for a little while ;-)

Its not about your shiny tools, its about the value you deliver using them

Shagility — Fri, 27 Feb 2026 11:45:47 GMT

I caught up with a data friend for a virtual coffee this week, they were lamenting the whole “single source of truth” they were experiencing with an org they were working with.

It was the same pattern talks about in this draft book chapter:

Practical Data Modeling

What Data Modeling Is and Is Not

Here’s the revision of Chapter Two for Mixed Model Arts, where I discuss various definitions of data modeling and bring it into the present day. We are no longer modeling just for humans, but modeling for humans AND machines…

3 months ago · 32 likes · 3 comments · Joe Reis

A quote from that chapter:

<-o->

“I’d just been hired as a consultant for a mid-sized e-commerce company that was hemorrhaging money. Their CEO pulled me aside within the first hour: “Joe, our inventory system says we have 50,000 units in stock. Our warehouse says 12,000. Our website shows customers that they can buy things that don’t exist. We had to refund $400K last month alone.”

“I spent the next three days spelunking through their systems. What I found was a horror show. They had an ‘orders’ table with 500 columns. Customer data lived in six different databases, none of which agreed on what a “customer” was. The product catalog was a single enormous spreadsheet that someone manually uploaded to the database every Friday afternoon. Date fields were stored as strings. Some prices included tax, some didn’t, and nobody could tell you which was which.”

<-oo->

My data friend was telling me about the time and money the orgs CDO had spent implementing a “Modern Data Platform” and how they were now being asked to present what “outcomes” had been delivered for that expenditure.

Unfortunately when my data friend had to go and get a single number for the equivalent of Joes “Count of Stock”, they found multiple systems that had different counts.

And when asking various Subject Matter Experts (SME) which count could be trusted they all gave different answers.

But the SME’s were all aligned when they said the one count they didn’t trust was the count in the new data platform.

I joked that the CDO should probably google (or perplexity) the three envelope joke about now.

But realistically its not a joke. That is shareholders money that has been spent, its peoples jobs that will probably be impacted as a result of cost that seemed to have no value.

So lets say it again .....

Its not about the tools you use.

Its about the value you deliver using those tools.

End of rant and here is a suggestion.

If you are investing in new tools, or a new shiny data platform do three simple extra steps.

Define an Information Product using the Information Product Canvas

You can learn about it for free here:
Information Product Canvas
Information Product Canvas a pattern template, to quickly discover and capture, data and information requirements, in a repeatable way, so stakeholders love them and data teams can build from them
By Shagility
Take course on how to easily and quickly identify the value of the information that the Information Product will deliver.

You can find his course here: https://maven.com/nick-zervoudis/dpm-value-course
Have you data team build the identified Information Product at the same time they build your shiny new Data Platform.

And then use the number from #2 above to start justifying the value the new data platform is delivering.

What is the new Moat in the new "AI" Vibe Coding" world

Shagility — Wed, 25 Feb 2026 11:08:33 GMT

The moats of old are disappearing and im ok with that.

The thing I love about experimentation is it helps me coalesce some divergent ideas that have been floating in my head for a while, into some semblance of order.

A bit like writing does.

I wanted to get a handle on the latest state of “vibe coding” so decided to experiment with building an app using Claude Code and Opus 4.6.

I picked vibe coding an app for the Information Product Canvas, its an app I have wanted built for a while, but the cost to build it the old way never matched the value I found people were willing to pay for it.

Vibe coding in theory reduced the cost, the experiment was would it?

You can see the results of a few iterations of the canvas:

Plus a side experiment into the world of the Context Plane (couldn’t help myself).

But that not the point of this post.

Nick Pinfold is experimenting with his teams Agile Data Ways of Working and freely sharing his journey via LinkedIn comments.

In this comment

https://www.linkedin.com/feed/update/urn:li:ugcPost:7432018978505617408?commentUrn=urn%3Ali%3Acomment%3A%28ugcPost%3A7432018978505617408%2C7432267116197765120%29&dashCommentUrn=urn%3Ali%3Afsd_comment%3A%287432267116197765120%2Curn%3Ali%3AugcPost%3A7432018978505617408%29

He talks about how he is creating a Streamlit app that allows him to capture the IPC content as Context and use it to assist with the next steps in their Information Factory.

posted a comment on this Substack post:

On how she is seeing “datapacks” that provide the content of a book in a way it can be easily used in an LLM as having some value.

Both of these show that there is value in writing books and creating apps, but that the typical moat of both of those things has changed.

Nick can vibe code a IPC app as fast as I can, if not faster

Anna can take the ePUB version of my book and use it in a LLM as fast as I can. If she pays more for tokens and the latest models than I do, she can do it faster and better than I can.

So effort and expertise are no longer the moat.

Given I have always said I “Can’t Code, Don’t Code, Won’t Code” but now I can now create an App but just asking questions, Im pretty sure Effort and Expertise is not the moat it was anymore either.

But maybe experience is.

To create my app in a way that meant it was actually useful, I had to have experience:

experience using the canvas
experience working with multiple data teams on the problem the IPC solves
experience using apps to know what UX features were needed
experience to know that I needed to add Google Auth to provide an easy login
experience to know not to store any API keys in a way that were public
experience to know that I wanted to use Google Spanner as the backend data repository and Svelte as the front end language
experience to know I wanted a API layer between the backend data repository and the front end app
experience to know …..

I don’t have an answer to what the new moat is, but I have more clarity on what it isn’t after these experiments.

Fact-based modelling patterns with Marco Wobben

Shagility — Sat, 14 Feb 2026 10:46:07 GMT

Join Shane Gibson as he chats with Marco Wobben about the patterns within Fact-based modeling.

Listen
View MindMap
Read AI Summary
Read Transcript

Listen

Listen on all good podcast hosts or over at:

https://podcast.agiledata.io/e/fact-based-modelling-patterns-with-marco-wobben-episode-81/

Subscribe: Apple Podcast | Spotify | Google Podcast | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser | Deezer | Podcast Addict |

You can get in touch with Marco via LinkedIn or over at https://casetalk.com

Tired of vague data requests and endless requirement meetings? The Information Product Canvas helps you get clarity in 30 minutes or less?

Google NotebookLM Mindmap

Google NoteBookLM Briefing

Executive Summary

This document synthesizes key insights from a discussion between Shane Gibson and Marco Wobben regarding Fact-Based Modeling —also known as Fact-Oriented Modeling. The central premise is that modern data modeling has become a “lost art,” leading to significant “business debt” where organizations lose the context and meaning behind their data due to silos and rapid staff turnover.

The core solution presented is Fact-Based Modeling, a methodology that grounds abstract technical requirements in “administrative reality” by combining linguistic terms with actual data examples. By focusing on how stakeholders communicate (Information Modeling) rather than just how systems store data (Data Modeling), Fact-Based Modeling allows organizations to bridge the gap between business subject matter experts (SMEs) and technical implementations. This approach not only ensures more accurate system design but also provides the necessary semantic grounding for emerging technologies like Large Language Models (LLMs).

--------------------------------------------------------------------------------

The Crisis of Lost Context: Technical and Business Debt

The current state of data management is characterized by a widening gap between what is stored in systems and what those records mean to the business.

Evaporation of Knowledge: Senior experts with decades of organizational history are retiring or leaving, and the average job tenure (four to six years) is too short to maintain deep context.
Business Debt: This is the cumulative loss of meaning within an organization. When systems are built without documenting the “story” behind the data, the original business intent is lost, leaving IT to guess the context of legacy records.
The Context Gap: Technical optimization (how data is stored) often overrides business representation (how data is used). This leads to “tribal wars” where different departments use the same terms (e.g., “inventory” or “customer”) to mean entirely different things based on their specific departmental needs.

--------------------------------------------------------------------------------

Defining Fact-Based Modeling

Fact-Based Modeling is a methodology developed to capture domain knowledge by focusing on “facts”—statements about the business that are agreed upon as true within a specific context.

Core Components of the Fact-Based Modeling Approach

Grounding in Examples: Unlike traditional modeling that looks at abstract entities and attributes, Fact-Based Modeling uses “data by example.” Instead of discussing a “Customer” entity, a modeler uses a statement like: “Customer 123 buys Product XYZ.”
Binding Term and Fact: By combining the linguistic term with a concrete fact, modelers can identify misalignments quickly. For instance, seeing that one system identifies a customer by an email and another by a numeric ID reveals a transformation problem that abstract modeling might miss.
Information Modeling vs. Data Modeling:
- Information Modeling: Focuses on how humans communicate about data to reach alignment.
- Data Modeling: Focuses on technical storage, structures, and optimization.
- The Distinction: Information modeling is the “primary citizen,” while technical artifacts (SQL, schemas) are secondary outputs derived from it.

--------------------------------------------------------------------------------

Methodology: The Process of Fact-Based Modeling

Fact-Based Modeling follows a specific logical flow to ensure that complexity is simplified without losing essential nuances.

Scope the Domain: Identify the specific problem area (e.g., Sales, Emergency Room, Tax).
Gather Data Stories: Interview SMEs to collect verbalizations of how they describe their work.
Identify Business Constraints: Use interactive questioning to find the “rules” of the data. (e.g., “Can a citizen be registered in more than one municipality at once?”)
Identify Exceptions: Use the data examples to flush out the “edge cases” that SMEs often forget until they see a specific record.
Alignment through Generalized Objects: When different departments use different identifiers for the same concept (e.g., Name vs. Email), Fact-Based Modeling uses “generalized object types” to link these different views into a unified communication framework.

--------------------------------------------------------------------------------

Strategic Value and Modern Application

1. Automation and Efficiency

Fact-Based Modeling allows for a “context-first” implementation. Because the model is rich in semantics and constraints, tools can automatically generate:

SQL for database creation.
Data Vault or normalized models.
Database views that represent the original user stories.
Test data derived directly from the interviews.

2. Grounding AI and LLMs

LLMs are proficient at generating “fabricated stories” but lack organizational context. Fact-Based Modeling provides the “ground truth” needed to keep AI outputs accurate. By feeding an LLM the terms, definitions, facts, and business constraints from a fact-based model, the AI can perform tasks with a much higher degree of reliability.

3. Avoiding the “Generic Model” Trap

The document highlights the failure of massive, pre-built industry models (e.g., the IBM Banking Model). These models often fail because organizations do not know their own “edge” or specific context. Fact-Based Modeling allows a company to capture its unique business logic rather than trying to fit into a generic template that ignores their specific reality.

--------------------------------------------------------------------------------

Notable Insights and Quotes

On Complexity and Simplicity: “If the end product is presented and everybody goes: ‘Wow, is this it? Did it really take you that long... I could have done this,’ then I succeeded in making something very complicated very simple to understand.” — Marco Wobben.

On the collaborative nature of solving data ambiguity: “It is somehow a team effort to slay this beast of miscommunication until everybody agrees and understands each other. ... It’s all about, working together and trying to fight it. What are we not seeing? What are we missing? How do we tackle this? And a lot of that is just human interaction.” — Marco Wobben.

On the Definition of a Fact: “A fact is a piece of data that’s physically represented somewhere... I can point to it. It has been created. I’m not inferring it. It is something that is factually there.” — Shane Gibson.

On Party Entity Data Models: “The most expensive part of our systems is the humans and [understanding] that context... as soon as you design a system with ‘thing as a thing’ and that context lives nowhere else, I now have to spend a massive amount of expensive time trying to understand what the hell [it is].” — Shane Gibson.

On the concept of “business debt” created by rapid technological change: “This is the paradox where business wants to have changed faster. And it ruined the party by saying, we can deliver faster with this new latest tech, but neither party realized what they were losing along the way. So it’s technical debt, it’s business debt.” — Marco Wobben.

--------------------------------------------------------------------------------

Conclusion

Fact-Based Modeling serves as a bridge between the human understanding of business processes and the technical requirements of data storage. By prioritizing the “authentic story” of the business and grounding it in real-world data examples, organizations can mitigate the risks of technical and business debt, ensuring that their data remains a usable, understood asset even as technology and personnel change.

Tired of vague data requests and endless requirement meetings? The Information Product Canvas helps you get clarity in 30 minutes or less?

Transcript

Shane: Welcome to the Agile Data Podcast. I’m Shane Gibson.

Marco: And I’m Marco Wobben.

Shane: Hey, Marco. Thank you for coming on the show. Today we’re gonna talk about a thing called fact-based modeling, but before we do that, why don’t you give the audience a bit of background about yourself?

Marco: Ah, yes. Thank you. Thanks for having me. Yeah, background a lot of background there. I’ve been around a few decades. I first fell in love with computers when I was still in secondary school. That got me hooked when somebody showed me the break key on a keyboard and we could stop games mid play, change the code and resume.

And that was like, oh, this is magic. I need to figure out how to make this my toolbox. And I enjoyed making software, hacking software working with computers ever since. And. From getting a job onboarding people on Microsoft Windows and Office. Back in the day, I decided to just chase my own career, quit the job and started doing entrepreneur work, made custom software from design to end product for a number of startups. And then somewhere early two thousands, a professor knocked on our door and said, we have this source code of a modeling tool. And the kids that graduated on it, with, they took a different path and we need somebody to maintain it. And ever since early 2000, I’ve been working on fact-based modeling that I had to learn from the inside out.

So that’s where I am now. I’m being considered the expert currently. ‘Cause a lot of professors retired and the young people haven’t caught on for it yet, just and here I am talking about fact-based modeling or fact oriented modeling, if you will.

Shane: Yeah, it’s interesting. We before we started, we talked briefly about the fact that data modeling’s become a lost art. And actually, I think it’s coming back now with all the AI focus, it seems that data modeling is a term I’m seeing used a lot. But, if you think about your career path, that idea that you could start out with games as a way of introducing yourself to, to computers in the early days.

I was like, you I started out. We had a couple of computers at school. The old I think they went green. I think they were amber screens back then. And yeah, again, I got hooked by the games. And I wondered whether that’s the thing that’s been missing is gamifying data modeling.

Like actually making it exciting is, it’s probably the missing thing, right? Is if there was a game that you just happened to date, a model when you did it. Maybe that’s what we needed to make what we do sticky with with people that are coming up in their career.

Marco: that’s an interesting take on it. it’s interesting looking at my users, I’ve been maintaining this information modeling tool for years now, is that there seems to be more. Interest from people that are curious and like doing things right and talk about things. And this is, I can see that in, in a lot of gaming communities where, you know, in the old days it was like you play your game single player with this computer, right?

But gamification nowadays is so much more you can team up and you none of that stuff was there in the early years. Even graphics were not there yet. The communication aspects right nowadays is more and more prevalent and important and it’s. If you look back, what data modeling really is to get the technical requirements of what people actually needed the computer to do in store and how to manage it.

Seeing that come back a little I dunno if gamification would help, but there’s definitely more openness to let’s talk about it. And you can see it a little bit in the agile phenomena where there’s a lot of standups stakeholders, product owners, and everybody starts talking and communicating with each other.

So that’s definitely on the rise. I’ve seen a few data modeling efforts that actually try to gamify it and it’s like a, they have this data modeling tool and it has flashing text and animated tables and, I’m not sure if that’s really the end goal, but I can see how it might help.

Shane: Yeah. I was thinking more about gamification as in the adrenaline rush when you are successful in solving a problem not the flashing things. Because earlier in my career, I worked for a software startup in the accounting space called X. One of the reasons that they became successful was they had this gamification of bank reconciliation.

So pretty much you had bank transactions on the left and your invoices and all that on the right. And whenever a transaction came up on your bank statement, you pretty much dragged it and matched it to the right. And then that road disappeared and it was a very simple gamification process.

But there’s this adrenaline rush going in and seeing your bank rec with 50 transactions you haven’t reconciled, and just going bing, bang Bosch, and it becomes clear. And, I don’t, it didn’t have confetti. Actually that was one of the big arguments at the time was should it have confetti con?

Marco: more like a Tetris road disappearing from your screen.

Shane: Yeah. Yeah, exactly. And so you sit there going, maybe that’s it. So as I think about it more, when I’m conceptually modeling the adrenaline rush is when I create a map that I can show to a stakeholder and they just nod and go, yeah, that’s business reality. I get it. Yeah. That’s how we work.

When I logically model, it’s this idea of yeah, I can take that conceptual model and I can slam it into the modeling Pattern that I use and it makes sense. And then when I take that logical model and I make a physical model and the cloud database actually takes it and I can actually query it fast and it doesn’t cost me a fortune, and any question that I get asked can be answered with that data.

There’s that adrenaline rush, right? That gamification of each of those steps adding value to my life or somebody else’s. I don’t dunno, I just, I haven’t thought about it that way until you mentioned that you started out hacking games.

Marco: That is, it definitely is true for me because, as I add features that support user functionality and all that, and it’s like there’s definitely a rush when I see people use it and they go oh, this is handy. This is practical. This, makes my work easier. Then I get the adrenaline rush. Definitely on the modeling part itself. It usually goes through long cycles of deep talks about the subject matter at hand, which, there’s definitely an adrenaline rush, but not necessarily always the good ones. As a, as an example that I used a lot and some other authors put it in their book too, is I had a long interview with the subject matter expert and it took us about two hours to figure out one requirement, and there was a lot of talk on the type level that created confusion because, yeah, what is a customer really? So I have to get through that. And in the end there was a little bit of an euphoria, which is, the adrenaline in rush, if you will, where we finally nailed it. The subject matter expert re replied, somewhat baffled and in disbelief, and he says, I had no idea my work was that complicated. And then being able to write it down in a way that everybody suddenly understands. That’s definitely a moment of adrenaline and rush, if you will. It takes two hours of hard work. It takes a lot of interviews, a lot of digging. And then finally when you reach that point, and it’s like one of, I remember a quote in a book I forgot which one it was, but I the quote really spoke to me and he this man was speaking about information modeling on a data modeling level.

And he said if the end product is presented and everybody goes. Wow, is this it? Did it really take you that long and work so hard to just present this? I could have done this, and then his reply was then I succeeded in making something very complicated. Very simple to understand.

Shane: And that’s the key is to take that complexity and describe it in a simple way where everybody nods and either agrees or disagrees. I remember one where we literally spent three months with an organization trying to get the definition of active subscription. And the problem was we had three business units.

Effectively three domains that all had different definitions, but couldn’t agree. They actually had different definitions. It wasn’t around the model itself, it was around the plain language description of that term where either we added a term in front of it, marketing, active subscription, finance, active description.

So we were clear they two different definitions or they actually agreed active description was described in this way. And that human engagement was where the time was spent not creating a map with nodes and links on it. Not creating the database, but without actually getting to a stage that we ever agreed or disagreed with that term.

There was minimal value carrying on because we would present a number that people don’t agree with because the definition’s wrong, not the number.

Marco: in the years of information modeling that I, I call it information modeling, but it’s really just a fact-based modeling underneath as well. But I’ve encountered so many synonyms or harmoniums or it’s just, people just completely get confused and it becomes almost like a battle.

But here’s the thing, and I think you explained that as well. It is somehow a team effort to slay this beast of miscommunication until everybody agrees and understands each other. I just recently started watching the latest series on Stranger Things that it’s we have to team up to fight this monster from the underworld, which is, we can’t see it. We know it’s there. And it’s like, how do you fight it? And and it’s all about, working together and trying to figure it out. What are we not seeing? What are we missing? How do we tackle this? And a lot of that is just human interaction.

Shane: And finding ways of taking that expertise that we have that ability to take complexity in a business organization and try and create a map that has simplicity that we can all share. That is a skill, And it’s how do we take other people on the journey without going into a room for six months on our own or creating really complex ERD diagrams with many to many crow’s feet that, few people understand, And terminology is really important. And so it’s interesting that you talk about information modeling and then you also talk about fact-based modeling. ‘cause as soon as I heard of fact-based modeling, I naturally go to dimensional modeling and star schemas because that is where I first heard a definition of the term fact.

And my understanding is you are talking about information modeling rather than anything to do with dimensional or star schemas. Is that correct?

Marco: Yes. That’s funny ‘cause your first take on the word fact is how I met my wife at a data modeling conference in Portland in the USA. She was like, oh, fact-based modeling I’m doing something with data warehousing. I should go to that class to listen what this man has to say.

And it was just not the same fact. So even there, even in it, we have confusion of words, but the word fact really boils down to something that is maybe a little older even where database records really store facts as they happen in our administrative reality as I call it nowadays.

Because there’s a lot of the single point of truth. We need to get the truth out there, the reality and all of those things. But there’s something seriously flawed with that, is that we all perceive from our own bias and subjective reality, the world out there. So there is no such thing as truth. But when you store data and you consider that to be true in your world, then you can state that as effect. Effect as in, I’m writing this down and me and my colleagues agree on it. And so that’s where the terminology came from. Nowadays we trying to figure out, maybe we should call those claims instead because we all say something and we all think it’s true.

And sometimes, especially in data warehousing you collect data from different source systems. Yeah, but you can’t just say that they all state facts because some might be alternate facts. So let’s put it as all the source systems claim a certain statement about this is what happened.

Shane: it’s interesting, ‘cause we’re talking again beforehand about how we’ve both been in the domain for quite a while, but we’ve never really crossed paths. And I’ve heard your name a lot, but I’ve never really read a lot of your stuff. And one of the things I did do, I had to train a new team moving to the data space.

And so I was trying to describe the difference between facts, measures, and metrics. And what he ended up coming up with is a language definitions that I used and the definition I used was a fact is a piece of data that’s physically represented somewhere. So if I go into a database and I see a number.

That is a fact. If I go into a spreadsheet and I see a number or a piece of text, that is a fact because I can prove it existed. I can say, that factually it’s there. I can point to it. It has been created. I’m not inferring it. It is something that is factually there for me. And that’s kinda why I use fact.

And the reason I raised information modeling versus dimensional modeling is ‘cause as soon as I use that term fact, anybody in the data domain goes, oh, you’re talking about a fact table. And I’m like, no. And then for me, I defined, measure as a formula, some of this, that kind of thing.

And then a metric is a complex formula. So this over this based on that at this point in time. And so for me, I didn’t mind whether people disagreed with my definitions as long as they gave me a different definition. But that’s the three terms that I used that seemed to get clarity and understanding when I was talking to people who weren’t data experts.

So yeah, I go back to the true definition of that term. Fact is not a fact table and a dimensional model. And, maybe yeah, should we move to claim or should we just bring back the true definition of that term, that’s one of the problems we have in our domain is we have what we call pedantic semantic arguments about the the most non-interesting things for.

Marco: We are not gonna solve that because there’s so much when I do actually information modeling and we can come up with the word, I’ve used it in, in, in different environments, but we can all agree upon what the definition is for the word inventory. It’s the amount of things that we have and offer certain article, but then you go into different departments.

Sale comes up with three because they already sold a few. Purchasing says, we have eight because they already ordered a few. And then you go into the warehouse and the guy goes but I only have one on the shelf. The, what the hell is going on? No, even though everybody agrees they have different data. And as soon as you, you and I would speak about facts, then, I could say that the customer buys a product and we agree and we call that a fact, but it has nothing to do with the data at all. So the word fact itself is like inventory, is location is like customers that you can apply it to anything and it doesn’t mean anything. So getting too hung up on it. Is very tricky because then you will start tribal wars because no. This is what the definition for a fact is. But the reality is that the word fact the linguistical part of it, the term of it is used in different contexts. So if you wanna use it within your dimensional world, it’s fine. I’m not gonna argue with that. Is similar to calling something red, we will find it in different environments. I’m looking at a book that has a red cover. You look at the fire truck is also red. It’s, oh, we’re all good.

Shane: Unless you’re in a country where the firetrucks green. And yeah, it’s interesting ‘cause Remco Broekmans talks about the example he has of definition of a bed in a hospital. And where one group basically said, it is the metal frame that the patient sleeps on. And another part of the organization said, nah, hold on.

It’s the room where the patient’s located. And so if I looked at the data, I would see one was probably two meters by two and a half meters and the other one was probably five meters by five meters. And then I could say the fact that this beard is five meters by five meters and has no wheels confuses me because I’d expect it to be two by three with wheels.

Maybe that data’s gonna gimme a hint that the definitions are different. So getting into that. can you just gimme a helicopter view of how fact-based modeling works.

Marco: You already gave me some beautiful examples is that to distinguish the things. You also have to look at the data and what I see happening in the data modeling space is usually. The data is not that relevant. We’re all looking at tables, entities, classes and what kind of attributes they have, what columns need to go in and what are the relationships or foreign keys and all.

So there’s a lot of technical views on it, and some of it may be guided by the data at hand. But the data itself is a secondary citizen. And as I just stated with the example of inventory, is it’s only by looking at the data that you start realizing, wait a minute, we’re all calling this inventory, but we’re seeing different things.

So there must be a distinction. And instead of having, this, I call it these tribal wars, these linguistical fights about no, that’s not what I said. This is what it means. And all of that. I call that. Type level arguments, type level discussions. It’s abstract in a way. So what factory ended modeling does is two things, is first, how do I talk about my data? And I use the data in the expressions, so I’m grounding it. I’m not talking about customer buys product. I’m saying customer with customer number 1, 2, 3 buys the product X, Y, z. Where both 1, 2, 3, and x, YZ is actual data, is real examples to ground the discussion.

Because, if I use inventory and I don’t specify that, I look at it in a certain way by giving the data, we will not discover that we might have a difference of perception there. So why the factor oriented modeling or fact-based modeling is that really we need to figure out how. I talk about it, how you talk about it and how we can talk about it and ground it with actual facts, this actual data it’s not enough to say the customer buys the product because you and I would agree, but it’s only by giving the actual data examples that we might start realizing, wait a minute, I have numbers to identify my customers, and you might have email addresses to identify your customers, so there’s something else going on. So it is really digging that one level deeper into it, and not letting that go because in the fact-based modeling, we keep tying the examples data with the language, with the fact types. So it’s always that package so that we can at any point, transform our information models into any kind of artifact, but still show the example data to accompany the definitions, to accompany the structures and to illustrate that this comes with a specific data use. And I think that’s the biggest difference from what you look at data modeling or dimensional modeling or data vault modeling is that it’s all on a type level and it’s usually geared towards how do we structure the storage of data. And it’s less about how do you and I figure out how we talk about the data and how do we quantify with data that we’re talking the same thing?

Shane: What I found interesting was so I’ve used the who does what Pattern a lot, And I think, I probably got it from Lawrence Coors Beam stuff. ‘cause that was some of the stuff I read earlier in my career. And I’ll often talk about customer orders, product from employee in store.

And then I’ve also used data by example a lot. And again, I can’t remember where I found that patent, but I found it valuable, so Bob purchased three scoops of chocolate ice cream from Jane at 1 0 6 High Street in Swindon. And what interests me about the fact-based modeling when I had a quick look at some of your presentations, is it seems like you are combining both those patterns.

So you are combining both the term and the fact into that data story. So you are saying, customer Bob purchased product, chocolate, ice cream from employee Jane at store 1 0 6 High Street.

Is that what you’re doing? You are combining the term and the fact together to give much more context around that business reality to then help you in the rest of the work.

Marco: yeah, definitely. It’s very much that approach and of course tools give you different functions to, to do it in a different way. But the theory that was even developed in, mid seventies and continue to develop all the way up to the end of the nineties was really about how can we write it down in a way that non-technical people can actually read what it means.

And in the interviewing phase, in the workshops, in the explorative areas, is that, as I’ve given the examples earlier is that it’s not enough to just say, okay, we have a customer and we need a data system for that. It’s like we needed to also figure out how you talk about it.

And I think what is an increasingly more difficult problem is that where in the seventies, eighties, even nineties a lot of systems were built for a specific purpose, for a specific department, for a specific system that everybody knew the context of. So now if I’m doing a customer registration, then I’m just doing just that for my department within my group of employees.

And we all know the context. So everybody understands that if I put customer there, they all know what customer is. Now with the increase of it is that. It added a computer on every department, in every system, But they never integrated. So this is where data warehousing came in. But because it was very high context, implied the data warehousing, the BI team now had to figure out, okay, what if we pull the data?

What does it even mean? But we’re really trying to rediscover the context of where it is used. The human aspects of understanding need to be reverse engineered on top of the data structure. And this especially goes for what we’re currently seeing with all the LLMs. Yes, it can give great semantics or storylines, but does it really understand the context?

So this is another gap. And what the fact oriented modeling approaches is that we try to interview and capture the knowledge of the domain and subject matter experts. With all the context, with their language, with their examples and try to line that up with the data, the language and the meaning of some other department that uses a different system. And not just to be able to talk about that specific system, but to also align the communication across departments, across contexts. That’s really what got lost. If you have individual silos that you want to bring together into a data warehouse solution, that the missing piece is the original and authentic communication about their systems in the first place. And then the factor oriented approach allows those subject matter experts to talk about it in a way that they understand and can be verified by colleagues and can be transferred across. Context. And I think that’s the real added value that when systems were built within a specific context, the value of that was not really seen because it served the context.

Everybody knew that. where you see now an increasing in interest in, okay, we need to get back to our ontology, we need to get back to our semantics. We need to get back to meaning, we have to get back to information. So all of that was there and it’s still there if you don’t throw away the baby with the bath water.

Shane: I agree. I, we go through waves, so we see a wave right now outside the AI wave. We see a wave of bi me metric layers. This idea of having a layer where you can define a metric and then regardless of which system, the data’s coming from or you are hitting that same metric’s been applied.

And I go well, you know, years ago in the old BI tools we had end user layer or a universe. We’ve had that Pattern and then we lost it, and then we get it back. I’m also with you in terms of the way systems have evolved. We’ve gone from a mainframe where we had one system where there was a term called customer and there was one fact and that was it.

And then we went to client Server N Tier, and we ended up with seven. And then software as a service, 50 to a hundred. And with the AI wave now and Gen ai, we’re gonna see these one shot apps. We’re gonna see thousands of things that have created for one use case get stood up really quickly, and then probably disposed of.

And so this ability to have a language where regardless of the system and how many we have, we get shared context is really important.

And that example that you used, I just wanna come back to that because I hadn’t thought about it that way. So if I said customer orders, product. I have a thousand systems that involve a customer, an order, and a product.

I have absolutely no context whether they’re defining the same things the same way.

But if I use this data by example, if I say customer, Bob, ordered product, and then in another system I see customer bob@gmail.com ordered product, and in another system I see customer 1, 2, 3, ordered product, and in another system, customer A, B, C, ordered product.

What I know now based on Pattern recognition, is that I have a problem with the unique identifier of customer.

Like I can tell that in 10 seconds by seeing those four data examples and it’s something I know I need to solve.

Because to mash that all up, I need some form of conformity, right?

Some form of either shared identifier or a way of mapping it. And so I can see by this combining this term and this fact together, it gives me the richness to understand the problems I need to solve much quicker than any of the other techniques where they’re separated.

Marco: That’s true and it’s, you’re still some somewhat catching an optimal path because you already assumed in this example that all those systems had something called customer. And in one system it’s probably CST. Then you have to combine it with SAP, where it has X 3 0 4 as a table name and then you have a very abstract table called persons. So you can see where this is going because all of those systems, the storage and the structuring of the data in that system served only one purpose. Optimizing the IT part of it, the IT end product presented the data in a way that the business wants to work with it and wants to see it, but it doesn’t represent how it is stored. The storage and the management of the physical data is technically optimized. It is not with the business representation in mind necessarily. And this is what a lot of people in the data space have encountered too. It’s okay, now I have 200 source systems and I need to figure out what is what.

So is this email address, does that indeed correspond with my customer? In the other system where it’s identified with 1, 2, 3, is that even the same customer? Is it a customer in the first place? So there is so much not just technical debt for systems that are undocumented or not well documented or behind in documentation, but also a, what I increasingly call the business debt is that nobody knows what that meant to the business anymore. And this is the paradox where business wants to have changed faster. And it ruined the party by saying, we can deliver faster with this new latest tech, but neither party realized what they were losing along the way. So it’s technical debt, it’s business debt. In governments it’s even worse because there’s a massive gap between the legal articles made up by politicians full of compromises and loopholes to the actual systems used by government bodies.

And, now we changed the law. Which system did we need to change or vice versa? We’re looking at data here but we have no idea if we’re even allowed to have this data or even be able to look at it because we don’t know the legal articles with it. So everything in it scaled up in the past 30 years so quickly. It totally got outta control in a way where the next tool’s not gonna solve it. An LLM is wonderful, magical stuff, but it’s not gonna solve the real thinking issues. We can do data profiling because we were still not sure if we got the context right. We can do LLM, but we are still not sure if the relationships are correct. So there’s still that gap of knowledge that we lost and somehow need to reintroduce to make things really work.

Shane: And actually that’s interesting around that organizational context and losing that knowledge. Because if I think about it, if I was walking into a new organization and I wanted to understand the context of that organization, my natural technique was to find somebody who’d been there for a long time as a subject matter expert, that person that’d been there for 20 years because they’ve got the stories of how that context happened and why it happened.

And so moving to the uk I had to set up bank accounts. And so I went, and there’s a whole reason that I needed to do a UK domicile bank rather than one of the newer, easier to deal with banks. And so I went to one of the main banks, I created a personal bank account.

Took a while. All good. And then I went back to that. ‘cause now I’m a customer of being identified. In theory. Everything is easy. And I tried to create a business bank account and it forced me to create a new identity. I had to go through the whole identity process, even though I used the same email address, which was my form of identification, my identifier, I had to go and revalidate myself that I have a same address, same passport number.

And you sit back as a customer and you go, that’s just crazy bollocks. And then I talked to somebody that had worked for that bank, that subject matter expert who’d been there for a while. And they said, yeah, but you gotta understand that was two different banks, two different systems, and they haven’t been merged.

And therefore you may think you’ll be dealing with the same organization, but you’re not really. And I’m like, yeah, actually. Okay, that makes sense. Now why? Why did I not understand that working in data so often, but if we go back to that example of term, in fact, so yes, if the term changes, so now I see a customer and I see person and I see prospect and I see X 2 0 3.

If the fact is the same, if every one of those is customer Bob at Gmail X 2 0 3, Bob at Gmail, prospect Bob at Gmail, again, I’m getting more context, I’m not getting an answer, but I’m getting more information that can help me understand a problem to be solved. And as we know with data, there’s so many problems to be solved.

It’s such a complex space. When we hit reality of organizations, the way they work, the terms they use, the systems they use, the way they create and store data. But this idea of binding term and facts. Gives me some more hints because now I can say they’re all the same email address, are they the same term?

And then somebody will say actually no, when you see prospect and customer, it is a different rule and then somebody else will go. But you do know that they can change their email address whenever they feel like it. And you’re like, yep, seen that Pattern before. Okay, so it’s an identifier, it’s a unique identifier, but it’s not a consistent or persistent identifier.

There’s all these patterns and data that we know are gonna hit us, but by binding that term, in that fact, I get some hints at the beginning across multiple subject matter experts. So I can see real value in, in that part of the process early. And so talk me through that, You drop into a new organization.

You wanna start off with fact-based modeling. How do you do it? What do you actually do?

Marco: It’s a funny question. I, a lot of people ask me that, how do you start it? If you open up the books about this topic that either are written as a university. Proof of concept all the way to, self study. It always starts with gather your sources, figure out what is the domain about in the first place.

So there’s very much a almost top down approach where you go okay, we got sales, we’ve got production. so it’s the general area. You can’t just jump into the jungle and start describing all the little insects on the jungle floor.

It’s that’s not how it works. But usually there is a problem domain, there is an integration problem. and. So what needs to happen is that you need to be able to carve out some time with at least some business domain expert subject matter expert to sit down and say, okay what’s the issue here?

What do you, what are you doing? What does it do? And start writing it down. so far, not that much different from actual data modeling, if you will. But the distinction starts with the nitty gritty where you need to get things right. And it’s usually getting things right, not just to verify if you understood correctly, because that’s already a hard part as my two hour example earlier showed to, just get one line of requirements, correct. But that back and forth with language and examples is of great help in getting to actual understanding. But the major part is usually you need to work with a colleague that also needs to understand it. And with the current short-lived career jumps, if you will, is that you may find an expert, but he might be gone in four hours or as is happening currently as well. A lot of seniors that actually know the organization, they’re getting in a pension range and they just leaving the company. And what is left behind is usually short lived career steps, short lived managers that move on. There’s a lot of tempo where at the average job years is four to six years. So the knowledge actually evaporates while we’re looking at it, while we’re trying to document it. and that’s where the difference starts to rise between traditional data modeling that creates diagrams and type level schemas into a, if you compare that with factory and modeling, you have real user stories that you can, with a click of a mouse can pull up and you can read how it’s being used in language in the organization and how it’s being transferred to other departments. And I think that securing that knowledge, that semantic rich document, if you will that is something that I see that gets lost with the more traditional, more technical data modeling. It’s, yes, you have the traditional layers of conceptual, logical, physical. But still, there’s not really a story there. And the people that I talked with and explained this kind of stuff is that they all recognize what I’m describing and a lot of the architects and a lot of the data models will reply with, yeah, that’s what I do in my head. And then my question is, my obvious question is but does it leave your head, does it get written down somewhere so that if you leave, your colleague can take over? And then the answer is usually no. There might be a document somewhere, describes the use case, but that very quickly gets evaporated as well, because everybody starts looking at the artifacts and the technical schemas. So by having an environment where you tie it all together, that you cannot do one without the other. That was really the grounding for the discussions the proof of the pudding, if if you will. But it also gives you the anchors going forward to technical artifacts. So by capturing that from the domain experts, putting that in an information model and being able to generate technical artifacts, even SQL to generate database, it will still comment all that SQL with all the written language and examples. It will generate a database with test data that came from the interview in the first place. It will add database views representing the full user stories on top of the production data, representing the interview. So it really is that don’t throw it away, don’t throw it away. Keep it as long as possible so that everybody is able to understand and read what the actual data is and what it means and how it’s communicated.

Shane: that’s interesting because that is a form of context first implementation. And what I mean by that is with our product, we define the context first and then the technical implementation is hydrated. So I’ll create the context of a business rule, change rule, and then our system will hydrate that into physical tables and SQL transformations.

But that context that I create is the key thing we, we care about, right? That is our pet. The way we deploy it and run it is our cattle. And what we’re seeing in the new Gen ai, LLM world is that context is actually far more important for the LMS and the physical implementation of that context.

And then as I said, we’re I’m working with Juha Coer at the moment trying to write a book around how to concept model. And one of the things we came up with is one of the steps is define your domain scope. So like you said, find a subject matter expert who understands it.

The next one is get data stories. And then after that, identify events, then concepts, and then connections or relationships. And the key thing is, as you said, is when you talk to experts in this process, if they can articulate the steps they take, those steps are often common. They might use different terms, and they might do them in slightly different order in slightly different ways, but we all do it the same.

If we think about it consciously, it’s where we bring in patent templates, where we bring in artifacts that we use repeatably, that we get that repeatability, that ability for that context to be stored in a way that another person can use it.

So in my view, yes, it’s great if we have a system that does all that for us, but we don’t have to, if we just have templates that are reusable, that’s valuable on its own.

If we have a repeatable process that’s not a methodology, right? It’s not fixed, but it’s just a way of working that is valuable because it’s bringing that knowledge back. So I just wanna take you back to something that you said, So if I take this idea of term, in fact having massive value and that we get that with a subject matter expert, and by documenting it in that format.

That context is able to be seen and understood by many people other than us. You then said that it’s also a way of getting alignment. So if I get that term, in fact, if I get that fact-based model for inventory from three different domains,

How do you deal with the alignment problem?

Marco: I can illustrate it by an example. And let’s stay close to the examples that we already mentioned, but it’s more powerful, more generic and more diverse than that. But I think we would need a podcast of another two days that would explain all of the ins and outs, but, so really to just show you how that would work. And I have to for the listeners doing this in an audio only podcast, I’ll try to make it as visual as I can. So when I say Marco Ban lives in Urich, which is the city of, where I live, is that would be effect that you and I can agree on and, me and my wife, we definitely agree upon that.

So let’s consider that effect and the statement that is to hold some truth value there. But there’s something else going on because Marco Warban lives, INTA is a state statement that, my wife lives in nut, my kids live in, so there’s a bunch of statements.

They all express something that we can classify as the city of residence. So by stating multiple examples like that, we can type those kind of statements as city of residence as the fact type. Now within me saying Mark of wo lives inre, I actually embed knowledge in there, even though you and I can agree upon the actual fact. I’m also saying, wait, Marco Ban is the citizen, is the city. But it goes a little deeper because my citizen is not really a citizen. It is just how I identify a citizen. And in that identification I can see, wait a minute, there’s a first name and there’s a surname. Similar with the city of Urich. It’s not really a city, it’s we’re representing the city by storing the name of the city. So you can see that if you can visualize that there’s knowledge, almost like a graph in there where I started with the city of residence. I gave it the semantics lives in, but it also has structure. It has the citizen, it has the city, it has the first name, the surname, the city name, and all of that ties together into this single fact statement. Now I can populate that with different examples. I can give my wife a position in there and my kids. And so it is populated with all kinds of example data and then the subject matter experts is then post with a series of questions. Could it happen at the same time that Marco lives in re as ma Marco lives in New York?

And then he would probably say, no, there’s something wrong with that. By going through these interactive sessions, which is almost like a gamification of the interview if you will, is you discover the business constraints and then by discovering the business constraints, that will lead to a certain structure.

When it comes to data modeling. If I can live in only one city, then the city is probably an attribute in the citizen’s table. If I can have multiple cities where I can live because that’s allowed in our register, then I would probably need a linking table in the end in physical model. So these constraints steer how the data is structured. And this is the interesting byproduct and I’m not sure if I’m still on track of their question, is that. In the interviews with subject matter experts, we can find ourselves very easily in hours long sessions about, what is a citizen. But as soon as they cannot give me a proper example to illustrate what they’re talking about, I’m talking to the people that are working outside of their scope of expertise. So that helps steering that. So there’s an organizational alignment in my efforts to find the data, illustrating the information. Now, the alignment on the other hand is now I start identifying Marco at some Gmail address lives in. So now I’m identifying the same citizen, but I’m using an email address. Now, obviously, in, in official organizations and registers that would never do, but, let’s suppose we have a small tennis club and it’s fine, so what I find now is that I still speak of city of residence. I still speak of citizen and city name, but I don’t have first name and surname.

So suddenly that citizen, where it used to be first name and surname now is an email address, but it still identifies a citizen. So what happens in the information grammar, if you will, is that there is something introduced called a generalized object type. My citizen can either be identified by first name and surname or by email address. So in the modeling part, it’s very easy to find statements that either generalize, which means that it allows different ways of identification for it, or different ways to talk about it. I now, I am presented with a data challenge because which combination of first name and surname goes with which email address. So again, I need a different fact statement that now would introduce, Marco W has an email address called Marco such and such at Gmail, which links the two keys. So nothing in my communication has to be altered to support alignment of different ways of identifying it, doing data mapping, just as part of the verbalization. I have a department here that only works with names. I have a department there that only works with Gmail addresses. I can make them talk to each other ‘ cause that needs to happen sooner or later. And the way that they talk to each other is say yeah, you’re right. This market woman corresponds with that email address because I have proof of that.

So then suddenly you have a mechanism introduced in the communication on how to identify certain entities in your data administration in a unified way. So this is different examples on how these alignments work on both semantic level, on a data level, on an identification level. In short.

Shane: I was thinking about it slightly differently, but it’s exactly the same process, I think. So let me play it back to you where I think I got to. you are talking there about, okay, we have this term and the term has the same context definition but the facts that identify that term are slightly different, right?

And we can then do some alignment where we see these different identifiers.

Marco: Yep.

Shane: The one that I was thinking about is where a domain or a business unit has a completely different definition for that term. And another one does. And we identify that. So let me give you an example that’s real for me right at the moment around citizenship or residency.

Marco: Oh.

Shane: So I think I know what made me a resident in the uk. It’s when I got a certain kind of visa and I entered the country. And as soon as I did that, that’s the rule that tricked the tick in the box that I am a resident of the uk but when I deal with tax residency, it’s a different set of rules.

And so they’re slightly different, I have to live here, but then also I have to make sure that there’s certain things I don’t do anymore back in my original country of tax residency. And so they’re both residencies, but they’re slightly different. And then how would I articulate that using term and fact, How would I fact based modeling it. And the thing that you talked about is this idea of business constraints. If I can describe the business constraint of what our residency is versus the business constraint of a tax residency, again, now I’ve got two patterns and I can look at those two patterns, those two constraints and say they’re the same or similar or they’re very different.

And in this case I’d say they’re different, One is around how my financial transactions work and one is where my ass sits when I have breakfast for the majority of the year, ‘cause I can spend a certain amount of time outside the country and I am still a resident of this country.

And then I think the other thing that this Pattern gives us, which we know happens a lot because we see humans do it, is as soon as I give somebody those terms and facts. And a set of business constraint, a set of descriptions of how they behave. Humans love to point out exceptions, especially subject matter experts.

It’s yeah, you could be a tax resident in the UK if you do that, but actually if you do this one other thing that nobody ever tells you about, actually you are not That’s the exception. Oh. And by the way, system A knows about that, but System B doesn’t. And humans are really good at pulling out that.

I dunno, what would you call it? The knowledge that only they have. And like you said, when we used to work for organizations for 25 years, that knowledge state in the organization, now you’re lucky if it’s five. So that context, that knowledge of those exceptions disappears. I can see how this idea of defining or articulating business constraints, identifying exceptions, finding them to that term, in fact Pattern in an artifact that’s repeatable.

We can now shortcut, like you said, the need for a data expert and a subject matter expert to spend a week in a room going through this magical process that only two people can do. And make it a little bit more, not democratized, but a little bit more accessible and repeatable.

Marco: it’s an interesting example, but again, it’s, it touches the exceptions. Again, your example in itself is an exception. And it’s, it is too much to say about exceptions. But even talking about it as you just did from a tax context, Shane, the citizen dot. So even in our semantic and in our language use, oh, we already start distinguishing that. And then it comes back to do we identify the citizen in a text context the same as the citizen in a legal context. And to tie it back to my previous example, are those two the same citizen? And all three of them are different domains. There is the text domain, there’s the legal domain, and there is the domain where we might want to match if this is the same person, for whatever anti-terrorist rule organization or whatever. So there’s different domains and every domain has different rule set. So blankly calling them all citizen because in mentally we can say, yeah, it’s the same person, it’s the same citizen. But data administration wise, the data administration of identifiers for a citizen are not the same as the data administration for identifiers for a citizen in a different context. And I think again, that’s the big separator between and it causes a lot of confusion in interviews and workshops, is that. We humans unify that because we see that the reality, it is about one person. But what distinguishes that is that we need to talk about how we talk about the data. And that is not the same thing as the reality. And by separating those out, it makes it, in a way, it makes it easier to say, okay, but are we talking about the same data administration?

No. Then we’re separating those out and if we are talking about three different domains that need to come together in one data administration, then we need more semantics. Distinguishing the three.

Shane: a couple of things I want to loop around there. So again if we go back to the term actually, there is a citizen and a , physical residency and a tax residency because I’m actually a citizen of New Zealand still, but I’m resident physically in the UK and my tax residencies coming with me. But again, it’s not until we, we start using term and facts together that I can articulate that those are three different things, and it’s the business constraints and the exceptions that tell me there are different things. one of the things that I find interesting is this idea of focusing on the complexity and modeling that, is the area that’s gonna cause you a problem.

So if you ever do data bulk training with Hans and WinCo Brockman, they’ll, and I dunno if they do it anymore, but they used to talk about Peter the fly. And so the example would be if I am sitting in this ice cream store and I see Bob come in and order that ice cream and there’s an order id, So I can say when I put term, in fact, there’s a an ID in there of 1, 2, 3, 4, which is the order number. And if I want to know whether they actually got the ice cream. Is a separate part of that process. And on Peter the fly, I can see the ice cream being handed to Bob. So I know it happens if I’m in the room, but if I’m looking purely at the data, at the facts, I see nothing because there is no handover,

there’s no delivery ID that they got the ice cream. So I’m gonna have to infer that if there was an order and no refund turned up, that I’m inferring that happened, but it’s not a fact. And so I think, again, those terms and facts tell our stories and then as data modelers we can look at the exceptions that we know we need to worry about.

But I wanna go back to this idea of domain because it’s one that I struggled with a lot and I still do is a domain boundary can be anything. It can be a business unit, it can be a series of core business events. It can be a process, it can be a use case, it can be a team topology it, a domain is just a boundary where you say it’s in that boundary or it’s not right.

It’s in this domain. It’s in that domain. And I hadn’t thought about using the term fact and business constraints as a way of defining boundaries. For domains. So if I see, a term, in fact that’s all around my citizenship and I see another term in series of facts and business constraints around my residency and I see another one around tax residency and they are different, then I can use those as domain boundaries.

They might be too granular for the intent that I want to use it for. But what we’re saying is they are different. You write them down as words and numbers you can tell they’re different. So that gives us a form of boundary. And so that’s really interesting because it gives us a Pattern,

it gives us a formula of how we can say that this sits in boundary A domain A, and this sits in domain B because of these rules. And I find that really intriguing and valuable ‘cause it’s solving a problem that I’ve been trying to solve for many years. ‘cause it’s annoying me.

Marco: Yeah. It’s, it is. So there’s two you’re right there’s two big areas where you can see there’s a separation of domain. And so rules is one of them. It’s in a hospital systems for example, you might wanna need somebody’s birthday to be able to put it into the computer.

As we had this kind of operations, these treatments, it needs to go this to his insurance company and all of that. But if you’re brought into the emergency room unconscious, you have no idea on you. You’re still being registered as a patient. So the rule is entirely different, but yet we have a patient number somehow. And the definition of the patient is clear across the whole hospital. So rules determine some sort of context. Or, if you come in with an appointment, they probably know your birthdate. If you wanna send it to the insurance company, they have to have your birthdate. The emergency room doesn’t really care. So how do you align all of that? So then you see, again it’s we’re unified in the language, we’re differentiating on the rules, and somehow we have to integrate systems. The most important part is that independent of which system is being built, we need to figure out how we communicate first.

So that language always goes number one. And that’s basically the what fact-based modeling does. It allows people to talk about it from their context, and then given that specific context, you add specific rules for that context. The example where can I live in multiple cities, municipality wise, no, you cannot. You have to be registered in one municipality. Now, in a more generic way, you can have people registered to different places. Sure. I have an office here, I live there, I have a vacation home there. And it’s yeah, different places. But on a municipality level where it becomes more context specific, there is a rule that says you can only register in one municipality. So there’s that too. There’s, there is a way to generalize in a more abstract way. If Ft has been doing that for years, where they introduced the party model, is it a person, is it an organization? Is it, we don’t know, is let’s call it party. We’ll give it an artificial key and we’re good. That’s just the way to keep the IT system running. It has nothing to do with semantic or integration or alignment or whatsoever. It’s just a technical solution because we didn’t get the business meaning in the first place, or we are not specific enough to a specific context. So these problems will always occur. And I think that separates the data modeling from information modeling is where the data modeling is. Do we find alignment in how we need to build the system? Whereas information modeling is, do we find alignment on how we communicate about the data?

Shane: so again, you are making a context differentiation

between the audience. That’s the consumer of what we are creating. And you are saying if it’s stakeholders who aren’t data experts or IT experts, then we’re information modeling. If it’s data people or IT people then we are data modeling,

And the ability to have both languages, but then a mapping and sharing across those languages of where the real value is. And so like you, I’m not a fan of thing as a thing. I don’t agree we should ever use that term in our information models. We should never use party entity thing as a thing.

A thing is associated with a thing as a generic way of describing context to a stakeholder. I also don’t think we should put that in our technical systems. Years ago when we had mainframes and we had no memory and we had no disc and the infrastructure was expensive yes, it had value to us. Right now, the most expensive part of our systems is the humans and understand that context.

And as soon as you design a system with thing, as a thing, and that context lives nowhere else, I now have to spend a massive amount of expensive time trying to understand what the hell, how many things you have, how many things they are related to, and how many things those relationships are.

And that is an expensive piece of work. And I just can’t justify doing that anymore. But as you can tell, I’m slightly opinionated on that one. And it’s also, if I come back to the way we do that information modeling the grain of it, The detail we go to is interesting

because if I do it just based on terms customer orders product, it’s very different doing it based on terms and facts customer bob orders ice cream

Marco: Yes.

Shane: And so what you are saying is actually you are bringing more detail, a higher level of grain into that information modeling process earlier because it has value, so you’re doing more work upfront because you’ve found ways of taking that work that’s done early and automating some of the downstream work in terms of the technical implementations.

And so bringing, from an agile point of view, you are doing work upfront work in advance, which could be waste, but you’re found a way of taking that work and reducing the waste further down the value stream.

Marco: Yeah. Correct. It’s really that, and for those who are interested in this a lot of the information can be found on the website called casetalk.com. What it shows, and this was developed in in, like I said in the seventies, all the way up to 2000 is it’s not just, oh, how can we capture the language?

But it was really, how do we talk to the domain expert to come up with the appropriate IT system? And I think what happened really is that it is such a rich and detailed environment where people modeling traditionally with business knowledge in their heads doing the IT itself, which was very close bound to, organization. When the first computers came in, it’s like it was usually the domain expert that got an IT training. They knew the context, but nowadays it scales up so much is that you are an IT professional. You have no idea about any business. You just, your business is it. So getting back on that and trying to find that alignment with business is, it is of increasing interest, but it’s not the majority.

A lot of new young professionals base their efforts on the tools and the ability of tools and not as much as are we doing the right thing for the business because they don’t really care. They were not educated as such. There’s a massive gap there that where business looks at it is like you’re, you are the expert, so you tell me and then there’s this rift and. You mentioned data Vault a couple of times where, traditionally the hub is the natural business key, right? It’s not a technical key in the source system. So what is the business key? Then you have to talk to the business. And the rich information modeling does with all the fact base is that already mentioned Marco Ledge and Nutra.

So I already have the natural business keys right there. And with a push of a button, it can then be generated into a citizen table and a city table and even have artificial keys with minor annotations like city of residence. We might want to keep a log in time, we might wanna have history in there, but also when is he planning to move to the next city so it becomes bitemporal. And these are very simple flags in the information model that can be generated to a data model that says, did you want a data volt model or a normalized model, or did you want a adjacent schema? So the physical parts become automatable. The data model becomes automatable and having all the semantics and verbiage and examples, it makes it verifiable and readable by the business. And so that really what it ties in and some experiments show that precisely that combination is the real power to keep LLMs grounded as well.

Shane: I definitely agree on the LM front. We know that if we pass at the terms definitions of the terms, the facts, so data examples around those terms and their relationships and natural language business constraints into those lms, we get a much better response when we want to do a task.

Marco: Yep.

Shane: So that context is really important.

And that’s why, it’s interesting watching all the, bi semantic layer vendors try to say they’re context engines and you’re sitting there going, but you’re just at the end of the chain, you’ve just got metrics and maybe some definitions, but the richness is all sitting on the left of our value stream.

It’s all around problem ideation, discovery design. It’s not physical implementation of our consume layer for your cube. Yes, it’s got some value And so I’m in really interested to watch the reinvention of the market yet again. ‘cause as you said technology is often what people get taught.

A number of times I’ll talk to a data science student who’s been taught Python

And you go back and you go, but. How do you understand the business problem? How do you understand the value you’re gonna deliver if you build that ML model? And we’ve seen, as you said, we’ve seen over time data teams that don’t add value don’t survive.

Marco: True. And I remember an interview that was told that we’re, and not to talk them down, but it really points at the lack of education as well, or, the business debt and the technical debt. It’s just people are being trained in tools and in technical approaches where some of the analysts really said, what is your highlight of the day?

Is that when they discover insight, it’s like insight. Should it not have started with the insight as documentation, or what are we really doing instead of let’s discover it. That is, like you said it’s the end of the chain while everybody shouts. Shift left shift left. It’s yeah, why?

Because that’s where it started and that’s what we lost. From my perspective, it just starts with can we talk to each other and are we writing that down? How we do that? Because that’s what we need to do in the end. And whatever it system we built, whatever dashboard is being built, it actually serves to better communicate about what we are really doing. So it all ties back to that. And yeah, there’s a never ending story because the LLM is the next silver bullet. It’s gonna solve all our problems. And it’s not, it’s helpful, but it’s not the silver bullet. ‘cause we still need to get the authentic story, not the fabricated story.

Shane: Yeah it’ll help us understand where the context differs but it won’t help us actually define what the context is.

Marco: Yep.

Shane: At the moment. from the stuff I’ve done with it. Oh maybe the, the next generation a GI version will, or, maybe we all end up with standardized context that every business follows.

But we know that’s not true either, right? We saw that with tools like SAP, where it’s implemented vanilla and then $50 million and five years later there’s a bastardized customized version because that organization is different with air quotes.

Marco: and I can illustrate that by fairly simple examples is that, I mentioned that somewhere else. It’s like there’s not a bank in the world that I didn’t purchase the IBM banking model at the same time I dare any bank in the world to actually have implemented it. and that points to a massive gap is that, to be able to implement it, you first have to know what you already do. Which is a massive dilemma ‘cause nobody has the information models and then you are presented with a technical model, like the banking model that says, everything will fit in here.

It’s okay, but what do we have that actually fits? So there’s a massive dilemma there. And then in the end every bank will try to do it a little bit different. So they have an advantage over the competition. So they don’t really want to confirm to one standard, which points back at. There is no real universal Pattern because everybody tries to give it their own little edge, which means you have to capture that edge, not just build something that you think will fit.

You have to be very specific and in that I’ve seen systems database was designed with rigor and the software was developed three times over, but the database didn’t change. What happened is. A new wave of tech came, it used to be mainframe, then it became Windows, then it became internet, then it became mobile Data did not change the way the technology work, changed the organization, changed the business processes because, we’re not having a physical storefront for the bank, but we now we have a mobile app. So the process of working with the data changed, but the data itself didn’t change. So there’s a lot of dynamics going on and it really shows the importance of getting the data right and don’t throw away the story that came with it so that you can actually reinvent all the processes and all the software on top of it without losing meaning. But it also points at the fast changing world where we perceive everything changes all the time. So we cannot sit down to do a proper information model and have a data model set up, et cetera, et cetera, because we’re in a hurry. We need to deliver what? Nobody realizes that if you sit down long enough to have that information correct, you have the data model correct.

And don’t throw away the story that it will definitely give you a return of investment.

Shane: I’ve gotta say that the IBM data model was a masterclass in sales, The fact that you could sell a diagram, a picture for millions of dollars that nobody ever used, apart from putting it on their wall. That is a masterclass. But I go back just to close it out, I go back to that idea of terms and facts.

So if I have a retail bank, had a store. The terms and facts might have been customer Shane has an account. 1, 2, 3.

If I became a mobile only customer when we changed technology, I might see the term in fact change slightly, where it’s customer oh seven five, yada y has an account 1, 2, 3.

Now I can look at those two lines and I can see something’s changed.

The fact that relates to that term has changed and now I can have a conversation about is that just what you are showing me? And in the background it still says Shane, or did it never say Shane in the real data? It was always customer id. 1, 2, 3, 9, 2, 4. There’s a whole lot of conversations I have, but I know that something’s different and now I know what to have a conversation about with that subject matter expert.

As you said in the past, the subject matter expert, the data person and the IT person was the same person.

Then it became the same team. Then we became a business and IT team, and now we are a business subject matter expert team, a data team and an IT team. We are, we’ve team topologies have got more matrix E, more bigger, and that separation causes some problems.

So by using patents and patent templates and artifacts and shared language, we can close that gap again. And for me, this idea of fact-based modeling, this idea of binding terms and facts with a set of rules, with a set of exceptions as context is really valuable. We can use it in so many ways.

So just to close it out, if people wanted to hear more about this, read more about this, find out more around fact-based modeling, where do they go?

Marco: I think the quickest way into it is to just go to I personally wrote a little book which is published by techniques publications called Just the Facts. It tells the story and names a few examples that I mentioned in this podcast too. It gives you an overview from management to architecture, modelers and developers, and how communication and effects weave through all the disciplines. Obviously CaseTalk is the software tool to support all of that. But you will find links for actual books if you happen to have a copy of the DMBOK by DAMA . The older edition has a crippled article about it. The version two latest edition has a slightly improved article about it. It’s on Wikipedia sources enough, but the good starting point would be casetalk.com.

Shane: Excellent. Alright, thank you for that. I’ve got a new set of patterns and templates I need to go and read a lot more about. It’s gonna be me over the next few months again. But thank you for that

Marco: good. Thanks for having me and talk to you soon.

Shane: I hope everybody has a simply magical day.

«oo»

Stakeholder - “Thats not what I wanted!”
Data Team - “But thats what you asked for!”

Struggling to gather data requirements and constantly hearing the conversation above?

Want to learn how to capture data and information requirements in a repeatable way so stakeholders love them and data teams can build from them, by using the Information Product Canvas.

Have I got the book for you!

Start your journey to a new Agile Data Way of Working.

https://docs.cloud.google.com/bigquery/docs/shopify-transfer

«oo»

Google Cloud just added Shopify and Mailchimp to their list of data collection services

Shagility — Wed, 04 Feb 2026 08:27:24 GMT

Google Cloud just added Shopify and Mailchimp to their list of data collection services, deffo a case of slowly slowly catchy monkey.

Just under seven years ago Nigel Vining and I onboarded our first customer, The Good Registry, to give our MVP AgileData.Cloud platform a run for its money, to test what survived and what didn’t.

One of the first requirements was to collect data from Shopify.

We had a principle back then (a principle we still have today) to use the Google Cloud Data Transfer Services as our first cab off the rank for data collection.

Alas at that time there was no Google Cloud data collection service for Shopify, so Nigel crafted a set of patterns using the Meltano open source framework to automate the collection of this data.

Part of this work was to refactor Meltano so we could operate it as a “serverless” pattern on Google Cloud, rather than it operating under a 24/7 always on container pattern, that would cost more money.

We have reused this data collection pattern many many times over the last 7 years, another great example of our DORO (Define Once, Rese Often) principle.

I saw last month an announcement that Google Cloud have added Shopify and Mailchimp to their ever expanding list of data collection services or what they call “Data Transfer Services”.

If you haven’t noticed Google have been quietly expanding out the list of systems of capture that this service can collect data from.

To me it looks like they are starting to ramp up in this space and will become a serious competitor to tools like Fivetran.

Time will tell.

One of the many things I have learnt about Google Cloud over the last 7 years, is they may not be first to market, but when they go after a market, they have the firepower to do it at scale.

Google Gemini showed us that yet again.

You can check out the doco for the new Data Transfer Services here:

https://docs.cloud.google.com/bigquery/docs/mailchimp-transfer

Redesigning traditional data systems to support Large Language Models and AI agents with Mayowa Oludoyi

Shagility — Sun, 18 Jan 2026 10:46:30 GMT

Join Shane Gibson as he chats with Mayowa Oludoyi about redesigning traditional data systems to better support LLM and AI Agents

Listen
View MindMap
Read AI Summary
Read Transcript

Listen

Listen on all good podcast hosts or over at:

https://podcast.agiledata.io/e/redesigning-traditional-data-systems-to-support-large-language-models-and-ai-agents-with-mayowa-oludoyi-episode-80/

Subscribe: Apple Podcast | Spotify | Google Podcast | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser | Deezer | Podcast Addict |

You can get in touch with Mayowa via LinkedIn

Tired of vague data requests and endless requirement meetings? The Information Product Canvas helps you get clarity in 30 minutes or less?

Google NotebookLM Mindmap

Google NoteBookLM Briefing

Executive Summary

To effectively harness the business value of Large Language Models (LLMs) and AI agents, enterprises must fundamentally redesign their data systems. The current paradigm, built for direct human interaction via structured queries, is ill-suited for the speculative and exploratory nature of AI agents, leading to significant inefficiency, waste, and prohibitive costs. Agents do not simply query data; they “probe” and investigate, a process that generates redundant queries and consumes vast resources if left unmanaged.

A paradigm shift is required, moving toward an architecture that is inherently AI-friendly. This involves three core transformations:

Developing Multimodal Query Interfaces: Systems must evolve beyond SQL-only interaction to support both structured queries and natural language, providing the rich, multimodal communication channel that agents require.
Integrating Context as a First-Class Citizen: The historical separation of data and its descriptive context must end. Future systems need to store and surface rich metadata, business definitions, and operational knowledge directly alongside the data, providing the essential “grounding” that agents need to perform accurately and efficiently.
Implementing Intelligent Query Management: Data platforms must become active participants in the query process, capable of determining which agent-generated probes are necessary and which are wasteful, thereby preventing redundant execution and controlling costs.

This briefing document synthesizes these critical insights, outlining the limitations of traditional systems and presenting a blueprint for the next generation of data architecture designed to convert AI’s speculative power into tangible business speed and value.

1. The Imperative for Change: From BI to Foundational Data Engineering

The evolution of data roles provides a crucial lens for understanding the need for architectural change. The journey of Mayowa Oludoyi from a Business Intelligence (BI) Analyst to a Data Engineer highlights a critical realization: the ultimate value of any data product, be it a dashboard or a machine learning model, is contingent upon the quality and structure of the underlying data.

Motivation for the Shift: The transition was prompted by the understanding that “everything still come back to the data.” A significant portion of time in analytics and machine learning is spent on transforming and cleaning data, suggesting that more focus should be placed on the foundational data processes rather than solely on the end product.
Challenges in Skill Acquisition: This career transition was not a straightforward path. The primary difficulties encountered were:
- Lack of Mentorship: Finding a mentor to provide structured guidance—explaining the “why” behind learning paths and the nature of real-world problems—proved difficult. Most learning had to be self-directed through courses and books.
- Absence of a Structured Curriculum: Initial learning was described as “a little bit scattered” due to the lack of clear, process-oriented roadmaps. While tool-focused roadmaps (Spark, DBT, etc.) exist, the more valuable curricula focus on fundamental processes and principles, as “tools we always change... it is a means to an end.”
The Value of Mentorship: A valuable mentor is not one who teaches coding, but one who provides structure and context. This includes explaining the types of problems data engineering solves, the reasons for solving them, and the logical sequence of learning (”you need to learn A, before you learn B”). This contextual understanding is often missing from online courses and is essential for career progression.

2. The Core Problem: Why Traditional Data Systems Fail AI Agents

The central argument for redesigning data systems is that they were built for a different user: a human executing precise, structured commands. AI agents interact with data in a fundamentally different manner, and this mismatch creates significant friction and waste.

Defining the “Traditional Data System”

For this analysis, a traditional data system is one where the primary interaction model involves a user leveraging a structured language (e.g., SQL) to execute a query against a database (e.g., Postgres) and receive a direct result.

The Nature of AI Agents vs. Humans

The key distinction lies in the method of inquiry. Humans query, but agents investigate.

Speculation and Exploration: Unlike a human who formulates a specific SQL statement, an agent must often “speculate” to find an answer. It performs exploratory work, probing the data system with a series of queries to build understanding. This process is inherently iterative and less direct.
Probing vs. Querying: The interaction is better described as “probing” rather than querying. The agent is investigating the data landscape, which is fundamentally different from a human retrieving a known piece of information. As stated in the discussion, “agents don’t just query, they probe.”

The Consequence of Misalignment: Waste and Inefficiency

Simply layering an AI agent on top of a traditional data system introduces massive inefficiencies that businesses cannot afford.

The Context Gap

Traditional systems are designed to store data, not the rich context surrounding it. Agents, however, are critically dependent on this context for “grounding”—the ability to understand the data’s meaning, relevance, and structure. While data catalogs have historically attempted to solve this, they often failed due to the high manual effort required to populate them. With AI, this descriptive, natural language context is no longer a “nice-to-have” but a core requirement for system performance and accuracy.

3. Architecting the Future: A Blueprint for AI-Ready Data Systems

To address these shortcomings, a new architectural approach is necessary. This approach redefines the interface, integrates context as a core component, and introduces intelligent oversight of the query process.

3.1. A New Multimodal Query Interface

The first step is to create a different query interface that serves both humans and machines effectively. This interface must be multimodal, accommodating both:

Structured Query Language (e.g., SQL): For precise, efficient data retrieval.
Natural Language: To allow agents to process and leverage the rich descriptive context needed for grounding and reasoning.

This dual capability allows the system to support traditional analytics while simultaneously providing the necessary foundation for advanced agent-based interactions.

3.2. Integrated Context as a First-Class Citizen

Context must be elevated from an afterthought in a separate catalog to an integrated component of the data platform. This can be achieved by providing agents with access to various forms of grounding material:

Past Queries and Code: Providing an LLM with a repository of previously written, successful SQL queries serves as powerful, practical context. Mayowa noted this technique works “perfectly” for his personal use, allowing the LLM to generate new queries based on established patterns.
Code with Embedded Explanations: The most effective context combines structured code with natural language explanations. In one example, an agent’s performance improved dramatically when it was given access not only to a repository of transformation rules (code) but also to comments within that code explaining why each rule was created and what business purpose it served. This provides both the “how” (the code) and the “why” (the context).
Centralized Context Stores: The system architecture must include a mechanism, potentially a “meta store” or a new function within the database itself, to store and serve this context universally. This ensures that any agent interacting with the system has access to the same grounding information, promoting consistency and accuracy.

3.3. Intelligent Query Management

To combat waste, the data system must evolve from a passive recipient of queries to an active manager of them. The system itself should have the intelligence to “determine what query or probe needs to be executed.” By leveraging its own metadata, the system can identify and prevent the execution of irrelevant or redundant queries generated by an agent’s speculative process, thereby preserving resources and controlling costs.

3.4. The Rise of Specialised Agents

An effective strategy for managing complexity and improving results is to move away from a single, generalist agent. Instead, a team of specialized agents, each with a “bounded context,” can be deployed. For instance:

An “ADI the data modeler” agent focuses solely on data modeling tasks and is given context specific to that domain.
An “ADI the engineer” agent handles data transformation rules.
An “ADI de boss” agent orchestrates the workflow, passing tasks to the appropriate specialist agent.

This approach mirrors how human expert teams function and has been shown to produce significantly better and more reliable results by ensuring each agent operates within a well-defined and deeply contextualized skill set.

Tired of vague data requests and endless requirement meetings? The Information Product Canvas helps you get clarity in 30 minutes or less?

Transcript

Shane: Welcome to the Agile Data Podcast. I’m Shane Gibson.

Mayowa: Hi, this is Mayowa joining from Nigeria.

Shane: Thanks for coming on the show. Today we’re gonna talk about redesigning current data systems to power LMS and agents. But before we do that, why don’t you give the audience a bit of background about yourself.

Mayowa: thank you once again. Again, let me just say thank you for bringing me to the podcast and , like I mentioned earlier, my name is Maya . I started off my career working as a business intelligence analyst. And honestly, I still feel it’s still one of the most interesting roles and anybody , can, get involved in. It’s very interesting. And while working as a business intelligence analyst, this kind of gave me the opportunity to work several projects. I started off working in consulting and so that kind of gave me the opportunity to work on, several projects.

We have clients working in telecommunication, in banking, in health tech. And so , starting off that way, give me, enough opportunity to experience, different projects. And while working, I had the opportunity to working on several projects, like working on data warehousing, building data reporting, power bi, Tableau and all of those interesting data projects. But then as time goes on, I move on to. A new company after two years. And then in this new company, it is just basically a payment company that is in FinTech. And then, , while I was there, I did a lot of work around analytics, data engineering, and that was where my interest in de engineering started, evolving, and from there I left the company another after two years, and then I moved into a bank. And then this time around I started working as a data engineer. And why, working as a data engineer, as a bank have a lot of legacy system. There’s a lot of things, a lot of moving parts when it comes to data engineering in a bank. You talk about privacy, you talk about, building systems that are highly protected, all of those things.

So while in the bank I started, having this interest in. how to actually build, robust pipelines. So I wanted to move from the normal, building ETL. And so I started learning and that was how my career has, spiraled all through the years. And yeah.

Here we are today.

Shane: that moved from being a business intelligence analyst focusing on the reporting. Tablet, power bi, I’m assuming the data was served to you, so you grabbed the data you needed and you focused on visualization and user experience,

And then moving back into the engineering side, into the code to transforms the data, collects it, all that kind of gnarly stuff that, that’s a common path for people.

How did you find it? How did you find changing from, a set of tools and a certain, set of skills to then expanding them out into new tools and new skills? Was that an easy journey or did you find that step change quite difficult.

Mayowa: I’ll say for me it was not that easy journey. The reason being that the resources are there on online, most time the resources are not what translates to business value, right? So sometimes you have to struggle to get things done. So it was not an easy journey. But one thing that I feel prompted this shift in me was that at the end of the day I discovered that one of the most important process still remains the data. You can have beautiful dashboard, you can have machine learning models that are doing, fantastic. But everything still come back to the data. So I started asking myself the question, . I think I spend a lot of time, transforming data, cleaning data, whether to produce report or to do machine learning. And so I feel like it’s important to spend a lot of time around the data itself than even the product. I mean, The end product is really important, Because that is the whole point. But then I feel like it’s important to spend more time on Iran. So that was what prompted, but then I had to, look for a way to attend conferences, take courses online, and then that shift was not easy.

But, gradually I started building all the necessary experience that I need. Yeah.

Shane: And so often I’ll see people that make that change from being a, bi centric set of skills to that data engineering set of skills. Often they’re in an organization that actually encourages that change. So they’re in an organization, they’ve got a bunch of mentors, a bunch of people who’ve done it before and they can get a lot of help and mentoring about the things they need to learn and a lot of feedback.

In some organizations that’s not possible. So people will go and study on the site, they’ll go and find courses and they’ll try and learn it to help them then, effectively change organizations to change roles. Which way did you go? Did you find that you had mentoring and support within an organization to do the step change?

Or did you have to learn outside the organization you’re in and then change jobs to, to go into the new role?

Mayowa: honestly I couldn’t find a mentor. I had to take the other route. I went all out to, get information. I have a couple of people who, tell me exactly, oh, this is what you need to do. You can take this course. I have those people, I have people tell me that, but I didn’t have the opportunity of having someone. Who will hold my hand through the rubies and show me everything I need to know. So most of the time I had to go out there myself, pick up courses, pick up books to read and that was it. So that was the part that I took. But I understand what you’re saying. There are a lot of people who have the opportunity of having senior data engineer on the team who can tell them what to do, and then they learn through, working on different tasks, different projects along the, but I didn’t have the opportunity.

Shane: And if we think about the idea of a curriculum, an idea of these types of courses, these types of tools, these types of skills and learning them potentially in this order makes sense, so almost like a manifest or a curriculum of what you might want to do, rather than having to try and find it all yourself.

Did you find that, did you find that there were resources out there that gave you the idea of a bootcamp? Or did you have to go and talk to lots of people and cobble it together to yourself to figure out how you went through that path?

Mayowa: basically, my knowledge was a little bit scattered at the very start because there was no tailored curriculum to give me more a roadmap. These days I see a lot of roadmaps here and there, and I wish I had this when I was starting, right? But then when I start, when I started, I didn’t have this roadmap.

I had to learn. So at the end of the day, I have, a lot of knowledge and it took a lot of time, a lot of, learning to put everything together to make sense actually. But I think that problem has been solved. I think I’ve seen a lot of resources online, a lot of roadmaps that I think is doing justice to that now.

yeah. And those roadmaps, are they primarily technical and tool-based, or are they about the process and the ways of working as much as they are about the technologies? Because again, when I see, if I look at job ads or I look at what people say they do, I’ll hear DBT Spark, Databricks, I hear tools and technology versus the skills and the tasks that, you need technology to deliver them, but they’re just as important as the technology itself.

Shane: So what did you find?

Mayowa: I can share this with you at the end of the, session because I don’t wanna mention names. there’s this particular roadmap that I’ve seen, which I think is very good. When you look at the roadmap, they didn’t spend a lot of time talking about tools.

Tools can change, but they spend a lot of time around processes, which I think is really great. Things, fundamentals that I think everybody should know when it comes to data engineering. And even though that domain I think is really good. And I shared the same. Opinion. I see a lot of roadmaps where they talk about, spark DBTI, I don’t think that is the best. I think it’s because tools we always change. They are, tools are like, it is a means to an end. It’s not an end in itself. So I feel like roadmap that talks about the real process, the real principles.

I think they’re very great and I think that is what we help a lot of people to, to move up quickly. .

Shane: We met each other on the practical data community and it’s still one of my most, my favorite, most active communities I’m in. And yeah, I think something, something we should look to do is creating that roadmap or that curriculum as an open source roadmap and curriculum as part of that community, because lots of people have that struggle when they’re trying to get into the domain or drop a jump across the roles.

Where do you start? And so last question before we get onto the current data stuff because this whole onboarding of a person into a new career really intrigues me. So when you talk about, if you had have found a mentor who would’ve helped you through that process.

What would you have expected them to do with you? What would you want them to help with?

Mayowa: Sorry, can you repeat that again?

Shane: So you talked about the fact that, you were looking for a mentor and you struggled to find one.

Say you had found one, let’s say somebody had said, yep. Happy to help you take that next step in your journey. What would’ve been valuable for that mentor to do? What would you expect them to actually do to help you progress your career?

Mayowa: I think one major thing I look out for in a mentor, it’s not to. Teach me maybe how to write code, but rather just to show me exactly what I need to succeed and I’ll explain what I mean. So for example when I was trying to, break into data engineering, it was difficult to actually see somebody who have the experience of what happens in data engineering. There is a lot of resources on how to write SQL Python and all those things, right? But I need somebody who work in data engineering who tells me, this is the kind of problem we solve. This is why we need to solve this problem. This is why you need to learn for, that kind of structure. that is what I’m looking for, and don’t think it’s something that is readily available. In the market, or I’m not even sure. Most courses offer that even when they, offer these courses, online. So I think for me personally, I’m looking for in a mentor, somebody who is not just interested in, learn these skills, but tell you exactly the structure of learning. All right? This is important. For example, you need to learn A, before you learn. B, you need to learn B before you learn C, something like that. That is what I’m looking for in a mentor.

Shane: I think that’s the important part. I, so I mentor people every now and again, and I know a bunch of people that want to mentor but aren’t sure what the process is. And they often think it is more teaching. They think they’re gonna have to spend hours teaching somebody how to do something. And if you’re in an organization where you are mentoring somebody in your team, junior, then yes you probably will do that.

But if you are external to that organization, to that person, to me it’s about connecting with somebody, listening to where they’re at. Potentially making a suggestion what they might want to try next and why it’s about storytelling, one of the things I find is if you can tell a story from your career that I did this and this is what happened and this is why it happened, that’s valuable.

And then the last one is context. When somebody says, okay, why does everybody say you shouldn’t do real time streaming? The context is, it’s expensive still to do that versus batch nine Times outta 10, they don’t need the data straight away. They can live with 15 minutes or an hour. So everybody says they want it, but actually when they see the cost of it, they say they don’t.

That’s the context. And you tend not to find that story of the context of why does everybody say don’t do near real time? Unless you have to have, it is not well written somewhere. And it’s definitely I don’t often see it as part of the course, so Yeah. I’m with you.

Is it really is an hour, maybe to a week having a chat with somebody and just helping them with their career. And so for me I think about all the people that helped me, all the people that helped mentored me and my career, , I think people who have been doing it for a while need to pay it back.

So anyway end of end of pitch for people to mentor more people in the world. Okay. So let’s talk about this idea of redesigning the current data systems to make them better for LMS and ai. So when you kind of propose that as a subject, talk me through it. What do you mean by that?

Mayowa: I started thinking about the whole LLM. Thing some few months back. As a matter of fact just like I told you, I have a, an article I want to release, which I think I just wanted to talk about, the traditional data systems and what we need to get to that point where enterprise can take advantage of what LLM is offering right now. And this started off from, working and everybody’s talking about LLM, since 2023 and now everybody has been talking about LLM and the whole boss, and there’s so much, happening LLM these days. So I ask myself on the job right now, I use charge GPT, I use Claude for a lot of things, but I ask myself, how does the enterprise, how does the business still get value from these LMS at the end of the day? So that was all led to that thinking. And then I started exploring and I’ve seen very great articles and, research papers around that. So for me, the management team here or where I work, they’re not just interested in hype, they’re interested in value, so before they put money into any of these things, they want to know how much value they can get from it.

So sometimes it’s difficult to get a buy-in the management about, some of these things unless they see, what they can get from it. So that was why I started talking about, I started thinking about how we can, bring value to the enterprise through early. But I then I discovered that. the current system. I don’t think that value can be created immediately with the current systems that we have, if at all. There’s gonna be a value. When I say value I’m not saying, we’ve seen agenda doing interesting things like, a lot of people integrate agent to their GitHub repository.

that is good for the developer, but when it comes to the business, I don’t think there’s so much value that they’re getting from that. So I think for us to get to that point, there’s gonna be a need for us to rethink some of the processes and even the data system itself should, so that, that was how I got to that topic, to that theme that I gave you.

Yeah.

Shane: Okay and so there’s a couple way we can approach this. We can look at what you would see as a traditional data system And describe that, and then figure out what you think should change, or we could just. Identify some things, people, processes, technology design that you think when you look at that, that needs to change for the use of LMS and to add value back to stakeholders.

So I’m with you. We see lots of use cases of developers using copilots and LMS to automate or make their jobs easier, remove that drossy work. We see, some interesting use cases using those tools to automate business processes

Data. And then there’s this third. Kind of dimension, which is agents using data for stakeholders.

It’s not business process automation, and it’s not co-pilots for developers. And

Now everybody goes, oh, it’s text to sql, right? Ask a question get an answer. That’s the obvious use case. So which way do you wanna do it? Do you wanna describe a legacy system? And then what your change, or do you want to talk about specific use cases that you think are valuable to a stakeholder?

And then we talk about it from that lens.

Mayowa: I think we talk, we can talk about a bit of the two, when I was, trying to, just put some point together before joining I actually have three questions. I feel like it’s important for us to answer.

One of the most important questions is why do we need to redesign the data system? Why is that we can’t take advantage of the current system? I’m talking about the traditional systems, right? I think that is very important and then. We then need to answer the question, why can’t the current system be leveraged as it Because there are two different things. One thing is that we need to answer the question why we need to redesign the data system. We need to talk about why can’t the current system be used as it is? Unless even, maybe we can even add on top of that. what do the traditional data system looks like?

Maybe somebody’s listening to this conversation and does not even know what the traditional data system looks like. So we might even want to touch about that. And I think the final question that I think we should try to provide an answer to is how then do we redesign the, system that we feel or that we think will be able to get us to that level where we can, get massive gain from LMS and agents.

Shane: let’s do that. One of the things that we need to always recognize is we need to anchor our language in a way that everybody else gets a shared language. That we are using. So when you say traditional data system, I have in my head what I think a traditional data system is, and it’s probably not the same as yours.

Because mine’s based on, 30 years ago, 20 years ago, 10 years ago, there’s been, broad iterations of traditional data systems. What I always suggest people do right, is when they’re, they’ve got a mental picture in their head, they’ve got a map. It’s always useful to describe the map.

In a way that everybody goes, oh, that’s what you mean. But let’s start off, what do you wanna do? Do you wanna do, why do we need to redesign traditional data systems? Or why can’t we leverage them for lms? Take me away.

Mayowa: I think we should start with the why, we designed the data system. I think one thing that is really important is that we need to know that LLMs or agent, whichever way you want to call it, they’re different from humans, right? if you have used agents or if you’ve used LLMs, but whether it’s strategy or clo, whichever one, and then you use it around, data, you will know that they don’t ask once, they don’t ask once and then analyze. There might be a need for you to, ask again, change some things. And so because of that, an attempt for these agents or LLM to provide an answer they speculate. That means they try to do some form of exploratory work, so that makes them different from human.

So because they’re depending on whatever it is that you give to them, for them to now, think. So that in itself tells you immediately that they behave deriving from humans. And so it is the natural way we get information from data system cannot be the same way that these agents get, information for us.

I don’t know if that makes sense.

Shane: Yeah, so let me just play it back and see if I understand

What you’re saying. So if we look at the lms, the foundational models, they’ve been trained on large knowledge bases of text.

And so the early days, we’d ask ‘em a question, they would search that knowledge base and they’d come back with an answer.

And all we did was we’d ask a question, it would look into itself like a human would if it only could look at its memory. So it would go what do I know about that? Here’s an answer. And then what we found was actually we wanna provide context to it. We wanna provide some additional information that foundational model may not have or may not have brought to the front of its memory.

And so we started seeing techniques like rack, and prompt engineering ways of giving it additional instructions on how we want it to behave. Because the way my dentist behaves is different to the way a data analyst behaves. The prompts were behave like a dentist, behave like a data analyst.

And then the second thing is, here’s some other information you probably don’t have that again, a human would ask, if you said to me how many customers have we got? I’m probably gonna say to you, what’s your definition of a customer and where do we hold the data about a customer so I can go look at it and count it for you?

And you want to count today, right? Not last week or next month. And it’s okay to count it to one, So 1,001 customers is okay. Yeah. So there’s a bunch of questions. As a human, you would ask if you’re an expert in your domain. And so this idea of context and reinforcement is passing that back.

So we pass the LLEM prompts to tell it how to behave the data we want it to look at if it doesn’t have access to it already. So we give it access and then the context, additional information we know is valuable to it. So is that what you mean by, The change in what we have to provide versus a SQL statement, which is select this from there, and all we gotta do is give it the code and the data and the machine runs that code on that data and we’re done.

nothing else needs to happen. We don’t need to tell it anything else. Just run this code.

Mayowa: I think the key words you rightly explained it, but I think the keyword there is that because LMS are going to be, or let me say agents let me let, for, let’s just use agents, all right? because agents are gonna be one of the ways we interact with data, when it comes to LLM, agents are gonna be like the main use case, right? So I think the key word is that since agents are not human, there’s a higher chance that in an attempt to provide an accurate response to your queries they speculate. That mean they spend a lot of time, doing some exploratory work, and that in itself have impact on the data system. So because there’s gonna be a lot of redundant queries, things that are not necessary that you will need to fine tune and all of those things. And so because of that I think just, like you said, they have been trained on this massive test that might not even be relevant to what you’re saying.

But then they have to re something back for you. So I think the key word here is that they speculates, and in all of these, it’s an attempt to just give you a response.

Shane: Okay so again if I play it back, if we took a human behavior, it’s like me saying to you, here’s my one terabyte data warehouse. go and tell me how many customers I’ve got. But by the way, there’s no table called customer.

Mayowa: Yeah.

Shane: And now you’ve gotta do a whole lot of exploration, right?

You’ve gotta go and try and figure out what’s a customer called? What table does it live in? How’s it defined? And you’re gonna go and do all this work. And that work takes human time and it takes compute from the system because I’m gonna be writing queries. I think what you are saying is with the L lms, it’s the same, If we say to them, here’s a big blob of data. With no context and no constraints and go answer this question. cause with the reasoning models, and reasoning in quotes,

Going through and building itself a curriculum. It’s building a manifest. It’s saying, I’m gonna go try that.

That didn’t work. I’m gonna go try that. That didn’t work. And it’s trying lots of things. And because it’s a machine, yes, we sometimes see it tell us what we’re doing. But it’s doing a whole of work under the covers. And one of the other things about that is token cost. Because every time it does a task, it’s using a token that costs us money.

If we change the way our data’s platforms are structured, we can make the LMS more efficient, more effective, and require them to do less thinking because we’re giving them hints of what they should use. And when is that the angle you’re taking?

Mayowa: Yeah, correct. The fact that you mentioned the cost thing, make me remember, some of the reasons , so I wrote here you pay for waste. and this is the, one of the reasons why I feel like we need to redesign this is because enterprises, businesses don’t want to pay for waste, Redesigning this system is how we convert, this speculation to speed. Because it’s important for us to know that businesses whether small or large, not in any way we allow waste, So it’s important for us to know that agents. They don’t just query, they probe and they continue to, and there are a lot of redundancy at the end of the day, if we don’t, get this system right, because it is in the nature of agent to probe instead of just getting that.

Yeah. So yeah you’re absolutely correct. Cost is a very important

Shane: And one of the things is at the moment we’re not paying the true cost of those tokens. We pay $20. Okay, now we have to pay 200 a month. Okay, now we’re getting some constraints from Claude and those tools where we can’t just run everything forever before we run out of our, our limit of usage.

But we’re still not paying the true cost. And so while we can do things that are lazy right now and they work and don’t cost us a lot, it’s gonna change, right? And so, you’re gonna start getting the $30,000 bill and we might as well start designing our systems now to be cost effective and efficient.

And I think the other one is speed. If you think about back to your power bi Tableau days, there was a rule of how long you could let a user wait before that report rendered, it was a second or two, and then they start going, this is too slow. And then you think about how much engineering we put in to summarize the data or to give us that performance.

And now you go into an agent, an LLM, and you ask it a question, and then when it’s doing its reasoning models, it’s really interesting to watch how it just sits there and gives you some feedback. It’s working and it’s not dead, but it’s taking way longer to bring back counter customer, which I could have had on a Tableau dashboard in less than a second with what, with, a blog code.

So we’re still experimenting where these agents are best fitting. Okay. So what we’re saying is things have to be changed, Because just throwing an LLEM on top of your current data platform is not gonna be the most cost efficient, not gonna be the fastest it is gonna.

And that waste will hurt us. Okay.

What’s next?

Mayowa: I think we can now talk about what the current data system looks like, right? What I term traditional and like you said, traditional in this case, maybe means several things to several people, right? But for me, traditional just mean the current system where we are able to use languages like structure Korean language, SQL, I, so right now the way we work is we leverage. Platforms, databases, Postgres, name it. All right. And then we just run a query and then we get a result, and that’s what the current system looks like, right? But then when you look at the way agent works I think the right word to use is that agents don’t just use queries. what they do is not just querying. What they do is more like investigating, probing. they want to probe. So we need systems. That are beyond, of course, SQL is very important. Many times people say SQ is gonna be very, we are, we’re always gonna be, I see people learning SQL every day.

I still see a lot of university teaching sq l in their curriculum. So I’m not saying SQL is going anywhere, but I’m saying we need system that will accommodate for other things like natural language. Because these agents will need much more than SQL, like we said, they would need contests.

We need to provide some kind of contest. So there might be, maybe there’s gonna be a need for us to have interfaces that allow for not just the structure query language, but then maybe natural language. So the current data system that we have does not, have that yet. So there might be a need for us to. Incorporate these kind of interfaces, To accommodate for these agents to perform better.

Shane: In the past we’ve always had data catalogs.

So we’ve had this idea of a catalog that set across the technical metadata, the technical context, the technical structure. So I have these tables, I have these columns, I have these values.

And then from a governance and stewardship point of view, we always knew that adding context into that catalog had value.

So that table holds a record for customer. That table holds names, therefore it’s PII that is the table that you query. If you want a single list of customer. These other ones they don’t hold everything or they’ve got duplicates, There’s all this language, this descriptive stuff that was useful.

But what we learned was nobody ever does it. The cost of creating that context was way higher than the value in using it. And yes, we had big organizations where it got mandated and they were certain industries where you had to do it. And yes, we got big tools that automated some of it and that, but in my experience, data catalogs was the tool you bought and two years later you turned off or nobody used it.

I think with LMS and agents, I think that those types of capabilities where we, we bring this natural language context in as text, as description of information is highly valuable. But I’m not convinced that data catalogs of old are the right place to do it. And the reason I say that is I think they’re disconnected.

They’re exhaust they hoover up exhaust. So we hoover up some technical metadata and then we ask somebody. As an extra task to go and add the business context, the operational context of, how many rows are in there, what’s the data quality like and they don’t even have this idea of a gen context, you can’t really, at the moment store prompts or hints for an LLM for an agent, I don’t think they’re fit for, I don’t think they’ll fit for purpose in the original days, but I don’t think they’re fit for purpose now. So I’m with you. I think that there will be a new paradigm coming out about how we hold this language, this description, this context against everything to do with data.

And then that is highly valuable to the agents. And then the way we capture it, to me, it’s gotta be done as close to the creation of the code or the data as possible. So data engineers who are doing the work. Have it in their brain. They know how the accounting customer, I’m gonna create a dim, I’m gonna create a hub and set.

I know the logic because I’ve asked, and I’ve done that work. They’re the ones that we have to make it really easy for them to capture that tribal knowledge into a place that the agent can go and find it. Or maybe we just have probes in our head and then the LLM could ask us. ‘cause when you work in an organization, you probably do it.

You go, oh, I don’t know how we calculated that. And you go and ask Bob. And you know, you, you know somebody in your team that you can go and ask and you get that tribal knowledge outta their head. Maybe, we’ll, I’ve been facetious here, but maybe we’ll have probes in our head where the LMS ask us.

Yeah. So is that what you mean, that idea of that really rich, descriptive stuff? Yeah.

Mayowa: So like I said, it’s summary. I think it’s important for us to take advantage of these agents gonna be a need for us to have a paradigm shift from the normal, SQL against databases. There’s gonna be more things that are involved, like you mentioned, maybe catalog that maybe need to have the data system itself.

Maybe the databases might have something that stores a lot of context in memory, just to provide more context. But I think one thing that is important for this agent is that they need grounding, need to have a lot of information for them to act, anyway, so the current data system, we have to evolve to the point where we have some of these components, integrated into the system for us to take advantage of the agent.

Shane: Let’s just take that comment around probing from the way that you meant it, not me putting a probe in my head for the LL to get my knowledge. And the traditional way we’ve always done it is we’ve taken data from source systems and we’ve bought it into one place and we’ve typically put it in a database of sorts that, like you said, we can write SQL and we can get an answer.

And yes, we played around with no SQL databases and Hadoop and whole of other stuff, but we keep coming back to a database column that stores data in a certain way and allows us to use this standard language to get the data back out. Seems to be a valuable way of working. And then we’ve now got this introduction of MCs where we can effectively.

Create almost an API or a, a network port or an HTDP kind of endpoint for the LMS to talk to a system and get a response, So I need this, give it back to me.

And it’s not writing sql. it might talk to the MCP server and the MCP server might write SQL to talk to the database and get back and then hand it over.

so I look at it and I go, do we know that there’s a bunch of use cases where probing won’t work? And I can come to those in a minute, but why wouldn’t we just fundamentally change the way we work? So rather than grabbing the data from the source systems and put it in one place, why wouldn’t we just expose these MCP services and allow the LLM agents to always just do a one and done query?

Have you looked at it at also actually remove the data warehouse? No. We’ll come to the use cases. Why? That probably won’t work at the moment, but I’m just intrigued by that. It’s, just go away, ask the question, give the answer. If the context is bound around it, if the context is the thing we care about, that’s the pet.

And where the data live really is the cattle. It’s an interesting architecture change to what’s been 40 years of my life.

Mayowa: So I think is an interesting one actually, but of course, we can talk from ante toward the reason why that is not gonna be possible. But I think it doesn’t stop the fact that it make us start looking at things in a different way the way, we used to work.

like you said, I don’t think there’s anything stopping us from doing that. Even from hindsight you start thinking about the cost effect and, a lot of things that might, but I think that there’s absolutely nothing stopping us from exploring that data warehouse today and this is me just digressing. In my experience, the whole idea behind data warehouse, which is consolidating data from different sources and putting it in one place. When you look at it, if we ask ourself this question, has it really, solved the problem? Because I’ve seen places where there are still information that you can’t find in the data warehouse.

That warehouse supposed to be the source of truth, but there are still information that you can’t find in the data warehouse. So if that is still a problem, so what if we have, like you described, if we have a sea of data where we can just throw these CPS to, get information.

I think it’ll be about, but then. there will be a lot of things that goes into making that happen.

Shane: And a high chance of waste. Yeah. A high chance of

Wasted cost , slow. All those things that you talked about at the beginning that actually we need to focus on at the moment. I think whatever we designed would give us those problems.

But that’s at the moment because we’re in the beginning of this whole new wave and so we haven’t really done the rethinking, the re-engineering of what it would look like.

So where’d you get to in terms of, if you started with a blank piece of paper, what would you do right now to build a data platform capability that is LLM, agent friendly.

Mayowa: The first thing is that need a different query interface. Just, like I explained, we need a different query interface and this interface should accommodate for both natural language, And also the structure, query language, SQL and the rest, this interface accommodate for that. But at the same time I think to avoid some of the things that I’ve talked about, like redundancy like waste and all of those things. I think there’s gonna be a need for us to have the data system being able to, determine what query or probe needs to be executed. Just to prevent that waste. Because even when we have this interface that accommodate for several mps that allow agent to run, different queries against the data system, it is important for us to also put, some measures in place to know what query to execute. Because at the end of the day, whatever you get back, it’s not gonna be all useful.

So I’ll say an example, so if you say something like, oh give me the sea strain of the sea strain of maybe battery in the United States. An agent is just probably gonna run a query against different web servers, different webs and just print. Everything is not gonna be useful for you.

But when you take that into an enterprise environment where you need to manage costs, where you need to ensure that you maximize and optimize your queries, that you’ll see that is gonna be a big problem. So for me, I think the first thing I will look at is that we need a different query interface.

That we need to find a way that the data system, determines what query to execute. So for example, the database know, if there’s a way to, of course there’s gonna be a need to implement maybe metadata storage and all of those things to give an idea of what currently exists within the data system. So that determines immediately this is what is available. So if you run a query that is not relevant to what is in the device, you just don’t execute it. So that is the way it is in my head right now. So we need a new interface. We need something within the data system to determine what query need to be executed. is the way I will start designing.

Shane: What’s interesting is this idea that when we move from humans to machine, we get infinite scale. So that example you used of, what is the sales trend of batteries in the US

A human, I have some natural constraints. I have natural constraints of time. I can’t spend two years going and finding that number.

I have a natural constraint of knowledge, right? I’m gonna Google it. I’m gonna find some websites, and then I’m probably gonna even run outta time or get bored, and I’m gonna stop. Whereas when we talk about , the agents, it’ll scale. If you let it right it’ll search every website in the world.

If you let it, if you till it to, it can run a thousand human, I don’t know. What is it? CPUs for the computer, miles per hour for a car? What’s a agent cycle for? For number of human hours. So it, it can run a lot of human hours in a short amount of time for a cost.

Anything we do that is a Pattern based on a human constraint of I don’t have time, I can’t scale myself, that disappears. And we’ve gotta be really cognizant of that, or we’re gonna get massive waste again, like you say, I think the other thing is this idea of, if you look at human behavior, if you’re in a data team and you’re in a team of five to nine, you’re gonna find that normally you’ll have a data modeling expert, you’ll have a data collection, source system expert, you’ll have a data engineering expert, you have this experts and you are naturally gonna go and ask the expert for some help. Hey, I am looking at this and I need to model it. And I know we, we are a data vault is our standard. So can you just. Gimme help because I’ve never done it. I don’t do it very often. Or can you peer review And one of the things we’ve found as we’re building this out is that when we had a single agent, so our agent’s called 80 when she was the only agent we used to flutter. So we would say, go and do this work. And she’s just time slicing her skills all over the place. And we always got back an okay response as soon as we then broke her out into other agents.

So we have 80 80, the data modeler, 80, the engineer who goes and figures out the transformation rules and then 80 debos. And so we talked to Ada Deboss and she knows about these other versions of herself. Then she goes, oh, okay, the next thing I need to do is model that data. And she talks to a, the data modeler.

And what that means is we provide a, the data modeler, a bounded context. We give her very specific prompts around You are a data modeler. You’re not an engineer, you’re not the boss, you’re not the bi you, this is your job. These are your skills. We give her a bounded subset of context. you can look at the data structures but you can’t go and do data quality tests.

Because we’re talking about, in this case, conceptual modeling. That’s not your job. Another 80 will then take what you’ve done and make it better or do her task. And what we find is that specialization of skills and then that handoff, which happens in a team, but you don’t really see it, if you then programmatically do that with the agents, we seem to get better responses.

So I’m with you. I think this idea of. What interface, what language, what set of skills, what persona can be bounded and then handed off all over the place. And that’s, I think, where we’re gonna need to, I still think sql to query data if we need to do that

is gonna be done.

However, we’ve had great success giving images to the lms, not data,

but it’s expensive. It’s wasteful. So if I give it a photo of a an image screen capture of an Excel spreadsheet,

It’s gonna do really well with it. But it cost me a lot more tokens than if I give it the CSP.

So I think it’s, again, that trade off between outta the possible right now and then the cost and waste of doing cool stuff.

Mayowa: So I think part of the reason why I also mentioned that it’s, important for us to have this interface that accommodate for both natural language and also SQL is, right now, part of the way I work is I actually have bunch of queries that I’ve written in the past.

And I just dumped them in my LLM dump a lot of them. And right now I just ask questions. And because this LM already have history of my queries that I’ve used, TPV, how to calculate TPV, how to calculate TPC and all of those things. So I just say, Hey, can you give me, develop a query to, and it does that and then gimme the output.

So I think part of, what you’re saying is, providing contest is. There’s still gonna be a lot of work to be done around, contest, so giving queries that we’ve used in the past, or maybe anything that can provide more grounding. We actually, help resolve some of these issues.

I’ve seen it work firsthand. It’s working perfectly for me. There are a lot of requests that I don’t spend my time developing the query anymore. My, LLM just, does that for me,

Shane: And that’s a really good point, is this multimodal thing. Joe Reese talks about multimodal arts. And so this idea that no organization actually uses one data modeling technique for their warehouse, They may say they’re dimensional star schemas, but there’s some relational stuff in there.

There’s some sorts. Yeah. We always have more than one modeling technique, and I think we’re gonna end up with more than one language. And that multimodal language is gonna be important. And so I, I’ll give you an example we. Presented all our documentation to 80 for our product.

And so we describe what we call change rules, which are transformation code. And we have a set of language you have to use for it. And so we described, how the rules work, what the structure is, what the structure of the language is, and there’s some examples,

if you want to go and pivot, do this if you wanna un-pivot, right? But it was all in text. And so when you know, somebody comes in to our platform and they go, I need to transform this data and they just write in plain text, I need to transfer this data, how would I do it? She came back with an okay answer.

But it was just, okay. And sometimes it was just wrong, she would hallucinate and we’re like, yeah, that’s not how our product works. And then what we found was effectively we run a multi-tenancy architecture. So we stood up another tenancy, or one we already had for our partners called Alliance, and we pushed example transformation code, our example, change rules across every customer we had.

We took the rule itself, not their data, and we said, this is a rule we’ve applied before. So your, exact same example of yours, of here’s a blob of code, this is a rule that we’ve used before. And we then created an MCP server that 80 could see, to see those rules.

That’s when we broke off to 80 the change rule.

And so all she does is if you ask her how you can transform this data in natural language, she will go and search those rules. But she’s effectively searching a code repository that is highly opinionated and highly structured, the language is the same. What’s in the language is different. And our the response back we get is so much better now.

And so the thing we found though was while we had the rule, the logic the language of the rule, we had no context. We never wrote down

Why we were doing. So as soon as we added that, a natural language of this rule is to take stats New Zealand data, which is being summarized as columns and unpivot it to rows so that we can use it later when we need a row per record.

Then she’s ah, now she’s getting both the, the thing. the same as you are saying is that if you take your blobs of code that you use on a regular basis, and I think you, when you talk about TPV and TPC, you are saying there’s some metrics in there, if you then in that code, put in the definition of the metric, TBV equals and then a bunch of texts, and that’s as a comment in that code, then the LMS are gonna get both a structured piece of code and context.

They’re gonna get sequel and they’re gonna get natural language. And I think that’s where we’ll end up.

Mayowa: going back to your question, if you ask me what are the things I need to, think about if I’m going to design what this system should be, I think the interface is gonna be very important. And also, how to manage what query needs to be executed to avoid waste, redundancy. And, I think that is the way I’m gonna think about it.

Shane: So let’s take that example where you’ve got you and blobs of code that you have found highly valuable and you’ve already tested putting that code as a reinforcement model, given that context of the code to an LLE and been able to ask questions and get back the help that you needed.

And let’s say that you do extend it out where you actually define what those metrics are in, in natural language, so it has a richer context. And then you’ve got three other people in your team. Who need to do the same thing, like right now, what would you do? Where would you store it?

Where would you surface it? what interface would you use to create it, to share it? Like how would that work for you right now?

Mayowa: Now the way it works, this is just for my personal use. I’ve not had any reason to share with anybody. But I think that also, point to what I was talking about when it comes to, conceptualizing what this data system should look like, I think there’s gonna be a need for us to have part of the data system, maybe the database, a part of it that stores these I don’t know whether it’s gonna be a meta store, or whatever, that stores this information. And the reason being that regardless of who is running this query or who is submitting a probe, they have access to the same information.

And that information can help provide more grounding the LLM or the agent, or MCPO, whichever one that you’re thinking. I think that part of what we need to think about is a part of the data system. Maybe this can be the database. Now maybe this open a new opportunity for, database research to see how we can, if part of we can store, information that provides more grounding to JLLM.

So that is the way I’m gonna think about it, but right now I’ve not had any reason to actually explore.

Shane: it’s an interesting one because if we think about the fact that we’ve always stored our operational data, O-O-O-T-P data separately from our analytical data,

Our OAP data, that’s because in the past we never really had a technology that allowed both to be stored efficiently in the same place and queried, they were two different query pads, two different storage patterns, and that’s not true anymore. We’ve got tools out there like single store that say I don’t work with ‘em, so I dunno if it’s true, but they say they do that, yet nobody’s really adopting it, that I can see, we still keep it separate.

And so when we start bringing this idea of a context store Victor databases, we seem to be the thing, but actually a lot of the times you don’t have to use those. You can use anything. So I’m intrigued like you, to see whether we end up with yet another database,

We now end up with.

Databases that store the data and the context side by side. It is an intriguing place to be. And then do we end up with a thing that I call the context plane, which is the idea of a shared centralized layer of context? Or do we end up with a context grid? The idea that context is stored next to the data and then something else federates it, provides a grid type architecture. And so again, we’re at the early days where lots of people were exploring and saying what works and what doesn’t. So just on that, just to close it out, so this is something you’ve been thinking about in your spare time, right? You’re not part of a software company building this, your organization.

You’re not part of a team building it out for them. This is something that just natural, inquisitive, nature going, yeah, that’s cool.

Mayowa: Yeah. Pure, natural, inquisitive. I’ve read a lot of papers. There’s this interesting journal that came out of Berkeley. I can’t remember what the title is now, but it’s very interesting. They did a very fantastic job around this conversation too.

So yeah, for me, I’m not part of any team actively developing something in that space, but it is just something that I’m just interested in.

Shane: And the next step is you’re gonna start writing. You’re gonna start writing your thoughts, starting write your explorations.

Mayowa: That’s part of what I’m doing. By the time you release this I want to listen to this conversation again because some of the things that you said, I think that are really interesting. I want to look into them more deeply. So yeah, I’m just documenting my thoughts right now.

Shane: And my advice to people is document lightly and document early. So share lightly, share early. don’t wait for this podcast to come out, document what you think right now and then write it up, share it, and then think about it a bit more and then write it up and share it. because people often find the journey of the way you think, as interesting as the answer. And what I say is by writing, it’s forcing you to think with clarity. you think, you know something, you’re like, oh yeah, I think I know how that’s gonna work. And then you write it down and you’re like, yeah, no that’s bollocks, that’s not gonna work.

So that art of writing just makes you think, ‘ writing is structured it has beginning, middle, and end. it’s gonna force you to story, tell to yourself and validate what you think is gonna happen as, so yeah, my recommendation to everybody is, write small bits.

Push it out early, it helps you think it’ll change. And that’s okay. it’s not a book, It’s not something that you can’t change. It’s Hey, I thought this and now I think this, and I think that’s better than what I thought before. But I had to jump from A to B2C to D to get to E.

Alright when this does come out, how do people find what you are writing? Have you worked out where you’re gonna publish it?

Mayowa: If I’m gonna post, it’s gonna be on my LinkedIn if I’m gonna leverage any other platform. I also put it on my LinkedIn. So the time I’m done with this, I think everybody can find it on my LinkedIn.

Shane: Most of us are now writing on Substack ‘cause LinkedIn sucks for long form content. So I’ll encourage you to create a Substack yeah. And then till everybody on LinkedIn, that’s where the long form content is. ‘cause LinkedIn’s kind of killed the ability, which is really sad. ‘cause actually I’d rather just write in one place.

Excellent. People can see how you’re exploring and what you’re learning and what you’re sharing. I look forward to it.

Mayowa: Yeah. Thank you.

Shane: I hope everybody has a simply magical day.

«oo»

Stakeholder - “Thats not what I wanted!”
Data Team - “But thats what you asked for!”

Struggling to gather data requirements and constantly hearing the conversation above?

Want to learn how to capture data and information requirements in a repeatable way so stakeholders love them and data teams can build from them, by using the Information Product Canvas.

Have I got the book for you!

Start your journey to a new Agile Data Way of Working.

https://podcast.agiledata.io/e/building-data-services-with-ai-with-jason-taylor-episode-79/

«oo»

Building Data Services with AI with Jason Taylor

Shagility — Sat, 10 Jan 2026 08:27:16 GMT

Join Shane Gibson as he chats with Jason Taylor a former quant researcher who turned towards the light (or dark) side of data, to explore the practicalities and pitfalls of building data services using AI

Listen
View MindMap
Read AI Summary
Read Transcript

Listen

Listen on all good podcast hosts or over at:

Subscribe: Apple Podcast | Spotify | Google Podcast | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser | Deezer | Podcast Addict |

You can get in touch with Jason via LinkedIn

Tired of vague data requests and endless requirement meetings? The Information Product Canvas helps you get clarity in 30 minutes or less?

Google NotebookLM Mindmap

Google NoteBookLM Briefing

Executive Summary

This document synthesizes a discussion on the intersection of Artificial Intelligence and data services, drawing from a conversation between Shane Gibson and Jason Taylor (JT). The core thesis is that while AI, particularly Large Language Models (LLMs), has dramatically lowered the barrier to entry for building sophisticated data services—especially in parsing unstructured data—it simultaneously demands a renewed focus on rigorous process engineering, human oversight, and robust evaluation.

Key takeaways include the shift in the data profession from role-based identities to a more fluid, skills-based approach, where transferable skills are paramount. The conversation categorizes data services into three primary use cases: internal business health, customer-facing data access, and external monetization, with AI impacting all three. A central argument is that unstructured data parsing is now a “largely solved problem,” thanks to models like Gemini that can interpret complex documents and even images with remarkable accuracy.

However, this technological advancement introduces significant risks. The concept of “blast radius”—the potential negative impact of an error—is critical in determining the appropriate level of AI automation, from human-in-the-loop “assisted AI” to fully autonomous systems. The speakers warn against “vibe coding” and the tendency to treat AI as infallible magic, citing high-profile failures (e.g., Deloitte, a lawyer using ChatGPT) as cautionary tales. The “maker-checker” paradigm is presented as a crucial process framework for ensuring quality and accountability. The discussion concludes that data professionals must apply their foundational principles of logging, testing, and healthy paranoia to the AI domain, continuously evaluating models and cross-validating outputs to build trust in these non-deterministic systems.

--------------------------------------------------------------------------------

1. The Evolving Data Career: From Roles to Skills

The dialogue begins by examining the career trajectory within the data field, highlighting a fundamental shift away from rigid job titles toward a focus on underlying skills and attributes.

1.1. Breadth vs. Depth and The PhD Dilemma

The transition from a specialized quantitative (”quant”) researcher to a broader data professional serves as a key example. This move is framed as a strategic choice between depth (e.g., heavy statistics, requiring a PhD to compete at the highest levels) and breadth (a wider data skillset).

Market Dynamics: The market often favors broader skillsets, enabling professionals to handle more of a project’s lifecycle end-to-end. As Shane notes, “...it’s easier if you’ve got a broad set of skills to be able to pick up a gig or do a role.”
The “PhD Barrier”: In highly specialized fields like quantitative finance, a PhD can be a de facto requirement. JT comments on this pragmatically: “I don’t have a PhD and competing against PhDs sucks.” This has historical roots in the 1980s and 90s when finance began recruiting physics PhDs for their expertise in signal processing, which was analogous to financial market analysis.
Stereotypes vs. Reality: While the market may have stereotypes about needing a PhD for certain roles, the speakers question the universal necessity, pointing out that “not all PhDs are the same.”

1.2. Attribute-Based Career Planning

A core argument is that professionals should focus on their inherent attributes and preferred activities rather than chasing job titles, which can be defined differently across organizations.

Focus on Skills, Not Roles: JT strongly advocates for this approach: “I hate that the role-based mentality... is for somehow perpetuated.” This is reinforced by Shane’s example of survival analysis skills from genetics being applied to supermarket product placement.
Data Persona Templates: Shane is developing a book on “data persona templates,” a skills-based framework. By analyzing job ads with a custom GPT agent, he has found that despite varied job descriptions, the underlying skill requirements often distill down to just three core personas.

Key Quote: “data scientists are just quants, or quants are just data scientists with more subject matter expertise. Like it’s all kind of the same thing.” - Jason Taylor

2. Defining and Monetizing AI-Powered Data Services

The conversation defines a “data service” primarily as a data-centric offering that generates revenue, distinguishing it from internal data teams that support a non-data primary business (e.g., selling ice cream).

2.1. A Taxonomy of Data Use

Shane proposes a three-category framework for the use of data in an organization:

1. Internal Use: Understanding and growing the business.

2. Customer Support: Enabling customers to access their own data (e.g., in a SaaS platform or bank).

3. External Monetization: Exposing data externally to generate revenue, which can include direct data sales or enabling partners.

The focus of “data services with AI” is primarily on the second and third categories, particularly where data is enriched or processed for monetization.

2.2. Models of Data Services

JT outlines several models for companies that sell data:

Pure Enrichment: A customer sends their data, the service does “something fancy” to it, and sends it back. The process is monetized.
Raw Material Sales: Collecting and selling data, often via methods like web scraping.
Integration: Providing specialty knowledge on how to integrate and organize disparate datasets.

Companies like Bloomberg are cited as examples that successfully combine all three models.

2.3. The Impact of Generative AI

Generative AI introduces a new dynamic: non-determinism. Unlike traditional services that sell a predetermined, consistent product, AI-based services sell something that “may be variable at times.” This fundamentally changes the nature of the product and the processes required to ensure its quality.

3. Unstructured Data Processing: A “Solved Problem”

A significant portion of the discussion centers on the claim that AI has made the parsing of unstructured and semi-structured data a “solved problem.”

Key Quote: “the one that’s exploded the most by a massive amount has been unstructured or structured data parsing... I feel like that’s a solved problem. Now do, maybe that’s extreme.” - Jason Taylor

3.1. From Tesseract to Gemini

The progress in this area has been substantial. In the past, extracting text from a PDF with tools like Tesseract was challenging, and even training specialized models like Google’s Doc AI yielded good but not “that good” results.

Now, modern models like Gemini Pro can process complex documents—including financial statements, org charts, and diagrams within PDFs—with “remarkable accuracy.” JT notes his surprise when he drops a document in and says, “give me everything,” and the model understands the content and structure exceptionally well. This has massively lowered the operational barrier to accessing this data.

3.2. The Diminishing Moat of Domain Expertise

Historically, the competitive advantage (or “moat”) for data service companies like Bloomberg or LexisNexis wasn’t just providing the raw data (which is often public), but the “many years of highly skilled and trained... professionals augmenting that raw data... with context.” This organization and curation is what created value.

LLMs are now diminishing this moat. They have “come a long way” and can infer much of the context that previously required thousands of human experts. While there is still value in tribal knowledge—”not everything we know is written down”—the gap has narrowed significantly.

3.3. The Power of Visual Interpretation

A key advancement is the ability of LLMs to interpret documents visually, not just as raw text.

Image-Based RAG: Processing image data (e.g., a screenshot of a report) instead of just the text can be “wildly more beneficial” because the model picks up on subconscious cues like layout, organization, and what else is on the page.
Use Case: Report vs. Dashboard: Shane describes a project where an LLM successfully categorized 8,000 legacy reports by analyzing screenshots. The model learned the human-like heuristic: “If I see a single table of data, it’s a report. If I see multiple Widgety objects, it’s a dashboard.”

4. The Imperative of Human Oversight and Process Engineering

Despite the power of AI, the speakers stress that its non-deterministic and fallible nature makes human oversight and robust processes more critical than ever.

4.1. Blast Radius and Appropriate Automation

The concept of “blast radius” dictates the level of risk and, therefore, the necessary level of human involvement.

Low Blast Radius: A mistake in a marketing campaign might result in spam.
High Blast Radius: A mistake in pharmaceutical trial data could lead to a death.

This leads to a hierarchy of AI implementation:

4.2. Failures of Blind Trust: “Vibe Coding”

The discussion warns against the dangerous trend of “vibe coding”—uncritically accepting and deploying AI-generated output. High-profile failures serve as cautionary tales:

A lawyer who used ChatGPT for a legal filing, which included fabricated case citations.
Deloitte being forced to repay a government agency half a million dollars after using AI to generate a report that “hallucinated a whole lot of case studies.”

Key Quote: “If you hired a genius level person... Would you read their work after they generated it...? I don’t give a fuck how smart you are. I’m reading what you put together... I’m accountable. So why in any of these circumstances would you not check this stuff?” - Jason Taylor

4.3. The “Maker-Checker” Paradigm

The solution to managing AI’s fallibility lies in process engineering. The “maker-checker” paradigm, a common process in manufacturing and finance, is proposed as an essential model for AI workflows. One agent (human or machine) creates the output (the “maker”), and a separate agent reviews and validates it (the “checker”). This builds in accountability and a review system, much like code reviews (PRs) in software engineering.

5. Evaluation, Testing, and Trust in Non-Deterministic Systems

The conversation highlights a cognitive dissonance where seasoned data professionals often forget their core principles of testing and validation when working with AI.

5.1. The Underinvestment in “Evals”

“Evals” (evaluations) are the AI equivalent of software testing. This is seen as a “massively important” but “under invested area.”

Complexity of Testing AI: Testing an AI system is more complex than traditional code because there are more moving parts that can change: the underlying LLM model (which vendors can update), the prompt, the RAG context documents, and subtle variations in the input data.
Methods for Evaluation:
- LLM as a Judge: Using one LLM (e.g., Claude) to evaluate the output of another (e.g., Gemini).
- Testing at Scale: Running a large number of tests, including edge cases and “chaos engineering” style random inputs, to understand the model’s boundaries.
- Ad Hoc Testing: Even simple measures like asking the same question multiple times to check for consistency in the answers is “better than nothing.”

5.2. Logging and Healthy Paranoia

Data professionals are trained to “log the shit out of everything,” yet often fail to apply this discipline to AI systems. Logging the reasoning path of an LLM is crucial for debugging and understanding its behavior, especially when an unexpected answer is produced.

A “healthy degree of paranoia” is described as a beneficial trait for data professionals. This involves an inherent distrust of outputs and a commitment to cross-validation. JT states, “I still crosscheck things. When I write code with LLMs, I read all of it, like all of it, I see my role as I am the reviewer.”

6. The AI Toolkit and Professional Practices

The speakers discuss their personal toolkits and workflows, revealing practical strategies for leveraging AI effectively.

A key professional practice is to always review AI-generated code. A common red flag is when the code references an outdated model version (e.g., Gemini 1.5 when 2.5 Pro is current), which is a “dead giveaway” that the user did not read the code.

7. Future Outlook: Tribal Knowledge and Creativity

The dialogue concludes by speculating on the future of knowledge and human creativity in an AI-dominated landscape.

Gravitation to the Mean: LLMs, by their nature, gravitate toward the mean or average of their training data. This could create a problematic feedback loop where human thought becomes less diverse.
The Pendulum Swing to the Arts: As AI automates more rote, scientific, and predictable tasks (”sciences”), there will be a “pendulum swing” where society places a higher value on uniquely human traits like creativity, randomness, and artistic expression (”the arts”). JT states, “I am very bullish on the arts.”
The Future of Tribal Knowledge: While some may try to hoard their proprietary knowledge behind paywalls, the speakers hope that technology will continue to lower the barrier to recording and sharing information. This could accelerate the advancement of human knowledge, as more ideas are documented and built upon. The belief is that we have not yet reached a point where “all thought has been explored.”

Tired of vague data requests and endless requirement meetings? The Information Product Canvas helps you get clarity in 30 minutes or less?

Transcript

Shane: Welcome to the Agile Data Podcast. I’m Shane Gibson.

JT: I am Jason Taylor, or you can call me jt. Either one’s fine.

Shane: Hey, jt. Thanks for coming on the show. I think today we’re gonna have a bit of a chat around building data services with ai. But before we do that, why don’t you give the audience a bit of background about yourself

JT: Yeah, sure. I guess the simplest way that I usually explain it is beginning of my career was more quant research and data, and then I just gradually went towards the data. I don’t know if that’s towards the light or towards the dark, but um, let’s see. I worked facts at Palantir. Usually everybody wants to talk about buy side, whole bunch of different places.

Now I play in startup land because I have a death wish. I don’t know. Yeah working on a bunch of fun stuff,

Shane: what made you come from Quant To pure data.

JT: That’s a good question. And, and quant is a data gig in a lot of ways, right? And Joe Reese started in this as well, doing more quant research at one point in his career. Like I, I think there are different types of quants but I think a good majority of them, and as people have become more technical over time, they’ve become more data oriented.

Almost like sometimes I joke that data scientists are just quants, or quants are just data scientists with more subject matter expertise. Like it’s all kind of the same thing. But , why I think the very practical answer is I don’t have a PhD and competing against PhDs sucks.

Shane: I think it’s interesting. It’s this idea of breadth versus depth for me. I see people start out with heavy statistics, right? That’s what they love and that’s what they get into, and , that’s a very specific set of skills. And then often they’ll go into breadth. They’ll extend their skillset out and become more data and a little bit less steady.

And , my perception is because that’s where the market is it’s easier if you’ve got a broad set of skills to be able to pick up a gig or do a role and do more of the work end to end yourself than if you are a, a specialist with a really deep set of skills. I hadn’t quite thought about the PhD side, so I suppose, I always talk about, are you gonna be in the top 5% of your skillset? And I suppose and that area, to do that, you have to be a PhD

JT: So first of all, let me start by saying I very much subscribe to the thing you said where I want to capitalize on my strengths and there are certain things I know I’m good at. And actually Google came out with this cool tool where you can actually talk to it about a job and it will tell you it’s all AI oriented, but it will actually tell you attributes and things. I think you like to self describe your attributes and it helps pinpoint potential jobs that it believes in for you. And I, it’s actually really smart and I very much subscribe to this of attribute based or characteristic based job placement, if you will, as opposed to like, you know, there’s a ton of people that are like, oh, I wanna be a data engineer or a data scientist.

JT: And it’s great. Those roles are wildly different at different places. What things do you like to do? What attributes? So long-winded way of saying I very much subscribe to this. I think that there are certain things that I gravitated towards in terms of the attributes I like and the activities I like, like I like being analytical sometimes to a fault, which I think we all do in the data space to some degree. But then I also think the PhD thing maybe a little bit of, terrible history. But yeah, at some point in time, from a finance trading perspective people started leaning into math go figure. And there’s portions of finance that have always been in, in that space.

But at one point, I think it was in like the eighties if, yeah, more the eighties and May, maybe early nineties was when. They started pulling in like physics professors and the like, because people started looking across the aisle effectively and saying, Hey, you’re doing signal processing and studying these things, and hey, that’s the same as finance.

Like it’s the same thing. So people started looking at that type of math and that type of process that they were doing. And that’s, I think people started leaning into that, right? there’s a piece of this which is, I feel and I’m sure this will relate to AI really fast, but it’s just stereotypes, right?

Like people are like, oh, I need a PhD to do this. And I’m like, do you really, do you know what you’re doing? Do you think all PhDs are the same? But ultimately that’s a good chunk of the market. I’m not gonna discredit people having a PhD. I think that’s awesome. I definitely. Feel like I would’ve wanted to do that at some point in my life had I not made certain decisions.

But yeah it’s hard. There, those people are smart and it’s, especially on paper from a recruiting perspective you know how it’s some recruiters are just like, oh, here are the qualifications I was given, yay or nay. And that’s like

Shane: I think I think the recruitment process is broken. I think that whole industry is gonna be disrupted. I think we’re gonna move to more community or closed network based recruitment. Very much

JT: Aren’t we there though? Aren’t we already there?

Shane: yes we are for some, but yeah. If I look at, over here, the way the government over here recruits, my standard joke is somebody in a government agency generates a job request job description with ai, to which then the candidate uses AI to generate the CV to match the job ad to which then the recruitment agent then uses AI to see whether the CV matches the job ad, I was talking to somebody and they were advertising for an administrator. Office kind of administrator’s part of their process. And they were saying they had five perfect cvs, perfect matches all the experience, all the skills. And when they went to interview them, none of them had done anything near what the CV said they had.

So I think this personal recommendation, this personal network is gonna become more and more important. And the idea of skills \ being transportable. I remember many years ago when I was working at SaaS and I was watching one of the projects, one of the consulting teams did for a supermarket.

And they ended up using survival analysis to figure out placement of take away hot meals with something else. And, I didn’t do well at stats at school. I didn’t enjoy it. So it was always funny when I went to work for a company that was pure statistics. And I was like how does this work?

And they said imagine you got, a tall man and a short woman, and they have a child. They, there’s survival statistics on, which genes are gonna survive and will the child be tall or short? That’s what we use to decide whether we should put beer or Coca-Cola next to those meals.

And I’m like, I still don’t understand it, but it makes sense. And so it’s this idea that actually, like you said with the finance stuff, these skills you get in another domain and then they are really useful in an alternate domain. So focus on the skills, not the role.

and that’s where the value is. Yeah,

JT: Yeah, no, a hundred percent. I hate that that’s perpetuated. I hate that the role-based mentality for a lot of people is for somehow perpetuated that’s just how people think about career

Shane: I’m currently perspiring on my second book, which is around data persona templates. And so one of the things I’m doing is I need examples for the book. So what I’ve started doing is downloading job ads for data professionals. And then I’ve created basically a chat GBT agent based on the book for a whole bunch of prompting.

And I’m telling it to take those job ads and create the persona template for me.

And the persona template is skill-based, it’s all about skills. It’s really intriguing to take all these different jobs that look different, and then when you boil them down to what is the persona it typically comes up with only three,

Three core ones, there’s variations of it, but I’m intrigued by that. But this one’s not about data personas. This one’s about building data services with ai. Why don’t you explain what the hell you mean by that? And we’ll go from there.

JT: What do I mean by that? There’s a very easy rabbit hole here, which is like, what is a data product, which I’m going to avoid. We are not going to go down that hole

Shane: Ah, come on. Pedantic semantics says 45 minutes of us arguing a definition of a definition.

JT: Terrible. No, I’m not playing that game. What are definitions, Shane? Can we, yeah. Define definition. . If you ever see Joe Reese, make sure you ask him to define definition. Joe loves debating semantics if you didn’t know that already. So if you see him or hear him please ask him these questions.

He loves them. Anyways no. I’ll start with an interesting divide that I was actually talking to someone earlier about. I always find it interesting because the vast majority of the data space, usually when you talk to other people in data professions, they’re usually in some sort of supporting role, I’m gonna call it. And when I say supporting role, I mean that the data itself is not the revenue generating aspect of the company, Very basic examples. I work at a company that sells ice cream, so I help ‘em figure out how to best sell ice cream. But still the main thing that you’re selling their ice cream, right? There is an entire world of people like us that sell data. And I’ve lived in that world for a very long time. It largely focuses around things like finance, because finance is an industry that has to consume data in order to operate. There are other industries, I think marketing and advertising and things like that are also in, in this scope.

But there’s an entire industry of people that just sell data. So I think that, especially from a data services with ai, I think of a couple different things. There are the people that are pure enrichment based. Send me your data. I will do something fancy and send it back to you. There are the people just selling the raw material. Here’s the data I collected in some capacity. A lot of those people are doing web scraping, not all of them. And then there’s other people that do integration style work, and if you think about a Bloomberg, which is relatively a household name at this point, they do all of them, which is interesting, right?

They’re both consuming from the public domain. They’re also have, specialty knowledge around how to integrate the data, and then also how to enhance it if you give it to them. But now we’re in a new world, not to say generative AI is brand new. I think it’s relatively common knowledge.

Maybe not for everybody that generative AI’s been around for a little while. It’s just now very mainstream, very accessible. There’s 10 cajillion startups around it. But it’s very interesting because , of course the key. Aspect of it, which is it’s non-deterministic. So now I’m not selling a predetermined thing to some degree, I’m selling something that may be variable at times.

I think that, especially, services around AI today, there’s definitely no shortage of web scraping companies. , I think the one that’s exploded the most by a massive amount has been unstructured or structured data parsing, that’s exploded. And I feel and I’m curious of your opinion, and I’ll pause here. I feel like that’s a solved problem. Now do, maybe that’s extreme. Do you agree?

Shane: I think it’s like when people tell me data collections a solved problem. And every time we onboard a new customer, two or three of their source systems ones we know will and then there is always an outlier system where there’s a hundred customers in the world and the APIs are really badly documented.

And the data structures are a nightmare for us to try and understand. And I sit there and go solve problem my ass. I just wanna go back though. There’s there’s an interesting one. One of, one of the things I do when I kind of work with companies and I’m helping teams out we start this idea of this playbook.

And the idea of a playbook is basically a bunch of slides and they have two purposes. The primary purpose is to explain how the data teams work for any new data team member. So when you onboard, you read this and you you get a feeling of how they’re structured, how their workflows, what the culture of the team is.

And the second one is if you’re a stakeholder in the organization, it tells you how the team works so you understand how to engage with them how long you’re gonna pretty much wait, what your role is and that data work. And one of the slides that becomes really common for me now, right at the beginning, is saying that I can categorize use of data in three ways internal use to understand the health of the business and grow the business data to support the customers where the customer’s actually accessing their own data.

So typically as software as a service or a bank or insurance company. And the third one is where data is used externally for monetization. And that might be selling data that might be enabling partners to use whatever you have. If you’re an insurance company and you’ve got insurance agents out there that I treat that as external data, you’re giving them access to that data outside your organization based on your customers to make more money.

So I kind of like that framing, so what we are talking about when you talk about data services with ai, you are talking about that last one, data being exposed externally, securely to make money somehow. And it may not be selling data, but it’s definitely we are exposing that data and sharing it to monetize it.

Is that what you’re talking about?

JT: I think that is definitely correct. I also think it’s the middle one too which is like exposing their own data, if I heard that correctly.

The exposing their own data, I think is another one. Like a very common is just enrichment, it’s already my data. You may be adding or doing something to it, that then I’m monetizing the process as opposed to the data itself.

If.

Shane: Okay. I heard a podcast a while ago that was intrigued me, and it was around a massive US company that had digitized and augmented all the lawyer case study ti case history, thingies can’t remember which one it was. I seem to think LexisNexis, but I’ve probably got that wrong. And what was interesting here, what was interesting was, and it’s coming back to this idea of semi-structured or structured what they were saying is,

JT: too.

Shane: Yeah. You define definition. Actually, I have a definition of a, and then I was doing a group thing with Ramona and Chris Gamble.

And Ramona and I have a disagreement on the definition of structure that unstructuredand And it’s all around csv, right?

Anyway so what’s interesting about this is you think okay, we’ve got all this case history stuff, and it’s in books and there’s probably digital versions of those books. And so it’s a sole problem. Now I could go and grab all that content and digitize it and create a service that competed with them.

And yes, they’ve got market share. The, how do I find the market in that, potentially, but I, I get the impression it’s not cheap, it’s for lawyers. Nothing with lawyers is cheap. So I could probably disrupt them and Uber them, or Netflix them right?

JT: Yeah.

Shane: What the key was the augmentation.

It was the many years of highly skilled and trained legal professionals augmenting that raw data, even though it was unstructured with context and that context is where their moat was. So before we get onto, , that idea of, is the problem solved? Is that what you are seeing though, is that once you get this data and then you add additional context to it, that creates the value, that creates the moat. That’s harder for anybody else to actually go and breach.

JT: Yeah I’m just gonna start outright by saying yes, I agree with that, and I think that’s, again I’ll refer to Bloomberg, FactSet, s and p, et cetera, because that’s the world I know, which is, public company filings are public, right? You can go and you can download Apple or anybody else’s, 10 K or Q or whatever other filing. Cool, neat. There is some value there in being able to access that and make it easy to consume and blah, blah, blah. No discount to that. And there, there’s actually cool open source people doing that Exactly now, which I’ll come back to. But yes, I the work that they do to organize it, right? To your point on semantics and governance and all this stuff, it’s that organization that actually gives value to the data.

it’s not about just serving it up, it’s about making it usable with other stuff, or potentially integrating it, so on and so forth. So and is that all hardcore domain expertise? Some of it, not all of it. I imagine in the legal space it is far more, but the part I would want to come back to is that getting access to the data and the value provided in that just operationally dramatically lower, massively lower.

I don’t think anybody disagrees with that. I think that that integration, that domain expertise has also become more accessible. I think that even though, in that particular case, and maybe it was Lexi Nexus or something, has all these domain experts there I, I do also think that, lMS have come a long way. They can infer a lot of these things. It doesn’t mean they’re perfect, but it does mean that we’ve accelerated what usually took hundreds or thousands of individual people to fill those gaps. So the moat has diminished significantly.

Do I think that there’s still differentiation or IP and domain expertise? I absolutely do. things I’ve been talking to people about, like I I’m very comfortable with that idea. There’s a very basic idea, right? Not everything we know is written down, period. it’s just not written down, or it’s a little bit more out there. And at the end of the day, large language models, for the most part, gravitate towards a mean, that’s by definition what they do. Do I think that means they can’t learn these things? Not really. And I’m using learn very loosely in that context. Do I think it means they can’t learn these things?

Not necessarily, but I do think the tribal knowledge, et cetera. This is what people are trying to do with rag everything else, right? It’s just how do I shove my tribal knowledge into the thing so that it has the things it needs to do this. But yeah I, that is definitely what we’re talking about.

I think that the expectations and obviously what you can accomplish today is wildly different. And I think that especially from what you can do, the bar is tremendously lower. But I think the expectations are wildly off the chart. I think to the earlier point, unstructured versus structured There was a period of time when shoving a PDF into pick your favorite model, whatever that was sucked a lot. Even if the PDF was pretty modern, you could extract the text off great, cool, still not great. And we’ve come a substantial distance from that where, I, on a regular basis am processing PDFs just as things I’m doing as part of my day to day. And I really like Gemini. I’ll plug them. I don’t mind. I really like Gemini’s Pro model for the vast majority of things that I plug in there and bear in mind, these are all kinds of interesting financial statements from a variety of different providers. It could be org charts.

I’ve found org charts in there, which has been cool, diagrams, all kinds of weird shit. And they, with remarkable accuracy, just pull it off. And my favorite part is that, and not that I’m doing this professionally, but I think the first thing I always try and do just for absolute giggles, is I drop the document and I say, give me everything. And then I’m just like, let’s see what happens. Fuck it. Let’s see how good or how well it understands things. And I’m continuously surprised because, it wasn’t that long ago that, again you’d have a tesseract or one of these other platforms that’s very widely used and , widely accepted.

And even if you trained a model like Doc AI or any of these things, they were good, but they weren’t that good. Like I couldn’t just drop random shit in and be like, Hey, gimme, gimme the stuff. And then That’s awesome.

Shane: I remember zero days when, we talked about dropping an invoice or a receipt and having it just turn up in your accounting system with accuracy. And this is 10, 15 years ago from memory. Back then it was a hard complex problem.

Now it’s not. One of the things I think we’ve gotta be really conscious is, is this idea of blast radius.

So what I used to always say in the data space, ghost of data past is, I’ll work on a marketing campaign. Because, the data we’re gonna get is gonna be crap, and therefore the results we’re gonna get are gonna be okay for moving a lever, but they’re not gonna be accurate. But if I was working on pharmaceutical data for a drug trial, then , it has to be different because the blast radius are getting that wrong.

When somebody dies, the blast radius of an incorrect marketing campaign is you spam somebody. And I think this is gonna be the same with using AI against data services. The standard, you’ve heard about, probably heard about the one where a lawyer used AI for the thing to the judge and Yeah.

And and there’s a use case in Australia where Deloitte’s had to pay back half a million dollars to a government agency because they used AI to generate their very expensive review document. And it hallucinated a whole lot of case studies. So I think we’ve gotta be careful about where we use it.

But I can see that the domain knowledge it has now from all the data and tribal knowledge it stole is useful, right? If the blast radius is acceptable, that actually it’s good enough to look at that, apply some context and it’s cheaper and faster than a human doing it, and the impact of it being non-deterministic and getting it wrong.

JT: But on that note, and let’s, we can tie this easily back to governance and among other things and the, I guess the world that I’ve come from is very easy. In a lot of contexts to generate, I’ll use the Deloitte example because it’s fun to pick on them, to generate a report and just shove it out the door. Why do you do that? Fuck if I now, but it’s very easy to do that. And this is my current complaint with vibe coding as well, right? It’s just that people hit the button, they say, ah, it’s fucking magic. And then they shove it out the door and it’s for the love of God, do you read your prs?

Do you edit your own writing? Please go back and read what came out of the random black box. Places I’ve worked, especially, when we were at Palantir’s amazing. At shipping things fast, right? But I was with a bunch of people, I won’t say where, but like I was with a bunch of people.

I’d have to kill you if I had told you where. But and they were phenomenal engineers, very good at understanding the problem write code very fast. And I caught a couple times where I’m like, guys, just read each other’s stuff. Like it’s not, it doesn’t take, I promise it doesn’t take that long. And then, we got there really quickly. It wasn’t a big deal, but and they’re, again, all exceptional. So it was a fairly easy thing to do. But I know tons of people that are engineers that don’t read their prs or don’t have automated checks in place, or not linting, like all this.

Do you use Grammarly? If you write do you spell check shit? Like for the love of God? If you’re writing a paper that has citations, fucking check the citations. This is like basic stuff, and I think that people are getting so jazzed about the fact thinking it’s literal magic and just forgetting everything.

They’re like, what planet am I on? Hit the send button. Go for it. And it’s just yeah, just take. You saved 90% of your time that you would’ve otherwise spent writing it. You can spend another couple minutes just making sure it didn’t spit out garbage. And, I’ll rant on this for two more seconds.

If it was a person, if you hired a genius level person, let’s assume that you hired somebody tomorrow to help you with your job, that is a certified genius. Would you read their work after they generated it or would you just say here, okay, cool, and just submit it to your boss or your customer? I don’t give a fuck how smart you are. I’m reading what you put together, right? I need to know what it says. I need to know what I’m represent I’m accountable. So why in any of these circumstances would you not check this stuff like that? It just, it’s mind blowing to me

Shane: It’s an interesting one because as we know, l LM is non-deterministic. And so people go how do you trust that it’s doing the right analysis? And Juan Cicada had a great comment many years ago where he said how do you trust the human?

And I sat there and I was thinking, yeah, the number of times I’ve seen a data analyst come up with a number and nobody peer reviewed it, and we trust it because a human wrote some code, and I suppose the code’s deterministic, you can go and see the code and run it time and time again and get the same response right or wrong.

That response is, but it, that’s not the point. The point was you trusted that number, made a business decision and nobody peer reviewed the process or the code. And I’ll go back to , one thing you know is definitely with ai vibe coding at the moment, if you wanna see how dangerous it is. We’re a Google platform, so I love Google.

Actually I love their platform. I love their technology. I hate their partnering and I hate their marketing ‘cause it’s the worst in the world. But anyway just go on to Reddit onto the Google Cloud subreddit and watch how many students are going and buy coding with the Gemini API and pushing their code to a public Git repository and then getting whacked with a three to $30,000 bill within two days because their API key is publicly available and people are just scanning, get repos and grabbing those keys and slamming them.

Somebody should run their eye over the, and it’s like you just watch the unintended consequences of this democratization. But let’s go back to that image one. ‘Cause it, it’s interesting for me. So one of the things we did with one of our customers a while ago, they were moving from a legacy platform to a new data platform.

And they had, oh, I can’t remember what it was, but something like 8,000 Cognos dashboard reports. So they’re built over 20 years and

JT: People watch every single one though too.

Shane: They’re all active. When they asked which ones could we get rid of, the answer was none. And they’d spend a couple of months with a small team of really good data analyst bas trying to document them.

So all they really wanted to understand was how big’s the data estate, right? H how many of these do we have? What do they look like? Which ones do we migrate, rebuild, or migrate first? Which ones don’t? And what we ended up doing is we ended up building a tool called disco. Effectively they exported out all the Cogness reports as X ml.

Yep. That was definitely a disco

JT: I’m dancing for anybody

Shane: Uh, that’s right. Yeah. I have a habit of making t-shirts. So we have a T-shirt for 80 80. The disco people can buy it online if they really can. And so what we did was they basically pushed the XML files to us and then disco when and scanned it.

And we did a whole bunch of prompt engineering based on some patents to say, what’s the data model underneath them? What’s the information product canvas? What action and outcome do we think’s been taken, right? So we generated all this context and then pushed that back into a database so they could query it.

And that worked, right? There was some engineering we had to do because doing it for one XML file manually works, do it for a thousand repeatedly. You have to loop through. But the blast radius was small, right? Because really what they wanted is insight. And then what happened was they came back and they said, we documented the reports that were copied,

so this report looked like that report, but it had a new filter. And this report looked like that report, and it had one more column. So where people had just cloned the report and that helped them deju. But they came back with an interesting question and they said can you tell us which are reports and which are dashboards?

JT: Oh,

Shane: So, Hmm. Okay. And there was some business reasons why they wanted to do that. So what we tested was them uploading screenshots of the reports, dashboards. And what’s interesting is, yeah, Gemini and I think this, were back then, we, this is pre-pro 2.5, but even then it was good. It basically.

Did what a human did and said, if I see a single table of data, it’s a report. If I see multiple Widgety objects, it’s a dashboard. And it went through and categorized them and I was like, holy shit, that just makes sense. And the other thing I’ve been doing is uploading the information product Canvas as a screenshot.

So building a canvas with a stakeholder, taking a screenshot of it and putting it into the lms and then saying, give me the metrics, give me the business model give me a physical model. A whole lot of questions around it. Whereas in the past, what I’d do is scrape out the text for that and put the text objects in, whereas now I can just put an image in there and I get almost as good a response.

Now the key is the blast radius for what I want to do is understanding, , I want to understand quickly versus I’m not gonna go tell it to build an information product and deploy it. But yeah, I cut out a whole lot of effort and it feels magical.

JT: So I’m gonna, I’m gonna do two things. One yeah. Image based LLM use is awesome. I saw someone recently note how we’ve been talking about rag, but doing it purely on I image data as opposed to on the text itself has been wildly more beneficial. And that’s because, there’s subconscious cues there, there’s things that we pick up on when you look at the layout of the text, what else is on the page, how the text is organized.

And it’s not just about looking at the text itself. And that’s been hugely beneficial. So that’s, whenever I do data file processing today, extraction, structured, unstructured, that kind of stuff, it’s all visual oriented. I try to avoid scraping entirely. Now, I’ll give you a kicker, which I don’t think this is IP at all, but like a kicker is a Excel.

I can’t pdf an Excel document that’s just extra stupid, right? You can’t do images of Excel. But Excel is a wildly interesting thing. This is where we can get into structured unstructured, right? Excel under the hood’s, really what XML or that weird format that it uses, right? In all I would define that as semi-structured.

Other people may fight me on this, that’s fine. But I would define that as semi-structured. Because it has structure inherently, but it’s also variable in nature. So I consider that semi. But those documents are hard to understand because, hell, I’ve seen too many really shitty financial models that are just like 20 tables in one tab and I’m like, oh, for the love of God, why did you do this? Who are you and what kind of chaotic person? Organize your shit, like gimme a break. This is insane. No. Scrolling around thousands of rows left or right. This is wild. And you’ve seen these, like people build financial models, just the most ass backwards ways on the planet. But visually interpreting them, assuming that you can get away without the pagination or anything like that is, very good. It’s way better from a visual standpoint. But I think that the big thing, and I’ll circle it back to the main topic here, If you think about, now there are companies out there that sell data where, that have a data oriented process, right? And that is their main revenue driver. And then you think about shoving a large language model into that process in some capacity, mostly probably because it’s unlocking some new features for you, or you’re moving faster, you don’t need humans, blah, blah, blah. In some ways It’s not really different than it was before. And I know that sounds really weird to say, but the reason I’m saying that is because if this is your product you always had, whether you acknowledged it or not, you always had a need to set up proper process to ensure you have a quality product. So for me, when I think about large language models and their use in any of these processes, it’s all process engineering. Yes, context engineering, blah, blah, blah, blah. But like it’s process engineering really that we’re talking about. And especially. Once you start talking about multiple agents, I’m air quoting agents because that’s a different bag of tricks. But once you get, multiple autonomous processes interacting, like it’s all process architecture, right? You’re, I think the most common one that a lot of people talk about is a whole maker checker paradigm of you make it, I check it. That’s how it works forever and always.

We don’t cross lines that works reasonably well. There’s some sort of, accountability structure and review system and so on and so forth. Prs have the same thing and people still put in shitloads of additional automation, but it’s process structure, right? So even if my entire data product just for hypothetical sake is me just. Shoving a prompt and hitting play repeatedly and then sending it to somebody , you should still have some sort of review system. You should still have some sort of checks to make sure it’s not garbage, every major manufacturing, et cetera. Everybody has this and that’s why like, I still think the lawyer and the Deloitte example, I’m sure the PowerPoint Deloitte put out was huge and it had a billion references and it was blessed by the Pope and shit.

Like I, I’m sure it, it had all this stuff so it probably make it really hard. But we’re data people, right? Rip all the fucking things off, go cross validate them with a deterministic process flag. The ones that don’t like you, you can bootstrap your probabilistic process with deterministic shit.

It doesn’t have to just be like, everything’s tossed to the wind where you’re using a new tool just hail Mary and pray. It makes money and VCs will pay you I don’t understand that mentality, we’ve been doing this for a long time.

I am definitely the first one out the door to use AI and LLMs for things, but it doesn’t mean I’m gonna let it, drive my car care for my kid. Like I, I want

some structure and controls around no different than a human right? And I think it’s a lot of people poo poo the idea of treating LMS as human , making them human-like. But I think that it’s a very good analogy, it corroborates my feeling towards vibe coding, which is , why in the fuck would you approach an engineer and just say, build me a website, and walk away and think it’s just gonna work.

Like they’re gonna build something.

Shane: it’s even, worse now though, because you read people saying, my boss vibe coded over the weekend and gave me the code and told me to push it to production. Like there, there’s a problem.

JT: I have a feeling that’s clickbait though.

Shane: Yeah, probably. Although, I’ve met some managers. One of, one of the questions I’ve got around that Deloitte thing though was the first prompt in their agent always you telling it what the latest shape to use in the document, I like where pyramids this week or where circles or where matrixes, because you gotta change the shape of every document that, that was dark as.

And by the way,

JT: That’s where all the money comes

Shane: the interesting thing is this idea of make a checker. Is that what it make

JT: Make? checker. Yeah. It’s a process paradigm.

Shane: yeah. Is around factorization. It’s about repeatability. It’s around deterministic.

And then we have artists, we have craftspeople that make things that are just one and done. they make it once and it is not deterministic.

It is probabilistic. It is a piece of furniture

JT: I’m gonna fight you on this. I’m gonna fight you on this. I agree with you that if I am painting a painting, that it is easy to think about. I’m painting the painting and it just, I paint it and it’s over. But I don’t know, I don’t know if you do any art Shane, but I cook I’ll relate this to cooking. Do you cook?

Shane: Yes.

JT: Do you taste your food while you’re cooking?

Shane: Yes.

JT: Good. That is a good thing you should do. It’s not exactly make checker, but if you have a partner or someone you’re cooking for, sometimes you have them taste it, It doesn’t mean you have to have them check it just at the end and you get to redo the whole fucking thing. But at least having some process that ensures you are not going off the rails entirely. I make a lot of random stuff. A lot of the stuff I make I’ve never made before just because it’s fun and I always taste it midway through because you never know what might go wrong or you might, there’s little tweaks, more salty, more spicy, too spicy, that kind of stuff. But the make checker paradigm, I. that is a very particular paradigm . There are probably some ways that that Pattern is implemented where it’s you finish your thing, give it to me, I review, say whether you suck or not, hand it back to you. Might also be versions of that where it’s mid points,right? Like

Shane: Okay so let’s take that, make a checker idea and say that the process, that even an artist, a craft person checks it themselves, right? They may have another person that’s as experienced as them, or, but they are checking, right? They always checking their work.

JT: Hopefully to some degree.

Shane: And within, gen ai, I have three types.

I have what I call ask ai, which is where you ask it a question, you get back a response, you ask a question and you’re chatting with it. And then you go off and make the decision as a human, right? And ideally get another human to, to check your work. Assisted AI is where it’s watching what you are doing and it’s going based on what I know, you might wanna think about this,

so it’s prompting you. You can listen to it, you can ignore it, but you carry on, you finish that task. And automated AI is where the machine does it and no human’s ever involved. Yeah. It just happens. And so if we take that idea of PDFs And if we think about code being deterministic And LMS being probabilistic, and we think about if I wanted to just upload a PDF and get some tribal knowledge back about it. That is a probabilistic problem

I can put it up there. I’ll give you some stuff. I’m in the loop. I’m gonna make some decisions. So therefore it’s an ask kind of feature and the blast radius of me getting it wrong lives with me,

and am I make checker paradigm. If I was automating that PDF to go into my finance system and put in the number, then maybe I move to an assisted model, I upload it. The machines identifies all the fields for me. It comes back and goes, this is invoice number, this is the tax amount, this is the total amount, this is the supplier.

You happy, click go. So it’s a system, it’s automating all that Rossi ship, but I’m still making the final call of Yeah, that’s right.

Versus if I take it to fully automated. That’s when I’m dropping in a thousand invoices. They’re going into my finance system and a payment has being made, and I am not in that loop.

At that stage. In my head, I go back to code, I go back to deterministic code that is looking at specific places on the layer of that invoice and saying that is the number, and if there’s no number there, don’t take it from anywhere else. And so I would say at the moment, I would not use an automated gen AI solution in that use case.

That’s only because I haven’t tried it lately. Like you said, when I uploaded images two years ago, it sucked. I upload them a year ago. It’s got amazing. I haven’t, done it this week. It’s probably gonna freak me out. So where would you sit, right? When do you jump from assisted human in the loop?

Make a checker to let the bloody thing take this unstructured or semi-structured data and just human out of the loop.

JT: I’ll say a couple things. One, there’s the very obvious part that I’m gonna state because everybody’s gonna say crap about this, but security, obviously there’s a security element to this. We’re talking about financial statements, blah, blah, blah. Let’s remove that just for argument’s sake. So I’m gonna repeat again, we are removing the security concern here and the data privacy and all that shit, just to have a hypothetical conversation before people are like, eh, privacy, blah, blah, blah.

Shane: But hold on. What’s your definition of security?

JT: ask you, go call, call. I’m gonna give you Joe Reese’s phone number. You can call him. He loves to debate semantics. Anyways this is gonna sound funny. I don’t know where that line is, and I am actively and consistently trying to do it the automated way as much as possible, and I often equate a lot of these things to like meditation, right? This is all about building good practices. You have to set up the conventions in your mind, build the muscle memory to do the thing that you may not have done before. I’m with you it’s very easy for us as engineers to think about Hey, I wanna rip this one cell.

I know it’s in the same place every time off this document. Write the code to do it right? And don’t get me wrong, there’s an over-engineering element to this of throwing an LLM at a problem like that is definitely a bazooka. At an anthill, like a hundred percent. There’s also a time to market, I’ll call it component of this, which is how fast can you write that code compared to how fast I can go to a website and upload a file and ask a question.

I’ll quick draw with you and I’m willing to bet that I’ll win. And I’ll still get the same answer, right? And then there’s a middle ground if you really wanna fuck around, which is have the l lm write the code to do the deterministic thing. That’s a whole nother like I don’t have to have the LLM just do the work. I can have the LLM write the process to do the work. And then you get a little bit of, a little bit of both, Because it’s a deterministic process that was generated very fast. The, this whole name of the game is speed, how fast can I do X activity?

That’s our North Star. If we’re talking about expense parsing I literally just did this the other day, right? I dropped a PDF onto a platform and it read in the expense. Cool. Did I validate it? I didn’t actually, that’s not entirely true. I knew what the number was before I dropped it in and it happened to produce the right number.

And I was like, cool. So that’s my pseudo maker checker. Most of these platforms these days. And you made a comment before about. Trust. And trust and determinism, trust and code, right? Especially when we think about people. A lot of that trust is just based on transparency, Transparency, auditability, the ability to go back, And this is a very common paradigms in data. Like, how do I roll something back? How do I undo something? We were talking about this yesterday or the other day, right? Especially, from a version control has given us this wonderful sense of security. I can go back, I know what happened, blah, blah, blah, blah, blah, I can yell at Shane ‘cause Shane fucked it up. So we have blames. I think that, in this world where an LLM can do the work for you, again, from a cutting down time perspective, it does cut down the time. Maybe I put the document side by side with it, which is very common, of the old platforms and the new ones, Great. Look at the document. Here’s the value, here’s where the value came from now. Looking at a form, let’s just make this more complicated. ‘cause it’s fun, If it’s a financial statement, And there are a bunch of companies out there that just straight do this.

This is their business, right? Talking about AI data services. Their whole job is to take documents and PowerPoints, I won’t name names, but documents from investment funds as an example and pull off the values. Now there’s definitely an intelligent person out there saying, why the fuck don’t they just put the values out on an API?

And that’s a great question, but that would be logical and God knows none of this shit makes any sense. I know a lot of them put out these documents and they probably got fucking pictures on it and all kinds of stuff, and they’re, and hell, if they’re all the same, they’re definitely all different because why would they be the same?

That would make sense. there’s whole businesses centered around just ripping these documents, doing the OCRE, whatever. It’s one thing if it’s a table. And this kind of goes to the facts that Bloomberg stuff too. If it’s a table and you’re just like, I always want sell one, column one, row one, give me the number every time. Not super complicated, not a lot of assumptions that need to be made. And I think assumptions is one of the big things I think about when I think about LLMs and probabilistic patterns and things like that. And I can give you my convention there, but the number of assumptions it needs to make. Also relates to how much context you give it, how clearly you can describe the things it needs to do. So my usual grid of this is on one side it’s the number of decisions that you’d need to make. And then the other side is how much information you’ve given it, That’s any process.

It’s straight up any process. it’s for a human or a machine or anything at all. I could say, Shane, go make me a cake. I’ve given you no information. You have to make shit loads of assumption. You could make a cake out of mud and theoretically you’ve delivered, you’ve given me a cake, Or you could have made me a Lego cake for all I care.

That will also suffice, But that doesn’t necessarily meet what I have for expectations. So again long-winded piece here, but. If we did something more complex from a document processing perspective, A, hey, give me the revenue and the revenue’s in the cell, but then there’s four adjustments in footnotes and 12 other little things that you need to take into consideration. that’s how it gets complicated real fast. And we know in financial statements, this happens all the time. Like accounting for a lot of people seems like it’s a very route thing, and it is actually way more creative than you realize. So yeah, that’s where this stuff gets creative.

So to my process and doing things like this I always start small tasks. I always try to automate it if I can, if I’m comfortable with the security, et cetera. I always try. I have to start there to understand the bounds of what can be done or not done. And sometimes it’s process architecture to repeat that, or sometimes it’s how much information, am I clearly articulating the instructions, which is prompt engineering for lack of a better term, which is for humans, just fucking communication, which is always comical to me. I think that there’s no balance in some respect. You should always be trying to see what it can do again, I don’t think we’ve fully adapted to understanding I can use this every day. And that’s why I think it’s helpful for a lot of us to think of them as humans because it’s just oh, if I had an assistant, you’d know instantly what you’d have it do short of pick up my laundry, which it can’t do.

Shane: I think there’s there’s an interesting kind of thread in there that I’ve been thinking about, , and I’ll just unpick it. When we built our data platform, because we’ve been building data platforms for years as consultants, there were a bunch of core patterns that we knew were valuable.

And because we pay for the cloud infrastructure and cost, not our customer, we are hyperfocused on cost of that Google Cloud stuff, because it’s our money. We log everything. . Every piece of code that runs, we log the code that was run, how long it took you, we have this basically big piece of logging where we can go back and ask questions.

You, oh, are we seeing a spike on the service? Which customer you or customers is it one customer? Is it a volume problem? Is it a seasonal thing? Is it across all customers and Google are changing their pricing model, which they do on a regular basis. And so that’s just baked into our DNA. Yeah. Log the shit out of everything.

‘cause at some stage we’re gonna have to go and ask a question of those logs and we need that data. When we started moving into, AIing with our agent 80 we logged some stuff, but not everything. And as we saw our partners start to use her for really interesting use cases, they started saying, here’s some source data.

What kind of data model should I start modeling out? ‘cause we haven’t seen it. They have a piece of data and they wanna transform it. And so they’re saying, what we call transformations, we call ‘em change rules. What should the change rule look like? Talk me through how to create it.

And we just we saw those questions happening ‘cause we were logging the questions and, we started doing more context engineering to give her better access to things that gave her better answers. And what we didn’t do was when we moved to the, and air quotes here, reasoning models.

We didn’t ask her to log the reasoning path

Every time we asked her a question ourselves when we were testing something, we would ask a second question strike away of, how did you get that? Because we wanna understand where she’s grabbing the context, the prompt from. ‘cause that’s what we wanted to tweak.

And what was interesting is, our principle was log everything up, the kazoo, and yet we move into this AI world and we didn’t do it. And it’s and it’s of course once we saw. Once we saw our partners asking questions and we’re like, how the hell did she get that answer?

‘cause it’s not the one we wanted her to have. And then we ask her the same question and we get a completely different answer. We’re like, okay. And so then what do we do? We just put, into the prompt or into the framework, effectively log the path for every question you answer. And now we have a richness.

So that’s the first thing is it’s really weird how as data professionals, we have this baked in principles and patterns for our entire life. And then as soon as we move into this AI world, somehow we forget what we always do. The second point’s around complexity. And so one of the things I do when I’m coaching teams is I will ask them to draw me a map.

Draw me a map of your architecture, draw me a map of your workflow, draw me a map of your data sources. Like just draw me maps, Because I, I’m a visual person. And essentially what you say about, giving an L-L-E-M-A layout, a map, and distance between things similarity or clustering of things.

They are visual indicators as humans, we’re really good at using.

Shane: And so when I’m working with teams, the reason I want a map is the number of nodes and links. The number of boxes. The size of the map will gimme an instant identification or understanding of complexity. You have 115 source systems there that are going through 5,000 DBT pipelines.

I’m gonna go, that is a complex problem. You show me your team structure and it’s got, four layers of teams all handing off to each other and they’re in pods and squads and there’s 150 of them. You’ve got a complex business organization and team topology.

And so what’s interesting, because I was just thinking about as you’re talking about it, is if I take those reasoning paths and I basically do some simple statistics And I cluster two things, how many steps did it take that will infer the complexity of the context it’s trying to use in the task it’s trying to do.

And the second one is clustering around reusable paths where it’s constantly doing the same thing. Means that is almost a deterministic behavior, Because it’s constantly going through the same path. Where we see an edge case, an outlier, where it’s gone through a completely different path, we are like, Ooh, is that because, different question.

You just went and hallucinated for some weird ass reason. Or, yeah, there’s something interesting there because it’s different and we know different has value, which comes back to one of the things around eval, So there’s lots of work and it’s a new hot thing in the market is how do I eval my,

Yeah.

We used to call it tests, right? And, we know what data people like testing go back to my point of principles and patterns that we apply for our data work every day. And then in the AI world, we don’t, let’s talk about the ones we never apply. And now you get this whole idea of judging, So the idea of, if I asked the LLM the same question four times, and then I go and determine whether the answers are similar, and if they are the same or very close, then I should have more trust in the deterministic capability of that answer.

And that’s a presumably, I haven’t actually bothered to go and see any research papers to say if that’s true or not.

But is that what you do? Do you tend to use that as an eval process when you are doing all this work to apply AI to reduce your effort?

JT: Yeah. I always try and set up test harnesses of whatever kind, and I put evals in that same bucket. I agree. And, in. Our book club, We’re reading AI Engineering by Chip, which is a great book, talking all about the different evaluation methods and types, et cetera. I agree with the fact that I think this is a under invested area. I think it is a extraordinarily important area. And I equate it to testing in code, I think that there’s, a ton of code out there that isn’t tested for whatever ridiculous reason, And it’s always, people wanna move fast.

There’s businesses, blah, blah, blah. But, even if that means ad hoc testing, which is what you’re noting right now, run it a couple times and see the answer, that’s better than nothing. I’ve done a variety of methods now. I do LLM as a judge. I am still back and forth on this. So short of this, if you’re unfamiliar for anybody else it, it’s effectively, kinda like an adversarial, I have a system that generates something and then I have another LLM that more or less evaluates it.

I’ve done it with different LLMs evaluating others. So I might have Gemini as my model and I’ll use Claude as my evaluator or multiple evaluators, things like that. Or have it asked different questions from an evaluation standpoint, I’ve done that a variety of different ways.

That’s been very useful. I think anything where you can run your tests. At scale. Scale might be, that’s a very overloaded term. A any, anything where you can run a lot of tests, not spot tests, A lot of tests and a lot of extreme tests, I always tell people the QA joke about the bar, you know that joke, right?

Shane: I may do, but tell it in

JT: I’ll tell it anyways. I’m gonna butcher it. But the joke goes something like, somebody builds a bar, qa tester tests the bar orders one beer order, 99 beers, order a million beers, everything works fine. First guy in the door says, where’s the bathroom bar explodes? Great joke. Very appropriate in this circumstance, right? Of you don’t know what people are gonna ask. And I understand that large language models obviously are an NLP thing. It works on natural language, blah, blah, blah, blah. There’s many things in that case that are open-ended, so to say. And I think that the chat paradigm, especially as a UX paradigm, introduces a wild world of open-endedness, That, and anything can happen, And I talk to people that are building, again, AI products, and I’m like, do you want me to come to your product and ask you what my favorite color is? Because what what’s your thing gonna do then? You may have built this wonderful chat bot for finance or something, and I’m gonna walk up to it and be like, how do I make a chocolate souffle? And it might gave me a wrong answer. I might be like, this AI is terrible. I walked up to my finance AI and asked it where to drive to for lunch and it didn’t work. I was like no shit, but there’s a piece of that. I was just like, it’s not intended to work. So the open-endedness is pretty wild. But I do appreciate, there’s a bit of, this was just like almost chaos engineering, which is just I just wanna let it like just blast it with like random stuff. I also think that, having a prebuilt dataset, you should have some tests almost like unit tests to some degree. Ask it a question, you should get an answer. Then the next problem to your point is how do I evaluate whether the answer’s correct or not? And that, that is hard in and of itself. But there are conventions, ways you look for certain keywords, values, things like that. So I do think that evaluation’s massively important. It’s very easy to spot check a couple questions and just be like, yeah, it’s good. Move on with your life. And then you’re handing people free cars because they fucked with your chat bot, So it’s massively important, And I think that the more we can in a manner I speaking, beat the shit out of the machine and like literally test it to the nth degree. Even if you are using a language model, I think that’s helpful because the language model’s gonna come up with shit you probably didn’t even think of, like jack the temperature up and just tell it, ask random questions and make sure it doesn’t ask the same question twice through all I care and then you, this is our make checker a little bit can go review and say, Hey, this make any fucking sense. Or did it just spit out random garbage? I think all that’s super important. Like evaluation I think is very understated right now.

I think people are catching on a little bit, but. You said it before it’s very comical right now that we’ve got this new toy and everybody just forgets who the fuck they are and where they are, and they’re just like, yeah, cool. Let’s do it. You’re just you’re,

Yeah. You’re a seasoned professional.

Shane: Ah, And an organization. I was talking to somebody the other day and they’re in a large organization where security is key. And then they were saying that their cloud provider, somehow the LLM part of the platform got turned on for the whole organization.

JT: Ooh, fun.

Shane: Wow. And so it’s interesting how this AI thing seems to change, behavior.

People, you and again,

JT: that case might be a mistake, just like a human error.

Shane: yeah, maybe. But you do see, organizations that care about security and governance and privacy, and then you see people in those organizations grabbing things and putting them in their personal LLM. cause it’s so easy now to take a screenshot, and yes, they’re doing the wrong thing, but it’s amazing how human behavior is so different. when we talk about evals, we’ve gotta go back to complexity. Let’s just go back to data. If I have a piece of data and I have a piece of code and I have a, an assumption or an assertion I can define what the assertion is.

I can run that code against that data and the data doesn’t change, the code doesn’t change, and it can tell me whether that assertion is right or wrong. That’s three moving parts in that test suite. Why we don’t do that a lot really intrigues me. Now let’s talk about that within Gen ai.

Oh, actually no hold on. I’ve got. The data or the thing that I’m using, right? The input, PDF piece of data, I’ve got the question that I ask it, which is effectively the code to a degree and I’ve got my answer, my assertion that I expect. Ah, but actually I’ve got an LLM model, right? Which may or may not have been updated by the vendor without me knowing about it.

Ah, I’ve got a prompt. I’ve got a piece of text that’s actually embedded in that process that may or may not have changed or be interpreted differently. Ah, I’ve got rag or context, I’m pushing at some other stuff. That may or may not have changed. Ah, the PDF’s exactly the same except the dollar value for the invoice moved down 25 pixels.

Now that’s not gonna make a difference or does it? This is where we get to is that actually now if you do nodes and links for the process, we have lots more of them. And therefore, the complexity of what we’re testing the things that can change is massive. And talking about change again, one of the human natures I’ve found is once I start using a model or a tool, I get stuck.

I use chat gpt a lot for writing, because that’s what I started out with. I use perplexity for searching now. Over Google. So I perplexity. Now I don’t Google. We use Gemini for our platform because we’re bound to the Google Cloud. I use Claude a little bit for MCP testing, but I don’t use it a lot because I don’t code and I’m sitting there going, I can’t remember the last time I actually tried to decide when I would work out whether there are better models for what I want to do.

For the thing that I use chat GPT for. So how do you deal with that? Given the models are changing all the time and that the models are, tend to be good at specific tasks how often do you reevaluate the tool or model that you are using in the work that you do?

JT: not as much as I think I should. But I evaluate the new ones as they come out. So , when GPT five came out, like I reactivated my open AI subscription, I really don’t like open ai. I don’t know what it is. I maybe they, I just like, feel like they’re like evil empire or something.

I, like everybody else started using their products when they came to market and, even before chat days. And I’m not gonna be that guy that’s just oh, I’ve been doing this before. But I used that stuff before and it was interesting and I was very curious about it, especially given like the work that I do. when, Chachi PT came out, I was definitely all over it and I was very fond of it. So I will tell you that I very often use for understanding information. I use Gemini, I appreciate that one. I am a Claude fucking junkie when it comes especially to coding, And I’m a heavy cursor user, like very heavy cursor user. And I will tell you my favorite thing that has come out period, that no offense to you, Shane, but I have definitely done some work while we’ve been talking, right? And this is my favorite thing. I have my ticketing system wired up to background agents or cursor.

So like I write. Reasonably verbose tickets that explain exactly what I wanted to do. And for context, like the scope of these tickets is like, Hey, you were taking in parameters one, two, and three. I don’t want that. I want it as one parameter that looks like this. And then I want you to do this with that. That’s the scope of tickets that I’m writing because I want to be in control of it. And it’s like a micromanagement approach. I write the ticket, I say Go and it writes it. And I just keep on with my day. So I could be like walking my kid or my dog or somewhere and I’ll just fire fucking tickets off.

I’ll be working all the time. It’s amazing. That’s my favorite feature by like a absolute mile and a half. And I love Claude 4.5. I think it’s amazing. And the code quality’s phenomenal and I’m probably paying them a disgusting amount of money. But it’s, I think it’s totally worth it right now because I can move epically fast and multitask.

so Gemini for information, Claude for code open AI for nothing unless I like, just want an alternative opinion. And I definitely consider them all like different people to some degree. I very much think of them like people, they all have their different quirks .

Some things are good, some things are bad. Some things are like, especially from a code perspective, some of the models are more overeager than others. Some of them wanna try and cover more edge cases than others. And I, sometimes I’m just like, no, just fucking do what I told you to do. I don’t need you to do 20 other things.

And I have all kinds of rules set up. So I have it set up in a way that it will do the extra things I told it to do and not the things I don’t want to do to some degree, which is very nice. I’ll make a separate point. And I’ve always been this way for better or for worse, but a healthy degree of paranoia I think is always good as data professionals, like healthy paranoia. There is unhealthy paranoia, but healthy paranoia. I am a big believer and I’ve met many people on my career and I think a lot of us agree with that level of just distrust, inherent distrust, And it’s not to say that we don’t trust everybody. I actually think I’m a reasonably trusting person, which I’m sure someone could take advantage of me just hearing that. But I still look up the things that I asked the LLM to do, to put this into context. I still crosscheck things. When I write code with LLMs, I read all of it, like all of it, I see my role as I am the reviewer. I am the checker, I always joke with people, it writes faster than me. That’s why I use it. It types faster than me, but I am always reviewing what it did. If it does extra things, great brownie points.

Take some, extra credits so you can, whatever, I’m very content with that. But I absolutely read everything as much as humanly possible. And sure, I do random experiments where I just tell it, build me some shit and I don’t really care. And that’s a fun experiment. But I’ve worked on a project recently that I’ll tell you about where, without going into too much detail, the project wrote something in a language I’m less familiar with and I read some of it and I definitely spent some time. I used an LLM to do this and to teach me about the other libraries, teach me about the other things that I’m just not as familiar with. So I spent some time doing that in, different LLMs, different context windows. So like hopefully there, there’s some, call it arb that’s happening there where, I’m hopefully getting the right information.

I also do Google things still. I don’t perplexity things, I think I perplexity things just sometimes ‘cause I wanna see what the answer is. But I do still Google shit. Cause , I’m not gonna trust it unless it’s like if it, if I asked it to write something in Beam, I better, I’m gonna go to the fucking beam website and check to see that it did it right.

And especially from a code perspective. And they may have fixed this, but my favorite like red flag reading somebody else’s generated code. One of my favorite red flags. There’s plenty of them. The comments and all this other shit. My favorite one for a long time was that, especially when you had it generate an AI model or something that used the language model, it usually wasn’t trained on enough information to know what the latest model was.

So it was always a really big dead giveaway when you generated code to use like Gemini, and it puts 1.5 in there. And I’m like, you didn’t read this I know you didn’t read this because it’s using a model that we all know is a year or two old now, but you didn’t spend any time just looking like that’s a really basic thing to look at. And you didn’t even look at it, , I usually demerits for that person. That’s a very glaring thing for me because it’s just read the code. It’s not that hard, I promise. It’s not that even if you skim it, it’s not that bad, it, it’s better to do that.

So to the point of do I evaluate the tools, do I do that? Yeah I absolutely still do. I definitely still spend time Learning about the stuff outside of my use of it as well. So that way again, same as a person, Even if you meet the smartest person on the planet, I still wanna have my own authority.

I still want to ha fact check people. This, maybe there’s a very existential philosophical comment here about between fake news and the internet and or the web, sorry, Juan, the web, So I’m a big proponent of cross validating things. I just believe that and sure, not everybody has time to do shit like that.

I definitely don’t do it for everything. There’s some things I just take face value, but I do believe in that, And I have, I have a little kid right now and I’m in the phase of why, which I love, I really love this

phase. I’m sorry, my, I think my wife hates it, but I love it because. he asks why he’s not at the annoying why phase, but he asks why for a lot of things. And I take it as an opportunity to go look them up

and we talk about things. And if he asks a subsequent why, I’ll look that up too. I don’t give a fuck. I love looking things up. But you, you should feel more empowered to do so. and I feel like people just are short attention span. They’re like, just don’t fucking care.

Shane: I think you’re right. I think that idea of trust, but verify and, coming back to your Gemini 1.5, they probably don’t know there is a Gini 2.5 because they haven’t done the work, They haven’t done the reps to become an expert or any experienced in it to know that actually there’s a difference between 2.5 PRO and 2.5.Flash.

JT: Yep.

Or flashlight, or there’s flash with dates after them.

Like there, there’s all different fucking shit.

Shane: Yeah, so just wanna look at time. I just wanna close out with two questions for you. So right at the beginning we were talking about the lawyer stuff and,

Expertise and tribal knowledge. And given that all the LMS have gone and stolen tribal knowledge that was publicly available and often privately available,do you reckon the world’s gonna move, that humans who have tribal knowledge are gonna stop documenting it in a way that it can be found?

Because actuallythe value now is tribal knowledge.

JT: I wanna say no from like a humanity standpoint, but I do think that, think the world will become a lot more polarized. I’m gonna explain this in a weird way, and I’m not intending it. I know that there’s a political backdrop and everything as we’re recording this, but if the model inherently gravitates towards the mean, always gravitates towards stereotypes, that kind of thing, and we are having a feedback loop to ourselves on this, our own opinions are going to be potentially squeezed, Like kurtosis and all that fun stuff, if you wanna get into that. Do I think that people will stop? I think people will use these tools in more automated fashions and then that will create its own probably problematic feedback loop. I do, however, think that the pendulum will inevitably swing the other way where. I think people will still use these tools, still use tech to create content that maybe they didn’t record before. I don’t have a substack or any of this stuff. Yes, I’m talking to somebody about recording a podcast, which is why the fuck not even though we’re on or no. It’s mostly from I wanna help their business kind of thing.

But I think that hopefully technology will lower the barriers of entry to recording more. I hope that inevitably forces the pendulum to swing the other way at some point in time. And I remember reading an article recently, something about someone saying how there’s no second internet, there’s no second web. Really, again, I apologize. I like Juan’s perspective on this. It is the web, not the internet. And I I, I make actually a mental note, like always to correct myself. So thank you Juan for that. But there, there’s no second web like we’ve trained these things on so much information. At some point in time I do think we’ll start recording more and then there’s a very interesting dynamic of, okay, now everything’s recorded. To your point, tribal knowledge at all. What’s left? I’d like, and maybe this is overly optimistic and potentially naive, but I’d like to think that if we continue to use these tools of whatever capacity, we’ll advance our knowledge faster and therefore there will be more knowledge and more interesting ideas and creative ideas and I’ll emphasize something different that is related to this question, which is. I am a huge proponent of the fact that I think that especially as the pendulum swings back and forth, I am very bullish on the arts, to your point of painting earlier, Art is inherently creative. It’s inherently random. It’s inherently wild. There’s no predicting that you can generate pictures and stuff.

We all know how shitty those come out. And some of them come out great, but for better or for worse, art is something I don’t think a lot of us understand. We still don’t understand how we work. And I think that in comparison to science and science, I apologize to anybody that’s in science. I’m not trying to demean your field, but I think that I’m gonna call sciences things that are route in some nature.

They have predictive process. End results, so on and so forth. So I’m very bullish on art and I think the sciences, quote unquote, are gonna be marginalized away, and that’s part of this whole cycle in knowledge, et cetera, et cetera, as it learns more. But I do hope that technology encourages more people to, maybe just do random shit, write down more, write down random ideas, write down everything. I don’t think that we’ve gotten to a point where all thought has been explored, right? That’s what we’re talking about to some degree. or all possibilities of thought or captured.

Have you seen Jepson stuff?

Shane: nah.

JT: Look, okay. I will plug Vox right now because I, it’s one of those things that you watch it and you’re like, whoa, like this is cool because. He’s trying to replicate, like evolution, an evolutionary thought as like part of his process. And it’s like a, I’m sorry Jepson if you’re hearing this, but it’s like a very sophisticated Monte Carlo in some respects. It’s try everything. Or random forest, whatever you wanna say. And it’s way more sophisticated than that. I’m generalizing here, oversimplifying, but I do think that there’s something that we have that the machine doesn’t yet in its creativity and that’s art. And I think that’s, and I hope that there’s a pendulum swing. And don’t get me wrong, I think that the vast majority of society is gonna gravitate towards me and it’s gonna be some massive in acidification of fucking everything.

But I do think that for people that are creative thinkers like that, this is the time.

Shane: I agree. I I think about data products without defining them or defining the definition of defining. And I’ve reviewed two or three that are currently being drafted around that. Some are what I would call bringing product thinking to data, and some are around a thing called data product.

That’s how I categorize them after reading the draft content. I know a couple of other people that are starting to write something around product thinking with data or data products as a thing. And my view is good. The fact that there’ll be five or 10 books in a similar subject space, the way they’re telling the stories is different.

the storytelling is different for every book, even though it is relatively in the same domain or subject.

And that’s great. ‘cause what that means is, they can write them ‘cause writing is easier now, and they can publish them because publishing is easier now. And then I can read them and I always pick up something new, something I didn’t know, something that entertains me or educates me.

And then I assimilate that into how I think, so I’m with you push more out. But I think we will see some people try to put paywalls up because that’s the trouble and knowledge is how they make money. interesting times. I think the other one and we’re outta time for this one, but I think given the fact that tokens are subsidized so heavily at the moment And we use the AI tools or the gene AI tools. To automate stuff for us because it is faster and it is cheap enough that we don’t have to care. It’s gonna change. And when we get out with the true cost of those things, then people who have built good automation who have built things that are efficient and optimized and aren’t lazy processes or lazy code they’re gonna be better.

And the people that just put a thousand PDFs in there with no context, no prompts and get an answer, probably your products will be one of the many AI startups that die a thousand deaths,

JT: Or maybe gets bought.

Shane: Yeah. And then let’s say, yeah, there is a difference between being bought and being Acqua hired.

JT: Valid. Valid.

Shane: So yeah. Again, what’s your definition of buying? Or what’s your

JT: Yeah.

Shane: of different lighting? alright.

JT: you put a flame to the, yeah.

Shane: Just to close it out if people wanna find you and find what you are thinking you’ve already said you don’t have a Substack, you might have a podcast

JT: Website. LinkedIn’s the easiest way to find me.

Shane: LinkedIn and Practical Data Discord. Because

you are one of the more active people in the community. That’s how we met.

JT: correct.

Shane: join us. It’s free.

JT: It’s a great, it’s a great community. I always advertise it to people. I’m like, just yeah who do you talk to about your business? Come talk to us.

Shane: Yeah. And we will talk back.

JT: Sometimes we might just tell you to fuck off, but that’s fine.

It’s also cool.

We’re very unhinged.

Shane: You might get a meme you’re gonna get a meme

JT: we’re a very unhinged group.

Shane: Thanks for that. That’s been an interesting chat. We kinda went all over the place, it’s good to talk to somebody that’s actually using AI to build data services to in production and ex and monetizing it as a, as, as much as possible, ‘cause that is an art or a science one of the two.

JT: art.

Shane: Alright, excellent. I hope everybody has a simply magical day. I.

«oo»

Stakeholder - “Thats not what I wanted!”
Data Team - “But thats what you asked for!”

Struggling to gather data requirements and constantly hearing the conversation above?

Want to learn how to capture data and information requirements in a repeatable way so stakeholders love them and data teams can build from them, by using the Information Product Canvas.

Have I got the book for you!

Start your journey to a new Agile Data Way of Working.

The Agile Data Big Book of Ways of Working

«oo»

Business Language Driven

Shagility — Fri, 09 Jan 2026 15:45:29 GMT

Few things are really new, sometimes we just need to remember the valuable things we used to do in our previous Ways of Working.

DORO : Define Once, Reuse Often

As I prep my presentation for Data Day Texas in a few weeks and also work on content for both the Pink and Blue Books I find that I am browsing, referring and reusing patterns and patterns templates that I have found, refined or defined over the last decade or two.

My method of storing these are ad-hoc at best, I think this year I will try to put them all into a Obsidian repository and then hooking up something like Claude to see if it makes finding and reusing things a little easier.

The results of that experiment will be a post for a different day, for now I find browsing the slides I have used for the many presentation, training and mentoring sessions I have delivered over the decade the best place to find things I can potentially reuse.

And as I do that I realise there are a few foundational slides that I use repeatably.

Some of these are the “maps” I have documented briefly here:

Intro - Agile Data Way of Working Blueprints

A big NO to frameworks and methodologies from me…

a year ago · 3 likes · Shagility

The Agile Data Big Book of Ways of Working

Problem People Diagrams

When I start out developing a new training course, a presentation for a conference, putting together a mentoring session or starting on content for a new “an Agile Data Guide” I like to think about the core problem I am trying to help solve…

5 months ago · 2 likes · Shagility

Business Language Driven

Another piece of content I reuse a lot is this slide:

I use this slide in almost every presentation I do that relates to data work or data modeling.

From memory this slide was based on content from Lawrence Corr that we reused as part of the one day “BEAM” course we developed and delivered as part of OptimalBI in NZ, the data and analytics consulting company I founded a few decades ago, .

It shows a fundamental principle that we apply as part of the Agile Data Way of Working.

The principle is we work with data by first focussing on the Business Language and then focussing on the source systems data structure or reporting needs after that.

I have talked to this slide many many times, but never actually created a reusable description for the principle itself.

The principle of being Business Language Driven

So let’s have a go at creating an initial definition now.

Using the Pattern of the Agile Manifesto

Here is a definition of the principle using the pattern from the Agile Manifesto:

Business language grounded in business reality
The highest priority is to define and deliver information using the language of the business as it actually operates, so actions are based on shared understanding rather than system or technical abstractions.

Using the Pattern of the Agile Data Blueprint

A component of the Agile Data Blueprint I deliver for organisations is a list of Data Management principles. These are documented using a pattern I think I got from TOGAF.

Here is defintion of the principle using that pattern:

Name
Business Language Driven

Statement
Information, data and actions are expressed in the language of the business as it actually operates, not in the language of systems or technology.

Rationale
Using business language anchors data work in business reality, reducing misinterpretation, rework, and translation overhead between teams. When data work reflects real business concepts and behaviour, teams align faster, reduce translation errors, and achieve better outcomes.

Implications
Stakeholders must invest time to assist in clearly identifying, defining and agreeing core business concepts and core business processes, while Data Teams must design, model and name data using those definitions consistently. This may require additional upfront collaboration and discipline, but it lowers long-term costs by simplifying communication, improving trust, and increasing reuse across analytics, reporting, and automation.

*Both these definitions were created with the assistance of my ChatGPT friend for the initial version and then edited by me.

IMHO both these definitions still need a few more iterations to tighten them up.

Information Value Stream

If we think about the pattern of Business Language Driven from the lens of the Information Value Stream:

The source system data structures and the last mile reporting requirements all sit on the right of this diagram, ideally in the Design step, but often in the Build step.

We want Data Teams to start by focussing on the left hand side of the Value Stream.

I keep wanting to use the term “Data Product Team” instead of “Data Team” as I write this content but in reality it is just Data Teams applying product patterns and pattern templates. They are still a Data Team.

We want them to understand the Problem that needs to be solved.
We want them to Ideate with Stakeholders on the options to potentially solve it.
We want them Discover which of those Ideas is the most Valuable, Usable, Viable, and Feasible.
We then want to prioritise the value of solving this problem compared to the value of all the other problems we could solve first.

To do these things we need to use Business Language not the language of the source system structures or the last mile reporting requirements.

Stakeholders either don’t know, understand or care about the source system data structure.

They understand and care about the Business Reality. They speak in the language of the business, not the language of the data structures or the technology.

With help Stakeholders might be able to articulate the Core Business Concepts and Core Business Processes that are contained within that Business Reality. Data Teams should count themselves lucky if they do.

Different data practitioner personas, different language of requirements

In my experience each of the data practitioner personas typically have a language to capture requirements based on their preferred way to work.

Personas that align with that of a Data Engineer will often start the process of understanding the requirements by wanting to review the structure of the Source Systems which the data that is needed is created within.
Personas that align with that of a Data Scientist will often want to follow a pattern based on Exploratory Data Analysis (EDA) starting by exploring the data values themselves.
Personas that align with that of a BI Developer will often want to start by wire-framing the final Information Product delivery type, for example a Dashboard.
Personas that align with that of a Business Analyst will often want to start with documenting and understanding the business processes.

Stakeholders will often align with the persona that best matches their career path or will default to wanting to start by describing the final Information Product output, such as the Report or Dashboard they want to use.

Data Teams have trained Stakeholders wrong

One of the mistakes we have made as data practitioners over the last few decades is teaching Stakeholders to talk to us in the language of Reports and Dashboards.

When Data Teams ask a Stakeholder what their “requirements” are, Stakeholders will often describe the Dashboards they think they need delivered. They are giving the Data Team a solution as a requirement not the problem to be solved.

That is not the Stakeholders fault.

Data Teams have trained them to think in the language of Dashboards, as that has been the hammer they have delivered to hit every little nail, aka every problem the Stakeholders have.

Data Teams have trained Stakeholders to be fluent in the language of the data platforms most common last mile delivery type, rather than the Data Team learning a new language.

This focus on legacy dashboards to define requirements and deliver solutions will bite Data Teams in 2026 as data platforms move away from legacy BI tools that build multi use Dashboards that are reused to solve multiple business problems and towards GenAI generated “One shot” BI Apps that solve a single problem well, and is not reused for anything else.

Missing T skills

Another common problem that drives Data Teams to focus on source system structures or reporting requirements, rather than Business Language and Business Reality is the loss of “Business Analysis” skills from the core Data Teams.

I wrote about Skills vs Roles a while ago:

The Agile Data Big Book of Ways of Working

Team - Skills vs Roles

It is a silo problem…

3 years ago · Shagility

And as part of that I talked about the shape of skills in a data team.

*I think its probably about time to review this list of skills and see if it needs to be iterated.

If you look at Data Team members who have traditionally have a role of Data Engineer or BI Developer, you will see skills similar to the diagram below.

Compare it to the typical skills for somebody that typically had a role of Business Analyst below.

People with a Business Analysis background and these set of skills typically engage with Stakeholders a lot more than people with an engineering background and as a result seem to use business language more often.

When I am working with a data team for the first time, I can often guess the language that team will use based on the primary skills or roles in the team.

Or I can review the things they produce and the language used within those things and guess at the primary roles and skills within that team.

Metadata Driven Information Factories

When I talk to data practitioners about being Business Language driven, they will often jump straight to the pattern of metadata-driven development.

As I was writing this article I did a LinkedIN poll on this.

https://www.linkedin.com/posts/shagility_as-a-data-practitioner-are-you-code-driven-activity-7414943978749480960-EgPO

In the post I wrote:

As a data practitioner are you code driven or metadata driven in your way of working?

Do you write code that creates the data structures you need.

- And then use those structures or that code to create the “map” of the data model those structures represent.

Or do you define the data structures as “metadata” and have code that reads that metadata and then automatically creates the structures.

- And then use that metadata to create the “map” of the data model those structures represent.

(I tried to abstract a whole bunch of different context languages into 2 simple patterns above, did a so so job)

I have been in the data domain for enough decades to have watched metadata driven data tools emerge and die with each new technology wave.

Our AgileData.cloud product is a form of a metadata driven capability, with a sprinkling of business language.

As the pattern for Context Planes start to emerge to support GenAI patterns, I think we will see these tools remerge.

But they will contain Context in the form of Business Language rather than metadata in a language of technology.

I look forward to the day we can capture this Context using the language of the business reality, and that powers the creation of the other languages for us automagically.

In Summary

One of the reasons I write is to help me think.

Sometimes I can sit down and write quickly and with clarity, as its something I have thought about for a long while and something I have taught other people.

Sometimes it is a new set of patterns or pattern templates and I yet to have clarity, so i write to think and iterate towards that clarity.

When its the latter I use a constraints model, I time box the content, to stop me going down endless rabbit holes.

This article was deffo the latter.

But the intersting thing is its based on a slide, a piece of content, a pattern, I have used for years.

So the fact that I still don’t have clarity on that pattern, is a surprise to me.

And means it probably never resonated with anybody else when I used it due to that lack of clarity.

In the famous words of the Terminator “ill be back” on this one.

Can AI tools bring back data modeling with Andy Cutler

Shagility — Tue, 23 Dec 2025 19:21:57 GMT

Join Shane Gibson as he chats with Andy Cutler about the art of data modeling and the potential of AI tools to improve the art.

Listen
View MindMap
Read AI Summary
Read Transcript

Listen

Listen on all good podcast hosts or over at:

https://podcast.agiledata.io/e/can-ai-tools-bring-back-data-modeling-with-andy-cutler-episode-78/

Listen to the Agile Data Podcast Episode

Subscribe: Apple Podcast | Spotify | Google Podcast | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser | Deezer | Podcast Addict |

You can get in touch with Andy via LinkedIn or over at https://linktr.ee/andycutler

Tired of vague data requests and endless requirement meetings? The Information Product Canvas helps you get clarity in 30 minutes or less?

Google NotebookLM Mindmap

Google NoteBookLM Briefing

Executive Summary

This document synthesizes a discussion between data experts Shane Gibson and Andy Cutler, focusing on the persistent challenges and future direction of data modeling within the modern data landscape. The central argument is that while data technology has rapidly evolved toward accessible, powerful cloud platforms, the discipline of data modeling has been neglected, creating a significant knowledge and practice gap.

The conversation identifies a “repeated and constant battle” to prioritize modeling over the immediate appeal of technology, which provides instant feedback that modeling processes lack. This issue is compounded by a decline in traditional mentorship, where senior practitioners historically guided newcomers. Modern data platforms from major vendors like Microsoft, Snowflake, and Databricks are criticized for lacking integrated, opinionated tools that guide users through modeling processes, forcing practitioners to manually implement common patterns like Slowly Changing Dimensions (SCD) Type 2.

The primary conclusion is that Artificial Intelligence, particularly Large Language Models (LLMs), presents a transformative solution. AI is positioned not merely as a code generator but as a new form of mentor and assistant. It can educate novices, generate starter models, and, crucially, act as an “antagonistic” agent to stress-test models for future flexibility—replicating the critical feedback once provided by experienced data modelers. The effectiveness of these AI tools, however, hinges on providing them with opinionated constraints and clear business context to generate practical, fit-for-purpose models rather than theoretical, unimplementable ones.

The Evolution of Data Platforms and Recurring Patterns

The discussion begins by contextualizing the current data landscape within a 25-year evolution of technology. This history highlights a recurring cycle of platform development and a significant shift from capital-intensive, on-premises infrastructure to flexible, cloud-based services.

From On-Premises to Cloud: The journey is traced from early-2000s technologies like ColdFusion and SQL Server 2000, which required purchasing and managing physical hardware (e.g., “compact three eight sixes”), to the advent of the first cloud data warehouses like AWS Redshift, and finally to modern platforms like Snowflake, Databricks, and Microsoft Fabric.
Democratization of Compute: This shift democratized access to powerful computing resources, moving from multi-thousand-dollar hardware purchases to pay-as-you-go cloud services.
Recurring Cycles: A pattern is noted where the industry moves from installable software to pre-configured appliances and now to cloud-native databases. Despite these technological waves, fundamental challenges, particularly in data modeling, reappear. As Gibson notes, “every technology wave it seems to become hot and then cold.”

The Persistent Challenge of Data Modeling

A core theme is the struggle to maintain the discipline of data modeling in the face of rapid technological advancement. It is often seen as a difficult, time-consuming process that lacks the immediate gratification of working with new tools.

A Constant Battle: Andy Cutler describes a “repeated and constant battle to make sure that data modeling is at the forefront of a data platform project.” He argues that modeling is frequently deprioritized in favor of focusing on technology.
Architecture vs. Modeling: A common point of confusion is the conflation of data architecture patterns with data modeling patterns. Cutler clarifies this distinction: “The architecture enables the modeling. The modeling is put over the architecture.” He notes that patterns like the Medallion Architecture are data layout patterns, not a substitute for disciplined modeling techniques like Kimball or Data Vault.
The Lack of Instant Feedback: A key insight is that technology provides immediate, binary feedback (it works or it doesn’t), which is psychologically rewarding. Data modeling, in contrast, does not. As Gibson puts it, “I can’t get instantaneous feedback that my model is good or bad or right or wrong... a model that you’ve created six months, a year down the line when all of a sudden something happens... the model isn’t flexible enough.” This delayed feedback loop makes technology more appealing to practitioners.

The Decline of Mentorship and the Knowledge Gap

The conversation highlights a critical loss of institutional knowledge transfer. As tools have become more accessible and projects faster-paced, the traditional mentorship structures that trained previous generations of data professionals have eroded.

The “Grumpy Old DBA”: Learning was often driven by experienced seniors, colloquially the “grumpy old DBA,” who provided critical feedback and guidance on performance, design, and best practices. This hierarchy of mentoring was essential on expensive projects where mistakes were costly.
Erosion of Foundational Concepts: With modern, abstracted tools, new practitioners are often not exposed to foundational concepts. The example cited is a user asking, “what is data persistence?”—a concept ingrained in older professionals who used tools that required manual saving (e.g., pre-cloud Excel).
Lack of Accessible Learning Resources: While foundational books from authors like Steve Hoberman and The Kimball Group still exist, formal courses and guided learning paths for modeling are less prevalent. Unless actively guided to these resources, newcomers may not discover them.

The Inadequacy of Modern Data Modeling Tools

A significant contributor to the modeling gap is the lack of robust, integrated, and opinionated modeling tools within major data platforms.

Vendor Agnosticism: Vendors like Microsoft, Databricks, and Snowflake avoid baking specific modeling methodologies into their platforms. They provide a “canvas” and “paintbrush” but “don’t help you draw the picture.” This forces users to bring their own process and often use disconnected, third-party tools.
The SCD Type 2 Example: The implementation of Slowly Changing Dimension (SCD) Type 2 is a prime example of a common, well-defined modeling pattern that largely lacks out-of-the-box support. Practitioners are still required to write custom code to handle historical tracking, even though it’s a fundamental requirement in dimensional modeling. Databricks (Delta Live Tables) and dbt (Snapshots) are noted as exceptions that offer some built-in functionality.
From Conceptual to Physical: There is a lack of end-to-end tooling within platforms like Microsoft Fabric that facilitates the entire modeling lifecycle, from conceptual design through logical design to the automated generation of the physical model.

Artificial Intelligence as the Future of Data Modeling

The discussion concludes that AI, particularly in the form of specialized LLMs, is poised to fill the void left by declining mentorship and inadequate tooling. AI can act as an expert assistant, a sounding board, and a critical partner throughout the modeling process.

AI as Educator and Mentor: For those new to the field, AI can act as a guide, explaining different modeling patterns (e.g., Dimensional, Data Vault, Third Normal Form) and helping to translate business requirements into an initial model. This helps bridge the knowledge gap. The tool Ellie AI is mentioned as a specific example of an LLM-powered tool focused on guiding users through data modeling.
From Generation to Antagonism: The most powerful application of AI is not just in generating a model, but in stress-testing it. The concept of using an AI to be “antagonistic” is raised, where the user can prompt it to find weaknesses and potential future problems.
The Power of Opinionated AI: An unconstrained LLM may default to the most prevalent pattern in its training data (likely Kimball modeling, due to the volume of public content). The true value emerges when the AI is given specific constraints and opinions. Key inputs that improve AI model generation include:
- Source Context: Providing the AI with source schemas and metadata.
- Design Patterns: Instructing the AI to use a specific, opinionated modeling pattern (e.g., “concepts, details, and events”).
- Business Boundaries: Using artifacts like an “information product canvas” to define the specific business outcomes the model must support, preventing it from over-engineering.
Multi-Agent Approach: A proposed advanced approach involves using multiple AI agents with different perspectives (e.g., one focusing on source systems, one on business processes, one on reporting outcomes) and having them “antagonize each other” to arrive at an optimal, pragmatic model that balances all constraints. This mimics the cognitive process of an experienced human modeler.

Tired of vague data requests and endless requirement meetings? The Information Product Canvas helps you get clarity in 30 minutes or less?

Transcript

Shane: Welcome to the Agile Data Podcast. I’m Shane Gibson.

Andy: And I’m Andy Cutler.

Shane: Hey, Andy. Thanks for coming on the show. Today we’re gonna have an intriguing track around data modeling and whether it’s been done not being done, gonna be done and how tools and AI can fit into, it’s it’s a passion of mine. But before we jump into that conversation, why don’t you give a bit of background about yourself to the audience.

Andy: Yeah, sure. Thanks Shane. so my name’s Andy Cutler, and I’ve been working. In the data space for about 25 years. So back in 1999 when I was working for a tiny little UK drum and bass record label called good Looking Records I was given the role of updating the company website and that ran on a technology called ColdFusion and I got quite into web design.

So I then managed to get myself a job, and I guess it was the first IT job that I had designing websites. Then that website agency bought a SQL Server 2000 license, and I know there’s older versions of sql. I’m, really aware of that. But that was the first version of SQL that I got my hands on, and I was tasked with building the data models and the store procedures to run CMS systems.

And then five years later, so mid two thousands, I am working in the data warehousing space. So I’d learned everything about database normalization, third normal form, so on and so forth. And then I was told to unlearn that all because I needed to de-normalized everything for data warehouses. So I’ve literally worked for the last 20 years in data warehousing.

The last few years has all been cloud-based. I actually started my cloud journey with AWS and Redshift. That was, when we’re going on for sort of 12, 13 years ago now. And then when Azure really started to motor with the products and services, I was then using tools like Azure, SQL data Warehouse.

That then moved into Synapse Analytics and for the last couple of years it’s been Microsoft Fabric. So really it’s been predominantly the Microsoft data space that I’ve been working in. .

Shane: Excellent ColdFusion and SQL server in the early days. That was back in the days where we had to buy our own hardware and put it under our desk or in a cupboard or, pre-data center. I remember back then I, we were doing some stuff and my first job and, we were buying compact three eight sixes and the argument was do we get an SX or a dx?

And I, I can’t remember. I think. Had , four mega memory and it cost the organization $35,000 New Zealand dollars back then. Yeah, times have changed. Right now you can spin up those kind of things by just putting your credit card in or even getting your free tier and . You got a massive amount of firepower in there.

Andy: Yeah, and that’s the thing is it’s also a little bit disconcerting now because all that compute is sitting in the cloud. Certain vendors like to show compute in certain ways and show you cause and other vendors, they like to obfuscate that behind other kind of terminology, which yeah, guess Microsoft do.

So you’re trying to map the cloud compute with what you’re doing with on premises, right? And saying, okay, I’ve got a certain amount of cause and I’m running this certain amount of workload on premises. What does that look like when. I go to the cloud and I’m then dealing with a service that I can’t exactly a hundred percent map to those on-premises calls, but yeah, I to, I totally get it.

You’re choosing, hardware to run that software. You’re configuring that software as well. You’re configuring that software to death to get as much as you can out of that hardware.

Shane: Yeah. And I think like you, I, when I had my consulting company, when Redshift and AWS came in we jumped on board really fast. ‘cause in those days, you’ve had to buy a big million dollar Teradata box or you had to buy some Oracle database and again, rack and stack it with some leased hardware. And when Redshift turned up, that really was the first cloud database for analytics. And it’s interesting how, it’s lost market share. It was first to market and it’s obviously been taken over by Snowflake and Databricks and a few others. So what’s interesting for me is we see these patterns getting repeated.

We see, databases where we used to have to install them, and then we see databases as an appliance where you buy the hardware and the database used to come with it pre-installed. Now we’re seeing cloud databases, Redshift being the first and some more oh, should I say, modern. So ones that solve some of the problems I’m gonna have to vacuum your database for anybody that’s dealt with a Redshift cluster before. and so one of the things that’s interesting is this idea of data modeling. Because, I’ve been in the data space for 35 years and we’ve always modeled data, but every technology wave it seems to become hot and then cold. It’s yep we model, and then no we democratize.

And people can do all the work without any conscious data modeling. So what are you seeing in the modding space and in your part of the world at the moment?

Andy: I am seeing, and I’m gonna speak honestly, and off the cuff here. I’m seeing a repeated and constant battle to make sure that data modeling is at the forefront of a data platform project. That it isn’t just about the technology and it isn’t just about the data layout patterns that we’ve seen.

One of my sort of bug bears is that I’ve seen a lot of articles on LinkedIn comparing architecture patterns with data modeling patterns, and I’m thinking, hang on, you are really comparing apples to oranges. The architecture enables the modeling. The modeling is put over the architecture. I can’t imagine a scenario where any CIO or CTO is gonna talk to, members of their team and ask them, should we architect our data a certain way versus modeling it a certain way?

No. They are complimentary. So in the last few weeks, I’ve been thinking about the data modeling side of things and asking myself is it because it’s hard to do? Is it because it requires thought and time and collaboration? Where sometimes technology is a little bit of an easier thing to do.

There’s still lots of human elements in using and deploying and working with technology, but I just feel that data modeling, you really have to have that community grounding where people are working together to get what they need from the data into a shape that’s actually useful for their business. So yes, that was my framing on that, .

Shane: It’s an interesting lens and for me I’m with you, right? I talk about. Data architecture layers. . So how we are laying out the architecture of our data across our platform. And I’m a great fan of layered architectures. I’ve used them for many years and I see massive value in them. And then once I have , an idea of the layers, then I can talk about what data modeling patterns we want to use in each layer. And I’m a big fan of mixed model arts, which Joe Reese talks about in my experience, I’ve never actually used a single data modeling Pattern. Even back when I was dimensionally modeling I always had a persistent staging area that was some form of native relational model that met the structure of the source system before I moved it into the dimensional model. So if I think about that in terms of data architecture, and then I think in terms about data modeling, then I think in terms of technology, and I asked myself this question if I was moving into the data domain. For the first time at the beginning of my career, where do I find any information around data modeling? Like how do I learn? Because the courses and the books that we had when we started have the books are still there, but the courses have disappeared to a degree. I think part of the problem is lack of accessibility , to that content for some reason. But the other thing that you just raised, and I’ve never thought about it this way, is as a technologist, if I want to earn a product, the product, I give it a go, I read some of the documentation. Like these days I’ll probably perplexity it. And I get to have a go and I get immediate feedback. So I can, install the software or turn it on in the cloud, and then I’m immediately able to log on and then I can give it a whack. I can try to use it and it will give me feedback. If I’m doing that with data modeling that’s not true. I can’t just turn something on. I can’t get instantaneous feedback that my model is good or bad or right or wrong or doing what I want. And yeah. It’s an interesting point you raised. Maybe that is the reason that people love to play with technology because they can just learn it against an instantaneous feedback and probably that adrenaline rush.

That first time you load the data up and your dashboard turns up with a pretty graph, you’re like, eh, that was pretty cool. That’s a bit of an endorphin hit. Whereas I created a data model in Miro. Yeah. That’s a picture. It’s interesting lens.

You’ve got do you agree that one of the problems is lack of access to content and ways of learning what data modeling is and how you use it?

Andy: I think it always has to be a constant conversation with the data people around modeling, and it always has to be brought up, but it always has to be. Marketed and communicated that the technology is just the starting point. Modeling comes into it as well. You are delivering a product that is the amalgamation of the technology.

Of course, it is the process that the organization requires that data technology to do, and you’ve got there through modeling and landing the data in the right way for that organization. If we go back to resources, and you touched on a point there around the books that we had access to and the resources that we had.

So yeah, I remember Steve Hoberman’s data modeling books. I then remember. The Kimball Group books. I mean they were, and I’m, looking at the Kimball reader on my shelf right now and the Dimensional modeling toolkit because , yes. You required database technology to implement that model, but the technology could be from Microsoft, it could be from Oracle, MySQL, Postgres.

It didn’t need to be a specific vendor. You were applying a process to a technology. So unless you are guided towards these resources, and I was told, I remember being told in the early two thousands to go, I think it was a database administrator who told me to go and learn about normal form.

Database normalization because the data had to be structured in a certain way to facilitate a transactional system, right? Database normalization. And then when I started to move into more of the data warehousing and analytics side of things, again, I was guided. I was told it was a MicroStrategy consultant that I was shadowing so that I could understand the data warehouse that he was implementing.

Again, that consultant pointed me in the right direction. He pointed me to towards books and said I’ll, I can teach you the basics of this now, but of course you are going to have to get hands on. You’re gonna have to learn this. And at at some stage, I was told that I would have to learn these things and.

Get hands on. And I think that unless you’ve got those people now who are guiding people to do that, people have gotta be very proactive and they’ve gotta be going out there and they’ve gotta be saying, okay, I’m building a data platform. I’m using this technology.

What else do I need in terms of my knowledge that I need to apply to that as well? So there is a certain amount of guidance, and this is, I hope, being passed down by people. There are, great people out there like Johnny Winter who are talking about data modeling and, we’ll get onto the data modeling AI topic in a little bit that I.

Essentially took from Johnny, right? Johnny was talking about this and I looked at this and thought, wow, this is, yes, this is relevant. I like this. But yeah, so that’s what I feel. The other thing is, if you look at vendors, are vendors necessarily pointing people in the direction of the modeling processes?

I’m a Microsoft data developer, and throughout the years there have been various books around how to apply things like dimensional modeling, a modeling patterns to Microsoft products, right? And even now, you can go onto the Microsoft Fabric learn documentation, and in the warehouse, the fabric warehouse documentation, there’s resources around dimensional modeling.

They will tell you how you can do that.

So the vendors, like I said, have been providing some of this guidance and some of this documentation, but it’s very it’s small fry, right? It’s a few pages in their documentation on how to do it. So you’ve gotta, you’ve gotta be mindful of the fact that when you are coming at these data platforms, that you’ve got to be doing the modeling behind it as well.

And it has to be front and center to a project. Yeah, so that’s what I think about that.

Shane: Yeah. And I agree with you that I, if I remember a lot of my learning was from that grumpy old DBA. When you used to deploy something and it ran like shit I remember the standard response from the grumpy old DBA is when you said, oh, my, my ETL loads are going too slow. They used to always reply with, compared to what, that was standard response. And then, eventually they go and help you tune it. They typically tell you it was a data model or your code that was wrong, not the database. And then, again, if I think back, I think you’re right that often the projects we are doing were expensive and expensive ‘cause the people and time. Therefore there was a whole hierarchy of mentoring because you couldn’t afford to have somebody just rush off and do something that didn’t fit the way everybody else was working. So there was always a mentoring process where somebody more senior experienced than you, took you on the journey to learn to do things the way they did. And I think what’s happened is as we’ve got new technologies, we’ve got new forms of democratization, and therefore people are able to do the work quicker and faster using those tools. And we’ve lost that mentoring process, that knowledge transfer outside the tool itself. And also people aren’t being taught what we got taught in the early days.

We did a podcast or a webinar and we’re talking about data layered architectures. And a template that I’ve been working on. And one of the questions in that was, is the data persisted in the layer or is it virtualized, right? Or is it temporal memory? And one of the questions we got online is, what is data persistence? And I sat back and went, holy shit. And one of the comments from Chrissy was doing the webinar in Ramona with me was, know what persistence is because we’ve worked with tools when Excel didn’t save. And if you didn’t hit the little dis get and persisted that data down to your laptop and your laptop crashed, you’d lose it. Now we’ve got Google sheets where you type, it saves, it persists the data. We don’t have to care about that. And so I think for some of us, we forget some of the core foundational concepts that we just learn by doing things outside of data to a degree that we’ve applied. So if we think about that, then. Is the problem with data modeling right now? Lack of tools really there is still a lack of data modeling tools in our data stack. Or is it a lack of mentoring and therefore is AI the answer? Do we actually now see AI bots or AI clones or whatever we wanna call them, that actually become the daily modeling mentors for people that don’t have a physical mentor like we did? What do you think?

Andy: I think that the technology itself, and I’m talking about it could be Microsoft technology, it could be Oracle, it could be Databricks, snowflake, so on and so forth. They don’t have any guided ways of creating a model. I reckon the last time that I used a piece of software that guided me through the process that was very aligned with a very specific data modeling technique was analysis services multidimensional, where objects inside this model.

Were named after a specific modeling Pattern. So your facts, your measures, and your dimensions. If we’re talking about Kimball dimensional modeling, so that was really the last time that I used a piece of software that was quite aligned and essentially had a wizard that would take you through a modeling process.

‘ cause of course now we’ve got different modeling processes we could model. Yes. Third normal form. We could use dimensional modeling, we could model it data vault, and no vendor is locking themselves into a specific Pattern, right? Vendors are saying, Hey, you can bring this specific type of modeling to our software so they’re not baking in a specific modeling process.

One of my real asks in terms of software is to start to bring in some of these modeling aspects. I would love functionality, like slowly changing dimensions. So this is a dimensional modeling concept, which adds historical records to a, a reference table, to a dimension table.

And this is just my experience with the software that I’ve used. I really only see, Databricks that have done this, slowly changing dimension type two in their Delta live tables or lake, Lakehouse declarative pipelines. DBT, they’ve implemented slowly changing dimensions in something called snapshots.

So it is there, but it’s a little piecemeal and it’s not necessarily being massively called out in terms of this is the modeling Pattern and this is the feature that we’ve implemented for you to be able to realize that modeling Pattern. So then you were talking about ai, right? And this is where we start to get into productivity.

This is where we start to get into how can someone with not much experience of something ask AI to help them. And this is a classic case of where AI can help and start to move someone towards understanding how they work with the business. How they would evolve a Pattern as well. And like you were saying before, , people can get hands on with technology, they can get cracking with technology.

It’s binary, something’s gonna work or it’s not. If you model some data, you might not know whether that model works that you’ve created six months, a year down the line when all of a sudden something happens. That means that the model isn’t flexible enough. It hasn’t been thought of it, it hasn’t, there hasn’t been enough collaboration to understand the impact Act.

I went to the Fabric London user group. I was talking at the user group, but I was talking on technology, right? I was talking about a specific feature within fabric called materialized Lake Views. A little, segue from our conversation, but it’s a feature. It’s a feature in a piece of software from a vendor.

Johnny Winter was there, and he was talking about data modeling. He was talking about, sunbeam modeling. And this is a modeling practice that I’ve used in the past only because Johnny Winter has been talking about it, because it brings together a couple of areas of data modeling that I have used.

I just never thought there was this, port manto of these things. Beam business event analysis, modeling from, Lawrence Corr and the, the great Agile, data warehouse or Jim Stato and Mark Whithorn, which I now know is the pronunciation who was championing sun modeling, in terms of, a central event and then your sunbeams were your reference data in terms of how you brought context to that data.

And of course that was just the modeling aspect. But Johnny then started to show a tool called LE ai and that really got me thinking. So this is a data modeling tool in which. It’s focused on data modeling. It’s focused on understanding how to build enterprise data warehouses, how to help someone and guide them through the process of creating a data model.

And it’s very different from something like a data warehouse automation tool, right? So there’s data warehouse automation tools out there, either wcap and such in which you pretty much have to know the modeling Pattern while you are using these tools. Whereas all of a sudden I’m looking at Ellie AI and thinking, oh, okay, I get this now.

It is essentially an LLM or maybe multiple LLMs that’s been trained on data modeling, perhaps on architecture, layout patterns, technology even. ‘ cause it can help it then shape how it. It helps the user as they prompt their way to a data model. And at first I looked at it and thought it’s just another AI tool.

There’s millions of AI tools out there now. But I did think, hang on, if this is an AI tool that’s helping people data model, this can only be a good thing. This could only be of benefit to people to use something like this to help them through the process of modeling. They’ve got the technology they can use, co-pilot or chat GPT to help them write code.

But they’ve got these tools with the subject matter expertise of data modeling to help them and guide them through getting to a state where they might not have these problems in 6, 9, 12 months time of a model. The isn’t flexible enough because they can prompt their way into flexibility. So that was what I was looking at specifically with the AI tools.

Shane, so I’m curious to think about what you think about somebody who doesn’t have much domain expertise in data modeling, but is working in the data space using , these AI tools for modeling.

Shane: there’s a lot to unpack there. I’ve just made a whole page of note. So let me go through from the beginning of what you talked about and replay it back and my thoughts around it. I liked your point around opinionated tools. If I think back in, previous of technology for the data space, the tool really supported one modeling Pattern. Yeah. And realistically it was Kimball, Kimball was the number one data modeling Pattern that I ever saw in data warehousing in the old days. And that was because wrote good books. He shared his content online with his blog, which was free and easy to access, and he ran great courses. So getting access to how to model, he was the most accessible piece of content that you could find, and it made sense. I’m a big data vault fan. I like the physical data vault modeling Pattern. I’m not a fan of the data vault bi methodology but I find that content and access to how to model using the data vault Pattern of hub sets and links is incredibly hard to find.

It is poorly written. It is pay. And so I think that’s why we see with the advent of DBT and when people started to realizing they had to model data consciously, we saw Kimball take off again, right? Because the old content is still valid. Just on that though, I was really intrigued about your SCD two comment. So let’s take a segue on that and I’ll come back to the AI stuff in a minute. Because I remember with the original ETL tools that we were using, there was never a native CD two node. We used to always have to bloody well write that node ourselves. then when the cloud analytics databases came out, would’ve just made sense for me for CD two behavior.

That Pattern of historical recording of change data to be a database feature. . Not be a piece of code to detect the change and store it, but just make it a feature in the database to say, this table is SCD Type two table, because the database could take care of that change detection the, end dating or the flagging of a current is record. And so it’s really interesting that SCD type two is one of the patterns that you use for dimensional modeling all the time. When you’re physically modeling using that Pattern, but somehow we still seem to be lumbered with write some code that deals with it. Is that your experience? Are you still seeing people having to write code to, to implement the type two Pattern?

Andy: Yes, I am basically, and this was even talked about a few days ago, so I was at a little conference in Birmingham called fab Fest, which was focused on Microsoft Fabric. I was speaking about a function called, or a feature called Materialized Lake Views, which is essentially like Databricks is Delta live tables or lake flow declarative pipelines as they’ve called now.

‘cause they’ve, they’re abstracting it away from Delta because it’s not just Delta that this technology supports, but also DBT and someone said, why have we not got this SCD outta the box functionality? Because it’s almost like slowly changing dimensions.

Have they transcended the specific methodology in which it would be used in? And what I mean by that is when you started learning Kimball and you started learning dimensional modeling, one of the sub-categories. Was dimensions and one of the sub subcategories was slowly changing dimensions and all of these different types.

So type two has probably now become the champion, right? It’s the one that will track changes over time by adding new rows of data and the associated metadata to keep it. Yes, there are, type four and type six and type three, and all of those have their reasons to implement them. Type three, you’ve got multiple columns which can store, the current data and then the previous amounts of data, but they are a little bit more difficult to implement.

So most of the time people will use slowly changing dimension type two, because that’s the one that is the most. Relevant, the most prevalent and the ones that’s easy to implement. However, unless you are using some tools that have this functionality in built, like I said, DBT have functionality called snapshots, Databricks in their, lake flow declarative pipelines.

You can declare an SCD type. It’s not massively prevalent and soaked in to the data landscape. So this person I remember at this conference was saying I’m still having to write my code to implement my slowly changing dimension. I’m still having to say these are the columns that I would like to track.

This is the key that I would like to join on. These are my columns that I’ve defined for my metadata, my from my two, my is active. If you have that, any other metadata columns that you want to be able to track your changes. Then, and my last point on this is we look at the medallion architecture right now, I’m not gonna go into the ins and outs of, should we call it the medallion architecture, because to my mind it’s a data layout Pattern.

It’s not necessarily an architecture, but it has these different zones of data. We know raw is bronze and silver is cleansed, and gold is modeled, but silver. Which is the cleansed data. It hasn’t yet got to a stage in which it’s been modeled to a specific Pattern. There’s a lot of advocates that want to apply slowly changing dimension functionality to that silver data, but it’s not modeled yet.

It’s existing at the same granularity as the raw data. Obviously it’s gone through dedupe, it’s gone through cleansing and all that kind of stuff, but it’s not modeled yet, but we’re applying something that was a modeling Pattern or a modeling feature into this store of data. So I find that quite interesting as well, is that this SCD has almost been extracted as a feature of a modeling Pattern and can just now be used as a way of tracking changes over time.

But To your point, I just don’t see enough of that functionality automatically added to database and data products. People are still having to do it themselves.

Shane: And that affects adoption because if we say that a simple technical Pattern of type two implementation, where we know what the patterns are, right? We know that is detect change, insert row, start date, end date. If that , what you like is active or as current as a flag. We could actually just add all of those, and I remember in the early days, when we were constrained on database technology where the cost of those servers was expensive, the cost of the licenses were horrendous. We had to optimize to reduce costs. You probably remember it, we would argue type one versus type two. And we would type one by default, and we would type two where we knew there was value because the cost of type two was higher than type one. Now with cloud analytics databases, we really don’t give a shit, we just type two everything because we can. And it saves us problems later. And people are more expensive than those databases. And so we can be lazy to a degree, but there’s value in being lazy. What’s interesting for me is that Pattern is a very well known Pattern, yet it hasn’t become opinionated in tools. And when we talk about data modeling patterns, do we go. Snowflake versus star versus data vault versus anchor versus hook versus unified star schema versus anchor, there’s all these other patterns where actually they’re a little bit harder to be opinionated about. and therefore they’re harder to bake into a tool. So if we can’t do the simple stuff, how do we expect to do the hard stuff? And then the last thing I’ll say before we move on to the next part of your point that you had earlier was your silver’s, not mine. I think medallion has been great because it’s reinvigorated the conversation around layered data architectures and the value of them. But when you talk about your silver as being cleansed, I talk about my silver as designed and I model. We have a an opinionated data modeling Pattern, which is concepts, details, and events.

That’s how we model and what we call our silver, our raw is historicized. We are effectively applying an SCD two type Pattern on our raw data for a whole lot of reasons. So again, the problem with medallion is, it’s a nice way of describing a layered architecture. But soon as we get into any detail of what you’ve got in your layer, it’s not what I’ve got in my layer, and that’s okay.

As long as you tell me that you are cleansing in silver and it matches the structure of your source, and you tell me you model in gold, I get it. Now I get your architecture by you just using those words that opinion you have applied. So I think that’s the key, is no more, there’s only one way to Medallia.

It’s what do you mean by silver? So let’s jump on then, and then let’s talk about tools because. In the past we had tools like Irwin, oh God, ea sparks. We had some really hard to use data modeling tools to draw diagrams of what our models look like, and typically they were completely disconnected. We found it really hard to draw a diagram for our conceptual and physical model and then get that model as substantiate in our database easily. And the modern data stack, in the previous wave, now that it’s dead, we saw tools like Ali and sql, DBM, we saw visual modeling tools come out. But what was interesting is they became category, they were a party or stack.

So if you had a on data stack, you’d end up with five to 10 different tools to do your end-to-end processing. And we’ve seen a lot of consolidation in the market. We’ve seen a lot of those tools that are part of the stack disappear or get acquired or become features in, in one of the other tools, we haven’t seen that for data modeling. We haven’t seen the data modeling capability being bought back into those end-to-end stacks. And again I don’t do a lot of work in fabric, but I don’t think, apart from the power bi SQL server analysis services part of the Microsoft stack, they’ve never really had a modeling tool, have they?

There’s no tool I would go into that would help me create a conceptual or a physical data model and instantiate the physical model in a database within the Microsoft stack, or is there.

Andy: So this is probably one of the most asked questions in forums around the data modeling aspect because as you’ve said, we’ve had, tools over the years that have enabled us to do data modeling. Even SQL Server Management Studio, which you can download for free has got this almost live data modeling process attached to it where.

It’s very much a physical design Pattern. You can’t logically design something in its interface in fabric. We’re still there. We haven’t got anything that’s gonna help us logically design a data model outta the box. There’s nothing that we can start with and say, okay, I wanna start with the conceptual data model.

I wanna go down into a logical design, and then finally a physical design of my lake house, of my warehouse. we’re having to still use other tools to be able to do that. So whether people still use, Visio, whether they’re using, other things, AATE or SQL database modeler.

I think even Toad, for these kind of modeling tools, one of the things that I didn’t like is when Microsoft did deprecate some of the data modeling tools that were available within Visual Studio, I just thought you’ve got to have something that can help a process. When I look at Fabric or even Databricks, snowflake, all of these sort of cloud vendors is, they are the canvas, they are the paintbrush, but they don’t help you draw the picture.

You need a process to go and help you draw that picture. And there are, there are cloud tools that help you do the data modeling. Some of them, I think a lot of them are paid because. To go from conceptual to logical, then physical is a natural progression. People want to be able to create the physical data models.

Ultimately, yes, some people might be a little bit annoyed that they’re forced to go through, the conceptual and the logical modeling processes before they get down to the physical. But then they want to be able to click a button and it generates the code necessary for them to run and create that physical model.

No, I don’t see anything. And if we’re talking about Microsoft Fabrics specifically, I don’t see anything in Microsoft Fabric that’s gonna help you from the conceptual all the way down into the physical.

Shane: And again, you’ve gotta be, software should be opinionated. So the way we do it is I create the conceptual model and then it generates a physical model without me doing anything because our physical model is opinionated. So our conceptual model is. open in terms of, the things you create, the concepts, the, who does whats, ‘cause I’m a Great Beam fan as well,

lawrence CO’s book is one of the ones I, read early. We used to, when I had my consulting company in New Zealand, in pre COVID, when things went online, we used to fly ‘em over as often as we could to teach our customers how to beam and event model their data.

‘cause it was so useful. And that idea of, your conceptual model of who does what, your core concepts, depends on the industry, depends on the business case, the usage, the actions and outcomes you want to take. So it is a, a little bit less opinionated to a degree. but once you’ve got that sorted, you can make your physical modeling Pattern incredibly opinionated. And that’s why I think these disconnected data modeling tools are struggling from what I can tell. And you’re starting to see them now bring in the ability to actually generate the ETL, the code to, deploy the model and load the model.

Because that’s the space you have to be in. You have to be able to create the model and test it, that it has value. And so when I come back to the AI tools. I’ve played a lot with the lms played a lot with the bots, and I can see as a helper friend, as an assistant, it allows me to ask questions and provide some context around the industry, the use case, and get back a starter model for 10. And that’s really useful. But if I think back to this idea of the mentoring I had earlier in my career by those grumpy data modelers what they used to do was they used to stress test the model. There was something magical about the way you could give them a data model they could just look at it and then they could call bollocks on the things you got wrong.

It was just that innate Pattern matching in their heads. ‘cause they’d done it so often and they could go, yeah. That relationship’s not a one to many. It’s a many to many. It’s not gonna survive. Okay. The rate of change on that table is gonna be horrendous. You’re gonna go blow out your Oracle instance. Yeah. Or you have to upgrade to Oracle Rack, which will cost you, two legs, one arm and your three newborn children. I wonder if that actually is where AI tools have to take us is effectively an agent model, right? Where there is one that helps us build the model and one that helps us stress test the model, which goes back to that point you made a long time ago which was, when we work with technology, we get instantaneous feedback that the technology’s working.

It’s not when we work with data modeling, we don’t. So maybe that’s where the AI tools need to take us, help it to create it. And then another agent, which is, grumpy old data modeler that tells you where you got it wrong. What do you think?

Andy: So I think AI can help all the way through that process. And this is where we start to look at AI as less as a technical tool that will help us do something and more as a sounding board, as something that we can ask it to be quite antagonistic with. As you said, and like I, I raised the subject earlier about AI tools and helping with data modeling is you could go to an AI tool and say someone has told me to design, x, y, z system for argument’s sake.

Let’s say it’s a data warehouse or it’s a lake house and we’re gonna use, a data modeling technique that is best for reporting and analytics. And the LLM might reply and say, okay, well here are a few. Modeling patterns and, dimensional modeling is the one that you would use for a data warehouse and so on and so forth.

So let’s say they then pick that and, carry on doing that, they’ll then ask you questions about, I’ve got all these different entities in my source system. How will a dimensional model help me? So it’ll then work through those entities and say, okay, you’ve got these entities that look like they can be grouped together.

Perhaps that’s a dimension and it’ll work through the process with you. So let’s say you’ve got that sorted, so you’ve applied your critical thinking and not just accepted everything that the LLM has given you. You’ve gone back and dah. Maybe you’ve Googled, maybe you’ve asked other people that are experts in that area to say, okay, you know what?

I’ve spent a whole day generating this model. It would’ve taken me two weeks if I had to learn the theory and then do it. What do you think? I’m sense checking it. Someone might come back and say, okay, we can tweak a couple of things, but actually that looks pretty good. Great. So we’ve got our starting point then.

You talked about stress testing and Yes. So this is where you can then ask the LLM to be antagonistic and say so this is my data model we’ve got here because the business want to be able to report on these X number of attributes and this is how they want to measure it. But I know that there are other source systems and things like that.

Can you tell me what problems this model might have in the future? And of course you essentially, you are asking the AI to try and do some future proofing for you and some troubleshooting. And it might come back with some generic questions about perhaps your product dimension isn’t, deep enough.

What about other entities that you might need to bring in that you are, that you haven’t yet got links in your fact tables, but it’s gonna surface, it’s gonna help surface potential problems for you to then deal with. So I totally get that as well. And then the third point that I wanna add is about anticipating changes.

We touched on, Lawrence Core and, the Agile data warehouse book a little bit earlier, which Yes is, something that I go back to constantly and, we’ve got beam in there to help us do this. But the agile data warehouse is also there to help us iterate over a model as well.

So perhaps we then say to the LLM, look this first version of the model, okay, we’ve gotta set it in stone now because we’ve got project deadlines we need to get the data in because the business are gonna build X amount of reports. And we tried to make the model as generic as possible.

‘cause we don’t want a report driven model. We want the data driven model in here, but help us understand what we might need to do to modify the model and be a little bit agile. So I would say that the usage of the AI tools to help us generate the model is just the first part. Most of it is gonna be about asking the model to be quite antagonistic about the model it’s generated.

Yeah I think that it’ll help us build the model, but the most important thing is antagonize the model, test it hopefully point people in the right direction in terms of future proofing and certainly surface issues that might happen in the future. They might need to fix those issues. I don’t think I’ve worked with a dimensional model yet that doesn’t incur a certain amount of technical debt in how it’s implemented.

But if you can mitigate those things earlier on, that’s just gonna be of benefit. Yeah, so that’s my thoughts on that, Shane.

Shane: Yeah, and that’s that problem between, doing a model quickly that gets value now and trying to boil the

Andy: Yeah.

Shane: for all changes in the future. Or an enterprise data model. Again, going back to that anti-patent, we have these days of, a data modeler sitting in a room for two years doing , one enterprise data model to rule them all that nobody implements. One of the things we’re doing is we’ve been experimenting in this space and we have a bunch of partners use our platform and we got one of them to experiment. And we had a use case around Google ads, so the partner needed to bring Google Ads data in and deliver it for a customer. And so that first part of the agent that opinionated agent that’s gonna help you do the initial model was really interesting. we found was, first thing was we were lucky that Google ads gave us effectively context about the tables. It brought in metadata that described the tables quite well. So that was really valuable for the agent because it now got a bunch of hints of what the source data looked like. The next thing was we have an opinionated design Pattern for the way we model.

And so , our agent already knew about that, so it knew what the rules of the game was, it knew it couldn’t dimensionally model it, it knew, it couldn’t anchor it, knew all the patents it couldn’t use. It knew what our patent looked like. So that opinion was effectively already in the agent. As a a bunch of rules. We also gave it the information product canvas that a partner had done, which is a description of what the actions and outcomes and business questions that need to be answered first. And what that gave was a boundary to the agent to say, don’t model all the Google Ads data. Only model the data that’s gonna support this outcome. So again, it gave it a boundary that led a lightly model. Now, what we didn’t do was, the bit that you are just raising is then stress test change, right? We didn’t say what’s gonna happen next, but deal with that in different ways at the moment. But I’m gonna think about that one really well, so boundary of opinion that was given to the modeling agent meant it did a good job, If I had just said to a, I’ve got Google Ads data and model it, I have a theory and I’m gonna go test this. I reckon every time I ask it that even though it’s non-deterministic, it’s gonna kimble model it. The reason I say that is if you think about why has Kimball and dimensional modeling become the number one modeling technique for DBT? Because if you are an analyst and you are moving into the engineering space and you hear that you need to model some stuff and you use an LLM or you go Google search, you’re gonna come back with dimensional modeling every time. ‘cause as I said, it’s the most freely available. Describe content in the world for data modeling in an analytics space, in my opinion. so therefore, the lms, who trained on everybody else’s content without paying for it, won’t go into that one. It’s gonna have the richest piece of content in the LLM for dimensional modeling. interesting question on that one is actually, if I go and ask an LLM to model it with no constraint, no opinion, I bet it’ll come back with Kimball Modeling.

What do you think?

Andy: And I think the LLM would probably not be displaying any emotion. So let me expand on that. So you touched on data vault earlier and you say that’s a, that’s a data modeling Pattern that you like. If we go back several years, actually decades we’re talking about Kimball versus Inman.

We were talking about dimensional modeling versus third normal form. Then Dan Linted comes along and we’ve got data vaults. There was lots of emotion involved in people comparing these technologies. In fact, all. Of those people, bill Inman, Dan Linted, Ralph Kimball, they all said, and it’s in their books, that these modeling patterns are complimentary.

Bill Inman would say your enterprise data warehouse can be third normal form, but for reporting and, for feeding into analytical tools. Kimball model, the dimensional model is great. Kimball would say, ah, okay, you can do that. But yeah, you can also do your, your enterprise data warehouse and dimensional modeling.

Okay. There might be a little bit difference of opinion there, but they were still complimentary. Even data vault, Data vault, you can’t just plug straight into analytics and tools. and I will. Admit and agree that it’s great for tracking changes over time in the lowest level of granularity, giving you the most flexibility to do what you want with it afterwards.

But a dimensional model is very good in plugging it in. But of course, we see all those debates out there. This versus this, the versus this, the LLM doesn’t will take those arguments into consideration, but it has no bias. It has no emotion attached to any of those modeling patterns. So I would say that LLM will probably come out with dimensional modeling because it’ll reason that, okay, you can store your data this way, but it’s not going to be what you need to design the model that’s going to be delivered to the business.

And I would like to test this as well and antagonize an LLM around these modeling patterns and ask it. Okay, I’m gonna be designing this. What do you think the best model is gonna be? And then I might add in a few, trip wires to it and say what about data vault? And what about this?

And I’m hoping that the LLM would say, yes you can use those modeling patterns, but it’s generally agreed that they are complimentary and that you can add on a dimensional modeling Pattern to a third normal form or a data vault. But I am hoping that the AI has less shall we say, emotion attached to picking that data modeling Pattern.

Shane: if it’s got no emotion, we should ask it for a definition of data, product, semantic layer

Andy: yeah.

Shane: We’d love to argue. I’m gonna, I’m gonna disagree with you on that one. And the reason is I don’t think LMS are reasoning, they are Pattern matching and tokenization based on a bunch of content. my hypothesis, and it is just a hypothesis, is the Kimble and dimensional content is being far more widely available. And therefore that model has a bias towards using it However. It’s just a hypothesis. And one of the things that Joe Reese has been doing as part of the practical data modeling community is just testing stuff live with a bunch of people. I’m gonna suggest you and I do that. I suggest that we figure out how to do a live session. bring up and, multiple LMS and with a bunch of other people watching or helping us. We just bash the snot out of it and try and see actually is there a bias for a modeling technique? But if I take that away for now your comment around some being really intrigued me, because Johnny’s mentioned it before and I struggled to find any content around it, it’s one of those data modeling that is actually quite hard to find anything about it. And good point. I need to get somebody on the podcast to come and explain it. So if I wanted an LLM to assist me in designing a Sunbeam model because I’m opinionated that’s the model I prefer for whatever reason, I think it’s gonna struggle and that will be another test, maybe we can give it a bash on that to, to see , so I think if you’re using an LLM to assist you in modeling and you are using a well-known Pattern that you are opinionated about, so data vault dimensional, third normal form potentially, you probably don’t need to use a lot of reinforcement with the LLM because it knows what you’re talking about.

If you wanted to bring in some of the more obscure modeling patterns, like something, I think you’re gonna actually have to pass it a reinforcement of that content. So one of the interesting things about Joe recently, what he does with the practical data community is he does live sessions where we all jump on screen share and we actually test out a hypothesis. And so one of the ones that he did was this idea of could we start with a business problem in an industry none of us knew. Get the LLM to help us understand the industry, create a conceptual model, move it to a logical model, move it to a physical model, and actually implement that physical model in a database. And that was fun. But what was interesting for me was observing the way Joe approached it, and he comes from more of a data science background than I do. So he tended, in my view, what I saw was approach it from an EDA, an exploratory data analysis approach. So he would be looking at the data sources and trying to understand the data that’s coming in, because that’s how we thought. Whereas for me, I come from more of a business background. That’s how I’ve been trained.

So more of that. Who does what business process stuff. A lot of the beam first part of the book. And so for me, I typically wanna understand the who does, what’s the core business events, the core concepts, and that’s how I model. so again, I think in terms of an AI assistant that helps you model, you probably want to train it around your modeling process.

Or maybe you don’t actually, maybe the LLM should decide what inputs it needs. Rather than you being opinionated on how you typically model, what do you think? First of all, which way do you model? Do you think source specific and understand the data first, or do you try and understand the core business events?

And then do you think you should be opinionated to the LLM about which way it should approach it?

Andy: I’ve always modeled from the source system side of things and then taken into consideration the requirements from the business. And I know why that is, and that’s because I am fundamentally a technical person. So I’ll always want to default to looking at the technical aspects of things. Hey, I can look at source systems, I can understand schemas, I can understand tables and columns and the domain values within that, and then I can look at matching that to what the business wants.

And I’ve worked with people that work the other way around. They are interested in the art of the possible. They’re interested in what do the business need to make the decisions, and then let’s go and find what we need from those source systems. So I’ve moved my needle a little bit more towards the art of the possible, right?

This is what the business ultimately needs in terms of decision making. If they need to be more operationally efficient, if they want to, take advantage of opportunities in the market, challenge competitors, whatever, I then go and look at those source systems and see what’s available. And then sometimes you really can’t realize.

Some of that data that is needed for the business, right? Because it’s just out of the hands of those source systems. And I suppose you could calculate it and you could infer it from that data, but I’ve tended to work that way. And then interestingly, when you were, talking about AI and Joe working through the kind of modeling aspects from that perspective, yes, I can totally see someone asking the model to work with them in a specific way and say, right here is my source system.

This is what I need to do. Now when those source systems, and if you’ve got access to schemas and you can provide that schema to an LLM and let’s, let’s say it’s An LLM that’s secure, it’s within your organization or it’s within your data boundaries. So you can give the LLM that schema and then you ask it to model that for you.

You may then ask that LLM I need to join it with other systems as well, and this is the system and this is the schema. So it could then help you generate the model to incorporate multiple systems or showing you some examples about how you can, join those systems together.

What I wouldn’t want an LLM to do is just go crazy with the business requirements and say, okay, this business operates in. This domain. So this is all of the data that they need to be ultra competitive and at the top of their game. And then I’ve gotta scrabble around trying to desperately find where I’m going to get this data from.

That the model says is going to help me build the perfect data model for the business. I guess I’m a data modeling pragmatist. I would look at the source systems and I would look at how they support the business in what they want to do. And then I would probably work with the LLM in that fashion.

I’d be saying okay, here’s the framing. Here’s the context. This is what we need to measure, but this is what we’ve got in our source systems, and this is the hard facts about what we’ve got in the source systems. Help me build that model that can, realize what the business wants, but work within the limits of what I’ve got with those source systems.

So that’s what I would do.

Shane: I was just thinking then, back to my point about, enterprise data model is sitting in the cupboard for two years to come out with the most beautiful data model ever. That’s basically a person going into an LLM now and saying you have no constraints. Here’s the industry. Gimme a data model from scratch that does everything .

It’s quicker. It’s not two years, but it’s just as unimplementable, that’s a bad word. One of the things we found was if we think about chat GT five and this idea that, we didn’t get the a GI that we expected, but what we got was a better interface. So instead of having to decide what type of LLM foundational model you wanted to use, you tell it what you want to achieve and it works out which model is the best fit for you. And one of the things we found when we were experimenting was. If we had one agent, our agent’s called 80. If we asked her to do everything, she was okay at it, but she wasn’t great when we broke her out into sub-agents. So we had agent, 80, the data modeler, 80, the what we call change rules, like the ability to write ETL and we gave her a clearer opinions and a clearer boundary.

We, we got a really good uplift in the accuracy, right? And the the evals that we got back in terms of, we got better responses that made our lives easier. so of the things when I’m teaching my canvas I actually use something that Lawrence talked about, right? Which is modeling based on source, modeling based on report or modeling based on business process. It’s one of the things that stuck with me for many years. And so now I’m thinking based on that. What we really probably need is an agent that we can go to and say give us a model based on the source system. And another agent. We can say, give us a model based on the core business events or business processes, or the who does what. And a third one where we go give us a model based on outcome, and then we give it the constraints of what we actually wanna achieve in the next iteration. What do we actually have to deliver and we get it to model it for us. So based on those three, tell us what the actual optimal model is to achieve this outcome and then stress test it for me. So that is getting a bunch of inputs, right? Because if I think about it. That’s what great modelers do. they take a stance, but then they always jump, your technical you source first, but then you go right now what are the core business events?

Customer orders, product good. I don’t need to worry about store ship’s product in this iteration, So I’m only modeling in that boundary. And then, okay, what do we know we have to deliver? Is it a dashboard, is it a data service? And then you are using that to iterate initial stance of that model until you get something that is fit for purpose.

If I think about it, that’s what we do as humans. So that’s probably the process we need to encourage the L LMS to do. What do you think?

Andy: I like that idea of asking each of those models to come up with their specific version based on those constraints. You do it by source system, you do it by business process, and then getting them to antagonize each other in terms of. Getting to a realistic result, taking on board each of those aspects.

And what I did make me laugh in my head when you were talking about, a great modeler is that in the Rocky films there was Apollo Creed who said to Rocky that, you fight great, but I’m a great fighter. And I think that about the modeling domain as well.

you can point at a person and say, you model great, but I’m a great modeler. And that’s just built up through experience, That’s just built up through battle testing models that you’ve created over the years and iterating over those models over the years. And like you said, in a couple of points before.

About the LLMs generating these models and coming up with dimensional models because that’s what the prevalent documentation will have. Those models have been trained over that documentation, so those models will be like, ah, okay, this seems to be the most popular way of doing things. So then when those LLMs are generating a model based on a source system, it’s gonna be based on the context of them understanding that source system and what the output of that source system is.

I look at something like Dynamics, which has generally been something that is quite difficult to model when so much customization happens within the platform. There is no real vanilla implementation of dynamics, which means that the LLM is gonna have to. Understand business context, not just source system because I guess the source system is just going to have all these entities that it might think I don’t have any information, because those sorts of things haven’t been discussed before.

Before. And as human beings, I guess we can reason over those things and we can hypothesize and we can make a best guess and iterate over perhaps the LLM can’t really do that because it’s been trained on previous data, previous examples, and it doesn’t have the ability to think outta the box if it hasn’t encountered something before.

Yeah, interesting point there, Shane.

Shane: Although I would posit that it probably has encountered it because it’s got access to information outside of data warehouse, data modeling. So it’s gonna have all the books on Dynamics implementations, it’s gonna have all the blog posts of people who have customized it. But yeah, I get your point, especially things like SAP, where, who knows how that bloody thing works.

It’s it’s gonna have more knowledge than I do on that. But less knowledge than an enterprise. SAP data modeler, who does it for a thing. so again, I think it comes back to being clear about when you want to provide an opinion when you don’t. So when you wanna provide an opinion, ‘cause it’s important that the LLM or the agent stays within that boundary for you or where you just leave it because it in theory has access to more expertise and knowledge than you have in that, in a specific space. And so if we go back to medallion, if we go back to layer data architectures, that’s actually a really good place where you might wanna be opinionated. Because if you are saying I want you to help me do a conceptual model, you probably don’t care about your layered architecture. But if you are saying, I want you to help me create physical models that I’m gonna implement, I know that. My layered architecture is I have a designed layer that is concepts, details, and events. And my consume layer, is a one big table. And I know that your cleanse layer is source specific data structured with data being cleansed. And your gold layer is, I’m guessing, a dimensional model,

if each of us had the same agent, but we put in those opinionated boundaries, we are gonna get back physical data models that are more fit for purpose for us to implement in our platforms of choice. And, realistically, if the L and N came back to you and said throw away your dimensional models and do one big table from cleansed, you are probably gonna look at that and go the cost of change is quite high.

I really need to understand why you’re making me do that. Because I have to learn a whole lot of new things and I have to rebuild everything. So I think, again being opinionated where you want to be opinionated, that makes sense is important. And then being free and hippie and open to whatever, the great a, a I agent in the world tells us as a start for 10 and then stress testing it with expertise.

I think that’s where we’re gonna end up. What do you think?

Andy: So I think that opinionated aspect is quite important in terms of how you approach the LLM because being opinionated about something means that you’ve got strong convictions in how you want to do something. Which then also means that someone else could be opinionated and have strong opinions in how they want to do something.

And as we know over the years when you’ve been working with other people that have experience doing these things, there might be a crossover, there might be some differences of opinions. How is the LLM going to know those differences of opinions that you can work out and collaborate on? Yes, it’s got this body of knowledge.

Hopefully the people with the opinions have been provided information, writing books, writing blog posts that the LLM can take on board and learn and use. But if I come to a project and use an LLM to create my medallion. I also ask it to create my data model based on my biases and based on my experience and the way that I wanna do things, I’m probably gonna guide the LLM to a certain conclusion.

Someone else is gonna come along and say I wanna start this project. I wanna lay my data out. Do you have some thoughts about how I wanna lay it out? I have used this in the past and then I wanna use data modeling to do it. It might come up with a slightly different outcome based on your own strength of conviction and your biases that you’ve put into the LLM.

It might come back with something that is generic at first, but then you might say to the LLM, actually. I don’t want my ization in silver, I want it in raw because that’s traditionally where I’ve put it. Whereas someone else could say, okay, why have you asked me to put my ization in raw? I’ve learned that it’s supposed to be in silver, but if we remove the LLM from this conversation for a second, it’s almost like we’d get the same outcome if it was just humans.

Because humans would go into an organization, implement a data platform. and based on their experience and biases, they would implement it a certain way, a different set of people could go into that same organization and implement it slightly different.

So I don’t think we’re necessarily solving the problem of these differences of opinions or the convictions that you have. It’s just using another tool to help, possibly reason, to help possibly have this entire body of knowledge and experience our disposal that we can interrogate and hopefully gets to a more human slash ai reasoned outcome.

That’s what I use it for. So I would use AI to help me model, and I Based on my experience, but I’m also not going to be arrogant enough to think I know everything there is to know about data modeling. So I also want to ask it questions about have we thought about it this way or have we thought about it the other way?

Just so that I can bring a little bit of critical thinking to it as well, and then evaluate the outcome. Yeah, it’s an interesting point there, Shane.

Shane: I like that. We’ve got to is, ‘cause it goes back to where we started, which is if you are new to the data domain and you wanna understand data modeling, there really isn’t a lot of great content apart from the Kimball stuff around. And there are other techniques. So now LMS give us access to ask questions and effectively get educated on what the art of the possible is.

What could I do? The second thing is, the mentors, the grumpy people who could just look at your model and tell you where you got a. Wrong. They’re not really so prevalent in organizations anymore for some reason. So again, using the LLM to provide that expertise, that rigor, that stress testing, that feedback is really valuable. There is another lens, but we don’t have time to go into detail, but I’ll just drop it in here ‘cause maybe we come back and have another chat about this idea when you’ve thought about it a bit, which is centralized platforms may disappear. So if we think about CRMs, they are centralized platforms that everybody uses with a core bunch of features, they’ve being, built in a way that become reusable. And what we’re seeing now with vibe coding is actually you could build one feature, one app that does one thing really well really quickly. and it doesn’t need to be part of a shared platform. And that’s gonna be a really interesting change in the market when that happens. That also means that you can buy code something that has 30,000 lines of code. And while you should know what it does, potentially, you don’t need to care because if it’s safe and secure, it does the job actually. You don’t need the expertise to understand how it does it. we apply that to the data domain and this idea of moving away from shared data platforms, that I’ve created this information product that does one thing well, and I’ve got the LLM to design, five different architectural data layers using 17 different data modeling techniques and a bunch of code, I don’t understand, as long as it doesn’t have to be reusable, then maybe we don’t care and again, I’m old. I find reuse really valuable. I find expertise and shared language really important. But maybe, the New World is one and done information products where the LMS are. Doing everything completely different each time you deliver a product. But we’re out of time.

So I’m gonna leave that one on the table, maybe have a think about it, and that could be a, a good follow up conversation around how would we use AI and LLMs to remove the need for reuse and shared platforms in the new Age Worlds. But before we finish off how do people find you? How do they see what you’re reading, what you’re writing?

You’re obviously spending a lot of time going to some great conferences what those conferences are and how they can find you.

Andy: Yeah, so I think the first thing is Link Tree. So Link Tree is just is basically my go-to in terms of, I. A jumping off point for people to find me. So that’s basically, link Tree slash Andy Cutler, so A-N-D-Y-C-U-T-L-E-R ‘cause that’ll take you to my company. So data high.com.

That’ll take you to my community blog, which is serverless sql.com. There’s my Blue Sky account, there’s YouTube, which is data high, and then my LinkedIn as well. So the Link Tree, Andy Cutler will take you to everything You need to do most of the conferences. I am in the Microsoft space and predominantly fabric, which is, which is where I, I spend, as the Fresh Prince of Bel Air would say most of my days.

The next conference is over in Oslo, so that is fabric February. So that’s gonna be over in Oslo. There’s obviously SQL Bits, which is, one of the UK’s biggest data conferences as well. so I would say, I generally post on LinkedIn blog posts, opinions, lots of conversation as well.

And as I said, yeah, Tre Andy Cutler, that’s where you’ll find me.

Shane: Excellent. Alright, hey, thanks for a great chat around AI and LMS and data modeling and I hope everybody has a simply magical day.

«oo»

Stakeholder - “Thats not what I wanted!”
Data Team - “But thats what you asked for!”

Struggling to gather data requirements and constantly hearing the conversation above?

Want to learn how to capture data and information requirements in a repeatable way so stakeholders love them and data teams can build from them, by using the Information Product Canvas.

Have I got the book for you!

Start your journey to a new Agile Data Way of Working.

https://podcast.agiledata.io/e/the-pattern-of-metric-trees-with-timo-dechau-episode-77/

«oo»

Recording the path to your "AI Agent" responses

Shagility — Mon, 22 Dec 2025 16:53:49 GMT

How did you get that answer?

As Nigel and I keep testing out different use cases for where our Ask ADI capability can help data professionals reduce the cognition and effort required to do complex data work we stumbled on an interesting pattern we both use.

Building the House while living in it

As we are using ADI in anger to do the data work for our AgileData.team Fractional Data Service customers and we are also extending out the capabilities in our AgileData.cloud data platform at the same time, we found we would often use ADI to help us with a task and then ask her “How did you get that answer”.

This would often be triggered with ADI giving us one of three responses:

Something that was right
Something that was wrong
Something that was unexpected

Something that was right

For these ones we used the answer and moved on to the next step in the data work, sing with joy that she is doing what we designed her to.

Something that was wrong

She would give as an answer that could politely be called hallucinating, in the data world I typically think of it as just being wrong.

In this scenario we always want to understand the path she took to return the wrong results. So we can figure out what we would change to make her more accurate in the future.

So we would ask her “How did you get that answer”.

Something that was unexpected

She would reply with something that made us go, wow that was unexpected, but actually bloody good, how the hell did she get that response?

Which meant we would next want to understand the logic path that she took to get that response.

Hence the follow up “how did you get that answer?” question.

In hindsight the pattern were doing was a form of eval, which we were doing repeatably but manually.

And then we started scaling it

As we tweaked the Google Gemini models we used, we tweaked what and how we stored Context in the Context Plane, we tweaked the prompts and reinforcement objects we stored in the Context Plane and made accessible to ADI etc, we decided she was good enough to be out in the hands of our AgileData.network partners.

To be clear they were well aware that she was still in an “discovery” mode, and not to be let lose directly on their customers without the partner in the loop, and was not ready to be given directly to the customers themselves.

But as one of our core AgileData principles is co-design, its how we can scale what we do so fast with just two co-founders. We know that putting patterns and features into the hands of our talented partners is a much faster way to iterate and scale.

WTF did they just try to do?

And of course you can imagine what happens when you put a fairly permissive AskAI capability into a data platform that covers the complete gambit from data collection to data consumption, and enables every data task in-between, into the hands of data practitioners who are by default very early adopters, who are working across multiple customers, in multiple industries and are trying to be at the edge if not bleeding on that edge ….

They started doing things that made us say, why the fook did they try and do that, what were they trying to achieve?

And more importantly, how the hell did ADI come up with that answer, and was it right or wrong, or excitingly unexpected?

Log everything

One of the patterns we apply to the AgileData.cloud is we log everything.

So of course we were already logging the question they asked ADI and the response they got.

But this didn’t help us work out the rest.

Manual processes don’t scale

So of course the first thing we tried, was taking the question they asked and typing it into our own AgileData Tenancies.

You can imagine what happened.

The data in our tenancies is different to theirs.
You typically wont get exactly the same answer to the same question with a LLM based model.

Epic fail and not a lot of use to help iterate ADI’s behaviour.

Not just “how many sales were there”

The natural language and non-deterministic behaviour of the Google Gemini models we use under the covers for ADI is where a lot of the value resides.

And this means she can help with a lot more data tasks than the typical “Text to SQL” use case every data vendors is chasing as table stakes these days.

And so the questions being asked and data work being done by our talented partners was a lot more than the simple “how many sales were there”.

Which meant we couldn’t just implement something like a judge pattern to make sure ADI was returning the correct number to each question.

Iterate with simplicity

As part of our Way of Working we alway try and decompose the work to be done into the smallest chunk possible and to start off with simplicity and add complexity later.

Automatically log what path was used

First thing we did was to extend the logging to include the path ADI took to get the answer, so we could review it after the fact.

Originally this was just logging in the background, but we found that this logic was actually useful to the Data Practitioner, so we surfaced it as part of the ADI response.

You can see the path being returned here in the language of as set of assumptions.

Ask for feedback

We still don’t know if the path used was actually the best path and if the response helped the partner do the data work quicker and easier, so we reuse the feedback pattern from social media products so they can quickly give us an up or down thumb for each response.

As our partners know they are co-designing with us, this quick and easy feedback loop provides value, without slowing them down from doing the data work and delivering value to the customer.

Should I trust the response

We also decided to experiment with providing a confidence score on each ADI response, we find this useful when evaluating the responses after the fact, it will be interesting to see this helps our partners or not.

Another Use Case

Here is another use case where we Ask ADI to help us model the data from Google Analytics.

Again we are logging the path ADI took to respond, but with slightly different language to make it fit the Context of the question more.

And ADI is providing a suggest next step and asking if she can help.

And as mentioned before the user will always go somewhere we don’t expect.

The point of these is not that the Gemini LLM provided definitions for commonly used Google Analytics concepts, any LLM will do that, you can just use ChatGPT on its own etc.

But the fact is we end up with full visibility of what was asked, what the answer was and who that answered was derived.

And with that visibility we can decide the most valuable use cases to iterate Ask ADI for, based on the things our AgileData.Network partners actually need them to help them with.

And one more thing

And you can see where we will take this.

Given ADI is embedded in the data platform that the data practitioner is doing the actual data work in, we can increase the level of assistance over time.

ADI should probably respond to the Google Analytics data questions, with a response that is tailored on how you do this work in the AgileData.cloud and use the patterns and language we use in that platform (Information Product Canvas, Concepts instead of Entities etc)

ADI should probably create the Concept Model for those Concepts.

ADI should probably populate the Business Glossary with the default definitions for those Concepts.

ADI should probably use the Context we have defined before for Google Analytics to differentiate between a Pseudo User and a User.

ADI should probably rehydrate the Change Rules (data transformations) needed to populate those Pseudo Users from thw GA4 event data.

ADI should probably …. [insert use case we see our partners do, that we have never thought about here]

Patterns you can adopt.

Here are some simple patterns you can adopt as you continue your “AI” journey in your organisation:

Log all questions asked and all responses given by your AskAI feature, somewhere you can see and query
Log the path your LLM took to provide the response
Find a way to gather feedback as you scale
Put it in the hands of your early adopters as soon as possible and let them help you co-design the most valuable areas to iterate with next.

The pattern of Metric Trees with Timo Dechau

Shagility — Fri, 21 Nov 2025 18:09:16 GMT

Join Shane Gibson as he chats with Timo Dechau about Metric Trees

Listen
View MindMap
Read AI Summary
Read Transcript

Listen

Listen on all good podcast hosts or over at:

Listen to the Agile Data Podcast Episode

Subscribe: Apple Podcast | Spotify | Google Podcast | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser | Deezer | Podcast Addict |

You can get in touch with Timo via LinkedIn or over at https://timodechau.com

Tired of vague data requests and endless requirement meetings? The Information Product Canvas helps you get clarity in 30 minutes or less?

Google NotebookLM Mindmap

Google NoteBookLM Briefing

Executive Summary

Metric Trees: A Framework for Strategic Alignment and Business Transparency

Metric trees are a powerful framework for visualizing the mathematical and logical relationships between key business metrics. Their primary function is to deconstruct high-level, often non-actionable, “output metrics” like Monthly Recurring Revenue (MRR) into a hierarchy of actionable “input metrics” that individual teams can directly influence. This transforms abstract company goals into a tangible map that guides strategic planning and execution.

The core value of a metric tree lies in its ability to serve as a shared language and a visual representation of the entire business model. It fosters alignment across disparate teams—such as product, marketing, sales, and data—by clarifying how their specific activities contribute to top-line objectives. By creating this transparent map, organizations can have more focused conversations about priorities, diagnose performance issues, and measure the true impact of strategic initiatives.

Implementing metric trees effectively requires a blend of product-thinking patterns and data discipline. Methodologies like Event Storming and Domain-Driven Design are crucial for identifying the core processes and high-value “heartbeat events” that underpin the business. Ultimately, a metric tree is a strategic tool for planning, communication, and high-level monitoring; it complements, rather than replaces, the need for deep-dive, explorative dashboards.

The Challenge: Isolated Metrics and Disconnected Teams

Organizations often struggle with a fragmented understanding of performance, driven by metrics that are viewed in isolation and teams that operate on different “planets.” This disconnect manifests in several key challenges:

• The Problem of Isolated Metrics: Stakeholders are frequently presented with long lists of potential metrics (the “PDF with 100 SaaS metrics” problem) without any context for how they interrelate. A metric like MRR, viewed alone, offers little guidance on what actions to take. As Timo Dechau states, the critical missing element is the “relationship” between metrics, which is essential for making them actionable.

• The “Iceberg” of Complexity: Metrics that appear simple on the surface, such as “Active User” or “MRR,” conceal immense complexity. Defining these metrics accurately requires a deep understanding of the product’s specific use cases and the business’s various edge cases, a process that can take weeks or months. A well-defined metric can “change the track for a company,” but a poorly defined one creates confusion.

• Siloed Data Cohorts: The data domain itself is often fragmented. Cohorts focused on the data warehouse, product analytics, and data science frequently use different techniques and technology stacks, even when working with similar data patterns. Product analytics, with its focus on behavioral event data and sequence analysis (funnels, cohorts), has historically operated separately from classic Business Intelligence (BI), creating what Dechau describes as “two different planets” that rarely interact.

Defining the Metric Tree

A metric tree, also known as a driver tree, is a visual framework that maps the relationships between metrics, showing how lower-level inputs drive higher-level outputs.

• Core Concept: It functions as a deconstruction of a primary business goal into its constituent parts. Every metric has a relationship to another, and the tree makes these connections explicit.

• Primary Function: Its purpose is to translate a high-level, non-actionable output metric into a series of actionable input metrics.

◦ Output Metric: A lagging indicator that reflects past success (e.g., Revenue, Profit). It is difficult for a team to influence directly.

◦ Input Metric: A leading indicator that teams can directly control (e.g., New Accounts, Conversion Rate from Trial).

• Structure: The tree is a hierarchical model that can often be expressed as a mathematical equation. For example:

◦ MRR is composed of New MRR + Expansion MRR - Contraction MRR - Churned MRR.

◦ New MRR can be broken down further into New Subscribers * Average Plan Price.

◦ New Subscribers can be derived from New Accounts * Account-to-Subscriber Conversion Rate.

This decomposition continues until the metrics at the lowest level of the branches are things a team can directly execute against, such as running more webinars to increase New Accounts.

The Strategic Value of Metric Trees

The primary value of a metric tree is not just in the metrics themselves, but in the clarity, alignment, and strategic conversations it enables.

A Shared Map for the Business

The metric tree acts as a universally understood “map” of the business operating model.

• Creates a Common Language: It allows different departments to point to the same part of the map and understand how their work affects others, breaking down communication silos.

• Fosters Transparency: It makes the mechanics of the business model clear to everyone. For many employees, it may be the first time they see a clear illustration of how the company generates revenue.

• Reveals Interdependencies: The map highlights how initiatives in one area can impact metrics elsewhere. Dechau notes that workshops to build these trees often lead to “eye-opening” moments where teams realize their actions might be inadvertently hurting another revenue stream.

Driving Action and Measuring Impact

Metric trees connect daily work to strategic goals, making it easier to plan and measure initiatives.

• Connects Strategy to Execution: Teams can clearly see their area of influence on the map. A marketing team knows its efforts to generate new accounts are a direct input to the company’s overall revenue goal.

• Measures Initiative Success: A specific metric tree, or “sub-tree,” can be built for a new initiative. This allows the team to define success upfront and provides a “control instance” to validate whether local efforts (e.g., A/B tests) are creating a meaningful impact on the larger business goals.

• Identifies Opportunities: By populating the tree with data, teams can spot areas of high potential. For instance, a part of the funnel with high volume but low conversion rates becomes an obvious target for optimization.

“If a metric is not actionable, it will have a hard life. It lives lonely on this dashboard and no one has an idea what to do with it.” - Timo Dechau

Implementation and Good Practices

Building and using a metric tree is a strategic exercise that requires a structured approach and an awareness of its limitations.

Starting the Journey

1. Map the Process: The first step is to gain a deep understanding of the customer journey. This is best achieved through a collaborative workshop, like an Event Storming session, involving people from all relevant disciplines who can map the process from initial awareness to long-term retention. This process naturally identifies the key milestones and potential metrics.

2. Align with Strategy: It is critical to sit down with the leadership team to understand their strategic priorities for the next 6-12 months. This alignment ensures that the initial metric tree focuses on what is most important to the business, which dramatically increases buy-in and adoption.

The Art of Event Tracking

A robust metric tree is built on well-defined data. This requires moving beyond simplistic interaction tracking to a more meaningful model of product usage.

Tracking Interactions

Capturing every click, scroll, and granular user action. This is the “take everything and look at it later” approach.

A high volume of noisy, low-signal data that is difficult for human analysis and disconnected from business success.

Tracking Product Usage

Applying Domain-Driven Design to identify core business entities (e.g., Account, Subscription, Project) and their lifecycles (e.g., Created, Updated, Deleted).

A small set (~15-20) of high-value, meaningful events that directly reflect product usage and can be used to build core metrics.

A key goal is to identify the “Heartbeat Event”—the single, central event that proves the product is alive and delivering value. For Slack, this is sending a message; for Miro, it’s adding an asset to a board. This core event can often be used to define multiple key metrics.

Practical Considerations and Limitations

• Keep it Simple: Avoid the temptation to “boil the ocean” by mapping every conceivable metric. An overly complex tree with 90+ nodes becomes an un-operational “monster.” It is better to start with a simple model and use sub-trees for specific initiatives.

• Acknowledge Timelessness: A standard metric tree exists in a timeless space. It does not inherently account for the time lag between an action and its result (e.g., an increase in new accounts may not impact revenue for 60-90 days). This requires separate cohort analysis, which does not fit the tree structure.

• It’s Not a Dashboard: A metric tree is a tool for planning, communication, and high-level monitoring. It is not designed for deep-dive, explorative analysis to find the root cause of a problem. Dashboards are still required for that function.

• Who Does the Work? The skillset required—combining product thinking, data expertise, and strategic facilitation—is often found in senior data leaders. Heads of Data are well-positioned to lead this work, as it provides them a strategic lever to connect their team’s efforts directly to business value.

The Role of North Star Metrics

The concept of a North Star Metric is closely related to, but distinct from, the top-line metrics on a business-focused metric tree.

• Correct Definition: A North Star Metric is fundamentally tied to successful customer experience. It measures the value customers receive from the product. Revenue is the result of delivering this value, not the value itself.

• Common Pitfall: A frequent mistake is labeling a revenue goal like “New MRR” as a North Star Metric. This confuses an internal business outcome with customer success. Shane Gibson jokes these should be called “South Star” or “East Star” metrics instead.

• Relationship to Metric Trees: The North Star Metric is an ideal candidate for the top of a product-focused metric tree, with input metrics below it defining the user behaviors that lead to a successful customer experience.

On North Star Metrics: “The concept of a North Star metric is it’s always bound to successful customer experience, it has nothing to do with revenue. Obviously we hope when people have a good customer experience that... it has a causal connection to more revenue.” - Timo Dechau

Future Outlook: Metric Trees in an AI-Driven World

As technology evolves, the principles behind metric trees become even more critical for managing complexity and providing essential business context.

• Context for LLMs: A defined metric tree provides a structured map of the business model. This is invaluable context for AI agents and LLMs, enabling them to perform more accurate “what-if” simulations, brainstorm strategies, and answer complex business questions.

• Governing Democratized Development: The rise of AI will enable the rapid creation of thousands of small, single-purpose applications (”AI slop apps”). This will create immense data complexity. A framework of core concepts (e.g., a universal definition of “user”), governed events, and a central metric tree will be essential to ensure these new applications contribute to meaningful business goals rather than creating data chaos. The metric tree provides the “principles, policies, and patterns” needed for this new landscape.

Tired of vague data requests and endless requirement meetings? The Information Product Canvas helps you get clarity in 30 minutes or less?

Transcript

Shane: Welcome to the Agile Data Podcast. I’m Shane Gibson.

Timo: , And I’m Timo Dechau.

Shane: Hey, Timo. Thank you for coming on the show today. I would like to talk about metric trees, something that’s intrigued me for a little while, so it’s great to have somebody who knows a lot about them on the show. But before we rip into that, why don’t you give the audience a bit of background about yourself.

Timo: I didn’t start out in data. I don’t know. I, it is interesting question would be if anyone actually ever started out in data. I started out in product. I will give some connections to metric trees later. Why? It basically was really important for me to connect it with product. . I started out on product was very quickly annoyed by people doing features just by gut feeling. I wanted to have a different kind of layer, so therefore I started to introduce analytics data to it. So this was really the early days. It was not even called product analytics at that point of time. The term came later. , So I spent a lot of time while walking through different kind of product roles to always setting up data setups for that. Also did the first connections there with data warehouses. Just a slightly little bit. And then after eight years in product, , I decided to go all in data.

So it was more focusing on the analytics side of things. So I’d say classic marketing analytics and product analytics. At that point of time, it was the high time in Berlin where a lot of e-commerce startups were really growing a lot. And so I was doing first data warehouse setups mostly in the marketing analytics space. Like people wanted to have their own attribution models and these things. I did this I spent then a lot of time in this kind of area. And then at some point I thought, okay, I want to go back to the roots and want to do more product analytics again. But I wanted to do product analytics in the data warehouse, , which required some things that were not so common at the time.

So one is like an event data model, which some people had some experience with, but not. Potentially for product use cases. , And then the second thing is there was no approach, like Amplitude postdoc were the classic tools that you do product analytics, they then were possible to put them on top of a warehouse. But then I figured out actually there, most of the time they’re too complicated for just doing product work. So in the end, you have to have a metric first product analytics approach. And this is where I came back to metrics where I never really had a good relationship with them.

So we might talk about this in this session. But I had to basically revisit my relationship to metrics. And this is where I came across metric trees again. I had them already at university. And yeah, starting to dive deeper into this.

Shane: , I’ve followed your stuff for a long time, so thank you for sharing so much great content. And what’s always intrigued me is in the data domain, I could almost see a data warehouse cohort, a product analytics cohort, and a data science cohort. And , although each of those cohorts are using very similar patterns, very similar techniques.

They always seem to use different technology stacks. And then we had the whole, what was it? Composable, combustible, whatever it is. Yeah. Yeah. So that idea that you don’t need a dedicated product analytics tool. You can just do it off your cloud data warehouse or cloud data warehouse database.

So it’s really interesting that there was almost a separate track, wasn’t there for product analytics and the techniques you used and the technologies compared to some of the other, cohorts or tracks in that data domain.

Timo: It is super weird because like sometimes you talk to people in the data space. So let’s say who do classic bi and , this was my problem when I was starting out. So because I was looking for specific way how I can model product data, the same as marketing data. Marketing data is a bit less, but like it’s a lot behavioral data.

So a lot of event data. And you do sequence analysis so funnels or cohorts and something like that. And it was really a hard time to talk to people who are in the classic BI space and tell them . Most of the data models don’t really work for me because I have to do sequence analyze.

So I cannot create 200 fact tables. It would be a bit crazy. And , I really had a hard time to explain, but it was also interesting because I learned a lot from them. I think some learned a little bit from me but it’s interesting because everything calls data, but I always call it they live on two different planets and they rarely visit each other.

I think it’s getting more, it’s getting better. So I think there’s more conversion now and something which I, for example, work a lot on. So my last, let’s say my last mission, maybe it’s, if it’s my last, I don’t know, but my biggest one is really crossing the whole marketing and product behavioral data with revenue data which then makes it really interesting if you do this.

Shane: We had a customer and, they were a scale up.

And so we needed to do some work around . Standard metrics for a startup. The pirate metrics, a r, all those kind of good things. And when you first go into those areas, you go, how hard can it be? And then you find out

And it’s damn this should be a solve problem right now.

The core metrics for startups, should be well-defined. They all use, standard type of software as a service products for capturing subscriptions and signups and, all those kind of things. There’s three or four of them that everybody tends to use. It shouldn’t be hard, it should be patterns out there that you could just implement and make it really easy. And again, whenever you say those words, you know it’s gonna bite you in the bum and then you start doing it and you’re like, actually for some reason there’s a whole lot of complexity just hidden under the top of that layer that bites you.

Timo: I just do MRR calculation. So it’s let’s say technically these are all the same metrics. So there’s, let’s say a whole metric set around MRR. But the tricky thing is , how do you define it under the hood? Because everyone comes to you and in every case, company tells me like, no, we have a very standard subscription model.

There’s nothing crazy in it. And then I always know, okay, yeah, this will hold for two weeks until we do the first investigation into all the edge cases, and then we will just cover all the edge cases. Ah yeah, we, there we had to do something different. Oh yeah. And then yeah, we had this switch from one subscription to the, and so it’s always messy.

And that’s the same with product stuff, I think the pirate metrics are still a really good framework to understand how product, and let’s say this business around is evolving. But let’s say just go for something like retention or active user or activated. So activated by default is in the end one metric, but it can take you weeks and sometimes months to redefine how you basically calculate this because it really comes down how the product works, what kind of use cases are you actually solving.

So it’s a fascinating, interesting thing, but it’s also good ‘cause in the end it’s one metric that you just see, but it’s an iceberg. So on the top you just see this one metric, but quite complex underneath. But this is where the fun is. If you really do a really good job there and the definition and so on, it can really change the track for our company.

Shane: Even then though, there’s still outside of the metrics, there’s some core patterns you’ve gotta decide from an engineering point of view. So again, when I look at product analytics, when I look at event tracking in a software product, typically you have a choice of pull every event. so effectively pull event logs to say lots of things happened, and then apply the events you care about after the fact.

So more of a Lakey type pattern. Or only define the events you want in the product when at the beginning. And then only bring those in and use those. And if you want to look at a new event, you’ve gotta go and define it again. And there’s a trade off there, right between this,

If you get every event, you get this wash of noise and it’s really hard to figure out where the signal is.

But when I worked with a lot of product teams, they seemed really reticent to define the events that were important. It was easy for them to say, I’ll just take everything and we’ll look at it later, versus this is the event that will tell us that this feature was successful.

That’s actually quite a hard metric to define,

Timo: it is, this is this is another this is my second passion topic and this is the one where I wrote a book about. I think the problem is when you go into this, you have to distinguish between tracking interactions and tracking actually product usage. And so most of the setups are tracking interactions.

Where people click, and it’s quite understandable why you do this because you have to define something. So it’s not it’s not everyone’s job to define what you want to track. So it’s usually a side job that you have to do in between. So the, let’s say the most approachable way to do this is you open up your own application, then you see, okay, where can people actually do something?

And so they’re like, okay, they can click here, they can click here and they can do this. And this is what I mean you track interactions. If, let’s say, if this whole output is just for machines, then it might be. Okay. We are still not there. That, let’s say you can collect all these, let’s say very noisy, very granular data, and then you have something running over it, finding some patterns.

So far I didn’t really see things that go in this direction, but it might come at some point, but when a human has to analyze it, this doesn’t work. It’s far too far away from what actually defines product success or what actually defines business success. So it has too many noise, as you said, and so therefore what I ask product teams is define the use cases and define the jobs to be done.

Define the entities that make your product. So in the end, it’s classic domain driven design, what I do with them. And so okay, let’s define the different kind of entities that build up your products. It’s usually five or six. And then we define how does the lifecycle look into this? So let’s say we have an account can be created, can be updated, can be deleted. So three events updated usually you don’t need because there’s not often a business value in it if someone update an account. So who would optimize on account update updating usually. Yeah. Any there edge cases. But, and you can do this for everything else. You can do it for subscriptions. You can do it for whatever your product is built off. And then you can just end up with. This is usually my take. You end up with 15 events and the same, you end up with 10 metrics. It’s sometimes really quite crazy if you really have a good setup. So right now, I was just working on a project where we had one event that was defining six core metrics that could explain if the business is running or not. Because there’s always, this is always my interesting take when I work with the client is in every setup you have something which I call a heartbeat event. And this is this is a central event. If you just need to track one event, this is the one that tells you is your product still alive or not.

So for Slack, messages for Miro, if let’s say an asset, it’s added to a board or so, you have always this one event where you know, okay, when this is still coming in, things are good, we don’t know about it, but they’re still there looking good. And you have to get to this level, then you can basically tame the chaos that product analytics can cause.

Shane: It sounds a bit like a North Star metric. This idea that yes, all the other things are important, but actually if you can define the one thing that really is the core, focus on that and the other things are useful, but if you don’t focus on that one thing,

Timo: I always have to fight a little bit for the North Star metric because there are so many definitions out there, and I think the worst case is when someone comes to you and says yeah, new MRIs or North Star metric where you say, no, it’s actually not the, let’s say the concept of a North Star metric is it’s always bound to successful.

Customer experience, it has nothing to do with revenue. Obviously we hope when people have a good customer experience that let’s say it, it has a cau connection to more revenue. And then the North Star metrics is usually also built up by two or three input metrics that lead to it. But in the end, it’s the same what you said.

You have a set of three metrics that could explain how is your product actually delivering value at the moment.

Shane: I’m with you is that, the North Star metric is success for the customer, not success for the business because if the customers get the success that you promised them, then your business will grow. But yeah, time and time again, I see North Star metrics being internal metrics. And it’s maybe we should just go read the definition again or call it something else, south Star Metric or East Star. Like just give it a different name. That’s okay.

Timo: Yeah. You can always call us like this is our, yeah, I don’t know. I think North Star Metrics is just a great name, so it’s a great label. So you lo everyone understands what it is. It sounds great, like adventure. So this is our North Star. We have to go there.

Shane: That’s my standing joke is that as data people, we laugh about the fact that our stakeholders can’t define active customer, active , subscription. Yet as in the data domain, we argue about the definition of a semantic layer, a data product, a north star metric, day in, day out.

So we’re as bad, if not worse than them. So on that note let’s go and talk about metric trees. So if somebody said to you, what the hell is a metric tree? How would you describe it?

Timo: First of all the same. I think it might be also a definition problem. So some people call it driver tree. There’s also like the term of kpit. I’m not really good in definitions. I know that they have slightly different variations of it. So I give you my explanation and my definition of it. So one problem, what you often have with metrics is when you take a metric alone, so let’s say you have a software as a service. So you’re not really sure what you should measure. You are on LinkedIn. One of this posts where someone is posting, Hey, I just compiled the greatest collection of software as a service.

Metrics, like common metrics, and you get my PDF with this 100 metrics. These things never really work for someone. And the problem is why they don’t work. It’s because everyone picks then a metric and looks at it in isolation. So I don’t know. Yeah. For example, MRR, monthly recurring revenue. So what is you have a standard alone there and then you don’t really have an idea what you should do with it.

Because two things are missing and what defines the metric? We mostly is it’s missing a relationship. So let’s say every metric has a relationship to another metric. They have I cannot even think about one, which is really standing alone. So usually you have a relationship and why do you need this relationship? Because not every metric is very actionable. So we talked about this North star, when you say, Hey, revenues are North Star. So the problem with MRR monthly recurring revenue is. It’s an output metric, so it’s something that comes out at the end. So when the CEO goes into a meeting and let’s say assembles all the heads of the different teams and says we have to increase MRR, everyone would say, yeah, sure.

Sure it’s our business model, so it makes sense, but no one would immediately come up with an idea, oh yeah, sure, we should do that to increase more MRR. No, you usually break it down. So you break it down into, oh, okay. When we need MRR, and maybe we need more accounts because they could end up in a subscription and then this would be new MR. But then this is interesting because this is already the first step to build out a metric tree because then you have MRR, and then you say, okay, what is actually making up MRR? New subscribers multiply it with an average. Plan price or average price, and then new subscribers you can build up from new accounts with a conversion rate. And then you get into your first version of a metric tree. And I think the nice thing about a metric tree is that it explains how you can get from something that you will end up with to something that you can directly influence. So if you go to a marketing team, tell them , Hey, we need more new accounts, they usually have an idea what to do. So they will like, oh yeah, okay, maybe we have to run more webinars or we have to invest in a podcast or whatever. And so there you make it actionable. And at least this is my driving force. Why metric trees are interesting because they help you to break down something that you want to achieve to something that you can actually do. And this is the tricky thing. So if a metric is not actionable, will have a hard life. It lives lonely on this dashboard and no one has an idea what to do with it.

Shane: So if I think about it what you’re really saying is there’s a bunch of metrics and they have a relationship to each other.

And we treat it like a tree. So if we think about an HR human resources org chart where we typically see a tree of people and people report up this tree, what we are saying is if we can find the relationship metrics that behave that way, so these metrics support this other metric which supports this other metric, then that relationship helps us work out what we need to change in our business to move one or many of those metrics.

Is that kind of the concept?

Timo: Yeah. That’s the concept. You can also use it for that. Where I like to use it is to really try to explain how the business works on a, let’s say, common concept that at least the data team understands and that the business team understands. And this is something which I would say the metric tree so far worked best.

So I tried different kind of formats to bring both teams together. We did it. Okay, what kind of events do we have to measure? Usually it doesn’t really work well because it’s too abstract and, You sometimes find common ground, but it’s too far away from what people really care about. If you really build up a metric tree, you have really interesting conversations with, let’s say, when I do these workshops where it’s usually with startups where it’s easier to get relevant people into one room.

So then we have people from all the different kind of disciplines. And we have really good conversations because for a lot of people it’s for the first time that they see how actually the revenue is coming together in the company. So this is let’s say often, unfortunately for data people, it’s the first time that they see, oh, this is how we calculate revenue. Because it’s usually one person who does this part, but not everyone and then for products the same, they’re like, oh, okay. I was not aware that we also do this to get revenue. So it’s a really interesting exercise that can, open up a lot of black boxes and understands how the whole system’s actually building up. And then. Obviously, if you go a little bit more down, then also, all the different teams might find a place in this tree where they say, oh, this is actually where I work hard for the data team. They usually don’t find a place there. But they’re the ones who can build the metric tree and can provide the metrics. But let’s say you can have a part where you can see what product can do, you will have a part where you can see what sales do. Then you will have a part where you can see, oh, this is actually the influence area of marketing. And then in theory you can analyze how these different kind of levers that you have, let’s say on a lower level, how they’re actually attriting to the whole final output metrics. Every time I did this kind of format in a workshop, it was always some kind of eye-opening in there where someone said, oh, I never looked into this. Or someone even said actually right now, I think with some initiatives we are starting to hurt our revenue and so on.

No, no one has recognized that because now we see it for the first time that when we do this, let’s say this stream of revenue will have a problem, which no one really thought about. And so this is something where metrics we definitely can help to bring transparency into. Let’s say the mechanics, how the business actually making money that often is completely overlooked because a dashboard in theory does it, but it doesn’t have the structure.

The tree structure is nice because , someone can look at it and immediately understands what it does. Yeah. So go from the bottom up to the top or the other way down.

Shane: I think that’s part of the core value of it, is that simple map. ‘cause what we know is we know we, there’s a lot complexity and when we draw a simple map, a simple diagram, two things happen. People can visually ingest it and understand it. There’s something about a human looking at a map that just naturally sees the pattern.

And the second part is you can point to part of the map and everybody knows they’re talking about the same thing. So that idea of different parts of the organization, different silos, understanding how they affect another metric. So you know, your example, how do we increase monthly recurring revenue while there’s a bunch of metrics that the marketing team can affect, number of ads placed, number of prospects found optimizing the funnel from prospect to signup.

And they can look at that part of their map and go I can do some work in there. And that in theory will increase our monthly recurring revenue, because that’s up the tree. So that idea of a map gives us understanding and also gives us a clarity of a shed language, i’m focusing on this part of the tree.

Timo: Yeah, I’m very happy that you come up, that you say the word map because this is actually what it is. It’s one map that can be very useful. So at least when I create one for the projects, it’s not that we constantly work with it. So we might get to this point like how operational are metric trees, but where I always use it is when we come together and discuss big things.

Okay, what kind of metrics we should focus on? Let’s say what kind of initiatives we should look into. So this can always help to have a then sitting around and when we say, okay, look, now we want to focus on this kind of area, you can always look then in the, let’s say in the tree.

And the nice thing is when you have the tree with some data, then you can see, okay. We focus on this kind of area, , because we have the feeling we have a lot of volume here. But we can see for example, the conversion rates in this kind of areas are not really high. So there’s, let’s say there’s quite nice potential for us to improve things just slightly a little bit, but see a big impact because we see that a lot of push comes in there. , And so then we can analyze how this works. For that, it’s really nice to have it because again, everyone immediately knows where we are. So it’s not that you have too long. Explain how this might have an impact on revenue, because immediately everyone sees that

Shane: The good thing for data teams is it means they don’t have to go look at yet another stupid strategy PowerPoint with four boxes that have no context and no understanding and no data.

You might have to, but at least they can say that, that part of the pyramid or what are we doing this week? The circle, whatever the latest consulting picture is for for selling a story.

How does that map to the tree? One of the things you said though was data teams would struggle to figure out which metrics are theirs and that, that kind of gave me an idea. So, you know, One of the things I work on a lot is this idea of an information product, and one of the key things I teach teams is focus on the action and the outcome.

If you don’t understand what action the stakeholder’s gonna take and what outcome’s gonna be delivered from that action and the value of that outcome, then really there’s a risk. You’re doing data work that has no value. While we say stakeholders, were accountable for that actually as data teams, we should be as well, we should be holding our stakeholders to account that actually they can describe the value, at least the outcome,

That’s been taken.

So what would be interesting is if there’s a metric tree in place, a data team, which should be able to point to the metrics that they’re doing data work for or with to improve, i’m working with the marketing team to use data to reduce the time between a prospect being identified and signing up for an account.

And so they should actually be able to use the metric train to show where the data work is adding value to the organization in conjunction with those stakeholders, those different business operating groups,

Timo: yeah, exactly. I think this is where it can really play a really nice role because can sit in these, let’s say you work with the marketing team. Marketing team does some planning for the next two or three months. So they have some ideas let’s say they come up with three initiatives, what they want to do. And so you can take every initiative and you can then say, let’s say, okay, here on the big company metric tree these initiatives. They’re trying to improve these kind of areas. And then we can use this metric there as always, as our our control instance to really see, okay, do we see some impact? Because that’s a tricky thing. You can run a lot of initiatives and locally looks really great, but when you look at it on the big picture, everyone is yeah, I don’t know. AB tests look great, but I don’t see any kind of uptick somewhere. So it’s definitely nice to have, okay, this is the one big metric that it should, let’s say influence and let’s see if we can fire it up enough that we can see some influence. Then what you can do. The second thing is let’s say you can create different views of metric trees. So you can do this very high level for a company, but then you can also take every initiative and just build a metric tree for every initiative.

This is something which I often like to do because again, it helps you to understand, okay, how do we measure the success of this initiative? So you might have this one big metric that we identify in the company metric tree that let’s say is at the top end. And then we break it down for this kind of initiative.

And so then it really comes, let’s say you want to improve conversion rate. So then we can build the whole surrounding around this conversion rate and maybe even break it down by three different channels because this is what we are trying to achieve. You’re trying to get more people from this kind of channel.

So then you basically bake strategy into the metric tree and the strategy connects. Directly to the initiative that marketing is driving. And then everyone has a very clear idea what you do. And then also you will build something that in the end, can marketing and you can use to explain if this initiative was successful or not. , And you can validate it before you can ask people, okay, look, when we deliver this, does this make sense to you? When, let’s say when the initiative is over in four weeks and we report on this metrics and we define, okay, success looks like when we move this part, I don’t know, by 10%. So does everyone agree with that? And then because I have the feeling it makes it for a lot people easier to think in that way. So that they not don’t say, yeah, I don’t have really idea if I should agree or not to this because at least, yeah, I think metric, at least they understand. They might ask, okay, how do we define this kind of metric? But that’s fine. And so then we can spend some time to say, okay, how we define it. But I like this approach a lot. Also like to not always see the metric tree as this one big, okay. We explain the whole business model, but really to use it to explain how I run a specific initiative. It’s also nice for me. Let’s say personally when I work on these, let’s say, as a supporting part for data for these initiatives, it’s a great brainstorming tool for me. So let’s say someone comes up with this initiative, I do a first version of a metric tree. Then I say, does it really represent what they’re actually doing there?

And then I say, yeah, to a very generic part, but maybe not customized enough. So I’m always trying to tweak the metric tree that the people who run the initiative immediately find it in there. So let’s say I do something for e-commerce. And the e-commerce is really pushing and getting high loyal customers. Let’s say , they really want to improve the segment of high loyal customers who buy all the time. There’s these, let’s say, high buyers that you sometimes have. So when I just report on let’s say new customers or returning customers. Then I don’t really have it covered because yes, we are going for returning customers, but for a special segment within returning customers.

So therefore what I can do is I can say, okay, I break it down in three groups and new returning and let’s say VIP customers and then with the VIP customers for the first time, I make it visible what kind of impact the initiative right now has. Or let’s say I can see, okay, how is the share of these users is growing or not growing or whatever. And so this gives me a lot of, playroom to build something that can really support what the business is trying to do, but do it in a language or in a way that at least most of the people understand.

Shane: All righty. So I’ve got a shit ton of notes right now. So let’s let’s go through them one by one, because there’s so much gold in there. So the first thing you pulled out is this idea that actually metric trees is a shared language. Yeah. So you can, you’ll define a metric tree for what you think you heard from the organization of how their business operates.

And by putting that map, by putting that tree in front of them, they’re gonna identify where you’ve got it wrong. They’re gonna look at it and go yeah. Oh no, hold on. We don’t do that. Or we are different or, I don’t understand. It becomes that visual shared language of an entire business and their operating model

Timo: it maps the process to some degree, and that makes it easy for people to see, okay, is it actually what we do?

Shane: Then the second thing is you said when you want to go and do something new, you wanna do an initiative or some investment or change a process or go into a new market, you can pick the metrics you think you are gonna impact and you can guess how much you’re gonna impact them by. I remember many years ago, it was probably late nineties, early as two thousands, I got into the whole balance scorecard thing.

I was working for a vendor balance, scorecards were hot. A couple of ‘em were trying to build software. And one of the things was, you had this metric and effectively you could put a budget on it. We are gonna increase that by 10% over the next quarter. Now the software itself was pretty hokey.

But it was the conversations around what are we doing and how’s it gonna influence that metric. So that’s one of the key things you called out , is by saying that if we do these things, let’s guess which metrics are going to change, is that what you’re saying?

Timo: Yes, exactly. And then also really make sure that you measure this metric in the right kind of way. So that, for example, let’s say you have a corridor that you can see, okay, is this a normal movement of the metric or not? Or that the initiative is big enough that it can move the metric. This is a tricky thing. But often. At least when I work in product or marketing is that often you see a lot of initiatives that are just, let’s say they’re very tiny bits. ‘Cause I don’t know, they’re not really bold. And then it, you will not see anything. And you can even tell this before, okay we try to optimize 5% of the typical audiences that we get in there. So we are working on this. There might be still a strategic reason for that, but then you can still make it clear. It’s okay, so we work on that. But because , let’s say the sample or let’s say the audience is so small, we will not see an impact when we look at the completely blended global metric. So then, for example, you have to break down this metric maybe in different kind of path that you can still see something. But this check-in really helps to see, okay, how do we actually want to see if it makes an impact or not? ‘cause sometimes just because of the setup, you won’t see anything. You can already know this, that it might not happen.

Shane: And that was one of the problems with the balance scorecard. So we had this idea of cause and effect, we said that, I think you call them input and output metrics, but this idea that if I improve this metric on the bottom or the left then it has a cause and effect with a metric above it or on the right and.

Back then, we really wanted to find correlation, we really wanted to say well, actually, if we just throw all the data at the machine, the machine should be able to find causality or correlation between the metrics, I should about to codify this. It should be accurate. And we never got there.

We still don’t have the technology or the patterns to do that at the moment.

Timo: no. I don’t think so. So there, this is definitely not my area of expertise, so I know some people do some things like that, so they try to find out, , because you have some relationship when you build a metric tree that are not deterministic. So let’s say a classic metric tree, you can write as an equation because it’s basically that’s also quite nice that you can do it. But sometimes, for example, let’s say you have a metric which is called active users. And so you have a very well, well-defined active user definition where you say, okay, the people don’t just show up, they also do valuable stuff within your product. And so once they do it, and they do it within the last 30 days, we basically flag them as active users. So there’s definitely a correlation between active users and at some point starting a subscription, let’s say you have a free plan, and so you have an active user and a free plan. So there’s a correlation between the both, because you might assume, okay, someone has to be active to end up in a subscription. But let’s say it’s, it’s it’s not a direct connection, so you cannot really say, okay, whenever we get someone as an active user, we have a probability of, I don’t know, 57% that they will end up in a subscription. I guess this is more straightforward to calculate, but even there, I never really, came across, let’s say, super good models that can , predict this very good.

If someone knows this, please let me know. Ping me on LinkedIn. I’m always interested

Shane: Now the answer’s just AI slop, right? You can just AI it

Timo: Yeah

Shane: and maybe you can, maybe it’s good, that non-deterministic patent matching will be really good at this. I dunno, I haven’t tried lately,

Timo: So if we go to this, what you had before, like we, we create a lot of noise and we might throw it in and we might find something yes. That’s, yeah. But I think for correlation, analyze is potentially not,

Shane: but the key is the. Visualization of a map, the conversations of what on the map, the conversations of what you’re doing to improve the numbers on the map, the relationships that actually is where the value is. So yes, we could get programmatic deterministic correlation across the metrics. That would be awesome.

But actually the conversations so the next one is, our natural reaction is to boil the ocean.

If I think about an information product that’s lifetime value. I tend to say to organizations why don’t we break it down? We know that to do a lifetime value model, we actually have to have revenue cost to serve churn, there’s a whole sub submodel sub products that have value. Why don’t we build the revenue model first? . And give you a revenue information product. Why don’t we do the cost to serve second, and then over time we’ll get to that lifetime value. I can imagine with metric trees, the natural reaction for some data people is to define every metric.

So draw the map end to end. Define every metric before we even start doing anything. Where for me I like the ability to change fast. So for me, I would be going maybe sketch out a map really quickly as kinda like a blueprint of what we think , it looks like, and then focus on some metrics and define them and build them and deliver them and monitor them and use that information.

And then. Kind of color in the map over time as you learn more, ? How do you do it?

Timo: so I had a phase where, let’s say I was in the rabbit hole and I did one exercise where I was trying to map my whole content production as the most extensive metric tree possible, and it was, I still have it somewhere in Miro. It was really a monster. It was like no one could ever implement this. I think it had in the end, I don’t know, 90 nodes. And obviously this doesn’t work, so I could easily see that. It was a nice exercise, was a nice thing to do over the weekend. But it’s nonoperational so it doesn’t provide any operational value to do this. And by the way, it would not even possible to track the whole thing because it would include a lot of attribution stuff, which is not possible. So the same, what you say, I think like the model that I used now is. Try to really keep it simple. I don’t know. Let’s say also try more to work with sub trees and don’t be too, let’s say don’t be too deterministic with it. So give yourself the liberty to say, Hey, I create a, let’s say a specific tree for this kind of new product feature that gets introduced and have no idea how it connected to the other tree.

Now that’s fine. I live with that because local optimization is still better than no optimization, so therefore be nice to yourself. But no, you definitely have to keep it. Quite simple and then you have to know what you can do with it. So I know that some people use it for root cause analysis, and I think for that it’s really quite nice because I could you, you can say, okay, look, we lost so much new revenue let’s say, compared to last month. So can we follow up with the tree to really see where do we see where stuff got off? So it can help you to, let’s say, get a starting point, but then still, the deep dive is something different. The metric tree will not tell you where it happened and then. Another thing that I discovered was, is often overlooked, especially when you talk about metrics and the metric tree is, it’s really big risk is you have time is completely out of this thing. So the metric tree lives in a timeless space. So the problem is for example, you increase new accounts in the example that I had before. And then you wonder, okay, why don’t I see any kind of new revenue? Yeah. Because it happens in 60, 90 days. So whatever your model is or whatever the average time is that people usually take to upgrade into subscription.

So the metric tree always does a very bad job where things, let’s say down in the branches are already impacting and you see the impact three months later at the top and you have no idea where it’s coming from so if you would do it properly, you would do a cohort analysis where you cohort everything that happens something.

But this doesn’t work on a tree. I think the important thing about working with trees is really to know the limitations, whether it’s great or not. As I said, for me, mostly for planning, for brainstorming, for communication. I guess some people use it also. For these check-in meetings, let’s say have a weekly meeting, you check quickly, okay, how’s the business performing in specific areas? I think that can work as well. But it’s not like that. Some people thought, oh, get rid of the dashboards and we just do metric trees. So no, that doesn’t really work. You still need specific, let’s say explorative dashboards to figure out why a metric is actually looking a little bit wonky this month.

Shane: I think back in, in the balance scorecard days, there was this concept of simulation. , It was almost like a digital twin. And although that term wasn’t around then from memory, so you could actually go in and say, if I change this input metric by a certain amount what’s the flow through?

And from memory we were inferring a delay. like you said, number of prospects coming into the funnel, how long would it be before they create an account and go into a pay plan? How long would it be before that money turns up if it’s a 30 day window for the bill? And so you bake that into the model to a degree so you could simulate some changes, but, that was quite technical.

Timo: What I often do is I take a metric tree and I just break it down on a spreadsheet and then create something which we can call a growth model, where I then have the metric tree on the left defining all the different kind of rows, and then I can put the timeline on the right in there, and then I can just model what you just described.

I can do some forecasting. And for me, the interesting part is I see the mechanics of the business and then look, I can change some conversion values and can see, okay, what impact does it have when we actually increase this by 10% and so on. And then I see it how it plays down. So it’s a very basic and amateur way to do forecast simulation, but often quite enough for most of the stuff

Shane: And again, , that relationship’s an interesting one and you just made me think about it. So we’re doing a whole lot of work in what we call the context planes. It’s this idea of taking the context of our data and bringing it back into, for us it’s centralized.

And we think about it as four types of context we think about as business context. Actions, outcomes, glossaries descriptions, those kind of things. We think about it as a structural context, so physical schema data types and all those kind of things. We know about our data. We think about it as operational context.

So when was data loaded, when was it refreshed? What’s the quality score? Those kind of things. The last one we think about is agent context, the prompting, reinforcement, those kind of things. And so there’s a bunch of object types. We have, so we have, actions as an object type outcomes, an option type metrics as an object type a fact, a value as an object type.

Then the last part of it is the relationships across those object types. And the reason we’re working on this is this context map. Effectively, if you give it to an LLM it’s really good at using it to answer questions. So you can do what we call blast radius. If I change this bit of code.

If I change this object, if I change this metric, what’s impacted. And so thinking about metric trees, they’re giving us a relationship across those metrics, which is effectively describing the business model for the organization. And that relationship would be really valuable to an LL lm if you’re using an AI agent because you can say, if I touch this metric, you know, what the relationships are infer what’s gonna happen, and that actually might be a non-deterministic way of getting a simulation quicker than having to build really complex data models.

Timo: Yes. So I was experimenting a little bit with that. Let’s say in the easiest way, you just do a copy of of your metric tree and put it in and then say, okay, can you please run this in specific ways. I also did some experimentation with, let’s say, descriptive YAML format for metric trees was not really happy with that. So I definitely, so far I have abandoned the idea to do this. But no, you’re right. Also because when you put it in LMM, then for example, you can ask, okay, can we run a cohort simulation on top of that? So let’s assume we run this initiative in the next months where we think, okay, we will increase this kind of metric, and then we want to see how the impact will look like over the next month.

And what I often see. For example, when I have, let’s say full data set up for let’s say a startup. And then the metric tree is always you can use it in an l and m and then you can say, okay let’s brainstorm on this. So right now we have this problem here, or we want to dive deeper into retention.

It’s this is the whole picture, so can you help us to break it down? So can you give us different kind of versions of what, let’s say what would the level underneath look like? And still at the moment, this is my favorite LMM workflow or let’s say use case is really to use it as this massive brainstorming machine where you can say, look, okay, let’s look at this angle.

And I can say, oh, okay, let’s slightly change this kind of angle. If we bring this, and then because you have so many things, then. Mapped out to you so you can immediately say, oh yeah, this is the direction we should follow on. Let’s go deeper there.

So it definitely helps to not do crazy stuff because you give it a, let’s say, a form or structure

Shane: And then my natural reaction is, oh no, because LMS are non-deterministic, and so the simulations it’s gonna run is wrong, but we’ve just already said that, you Correlation of cause and effect across the metrics is hard to do if not impossible. So a non-deterministic engine is just behaving like a human going, this is what the patterns we see.

So actually using it to do those what if analysis, those simulations, those, how are those things related is just as valuable as humans doing it. I hadn’t thought about it that way.

Timo: and the Usually nowadays would, take the other approach and would write a Python program.

Often, when you ask these things, they say, yeah, let me write a quick script for you. Usually does that, and then you can just check.

Shane: Interesting space. I think , we’ve seen metric layers come out as bi semantic layers. And they’re not, from my view, they’re not making it right. They’re not really getting traction on the market. Maybe we’ll see a reinvigorated of the balanced scorecard products as metric three products.

‘cause there’s probably some value there. Alright so let’s just go back to basics, i’m rocking into an organization, or I’m an organization and I wanna start off this journey. We’ve talked about the map is the most important thing, the shared conversation. So we need to build that out over time.

And we’ve talked about doing it, step by step, don’t bore the ocean. We’ve talked about the fact that actually defining metrics are hard. . So each one of those you’re gonna think is, oh, it’s only MRR. How hard can that be? You’ll find out. So we know that each metric to define it and implement it actually is a lot harder than we normally think and takes time.

Where do you start? , So if we think about this idea of input and output metrics and metrics in the middle that we’ve got this idea of a heartbeat metric, the core metric for your organization, that actually is the one you want to look at the most, and that’s gonna be made up of a whole lot of input metrics, we know that. Where do you start? Do you tend to start at the input side? Do you tend to start at the output side? Do you tend to start in the middle? , How do you decide which metric to do first?

Timo: Huh, that’s a good question. I start on two areas. Area number one is I usually bring, let’s say, a set of people of this company into one room. That can explain, let’s say the whole customer journey or the whole customer flows in this company to me. I usually work with, let’s say startups, which are between seed and series A.

So therefore, the whole business model or the whole business processes are not super complex yet. , So it’s definitely possible in three hours to map the whole thing out. So I do a classic event storming session. So not classic event storming is a little bit broader and it does more things. So like I do a reduced version because I’m just interested in specific things. But. When you ask a company to explain how the customer’s basically going through the whole process, then you will identify these things. So you will identify, okay, what are the actually important points that they have to come to that in the end? Like it means success for you. So we will find this one place usually pretty quickly where we say, okay, this is a sweet spot. So that we definitely have to have a metric for that. We just have to see how we define it. For example, an active user is a North Star metrics in general. It’s it’s a definition process that takes longer because you come up with a hypothesis where you say, okay, people have to, okay, I’m just making this up, let’s say when you’re miro, so they have to create one new dashboard every month. They have to share at least two. And then they have to add at least 20 cards on a board or so, so then you would say they’re active. So this definition has to be fine tuned over time to see, okay, does it actually stick? This comes later. So I start with understanding the whole process. This is part number one, part number two, and this was something that I learned later. So I started very early on with these event storming maps because I use it already in other projects for other things. They’re really great to understand the process very quickly. The part that was always missing for me, that came later for me was. I sit down with the leadership team to understand strategy. I really want to understand where they want to go in the next six months. So what is the thing they want to improve which kind of direction they want to go, because from there I get the priorities, what you just asked.

Okay, where do we get started? So what do we have to build out first? I always try to bring it as close as possible to the strategy that the company right now is doing, because if I do it, I usually get a lot of more interest by management teams. By everyone I mean then by everyone else because they have to report towards this. I won’t do this, people would like. Yeah. It’s interesting, but it doesn’t really. Let’s say count into our current initiative. So if I fail to support a current push, strategic push into something, then usually I get a really low adoption rate. So this is like the second, I would say most essential part that I really try to understand, okay, what they’re trying to achieve in the next six months.

Shane: So once you’ve blueprinted out or, event stormed the kind of, I, I think of it as a blueprint, you’re creating a hypothesis of what the metric tree might look like when you’re finished. Then what you’re saying is you then look and talk to ‘em about their strategy to figure out where you should zoom in on you’re basically saying as part of that event, map that metric map.

And based on your strategy. I think we should do these ones first because they seem the most valuable, the ones that’ll get the most traction or have the largest conversation and adoption internally, which is the goal, is to get more people to understand it and use it and go, yeah, these are valuable.

Let’s do the rest of them.

Timo: It’s even slightly different. It’s not like just highlighting the areas. It’s often there’s a different version of a metric tree that I use. So you can take a metric and you can break it down, buy specific things. So this is the classic thing that you do in BI as well. So you have sales, you can break it down by something. You can do the same thing with the metric tree. So you can take a total metric and then you can break it down by some kind of segmentation, whatever it is. In this segmentation, I can try to incorporate the different kind of strategy. So I want to have, let’s say some, okay, let’s say, miro, I’m always using Miro because most of the people have used it before, so let’s say they push into AI supported workspaces. So then I’m trying to get up metrics that will highlight these, let’s say the adoption of AI supported workspaces versus non AI supported workspaces so that everyone on the first glance can immediately see, are we making progress with that or not? And so it’s really like creating a variant of the generic metric tree to say, okay, this is how it would be useful for the current strategy.

Shane: Okay so lemme play that one back because I just I want to make sure I get it. And is this idea of the buy statements of these metrics, I want this metric buy channel, before we started, we talked about the tools we use to create our content. And we talked about the fact that I use InCast for recording, I use Descrip for editing and I use Podbean for publishing.

And now what I’m seeing is a convergence. Yeah. I’m seeing each of those three products add the other features , that I do need into their product. And I was ranting about how that actually degrades the value of the one thing I use ‘em for. So let’s say that we’re zencaster and we wanna move into the real time streaming and video editing.

So what I’d be looking for is the metric of. Active, whatever, maybe active user, maybe active sessions. And then I wanna be saying, okay, how many people are moving from audio recording only to video recording? Because that’s a input metric. If they’re not doing that, they’re never gonna use our video editing feature.

And then how many of those move from video recording to video editing and then segmenting it maybe by region? And we say actually the US is the target market, so how many people? And by doing that, we’re effectively going, there’s a really small metric. Effectively, we’ve broken it down to its smallest part, but it’s there to support our strategy of getting more people using our video editing.

And so that’s an okay, that, that makes sense for me.

Timo: No, Is a perfect example. So exactly like that. And usually it makes it a lot easier than to talk to the people and people who find it very interesting. They want to see the results from that.

Shane: And you can get instant feedback. ‘cause somebody can say, actually yes, we are going after the video editing market. That’s one of our initiatives. ‘cause we all know that most organizations have 52 initiatives every quarter. But they can say, that’s not our most valuable initiative. This one over here actually is our most valuable and that’s what we should work on.

So when you are talking, you’re bringing a whole lot of patents to the table to support the metric trees. So you are talking about event storming. You’ve talked about jobs to be done. You’ve talked about domain driven design, you’ve talked about journey mapping.

There’s a whole lot of really valuable product and data and agile patents you’ve just talked about.

Timo: You can see. where I’m coming from. So it’s there’s a big product influence in this whole thing. Yeah.

Shane: They’re all valuable patterns that support what seems to be a simple pattern of metric trees, which is define a metric and say how they relate,

Who does the work then? Because if I talk about a data team, often I don’t see data teams applying product thinking or product patterns, they’re data teams. And so you get this idea of a data product manager now who kind of bridges it, but who is the type of person that would do this work given the level of patent understanding and experience that you actually need to make metric trees work to the best ability, not just defining a metric on a dashboard who do you see doing it?

Timo: That’s an excellent question. I don’t really see data product managers. Obviously because I don’t really see them happening so much. So from my experience when I talk to some people who use metric trees, I would say most of the times these are head of datas leading the data team.

Why is it good fit? Because they have a strategic role and some of these people struggle a little bit to , have a strategic role because they love data stuff. So it’s a tricky process to go from something which, was very operational to something which becomes more strategic. But let’s say the one or two people that I had long talks about were all heading data teams and they were. Completely concerned about, it’s okay, how can we do a better job to connect what actually the business is doing? They also let had a high frustration level, to be fair. So they define Northstar metrics without us, and so they, they do all these crazy things where we actually, as a data team should be involved, but there’s a reason why they don’t talk to us, so maybe we are not really well prepared for that.

And so this is where they invest their time. And I think for someone who’s leading a data team, I think all these things are good things to learn because they have a very strategic aspect to it. And You have the possibility to understand the business, which helps a lot.

Let’s say it’s the head of data, you are the. Chief sales officer for all the data work. So therefore, to have a good idea where your work makes most of the impact and gets you, let’s say, gets your team a lot of fame is definitely not bad to learn. So I would place it most likely there to not also create a different new role for that.

Shane: And it comes back to that value of the data team, value of the things they’re doing. And in my head, so within the information product canvas, we have this area called business questions, and the questions that we want to answer with that product. And from there I can infer metrics relatively easy.

And so therefore, if I think about it as a map, it goes, business question is supported by a metric. And an action is supported by a business question. If I answer that business question, what action are you gonna take? Now, when I think about metrics in a metric tree as more a business model, I’ve gotta think about where it fits in my, in the model in my head, because now I’m going, oh, actually, hold on. They’re not really a subset of a business question. They’re sitting in this patent map in a different place, and . I’m gonna need to think about that one. ‘cause that’s changed my thinking around metric trees, because

I just started off this conversation around it’s just a metric and yeah, I need a metrics layer rather than it is actually a simulation or a visualization of a business model.

Timo: Yeah. Yeah. Business process. So like both, It’s more in this direction.

Shane: yeah. That’s perfect. All right. So if people want to find you and find your writing and you run a course that teaches some or all of this stuff,

Timo: yeah, not yet. Okay, so a guest and I, so if, let’s say, if you look for Metric tree content, so you will come across, a small group of people who wrote about it. And so a guest is one, I’m another one. There are some others, Arby has written a lot of stuff about it. Ollie as well.

So we are working on a course but we want to do it right, so therefore this takes a bit of time. So we did already some iterations. We have some ideas now in general, if people want to read what I’m writing. So you can go to de odeo.com where I write my newsletter. We sometimes write about metric trees. Metrics, product analytics event data models is usually like what I write about. And sometimes I do this as well on LinkedIn. You can also just follow me there if you have a direct question. Or you can also just write me directly on LinkedIn. I usually try to answer them.

Shane: Excellent. I read your content. It’s great. You can tell when somebody knows your craft and then they spend the time to figure out how to write something that teaches it. ‘cause I dunno about you. I find I can write content quickly if I don’t worry about simplicity. But when I try and write something to explain a complex thing with simplicity, that takes me a long time to get it where I’m like, yep I think I’m happy.

And then testing it, give it to somebody and they can tell me what it actually means. That is a lot harder. And the way you write, I can read it and go, ah, actually I think I get it.

Timo: That’s good. No, like I just wrote a, I think this is the longest that I ever wrote, like an 8,000 word piece about the history and future of digital analytics. And I think this thing was brewing in my head for four months back and forth with different kind of variations. So like some sketches on a paper and then, ah, I’m not really happy.

No, this is the wrong direction. And so yeah these things take long until

Shane: so on that one , what we’re seeing now is we’re seeing generation of AI apps are incredibly fast and, 30, 40 years ago, we used to struggle with one system sitting on, mainframe or a, or whatever and getting the data. And then we moved to enterprise resource planning and CRMs.

So we always ended up with five to 10, and then we moved to software as a service. So we ended up with 15 to a hundred to a thousand systems. Now, with the ability to spawn up, an app in a heartbeat, we are gonna end up with a problem of, a thousand, 10,000 systems that capture data for an organization that allows that to happen, that’s gonna change the way we do product analytics, isn’t it?

Or is it not? Do you think that the techniques and patterns and technologies we have today are gonna be able to handle this idea that there are 20,001 shot apps in an organization that are being built for a very small use case for a very small set of personas.

Timo: That’s a very interesting question. I would say in general, not so for example, what really took me, some time to get to this point, others came to this point as well. Unfortunately it’s not so much teach and product analytics that in the end, the tricky thing in product analytics, where do you find an high abstraction layer that still explains the product enough, but it’s simple enough that you can, basically do some calculation and good stuff around it.

So , what really works well is the user state model, which is in the end a gross model. So you say, okay, we have a new user. You new user can become activated. It become active, it can become at risk. When it stop being active, then it can be dormant when basically nothing happens anymore, and then it can be resurrect and whatever.

So you can basically create a loop where people can move through different kind of states. If you take this kind of model, then you can have 400 things under it, as long as you can map the things that are happening in this. To one account. This I is always the tricky thing in product analytics. So as long as you can do this, you can still then abstract it on this high level. You can still say, okay, people do different things. Or , let’s say we capture different things, what the people are doing and let’s say these 100 internal tools that we are using, as long as we can bubble them up into one place where we can say, okay, look, when people show these kind of activities, we flag them as an active user. I would say the system still would work. I would say this is the only escape. If you go the, , classic product analytics approach where you try to track everything and then try to figure something out, obviously this is something you cannot win. You really have to go on a very high abstraction layer. I think the tricky thing is really how you do identity stitching. So let’s say the basic function of making marketing and product analytics work is you have to combine all these different kind of things. So if you cannot combine all the different kind of signals happening somewhere then obviously you cannot analyze them for this one account.

So then you basically have 20 phantom accounts that are actually one, but you have no idea that they are. And this I think, can be really an issue, especially when these tools can pop up everywhere and the company, you don’t have a concept to make sure, oh at least we should identify the people in similar ways everywhere.

Shane: Actually, you’re just giving me a. Visual map in my head of how you put all this together potentially. So effectively what I think you are saying is, when we have 10,000 apps that are one shot apps, the first thing we have to do is make sure we’ve defined the core concepts of our organization.

So concept of a user. Yeah. And that’s important. We know that anything to do with a user is important to our business. So that concept needs to be defined and held. And then if your application is touching a user, touching that concept, then you need to be aware of that. So you are touching the user concept, and that’s an important concept for us.

And then we have the metrics that are a form of statement around active user but also state of user, so there’s a state flow. So your application actually has to be able to do whatever it needs to do to tell us when there’s a state change for that. So we can define active or inactive.

Timo: the interesting thing is usually I like to model the state changes, so I don’t want to have the applications to refine it because we will play around with this. So we will have different definitions over time. We might even break it down. We might have six definition of an active user. This is a great stuff that you can do in a data model, in a data warehouse.

I usually tend not to have it already on the application layer. I just need to track all the things that are happening. This is why events are nice. ‘cause events I can use to derive a state change.

Shane: But to do that from a government point of view when you’ve got 10,000 apps, is you’re gonna have to say to them, in your app, when a user changes to this state, you need to push it out as an event that I can see.

That has to be a governed thing, is you have to actually do these events.

‘cause these are the core events, if you don’t do this, your app is not valuable. And then how do we know it’s not valuable? What we can do is show them the metric tree and say, if you’re not telling us that you are changing the state of a user, then this metric won’t move. And if this metric doesn’t move, then these other metrics don’t move.

And they’re our core North star metrics. Internal metrics, sorry, our estar metrics see I got it wrong then is like our internal metrics are North Star. No they’re not. North star metrics are about customer. So they’re our internal metrics, but they’re the important ones. So if you can’t tell us you’re improving that metric, then why the hell are you building that app?

And that it’s A combination of all those patterns and defining them early so that all those AI slop applications actually fit into this governance

Timo: I definitely have to make a case for AI slop applications. So I privately love to build them. Because now, for the first time, I’m a former product person. I still am a product person at heart, and my biggest problem was always. I need to see things that I can make decisions. So you can create a wire frame, sure. But it’s not really to have an app in front of you. So now you really have the possibility you can go down three ways. I would say, okay, I would do it like this. You can compare it and you can say, yeah, no, this makes totally more sense. So I think it’s really great what’s happening, but you’re completely right.

So it will create an interesting complexity for us.

Shane: Oh, we’ve seen this before and democratization is brilliant. The ability to put the tools in the hand of people that aren’t professionals in their art is great. , That is massively variable. We’ve seen it time and time again, but we’ve also seen the impact when we don’t.

Bring in the principles, policies and patterns that are useful. DBT allowed a lot of people to write transformation code, which is great. It removed the bottleneck of centralized data engineers who were never allowed to do it fast enough, but what we ended up with is 5,000 DBT, and I’m using air quotes here, models, and we lost the definition of a data model, of a conceptual model of those kind of things because we didn’t apply the policies, patterns and principles that were valuable. So I can just see with the ability for democratization of building apps, which is great, we are gonna have the same problem and for me, this idea of events defining them conceptual model and a metrics tree could be the things that we use to create those principles, policies, and patterns

Timo: that’s true. I also never thought about this in this way. I think it totally makes sense. No, you’re completely right. I’m a big believer in Democrats on one side, but on the other side, have a really good foundational concept that will tell you, where you define specific metrics that will tell you if the thing still work. So for DBT, my classic metric is how long does it take a new member of your team to understand the model and make a production commit. If you have 5,000 models, that’s will take eight.

Shane: Where, whereas my metric is what was the original time from a stakeholder saying they had a problem and being served with something that solved it with data and you’ve moved to a team of, 10 new analytics engineers and DBT has that time come down. cause if it hasn’t that’s great.

Busy work. Thank you for hiring more people and being really busy. And the other one that I often talk about when I run my course is the clock starts when the stakeholder says they have a problem. Not when that problem hits the Jira queue. Prioritize for the data team because if that takes three months, actually the stakeholders already said it’s three months late.

And it’s not the data team’s fault because they’re not allowed to work into the hits the team. But if we think about nodes and links and metric tum system maps now we just wanna focus on the prioritization process because that’s where it’s broken. That’s three months. And if the team take a day, it’s still three months in a day as far as the stakeholders.

So same kind of patents as you, eventually some form of storming, some form of jobs to be done, some form of. Domain driven, bucketing, some form of journey mapping. We can apply that to the way teams work as well. It’s the same set of patterns and they have value. Yeah, end of rent. But , I can see that metrics tree and that event definitions of those core events has been really valuable in the, in, in the democratization we’re moving to.

Excellent. All right, so at the beginning , you said, oh, I’m not sure we can talk about metric trees for an hour. And I said we’ll talk about lots of stuff as we did. . Hey, look, thank you for coming on the show. It’s been awesome. I’ve learned lots and I hope everybody has a simply magical day.

«oo»

Stakeholder - “Thats not what I wanted!”
Data Team - “But thats what you asked for!”

Struggling to gather data requirements and constantly hearing the conversation above?

Want to learn how to capture data and information requirements in a repeatable way so stakeholders love them and data teams can build from them, by using the Information Product Canvas.

Have I got the book for you!

Start your journey to a new Agile Data Way of Working.

https://podcast.agiledata.io/e/diagramming-to-understanding-your-data-estate-with-rob-long-episode-76/

«oo»

Diagramming to Understanding Your Data Estate with Rob Long

Shagility — Mon, 13 Oct 2025 20:09:29 GMT

Join Shane Gibson as he chats with Rob Long about diagramming patterns that you can use to quickly document and share your data estate.

Listen
View MindMap
Read AI Summary
Read Transcript

Listen

Listen on all good podcast hosts or over at:

Listen to the Agile Data Podcast Episode

Subscribe: Apple Podcast | Spotify | Google Podcast | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser | Deezer | Podcast Addict |

You can get in touch with Rob via LinkedIn or over at:

The article we talk about in the podcast episode:

AtLongLastAnalytics

Data Producer-Consumer Diagrams: Understanding Your Data Estate

Read time: 8 minutes…

a year ago · 1 like · AtLongLast Analytics

Tired of vague data requests and endless requirement meetings? The Information Product Canvas helps you get clarity in 30 minutes or less?

Google NotebookLM Mindmap

Google NoteBookLM Briefing

Executive Summary

This document synthesizes key insights from a discussion with data strategy consultant Rob Long regarding the use of Data Producer-Consumer Diagrams as a powerful framework for understanding, managing, and communicating the complexities of a data estate. The core concept involves mapping the flow of data through a series of Producers (entities that generate/capture data), Consumers (entities that use data for a purpose), and Handovers (the interfaces between them).

The primary value of this approach lies in its ability to create a unified, workflow-driven view that integrates disparate organizational artifacts like organizational charts, technology architecture diagrams, and data lineage maps. These diagrams are not meant to replace existing technical documentation but to complement it by providing a more accessible, story-driven perspective. They can be tailored in granularity—from high-level conceptual models for executives to detailed technical maps for engineers—allowing for targeted communication.

Key applications include clarifying team roles and responsibilities, performing impact analysis for technology changes, defining the scope of data contracts beyond mere technical schemas, and identifying process inefficiencies. By visually representing the interplay of people, processes, technology, and data, these diagrams provide the necessary scaffolding for informed, strategic conversations and decision-making, ultimately reducing organizational friction and aligning data initiatives with business objectives.

--------------------------------------------------------------------------------

1. The Producer-Consumer Framework

The foundation of this methodology rests on a simple yet powerful set of definitions derived from systems thinking, providing a common language to describe any data workflow.

Core Concepts

Data Producer: An individual, team, or system responsible for generating or capturing data. Examples range from a physical sensor reading temperature to a CRM system capturing customer interactions.
Data Consumer: An entity that receives and utilizes data for a specific purpose, such as answering business questions, training a model, or generating a report.
Handover (or Link): The critical interface where a Producer transfers data to a Consumer. This encompasses the mechanisms and agreements governing the exchange, including data contracts, quality checks, compliance rules, and delivery formats.

This terminology is analogous to the systems thinking concepts of “Nodes” (where a job is done) and “Links” (the handover between nodes).

Fundamental Value Proposition

The central benefit of this framework is its ability to unify disparate views of a data estate into a single, coherent narrative. As described by Rob Long, “it helps give you a unified view of your data estate... together they give you that workflow driven kind of diagrams which help unify everyone and reduce organizational friction.”

Complements Existing Artifacts: It enriches traditional documents like vertical organizational charts by providing a horizontal, workflow-oriented perspective. It integrates views of people (org charts), technology (architecture diagrams), and data movement (lineage diagrams).
Workflow-Driven Perspective: The diagrams focus on the end-to-end flow of work and data, clarifying how value is created and transferred across teams and systems.
Establishes a Common Ground: By using a simple, intuitive model, it allows technical and non-technical stakeholders to engage in meaningful conversations about complex data processes.

2. Key Applications and Use Cases

The producer-consumer diagramming approach is a versatile tool with a wide range of practical applications for data strategy, architecture, and team management.

Mapping a Multi-Layered View

The framework is capable of mapping multiple organizational dimensions onto a single diagram, providing a rich, contextualized picture. This includes mapping:

Data systems and their interactions.
The flow of data through different architectural layers (e.g., Bronze, Silver, Gold in a Medallion Architecture).
Team design and the boundaries of responsibility.
The specific technologies and tools used at each stage.
The personas (e.g., Data Engineer, Business Analyst) involved in the workflow.

Variable Granularity for Diverse Audiences

A key strength is the ability to adjust the level of detail to suit the audience and the story being told.

High-Level (Executive View): A simple map with a few nodes (e.g., “Source System,” “Data Lake,” “Data Warehouse,” “Reporting”) can tell a clear, conceptual story without overwhelming detail.
Fine-Grained (Technical View): The same map can be expanded to show intricate details within each node, such as specific data quality rules, data mastering processes, metric definitions, and the technologies involved.

This flexibility allows for the creation of “variants very simply which tell different stories for the audience.”

Enhancing Handovers and Data Contracts

The framework places significant emphasis on the “handover,” treating it as a critical point for negotiation and clarity.

Identifying Waste: It helps uncover process inefficiencies, such as when a producer generates data that is never used or when a consumer needs information that is not provided, forcing them to perform redundant work.
Informing Data Contracts: The model is crucial for defining what needs to go into a data contract. It pushes the concept beyond technical specifications (schema, field types) to what it truly should be: “an agreement between two parties... a negotiation between the producer and the consumer about what’s needed for both sides to do their job well.” This includes non-technical aspects like documentation levels and support expectations.

Facilitating Feedback and Agile Processes

The diagrams inherently support modern development practices by visualizing necessary communication channels.

Feedback Loops: In reality, data flows are not purely unidirectional. The model highlights the importance of feedback loops from consumers back to producers to report errors, document issues, or request changes. This moves teams away from “throwing things over the fence” and towards collaborative problem-solving.
Agile Gates: It provides a structure for implementing agile patterns like “Definition of Ready” (what a consumer needs before starting work) and “Definition of Done” (what a producer must complete before handing off work).

3. A Practical Example: Mapping a Data Workflow

A concrete example illustrates how these concepts are applied to map an entire data pipeline, layering multiple dimensions of information.

Scenario Overview

A common data workflow can be visualized with five primary nodes:

Source System
Data Lake
Data Warehouse
Analysis/BI Layer
Report

Layering Information

This basic flow can be enriched with additional layers of context to tell a more complete story.

Derived Insights and Strategic Questions

This single, relatively simple diagram immediately provides significant value:

Quick Comprehension: It tells an instant story. An observer can see, “you’re a Microsoft stack... I’m not seeing Snowflake, I’m not seeing Databricks.”
Generates Insightful Questions: The map acts as a catalyst for deeper inquiry. For instance:
- Are the Data Analysts read-only in the Data Warehouse, or can they write transformations?
- Where are business metrics defined? Are they only in the Power BI semantic layer?
- Is there a mismatch between the GUI-based tools (Power BI, Excel) and the skills of a potential hire who is a “hardcore Python coder”?
Identifies Boundaries: The diagram visually delineates responsibilities. It becomes clear where ownership lies within a single team (e.g., Analyst BI to Report) and where cross-team handovers occur (e.g., Data Engineering to Data Analytics), highlighting areas that require formal processes for communication and problem-tracking.

4. Strategic Benefits and Implementation

Beyond tactical mapping, the producer-consumer framework is a strategic tool for driving change, fostering alignment, and building robust data capabilities.

Driving Informed Decisions

The diagrams provide “the scaffolding for meaningful conversation and decision-making.”

Impact Analysis: They make the “blast impact” of a proposed change instantly visible. For example, replacing a BI tool is not just a single change; the map shows it could necessitate replacing “one-third of our data layered architecture.”
Challenging Assumptions: By providing a concrete reference point, the diagrams allow stakeholders to challenge decisions with an informed opinion, moving conversations away from pure conjecture.

Collaborative Creation and Alignment

The process of creating these diagrams is as valuable as the final artifact.

Workshop Approach: A highly effective method involves collaborative workshops where a team jointly maps its processes. This is a quick way to document workflows and often reveals stark differences between how leaders believe work is done and how it actually is.
Synthesizing Perspectives: Another successful technique involves creating separate diagrams with different groups (e.g., technical contributors, managers, executives) and then bringing them together. This process uncovers misalignments in definitions (e.g., what constitutes a “data set”) and processes, leading to the creation of a new, shared “truth” that becomes the adopted standard.

A Tool for Strategy, Not Just Technology

The framework encourages a holistic view of data strategy, aligning with the “four pillars” of People, Process, Technology, and Data. It forces a focus on foundational questions before technology selection:

What does success look like and how will it be measured?
What team design and ways of working are needed to achieve the goal?

The architecture then becomes “a means to the end... how you achieve the goal, it’s not the goal.” This helps in designing a “minimal system to achieve the goal and to hit success” rather than an over-engineered solution.

5. The Underutilization of Systems Thinking in Data

Despite the long history of systems thinking in fields like lean manufacturing, its application in the data domain remains rare.

The Communication Gap

The reluctance to adopt these end-to-end mapping techniques may stem from several factors:

Hyper-Specialization: Data roles are often highly specialized (e.g., Scala developer, dbt modeler), with practitioners not always being taught to consider the entire system.
Lack of End-to-End Onboarding: Unlike factory workers who “walk the line” to understand the full process, data professionals are often not onboarded with a holistic view of the data flow.
Communication Skills: There is a recognized gap in “soft skills,” particularly the ability to communicate complex technical ideas to diverse, multi-disciplinary audiences.

Bridging the Technical and Business Worlds

Producer-consumer diagrams are a vital tool for bridging this gap.

Making Complexity Accessible: They distill complex ideas into simple, understandable stories. As Rob Long notes, a key skill is to “make information accessible, whether that’s just choosing the right type of diagram, using the right vocabulary.”
Educating Stakeholders: The diagrams can be used to explain technical concepts like data lineage at a conceptual level to executives. This isn’t to make them experts, but to build awareness so they can better understand the value and necessity of investments in areas like data contracts and observability.
Fostering Business Acumen: The framework encourages data professionals to become more business-driven by understanding how their technical work fits into the broader value stream, aligning with the principle that “as companies want to become more data driven, data engineers should want to become more business driven.”

Tired of vague data requests and endless requirement meetings? The Information Product Canvas helps you get clarity in 30 minutes or less?

Transcript

Shane: Welcome to the Agile Data Podcast. I’m Shane Gibson.

Rob: I’m Robert Long.

Shane: Hey, Rob. Thanks for coming on the podcast. Today I wanted to have a chat to you about this article you wrote that I really loved. It was called data Producers Consumer Diagrams, and Understanding Your Data Estate. But before we rip into that, why don’t you give the audience a bit of background about yourself.

Rob: Yeah, I’d love to. So I got started with data in academia. So I actually have a PhD in applied mathematics and geophysics. So I was doing mathematical modeling and numerical simulation of planets and stars. Things like sun spots and storms on Jupiter. And without realizing at the time what I was doing was bringing machine learning and data engineering to that field. ‘ cause these simulations generate big data and we were implementing. Supervised machine learning, but it was still a step up for that discipline group of people. Since then, I’ve dabbled all over the place, so I had a couple of years where I run a software as a service, helping schools in England improve their national ranking. So again, very data centric. And then throughout my career I’ve gone through and I spent a couple of years as a senior data engineer in defense in the uk, so really helping ‘em upskill and using cloud platforms, how they do data, and was using GN AI to help with content creation, so scripts, audio imagery, things like that. And now I’ve moved from the UK to the US at the start of this year, and I run a consultancy where I focus mainly on enabling analytics. So lots of data engineering work and data strategy. And that later part is really where this kind of producer, consumer framework came from as I’ve been helping. Companies new to data Build the foundation. So really getting them set up on the right path. And that’s how this kind of came to be.

Shane: Let’s do a little bit of anchoring around terms, ‘cause you use slightly different terms than I do, so we’ll anchor those. So just talk me through this idea of how you define a data producer, a data consumer, and the concept of a handover.

Rob: So I think a producer, a very broad strokes, is an individual, a team or system that produces, which means to generate or capture data. So that could be thermometer, reading temperature, it could be a CRM system, whatever it is. A data consumer is someone who takes in that data for a purpose. And so there you’re thinking they need data to look a certain way so they can use it to answer certain questions. And then a handover is really this interface with the two meet. So that’s where a producer is literally handing over, in this case, data to a consumer. And so there you might, think about terms like data contracts you might have quality checks, compliance, whatever it is. And so I build these networks on those three ideas really.

Shane: So the language I tend to use is I was a great fan of a TEDx video called How to Make Toast. And it’s around system thinking, so very much the same. And he talks about nodes and links. So for me, I talk about nodes and links. Node is where a job is done, something happens, and then the link is the handover to another node.

Data producer in terms of data’s created in the CRM or a sensor is creating an event that is a node and when it moves to somewhere else to get worked on or consumed, use a link to the next node. I’ll jump between your terminology and my terminology all the way through as I always do. So we’ve got this idea of producers, consumers, and handovers or nodes and links . How do we use them? ? Why do we care? ? Where’s the value?

Rob: I think the value is really in unifying other things that already exist. So lots of companies, for instance, might have an organization chart, which gives you an almost vertical view of the roles or teams and where they sit. But as we know, lots of data work happens horizontally. But there again, it’s not to replace org charts, but it helps enrich them because these producer consumer diagrams are very workflow driven. We’re trying to capture a flow from start to end. so that’s one use case where it lets you overlay these different artifacts you already have. As I said, an org chart is typically representing people. You might have your architecture diagrams, which represent technology. You might have data lineage or flow diagrams, which capture how data’s moving through these systems and. All of them, to some degree, you can represent as producers and consumers ‘ cause they’re either generating or using data. And so I think this is really powerful ‘cause it helps give you a unified view of your data estate. All of these artifacts are really good, but together they give you that workflow driven kind of diagrams which help unify everyone and reduce organizational friction.

Shane: I agree. I think the idea of using a map or a diagram to visualize a concept is really valuable as long as we don’t overload it . We don’t make it too complex in itself. So those maps should be as simple as possible. And so we can use it to map systems. We can, like you said, we can use it to map all charts.

We can use it to map flow of data. We can use it to map data layers and the rules on each of those layers. We can use it to map team design. We can use it to map a flow of work as you say, somebody works on something and then hands it off to somebody else who uses that. I think, if we go back to some of the original data mesh thinking there was this idea of input and output ports.

So we’re talking about that, somebody does a bit of work and it gets handed over to somebody else who does another bit of work, and we just visualizing that workflow or that flow. Is that how you see it?

Rob: That’s exactly it.

Shane: Let’s give a an example, concrete example. Talk me through, an example of one of these maps.

Rob: Going through an example is really useful. ‘cause there’s also this concept of you can look at different granularities and I showed diagrams of that in the article where at a very high level you might just have your external system, which is producing data, and then you bring it into internally into your architecture where you have your own processes for data.

So there we’ve got the external being the producer, the internal being the consumer. But if we look more deeply internally, we might have a data preparation layer where we ingest, do quality, create our data models, and then the reporting function, which takes that middle block, which is now consuming data and producing data. So you get this cool thing of, as you add nodes, things can turn from just being a producer to a producer, consumer. And so I think there, that’s an example where these networks are dynamic, but they also can have as much information as you want ‘em to have. So to your point, at a very high level, you might want a very few number of nodes and keep it almost like a conceptual model. And then as you go down, you might want to get into the real nitty gritty, detailed view to understand exactly for instance, which technologies are doing what, which teams are responsible where, and things like that.

Shane: I think the level of detail can vary. So for example, if we think about layered data architectures, which have been around for donkeys, but are now hotten and popular again because of the medallion architecture, we can have a very high level map of nodes and links where there’s five, source system, whether it’s produced, data’s generated, bronze , silver, gold and then the consumer.

And that’s only five nodes, and it tells a very simple story. Now, what we know as data people is there’s a whole lot of extra complexity. Where’s the data quality rules been applied as the data contracts across each of those nodes. There’s nodes within that. Whenever we get into silver, how are we conforming data?

How are we mastering data, how we bring in and reference data a whole lot of things, where are we defining those metrics? So we can take that very simple diagram and then with only the five nodes, and then we can break it down to even more detail when it makes sense, so we can present the same system with two different stories depending on who our audience is and the story that we want to tell.

Rob: I think that’s really powerful. That’s exactly right, is as you said, you might want the data engineering team to understand what goes on within the silver layer, but your business analysts who are consuming at the end, they. Don’t necessarily need to know care or even have access to the data that’s there. And so that’s exactly right. It’s powerful because you can create variance very simply, which tell different stories for the audience and it lets you again, unify all those other bits as needed.

Shane: One of the other ways we can use it is this idea of when there’s a handover or when there’s a link, what’s actually involved. So what is handed over? Is it just a blob of data? Is that a blob of data with a schema? Is it a document? What does the person hand over to the next person?

What does the producer hand over? And what does the consumer expect? What do they need to be handed over to them to do their step in that process? And a lot of people don’t think about that. We don’t bring in that lean system thinking to say, actually, where’s waste in terms of we, the producer generates some stuff that’s never used, but also waste because the consumer needs some information that’s never provided and now they’ve gotta go and do all that work again and introduce that wasted effort that’s already been done because nobody told them that it has been done or what it was.

Rob: It plays into a bigger story of kind of, in a very ideal case, if everything’s perfect, these are very unidirectional networks. You take data from source process it, it’s perfect. So it’s consumed. In reality, it’s more complex because you want to empower consumers at each handover to be able to report errors and feedback and say, you’ve given me this with a bunch of columns, that I dunno what they are ‘ cause they’re undocumented or they’re new in the API or for whatever reason. And so it’s really important to have both the producer driven flow, which is in my mind, left to right from source to target, but to have the feedback loops that go in the other direction. And so then you can have both the functional benefits of fixing errors quickly, knowing that they exist, what have you, but also from a very almost people level, remove some of the idea of playing the blame game because you’re no longer throwing things over the fence. It’s about having well-defined processes so that everyone can work together to fix these problems, and I think that’s really important.

Shane: Yeah, and if we look at it again from that system thinking, we can have different lenses on what those processes are. So for example, if we look at the flow of data, we can look at data contracts. And we can look at data contracts more than just system of capture to the people doing the data wrangling.

When we get it into that rule layer, we can actually have data contracts between every node and link. So if I’m transforming some data and creating a metric, what’s the contract for that metric? Is there a standard YAML format? Is there a certain descriptive metadata I’ve gotta store?

We can actually put data contracts in between each of those steps.

If I look at way of working, we can bring out some of the patterns from Agile and Scrum where we can do definition of ready, end definition of done. So the consumer can say, Hey, if I don’t get given these things, then you don’t meet my definition of ready.

Therefore, I’m not gonna start work because I don’t have the tools I need to do the job. I need to do. As a producer, I can have a definition of done. Here’s all the things I would expect. Myself or anybody else in my team to have done before I’m saying the work’s done, it’s ready to be handed off to the next node or the next consumer.

So we can bring that idea of gates and patterns and patent templates into both the way the data works, the way our data platforms work, and the way our teams work.

Rob: Exactly that, and that’s why I really like this idea and I think that. , It’s always gonna come up by the idea of data contracts, which is, that acceptance criteria between producer consumers for data sets effectively. But, producer consumers are really important for telling you what needs to go into a data contract. ‘ cause it’s far more than just technical as you said, because if the analysts are receiving data from an engineering team, they need and expect a certain level of documentation, formats, whatever it is. And so it is really about, yeah, removing friction and unifying those requirements and expectations from both sides. So that’s exactly why I really like these. The framework is the Pattern to use.

Shane: And I think one of the things about data contracts is everybody looks at it from a technical point of view. They look at it as what is the schema, what is the field type? What is the grain? But they forget the actual word contract, which basically means an agreement between two parties.

So it is a negotiation between the producer and the consumer about what’s needed for both sides to do their job well. that’s how we should treat it, we should actually negotiate the contract, not just treat it as a technical task.

Rob: I a hundred percent agree, and I think that’s probably improving, but at least in lots of cases, I’ve seen , data contracts are taken to be technical andnot have the surrounding pieces, which I think I agree with you. They’re missing

Shane: I’ve never seen a data contract that has the level of documentation involved in the handover unless it was an API contract. . Which is in theory self-documenting. But we don’t put, APIs between our bronze, silver, and gold layers. . We should probably we should definitely put contracts in between each of those moving parts.

We don’t put data contracts in between the way we capture requirements as a business analyst and hand them off to a data team. If we are running separate teams that do requirements gathering in a separate team, that doing bill, which again, I don’t actually suggest you do. Put them in the same team, solves a whole lot of problems.

So one of the other things you have done is you’ve used that idea of consumers and producers to actually map it to tools and technologies that are being used. So just wanna talk us through how you do that and where the value from , that Pattern is.

Rob: As I said, I view these as a way to unify these different lenses, technology people, process data. And so when we talk about, for instance, ingesting data from an external system, we’re gonna use a technology to do that. So I’m gonna use the example of within the Azure ecosystem, because that’s what I mainly work with. But any ETL tool, for instance, Azure Data Factory, you can then, say this edge of this diagram is handover, is going to be performed by Azure Data Factory. And then internally, once you’ve ingested it and gone through your medallion architecture, that’s gonna be some data warehouse, Postgres, synap, synapse analytics, whatever data platform you are using through to, once you get to the reporting stage, you’re gonna have a BI tool, typically, whether that’s Python, power, bi, Tableau. And so really the technologies in general map to the handovers. There are cases where they can be the nodes as well, the producers or consumers. And then I think what also that lets you do, again, the benefit of that is. It lets you have an informed decision about do you really need this technology? So when you’re thinking about cost, performance capability, when you’re talking about does your team have the skills to use this technology or do you have to learn something new again, by bringing it all together, you can make an informed decision about that. So it’s much less the preference of the architect or listen to a vendor pitch and it lets you get that view of what you need to use.

Shane: I think the other thing I liked in the diagram you used as an example is you also bring in the personas. So you are mapping out the flow of the data effectively. Then the technologies, the tools that are being used in each step of that flow, and then the personas, you’re expecting to use those tools.

So you are differentiating between the tools the data engineers will use and the tools are a persona of data analysts.

Rob: Yeah, and as I said, a lot of the people I work with and my clients are. Less mature with their data capability. And so for them, a lot of them start with the Google definition of data engineer or data analyst. So it’s much more useful then to use these types of diagrams, map the technology, and really distinguish the responsibilities of the personas so then they can hire the right people, get the right skills. But it also again reduces friction because if you are brought in as a data engineer and you’ve got clear roles and responsibilities, you’re not gonna kick up a FU because you’ve been given something that you deem as pure analytics or whatever the case is. And so I think again, that’s really useful to have because it gives you that unified view of your data estate and lets you make informed decisions about technology, team structure how data flows from source to target.

Shane: The thing I like about it’s, it tells me a story that, it’s a map at a high level and I can understand it, I can embarrass some stuff and I can get a bunch of questions. So the one I’m looking at, you start off by saying there’s a source system that goes across to a data lake that goes across to a data warehouse, that goes across to a analysis BI and then across to a report.

So I can look at that and go. I think you’re running a three tier architecture, data lake, data warehouse, and an analysis bi layer. And then you’ve got source systems of data coming, being produced and then reporting content on the right hand side where people go and use it. And then you are breaking it down to tools.

So you’re saying that effectively as Data Factory has been used to extract and load the data into the Lake, synapse analytics has been used to Mung and wrangle the data. Power BI has been used to create the analysis PI layer and then PowerPoint Excel has been used for the primary reporting.

And then you’ve got the idea that data engineers sit on the left and do half the work and data analysts. And so I can look at that and I can go, good. There’s no omni there’s no thought spot. . I can get an understanding quickly of the technology and I go, yeah, you’re a Microsoft stack, i’m not seeing Snowflake. I’m not seeing Databricks. , I know that effectively you’re gonna be on Azure, not GCP and not aws. aws. I can see you’ve got two sets of personas in there. And that raises a question because in the way you’ve laid it out, it looks like the data analysts are reaching back into the data warehouse.

So I can have a question of are they read only or are they actually able to write transformation code in the warehouse? Your analysis bi layer is Power bi, so I’m just gonna assume that it’s dxi. Semantic key thing, and I’m gonna ask you questions around where metrics defined. Are they only in that layer or are they somewhere else?

Your data lake, I’m gonna assume it’s Azure Data Lake Gen two or something like that even though you haven’t specified it. So I’m gonna ask you those questions so again, I can get a really quick map of that environment, of that estate, and I can ask you a whole bunch of questions.

For things that I just want to know because I need to know, or I’m just nosy and I wanna know. And that’s the value of these maps.

Rob: Yeah, and I think what you’ve just touched on is really useful in that you can create different views like we discussed about earlier. So you might have this, which is your more executive focused story, and then for your technical users, your power users, whomever. You might have that more finer grain that you’re talking, where you go into more specifics about technology, about the rules. And again, you could also go more in detail. For instance, in my experience, a lot of frustration and friction comes from the handovers is that mismatch of expectation, definition of ready, definition of done. And so this lets you account for some of those. It lets you see quite clearly who owns which parts.

Is it within one team? So the analyst bi to the report layers are both owned by the data analysts. So the part of the story there is that they should be able to do that internally. Whereas if you look at the handover from data engineering to data analytics, there, we might need some cross team and you have to work out the processes for that. But again , you’re informed of that with this diagram so you can start to flesh out how that’s gonna work. Who’s responsible for what, how communication happens. If it’s a Slack conversation, a Jira ticket, whatever you want to use for that kind of problem tracking. The extra layers of information you can get when you go to finer granularity.

Shane: And I think it’s that idea of a boundary because my eye is naturally creating boundaries and saying these things these nodes, these links are in the boundary of a data analyst, and these other ones are in the boundary of an engineer. And now I’m gonna worry about whether the boundaries look weird to me or whether there’s an the touching of the boundaries or an overlap of the boundary.

One of the things you do though, when you diagram this is you’re effectively bucketing or putting together the consumer and the producer node with no line. And I’m assuming that your visual way of saying that a person consumes something and then produces something, and that in my head is within the node before they hand off to somebody else.

Is that just a visual style of the way you do it?

Rob: It is. So it was trying to capture the logic that, for instance, the data warehouse, it consumes from the previous node, the data lake, and that same data warehouse produces to hand over to the BI tool. So again, you’re right, you can look at this and view this in different ways. For me, it’s really just about having the node label as either producer or a consumer or a producer, consumer pair. And yeah, there’s different ways of doing this. This was one of the simplest ways that I could fit quite a lot of information into a easy to read diagram. And so that’s a complete design choice that I’ve had success with.

Shane: And I think it’s just a choice of language now. I understand that when I look at those, they’re effectively an ensemble. . And I’m looking for ensembles where these consumer producers, which means these in and outs within that node, or I’m looking for one where there’s only a producer or a consumer task effectively.

Again, it helps me tell that story

Rob: And in general, obviously you very rarely want to produce data that’s not being consumed. So in typically they will come in pairs, and obviously there’s always the question of where do you end the diagram. So in the case we’re talking about, the reporting layer says consumer producer, but the producer’s not connected to a consumer. So again, the bit that’s missing from there is that there is a decision maker, an executive who takes that report and makes a decision, takes an action. And so that’s also. Where I say it’s as much art as science when you make these kind of diagrams is there’s no rule for granularity or your starting end points most of the time.

There is a logical start point, but where you decide to kinda end the flow. As I said, it can vary a lot and it really just depends on what you’re trying to convey with that particular story.

Shane: And that’s the important part is you are telling a story. So I could take , that five node one you’ve got of source system data, lake, data warehouse analysis, BI and report. And I could extend it to the right and actually say actions that are taken off, those reports and outcomes are delivered by those actions and value that’s delivered from the outcome.

If that’s the story I wanted to tell,

Rob: yeah, and then you’d obviously sell a third persona, the executive or whatever. It’s Exactly. That’s how easy it is to iterate these and see the difference between where you are. And a proposed change to your process, framework, technology stack, whatever it is, it directly gives you the implications and new connections you need to manage and prepare for.

Shane: And that’s the other part is it helps with the story. So for example, in this scenario, if I said, oh look, I want to introduce a new BI tool, consumption tool, last mile tool I’m even gonna have to find one that is compatible with the power bi semantic BI layer. If not, we know that the blast impact of that decision is we now actually have to have two semantic BI layers or replace the power BI one with one that actually serves multiple last model tools.

So again, I can point to something and say, if we do this, then we are gonna affect that. . And visually point to the map so people understand, oh, holy shit. Actually that’s one third of our data layered architecture that we were replacing. If we’ve only been working on it for six months, maybe.

If we’ve been working on it for 10 years and we can’t programmatically migrate it, we’ve got a lot of work coming. So again, it’s really valuable to be able to point to parts of the map and say, we’re talking about that. We’re talking about the uk, not the us. That’s the value of the map.

Rob: Exactly. That’s it.

Shane: And then the other thing we can do, so we can extend left and right, and we can also change the grain effectively so we can add more rows. So for example, at the moment we’re saying that all source systems are producers and they’re the same. But if I wanted to, I could create different rows,

I could say we’ve got relational versus SQL sources, or we’ve got streaming versus batch sources. So if I wanted to, I could just add more rows more nodes and links as a row to add more complexity or tell a different story to this map. Couldn’t I.

Rob: Exactly. You can add more rows in terms of different sources or even like you said, making it modular and taking out the synapse piece and say, what if I replaced it with Databricks? And then you can obviously also account for new data. So in this case, the data warehouse is just producing datasets to give to the analytics and BI tool. But you might decide that you want to also capture system data from your warehouse. So from Synapse, maybe. How long your spark clusters have been running, how many outages you’ve had, how much it’s cost. And so you can also kind of, not add new rows, but add an extra dimension where it’s not just going left to right, but you have metadata that kind of goes up and down as well. So you can add different directions or paths possible in your network.

Shane: So do you tend to. Design the complex map first and then simplify it to tell different stories, or do you tend to start off simply and then add the complexity as different versions as you go? Which way do you work?

Rob: So I do something very weird, or I start with a super high level simple flow. So I really just want to get the conceptual model right and then I skip the middle bit and jump to the perceived finest grain and do the real in depth these data quality rules, these data contracts, these technologies. Then I backfill the middle bit because I find it really easy to question the requirements at that very high level. And then by jumping to the really low level, find the real complexities. So like I said, if there’s a skills mismatch with the team versus technology, which then I can propagate upwards. I’ve got a few horror stories where I did go from top to bottom, got to the bottom, and then realized something didn’t work.

And you have to backfill the whole thing then. And even though you’re only changing one node, it inherently sometimes changes the nodes that it’s paired with, obviously ‘cause things work differently and connect differently. So that’s how I work. I think it can work both ways, starting at the very high level or very nitty gritty, detailed level. I think it’s a matter of preference and how good your initial requirements are. I find the better the requirements. The easier it is to start at the height level and then go from there?

Shane: And then you are handcrafting these, you are drawing them as if they’re pictures. You’re not using a tool to help you with this.

Rob: A little bit of both. So I do start very conceptual hand drawing in a tool like Draw io, but I do automate some of this with Python in terms of just treating it like a knowledge graph. And you can def create a simple CSV file with your nodes and edges and get it to populate a graph. So I do a hybrid approach to get these ready.

Shane: It’s interesting that there’s a gap in the tooling. I think at the moment, we used to have things like Spark Enterprise Architect, which was a horrific product to try and use. , I used to do enterprise architecture and I’d go into an organization. I was forced to use that tool. I’d if I knew in advance, I wouldn’t take the gig just made you so slow. And such waste. And like you, I just use draw io. But it’s a graph problem, we’re talking about nodes and links effectively ins and outs and relationships. So surprising that actually there’s not a great tool

For defining it, but then also creating the simple stories.

Rob: I say, I think the closest I’ve seen is mermaid js, so it’s diagramming as code. So it works, but you just don’t get the customization to make it more user friendly. So it’s great for spinning up dirty diagrams that a technical team would love, but to sell this to an exec, it falls short exactly as you’re talking about.

Shane: You’re telling a story, so therefore, it helps if the story’s attractive to look at not ugly. And yeah, I’ve done ones where I’ve used some of the lms, so I’ll put it in as a CSV, get it to gimme the mermaid and tax and then put it into a mermaid viewer. But it’s ugly,

It tells a story, but not in the most attractive way. I think the other thing is, again, overcomplicating at your risk. So if you can keep it left or right in an English speaking country, people will understand it as soon as you start branching off to if DL statements where you’re coming along and you’ve gotta go up and or down and again, that just increases the complexity of the diagram and the story you’re telling.

So if that’s the story you need to tell, then do it. I think the other thing is how many lenses or dimensions you bring to it. So for the one that you’ve got here, there’s a flow of the data and the data layers. There’s the technology and then the personas. And I find that three is normally the maximum you get to before, again, you start bringing complexity in.

So if you add another two in there, you’ve gotta do that consciously. You’re consciously saying, I want to make this a more complex technical diagram than a simple map.

Rob: Fully agree. And again, it’s all about knowing your target audience and the story you wanna tell. So again, I think these are, these are not your super technical architecture diagrams, and they’re not meant to replace them. They’re meant to compliment them by giving a view which is consumable by. Your executive team, your managers, whomever else, decision makers who don’t know what the icon for Databricks is, for instance. So when you show them an architecture diagram with networks and tool specific icons, it’s noise and this is about almost filtering that into something useful for that audience.

Shane: one of the ways I use this nodes and link format and this idea of maps is, as a workshop with data and analytics teams when we want to change the way they work. So the way it works is I get the team together. I basically put something on the left and something on the right, either on a wall, if it’s in person or on a virtual whiteboard.

If it’s not. So an example would be data sources on the left information consumers on the right. And I asked ‘em just to brain dump. Brain dump all the data sources on the left and brain dump all the people that use whatever you produce on the right. So I start giving, a bit of a map. And then I say to them just use some stickies and do a stick a node for everything that you do to get the data from that left to that.

Right. It’s always interesting. So some people do very high level stickies. Some people do very detailed. At this stage I don’t care, I don’t give them any boundaries. And then once they’ve done that, I get ‘em to group those stickies together. So effectively, where you’re doing the same task, put it in the same area.

So your idea of, where it’s in, in and out, and it looks like it’s the same. And from there we now have a flow of work. I can get them to use the dot Pattern to say, where do you think it’s broken? Where do you wanna invest and change? And a whole lot of other things. But I find it, a really quick way of getting a team to document their processes without endless interviews and documentation.

And also it’s really interesting when a leader sits in the room and they use the words, but that’s not how we do it, right? We do this and the team just laugh at them and go, no, this is always how we do it. Or when you have two teams that actually have completely different processes, there’s some things that are shared and there’s some things that aren’t kind of looking at from that lens, gimme some examples of how you actually create these diagrams. Is it just you? Have you ever done it in a teaming environment?

Rob: So there’s two ways that I’ve had good success. One is where I’ve been brought in as a almost contractor, and so they’ve given me the requirements and a few hours of someone’s time, so I go through the requirements with them and then create this myself. Then the value is really in the playback and discussion session. But like you, I’ve also had success doing a whiteboard activity where we just go through with maybe one of the data teams, the kind of technical hands-on person to create the flow. Then we go with their manager independently to create the flow. Then we go with the exec level and create the flow. And then you get, like you said, very different viewpoints of how they think things are working. And that’s how you get real change at a process level because it’s not what people thought it was. And so I’ve had great success with that. And then it tends to end with a session where we all come together, put them all up, and go through and discuss differences, similarities, and often we come up with a new truth, which is then the one that’s adopted and implemented. So it’s normally some Frankenstein monster of all of them. But it turns out that’s the one that’s useful because it brings together the different views people have and brings together the best of each way of working. And so that’s really where I think the best lessons learned are and how you actually make change using this as a tool. Yep.

Shane: And I can imagine when you’re doing that again, now you’ve got say, three maps, right? To keep it really simple. So the three expectations of how the system works to help get. Agreement. You can point to parts of the map, you can point to the executive part of the map where it’s got the word AI agent and you can say nowhere in the other two maps.

Does that have any idea of LMS or a, agentic behavior. So we need investment, we need to add that node because it’s just missing. We’re not investing in that right now. So by pointing to things in the maps, you can get agreement to add things and then you can get agreement to take things away, I’m assuming.

Rob: Yep, exactly that. And another similar example is where you might just need to end up updating their business glossary. So it might just be, and this is real case, where the ic, so the technical contributor versus the executive had different definitions of dataset, and so their flows look very different. And it’s just because. Dataset was taken for granted to mean dataset, and there we just, went into the business. Glossary added a new entry, and moving forward communication was easier ‘cause they were talking the same language. So you can both modify things and update how they, work.

Shane: Yeah, actually that’s a really good point is that when you’re doing these maps, you need to be iterating a bunch of definitions of business glossary so that if there is a box, if there’s a node and it’s got a word, that word needs to be defined. . Because otherwise everybody’s gonna look at the map and go, Christchurch, I, that’s Christchurch, New Zealand, no, it’s Christchurch in the UK endorse it. Oh, okay. Actually, it needs to say Christchurch, New Zealand and Christchurch uk. Otherwise we are gonna look at different part of the map thinking it’s the same thing. I think two other areas I’ve seen value during this process, when I get bought in to do data blueprints for organizations, I will create these as a way of articulating a story like you said. . I will help it for me to understand the thing I’m trying to map, the system I’m trying to define as a blueprint. And then use that to test and iterate and get feedback on whether I’m on the right track for what the organization thinks they want.

And the second one is when you go in and do a review or a stock take. I do exactly what you said as well. I will talk to the technical people and get them to help me draw the nodes and links diagram or get them to do it. And then I’ll go read the technical documentation, the solution design any documentation and see how well it conforms to their understanding, because that means one or two things, the documentation out of date, which is typically the case or the people have an impression of how the system’s working, but that’s not actually what’s happening.

So those two use cases , for this Pattern, I found is really valuable as well.

Rob: Yeah, I think we’ve got a very shared experience with those. The last one, which just came to mind was there’s an educational piece, which is really interesting in that when we talk about these nodes and edges, producers, consumers, as I said, I like to use ‘em for different use cases, but for instance, executive level people may not know what a data flow diagram or data lineage is, but if they’re happy with this as a conceptual map, source data warehouse. Analysis, so on. Then you can give the example, for instance of if you may instead make the nodes data sets that then if you think about it, it’s really just a data flow diagram or a data lineage diagram. And so there’s a nice educational piece where something that they’re comfortable with can help them understand something more technical that they don’t run into. And I think that’s actually never the intent, but always a nice side effect of using these.

Shane: I think we need to be careful there though, because all data lineage diagrams look awful. They look incredibly complex. They look like a, Frankenstein version of the London Underground. And therefore we need to be really careful that telling that story has some value.

Because it’s a tool for data professionals to be able to go into the detail to find the bit of nodes and links that they need to solve a problem. I think it’s like data models, and, ghost of data past the enterprise data modeler would print everything out on a or zero, have it up on the wall and be very proud at the size of their data model, the number of nodes and links and how complex it was. Nobody else gave a shit. In fact, it actually did them a disservice because it was like, I don’t understand that. And I think we’ve gotta be careful with data lineage as well. That actually it’s a tool for us for data professionals. It’s not a great map for information consumers.

So I’m with you and understand the concepts and people can look in and go, oh yeah, it looks like a really complex version of what I understand. I probably not gonna go near it ‘cause I don’t need to.

Rob: So yeah, exactly that. When I say to help ‘em understand what it is at a conceptual level. It’s not an excuse to start showing up with those lineage diagrams. However, , it does mean if they know what a lineage diagram is and consequences for contracts, observability, things which can affect cost value, have implications, it’s then easier to get time or money assigned for those types of projects because they have some awareness of it versus you rocking up out of the blue, our observability is a mess. So I fully agree. I’m definitely not saying to try and turn the execs to data lineage experts, but again, just to increase awareness. Something I say a lot is just as companies want to become more data-driven. Data engineers should want to become more business driven. And so it’s all about that.

Just, I’m not expecting data engineers to know the business through and through, but we should be aware of key metrics, processes. . Streams of revenue, what have you. And so I think it’s just about the conceptual awareness more than anything else.

Shane: I think the other thing is the complexity of the map also tells me a story. In our product, we run a relatively simple three-tier architecture. We have history, design, and consume. And then within our design layer, there is effectively three objects you can create. You can create a concept object, which is a list of keys for a thing, customer, supplier, employee product order payment.

You can create detail about it. Customer name, product skew, order quantity, payment dollar amount and you can create an event, a relationship between them, customer order product and that’s it. There’s only three types of objects you can create. And what that means is when I look at our lineage graph, it may have lots of left to rights, but the number of columns in it is very light, which means when I have to troubleshoot, I have a very short conversation with myself.

Is it the history layer? Is it one, the concept detail or event and design, or is it the consumer layer? When I go look at other organizations and, we’re creating a transformation code that has lots of crate tables temporary tables in between, and I’ve now got 16 columns for a relatively simple transformation.

That complexity comes with cost.

And we’ve gotta really understand that. And then the other thing we can do is, if we think about it if these maps become context, if they become metadata, if they become something we can query, we can actually put a boundary around a map and say, Hey, if we replace the source system, how many of those nodes and links, how many of those consumer and producer ensembles need to be touched?

Oh, 250 out of how many? Out of a thousand. Okay. So what we’re saying now is we actually have to refactor 25% of our entire state, and we can get a sense of the cost of that change, at a really high level not in detail, but we can start to really understand how much of an impact on the system we’re gonna make when we make these types of changes.

Rob: Yeah, I don’t have anything to add. That’s exactly right.

Shane: Which again, comes back to actually turning this. Context, this metadata into actual data we can use is probably something that we should think about a lot more. Because I’m like you, I just draw them. I’ve thought about automating them to make my life of drawing them easier, but I haven’t actually thought about using them as a global repository to help me make better decisions.

Rob: Yeah I think there’s something really interesting there because as I said, I see these very much as a compliment to those other artifacts that already exist. Your org charts, your architecture diagrams, data dictionaries, business ies, so I think there is a really powerful layer there that if you could bring all this together as your context layer and then yeah, use that again just as context.

I think there’s something really powerful you could do with that. I dunno, of anyone or anything that’s implementing that or even thinking about that yet.

Shane: And it comes back to ea sparks. And all those standards and the tools that complied with those standards or applied those standards, that’s what they were doing. They were bringing all these different dimensions of lenses about everything we know about an organization. The problem was the tools were just horrible to use.

They weren’t friendly. And then the diagrams they produced were ugly. So I think that’s the key, is you’ve gotta make it easy to create and you’ve gotta make it easy to consume producer and consumer.

But do it in the way you define your system as much as the way you define, the way you work, the way you define your technology, your architecture, and your data flows.

So yeah, I think that’s a good point.

Rob: ‘ cause that’s really what I try to do. I’m a big fan of Dylan Anderson’s posts about people, process, technology and data, the four pillars. And I always use these in that context of bringing those four together and. How you build a data capability or data strategy, which in turn helps you achieve your business strategy, right?

So it’s all about that kind of hierarchy of do what we can and hope that we’ve done enough for it to propagate up into something more meaningful.

Shane: I agree. So when I do the data blueprints, I focus on team design and ways of working flows of work as much as I focus on architecture. ‘cause I got sick of strategies or designs that were just a bunch of technology boxes and none of the other things that were important. The other thing I do add though, is I start up with measures of success which is, if we’re gonna spend all this money changing what we do or implementing this new platform or whatever, what does good look like? How do we measure the investment was worth it, is that increasing the number of self service?

Being done by people outside the data professionals. Is it data or information being delivered faster or with a higher quality? What does it actually look like if we are successful after we spend all this time and money? ‘cause I’m surprised at the number of people that don’t even think about that.

They just deep dive straight into the architecture map,

Rob: yeah. Whereas I think, again, I’m fully aligned with you stuff like the architecture map are almost a means to the end. It’s how you achieve the goal. It’s not the goal. And so I also start with the people and process driven part. What do we want to achieve? Ways of working to achieve it. How do we know we’re successful? And then as I said, things like then your team design or responsibilities, they then inform my architecture choices. A lot of times I’m never gonna suggest to someone who’s got two data people on their team. To try and pick up you a enterprise level data platform plus storage, plus BI, or whatever the case is.

It’s all about almost the minimal system to achieve the goal and to hit success. And that’s again why I think this just helps my thought process. Keep it simple, keep it trackable, and just have impact versus bells and whistles, which you can always add on later as and when you need them.

Shane: And again, going back to that diagram you did, I can also see, ways of identifying who we’re gonna hire. And the one that you drew, it talks about power bi bean as the semantic bi layer PowerPoint, Excel bean as the primary reporting last mile tool. Then you’re gonna hire analysts that are used to using gooeys draggy, droppy those types of things.

Maybe a lot of Excel, which makes sense going back to the DAX formulas. But if you get somebody that’s, hardcore Python. Coder who wants to just use ACL I there’s gonna be a mismatch between the system that’s in place and their expectations. Now, you can deal with that by giving them a different set of tools, but now we’re gonna have this conversation of how does the hell does a CLI with Python code talk to the semantic BI layer?

Because, sure as shit, they’re gonna wanna punch back into the data warehouse layer and use the data or even back into the lake because that’s what they’re used to, the way they’re used to working. And that’s okay, as long as you understand there’s a mismatch and you’re gonna have to change something.

But if you go in there thinking you don’t, now you’ve got a problem. And I can point to you where that problem is. It’s a mismatch between skills of your analysts and the system you’ve built. So I think that’s important. Again, taking these different maps, different dimensions, and being able to compare them.

Rob: yep. And I think as a consultant, it gives me some validity. I’ve not just given them a list of tech, I’ve not just given you an architecture diagram that you probably don’t understand. I’ve given you not just the tools, but the personas, the workflows. I’ve talked through how I went from your requirements to this proposed solution. And it again, it’s transparent. It lets you have useful discussions with people. It lets you align their priorities, whether it’s cost, people, performance, whatever it is. And that’s why I really like this. And as you said, that kind of mismatch, identifying it early, you then get to make an informed decision. I know I’ve said informed a lot. That’s what these diagrams give you. They give you informed decisions from the start before you are too committed to anything.

Shane: It’s also a decision that can be challenged because again, I can point to a box, I can point to a node, I can point to an ensemble, I can point to a line. I can point to a consumer, producer peer and say, that doesn’t make sense to me. I can point to a handoff and say, that looks like it’s missing something, or that handoff looks like it’s waste.

I can now start to challenge some of those informed decisions with an informed opinion.

And I think that’s important. Again, it becomes less conjecture. . Still an opinion, but I can actually try and get some clarity on where I’m disagreeing.

Rob: yeah. It gives you the scaffolding for meaningful conversation decision making. It’s less about opinions or people there’s still opinions, but it’s less about opinions without context and more about how opinions fit into a workflow, which involves technology, people, data, all the different parts. Yeah, so I fully agree.

Shane: One I remember back in, in the ghost of Dana past when we were doing big requirements up front, so I used to hate it, every now and again, there’ll be one of these mega projects, transformational things, and you used to get a list of requirements and number of the bloody things.

And that was effectively the input into any of the system plans and your blueprints. And I used to map the requirements to the nodes, so this node supports requirements 54 B, 27 Cs. , It helped mitigate some of the arguments that, where did this come from? But I’m not sure the, the juice was worth the squeeze. I kind of found it waste,

What about you? Do you actually map any of these back to requirement statements?

Rob: It depends on the level of the requirement statement, so I definitely don’t religiously apply it to all of them, but I think they are the requirements in general. Give me the context and that’s how I treat it in general. If the requirement is you need to ingest data from system A , that informs some choices. If your requirement is less specific and a bit more hand wave your conceptual, then I’m definitely not going to slog to try and assign it to a node. I’m going to use it as broad context so that then if when the discussion comes up, I can describe my choices in context of that. But that’s it. So I think, yeah, like you in the early days, ‘ cause this came from experience, the reason why I’m attached to this idea is I was working a job where none of this was in place and so we had to do this just out of necessity. And so I think, yeah, I’m like you, I started by very rigidly trying to. Almost one-to-one map the requirements to the flow and it doesn’t work. Or you end up with very rigid workflow and you don’t have any kind of freedom to make something better. And so now I yeah, I put, I’m selective over which ones I directly incorporate versus use context text.

Shane: I think actually as I as I was thinking about it as well I often use the requirements to identify where I have complexity in my map. For example, if I have a requirement that data’s gotta be able to come from the source system, the system of capture or production and be available in a last mile capability to a in consumer.

In less than two seconds I’m gonna have some kind of streaming architecture. And so if I look at your map, yeah. That is a typical batch orientated architecture. Like I could stream it, but I’d be really surprised if it would be streaming with those layers. So if I then have to do another flow, .

If I have to do another row on that diagram, which immediately makes it more complex and that’s only so I can stream, then I gotta justify where that came from and then potentially I could rearchitect the layered architecture. You have to be stream only. If it meets every other requirement, but now it can go back to what’s forcing me to have that complexity and is there any way I can remove the complexity without introducing more complexity?

. Because sometimes having only one row means it’s trying to do too many things. . And therefore it’s even more complex. You’re just hiding it. So yeah, I think actually thinking about it, those broad brush requirements help me again, put a boundary around things and say, I have to do this for these reasons.

If those reasons aren’t valid or important, then I can stop doing those things.

Rob: Yep. I also think. Some requirements are , very specific. And like we talked about earlier, they might only be applied to a finer grain version of this flow. So when I worked in defense, you can imagine there was quite a lot of strict rules about data encryption and personal information masking. So there, I wouldn’t worry about showing that, for instance, the high level workflow, but in the more data engineering focused one, that’s where I would include it. So again, there’s a trade off there in decision about, it’s almost twice the work, but it can be twice as impactful to have those two green views of this one system. One for the kind of exec level, one for the technical level.

Shane: But again, that helps the collaboration conversation. So if you tell me that we need to mask people’s personal identifiable information in that scenario, their names, their date of births, maybe some of their deployment information. I’m gonna ask you, where are we masking it?

Am I masking it in the lake, the warehouse, and in the bi semantic layer?

Or are we saying actually the lake can hold the raw data? When we get to the data warehouse layer, then we’re gonna mask it, which means we now need to control access to that data lake layer that only certain people with certain, authority can see the data in there. So now I have a boundary, nobody’s allowed into that layer unless they pass a certain security level. So again, they’re just helping us make tradeoff decisions and also have a conversation about what happens, where and what doesn’t. And what does the contract look like?

Rob: Yeah, exactly. And then again, it gives you that extra view, like you talked about kinda access control of, if you’re in the cloud, there is a different view of this diagram where you might talk about networking or access management. And again, the exec level probably don’t need to see that, even if it’s a requirement, you might just have that as a bullet underneath with a check mark. But then in the more technical view, you really say, we’ve made this group which has these permissions, and that’s how we’ve satisfied the requirement. Again, it’s all about, yeah, controlling the information you present and which grain you capture those requirements.

Shane: So this idea of nodes and leaks has been around for ages. This idea of system thinking, it came outta lean manufacturing. It’s been around for a long time. The idea of business process mapping and understanding the flow of work being around for ages. The idea of enterprise architectures and diagrams that hold the ability to tell different stories at different levels, been around for ages.

Why do you think in the data domain, it’s very rarely used?

Rob: It’s a great question. I dunno the answer. I think in my experience, data, people have always, for whatever reason, decided not to learn from, say, software engineering, like data engineers. And so data folks tend to be, can be very technical. They’re almost very cultish in like they do data. And I think there’s very few people relative to the number of data folks who actually understand the business and how processes work and how to communicate that. So I think one of the biggest things I learned from academia that I’ve brought to my career is communication is presenting to multidisciplinary audience audiences. And I think that’s something that it’s just skipped if you go from undergraduate degree to an internship and you just that, that piece of learning is missed a lot of the time. And so I’m not saying that’s the only other main reason, but it’s a practical reason I think that I’ve seen in my career that prevents this kind of thing from picking up direction.

Shane: I think we segment the work into hyper specialization and then we start at the lowest level. So we introduce the idea of, a data engineer that’s gonna write code using. Or we bring in, that you’re gonna be an analytics engineer that’s gonna write a model in DBT and we don’t teach end-to-end system thinking as a framework, as a set of patterns that you’ve gotta understand first before you can go and do the work.

If you walk into a factory, and again, I worked in the factory as a kid 30 odd years ago, but you walk the line, you’d understand the flow when you get onboarded, the flow of work from the beginning of the factory to the end so that you understood where your station was what your part in it was.

And I don’t think we do that in the data world. I think the other thing is and one of the reasons I wanted to get you on is that article you wrote, it was simple to understand, for me at least. The thinking is really aligned to the way I think, and that probably helped. But what I often find, especially if I look at academic writing.

It’s really research based. It’s quite technical. It’s lots of complex ideas that aren’t distilled in the story I can understand. , I read them and I’m like, ah, I don’t get it. And so to take that complexity and write it with simplicity is actually really hard. Again, big ups to you for writing an article that distilled what is quite a simple idea, but can be complex down into something that’s easy to understand in the written word.

Rob: Yeah, I think part of that comes from the kind of, even though I’ve jumped fields a lot in my career, the one consistent has been an interest in mentoring and developing others. And so to do that, you really need to be able to make information accessible, whether that’s just choosing the right type of diagram, using the right vocabulary, whatever it is. So a lot of these ideas that I am drawn to. Yeah, it’s often something which I’ve not seen someone else explain quite simply. So I take up the challenge to do it because I think it’s has value to even if only, yeah, two or three people read that and say, I now get it. I’m happy with that.

That’s, that’s worth it. But I think that kind of thinking as you said, the systems thinking, the context, being able to understand how these systems come together, the modular parts that they’re made up of, and then how to communicate that not just between technical teams, so not just between data engineers, but also data engineers and data analysts, but also data engineers with managers with C-Suite. That is I think, one of the biggest gaps. And especially now with, lots of buzz that you see about, we don’t need junior engineers. There’s not just a technical deficit there just because you can outsource some work to LLMs. There is a real issue of new starters, not picking up in quotes, the soft skills, communication skills, and learn how from the start to talk about these ideas, to communicate these ideas. And I think that’s something which is a concern for me and something which I’m podcasts like this, I think do a really good job of starting to address the gap of giving people another avenue to learn something like this quite easily.

Shane: Yeah, I think there’s the whole argument right now about, role of the junior or the role of the senior when we all get 10 xd it’s gotta be interesting because I think education has to change. Because the skills that we learn, the technical skills of how to code are gonna be supported by, the tools of the future, which actually just leaves us the system problem, actually understanding how to daisy chain those tools and that code together to become far more valuable.

But just go back to that writing process. I find it relatively easy to write complex words, I can just brain dump and I can just write lots of complex stuff. But to then refine it down into something that is clear and simple and reduces the complexity and increases the cognition when you’re reading it.

I find that the hard effort, that’s where I’ve really gotta focus and iterate time and time again and spend my time. Is that what you find, or is it like the way you do your diagrams, you find it slightly different in terms of the process you use?

Rob: No, I think it’s quite similar. Again, from my academic work, I would write quite technical, scientific papers, but then I would convert those into PowerPoint presentations for conference presentations and stuff. And so I theory writing stuff like this the same way. So I go from a technical idea that I understand and then I always start by creating the diagrams that capture what I’m thinking and then the words I can just naturally fit around that. ‘ cause once I have the core concept in images, I can just walk through the process of what do you need to know to understand this image? Then what do I want you to take away from this image? And so that’s always my writing process of taking some complex and making it more accessible.

Shane: I think again, that idea of writing some complex words and then drawing yourself a simple map and see what’s missing or what needs to be added, that helps that visual to written and back and forward

Rob: That’s why I always have a notepad on my desk. ‘ cause I always find, even if I’m reading or learning something, if I can draw it out, I can understand it because I’m a very visual learner, like you said. I think so, yeah. That’s how I go through this.

Shane: So if people wanna follow you and read what you’re writing and get some more of these cool ideas, these cool patterns in a way that’s simple to understand, how do they find you?

Rob: So the two best places to find me are on LinkedIn. So Robert S. Long, I think is my handle. And on Substack at Long last Analytics. So that’s the name of my consultancy, my surname’s long, and I like cheesy things, so I like the at long last, I’ve solved your problem aspect, so that’s where you can find me.

Shane: I’ve been reading your stuff for a little while and before you mentioned that, I only just got the joke as I was, as we’re doing this podcast. I was like, ah, actually, hold on. That’s your last name. So well done that. I like cheesy as well, but normally I pick it up a lot quicker than that.

Excellent. All right anybody wants to read what Rob’s writing go to at long last Analytics on Substack hook him up on LinkedIn. Otherwise, I hope everybody has a simply magical day.

«oo»

Stakeholder - “Thats not what I wanted!”
Data Team - “But thats what you asked for!”

Struggling to gather data requirements and constantly hearing the conversation above?

Want to learn how to capture data and information requirements in a repeatable way so stakeholders love them and data teams can build from them, by using the Information Product Canvas.

Have I got the book for you!

Start your journey to a new Agile Data Way of Working.