Join us for the next AURA session in the app.

Got a Question you want to ask and have answered in the next AURA session, leave a comment and we got you!

Leave a comment

The Problem

The problem to be solved is that we collect data from multiple platforms and multiple publishers, and the Content Titles don’t align, so being able to show a single measurement number for a piece of content is a challenge.

For example the Content Titles used for Shortland Street on Linear are often slightly different to the Content Titles used for Shortland Street on Video on Demand.

When we get to YouTube then the problem get a lot bigger, content publishers want to be able to tweak the titles they use to try and increase engagement and reach, but we always need to know which funded content those YouTube videos relate to.

The Blueprint

As we build this out we always start with a light architectural Blueprint so that Nigel and I are on the same page and talking about the same things. (we call this our agile-tecture sessions)

For this one we ended up with 5 pattern components we cared about:

1. Matching Engine
   (AutomatedAI)
   The Pattern(s) that create the suggested matches for Content Titles

2. Matching Store
   (Operational Context)
   The place we store those suggested matches

3. Human in the Loop
   (AssistedAI)
   The interface that allows a Human to be in the loop and approve/decline/pass the suggested match

4. Mastering Policy
   (Business Context)
   The place where we store those human matching decisions as policies

5. Mastering Execution
   (Business Context and Code)
   The business logic that applies those matching decisions in a repeatable way every time we collect or consume the data

Experimenting with Agility

We tend to work on these architectural components in parallel as a decision for one will often impact the decision for another.

And we will often discover something we haven’t thought about that means we end up needed an additional component or two.

For example what we decide for #2 and #4 impacts what we build for #3.

We can use a relational physical data model or a graph model and we can use Spanner, Spanner Graph, BigQuery, BigQuery Graph. These decisions impact what we build for #3.

But also the design of #3 will also impact teh choices we make for #2 and #4.

In the past we had awesome humans help us prototype our app, Chris did the awesome Design and Richard did the awesome Build. Nigel had of course automated the Deploy.

The challenge was the cost of change. It reduced our agility.

So I have been experimenting with the use of Claude Design and then Claude Code to see if we can reduce the time and cost of those experimentation loops.

My Claude Design Process

Here is the process I go though using the Human in the Loop #3 component above as the example.

1 - Pattern Storming ideas, the Town Plan

The first thing I do is give it some light instructions on the intent of the app and then ask for 10 options.

As you can see I don’t worry about spelling etc.

Claude Design came back with this:

What I am doing in this step is looking for UI design ideas.

I always have an idea of what I think these screens will look like before I start the design process and how they will flow. But often Claude Design will come up with a UI or UX pattern I hadn’t thought of, and I then get it to help me rift with those ideas to see if they have legs or not.

2 - Explore the possible Houses on the Street

Next I step back and look at the UI Patterns across all the options. Either one will stand out as the my clear preference and I will start iterating that or I will want to experiment at the higher level, lets call it the town plan level.

In this case I went with the town plan level and so asked for another round.

If you have seen any of the designs in our AgileData UI you will see how Claude Design is re-using the “tile card” component from our Design System for these iterations.

While I love reusing that design component, i’m not sure it fits well for this use case. So we need to iterate that.

3 & 4 - Design the Rooms in the House

One of the things I love about Claude Design os the way it numbers each option. So it allows me to say, take this bit from 2b and this bit from 2d and then give me new versions

It also allows me to add comments to specific parts of each option and then get it to generate new versions based on those comments.

In this flow I did both.

I wanted to focus on one Component and validate it is what we want, so first I picked a specific option and asked for a new version of it.

Focusing on the design of one room in the house so to speak.

It still wasn’t quite want I wanted so back to designing all the rooms again.

This time I provided more guidance using comments. This allows me to tag comments about different parts of the options, highlight the bits I liked and want to keep and the bits I don’t like and don’t want reused.

5 - Design the flow between the Rooms in the House

Once I think I have an idea of the moving parts, I start to refine each one.

In this round I want to focus on the different flows through the screens depending on the Matching Pattern that was used to selected the recommended match.

I want to be clear to the Human in the loop which matches are pretty confident and which require more curation.

6 - And back to the Furniture in a Room

Once I ham happy we have the high level flow of the screens sorted, then I start to focus on the detail.

This time I want to return to focussing on the components within the match that provide the context for the recommended match.

I can’t quite decide how I want those to look, so rather than specific instructions for that part of the design, its back to the “give me 10 ideas” pattern storming

7 - Iterate a specific item of Furniture

8 - Time for the Human

I could keep going iterating this way for ever, but I would quickly hit the diminishing returns anti-pattern.

Plus the closer I get to supposed Done Done, the less likely the people who will validate these screen designs will feel like they can can provide feedback or suggest changes.

So now its time to create a prototype app they can try and see what they like, dislike and hate.

Claude Design creates an prototype you can click though. Haven’t tried that in anger as a way of getting feedback, so keen to give that a try and will report back later.

Mature Design System

One of the gifts we have is Chris our Designer was fastidious on creating a Design System for all the UI components he designed.

This was in Figma and we were able to import that from Figma into Claude Design, as the basis of the design system for prototyping.

I think that saved us a lot of time and effort in the initial setup.

Claude Design will still ignore some of the design system components, specifically fonts for some reason, so I always have to reinforce for Claude Code that its to use the Prototype designs as the ideas not the spec and it must reuse the AgileData App Design System when its creating new screens for our AgileData App UI.

Define Once, Reuse Often (DORO)

One of the things I keep in mind throughout this design process is reuse.

This series of screens will be used for more than just matching Content Titles, we will use it to help match Customers, Suppliers, Products etc for other AgileData Information Products for other customers.

But I don’t want to over engineering it right now, so I focus on the use case in front of me.

That is a bit of a balancing act.

Next Steps

Next I will get feedback from the key stakeholders on how they find this design based on the Claude Design interactive prototype.

And I will share it with Nigel as context for our conversations on what we are going to do for the other moving parts,

The key being this was a minimal investment in time and effort to get this far, and so the cost of change is low.

Its the value we get from this AssistedAI pattern.

And that give us agility to iterate this as much as we need to.

Join us for the next AURA session in the app.

Got a Question you want to ask and have answered in the next AURA session, leave a comment and we got you!

Leave a comment

Subscribe now

Subscribe now

Subscribe now

Join us for the next AURA session in the app

Got a Question you want to ask and have answered in the next AURA session, leave a comment and we got you!

Leave a comment

Join us for the next AURA session in the app

Got a Question you want to ask and have answered in the next AURA session, leave a comment and we got you!

Leave a comment

Join us for the next AURA session in the app

Got a Question you want to ask and have answered in the next AURA session, leave a comment and we got you!

Leave a comment

Another AURA session is booked!

and I will chat on Monday, 21st August 2026, 8am MDT / 3pm GMT.

You can join the Substack Live and ask your questions live here:

https://open.substack.com/live-stream/295955

If you cant make it, the recording will be published on the Agile Data N’ Info substack after the live session.

Feel free to ask any questions you have in the comments here and we will answer them in the live session.

While you wait for the session you can read Anna’s awesome content here:

When Data Met Product

A substack about how product management can help drive better outcomes with data - frameworks, case studies, and lessons learned from a product leader in data & AI platforms.

By Anna Bergevin

The AURA Session Format

AURA is aimed at working Data Practitioners who are tired of polished webinars and want the real, in-the-weeds conversation, where the live audience sets the agenda and the unedited recording gets published afterwards

Data Practitioners get to ask us questions and we will try to answer them.

There is no agenda, each of the AURA foursome have “a very particular set of skills, skills they have acquired over a very long career …”, the live audience will drive the focus of the conversations.

If the room goes quiet, we will just chat amongst ourselves about what we see is actually happening in the data world right now and what we each worked on and wrestled with last month.

Come join us

It will be quite the ride!

Another AURA session is scheduled!

and I will chat on Friday, 28th August 2026, 3pm EET / 1pm UK.

You can join the Substack Live and ask your questions live here:

https://open.substack.com/live-stream/320306

If you cant make it, the recording will be published on the Agile Data N’ Info substack after the live session.

Feel free to ask any questions you have in the comments here and we will answer them in the live session.

While you wait for the session you can read Juha’s awesome content here:

Common Sense Data

Writings on data that (should) make sense. Tackling the difficult non-technical problems relating to data modeling, data governance, data team operating models, and many more!

By Juha Korpela

The AURA Session Format

AURA is aimed at working Data Practitioners who are tired of polished webinars and want the real, in-the-weeds conversation, where the live audience sets the agenda and the unedited recording gets published afterwards

Data Practitioners get to ask us questions and we will try to answer them.

There is no agenda, each of the AURA foursome have “a very particular set of skills, skills they have acquired over a very long career …”, the live audience will drive the focus of the conversations.

If the room goes quiet, we will just chat amongst ourselves about what we see is actually happening in the data world right now and what we each worked on and wrestled with last month.

Come join us

It will be quite the ride!

Join us for the next AURA session in the app.

Agile Data N’ Info

AURA - Chris Gambill - Friday, 10 July 2026 - 1pm EST / 6pm GMT

Second AURA session is booked…

Read more

2 months ago · Shagility

Got a Question you want to ask and have answered in the next AURA session, leave a comment and we got you!

Leave a comment

Its been a while since I was back riding on the AgileData N’ Info podcast horse so to speak.

So decided time to mount up and this season I am trying something a little different.

I have been lucky enough to meet some awesome data experts over the years, and extra lucky to be able to chew the fat with them on a semi regular basis to discuss what we see happening in the data, product and AI domains.

Its also a great opportunity for me to ask them my latest burning questions.

So I thought what would happen if we had those chats in public and let the audience ask questions.

Me + one data expert, ~1 hour, fully live, audience drives it, published raw with no editing.

Then I thought you know what is better than 1 data expert, 4!

Four regular guests on a monthly rotation, one guest a week.

And so I asked four awesome data experts if they were up for it and they all said yes!

This is going to be an experiment.

• We will use Substack Live for it (first time I have used it)

• Who knows if anybody will turn up and ask questions (but if they dont that is ok, we will just chew the fat anyway)

First awesome data expert scheduled in for next Wednesday, 1st July.

If you want to see the schedule as we build it out, there is a new section for it here here:

agiledata.info/s/aura-a…

Come join us, it might be quite the ride!

“One Shot BI Apps” aren’t good enough for us

As I build and deploy Information Products in production, I have the benefit of the deployment and running of the Information Products being completely automated using the AgileData Information Platform templating system built by Nigel a wee while ago to enable this.

This means I can focus on what the Information Product front end looks like and what data and information it presents to our customers users, rather than how I deploy and manage them.

But as I build more of these Information Products in production I find the need to define and reuse ‘widgets’ in a repeatable way is becoming compelling.

For example I typically need a Date Picker in an Information Product.

But I don’t want Claude Code to randomly generate a new one with different code, different features and different styles every time, I want a default and repeatable version that I can choose change or replace if I need to:

Same with Page Intro’s, I want a certain style:

And with KPI Cards, I want to be able to define a standard style and have all new Information Products use that style:

And in the near future when I add to the KPI Card component the ability to see period on period change, which I know I will be doing soon, I want all my Information Products I have already deployed to inherit that change, where it makes sense.

Define Once, Reuse Often (DORO)

This behaviour is just following our DORO (Define Once, Reuse Often) principle.

Design System

Luckily there is a well proven Pattern that solves this reusability problem called a Design System.

I am out of Claude Code tokens as I write this so over to my other friend Perplexity to find the quick answer for me:

Claude Code of course has access to a lot of content about Design Systems and so with some simple prompts it can start to build one out as I am building Information Products.

There is a whole problem space on how you build these components in a way that can be inherited, but that is a problem and Pattern conversation I will write up in another article.

Making our Design System visible

One of the other challenges is we are not the only people reusing the AgileData Information Product templating capability.

Our AgileData Network partners also use it to build, deploy and manage Information Products for their customers.

So how do I let them know what Design System components are available so they can choose to reuse them, customise them, or ignore them and build their own?

Luckily again there is a proven Pattern that solves this problem.

Companies like Google solved it when they published their Design Systems like Material.

https://m3.material.io/

They created a website that let you explore and see the components that were available.

So that is the Pattern I reused.

First an overview of the Design System Context.

Then each ‘thing’ is visible.

And each Component is interactive so you can quickly see how it works:

And then provide visibility as things constantly change:

Dynamic Context not Static Documentation

In the past these Design System websites would be static, i.e they would be generated and then hosted so the AgileData Network partners could view them at will.

But that always took effort to keep it up to date.

Yes I could automate that refresh each time I get Claude Code to iterate the design system, but given how often I am iterating it at the moment just to get to “BI tablestakes” (that is a whole nother article) it will burn tokens constantly updating it, and there would always be those times where Claude Code will “forgot” to update it.

So applying our “Context not Code by Default” principle, I have embedded the capability to generate the Design System into the Design Systems / Information Product templating system itself.

What this means is when an AgileData Network partner wants to see the latest version of the Design System they just ask Claude Code to generate it. Claude will then read the Design System, generate a standalone localhost app and the partner can explore it to their hearts content.

That way it is always up to date, as it is generated when it is needed using the Context that is current.

(and it uses the AgileData Network partners Claude Code tokens, not mine …. )

Iteration One Done

This is just the first iteration of the Design System.

Now we need to test it for a while with our AgileData Network partners and prove it actually makes their lives easier and what we need to change.

As I have been working with Claude and Claude Code to do some “vibe coding” development over the last wee while I have also been working on a “harness” that is designed to make me more efficient.

Its following the DORO (Define Once, Reuse Often) principle I try and follow.

One of the surprising thing (to me anyway) is when I mention the concept of a “harness” to other people I sometimes get blank stares.

I actually did a LinkedIn poll and this feeling was reinforced by the reponses.

https://www.linkedin.com/posts/shagility_based-on-a-few-chats-and-a-bunch-of-reading-activity-7458771767482839041-cO_e/

No seriously what is an “AI Harness”

One of the problems I have is I can’t share the harness files we have developed within our AgileData platform and I don’t want to share the harness files from my PersonalOS version.

I thought about mocking some up, but then they would’t be real examples and so wouldn’t be that helpful in showing what they do, how they do it and their real value.

I actually started my harness journey based on playing with OpenClaw, listening to a few podcasts on the subject and chatting to . Nick did an in-person meetup on how he built his PersonalOS harness and how he uses it, I learnt a lot at that session.

You can watch a version of this talk online here:

There is also a great article by on LinkedIn that explains what an AI Harness is:

https://www.linkedin.com/pulse/i-learned-ai-harnesses-you-should-too-andreas-kretz-v2okf

Claude Write up on how my harness works

One way I thought might be useful to share this part of my journey is to get Claude to go through all the chats and code we have worked on together over the last we while to create my harness and explain it.

This has been based on iterating the harness for three specific use cases:

1. To support my PersonalOS

2. To support the vibe coding of the open source Pattern Template Standalone Apps I have been creating and sharing. (https://github.com/AgileDataGuides)

3. To support the creation of an Information Product templating system in the AgileData platform and using it to build and deploy Information Products for our customers.

The key is each time I work on one of these use cases, the learnings from doing that work is used to enhance the harness. And hopefully makes that work quicker and easier for me next time.

So here is what Claude wrote.

Universal harness     — applies to every Agent
   Family templates   — shared patterns for one Agent family
      Real Agents     — specific deployments

Weekly retro.md (per Agent)
   ↓ proven across sessions
"Retro learnings" in wow.md (per Agent)
   ↓ would benefit other Agents
Harness inbox (cross-Agent review)
   ↓ accepted by curator
Universal conventions / shared skills (every Agent)
   ↓ sync
Sister Agents pick it up

proposed_destination: agiledata     # universal Agile Data rule
# or: custom-app-template           # all Agile Data custom-app Agents
# or: demo-template                 # one Agent family
# or: shagility                     # personal preferences only

Markup version of the text at the bottom of the article if you want to copy and paste it into your own LLM.

Some More Context

I asked Claude to explain:

1. What the harness is

2. How it is inherited across “agents”

3. How it differs from a bunch of skills.

When it talks about “Agents” there is nothing fancy happening, its is just a different directory on my MacBook, that has separate code in it that I am working on for a specific Use Case.

Nothing I am doing is unique or magic, a lot of people are building these harnesses as they use these tools.

Why not just use somebody elses “harness”

There are a bunch of harnesses out there you can download and use.

I tried a few and found they didn’t help me as much as I had hoped.

It might have been the type of work I am trying to use them for.

It might have been my lack of coding skills.

Try one and see if they help you.

Part of this whole journey is learning as I am doing so I also naturally leaned towards craft my own.

My Agile Data Way of Working Language

You can see the Agile Data language I use coming through strongly in this harness:

• Persona

• Policies

• WoW

• Retro

• Defintion of Ready

• Definition of Done

• TODO / DOING / DONE

• Press Release

I have naturally being applying the Patterns and Pattern Templates I coach a human Data and Analytics team to use to my machine buddy.

Others use different terms for the same things (Identity, Soul etc). They may also include the things I hold in multiple files in a single file for their harness.

And I do worry that applying Human centric patterns to the Machine may not be the optimal approach.

A journey not a proven Pattern or Pattern Template

All the above is just a brain dump on part of my journey so far.

It is not a well formed Pattern or a tested Pattern Template.

But hopefully Sharing it in its half arsed state is still Caring.

Markdown Version of the Claude Content

# The Agile Data Way-of-Working Harness

A reusable convention package that gives every AI-assisted coding session in our repositories the same Way of Working — the disciplines, guardrails, and patterns we've evolved over months of working with AI coding assistants.

## The problem

AI coding assistants are stateless. Each session starts fresh, so the team re-explains conventions, hits the same gotchas, and watches the same mistakes recur across Agents. When a learning emerges in one Agent, there's no clean path to apply it everywhere it matters.

## The structure

Each repository carries a small, predictable bundle the AI reads at session start. Together this bundle defines a specific **Agent** — the AI is no longer a generic assistant; it has an identity, hard rules, a way of working, and a memory of what it has learned.

**Agent identity** — four short markdown files defining the Agent's character:

- `persona.md` — what this Agent is, its voice, who it serves
- `policies.md` — hard rules and guardrails (will not / will always / boundaries)
- `wow.md` — Agent-specific way of working + accumulated retro learnings
- `retro.md` — running weekly log of new learnings

**Universal conventions** in a single `CLAUDE.md` that the AI loads automatically:

- Task tracking (TODO / DOING / DONE files)
- Commit discipline (logical chunks, no work pileup)
- Definition of Ready / Definition of Done checklists
- Press release format for user-facing changes (working-backwards style)
- Estimation as t-shirt sizes (XS to XL), not minutes
- Pre-push verification (explicit checklist before push to auto-deploy branches)
- Design system compliance (every UI value traces to a design token)
- Scoped tool / data access per Agent (prevents wrong-environment bugs)

**A skills system** — discrete procedures the Agent invokes by name when triggered (e.g. *"draft a step-page"*, *"verify after change"*). Skills load on demand, so they don't bloat every session's context.

## The inheritance hierarchy

```
Universal harness     — applies to every Agent
   Family templates   — shared patterns for one Agent family
      Real Agents     — specific deployments
```

A new Agent cloned from a family template inherits all three layers and starts with a clean identity it customises. Every layer follows the same convention set.

## What it gives you

- New AI sessions reach productive work in minutes, without re-explaining basics
- Conventions stay consistent across Agents without anyone policing them
- Bugs caught once stay caught — they land as policies or guardrails
- AI estimates and pre-push verifications are predictable and comparable
- Cross-Agent learnings compound rather than re-occurring

## What it doesn't do

- Replace human judgement on architecture or product direction
- Generate code without context — good prompts still matter
- Work across tools that don't honour structured conventions

The harness is small, opinionated, and modular. Drop in what fits, leave what doesn't.

---

## The retro process — how learnings compound

The retro flow is the engine that keeps the harness alive. Without it, the conventions calcify and the harness becomes a relic. With it, every Agent that's harnessed benefits from every learning every other Agent surfaces.

### Three tiers, one direction

```
Weekly retro.md (per Agent)
   ↓ proven across sessions
"Retro learnings" in wow.md (per Agent)
   ↓ would benefit other Agents
Harness inbox (cross-Agent review)
   ↓ accepted by curator
Universal conventions / shared skills (every Agent)
   ↓ sync
Sister Agents pick it up
```

### Tier 1 — Weekly retro per Agent

Mid-session, when a learning emerges, it gets added to `retro.md` under the current week's heading. Examples of what lands here:

- *"Tooltips inside scrollable tables need to portal to `<body>` or they get clipped by overflow."*
- *"Don't fabricate descriptions when the catalog is silent — use an explicit `(no description in catalog)` fallback."*
- *"Status of an Information Product is about trust level, not dev stage. Code-complete + data-unvalidated is `experimental`, not `live`."*

The retro log is cheap, time-boxed, and read by humans during the weekly review.

### Tier 2 — Promote into wow.md when proven

If a retro entry has stuck across multiple sessions and continues to serve the Agent well, it's promoted into `wow.md` under "Retro learnings" — becoming a permanent Agent rule. The model reads `wow.md` at session start, so promoted entries shape future behaviour automatically.

### Tier 3 — Promote to the harness for universal value

A learning that would help every Agent, not just this one, gets sent to the harness inbox via the `retro-promote` skill. The promotion is a single markdown file with a frontmatter block declaring its proposed destination:

```yaml
proposed_destination: agiledata     # universal Agile Data rule
# or: custom-app-template           # all Agile Data custom-app Agents
# or: demo-template              # one Agent family
# or: shagility                     # personal preferences only
```

The curator reviews the inbox in a separate session, applies the universality test ("would another Agent benefit from this?"), and either accepts into the appropriate layer of the harness, rejects, or defers.

### The reverse path — sync down

When the curator accepts a learning into the universal harness, it doesn't magically appear in every Agent. Each Agent pulls it on the next `harness-sync` call. This explicit pull preserves Agent autonomy — a fork can choose not to sync a change it doesn't want.

### Why this matters

Without the retro loop, the harness becomes static reference material that quickly drifts out of date. With it, the harness is a living convention base that gets sharper every week. The cost of a hard-won learning is paid once; the value compounds across every Agent that follows.

---

## How this differs from "just a collection of skills"

It's tempting to imagine the harness is just a pile of skill files. It isn't. Confusing the two leads to a system that feels organised but doesn't actually change behaviour where it matters.

### Skills are tactical. The harness is strategic.

A **skill** is a discrete procedure for a specific task: *"draft a LinkedIn post from a published article"*, *"verify the changed code by running the test suite"*. Loaded on demand, by keyword match, when the user asks for the thing.

The **harness** is the identity, conventions, and rules of engagement that apply BEFORE any specific task starts. *"This is what this Agent is."* *"Here's what we never do."* *"Here's how we track work."* *"Here's the press release format every user-facing change must follow."*

A skill answers *how* to do a particular task. The harness answers *who is this Agent*, *what are the non-negotiables*, *how do we operate*.

### Skills load on demand. Harness conventions are always-on.

The model only loads a skill's full body when the user's task triggers its description ("write a press release" → load the press-release skill). Without that trigger, the skill sits silent.

Agent identity (persona, policies, wow) is loaded into the system prompt every session start. Every conversation, every response, the model is already operating inside that frame. That's why a stray "wrong tenancy" mistake gets caught — the policy is always in scope, not waiting for a keyword.

### Skills don't enforce discipline. Conventions do.

You can write a skill called *"do the Definition of Done"*, but it only fires if someone asks for it. The DoD checklist as a convention in `CLAUDE.md` shapes every response by default.

The pre-push verification rule that catches wrong-tenancy data, the press release format that protects brand voice, the t-shirt sizing rule that stops minute-level commitments — none of these would be reliable if they were skills the user has to invoke. They have to be the air the Agent breathes.

### Skills don't propagate. The harness has a sync path.

If you write a great skill in Agent A, it stays in Agent A. Agent B's session has no idea it exists. The retro flow + harness sync is what makes a learning in one Agent show up everywhere it matters.

A pile of skills with no propagation path gets reinvented in every repo. The harness pays the cost of a learning once.

### Skills don't carry Agent context. Identity files do.

A skill called *"build a marketplace page"* is the same procedure no matter which Agent it's in. But what a marketplace page *means* for an Information Product Agent with five live products is different to what it means for a single-product demo Agent. The persona, policies, and wow files give the AI that Agent-specific context BEFORE it reaches for any procedure.

### The relationship

Skills and the harness aren't competitors — they're complementary layers:

| Layer | Loaded when | Best for |
|---|---|---|
| **Agent identity** (persona/policies/wow) | Every session start, always-on | Who this Agent is, hard rules, way of working |
| **Universal conventions** (CLAUDE.md) | Every session start, always-on | Rules that apply to every Agent regardless of context |
| **Skills** | On-demand, by keyword trigger | Procedures for specific recurring tasks |
| **Retro flow** | Weekly + as learnings emerge | Keeping conventions alive and propagating wins |

Take any one of these layers out and the structure leaks. Stack them and the AI session arrives with Agent context already loaded, universal rules already in scope, the right procedure ready to fire when triggered, and a clean path for what it learns to compound into the next session.

That's the harness.

In this episode, the conversation explores the intricate world of Focal data modeling and identity resolution. Patrik explains how the Focal framework operates as a highly abstract, metadata-driven engine that creates an extremely agile, non-destructive architecture. By focusing on understanding the business and using documentation to generate code, Focal ensures that your architecture and documentation are always perfectly in sync.

The discussion also dives deep into the complexities of identity resolution, clarifying the crucial differences between stable “identities” (surrogate keys) and system-specific “identifiers”. Patrik breaks down how the Focal framework prevents key collisions and achieves seamless key integration across multiple source systems using a specialized identifier table.

Whether you are curious about ensemble modeling techniques, how AI and metadata-driven automation are changing data engineering, or want to learn about the new Daana command-line interface (CLI) that makes implementing Focal easier than ever, this episode is packed with valuable insights for data professionals.

Listen

View MindMap

Read AI Summary

Read Transcript

Listen

Listen on all good podcast hosts or over at:

https://podcast.agiledata.io/e/the-patterns-of-focal-data-modeling-and-identity-resolution-with-patrik-lager/

Listen to the Podcast Episode on Podbean

Subscribe: Apple Podcast | Spotify | Google Podcast | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser | Deezer | Podcast Addict |

You can get in touch with Patrik via LinkedIn or over at https://daana.dev

Google NotebookLM Mindmap

Google NoteBookLM Briefing

Executive Summary

The Focal framework is a mature, metadata-driven approach to data warehousing and integration that prioritizes non-destructive architecture and business understanding over manual coding. Developed and refined since 1998, Focal employs a specific form of ensemble modeling that decomposes data into three core components: keys, descriptions, and relationships.

The framework’s primary strength lies in its “documentation-first” methodology, where a massive metadata layer (consisting of approximately 80 tables) generates the physical implementation. This ensures that documentation and code remain perfectly synchronized while making the physical layer “cattle”—disposable and easily regenerated. In the context of identity resolution, Focal provides a robust technical pattern using an “Identifier Table” to map multiple, changing business identifiers to a single, immutable identity. As the industry shifts toward AI and Large Language Models (LLMs), the framework’s highly patterned, metadata-rich structure positions it as an efficient alternative to traditional, manual coding practices.

--------------------------------------------------------------------------------

1. The Focal Framework: Philosophy and Automation

The Focal framework is distinguished from other methodologies by its tight coupling of data modeling patterns and a technical automation engine. It is designed to create a “non-destructive architecture” that allows for rapid changes without affecting existing structures.

The Documentation-First Approach

A central tenet of the Focal framework is that the value of data warehousing lies in the understanding of the business, not the code itself.

• Metadata over Code: The framework captures metadata models, mappings, and rules into a repository. This metadata then generates the necessary code and holds comprehensive lineage (business, technical, and operational).

• Disposable Code: In this paradigm, metadata is treated as a “pet” (highly valued and preserved), while code is treated as “cattle” (disposable and easily replaced).

• Synchronization: Because the documentation (metadata) creates the code, the two are never out of sync, solving a common problem in long-term data warehouse maintenance.

Structural Complexity and Learning Curve

Despite its efficiency, Focal has historically faced a low adoption rate outside of specific European regions due to its abstract nature. It is often described as an “engineering dream” that requires a significant shift in mindset from traditional relational modeling.

--------------------------------------------------------------------------------

2. Focal Modeling Mechanics

Focal is a form of ensemble modeling, similar to Data Vault and Anchor Modeling, but with distinct differences in how it handles attributes and normalization.

The Three Pillars of Focal

The model is built on three specific structures to ensure agility and minimize the impact of change:

1. Key Structure: Manages and controls the unique identifier.

2. Description Table: Holds the descriptions of data using “typed tables.”

3. Relationship Structure: Manages associations between keys without the use of traditional foreign keys.

Atomic Context and Named-Value Pairs

Focal utilizes an abstract method for storing attributes:

• Typed Tables: Rather than standard business columns, tables use generic columns (e.g., start_timestamp, end_timestamp, value_string, unit_of_measure).

• Atomic Context: Data is grouped to answer one “atomic question” per row. For example, a monetary value and its currency are stored on the same row because one cannot be understood without the other.

• Human-Centric Querying: The model strives to ensure that all data necessary to answer a basic human question is available on a single physical row, effectively reaching a state similar to fifth normal form.

Comparison of Ensemble Modeling Techniques

--------------------------------------------------------------------------------

3. Identity Resolution and Key Integration

Identity resolution in Focal addresses the challenge of recognizing the same entity across disparate systems where identifiers may overlap or change.

Identity vs. Identifier

The framework makes a critical distinction between two concepts:

• Identity: A stable, immutable surrogate key that represents a unique instance of data throughout its entire lifecycle in the analytical system.

• Identifier: A business-derived value (e.g., account number, email) used to find an identity. Identifiers are unique within their own data space but may change over time.

The Identifier Table Pattern

Focal uses a technical “Identifier Table” to manage the relationship between multiple identifiers and a single identity.

• Key Collision Avoidance: By using source-system prefixes or concatenated strings, the framework prevents different entities with the same business ID from merging.

• Multi-Identifier Mapping: The table allows multiple identifiers (e.g., a Social Security Number and a Customer ID) to point to the same surrogate key.

• Key Propagation: When a new system introduces a new identifier (e.g., a local ID) alongside a known one (e.g., SSN), the framework automatically propagates the existing identity key to the new identifier.

--------------------------------------------------------------------------------

4. Performance and Cloud Optimization

As data architecture has moved to cloud platforms like Snowflake and BigQuery, the Focal framework has evolved to manage costs associated with columnar storage and large-scale lookups.

Single vs. Multi-Identifier Modes

• Cost Efficiency: Performing lookups for surrogate keys on billions of rows is expensive in the cloud. Focal now allows entities to start in “single identifier” mode (using natural or hash keys).

• Automated Migration: If an entity eventually requires multiple identifiers, the framework uses metadata-driven scripts to automatically migrate the data from a single key to a surrogate key structure across all related tables and relationships.

Storage and Indexing

• Relational Databases: Focal tables are “deep” rather than “wide,” making them highly efficient for indexing due to their consistent, repeatable patterns.

• Columnar Databases: The framework’s reliance on unique values and patterns works well with the compression algorithms of modern cloud data warehouses.

--------------------------------------------------------------------------------

5. The Impact of Artificial Intelligence

The emergence of AI and LLMs is viewed as a turning point for metadata-driven frameworks like Focal.

• Devaluation of Manual Coding: As AI becomes capable of generating code, the value of manual programming skills decreases. The primary value shifts to the ability to design and understand complex systems and architectures.

• Machine-Readable Architecture: Because Focal is entirely pattern-based and metadata-driven with no exceptions, it is exceptionally easy for AI to interpret. An AI can read the metadata layer to understand the data layer and answer business questions without the “cheating” or “bespoke” code often found in manual implementations.

--------------------------------------------------------------------------------

6. Significant Insights and Quotes

“The value of data warehousing... was the understanding of data. The value was not in the code. The code had a secondary value in the architecture.”

“Your coding skills are not that valuable anymore. Your ability to design and understand systems, how a good design and a good architecture is, becomes much more important than being very skilled at programming.”

“The metadata is our pet; the code is cattle... we end up protecting or caring about that metadata... because the code becomes disposable.”

“Focal... can create an extremely non-destructive architecture that is very easy to change and add data... without affecting anything else in your architecture.”

Transcript

[00:00:00] Shane: Welcome to the Agile Data Podcast. I’m Shane Gibson.

[00:00:04] Patrik: This is Patrik Lager.

[00:00:06] Shane: Hey, Patrick. Thank you for coming on the show. Looking forward to this one today. Today we’re gonna be talking about focal data modeling and identity resolution. But before we rip into that, why don’t you give the audience a bit of background about yourself.

[00:00:19] Patrik: Yeah, I’ve been within the data area for the last 25 years working with data warehousing mostly, but more or less just integration is a lot of work. I worked in SOA projects and other kinds of integration areas as well. I’ve been around and seen a lot of things on the way. I also have been one of the main developer of something that is called the focal framework and the Focal modeling Method. So I’ve been working with that since 2004. Or less. The focal came along 1998 by a man who invented it more or less, but then he and I, plus a lot of other people, worked [00:01:00] a lot with developing it and moving it forward, so to speak. So that’s me.

[00:01:04] Shane: focals been around for a long time, but it’s not well known. In some European countries is relatively well known, but in the rest of the world it’s not. Why do you think that is?

[00:01:16] Patrik: The focal framework has a Achilles heel, if you want to call it. It’s an extremely abstract way of using data, but what it makes, happens, it’s, it can create an extremely non-destructive architecture that is very easy to change and add data and change data without affecting anything else in your architecture. Which makes it very agile and very efficient when building things. But as I said it’s an engineering dream, so to speak, has been a lot of smart people working with it. So it’s not easy to take in to learn. And that has been, its Achilles heel, so to speak.

[00:01:57] Shane: And when you talk about focal being a [00:02:00] framework, can you just describe what you mean by that? Because a lot of times we see data modeling patterns that are described as frameworks or methodologies and really what they are as a good way of structuring data and not really much around it. Or sometimes we see methodologies that are agnostic around the data modeling patterns that you can use.

[00:02:19] Whereas from what I understand, focal kind of binds those two things together closely. The framework and the modeling patterns are coexistent is that right?

[00:02:28] Patrik: Yeah. The, the focal framework as it all started with the focal modeling technique, but then it, it became a focal framework was more for documenting. Really what we ran into in all of these projects we work was the documentation always got outta line from what we implemented.

[00:02:47] It doesn’t connect it after a while. So the idea was that we would capture all the metadata models mappings rules and stuff like that into a framework that then would generate the [00:03:00] code for us and generate the things and hold lineage and everything in it. So that became the framework, so to speak.

[00:03:07] You could call it a automation engine, but it’s much larger because it also works as a. A vocabulary and description models holds all kinds of lineage, business lineage technical lineage, operational lineage. It’s all over the place, so to speak. The metadata model is pretty huge, is like 80 tables containing all kinds of metadata around what you’re doing.

[00:03:30] So that’s the framework, so to speak.

[00:03:33] Shane: so if I play that back it’s a technical framework on how you can implement and automate the focal patterns. So it doesn’t take care of people and process, it doesn’t to talk about standups or the way teams are structured. So it’s more in that how you build a technical framework to implement and execute focal without having to handcraft code on a regular basis to make it work.

[00:03:55] Is that right?

[00:03:56] Patrik: Yeah, exactly. The thing was that we, at one point [00:04:00] we came to the conclusion that. The value of data warehousing or the value is the understanding of data. The value was not in the code. The code had was a secondary value in the architecture. So we wanted to make the focus on understanding what you’re doing, understanding data, understanding the business, so that was the focus, and then could use that knowledge to generate the code instead of writing it, and then also always be in sync.

[00:04:29] The documentation would always be in sync with the code because the documentation creates the code.

[00:04:35] Shane: And there’s been a few of us that have been working on this idea of metadata driven data tooling or, active metadata or config driven. Now the new buzzword is context driven. I always saw three patterns in the world. There were the, let me just write a blob code. The blob code executes, it’s it’s on its own.

[00:04:57] It has no relationship with anything else. It’s pretty much gonna do with the [00:05:00] heavy dating work from the left hand side, the raw data history all the way through to consume. And then the next one is tooling that allows you to write blobs a code, but then produces the documentation after the fact.

[00:05:13] WhereScape was probably old enough to remember that that was one where, a lot of the value was when you generated your code, it created the documentation for you. And if I look at people using DBT, a lot of it was around that documentation, that lineage that you got outta it.

[00:05:26] And then there’s a smaller group like us where we said, create the documentation first. And then let the documentation effectively generate the code, So we define the metadata, the context, and then that code will get hydrated based on that core. And we end up protecting or caring about that metadata that config because the code becomes disposable.

[00:05:48] The metadata is our pet the code is cattle and DevOps language. And we see tooling come outta that metadata driven Pattern over the last 30 years, [00:06:00] but it never really gets traction. Why do you think that is? Why do, and we still keep doing it because we know it’s much faster for us.

[00:06:06] But why do you think that this middle ground where people can write code and then document it after the fact always seems to be more popular than defining the metadata, the logic, and letting it hydrate the code for us.

[00:06:19] Patrik: The love of the code. That’s the thing. everyone who is a developer loves their code. They are proud of it. They want to do it. They want to show how good they are at it. And then they come a tooling, a framework that you do not need to be good at writing code. You need to be good to understanding the business and describe the business and describe models and create things like that. And it’s hard to get organizations to accept that because you get often a a lot of fight back from the development side that says you can’t write as good code as me. And I’m better at that. And then always it comes. One of the biggest thing is often that they say, but can you do this? And [00:07:00] you say, no, you can’t do that. Okay, then it’s useless. Even if you can do 90% of the organization because you can’t do the 10%, it’s useless. So they. Drive that line up in again to their own management side. And then it’s, it becomes hard to go, to come into companies doing that. They probably feel threatened as well.

[00:07:20] The interesting point is now with ai, there’s no way out anymore. This is what’s gonna happen. Your coding skills are not that valuable anymore. Your ability to design and understand systems, how a good design and a good architecture is becomes much more important than being very skilled at programming.

[00:07:42] Yeah.

[00:07:42] Shane: Also writing code by hand takes time.

[00:07:45] It’s busy work. So you can be busy writing that code and you’re not having to engage with the stakeholder side to understand the value, deliver that value incrementally. It, it’s not unusual for a data team to be able to do busy [00:08:00] work for a month and actually deliver nothing to a stakeholder.

[00:08:03] We, we somehow have this expectation, and I’m with you, I think the LMS and Gen AI are gonna reduce their expectation timeframe that a month, doing busy work is is not gonna happen. Vibe code and LLM or use a metadata config driven framework pick your choice. But I’d rather leverage an architecture that somebody else has created for me that I know works than let the LLM guess at an architecture that hasn’t been proven before.

[00:08:29] Okay. So framework is the technical implementation of fo. Focal started out in 1998, I think you said, as a data modeling paradigm. Do you wanna talk me through the focal modeling just for people that are more familiar with other modeling techniques and how it’s the same and how it’s different?

[00:08:46] Patrik: Yeah, the focal has been from the first point really method data driven. But it came up with the thought of three parts within the model where you controlled the key and that was its own structure. [00:09:00] And then you had a table that represented the description of data, and then you have their own structures for relationships.

[00:09:09] So you never had foreign keys between tables. And this was not my idea, this was the one who invented it. That this is how it should work. Because he had seen that complexity growth and that changes always happened. And when changes affects the model and when it affects the code, it becomes much slower to change.

[00:09:32] And a data warehouse after five or 10 years becomes very complex. So he was trying to find a way to do, to lessen that effect. And the thing is, one of the things that is making it abstract from a point of view is that the tables that are holding the descriptions are what we call typed tables. You don’t have normal business attributes, columns in the tables. They have generic columns like [00:10:00] start timestamp, end timestamp value string, the numeric string unit to measure. And then you have a key or a identifier of the information that lies on the row. So it’s what we call an tight key or an atomic context key.

[00:10:18] But what really says is it’s like a name value pair, if you know what that is. But it’s the inventor, he said it’s modeled name, value, pair, because it’s not one value and one key. It is one key. And you can have up to five different attributes if those attributes belongs together. So this is one of the, one of the problems with Focal, everyone you talk about it is it has a lot of underlying philosophical ideas and theories that doesn’t match how we do normal data models. So it’s always an abstract step to take in your mind, but why do you do that and how does it work? But from a, from, if [00:11:00] you want to simplify it very much, you would say that the tables that hold the description are named value pairs. You could say. So you have a ID or a key or what you normally that name and that says what is in the value string. And then you can add new attributes. They are just new names and they hold their own values. So you never change the table, you just add new types of data into that. So that is one of the biggest differences with normal data modeling really. So it’s it’s built for agility. More or less change, do non-destructive change.

[00:11:37] You can add attributes, remove attributes, and you never change the code. You don’t change the tables, you just do the change, so to speak.

[00:11:45] Shane: So let me play that one back. So you talked about first object is a key. Second object is a table which describes the values effectively and the key, and there’s some unique technical architecture around that, or modeling architecture compared to most [00:12:00] patterns. And then the third object is relationships,

[00:12:02] the keys that have a relationship at a point in time. my understanding is that is a form of ensemble modeling, You are decomposing the key from the descriptors of the key and you are decomposing the relationships and holding those separately. So that’s true, right? Focal is a form of ensemble modeling.

[00:12:20] Patrik: Yes that it is.

[00:12:22] Shane: Excellent. And then there’s two other ones that I know of, There’s data vault as one form and anchor modeling as another. So what I might do is just actually we’ve never had anybody on to talk about anchor. I need to do that but let me try and just play back what you told me around focal and how I think it’s different to data vault and anchor modeling.

[00:12:40] And I’ll get the anchor one wrong ‘cause like I said, I haven’t had anybody on to explain it properly. So the first thing is both anchor and data vault hold the key in a separate object like focal does. So that’s the same, correct.

[00:12:53] Patrik: Yes,

[00:12:53] Shane: Okay. And then in data vault we’ll hold one or many satellites that hold the descriptors or the [00:13:00] attributes against those keys.

[00:13:02] But we will hold a table that has, or a view that has many columns. So if I have person’s name and person’s age or date of birth, I will hold those as columns in the satellite and anchor. My understanding is I would hold each of those attributes as a separate table.

[00:13:20] Patrik: yes, that’s

[00:13:22] Shane: And what you are saying is, in focal, what we do is we use a named peer key.

[00:13:26] So we’re making those columns rows. However, there’s an extra little bit of complexity or value in there that you can hold multiple values against one pair. So I could decide to have name and date of birth. And what sounds like a Jason string is it’s a way of. Typing an object and having a subset of values against it.

[00:13:50] Is that right?

[00:13:51] Patrik: No the underlying theory, you can do that, but that’s not why it was designed for it. It is designed for what we call an atomic context. And the atomic [00:14:00] context is, so the key but we hold is we call it the atomic context. Key is a group of attributes, one, two, many, but five more or less is what we have in the tables, and it is supposed to answer one atomic question. So if you put a shoe number and first name, you don’t have an atomic question for that, right? Because you want to ask what’s your first name? What’s your shoe number? That’s two different questions. So they would be two different atomic context. But on the other hand, if I would say, what is the loan amount I have on my loan that I, my sign, my original loan amount, I might say it is 1000 euro.

[00:14:43] Because you can’t answer the question of a monetary value without its currency. So you need to have the value and the currency on the same row. Therefore, we have, as an example, we have a unit measure that we call unit to measure column that you say then, so you use the [00:15:00] Val Nu column and you say That’s 1000.

[00:15:03] And then in the unit measure it says Euro. So then you answer the atomic question. And as another example, more complex one is like a balance. That you have a balance. And to understand the balance, which is a point in time data, is only true at a certain point in time. You need the balance date. When was this balance value true? And what was the unit to measure of the balance? So obviously sudden you have three attributes within the same atomic context to be able to answer the atomic query. so this is more like from a human perspective think how does human answer ask questions? you can ask who are you?

[00:15:39] And you will get a lot of things. But if you ask an atomic question, the idea is that all the data should be on the same row. To answer that in the physical table.

[00:15:48] Shane: So let’s go back to that example then. So if I’m holding balance on my bank account, then I need to know what the point in time, the date time of that balance is. And I probably also [00:16:00] need to know what currency it is. So what you are saying is you hold those attributes together because actually they need to be together to answer that business question or to provide the context of one of those numbers.

[00:16:13] And therefore, rather than holding it outside of the physical structure in some form of, physical data model where I know I’ve gotta go join that with that, or query that row and grab these three columns. You’re actually saying by bounding them together in a row, the answer’s already there, You have to grab them all because we went to the effort of putting them there because that’s the information that you need to actually answer a question around what is my balance.

[00:16:40] Patrik: Yeah, so you can almost see it like a normalization of data. Because if you would have a, if you had a satellite in data vault, you could have , these three columns that we talked about, the date the value, and the currency. And then you could have what kind of classification balance is this?

[00:16:56] What is, so you just added up. But as we do [00:17:00] that, we break it out. So these, this answer one question, and these columns also these questions and break it out in rows then and these on, in, in in anchor. They go all the way. They do not care about how data from a human perspective is represented.

[00:17:17] They just break it out totally mechanically. One attribute is one table. So you have the value in one table and the currency, another table, and the date in another table. And then they generate views that joins everything together. so it’s three different approaches. Sometimes when I talk to last Runback, who is the inventor of Anchor, we talk about it, he says, anchor is in six normal form. And he, and not from a truly formalized perspective, but he says it feels like focal is fifth normal form somewhere around there. And then we have data was very small, third normal form.

[00:17:53] So it’s, you could see it like that. It’s more normalization is.

[00:17:57] Shane: And it would be interesting when you start putting [00:18:00] LLMs across it, because with data vault we’re gonna have to provide the context of which columns, which attributes in the satellite actually have value given the business question and anchor, we’re gonna have to say which tables right? Effectively need to be put together to answer that question.

[00:18:16] So that context is held outside the data. But in focal, what I think I’m hearing is actually the data has context as well. So if I give it, that name peer key and it can see that there is a balance, a currency and a date time. It’s gonna be pretty good at saying those three things are important to ask the business question of what was my balance, in New Zealand dollars.

[00:18:38] Okay. I can see doing it. One of the things that people often say against any of the ensemble modelings is, oh, storage is too expensive, it’s too hard. And so I can see that the estate currency, we’re gonna store currency again and again, right Where it’s bound to the context of one of the other attributes.

[00:18:57] In my view, storage is cheap now. And [00:19:00] actually with coer databases actually querying across tables as more expensive than querying within a table. But what have you found that’s one of the pushbacks with focal is data storage and duplication is bad. ‘cause it was 30 years ago when we had mainframes.

[00:19:15] Patrik: No never run into the issue of of storage. Really. And more, when we move really far back in the early 2000 when we had the normal relational databases the easy thing with the focal was that it was so easy to do, set up good indexes because the tables looks exactly the same all the time.

[00:19:34] And the, you could say the, the logic for True Delta always looks the same and always how you query it looks the same. It’s always the same. So, it was very easy to set up good indexes. So you even then the tables became very deep. It still worked very well because the indexes was so well defined and the Pattern was reusable all the time.

[00:19:57] So it just worked, so to [00:20:00] speak. What I’ve seen on the other side is that when we moved up into the cloud area where indexes is not so important anymore or important, but they aren’t, doesn’t exist, so to speak, true indexes but they are column narrative, right? Often snowflake and if you look at BigQuery, so on and so forth it’s really efficient because it doesn’t need to store everything, right?

[00:20:24] It’s just store one unique value. It is a really good what do you call it when you press data together? yeah. so focal works very well with those formulas for doing that. So it becomes very small in storage instead because it doesn’t need to store each and every value in the engine, so to speak.

[00:20:46] Shane: And then from what you’ve said the focal framework and the modeling Pattern has basically described the types that sit in those name peer keys. So there’s a dictionary or a shopping list of types [00:21:00] that, that you need because they’ve been proven over the last, what is it, 20, 30 years.

[00:21:05] So you’ve probably gone through and figured out all the types that are needed, pretty much in, in most examples. And you get those kind of free when you are using the methodology or the patterns.

[00:21:14] Patrik: Yeah. You always have to keep in mind that the focal today is the data layer, so to speak, where the data is size is very crucial. See, machine driven is not for the human eye. It’s been very specialized to be fully met data oriented. So there you can’t really look at look at the table in, in the focal data model and understand what kind of data is there, you need to add them.

[00:21:41] Then metadata layer or knowledge layer upon it, that is directly related then. But but everything describes what is there, how it is stored, and everything is up in that layer. So that layer is the, the heart of the solution, so to speak. The physical layer is more just [00:22:00] how do we make this reusable?

[00:22:01] How do we not care about what the human think, care about good engineering sort of principles? And that has been very effective. And now when AI comes along, it’s all of a sudden it’s shit. the machine understands it without a problem, but a human thought was hard for a machine.

[00:22:18] It seems like the AI goes like just, okay that’s not hard.

[00:22:22] Shane: Because it’s patent based, right?

[00:22:24] Patrik: Data, everything is described. So he just reads the metadata layer and then he understands what’s in the data layer. And then you can just ask him questions

[00:22:33] Shane: and there’s no exceptions

[00:22:34] because it’s meta data driven. . There’s no, oh Bob came and did that one and he cheated and Yeah. That’s an exception that you’ve gotta deal with. .

[00:22:41] Patrik: No. So that’s, yeah, it’s fully patterned. So it’s, it is, as you say, there is no exceptions that it has to take into consideration

[00:22:48] Shane: So one of the negatives that you often hear around modeling patterns like data vault is consuming, the data is hard.

[00:22:56] There’s a lot more tables, a lot more joins, even though it’s pattern [00:23:00] based, people that try and query it directly have a little bit more work. And what we always say is your consume layer should be automatically generated,

[00:23:07] it’s a pattern based design pattern. So creating the consumption patterns can be automated and you can use one big table with. Dimensional modeling or whatever, anything that flavor that you prefer to query the data. But those patterns for querying, it should be automated.

[00:23:23] I’m gonna assume that for focal, it’s the same that you wouldn’t expect a human with Tableau to go into the core focal data model and try and figure out how to query it.

[00:23:32] Patrik: No, it’s certainly not. No. It’s the focal framework itself creates, automatically creates views more or less like anchor does. So it you as a human never reads the physical tables. You go through the views, which will pivot the data into this is the customer view.

[00:23:50] This is a customer table. It looks like a normal table. When you look at it with all the column names and stuff like that underneath, it’s a totally different thing. But for a human, you as [00:24:00] go in and. Access the views and it looks like a normal table. So that’s how we have abstracted the complexity.

[00:24:07] So you don’t work against the physical the machine does that. You just read the views and you don’t have to care how complex or how strange or abstract it’s underneath really. So that’s more or less how we work with it. You don’t go in and hand code a dimension from the physical tables?

[00:24:25] No.

[00:24:25] Shane: And that’s part of the framework. When you talk about your metadata repository, having 80 tables, that’s where you’re gonna define these keys. These name peer keys, the relationships, and then it’s gonna generate the physical structure under the covers using the focal modeling technique. And then it’s gonna also generate the views that you need to make that data useful

[00:24:47] and usable when you query it.

[00:24:49] Yeah.

[00:24:49] Patrik: Yeah.

[00:24:50] Shane: And that’s the value of the framework over and above the modeling pattern, correct.

[00:24:55] Patrik: Yeah. I think I, I agree that most of them, both from anchor and data [00:25:00] vault and focal and ensemble models, it is harder to write queries against it. You can’t really dodge that’s how it is, but there is a reason for it, and that’s what people forget. So you can’t say it’s just as easy as a dimensional model.

[00:25:16] No, it isn’t. But there is a reason why the ensemble models looks and works like they do, which from an architectural perspective and longevity of the system and changeability and stuff like that. these layering ideas or have layers, is that you have a layer which is specialized on a. Specific area and use the modeling technique that support what you need to do in that layer. Have another layer that makes it easier to do other things, but don’t suboptimize the layer by trying to do everything in one layer. you have to have sort of a system design mind to understand when and where to use things.

[00:25:56] Shane: Yeah, that idea of a layered data architecture and use the modeling techniques [00:26:00] that have the most value for the problem you’re trying to solve. And then automate them and standardize them so they’re part of your architecture. Not little things bolted on the end that, people forget that for this one we somehow use something that was slightly different.

[00:26:13] Last question around that though. The focal framework itself, it’s open source, correct.

[00:26:17] Patrik: at this point there is a focal CLI that we are about to release it. It’s a lot of testings that goes round. The focal framework itself is not in GitHub at this point because more or less, it’s just so hard to understand it. So we need to abstract it, give you tools to use it, so to speak, so you don’t have to be a focal expert to use it, because that just takes too much time, more or less

[00:26:45] Shane: So the other thing we were gonna talk about is how Focal helps us with identity resolution. So this idea that, you know, one of the largest problems always is we see the same person in multiple different places and [00:27:00] systems and we want to stitch them together or the same organization.

[00:27:03] And that is a big problem that most people ignore, right? They focus on the simple problems when they bring out new patterns, frameworks, methodologies. They do the nice, customer orders, product simple ones. So let’s deep dive into a identity resolution and how Focal helps us with that.

[00:27:21] Patrik: Okay. Let’s let me set some concepts first around it, on why we’re doing it just for the audience. So when I’m talking about entity resolution or key integrations, I also used to call it is that we have two things. Two concepts, and that’s one that you have something is called an identity, and that is something that is stable, immutable key that represents an instance of data. And that key should be it doesn’t have to be, but it should be unique throughout the whole analytical system. And it’s [00:28:00] identifies a thing on event unique over its whole life cycle. So it’s immutable. So that’s an identity when I call it identity. The other one is to call an identifier, which is a way of identifying a thing in event it has to be unique within its data space. Might not identify the instance as its full lifecycle, but we should strive to find one. So these are two different concepts. The identity often is called surrogate keys is normally represented as a surrogate key in your system. And the identifier is what people sometimes call business identifier or something like that. But it’s good to understand that you should keep those two apart. That there are two different things and that they are there for, to handling things differently. The other thing is what are we trying to do and what do we not want to happen when we do the sort of integration?

[00:28:53] Key integration is that one thing we don’t want to happen is key collision. The idea is that we have two [00:29:00] different systems both holds accounts and they have an account number series on them and we send them into our account focal or hub or whatever. And we use the account number as the identification because it’s the business identifier, but for some reason these numbers are overlapping.

[00:29:17] So we can start getting key collision. That means that. Two different accounts that are two different instances in the real world, start writing their data on each other and changing data back and forth because they are not truly the same account. That’s a key condition area that we want to avoid. The other thing is that we want to achieve key integration. And and that is the thing that we want to, if they are the same thing in two different system, we want them to integrate on the identifier or the key, so to speak. And the, this is the hard part when working with this is to understand. [00:30:00] How unique should I do something and make something and still make it integrate? So let’s say you could have this account thing and you go okay, but I set source system ID in front of it, and then the account numbers will be unique. Yeah. And then you have two different customer systems and the same customer lives in both system, but you just go, wow.

[00:30:21] But we set the same system ID in front of the customer ID, and then it’s unique, but then we all of a sudden don’t get the key integration that we want to happen because we have made it too unique for how it works in the it. So this work, as you said, this work with integration and entity resolution is. It’s really hard work. It’s not something that just comes along freely is you have to put time and effort to understand the IT landscape, the business processes. How does data move? Where is the unique instances of things and where do they live and how do they move in the IT landscape [00:31:00] to be able to understand how should I build my identifier, because this is what we talk about, so we need to create an identifier. Some people talk about business identifiers. for me, that doesn’t ring correctly because I talk about I normally want to say integration identifier because for me, business identifier is something that the business can use to identify things, but it doesn’t have to be unique for their work, in their daily work.

[00:31:28] So you could have something that is true in one point in time, that it’s unique, but it can change. And it’s still, the business uses it because it works from an operational perspective. But from a analytical system where we historize things and keep track over time how things changes, we need identifiers that stay true throughout the life cycle as much as possible. So there is always that kind of work we need to do to understand how [00:32:00] do we create the identifiers so they will truly integrate and not end up with a key coalition. So it’s hard work, more or less

[00:32:10] Shane: There’s, there was a lot in there. So let me unpack it and try and map it to some patterns that I understand, and I’m probably gonna get them wrong. So I’ll pause after each one to make sure that I’ve got it. So the idea of key collision, so let’s use let’s use accounts that, in system one, we have an account, which the account sequencing numbering is 1, 2, 3, 4,

[00:32:32] so it’s a number. And then in system two, we have a separate series where it’s A, B, C, and based on that, if both of those are held true forever, we know there is no key collision, we can trust that as a unique. Account record, and we’d never collide. But if system one is 1, 2, 3, 4, and system two is 1, 2, 3, 4, and they’re not integrated anywhere else we know that potentially we’ll see that 1, 2, 3, 4 come through from each system, but for a different [00:33:00] account, account for Bob versus account for Shane or, account for savings versus an account for investment.

[00:33:07] And so our natural way of dealing with that is to put source identifiers at the front. Yeah. So system 1, 1, 2, 3, 4, system 2, 1, 2, 3, 4, and now we know we will never get a collision. So that’s the first Pattern that we typically do is that correct?

[00:33:22] Patrik: Yes.

[00:33:22] Shane: And that source and account combination, is that what you call an identifier?

[00:33:30] Patrik: Yeah, it could be, but it could be multiple. You could use multiple columns to create an identifier. So it’s depending on your physical realization of how you want to do it, you could keep the columns apart, but in the focal we, at least we concatenate all the different columns we need to create a unique string of data.

[00:33:48] Shane: Yeah, but that’s the identifier because we can identify that was true at that time. It’s a fact. We can see the data there. We didn’t make it up. We didn’t infer it. It’s a fact. Okay. And then we [00:34:00] know that if we want to say, this account system one is the same as this account system two.

[00:34:06] And now we don’t have an identifier that is exactly the same identifier. We have to infer that they are the same. And it’s like a surrogate key in my head. We’re trying to find this unique identity. I’m just trying to get my language right, the unique identity to say that actually is the one thing that looks differently elsewhere and it’s the same.

[00:34:26] Is that right? So we’re creating this unique identity ID that then everything’s bound to if we can match it. Is that right?

[00:34:34] Patrik: Yeah, I mean it’s the same for data vault people that you have the business key and then you have the Sid, right? It’s the same principle. You have an identification identifier, which is related to a surrogate key. So that’s just to keep them apart, so to speak. That’s what they have two different sort of workings in the system.

[00:34:54] Shane: And so obviously creating that identity ID, because that becomes this golden [00:35:00] id, It’s this immutable thing that we bind everything to when we find something that’s related in the future. And we can see changes over time, so if my account number goes from 1, 2, 3, 4 to 5, 6, 7, 8, 9 a year later and we bind that back to the identity Id, you still know it was my account.

[00:35:18] it’s identifier changed and sometimes happens, especially when we’re using email, When we use email addresses the identifier or name, we know that those identifier IDs will change over time and they cause us a major problem. So what we’re trying to do is find that, that golden identity ID for the thing the event, the person, people place or those kind of concepts who does what’s all those kind of concepts that we wanna manage.

[00:35:42] Is that right?

[00:35:43] Patrik: Yeah. Yeah.

[00:35:44] Shane: Okay. Lots of people talk about that and then they leave it to you to figure out how to do it. So take me through the next bit.

[00:35:52] Patrik: Yeah. So since we are talking about patterns let’s talk about how, first, how data vault does it and Anchor does it just to [00:36:00] make it see, and then we go down to how focal does it. So in the data vault, we already touched on it, right? You, we have the hub, and in the hub we have a business key and we have a c and a surrogate key.

[00:36:11] Now I know a lot of people have discontinued use of surrogate keys in, in the data vault area because they’re using hash keys. And there is a reason for that, and I agree that data vault should do that because they do not have the ability to relate multiple identifiers to one identity because it’s hardwired in the hub. So if it is a one-to-one relationship between the one identifier and it’s key. There is no really architectural functionality for the surrogate key. You can just as well just hash the identifier and then you have a key that always works through. Instead, when the reality hits us in the face, we all of a sudden understand that, oh, this one can be identified multiple different ways. So you could have [00:37:00] a customer ID and you have a social security number, and both are used to be identifying you in different systems and in data vault. if you use the customer ID and it has its surity or you just hash it, that becomes one instance of the customer. You do the social security number, it becomes another instance of the customer. Or then they use a same as link that says This ID is the same as this. ID from an instance perspective. So that’s how they Do it in their solution. In anchor, they point at an attribute table. So you have, in the anchor, you only have the surrogate key, and then they point at the table and one of the attribute tables, and you say, this one is also used as an identifier.

[00:37:44] So their code uses that for lookups. Okay, I, I have my identifier, I check there and then I can pick up from the anchor, I can pick up the surrogate key, I get the surrogate key, and then you have the pickup and they have the ability that they can choose, okay we want [00:38:00] another attribute as well as part we could be able to identify with social security number as well.

[00:38:06] And then they can point in the table that also social security number. The issue is that they have to generate the code or change the code because it has no generic solution around it. So it’s. The code is changing. It’s is a destructive change in the architecture. When you need to add a new way of identifying things, then you get a destructive change in your code.

[00:38:28] So you need to change your code. In Foco, we have what we call an idea for a table, which is to identify a table, which sole purpose is to integrate, to be used, to create integration and hold the relationship between the identifier and the key that are created. And how this works is no magic in the focal framework.

[00:38:53] As of today, there is no fuzzy logic solution or things like that. It’s only [00:39:00] purpose and only functionality is that you can, connect multiple identifiers to one key if you have them in the same table. More or less. So let’s say you have a table where you have your customer ID in the table, and you also have the social security number in the table, probably from the source, right? You can then point out and say, the customer Id identifies the customer, and the social security number identifies the customer. How do we know it’s true? They are on the same row in the table, right? So we know by how the built that they belong together. There’s no magic really.

[00:39:36] It’s just we can just point out multiple things and those different columns that we use to identify becomes their own identifier stream. Now, in the identifier table, there is, I’m not gonna talk about the whole, because there’s a lot of technical things in it as well, but from a conceptual point of view. You have one column that is [00:40:00] called identifier, and you have one column that is called key. And the thing is that each row represents the relationship between an identifier and a key. Now in the code, if you point out social security number and you point out the customer ID as identifying the identifier code will then put both of those into the same SQL, and then he will check for each of those. He will do multiple lookups, one for each. We will look up with the string for the customer ready, we’ll check against the identifier table. Do I have a key for this, yes or no? Okay. And then he checks for the social secured number. Do I have a key or not? In my identifier table, if there is no key, new key is created. Both of these becomes pivotal to its own roles. So the social security number becomes related to customer [00:41:00] key one. the identity, the customer id, key one, and the social security number will also be aligned to the customer key one, so to speak, this surrogate key. So now they are aligned. The, one of the interesting point is, let’s say we also have something else I don’t come up with a good example, but some kind of other identification of a customer, maybe in the same table, but could also be in another table. So in another table, in another system, we have the, let’s say the social security number, and they have a local identification of the customer as well in another color.

[00:41:36] And they want, they need that because they have other systems that. Works with that identifier in the ILE landscape and not the social security number. so the thing is that when they map that system and say it’s a social security number and this local ID that identifies a customer from this source, when that goes in and checks the data in the identifier, he will check the [00:42:00] social security number and he will pick up a key number one, right? And then he will go with a local and he will find no key. But since he already found the key, he will propagate that key to the local ID and that will become a new row in the identifier table. The row that represents the social security numbers, relationship to the key is not multiplied, it’s just exists only one instance independently.

[00:42:26] How many system that uses it. So it just becomes one instance, one row per. Unique string, so to speak. So all of a sudden then you have three different identifiers related to the same key. Now the fascinating thing about this is that since the table is just identifier and key, and when you instantiate this Pattern, you do it for the concept.

[00:42:51] So you have customer identifier and then the table will call it customer identifier. And in the column there will be a customer, [00:43:00] IDFR column and customer key. And do you do it for product? Same thing. Product, ident. so from a patent base there is identifier, key idea, four key, that’s it. And then you just add what kind of entity you want to use this identification for. And the code is exactly the same for independently or which entity. You can also say independently of what industry you do it for, it doesn’t change. It is the same. So the table and the code works the same way independently or who you want to do identification on. So this Pattern is for me, it’s not really a focal Pattern. It is part of the focal solution. But from my point of view, you can use this for any modeling technique really. If you want to be able to create a surrogate key, you just create this look up table, so to speak. And you can have it for dimension modeling. You can have it for data vault if you dare to break the hub thinking. [00:44:00] And you can do it, you could really do it for anchor as well or yeah, whatever modeling method you want, as long as you, you like some sort of, I need to be able to have multiple ways of identifying things. But I don’t want duplicates in my data. I just want one surrogate key for it. It’s totally reusable for anyone to use.

[00:44:19] Really?

[00:44:20] Shane: So let me play it back to make sure I got it right and then we’ll jump into a couple questions. So what we’re doing is we’re holding a table, let’s call it table. And in that table there is a golden unique identity, something we are generating that we, that never changes to say we, we have an instance of a thing, and then we are binding that to identifiers we see.

[00:44:41] So in your example, I might bind it to an email address and a social security number and a passport number and a driver’s license number because each one of those are identifiers of a person.

[00:44:53] So what we end up with is we end up, in that scenario, we would end up with the golden key that we’ve [00:45:00] generated four times four rows, and then each of the related identifiers, social security number, email address, license, passport number, and the benefit of focal is.

[00:45:11] The framework does that automatically for you, so if you go and say, that social security number is the same as related to that driver’s license number because you have a table that tells you that, then the focal framework is gonna go and do an upsert into that golden table.

[00:45:28] If it hasn’t seen it before or say we’ve seen it before, I know that relationship exists, we’re okay. Am I right so far?

[00:45:34] Patrik: Yes.

[00:45:35] Shane: And so the benefit of that is if I look at something like data vault where we have same as links. I’ve gotta code fire the code. I’ve gotta build my old framework to go, how do I do these relationships?

[00:45:46] And frankly, I’m not so sure, but probably the same. And then, so that’s the first part of the problem, how do we codify or patternise the identifications of these keys without having to manually do it? And then the second one is [00:46:00] how do we then use that in all our queries to be able to let me query any data with any of those identifiers that is valid at the point in time.

[00:46:09] And that’s the bit that you are saying in the focal framework. You still you haven’t patternised or solved that automatically yet that still requires some human to do that query side of using that data, or is that something you have solved?

[00:46:23] Patrik: It’s depending on what, because, ‘cause from a focal perspective we more or less never use the identifiers for anything else than creating the relationship to the key, so to speak, if there is multiple ways of doing it, multiple identifiers. So I will talk about that a little bit later. But so the identify table is a technical table. It’s not about using as a business representation of things. , You have your key, that is a unique instance or something if you want to search or do a query on the social security number. Then the social security number also is stored in the descriptive data [00:47:00] as its own. It’s its own representation. So you search there for a social security number, and we have said that there is a, there’s a separation of concern here. We should have, the identifiers should be able to live and develop however they will need to be to achieve integration. While the business identifiers should not be contaminated by that. So the business identifiers is held on their own in its free form, it’s clear. So if we do a concatenation in the identifier table, it’s not done in the, on the business identifier. Social security number. Social security number if we need to say. Sweden social secure number, and Russia social for some reason. Whatever the number the idea or the, in the scripted table, it’s just the social secure number clean, so to speak, as a business from a business perspective, which means you could get up duplicates when you search for that. You could get two rows, one Russian and one Swedish guy. [00:48:00] But yeah, that’s how it works. Then you just have to choose what, whatever you want to have it, what do you want to see? So that’s a separation of concern in the architecture the identifier table is a technical table should not be accessed by people, so to speak, because the key itself is really what hold binds it all together

[00:48:19] Shane: And then again, because it’s pattern based, you’d assume that LMS would understand the role that technical identifier table plays because it’ll see it being used the same way every time, and therefore it should use it more effectively than we’ve got bespoke tables or other things that are used in different ways.

[00:48:38] And then I think you said it doesn’t solve the whole fuzzy matching problem, but again, there’s patterns and code bases out there that allow you to do it. So all you’d be doing is injecting the fuzzy matching logic. To determine what rows you need to insert or check for in the identifier table.

[00:48:57] Right.

[00:48:57] Patrik: No, there is no, you can do it, [00:49:00] but the code as it is today is just you have it on the same row. They mean the same thing, two different columns. Yeah. So there is no magic. I usually say that there is no magic. It’s just.

[00:49:11] Shane: Okay. we talked about three object types. We talked about a thing that holds the keys, a table that holds the descriptions or the attributes as name, peer values, and then something that holds the relationships of the keys, this identifier table that holds this unique identifier and the system identifiers.

[00:49:29] Is that a different object type or is that reusing one of the, just to get, I got a funny feeling. I know where you’re gonna go with it, but Yeah, it, is it a different object type or is it just reusing one of those? A couple of those three.

[00:49:41] Patrik: No, it’s an, it is a different object

[00:49:43] type In the focal. We have the focal, there is a focal table that holds the surrogate key Exactly. As a anchor key anchored us. It’s only a surrogate. As I said, the identified table is a technical table to achieve integration on the [00:50:00] key, so to speak. So it’s its own type.

[00:50:02] But normally when I talk to Hans, we talk about, he says, yeah. The concept is the key, even if it’s two tables.

[00:50:10] Shane: so like we have with Data Vault where we have, HALs and SALs and these other things that we need to do from a technical point of view that break the, there’s only three simple type of tables you need. I’m assuming within focal, there are other instances of that where there are other technical tables or objects that we need for certain things, or is this the only one?

[00:50:31] Patrik: this is the only one

[00:50:33] I mean from a physical data modeling perspective, but then you have the whole metadata layer, but that’s another animal, so to speak.

[00:50:40] Shane: then we get to the problem of point in time reporting. And so if I changed my email address as one of my identifiers, at a certain point in time, it’s gonna solve that problem anyway because I’m now going to automatically get another row in that identifier table to say, against this golden [00:51:00] key, I’ve got another identifier.

[00:51:03] And then in my queries I’ll be able to use that to say, was it this or was it that? Okay. Yeah. I’m just trying to think around the whole, changing of of identifiers and then how that goes back.

[00:51:14] Patrik: I mean there is always that There is, yeah, there is no magic. So if you don’t have a stable identifier and you change your email address, you only have email addresses an identifier, and then you change the email address and you don’t have a column that says this was the old email address or something. It will create a new key because he doesn’t know about the previous email address. but if you have a social security number and an email address, the social security number will be stable and you change the email address. It’ll add the new email address to the same key.

[00:51:48] Shane: But if I had a table that held a change of email addresses an event, and so that table held previous email and new email,

[00:51:57] then it should pick it up against that golden key that [00:52:00] the email address had changed. And so there’s just another identifier against that key. Correct?

[00:52:04] Patrik: Yes. Then it would do that

[00:52:06] the identifier table doesn’t care which column you use. So it’s just a generic column of identifier. You just have a string of values. That one is one you check. There is of course the ability to going up to the metadata layer and see. What attributes in the source was used to concatenate this string. So it exists up there, but in the physical from a code perspective, it’s just, here’s a string. Does that string exist? When it checks, when it do a looks up, it doesn’t care if it’s what column you use to create that string.

[00:52:39] Shane: And again, it goes back to that idea that the metadata or the context is held outside of the physical data, so if you’re using an LLM you need to make sure that both are exposed because if you give a physical focal data model with its , physical data to an LLM, it’s gonna struggle to understand the context of how that works.

[00:52:57] Patrik: Yes. Very much yeah.

[00:52:59] Shane: Okay, that [00:53:00] makes sense.

[00:53:00] Patrik: I can just do another thing around this because when focal started and for the up to, yeah, 2014 or something like that, and 15, the, or even later. But anyway the identifier table was not optional. It was always, you always have a surrogate key. That was how it worked. But when we moved into the cloud area, we the cost of doing a lookup when you have a lot of rows costs every time. So if you have an entity that does not have multiple identifiers, why should you use a lookup? Why do then you can just as well hash the thing or use the natural key, right? So in the focal framework today the multi identifier functionality is the optional. You always start with a single identifier point of view. So you set up your entities and if you don’t have any , multiple identifier at this point, you just run it as like data vault [00:54:00] or something. It’s just natural or hash keys. But the thing is then that when you come to a point in your development that you all of a sudden know, okay, we need both social security number and email address to make this work. Then you can, in the focal framework, then just point at the entity and say, now this entity has multi identifier entity. And then that gives you the ability to map multiple identifiers to that as you do your mapping and say, this identifies and this identifies. And then the framework will also migrate your data for that entity. So it will. Propagate the new surrogate key that is generated. Why when you deploy, before you load it deploys and take all the wall data, integrate it to the surrogate key that you have created and populates all the tables that are involved. this is also when you think about it from a ensemble modeling perspective, is you have an entity and it has three relationships, right? And you point at that entity and says, this one is now [00:55:00] multi identifier. So it has to run on a surrogate key. It also has to migrate the data in all the relationships. And once again, since everything is metadata driven and pattern based, once we figured out the script on how to migrate from the normal key or natural key to surrogate key, it works for every entity. It was just a one go. You needed to figure out how do you do this? How do we move what? What kind of things do we need to do?

[00:55:27] And once that is done, it was repeatable. So it became repeatable. So you don’t have to think about it either. Oh, how should I migrate then? No, it’s just point that it say this is multi identifier now, and then it would migrate itself.

[00:55:40] Shane: And this is one of the value of these frameworks that are put on top of these modeling patterns is that they have been built out over so many years, decades, they’ve seen lots of edge cases, and the framework then gets updated with those edge cases and then. That code gets tested. That [00:56:00] pattern gets tested at time and time again by many people and many organizations and many industries.

[00:56:05] And it pretty much becomes bulletproof versus a senior engineer writing it for the first time or, go and find a blo of code from somebody, and then obviously you’ve got all the get repos, you go and harvest, and now the new one is your vibe code on the LLM.

[00:56:18] But none of that is proven versus you go to a framework that’s been tested over decades and you know that those, each cases where they’ve been dealt with have been proven. One of the questions I always have is, when do I ever trust an organization who tells me they don’t have a secondary identifier?

[00:56:35] And my answer is never, because they’ll tell you they’ve only got one. And then, three to six months later another one will turn up in the data and they’ll be like oh yeah, sorry, I forgot about that. And so for me, yeah, I probably would just go with multi identifier in the beginning and wear the cost of the surrogate lookup

[00:56:52] Patrik: at the same time, you don’t need to when you have the migration script, so you just, you run it as single until you need [00:57:00] multi. Just automatically migrates your data over to multi identifier,

[00:57:03] and then it’s just for that, and then it’s just for that

[00:57:06] entity also.

[00:57:07] Shane: if I’ve got billions of rows.

[00:57:09] Patrik: This is something we have seen when we worked in BigQuery and Snowflake. This is really important from a cost perspective, really. Because when it’s huge data sets, it costs a lot to do those lookups, even if they’re really efficient. So still costs data especially since there are no true indexes and stuff like that on these platforms. So you, even if you do a very good setup for. For the order of data and petitioning of data and what, can do in these areas. It still costs money. so that was the decision why we said, we know that a lot of entities will get multiple identify over time, but until they do not need it, we can run it on single identifier code, so to speak.

[00:57:56] Shane: Yeah. And it’s that idea of optimizing where it makes sense.

[00:57:59] You know, [00:58:00] 1, 1 1 of the things we do is, we will for smaller volumes of data, when we see an event, we will just do an upsert where we see larger volumes of data and we know some things are gonna happen. We might do a petitioning replace, but the system takes care of it,

[00:58:15] because it. You optimize it for, okay, if it looks like this, do that. Because we know over time that’s gonna save us money. And if it looks like that, do this and we can switch between if we need to. But it’s that baking in optimize at the right time at the right place. And don’t be afraid to change it later if you can change it safely.

[00:58:32] Yeah. Whereas in the past we would never touch anything like that. You say to somebody, oh, can you go into your dimensional data warehouse and just rekey all the surrogates for me? Yeah. That’s gonna be a dangerous change, right? Because yeah, it hasn’t been designed to allow you to do that.

[00:58:48] Patrik: Oh, exactly.

[00:58:50] That’s

[00:58:51] Shane: excellent. All. If people want to get hold of you, if they wanna understand more about focal, if they want to learn more about the modeling [00:59:00] patterns and the framework, where do they go?

[00:59:03] Patrik: I would say you can either go to LinkedIn to my profile, or you can go to Daana dev, which is the project where we’re working with with a focal framework creating CI around it so we can, you can easily test it.

[00:59:20] Shane: And give me an example of what you mean by building a CLI out for this. What are you looking to achieve?

[00:59:25] Patrik: But it’s more like. Doing an abstractional layer of of the focal framework since it is complex in its sort of design to make it easy for people to use. So the Daana CLI works with YAML files where you set up your models and your mappings and then that get processed into the framework and then the framework creates the code and stuff like that. So it’s more like just ease of use. So instead of trying to figure out how you’re gonna use the focal framework, there is already a command line interface where you can [01:00:00] do all the things you want to do with it, so to speak.

[01:00:04] Shane: The metadata tables and the physical tables, they’re technology agnostic, right? So you can run those on any of the data platforms or data technologies.

[01:00:14] Patrik: Yeah, and not, there’s always these small quirks in SQL that the different platforms uses. the interesting point is a focal framework have had Oracle sgl, L Server snowflake, BigQuery Postgres. But we are now in the middle of our sort of re visiting the architecture. So now we have pulled us down to Postgres and BigQuery. Where we start to build up all the different platforms again. But it’s still certain parts are platform specific. But since it is metadata driven, the it’s not that hard. Once you get it to work, it’s just shift.

[01:00:58] And now with ai, [01:01:00] LMS also, they can easily translate your patterns. So you can go now I want this to work in Snowflake and just show them all the patterns, and then they just transform it over. You have to test it, of course, because it’s lms, you don’t know what they’re gonna do. But it goes very quickly to, to move to a new platform at this point.

[01:01:21] Shane: But if somebody was starting out and they, wanted to reduce the uncertainty of things they need to learn, then you’re saying Postgres and BigQuery is probably the safest way to start out, to learn on it

[01:01:30] Patrik: now you can go in for the homepage, that’s Daana dev and sign up for be a better be tester and try it out. And then we have building up a community around it because people was to be able to contribute and stuff like that.

[01:01:47] So yeah, we see where it goes for me it’s really fun because all of a sudden the focal framework becomes accessible for people that, that doesn’t want to put the energy or effort trying to [01:02:00] understand underneath the hood. And that’s really fascinating, I would say. So it’s really fun.

[01:02:06] Shane: And even if we look at something like data Vault, which is probably a lot more well known across the world even then it’s still, you’ve gotta learn some stuff. So anything we can do to remove that complexity and make it for people to, simple for them to use the patterns.

[01:02:19] And that’s a good thing in my view, right? But it takes time and effort to make something that’s complex under the covers simple to use and execute and get that value of automation and. Optimization out of it.

[01:02:31] Patrik: Yeah. Yeah, exactly.

[01:02:33] Shane: Excellent. Alright, thank you for that. It’s been a great chat. I now know a lot more around focal than I did a while ago.

[01:02:39] So thank you for that and I hope everybody has a simply magical day.

[01:02:44] Patrik: Same. Have a nice day everyone.

«oo»

Struggling to gather data requirements and constantly hearing the conversation above?

Want to learn how to capture data and information requirements in a repeatable way so stakeholders love them and data teams can build from them, by using the Information Product Canvas.

Have I got the book for you!

Start your journey to a new Agile Data Way of Working.

Buy the Agile Data Guide now!

«oo»

Listen

View MindMap

Read AI Summary

Read Transcript

Listen

Listen / watch over on Youtube:

https://youtu.be/GjSD1pahPRE?si=1yg8gNsCb_4PvJWN

Google NotebookLM Mindmap

Google NoteBookLM Briefing

Executive Summary

This briefing document synthesizes key insights from a strategic discussion regarding the definition, implementation, and governance of data foundations. The analysis shifts the focus from purely technological solutions to a holistic framework encompassing management, organizational culture, and business context.

Critical Takeaways:

• Definition of Success: A data foundation must be predicated on measurable business outcomes—specifically “up and down arrows” such as reduced time-to-decision—rather than the implementation of tools.

• Governance vs. Management: While debate exists on whether governance should be an integrated part of management or a separate “legislative” body, there is a consensus that current data governance often lacks the power to enforce rules effectively.

• The Three Ps: Effective foundations rely on Principles (cultural preferences), Policies (immutable rules), and Patterns (reusable templates).

• Context-First Approach: Metadata should no longer be treated as an “exhaust” of system activity. Instead, “Context” (Business, Structural, Operational, and Agent) should be defined upfront to hydrate and drive data systems.

• Organizational Culture: Organizations must identify as either Pioneers (experiment-led) or Town Builders (process-led) to align their data strategy with their operational reality.

--------------------------------------------------------------------------------

1. Defining the Data Foundation

A data foundation is often misconstrued as a collection of technology stacks (cloud databases, ETL tools, etc.). In practice, it is a multifaceted subdomain of data management that integrates architecture, platforms, ways of working, and governance.

The Problem of Silos

The separation of data governance groups from data teams creates significant operational friction. Management is the overarching discipline; if data is not being governed, it is not being managed. A foundation must bridge the gap between “artisan” manual work and “automated” factory processes.

--------------------------------------------------------------------------------

2. Measurement and Strategic Alignment

Before investing in technology, organizations must answer: What is the definition of success?

• Success Metrics: Success should be measured by tangible indicators (up/down arrows). For example:

  • Down Arrow: Reducing the time taken to make a decision.

  • Up Arrow: Increasing the number of business questions answered.

• Business Questions vs. Decisions: Framing requirements as “decisions” often leads to vague answers. Stakeholders provide higher-quality requirements when asked for “business questions” (e.g., “How many customers do we have in this region?”).

• The Strategy Story: Data strategy is simply the story of how data will execute the broader business strategy. Without a clear business roadmap, a data strategy becomes an empty “PowerPoint deck” without execution value.

--------------------------------------------------------------------------------

3. Structural Frameworks: Value Streams and Factories

Data management functions through two concurrent value streams that must be balanced to avoid “building a plane while flying it.”

The Stakeholder Value Stream (Product Thinking)

This follows the progression from problem identification to value realization:

1. Problem Statement & Ideation: Defining the issue.

2. Discovery & Prioritization: Identifying the most viable solution.

3. Design, Build, & Test: Creating the product.

4. Deployment & Maintenance: Ensuring long-term value.

The Data Factory (Engineering Process)

This focuses on the internal mechanics of the data team:

• Identifying data layers (Staging, Raw, etc.).

• Establishing hand-off points between team members.

• Automating testing and deployment to reduce “blockers” in the factory line.

--------------------------------------------------------------------------------

4. The Governance Framework: Principles, Policies, and Patterns

A foundational blueprint should be structured around three tiers of guidance:

The Power Gap: A major critique of modern data governance is its lack of empowerment. Unlike financial governance (overseen by a CFO with auditing power), data governance often lacks the authority to “shut down systems” or “fire people” for policy violations, rendering many policies as mere “wishes.”

--------------------------------------------------------------------------------

5. Cultural Archetypes: Pioneers vs. Town Builders

The implementation of a data foundation depends on the organization’s cultural starting point:

• Pioneers: These organizations prioritize experimentation. They go out, settle new “land” (use cases), and see what works. Governance in this model is iterative, hardening into policy only after successful patterns are discovered.

• Town Builders: These organizations prefer to build the infrastructure (the “town”) first. They establish the roads, schools, and zoning (rules) before inviting residents (users). This requires heavy upfront design but provides immediate structure.

Failure occurs when an organization’s stated culture (e.g., “we are pioneers”) does not match its processes (e.g., “we have heavy, slow approval gates”).

--------------------------------------------------------------------------------

6. The Shift to “Context-First” Data Environments

Modern foundations are moving away from treating metadata as a byproduct (exhaust) of technical processes. Instead, they embrace a Context-First model where context hydrates the system.

Four Tiers of Context

1. Business Context: Definitions of entities like “Customer” or “Product” that remain stable regardless of technology.

2. Structural Context: The layout of tables, fields, and system architectures.

3. Operational Context: Tracking who accessed what data, query performance, and decision logs.

4. Agent Context: The prompts and skills provided to AI/LLMs to guide their behavior.

Data Contracts

A data contract is a formal agreement between data producers (e.g., software engineers) and consumers (e.g., data engineers) regarding schema, frequency, and quality. Governance teams should mandate these contracts to ensure that data moving from operational systems to platforms is audited and reliable.

--------------------------------------------------------------------------------

7. The Role of Modeling and AI

As the industry moves from the “Decision Era” to the “Action Era” (automated actions driven by AI), the importance of conceptual modeling increases.

• Conceptual Modeling as Foundation: The conceptual model is a representation of business behavior, not data storage. It should be defined without looking at source systems.

• AI and Metadata: AI (LLMs) can “vibe code” applications and replicate front-end structures, but they struggle with back-end logic and physical modeling if they lack the underlying business context.

• Physical Modeling as “Cattle”: Physical modeling techniques (Data Vault, Star Schema, Big Table) should be treated as disposable. If the conceptual and logical context is strong, the physical implementation should be able to change (”be hydrated”) as technology evolves.

Transcript

Mustafa: Hello, Shane. Hi. How are you?

Shane: Hey. I’m good. I’m good. Good to see you again. Thanks for having me on again.

Mustafa: Pleasure. Pleasure. Thanks for filling up the slot. Actually, one of the guy pulled out. He had some personal issues. Which is fair enough. All of us are senior guys. And we do book it, like, couple of quarters ahead and commitment can come in. Thanks for jumping in and making this flock fill up.

Shane: No problem. Hey, I’m looking forward to it. It’s kind of a subject that’s close to my heart anyway. So, as always, I’m sure to have an opinion.

Mustafa: Yeah. Because last, last week on our panel discussion, we had this session. What is the impact of not having a data foundation? So we all discuss this four or five panelists, and people really lied. This. And people always forget to click like on YouTube. But they always send messages one to one that they really like it. What is the importance of data foundation? But one obvious question start coming in. What is data foundation? Because for different people data foundation are different. Like data governance is data foundation. But data governance is huge. Then data modeling also comes into data Foundation. There are tons of number of data modeling techniques going in. So people are asking what it is. So you said you want to jump in, you want to solve this puzzle. Floor is all yours. Shane, let’s start it. What’s in your mind?

Shane: Yeah, I’m not sure I’ve solved it, but I’ve been working on it for a few decades, and I’ve got a set of what I call patterns and pattern templates. So. And kind of. I think about it like this. So my view, and it comes back to semantics, it’s all part of data management. So our foundational patterns, you know, patterns and governance, our patterns in engineering, our patterns and platform, our patterns and way of working, they all kind of, for me, come under that, that turn data management. And the reason I say that is I don’t like this idea that there’s a separate data governance group. And a separate data team. That silo behavior causes us massive problems. And so for me, it’s, it’s all within the same way of working. We may organizationally have our data team structures slightly different where, you know, people are focused and for scaling reasons and separate pods or groups. But I think governance is part of management. Yeah. And if we’re not governing our data, then we’re not managing it. So that, so that’s the first thing. If I kind of look at it from that lens, and then I look at. Kind of subdomains within data management, you know, we have ones which are around architecture and platform. And so when people say foundation. What I typically hear them talking about is technology foundations. You know, I need cloud analytics database. So I need an ETL tool. And what often happens in an organization is I’ll, I’ll get asked to come in and do some work with them either on a greenfields environment where they’re starting from scratch again. Or they haven’t had a data management capability or platform before. Or a brownfields one where they come in and ask me to do an architecture review. And as an ex technologist, my natural reaction is to get into the technology, right? Oh, that’s what they’ve asked me to do. Come in and look at our architecture, look at our platform, look at our tooling. And what I’ve learned over the decades of doing this is we need to step back and we need to look at some other things as well. We need to look at our way of working. We need to look at what our strategy, our team designers. And so over the years, I’ve kind of been building out this data blueprint template and set of patterns that I use. And what I’ve learned is the first foundational question that I always ask now, regardless of it’s greenfields or it’s brownfields or why I’m being bought in is what is the definition of success? Yeah. So if we’re going to spend this money, right, in terms of tooling and technology and people and change, and even if we’re using open source, let’s not. It’s not tell ourselves it’s free, right? Yes, there may not be a license cost upfront. But the five data people are going to cost you a million dollars in salary for a year if you’re in the US. Right. So there is a cost there. So let’s say we spent that year. We delivered everything we promised. Yeah, we made the changes that we needed to make. What’s the measures of success? Like what’s actually going to happen? And for me, I want to keep them really simple. For me, they’re just up arrows or down arrows. And I’m looking for things like, you know, when I often hear, we want self service. I’m like, well, that’s not a measure of success, right? That’s just a tool. It’s a pattern. So what does that give you? Oh, actually, we get quicker time to decision. Ah, okay. Right. So now what you’re saying is the time taken to make decisions in the organization based on data reduces right down arrow. Good. Or we, we don’t need to talk to the data team. Yeah, cool. That’s, that’s a process thing. What’s the measure of success? Oh, again, we can make decisions quicker. Okay, so we’re still back on that down arrow. Or how do we want to measure we’re successful with that? Okay, well, the number of requests going to the data team is reduced. Yeah. Or maybe we say, actually, it’s not self service is not the tool we’re going to use to deliver that measure of success. Maybe the data team are just going to get better and faster answering questions. Yeah. So the time to decision still goes down. But the tool we use, the thing we measure is how quickly the data team respond and answer that question. Yeah. And that should go up, right? We should be faster and faster. Yeah. Less time or more questions answered. And so that’s the first foundational piece I always think an organization should do. If we make this investment, what actually changes in our organization? What are the measure of success? What are the indicators, the arrows that are even going to go up or down that we can then prove that that investment was worth it? And I see lots of people not doing that foundational piece. What have you seen?

Mustafa: No, absolutely. Absolutely. As you, as I mentioned the other day, right, I have come up with this canvas, which is, of course, influenced by your canvas. Where you had gone. Last time we had one year back, we had this session. Right? So you’re absolutely right. I always ask, first of all, everybody knows what is value, but value is very subjective for me. Everyone want to create a value. There’s no doubt about it. Right? You ask anyone there. Yeah, we have a value. But how this value come up? So you need to have questions, answers based on the answer. You need to have actions. And based on the action, what you will actually have in your hand tangible measurable. That’s. I also look into it. So that’s also my first step. I always start with the canvas. Where I say. Upfront. Okay. Let me put in a different way. Right? I know because as you mentioned, wherever, whenever you go and talk to people, people are normally talking about technology. Normally. They have new generative AI, gente AI. We want to, we want to do something with it. Right. We have a car. There’s a new car in the market you want to buy. The question is why you want to buy a car? Right. So you need to have those things in your hand. So, yes, for me, that Foundation is also you put up your questions and answer the ones you are already be doing. Normally people want to explore new things. I tell them, do what you already are doing and how you can improve it. If you do not know the measure of improving it. Forget about the new things. I start from there once they know. Okay. Okay. We have been doing this with X technology or maybe why technology. And it has been giving us this value, but we need to go next level. Okay, let’s discuss. But what, what you’re achieving. So that’s how I also started.

Shane: Yeah, I, I use a frame in that similar but slightly different maybe. So let me talk through those. And it comes from the struggle that most data teams have. Differentiating between the products they’re building. If they’re a product team or the dashboards they’re building, if they’re not. But the things you’re building, right? The things they’re building using data and information that solve business problems, you know, help drive decisions, actions, outcomes and value. And so they like boxes of cereal right on the, on the shelf. You know, cocoa pops, honey puffs, right? They’re things that people are going to intangibly buy in your organization and use because it has value to their lives. And so we’ve got these products and then we’ve got the foundations of how we work. It’s the factory that builds those products. And data is a funny thing, right? We kind of cross people that have a factory. We want to build everything as a factory, fully automated, but for some reason, the data is always so shitty or the requirements are also new. They actually, we can’t just cookie cut a factory that works as a bunch of machines. So it’s this blend of artisan and automation. And people tend to confuse the foundational work around that, that factory and that way of working with the product we produce. And then what they’ll do is they’ll tend to focus on one or the other. You know, let’s do a founder. I don’t have a time to hear this. There’s still a foundational build. Right? It’s going to spend a 12 months. It’s always 24 to build in the foundation, change the technology, build in a lake. Right. And the problem with that is there’s no value to the organization. Yeah. Like, if we were building a physical factory, then maybe you could go and say, you know, there’s some steel, there’s a roof, there’s some machines. We’re getting there. But the problem with data is we, we don’t show how we’re building out that, that Factory way of working. And so 18 months later, we built our foundational, we’ve built our technology foundation in our factory. But we’ve delivered no value to the organization, and that causes a major problem. What’s the alternative? Well, the alternative is we just build ad hoc products. Yeah, we go in and we just build a product and deliver value. Right. Great. But then we get no automation. They changed our way of working. No ability to scale. And as we add more and more of those products, the business as usual over here to maintaining and just bleeds all the effort out of the team. So what we want to do is we want to balance those two things. Yeah. We want to build out the foundations at the same time we’re delivering value. And that’s hard because if you think of it, what we’re doing is we’re building the plane as we’re flying it. Yeah. And tell me any other industry when we do that, right? It’s kind of, you know, you don’t go, you know, if you’re a surgeon, you don’t go into an operating theater that’s emptying and go, oh, yeah, I’m going to do open bypass heart surgery, but I’m also going to build the machine that pumps the blood around. Right. And design the, the scalpel.

Mustafa: Yeah.

Shane: And so we’ve got to realize that we have to work on both of those things together. And an easy way to think about it is people always talk about people, process culture and Technology. Yeah. So the reason I’m looking left is I’m looking at my blueprint, right? And just going through all the bits that I make sure I fill out because that’s the foundational building blocks or patterns that I use. And so if we think about the people processing technology. The thing we don’t focus on after we’ve got our measures of success is the team design. Right. The version of the operating model. And that’s the next thing I always do is a foundational piece. As I go, what are the personas in our organization? Do we have this idea of data stewards or not? Yeah. And what do they actually do? Do we have a data engineer on how it looks engineer? Like, what’s the language? If we’ve already got a team that we already have. And how do they operate together, what’s their team design? You know, you can get things from team topologies. Are we working in small squads? Are we working in bigger teams? Do they do end to end? Do they do handoffs? And for me, that’s the next foundational piece of work, because by documenting the team design. And that flow of work, we get a foundational piece of visual hints on how the way of working is going to happen. And we can do that relatively quickly and early, right? We don’t have to over bake it. And once we have that, then we go into the next stage, right, which is what other foundational pieces do we need in place? But to come back ways of working and building products that have value, they’re two things that have to happen together. Otherwise we give ourselves in trouble with our stakeholders. Quite rightly, because we’re not earning value.

Mustafa: Yeah, absolutely. I see. Then what, what you are going toward is data governance, because nowadays I’m building a kind of data governance strategy for someone. Right? They ask you to build them this thing. I said, okay, let me build it for you. So their first question was, what framework you will bring in? What tangible thing we will have in our hand? I said, no, you can’t start from the what first. You need to understand what is why then how you’re going to do it and then the technology comes in. They start asking you which tool you will use. I said, stop for tattoo. Don’t talk about tools. First go towards governance. Because, because nowadays we have been discussing this CDO rule, Chief data officer. Right? So, like, what this role will do is. They will create a data strategy with the business. They will create a data governance strategy with themselves and they, of course, did us two words. And then there are data management strategy comes in. These three things comes right. Of course, first is business strategy. Then data strategy, then governance strategy and then data management strategy. Right. So what you are going toward is yes, as I previously just mentioned that I normally tell them, hey, if you want to do something, a data product, let’s take an example, how you are doing things now. People normally forget about it. People think that let’s take an example of generative AI. A tool came in. Hey, let’s bring the tool and let tool give the tool the data and let tool tell us what use cases they can bring it out. Right. That was the reason failure of generative AI. So I told them, no, tell me, even if you are doing manual, get me with your data analyst. What you are doing manually or semi manually or maybe something else. Let’s see that. Which is the actions can be manual or semi manual XYZ and then automated. And from that practices, then personas will come out. Stewards will come out. Processes will come out. So processes has to be predefined processes. It cannot be bring a tool and tool will bring a process. You need to have a process minus technology. What do you think?

Shane: So I think the, there’s a whole lot of bundle in there, right? So let me kind of unpack it.

Mustafa: Yeah.

Shane: So I think one of the points you raised and at some point I kind of missed is one of the first things I’ll do when I’m working with an organization on their foundations, as I will observe. Yeah. Because I want to understand what the current state is because it gives me a whole lot of hints. If it’s a green fields, sorry, if it’s a brownfield environment, right, I’ll observe the data team, obviously of the way they work. I’ll do some workshops with them to understand that. I’ll talk to the stakeholders to see where their real pains are. If it’s green fields, I still do the same thing because even though there’s not a data team and there’s not a data platform, there’s a whole lot of data work, right? It’s just been done in excel and outside.

Mustafa: Somebody else is doing it. Yes.

Shane: Yeah. Yeah. Somebody’s doing the data work, right? Normally. And I will create some very light artifacts around current state.

Mustafa: Yeah.

Shane: Because that helps us understand what those measures of success are. Right. What we have to change, and if we change things, what success if we make those changes looks like. Second pointers, and you kind of mentioned it is. The data strategy is just a story on how we’re going to execute the business strategy. So if somebody comes in and says, do a data strategy, it’s like, well, what’s the business strategy? You know, and if I get a slide that makes no sense to me.

Mustafa: Yeah.

Shane: The conversation in is, well, I don’t understand how we’re going to tell a story around how data is going to achieve that because that doesn’t, I don’t understand the strategy. You know, like what’s measure success for the organization, make more money, reduce costs, reduce risk? Yeah, great. But actually what we’re going to do, right? Like what actions in the next two years are we going to increase market share? Are we going to sell more products? Are we going to reduce cycle time for a patient coming into a hospital? Right? Like what actually is our strategy and our roadmap to achieve it? And from there we can say, right, how does data support it? What I see happen a lot. And again, one of the things I do when I come into an organization do the blueprint, the first thing I ask them is, could you please give me the four other data strategy documents that you’ve had from forever data consulting companies? Because, you know, they’ll be really useful. And they say why? And then my standard joke is I just want to know whether triangles, circles, squares or matrixes of the flavor of the month right now. But that’s not really the reason. The reason is there’s always good information in there. You know, there’s spent some time with some good people to create this document that’s a strategy. And that is valuable. It shortcuts how much work we have to do going forward. We still need to revalidate and make sure it’s not out of date. But the reason they have four other ones is because they got a PowerPoint deck. With some pretty pictures that tells them a strategy and there is no execution plan. There is no roadmap. There is no measures. I’m going to come back to it. No measure success. There’s nothing telling them whether this strategy is working or not and whether they should change it. So based on all that. The other thing I talk about is the blueprints we do should be light. Yeah. So we’ve got this again big design up front problem. We kind of want to see a bit of a blueprint, which is a vision of the future and how we’re going to get there. Measure success team design, architecture, ways of working processes. But we don’t want to spend a year doing that because a year of that work has minimal value. So we’ve got to be able to do these light artifacts, right, and then iterate them as we go forward. And then I’ll come back to the last one, right? Data stewards. So my question there I always have for an organization is, okay, if our team design has a role or a persona called data steward and we know what they’re going to do, right, what, what their job is, what their persona is, what their role is, what their skills are. What changes. And what will happen, right, is I will often hear we get better data quality. And then I’ll go back to the team design and I go, really? So what we got, right, is we got this data governance group, and this is typically a large enterprise. Right? Data governance group over here. Data team over here. Data management across the top. And you’re telling me that the steward is going to change the quality of the data. Because, hell, the ETL developer going in there and writing code in dbt, right? Yeah, they got no accountability on quality. It’s just wrong. Right. And so we’ve got to tie back to that foundational piece of if that metric, if that measure of success of increased data quality, whatever that means improves, like which group of people are responsible in the team design and the way of working for making that change. And that’s how I think about it, right? Like almost like Lego pieces. We kind of. Plugging them in. And the problem is the Lego pieces are always correlated. You know, your team design, your measures exist. They change the way you work. They change the way the factory works. They change the technologies you need. You know, my close-up one the last couple years put in the data catalog. Well, yep. The invisible context for an organization has value. But needs to start talking about, well, what’s the team design for the catalog? Right? What’s the measure of success? Is it number of data assets logged? I’d say that was a weak measure. So you can see how they all kind of, the foundational pieces all kind of interrelate. And you’ve got to be really good at choosing small versions of each of them and then iterating them and see how they then work or don’t work with all the other foundational pieces. And then you kind. Of doing systems thinking, right, you’re tuning the overall system of the organization over time. Does that make sense?

Mustafa: It makes sense. So for me, this is the first part. Right? Because then, so you, what you’re referring is when we start a journey. What we need to do, we need to understand the success factor, as you mentioned, the image. I call it decisions. Nowadays I’ve started talking like four hours. We had operationalization error where this called ERP, sap oltb applications came in. The second era was analysis era. Which was, which brought in data warehousing, task schema, snowflake. Then third era was analytical error. Analytical analyze data mining protection forecast data science stuff. Now the current era, what I define is a decision error. Everything has to start. With a decision. What you want to do. Starting point, what decisions will be impacted with this initiative? And from that everything. Your tickle back, you can, you can bring it back. Okay, I’m going to take a decision. Like buy go on the cloud from on premises to cloud. That’s the one decision. Now what you need, I will find it out for you. So that’s, that’s the starting point foundational that the starting point of a journey. Now let’s go to the next step. Next step will be your, I would like to bring your canvas into it because once we go into artifacts. Do you think, I think you will say that and I want to emphasize on it. And I’m following that as well after my canvas. Where I finalize a product or a requirement. Next, my canvas is your canvas, by the way, which I take to the canvas to the, to the customer and start asking questions. And as in your book, you mentioned the moment you customers start thinking about something else. Stop over there. Whatever is on top of their mind, that is what your initial requirement should be. Tell us about that, that foundation stuff.

Shane: Yeah. So. So it’s interesting when you said the word decision, I naturally took to a business decision using data because I’m a data person, right? So I’m like, which decision do you want to take? And then I naturally went, well, actually, we’re in the era of decision right now, but. We’re accelerating so fast. We’re going to be in the, in the wave of era of action. Next. Right. We’re going to stop talking about decisions and which action needs to be automated. So that’s where I went. Right? And then you qualified it with, which a decision around a change we have to make. And that, for me, is, is the key, right? Is this idea that we’re changing something, changing the way we work? We’re changing the way the team’s designed. We’re changing the technology we use. And so I sat more and more talking about change. As kind of the thing we’re focused on because decisions and actions, as I said, always takes me back to the data world of the product we’re building, not, not our foundational way of working. And then, of course, we get into change management, which has been around for a long, long time. Business process reengineering. And we bring us back patterns that have had value and we kind of lost, but we, we won’t go there. So the way I think about the next step is I think about value streams and factories. And so a value stream is the end-to-end process from when a stakeholder knows they have a problem. And they put their hand up. I have a problem. I need some help and it needs some data to when we deliver something of value that solves that problem. And that is the process that involves things like problem statements, ideation on how we might solve it, discovery or requirements, prioritization. The left hand side of what I call my continuum. Which is, and I’m a great fan of product thinking in that space. Right. So not IT thinking, not projects, not, you know, rescues, none of those things that we, we get from the IT domain. I’m a great fan of pulling out patterns from the product domain on the left hand side of that. So identify the problem, idea how we can solve it, discover which solution is going to be the most viable prioritize which one we’re going to work on next. And then the right hand side of my value stream is around design, build, test and deploy and value. Right. And then maintain. So we kind of lost designing in our, in our data domain a while ago with the modern data stack. Sure as with AI now, we’re going to be bringing it back in because those designs are actually the context or the knowledge that llms need to do the work without us. So, you know, that design, build, deploy, maintain value phase. And so I want, I want the blueprint to have really simple, what we call nodes and links. Like, just think of them as stickies and lines. Like who’s defining the problem space? And if I have a problem as a stakeholder, how do I engage the data team? Or do I not? Do I have a team of ba’s back to the organizational design team design? How do they get into the ideation phase? Right. How do we know how we might solve it? How much time are we going to spend on each one of those all the way through? And then I couple that with the factory process. And that is as the data team. Okay, I understand now which problem I’m going to work on. What’s my process? You know, do I naturally just go and grab the data first? Yeah. Or do I grab, see what data is available and mock up something and give some feedback? Or do I go grab all the data or a thin slice? Yeah. What, once my data layered architecture? You know, which, which layers am I having, which ones are rules I have to comply with, what tooling do I have available? How do we hand off the work between the team members and that factory? How do we test it? Yeah. How do we deploy it? So that all the way from I’m going to touch a bit of data because I know what the problem I’ve got to solve is to I’ve deployed something that somebody can use and it has some value. And so I’m a great fan of a thing called tedex talk called how to make toast. And it’s just a visualization way of doing systems thinking. And so as part of that foundation, I’ll get the data teams and the organization to, to work through those two value streams, right? Those two flows of steps, nodes and links. And so when we do that, what that also allows us now is to visualize where we’re about to make change. Because we can start basically voting and say, well, part of the value stream is broken, you know, so perfect example. If I look at somebody’s prioritization queue, their backlog and they’ve got 554 jira tasks in the, in their backlog queue. And some of them have been sitting there for a year. I would say we have a problem on the left hand side. Yeah. Around ideation, discovery or prioritization. And then we probably also have a problem on the right hand side around our factory that it’s too slow to move the, the work, but we want to focus on the lift. Right. We have a whole lot of work there that can’t be a priority because we haven’t built it. If we look at the way the teamwork and the data factory, you know, we can see blockers, right? We can see that, you know, this piece of work seems to sit in, in the factory for too long. You know, if you think about it as a, as a product, like a actual factory building cereal, making cereal, you know, we can watch the machines and we can see bring in some lean thinking and see how long they take. And again, we can go back to measurement. You know, the time between these two things takes too long. Like, what can we do to reduce the time? Is that more humans, more automation? Doing less? They’re all choices we can make. So I come back to your point around change, right? We want to visualize the system in those two different ways, and then we want to make conscious decisions on what we’re going to experience and changing. And that’s the last point. They’re experiments. We have a hypothesis, right? If I change this thing. Then good things will happen. And sometimes bad things happen, right? My favorite one. Is bring three more people into the team because we’re a bit, bit behind on building our products. Okay, let’s think that through. Right? We’re going to bring somebody in and we’re going to put them on the ground with the team. And let’s assume we don’t have a lot of documentation around the way we work or the way our systems work. So now they’ve got to learn our way of working. They’ve got to learn our technology systems. They’ve got to learn our domain. And how’s that going to work? Well, the team doing the work right now are going to spend half their time bringing those people up to speed. Now, in theory, they’ll, they’ll help us accelerate in the future. But right now that change is going to cost us three months of working slower.

Mustafa: Yeah.

Shane: Is that okay? Right. Because, you know, that’s the reality of that hypothesis. Yeah. So I’m with you, right? And your foundational space. Think about changes, right? Changes should result in things being better. You know, faster.

Mustafa: Yeah.

Shane: Quicker, more fun, easier, those kind of words. Does that make sense?

Mustafa: Yeah. Yeah. It makes sense. It absolutely makes sense. Before you go to governance and decomp management again, chicken thing, right? I have those questions as well. But let’s bring from this data product concept as well, just for our audience as well. And just to head out your, your thoughts around it, because as we mentioned, the previous adults were about project errors. We build a project. We start a project. We build everything front end to end centralized. And we, we wish that we can do something out of it. Right. That was projects. And then this product era comes in. Which you still feel is there. For me. I want to did a product is perfect. You can have a data product or you can have a data as a product, whatever you want to call it. But decision I want to bring in decision, right? Because why sometime I conflict with data product or software engineering practices is because you’re again creating a product, product may or may not work. I want to attack just, just to share my thought, right? You can comment on it. I want to attach a decision with a data product. I don’t want to start a data product without a decision. Because you can create a product, you can have a super good dashboard or headboard or slicing, enticing, and you end up having more decisions benefit on top of it. So what’s your thought on it? That die die or data product have to map. With a decision rather than analysis or analytics?

Shane: Yeah. So a little bit of semantics, right? Just for people listening so that when I use different words, they know kind of where I’m at. So in my language, an information product, there’s a boundary of a whole lot of things, data code, everything, visualization in a little box that solves the business problem. Yeah. So that’s, that’s what I call an information product. When I use the word data product or something like that, I’m talking about data essay. I’m talking about a table, right, that data asset identifying data products shouldn’t, but they tend to become just tables, right? Their data assets. So that’s, that’s my language. Right. And, and, you know, we’ve all had arguments around data product, data product thinking, data as a product. Right. And, and so I always think you just got to anchor your definition so that when you’re talking, people are clear what you’re saying. Right. It’s kind of the English to French problem.

Mustafa: Yeah.

Shane: But I’m with you. Right. When I talk about information product, I talk about the problem to be solved. The decision to be made. The action that will happen that needs to be happened based on that decision. The outcome that will happen if I take that action and it’s successful and the value that will be realized if that outcome is, is positive. And so that’s, that’s my thinking frame, right? And that comes from the product world. And then I make in the, in the canvas, which is a requirements gathering template. I change the language from decision to business question. Because I’ve found that if I say to a stakeholder what decision do you need to make? I don’t tend to get good answers back. Yeah. I tend to get. Fluffy words that I don’t understand and I can’t figure out how we can give them information to support that decision. Not always, right, but nine times out of ten. If I ask them what business question they want to answer with data and they start with who, how long, how many, how much, where, why? I will get back five to ten good business questions. And, you know, there’s things like, you know, how many customers do we have, which general are they coming from and how much money do we make out of them? Yeah, they’re good business questions. I can then ask them, okay, if I answer those business questions, what action will you take? Yeah. Oh, well, I’ll identify the customers with low margin and which general they’re coming from. And we’ll either, the action will take us either remove that channel or put investment in to make the people coming through that channel more profitable. Right. So what I’ve gone from as a business question to an action and then the outcome and the value. And these are decision in between, right? You’re making a decision to shut that channel down or you’re making a decision to invest in those customers through that channel to increase margin. But it’s kind of inferred, right? Now, what’s interesting in the AI world. Is with LLMs, does that change. The model, the pattern? Right. And I haven’t finished that work. Because what would happen if I told the llm that this action I wanted to make, is that enough now? And these are questions I haven’t answered, right? Because we’re into the, we’re moving into the action era, right? We’re moving away from the decision era and we’re moving into the action wave, I think is where the world will go. But yeah, so that’s where I’m at, right, is I talk about business questions that you need to answer and then what action will you take if you answer it? And then I get really strong requirements that a data team can, can build based off.

Mustafa: Awesome. Awesome. We are on the same page till now. Now let’s go to the next stage.

Shane: Yeah.

Mustafa: Because normally what we have just defined in last three, we move so quickly here. It’s already 35 minutes. So fun. Right? This is what, what normally I call it when I went into your canvas when you had a discussion. What you’re doing, it’s by right. It should be an obvious thing for me. It’s not any journey. All these things should be obvious. But people have forgotten about it. Because they were time pressure. There are so many pressures and they want to do it things very quickly. That’s why they forgot about these things. But your canvas says, right? You have to start with why and then how and what all those finish and stuff. Now let’s go. Okay. We as a customer, customer says, okay, we agree with you. That we have defined these basics. Now what? Now what is next stage of data foundation? Because the moment you have a data strategy, which we just discussed why thing, everything. Now there’s a governance and in parallel there is management data management. Two things underneath. Right? That is where I feel is an egg and chicken. Because if you create governance, which include principles. Procedures, principles, policies, procedures, metrics, tools and responsibility, all those things. Right? That you define and you are telling your workers, do it. But sometimes what happens doing ground reality is much different as compared to what we define. Now you go the other way. You say, okay, let’s build something and learn from there and then create a governance practice policy, blah, blah, blah. I normally give an example. You, you, you have a new colony. We have a open land. People came and start building houses. In the middle. You have a road can go through. You don’t have any direct any hurdles over there. But once kids comes out. They can run across the roads. Now you have a, you need to have a policy or you need to have something or maybe a divider across both sides so kids cannot go through. Now kids close up. They start playing. They play with cricket. We, we both know cricket with hardball and this stuff that can damage the cars who are going fast. Now you create bigger walls. So now these are the best practices or experience coming in which goes into a governance documentation. That if you build a new load, you need to follow this, right? So it’s an thing that is where normally wherever I’m talking, they are confused. What is foundation? Because if you build up front, we might not need it. But if you bring later, maybe it’s already broken. And we go under compliance. Way to start governance and management practices, various stuff.

Shane: So I remember my language governance as part of management. I’ll just keep reinforcing that. So this is really interesting, right? And this is something I’m working on right now.

Mustafa: On.

Shane: And I kind of think I have a set of patterns now, but I’m just testing them. And it’s one I’ve struggled with for decades, right? Because it comes back to this idea of building the plane while you’re flying it, right? Like how much of the plane needs to exist. And experimented for years around this and never really got traction. And so the framing of the pattern that I have in my head right now is this idea of pioneers and town builders. And you need to decide which of those two patterns is your organizational culture. And that is a decision that you need to make upfront. Because that frames everything else. Now everything else is just a lego block, right? It’s a pattern. It’s a patent template. I’m doing lots of work on those at the moment. For a whole reason. I mean, we could talk about in a minute around why I think templates actually help solve the problem that we had, which is we lost a lot of our domain expertise when, when we went to democratization, when we bought more people into our domain to do the work that we used to do as experts. We lost the education, we lost the training. And that was a bad thing, but it was also a good thing. So I think we have to teach people to do the work we used to do. And ways that had value. Maybe not the ways we used to do them, but in the same pattern they had value. Like data design, right? We should always do a light data design because it is a valuable thing, but we don’t need to spend six months doing enterprise data model anymore. So I think these patent templates are actually the unlock to teach people to do what we’ve got taught over decades to do. But I’ll go back to pioneers and town builders. So are we going to go out and experiment. Heavily. Find things at work? Buy things that don’t, and then start codifying those into our town. And build the town after we’ve pioneered the settlements. Is that our culture? Or do we want to actually build the town? And then get people to come and live in it? Right. Is that our culture? Now, when we build the town again, we can’t spend two years building the town before we let them come, right? So we’ve still got to do it and they’re iterative and a way that has agility. So let’s say that that’s a culture decision. And it’s an important one. And then what I can go back to is I can take that decision that organization has done and I can look at our team design, our measure success, our value stream and our data factory processes that we think we’re going to run with and go, well, you told me that you’re going to be pioneers and yet I see heavy team design, heavy value stream processes, right? So you actually, that’s not your culture. Like one of those two things is out of sync and we need to fix it. And so then, okay, let’s say that we pioneer all we town builders. The key thing is we have to think about principles, policies and patterns. Those are the three P’s that I use. And so again to anchor my language. A principle is like a culture thing. It’s like an agile minute based to us. We prefer this over that. We prefer small teams of three over big teams of 10. Yeah, we prefer delivering value earlier, even though it’s riskier. Than waiting three months and delivering value later than as higher quality. Or we prefer delivering value later with high quality over early with high risk. They’re just principal decisions that we want to make. And they set the culture of the team and the way the organization works with data. Policies for me are immutable rules. They are things we will get fired for. Yeah. So you should never store a customer’s name date of birth and passport number and clear text. Right. That is a policy. But we have to enforce that policy. So if I ever see a customer name date of birth and passport number in an excel spreadsheet. Then we’ve broken that policy. And the person who’s done that. Should be. Censured or fired. Right. And what happens though is when we talk about data foundations and especially data governance, we treat policies as wishes. And I’ve done it, right? I used to write these principle and policy documents that were massive and they were kind of rules, but they either got fired. You know, I work for a bank and they had all these policies. And yet I saw them posting a customer’s name. Their account number and their last transaction in a teams chat because that was the, yep, because that was the only way that team could solve the problem for that customer at the right time. And so they were breaking the policy. Now, the reason they broke the policy was because no other system was in place that let them solve the problems and they had to solve it. So they solved it with the tools they had. But what we should be saying is that team’s breaking the policy. What do we need to do to change what that team’s doing? So they actually don’t break the rules? Yeah. So policy is very small targeted set of rules. Like you said, the police state, you know, police things, you know, you, you can’t drive a car at 200ks an hour past a school road, you know, during school time or you will be locked up in prison, right? Quite rightly. Um, so that’s there. And then patterns are all the other things that are useful. Yeah. They typically what I see people write policies as if you do this, it’s valuable. If you do this, it’s useful. You should kind of do that. And that’s what I look at those three P’s principles, things we should subscribe to this versus that policies rules that I’ll get fired if I actually break them. And so I’ll come back to that. If we say that that’s the rule, then we even need to automate detection of when we break the policy. Or have auditors in our organization that will go out and audit that those policies aren’t being broken. If you don’t enforce a policy, then actually it’s not a rule, right? It’s a wish. And then patterns, what we should do is have people that share their patterns or people that define their patents. So let’s take that back to. The idea of pioneers and town builders. So if I’ve got a pioneer organization, yeah, then I’m expecting to set some principles upfront. You know, some disperses there. I’m expecting those pioneers to iterate around the patterns. And then I’m expecting to discover the policies that are important. Yeah, because they’re going to do some things that are wrong. And then I’m going to go, yeah, no, actually, we need to harden that. Right. And so we’d then bring that back into a policy. If I’m a town builder, I would expect the principals policies and patents to be built by the town building team, the data team as they work. Still iteratively, not two years up big build up front. But I’d be expecting them to do that work. So then it comes back to team design. Right? Because what I’m then saying is data governance is just a skill and a set of roles within the people doing the data management work. It’s not a separate team. But you can make it work. If you’re a town building organization, you can have a separate governance team, you know, like a team of architects or a team of town planners from the council that make sure you’re actually buildings of high quality that aren’t going to fall down in the next rate. Right. But you’ve got to decide which of those you are. And then bring in everything else to align with that way of working. Does that make sense?

Mustafa: Yeah, that makes sense. But I want to, I want to throw a challenge over here in my point of view as well. Regarding governance and, and data management. Right. As you correctly mentioned that. When we are doing stuff, then, then governance, as you mentioned, is part of the management. Right. But, and you, you gave an example of pioneer or Greenfield or wrong food kind of stuff. Right. But for me, in data world, right, because everyone wants data. Decisions on data or value out of data. And for me, they might not be any organization. It will be hard to say that anyone who do not have data, everyone have data and everyone have one or the other way data management practices already there. Because we have come from four hours operationalization analysis, analytics. And now decision you’re saying actions as well. But I bring action under the umbrella of decision. Right. Because without any action decision cannot happen. Right. So now the thing is. I want to bring this anger into it. That that era where you do things and learn and create practices. And policy, as you correctly mentioned. I always principle. Policies and procedures or patterns, these things. Right. I think these are these templates are already there. I think that is where I am already also telling customers that either you are, your domain specific, your banking or telco or some one of the, one of the domain you are holding. And there are a lot of patterns and templates already there. So I. Am on the other side of the table with apologies to you. Is I don’t want to mix governance with management. Because if you merge governance with management. Governance change based on what you’re doing. I feel we have come out of that era where governance practices are already matured. I feel you can bring up the template with by the, by the smes subject matter expert, as you mentioned. Last time subject matter expert was the one who doing something. Now we brought other domain people into the data world so they can do things for us. And that is there, I think they’re all chaos started happening. Right. So I feel data governance need is kind of a law. You’re creating a town. The data governance is law and legislations, whatever you want to call it. And managing is doing the construction. Now there are best practices when you’re building a town. You already know which kind of housing Society is going to have, which kind of knowling you are going to have. You need to have a police station. You need to have how many hospitals, how many pharmacies you want to have. So I, I feel this team need to be little small team. I’m not saying a bigger team, small team on governance need to be there. And there has to be domain expert and they have to be data savvy. Well, I can, you can say I’m a biased person for a data because I am from data. I always emphasize that in data governance, data people need to be there. Because once data people there, they already know what they are building and they can create principles, policies and procedures and patterns based on their past experiences. We don’t need to reinvent the wheel. I think we are already 10 minutes left already. Do you have a hard stop?

Shane: No, no, I’m good so we can carry on if it’s okay with you.

Mustafa: Yeah. Do stop me because I’m just throwing my ideas because I’m building a data model strategy for someone. So I was telling them, bring up the best practices, right people, because skill set is very important. Nowadays, because of AI, AI is a black box. It creates something. I have an example I gave into last panel discussion. I give you to you now as well for your thought process. Right. That if you don’t have this, then everybody will do whatever they want to do based on their experiences. And you will have multiple flavors of data governance. If I go with your mindset that data management need to have a data governance, do you see this risk multiple flavor of governances.

Shane: Yeah, yeah. But.

Mustafa: Oh, you don’t mind that.

Shane: No, I do mind it, but I think it’s the reality at the moment. So again, I’m going to slightly disagree with you but agree with you at the same time.

Mustafa: Absolutely. And on the, I want to hear it out for my. I can.

Shane: Okay. So let’s, I’m going to put a stake in the ground and then I’ll talk around and come back to it that I don’t believe we have data governance. I don’t believe I’ve ever seen it. Right. And let me justify that. So if I’m in an enterprise organization. We have financial governance. Right? Like we have people who are accountable, CFO to make sure that our financial records are accurate. And we don’t overspend. And then they have a bunch of principles, policies and patterns that help them do that job. They have a thing called a profit and loss. It is a pattern. That every accountant applies like a balance sheet, like a trial balance, like a cash flow statement. That is part of the practice. As part of their way of working, you get qualified as an accountant, you know what those patents are. You apply them. You are ordered by them. You are held to account. They are governing artifacts. And the data domain, we have check. We have data modeling, but you can have any flavor you want and you don’t even have to follow it. We never audit any of our data models. Right. You can say that you’re storing data encrypted, but does anybody ever audit you? You can say that you can’t do this. But then people go and do it. And that’s part of the problem. And if we take our town plan, you know, the idea of building a city. We have governance because there is normally a council, a regulatory body that says if you’re going to get, if we agree that you’re allowed to build that town, here’s the rules. You know, you have to have a school. You have to have buildings that aren’t more than three stories high. And that governance body has power. To punish you if you don’t follow the rules. In my experience, it’s very rare for a data governance group to have power in an organization. Now in some organizations it does, but I find that in my experience a very rare occurrence. And so I then say back to my stake in the ground. I don’t think most data governance teams are governance teams. They’re not auditors. They’re not the CFO. They’re not the risk committee for a bank that will, you know, shut you down if you break the reserve bank rules. You try to do the right thing, but they’re not empowered to govern. And that for me is the problem.

Mustafa: See. That’s why I’m saying that skill set comes very obvious over here. When I say, of course you are right. Data governance comes under the domain of governance. Right. And there are financial governments. There are many other governance in the organization. Can you believe that in a financial governance strategy is built by non finance guide? You can’t think about it. The same way I feel for data governance. It has to be built by data. Guys. What you’re saying is absolutely right because I have been working in right in organization where I was not part of data governance. We never heard about. We never follow their policies. Reason being we already have way to justify. Right. Because they are not data people. That is my obvious first prerequisite is if someone is building a data governance team. Data people need to be there and senior data people because it’s about law. You have to implement a law. Law can only be implemented if people understand the law. Otherwise people will bypass the laws. That’s not my point of view.

Shane: Yeah, I’m going to disagree with you again. And the reason, and I haven’t thought about this, right? And so I’m kind of speaking as you’re making me think. So that’s great. Thank you. I go back to as data professionals, we have no practice. We have no rules. We have no certification that we’re held accountable for. Therefore, we can’t govern ourselves. What we are is we’re Practitioners. Thinking of the hoop, you know, I think I’ve seen CFOs, chief financial officers being the best data governance people because they come from a practice of governance. But like you said, they’re not data professionals, so they don’t, they can kind of know what to govern but not how to, how to make sure it is governed. Right. And so I go back to this user car analogy. You know, the rule is I can’t drive more than 100ks.

Mustafa: Yeah.

Shane: That’s the rule, right? That’s the governance in this area. Well, actually, as a car manufacturer, why don’t I limit the car to never going over 100ks? That doesn’t happen. Right. And it’s the same with data. Like, yeah, I can’t do this, but everybody does. So I think that’s the problem. And then we come to llms. They, they’re, they follow the rules as much, you know, they’re non-deterministic so. But everybody’s trying to make them deterministic. So let’s say we finally get that right and it’ll be really interesting. We ever do. The second problem we’ve got to solve is, is the rules aren’t clear. Right? They’re not described in a way that if the elderly was deterministic, it actually knows how to follow it. And that is a problem with hemp for 30 years. If I drop into an organization and I say to them, what are the rules? I get a conversation. I don’t get a set of rules. And that’s part of the problem that teams have struggled with. And that’s why I come back to this idea of, of patent templates. And what I mean by that, and the canvas was just one of them. Is if we have a thing that we fell out or do as we’re doing the work, and that tells us whether we are meeting the governance rules or not. We kind of make a whole lot of problems go away. Because I’ve got a, the first problem is I have to understand what I’m filling out, right? Like, I’m, I’m filling something out. I have to understand what that process is. So if I have to fill out a conceptual model. You know, I’ve just Concepts and relationships, you know, customer orders product from store. I have to understand how to fill out that concept model, that template, but I actually have to understand what I call a concept is, like, how do I Define what a customer and a product and a supplier and a store is? How do I Define that relationship? But once I’ve done that, now I’m by default, I’m going to start to get a template that’s got some data in it that can then test the policy of we will only Define customer ones. Yeah, probably a principle or a patent, not a policy. Right. Because you only get fired for doing, doing it wrong. But that’s kind of where I’m at the moment. Right. And so what I think from a data governance lens, if we, if the data governance team can’t believe the true stakeholders that are accountable for that government’s been applied, then they’ve got to be enablers, right? They’ve got to have ways of giving patent templates and ways of working and patterns to the data teams so that those rules are complied with. And that’s kind of where my head’s at the moment, right, is, is giving teams things that if they follow those things, the rules are automatically applied for them or flagged when it’s not. Yeah. But really think I’m still in the mid of thinking of that one. Right. So that’s why there’s not a, not 100 Clarity on, on that one.

Mustafa: Absolutely right. Because we, I am trying to tag this data Foundation where governance and management. Because let us bring another flavor for your food for your thought. Right. Matter data. Let’s bring matter data because for, for me, the data foundation which we are talking about, as we mentioned earlier in this session is your product can best practice as all we discuss decisions and blah, blah, blah. Those should be an obvious. Now those have surfaced because AI came up and start showing that what we are doing wrong. But those were supposed to be their practices supposed to be there. Now. Metadata. Because metadata is also for me is key because when AI comes in. Okay, let me say this example again which I gave in my panel discussion last week. What happened? I was not left coding like 15 years back. But this wipe coding everybody keep telling Mustafa go and try, try, try. Never ahead of time. Now I’m a jobless like few months. You know that. So I went into coding. I opened a tool called cursor. It’s a white coding tool. I started creating, replicating. LinkedIn. So it’s an awesome tool. I gave it a LinkedIn URL. It replicated 50 to 60% of LinkedIn. Those things did not replicate it, which was not. Possible to open without a username as it’s replicated. It treated the whole data model underneath. And it was working perfectly fine. Now the question is AI can create a product. Now what I did is. I picked up that data model. Put it into a different schema. And ask the same wipe coding to read that schema. And create. Linkedin like portal. It could not do it. Because when it creates that schema, it went from application front end and created backend. But when I gave it a back end, it could not create anything for me. It was a rubbish thing. Like which gave me an idea that, hey, because metadata of those tables are not there because when it created from linked in meta data was there. Conceptual, it went back. It created business model information model. It created logical model. It created conception model. Well, there are many kind of data modeling technique backend. It created that. Then it created a physical model. And it works. Now the second example which I had, it could not. Then what I did is I asked it create a business model on top of it. Informational model fcoim conceptual logical. I created all those and mapped with the physical model. And then I asked it, go and create a portal. 90. So for me, if AI. Asking you to start creating a product, it will create. But if you build a foundation of data product, AI might not be able to. Because metadata is not there. So now correlating our topic data foundation. If we, are we saying that. Creating modeling modeling, as I mentioned, business modeling information modeling, all those things should be a separate practice. And that is the foundation which we are talking about. And we, we can argue there are so many data modeling three and a half star schema data world focal enter hope. So many are there. Right? If we are talking about, are we talking about only up to the logical foundation as a logical modeling? Because after logical modeling, physical comes in. And that is where AI is unable to do it. Are we saying that we should emphasize or spend time building foundation of data modeling up to logical and which will be, by the way, part of the greater governance? If not data management, data management comes when you go towards physical modeling. Are we saying that. Okay, you, you can comment on it. You can consider for me physical modeling can will always change because new tools and Technologies will come up until unless we have a proper up to the logical model. Then these things can be reused. Are we saying these are the data foundation we are looking for or not?

Shane: Okay. So that’s a big question. So let’s, let’s go back to the core preface that every part of our foundation is a lego block. Yeah.

Mustafa: Okay.

Shane: So there’s small parts that we put together to work the way we want our team and our organization to work. And we don’t want to spend three years building out all the lego blocks. Because three years of great fun, but no value to the organization. And we don’t want to just do ad hoc where there are no lego blocks that we can reuse later. Right. So we want to be in that dodgy middle ground and we’ve got a whole lot of decisions like which Lego blocks are most important. Which ones do we do first? Which one do we use second? What order do we need to do them? Which one’s a small Lego blocks? We can spin a little amount of time, and that’s okay. Which ones do we actually have to spend a little bit more time because it just involves them?

Mustafa: Yeah.

Shane: So, so let’s use that as our framing and then let’s go back to the blueprint. Right. So we, we’ve understood our current state. We’ve gone and figured our measures of success. If we spend this money in time, this is how the organization is going to change. We’ve done our, our team design. We know kind of how the team is going to be structured and where they live in the organization and the skills and roles they have. We’ve done our value stream. Yeah, we know kind of the process from when somebody has a problem, all the steps we’re going to take. We think it’ll take till we deliver value. We have our data Factory kind of sketched out, we kind of think this is the way the data will flow and some of those decisions. What I, and we got some data principles. Yeah. Some kind of disv is that. We might have a couple of policies because we already know there’s some bad things we shouldn’t do. We probably don’t have any patterns yet. Then what we do is we then look at, for me, I look at kind of capabilities. Like actually, given all that, what do we actually have to put in place on day one? To, to make some stuff work, you know, get some products out the door that are governed, you know, that are managed. They managed. Right. And so I, I kind of then that’s where we start getting into tooling capabilities, technology now, right? Like, I need something that does this. I need something that takes the data out of those systems and brings it into the platform. Because we’ve made an architectural decision that we’re going to bring all the data into one place. Yeah, it might be a lake, it might be a data warehouse. That pattern, right? We haven’t made that decision yet. And we’re not done that in the past. I’ve always had metadata is important. But we’ve always treated metadata as an exhaust. I’ve always gone and said, well, we don’t click the data. We can’t build anything really important. We can’t store the data. Yeah. Again, if we can’t transform the data. Yeah. And if I can’t visualize or deliver something kind of thing. And so I foundationally always came back to those moving parts as being the first moving parts of this Lego blocks that have to be built out. And then I’d bring in some development practices. I’d say, well, you know, we want to be able to check our code in. We want to be able to deploy and test. And I bring in some other Lego blocks. And for whatever reason, metadata. Kind of always came last, right? It was kind of treated as exhaust. We did all these things. They’re not now need a catalog. Right. But over a week while I flipped my thinking and, and I kind of said, actually, we need to capture that stuff first and then hydrate everything else. Now I’m just going to go bound us to some language again. Right. So 10 years ago, I talk about metadata. I’ve talked about being metadata driven. Yeah. Tools that are metadata driven to find the metadata. The tool does everything else. And then I kind of moved on to config. Yeah, it’s config driven because we’re kind of configuring how the system should work and it should hydrate everything else for us. And then last year, there’s a big, big semantic war in the data world, which was basically semantic layer or context. Yeah. And I’m a, I’m on team context, and I think we’ve won that argument. Although our library sciences friends are now bringing an ontology taxonomies, our and RDF and we’re back into another semantic war. So I’m just going to use the word context. Right. And the way I position it now as patterns in my head is we have business context. Yeah. So definition of a customer is business context. Our domains are business context. We have a whole lot of context that is relating to the way we run our organization. We have structural context, you know, the way our tables are structured, the way our fields are structured, the way our systems, the, the things that I would have called metadata 20 years ago. We have operational context. The queries that have been run against our data, who accessed what data when, who exported what data when, what decisions, there’s a whole decision logging kind of thing coming through now. So there’s all this operational stuff that’s really useful from a context point of view. And then we have agent context. The prompts that we give the agents, the skills that we give them if we’re using something like Claude. The contents that we give it to help reinforce some of the models. Right. This context or things that, that drive the way the agents behave. And that’s how I think about it now. Right. And, and foundationally, I would start capturing parts of that as I go. But I, again, I’d flip the model. The way I think about it is we want to capture the context and then have that context drive everything else we do. So give you an example. If we say we want to base on all that, we decided that policies, that we’re taking in more of a town plan approach and we want to put in place some policies, right, that are complied with as we built that town. Or even if we’re doing pioneers, we say this, you know, one or two policies applying is can’t break. Yeah, they can build a town wherever they want, but they can’t build it on the land over there because it’s owned by a different organization. Right. So, yeah, don’t just know you’ll be fired if you, if you do that. Everything else is on. How do we capture those policies as context? And then how do we hydrate everything from it? Yeah. So how do we say, you know, you can’t store customer name. Yeah, it’s a policy, that’s context. And then how do we hydrate a system that actually checks it’s never been stored. And automates that formula as a test. Yeah. And that’s how I think about it. Right. So, yes, I agree with you metadata first, but truly metadata first. Right. We write the context first and then the system builds itself as much as it can off that context. Now, if we don’t have technology that can do that, that’s fine. We define the context first and then a human rights code that enforces that, uses that context. I think that’s where we’re going to end up in the new world and the new data world. Is everything that’s going to be defining context first and then hydrating all the other moving parts that we used to do as humans over time.

Mustafa: You’re absolutely right. Just to, just to bring in same example which you’re mentioning. Right? That. Policies has to map with the, if you have a proper metadata down there, then you can attach your policies with the meta data. Like you give an example of mobile number or ID card or a name or account number of the bank. Right? You cannot. Make those visible. That’s a policy. Now that policy need to be implemented somewhere by data management folk, which is engineers. Right? Data Engineers. Now giving this policy to them. They are not good enough to understand the business. For me, this policy need to tag with the meta data because in metadata, I normally segregate business metadata, technical metadata and operational metadata. I feel the foundation is business metadata. Because once you have defined business metadata, that won’t change. In an organization until process doesn’t change. Business meta data is about processes, how everything works, what is the definitions of blah, blah, blah, blah, all those things. And then policy attached to that particular business model. And now when management folks comes in. This policy get attached into the development activities as well. That wherever one example, wherever ID card is there, it has to be masked. So people don’t have to think about a developer, don’t have to think about it because it will go through a, maybe a lookup table or maybe something which automatically do it rather developer has to manually do it. So that is where just to bring my thought process back that data governance has to be little bit maybe coming era, you think about it again. In coming at a data governance has to be from the prospective of business matter has to be separate. Because if you give it to management team, they won’t care developers are who are the developers python developer or SQL developer. Their developers, we don’t expect them, as you correctly mentioned early in this session is we brought other domain people into data domain and start doing developers because they don’t care and we should not put this responsibility on them.

Shane: Yeah.

Mustafa: To find out what they need to mask, what is the policies. We make it. Again a principle. Principle is follow all the policies. That can one principle.

Shane: Yeah. But, but the least type of scenario, right? So, and this is where I go back to finance, because I think they’re well ahead of us. And, and some of this. You know, they, they have this idea. I’m sorry. I go back to governance. I see governance teams either taking one or two stances. They’re auditors or their coaches. If their auditors, what they say is, well, here’s the policies, and I’m going to come in and actually audit you. To make sure you’re following them. If the coaches, they’re saying, here’s the policy and here’s patents or tooling or things you can use that make sure that policy is enforced and they don’t have to worry. If I’m a finance auditor and I know that you’re using a financial system and you have a p m l. That’s a pattern based governance because I know that there’s no way the p l can not balance, you know, because it’s these things. And within the data world. I still, like, we don’t have the idea of microservices for our tests, right? Global ones. We have local ones, but we don’t have. A microservice that says, hey, that data, you know, anytime we see something that has ID in it, an ID value, it’s encrypted. By default. Right. The team doesn’t have to care. And actually, we used to have that in databases. And when we went to file systems like hadoopa now iceberg and we kind of lost a whole lot of the free that we used to get in made our life easier. So auditors or coaches have systems that enforce what’s right. And then if we’re auditors, if I look at people that do auditing, they actually have a plan on how they’re going to check that control, that policy. Right. What are we doing data governance? We say, here’s a policy. And I go, how are you going to go check that that’s been enforced? It’s the data team’s problem. Right. And again, we haven’t solved that. I think the last thing for me is if I go back to this idea of context. Being defined first and hydrating everything. Those capabilities don’t exist yet. And they don’t even, it’s not even that they don’t exist in, in the data teams. We need them to exist in an organization level. So, you know, if I say the policy is ID can’t be stored in clear text, then that policy has to be automatically applied in my data platform and my software as a service and my operational systems and my Excel spreadsheets and my SharePoint and my text file sitting on my local hard drive. Like, it’s a policy, right? It’s not just a policy for the data team. And that idea of defining context and having it hydrate all our systems, I haven’t seen that yet. We get it one day, right? But I think it, it’s a big technical ask to figure out how the hell you, you do that as well. Right? But any step we can do, which takes us. Away more and more away from this idea that metadata is exhaust context is exhaust light. We kind of look at what the system did and said that was the context. And we flip the model to define the context and get the systems to follow that context. As we do the work, I think that’s a good change in our foundational. So coming back, as I talked about, you know, the foundational technology pieces that I’d always looked to as data collection, data storage, transformation, visualization, and then a whole lot of other stuff around it. I’m now saying actually defining context early. Is the first foundational piece before I do any of those. So defining what sort of systems we have as context, where the data lives in them and then how that hydrate the data collection of that data into my platform. That’s what I do. Yeah. It’s a change of thinking and it’s also a complete change of working.

Mustafa: No, absolutely.

Shane: Yeah.

Mustafa: Just to close this session, right, you are absolutely right. That for me, at least people, most of the people think that data governance is more towards only data people. I join data governance business as well. Data governance need to be implemented as much as in data side as much in business side as well. It cannot be one sided and this aside we all know data store word is also become is one of the critical rule in data governance. That they have to work hand in hand with data governance team so they can make sure that business is also following the same thing. As you mentioned, it’s not about putting policies on data folks. It’s putting policy, even IT card if it is shown into your email, it should not show. Even it is in your excel, it should not show, which is on your desktop. So yes, there are a lot of technical. Depth or technical implementation need to happen. With the governance. So just to repeat myself, it had to business side. It has to be the it and data side as well. Following all those governance thing. So I think we have covered most of the thing. I want you to go to a data modeling as well, but we have already passed our dedicated time because data people are asking that when you just to throw in last question, right, people are asking, when you talk about data foundation, are you talking about data modeling? Physical data modeling, right? Even before logical like let’s go to one step back conceptual data modeling. Because logical data modeling is something which. Will. Normalize the tables and entities up to the level it can be implemented. Right. People asking me. That data foundation are you saying that we should finalize our decision that it will be star schema. It’s a data vault. Which kind of data modeling we need to implement. Is that the foundation? Just the last question.

Shane: So I think we have a foundational lego block of our data architecture layers. You know, how many layers staging raw, whatever we’re going to call them. Are they enforced or are they optional? How does the data move through a system? That is a foundational piece we do early because it kind of sets the scene. So we’re going to do later. And it’s, it. S harder to change later than it is to define upfront. I’m co-writing a book on conceptual modeling right now with your copola. So I’m very, very opinionated on this. But if I go back to this idea of context and hydration, my view is. A conceptual model is one of the foundational pieces that I would now bring in very, very early into a positive date as of part of data governance. Then logical modeling. Has some value, you know, where do the attributes fit? And then physical modeling. It’s important. But what I would do is treat it as a candle. Right. So I would say if I use the devops terms, my conceptual model is my pit. And then I should be able to hydrate a physical model. From that conceptual and logical model whenever I felt like it. Yeah. Data Vault, dimensional one big table activity schema. Right. I, I should treat those modeling techniques as disposable now because my conceptual and my logical models are my business context. And those physical ones are my structural context. So that’s how I think about it now. But most people don’t. Right. So then.

Mustafa: On this one. Absolutely.

Shane: Yeah. If you don’t think about that way, then you, you need a patent or a policy. On your physical modeling technique because you’re, you’re losing all the value of doing the context stuff up early. So, yeah, pick one. Yeah. And just be clear as an append. Like I should use data vault or it’s a policy. I’ll get fired if I don’t use data vault. Right. Which of those are, which are those two? Are you telling me? And then I take your point about governance. Right. So I was just kind of thinking about as you’re talking about it, and I kind of give you an example to close it out. There’s a lot of work around in the, in the data domain right now around data contracts. And again, let me, let me anchor my, my terminology. Right. A data contract is an agreement. Between somebody on the left and somebody on the right on how they’re going to share data. Yeah. So it discusses schema frequency, quality. It is a, an agreement. Right. And then we tested that agreement that’s been there. Typically, it’s an agreement between the software engineers and the data engineers because we’re moving data from a software system to the data platform. But I’ve seen lots of people now using it internally within the data platform between ETL jobs. Yeah, it’s a, it’s a contract and agreement that this table will look like this and this table will look like this. And therefore I can write the code in between. If we think about that in terms of true data governance. The data governance team should be mandating that data contracts are put in place whenever data moves from a software system to a data platform. And they should be auditing that actually happens. But we don’t. I never see that. Right. Yeah. It’s always a data team.

Mustafa: Don’t do it.

Shane: Thing, right? It’s a tool for them. But actually, just think about that, right? From a governor’s point of view, you know, when we go and have, we go and buy something from a supplier outside our organization, most organizations, every procurement team that governs there’s a contract in place. Right. We know what we’re getting and what we’re paying and isn’t illegal agreement. Data contract is exactly that, but for data, and therefore, and the way you think about it, the data governance team should be responsible for making sure contracts are in place and either auditing that they are or giving patents to make sure they happen by default. So, yeah, I’m like that. So, yeah, contracts, data modeling, they’re all foundational pieces. For me, I’d focus on conceptual and logical and treat high physical data modeling as just a execution problem. It’s cattle. I should be able to change my mind.

Mustafa: Yeah.

Shane: And the system should just absorb it. Right. I, I don’t think we’re there yet. Right. But I think that’s where we’re going.

Mustafa: I wanted to close this, but I have to throw one more question. As you mentioned, you’re writing a question writing a book on conceptual modeling. Because again, for me, the anchor is conceptual and somehow little bit logical as well. But conceptual is something which is, which is whole business up to conceptual and then goes to the management side. Right. When you talk about conceptual, are you saying that conceptual model of course need to have a business modeling information modeling all those techniques, taxonomies and all those things. Are you in your book by the way when your book is coming?

Shane: I don’t have a date. Juha has been much better at riding his parts of it that we need than I am. So I am dragging the chain and I’m trying to get that chain pulled up a lot faster. So, yeah, no, no idea when we’ll be done done on that one.

Mustafa: So now when you say conceptual modeling, are you considering all business modeling information modeling? All those things up to the conceptual modeling, are you saying that as a foundation?

Shane: So.

Mustafa: Last question.

Shane: So the book’s called modeling business Concepts. And it’s called that specifically for, for a bunch of reasons. And the, the. The goal of the book is to teach the steps that you need to create a concept model. Right. And so we’re kind of just working all the way through, like, what does that actually mean? And so we’ve identified 11 steps, right? So the first step is Define the domain that you’re going to work in. Like, don’t model the whole organization, try and figure out a domain. The next step is find a subject matter expert. Yeah. Find somebody in organization that knows how the organization works, and they’re going to give you the information you need to define those Concepts. And then we go through. And. It has the idea of where we Define the concepts and we Define the relationships, and then we identify where those relationships and Concepts are an event, a core business event. You know, customer orders product, the order is, is something that was really important to us. And then we have a visualization that we’re working on, like, how do we draw that map, that concept map, what does it look like? And what we’re doing is we kind of both aligning that map to what a conceptual data model map looks like with a few tweaks. Because that’s how we think about it, right? That’s how our expertise and experience has already done it. What’s interesting is I often use an event Matrix, kind of like a bus Matrix, but a version of it that kind of got from Lawrence core and has been stuff. And so I’m always looking at it going, yeah, we’re building this concept map, but actually maybe an event Matrix is another way of articulating that. And I think what we end up with just the core we’re teaching is the concept map, but we’ll use those event maps and those other visualizations as a different lens. The thing I struggle with is what is a business model look like? Yeah. I, I, because I always think business map, concept map, logical, physical, right? Like, that’s in my head. But I’ve struggled to find a business. I mean, the business canvas is kind of a business map, but nobody really treats it that way. And so, yeah, I’m very intrigued. If you can find somebody that goes, well, the thing we do before a concept map is the business map. What does it actually look like? That’d be, that’d be cool piece of content because I struggle to find a pattern or a patent template that makes sense to me. So I’m not sure I answered your question on that one because we’re still writing it, but it’s kind of a rep.

Mustafa: Because then you say conceptual, you are looking towards default conceptual because conceptual doesn’t come from the air. It has to come from business, as we said.

Shane: Lica. Right. Yes. Yes. So, so.

Mustafa: So. It has to come from business.

Shane: Yes. So when we talk about a concept map, it is a representation of the way the business behaves, not a representation of where the data is stored. Yes. So we should be able to do a concept map without ever looking at a source system. Either looking at a database.

Mustafa: Yes. Yeah. And just to answer you to find out, by the way, in my latest book, I am data media. I have added 37 data modeling techniques. So over there I go step by step. Maybe I can share with you. You already bought my previous book. Right?

Shane: Probably. I’ve got a whole bunch of books sitting left to me in paper, and I’ve got a whole folder of books I bought in PDF, and it’d be fair to say I have.

Mustafa: Yeah. So whoever. Yeah. So whoever have bought my previous book, I’m giving them free my latest book. So let me pass you my PDF of the latest book. Actually over there you can go into it, have a look. 37 techniques. So wherever conceptual before you can go and see all those techniques, what they do. I spend a lot of time because someone asked me how many typings are there. We have worked only five, six techniques. Normally we work three and a half. Star schema snowflake. Data vault. Logical physical. This is work. But actually when I really went into it from above, I discovered there are 37 data modeling technique either what most of them I didn’t know.

Shane: So, so you know how, you know how you said that you kind of went vibe, started vibe coding and you picked LinkedIn. Yeah. So it’s kind of, what I find is whenever a wave comes, right, the best way for me to understand the impact of a wave is to learn by doing. And so, of course, I always talk, used to talk about current code, don’t code, won’t code, right? Like, I, I tried to code in my early days. I was the worst coder there ever was. But I thought, well, okay, we got this new world. I’m going to go try and vibe code something. And so the, you know, the obvious candidate for me was I vibe coded the information product canvas. Yeah. So I vibe coded a small app, which means you can do it online. And then I published it as an open source thing. So you could download it and use it and hack and do whatever you want. And then because I’ve been doing a lot of work on this idea of context and context playing, I was like, oh, well, maybe we’d extend this out. So I vibe coded the business model canvas and lean campus. And then, of course, I couldn’t stop myself. And I started vibe coding the business event matrix. The data layered architecture checklist concept model, data dictionary, a glossary. But I purposely kept them all separate little apps, right? Because I’m testing this theory that these little micro apps, if you use them in your day job have value and store and capture context without you really realizing you’re doing it. So if you’ve got 37 different patterns. For the way we can diagram or canvas things, yeah, I’d love to work with you on it that we kind of just create little bespoke apps for these and publish them as open source. Because I think seeing by doing, like being able to fill out one of those things is a great learning mechanism for people to go, oh, you talk about a concept model and you’re talking about Concepts and relationships, and that’s it, right? No boxes and lines and a bit of car melody. I kind of get it now. Like, I got more questions, but I kind of played with something and it’s visual and I kind of get, I think I kind of get what you’re talking about. I think that’s a great way of rounding out the learning in our domain that we’ve, we’ve missed or lost. So, yeah, if you’re up for it once, you know, flip me your book. I’ll, I’ll have a look at it and then I’ll come back to you and go, it’s vibe code together. You have to move to claud, not, not cursor. But, yeah, and maybe we’ll do them and publish a couple and see what happens.

Mustafa: Same thing cloud is and cursor same thing both are lm behind it cursor is people like me who have lost.

Shane: Yeah.

Mustafa: Development like 15 20 years back. I can’t do coding in vs code. So if you are using clot, you need to have vs code as well. So I don’t know to go into there.

Shane: No, no, no, no. I use the claw code gooey. I I can’t, I can’t go into that terminal thing or VS code. No, no, I just, I, I go into something that looks like chat GPT and I go, like, I did one last night. I went and I said, I want a small app for metric trees. And then I went to bed. And then this morning I got a really bad app for metric trees. And now I’m yelling at it, telling it what it needs to fix. So, yeah, I mean, there’s. Interesting thing, right? That in theory, we can create this context layer for these apps, like the decisions and, and those kind of things that are being used when the app gets written back to your idea of building LinkedIn and, and hold the context of LinkedIn, not the physical data structure. And if we do that, then in theory, we can have Claude or codex or cursor or any of these things. Read that context of that app and then update it. But I’ve tried it. And, and again, I tried it a while ago. So what happens with the new stuff is you have to try everything every week because something bloody changes and it gets better. But, but what happened was the, the coding lms argued. Like codex would go, oh, Claude obviously wrote their, and it would refactor the code without making anything of extra value. And I kind of liken it to humans. I don’t know if you’ve ever seen it, but you walk into a brownfield site and, and the engineers that build all the ETL are gone. And the new people came in and, you know, they typically go, oh, that’s not the way you write that code. And they’ll refactor all the code. So it writes the way they wrote, normally write it, but it hasn’t changed anything. It doesn’t run faster. It doesn’t run safer and it hasn’t edited any new products. So, and so I find that llms are funnily enough doing that as well. So, yeah, we could give it a go. You know, I’ll give you a point cursor at it and see what it does.

Mustafa: I can share it with you. I created a training as well. It’s not very polished, but I’ll share with you. I have to polish it, but you can have a look all 37 techniques are in there I gave some examples. And I’m planning to get agent for each. I think I created agent for first four five techniques. It really, it was awesome because I talked, it narrated into a text. That text was in first prompt for the first modeling. That modeling created an output that one agent that output was an input for a next agent which can do the second modeling. So it was so awesome. And when I gave it to the cursor. Because all metadata was there. It was able to create the application very good, but it was such a big effort. Because someone has to create 37 agents. I didn’t have a time.

Shane: Yeah.

Mustafa: I learned in four or five that it’ll work. So maybe I’ll share with you.

Shane: Interesting thing on that is that lots of people are working now on skills for things like Claude, right, where all the expertise sits in it. And they’re really, really valuable. But there’s no feedback loop. And because we know that the lms are still probabilistic, not deterministic, which means that you ask it the same question 10 times, one of the times you will get a different answer. I’m still fundamentally believe we need a combination of the skills that do the work and visual techniques so somebody can understand the work that’s been done. So. And your scenario, a skill that I can ask it and it will create that model. And then a visualization of the model in a way that I can understand where that model is correct or not. And that, that becomes the problem. Like, how do you know it’s correct? Like, do you build another skill that tests it, what we call a judge mechanism? Like first skill writes it, second skill validates it. Yeah, we, we can’t, we’re still not really to move to a world where those things are right and therefore people without the expertise can just trust what it’s giving them every time. But, hey.

Mustafa: Meditation after the human. Why I feel it can be my own perspective. That why I was successful because I knew what I want. AI didn’t new because when I gave it a physical mode, it didn’t know. A physical model customer is just. Six or seven character world. It doesn’t know what is customer until I gave it a metadata that what is customer and customer can be different for different domains. For HR customer employee for HR is different. For a loan officer is a different. So I have to give a context. That is how it works. Never mind. I won’t take you a lot of time. We already one hour 35 minute. I will always chain. I think it was a great session. It become more of a data governance session. But I think data foundation has a big play of data governance as a quick play in data foundation. Because as we are discussing, you need to have right people at the right place. Because you don’t have a time to test and run and create something governance. I feel which still contradict with you is I feel that data governance team need to be separate. At least in this era. Because AI is doing their own shit until as right people do not know what they are doing. We are lost. That’s my prospect. Ive.

Shane: So I, if we say that data governance now behave like Auditors or risk managers or the CFO, then I agree with you. Yeah, that actually, it’s, it’s outside the people doing the work and they are governing that work is done right.

Mustafa: Yeah, that’s what I mean.

Shane: That’s a different pattern to what we see when we see data governance right now, in my view. Right now, we see some people trying to do their best, but they’re not empowered. Yeah. They can’t fire people. They can’t stop things happening. They can’t delete data.

Mustafa: Right.

Shane: They can’t shut down systems. They don’t have the power to govern. Like a council does if you’re building a house, like a CFO has if you’re paying for money, like a procurement team does if you’re buying something. So, yeah, I’m with you. We need to elevate governance to be governance and not management.

Mustafa: Yes.

Shane: And then, then I agree with you. Right.

Mustafa: Before you convince me. Okay, thanks. It was such a talking to you. I think we can have more sessions if you share. Take care. Have fun.

Shane: Thank you very much for having me. We’ll talk soon.

Mustafa: Yeah. Have a good week. End.

«oo»

Struggling to gather data requirements and constantly hearing the conversation above?

Want to learn how to capture data and information requirements in a repeatable way so stakeholders love them and data teams can build from them, by using the Information Product Canvas.

Have I got the book for you!

Start your journey to a new Agile Data Way of Working.

Buy the Agile Data Guide now!

«oo»

The reasoning behind this principles was creating a BI tool or BI platform had too many table stakes from day one to have a chance of succedding.

Too many expected features, too much competition, too much time, effort and moolah to build something that could compete with the established players.

So we used third party tools. Looker Studio, Power BI, whatever the customer had already paid for. We focused on the hard bit underneath, the data platform, the data, the business context, the patterns, the scaffolding.

All bound with the core principles of being able to scale at will, and rin on the smell of a oily rag ( well based on the cost of oil before the latest middle eastern “adventure”)

That was the right call and has lasted us well for the last seven years.

Then vibe coding turned up and we revisited our thinking.

TL;DR

We are betting the BI platforms of the previous waves become legacy and “one shot BI apps” become the norm.

Bold statement right?

Let me give you some context (after all without Context all we have is data)

Why we think “One Shot BI Apps” are the new black

Up until now if an organisation wanted a bespoke Information Product / BI App that did exactly what a consumer needed, they had to pay an expert development team to build it and then maintain it.

That took time, effort and money.

So instead they compromised, they bought a BI platform, a general purpose tool that kind of did what everyone needed but never quite did what anyone actually wanted.

Stakeholder don’t actually want a dashboard with 24 areas and 47 filters they have to squint at to find the one number they care about. That is just what they have had to live with until now.

They want an app that answers their specific question or even better tells them the next best action. They want an app that works the way they think and the way they work, with nothing else getting in the way.

Data teams doen’t want to spend three iterations building a Power BI report that gets used twice and then someone asks for “just one more chart”. They want to deliver something that actually gets used, that drives proven organisational outcomes and value and most of all and makes somebody’s life better.

Stakeholders and Data Teams want something that does one thing well, enables one specific job to be done, and gets out of the way.

The BI platform was always a compromise. It existed because bespoke was too slow and too expensive.

And that where vibe coding using tools like Claude Code have changed the promise.

They promise:

• Low cost :: Making bespoke cheap as chips.

• Speed to market :: Minutes and hours, not weeks and months.

• Democratisation :: Anybody who can describe what they need can build it.

• Cost :: Less than your annual BI platform licence, and your big server/services infrastructure costs by a long way.

With all these promises why would they keep compromising?

We have seen this movie before

We have seen promises like this before, promises that are early in the hype wave, and then eventually get grounded in reality

We have seen how every time we democratise access to something in the data domain, we eventually end up with sprawl.

• The OLAP wave democratised access to data. What did we get? A mess of cubes nobody could find or trust.

• The Tableau wave democratised access to dashboards. What did we get? Thousands of dashboards, most of them showing slightly different numbers for the same thing, or a slightly different number of data columns.

• The dbt/modern data stack wave democratised access to data transformation. What did we get? 5,000 dbt models with no actual data model, anybody?

Now we are about to democratise access to building BI apps. And the sprawl problem is going to make the previous waves look like a gentle ripple.

I have written thoughts about some of these anti-patterns in the new “AI” wave before:

But what if you already had the scaffolding?

Luckily we have spent the last seven years building out a data platform that is both opinionated and has all the scaffolding that should (in theory) help manage this inevitable sprawl.

So we started by creating a simple templating pattern in our platform. A way to quickly build and deploy a “one shot BI app” using your favourite LLM tool. Claude, ChatGPT, whatever you prefer.

Then we let it loose with one of our talented early adopters Network patterns, to see what he would actually do with it.

We were amazed with what he started creating.

Next we used it to build a prototype for a customer who wanted us to upgrade the capabilities they had been using with us for the last few years.

They loved it.

So once more into the breach, my friends.

“Can’t code, won’t code, don’t code” tries vibe coding

One of my frequent sayings is

But I wondered, with the latest tools like Claude Code and the latest models like Opus 4.6, is that still true?

I find I learn best by doing.

So I thought I would experiment with creating an app using Claude Code to see how the process would work for somebody like me. Somebody who understands the patterns and requirements inside out, but has spent decades successfully avoiding writing code.

Now I could have vibe coded a “one shot BI app” for a customer, or built something with public dummy data to experiment and learn.

But the Information Product Canvas was the obvious first cab off the rank.

I have been iterating that pattern template for over a decade. I know the 12 areas, how they relate to each other, what the user experience should feel like, and what the anti-patterns look like.

If Claude Code got something wrong, I would (hopefully) know immediately.

Plus I had “build an IPC app” gathering dust on my backlog for years. I never wanted to spend the moolah to pay somebody to build it.

Vibe coding supposedly changed the economics.

The result

One of my other common sayings (apart from #Whoot!) is

I published the resulting Information Product Canvas app as open source.

You can read about what it does and how to get it running over on the Information Product Canvas companion site:

Information Product Canvas

The Standalone Information Product Canvas App

Do you prefer the feel and joy of reading a physical book…

Read more

4 months ago · Shagility

And grab the code from GitHub:

But of course I didn’t stop at one app

For those that know me know when I find something interesting, something that looks like a useful pattern, but I cant describe that pattern with clarity results in something I can’t leave well enough alone.

The Information Product Canvas app scratched one itch.

But it also made me realise how quickly I could experiment with ideas and templates that had been stuck on the backlog for years. Pattern Templates like the:

• Business Event Matrix;

• Concept Models;

• Business Glossary;

• Layered Data Architecture Checklist;

• Data Dictionary;

• Data Contracts;

• Data Asset Catalog;

Ideas I couldn’t justify spending the time or money on before, but could now explore by whispering sweet nothings to Claude Code in the background for a few hours, while still doing other more important work.

So I have kept building. More standalone apps, all connected by a shared backend.

Some of them are getting close to useful. Some of them I still hate.

I will publish each of them as open source standalone apps as they get to a stage where I don’t hate them.

If you have been following my Context Plane experiments you will start to see how these pieces fit together.

https://agiledata.info/t/context-plane

If you want to help build them with me, just sing out and let me know, the more the merrier.

The bit I haven’t been sharing

As part of my Sharing is Caring mantra I realised I haven’t been sharing my journey as I learn the process and pros and cons of vibe coding something that needs to be actually used, as a person in the data domain who can’t code.

So as I keep working, off and on, on my vibe coding process and building out apps, I will post articles on what I have experienced and my thoughts around it.

What worked. What went sideways. What surprised me. What I learned about building software through conversation rather than writing code.

If you are a data person curious about vibe coding, or a builder wondering what it is like when someone who “can’t code, won’t code, don’t code” picks up Claude Code and starts whispering sweet nothings to it, hopefully the series will be useful.

If not, maybe turn off substack notifications for this site for a little while ;-)

It was the same pattern talks about in this draft book chapter:

Practical Data Modeling

What Data Modeling Is and Is Not

Here’s the revision of Chapter Two for Mixed Model Arts, where I discuss various definitions of data modeling and bring it into the present day. We are no longer modeling just for humans, but modeling for humans AND machines…

Read more

6 months ago · 32 likes · 3 comments · Joe Reis

A quote from that chapter:

<-o->

“I’d just been hired as a consultant for a mid-sized e-commerce company that was hemorrhaging money. Their CEO pulled me aside within the first hour: “Joe, our inventory system says we have 50,000 units in stock. Our warehouse says 12,000. Our website shows customers that they can buy things that don’t exist. We had to refund $400K last month alone.”

“I spent the next three days spelunking through their systems. What I found was a horror show. They had an ‘orders’ table with 500 columns. Customer data lived in six different databases, none of which agreed on what a “customer” was. The product catalog was a single enormous spreadsheet that someone manually uploaded to the database every Friday afternoon. Date fields were stored as strings. Some prices included tax, some didn’t, and nobody could tell you which was which.”

<-oo->

My data friend was telling me about the time and money the orgs CDO had spent implementing a “Modern Data Platform” and how they were now being asked to present what “outcomes” had been delivered for that expenditure.

Unfortunately when my data friend had to go and get a single number for the equivalent of Joes “Count of Stock”, they found multiple systems that had different counts.

And when asking various Subject Matter Experts (SME) which count could be trusted they all gave different answers.

But the SME’s were all aligned when they said the one count they didn’t trust was the count in the new data platform.

I joked that the CDO should probably google (or perplexity) the three envelope joke about now.

But realistically its not a joke. That is shareholders money that has been spent, its peoples jobs that will probably be impacted as a result of cost that seemed to have no value.

So lets say it again .....

End of rant and here is a suggestion.

If you are investing in new tools, or a new shiny data platform do three simple extra steps.

1. Define an Information Product using the Information Product Canvas
   
   You can learn about it for free here:

   Information Product Canvas

   Information Product Canvas a pattern template, to quickly discover and capture, data and information requirements, in a repeatable way, so stakeholders love them and data teams can build from them

   By Shagility

2. Take course on how to easily and quickly identify the value of the information that the Information Product will deliver.
   
   You can find his course here: https://maven.com/nick-zervoudis/dpm-value-course
   

3. Have you data team build the identified Information Product at the same time they build your shiny new Data Platform.
   
   And then use the number from #2 above to start justifying the value the new data platform is delivering.
   

The thing I love about experimentation is it helps me coalesce some divergent ideas that have been floating in my head for a while, into some semblance of order.

A bit like writing does.

I wanted to get a handle on the latest state of “vibe coding” so decided to experiment with building an app using Claude Code and Opus 4.6.

I picked vibe coding an app for the Information Product Canvas, its an app I have wanted built for a while, but the cost to build it the old way never matched the value I found people were willing to pay for it.

Vibe coding in theory reduced the cost, the experiment was would it?

You can see the results of a few iterations of the canvas:

Plus a side experiment into the world of the Context Plane (couldn’t help myself).

But that not the point of this post.

Nick Pinfold is experimenting with his teams Agile Data Ways of Working and freely sharing his journey via LinkedIn comments.

In this comment

https://www.linkedin.com/feed/update/urn:li:ugcPost:7432018978505617408?commentUrn=urn%3Ali%3Acomment%3A%28ugcPost%3A7432018978505617408%2C7432267116197765120%29&dashCommentUrn=urn%3Ali%3Afsd_comment%3A%287432267116197765120%2Curn%3Ali%3AugcPost%3A7432018978505617408%29

He talks about how he is creating a Streamlit app that allows him to capture the IPC content as Context and use it to assist with the next steps in their Information Factory.

posted a comment on this Substack post:

On how she is seeing “datapacks” that provide the content of a book in a way it can be easily used in an LLM as having some value.

Both of these show that there is value in writing books and creating apps, but that the typical moat of both of those things has changed.

Nick can vibe code a IPC app as fast as I can, if not faster

Anna can take the ePUB version of my book and use it in a LLM as fast as I can. If she pays more for tokens and the latest models than I do, she can do it faster and better than I can.

So effort and expertise are no longer the moat.

Given I have always said I “Can’t Code, Don’t Code, Won’t Code” but now I can now create an App but just asking questions, Im pretty sure Effort and Expertise is not the moat it was anymore either.

But maybe experience is.

To create my app in a way that meant it was actually useful, I had to have experience:

• experience using the canvas

• experience working with multiple data teams on the problem the IPC solves

• experience using apps to know what UX features were needed

• experience to know that I needed to add Google Auth to provide an easy login

• experience to know not to store any API keys in a way that were public

• experience to know that I wanted to use Google Spanner as the backend data repository and Svelte as the front end language

• experience to know I wanted a API layer between the backend data repository and the front end app

• experience to know …..

I don’t have an answer to what the new moat is, but I have more clarity on what it isn’t after these experiments.

Listen

View MindMap

Read AI Summary

Read Transcript

Listen

Listen on all good podcast hosts or over at:

https://podcast.agiledata.io/e/fact-based-modelling-patterns-with-marco-wobben-episode-81/

Listen to the Podcast Episode on Podbean

Subscribe: Apple Podcast | Spotify | Google Podcast | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser | Deezer | Podcast Addict |

You can get in touch with Marco via LinkedIn or over at https://casetalk.com

Google NotebookLM Mindmap

Google NoteBookLM Briefing

Executive Summary

This document synthesizes key insights from a discussion between Shane Gibson and Marco Wobben regarding Fact-Based Modeling —also known as Fact-Oriented Modeling. The central premise is that modern data modeling has become a “lost art,” leading to significant “business debt” where organizations lose the context and meaning behind their data due to silos and rapid staff turnover.

The core solution presented is Fact-Based Modeling, a methodology that grounds abstract technical requirements in “administrative reality” by combining linguistic terms with actual data examples. By focusing on how stakeholders communicate (Information Modeling) rather than just how systems store data (Data Modeling), Fact-Based Modeling allows organizations to bridge the gap between business subject matter experts (SMEs) and technical implementations. This approach not only ensures more accurate system design but also provides the necessary semantic grounding for emerging technologies like Large Language Models (LLMs).

--------------------------------------------------------------------------------

The Crisis of Lost Context: Technical and Business Debt

The current state of data management is characterized by a widening gap between what is stored in systems and what those records mean to the business.

• Evaporation of Knowledge: Senior experts with decades of organizational history are retiring or leaving, and the average job tenure (four to six years) is too short to maintain deep context.

• Business Debt: This is the cumulative loss of meaning within an organization. When systems are built without documenting the “story” behind the data, the original business intent is lost, leaving IT to guess the context of legacy records.

• The Context Gap: Technical optimization (how data is stored) often overrides business representation (how data is used). This leads to “tribal wars” where different departments use the same terms (e.g., “inventory” or “customer”) to mean entirely different things based on their specific departmental needs.

--------------------------------------------------------------------------------

Defining Fact-Based Modeling

Fact-Based Modeling is a methodology developed to capture domain knowledge by focusing on “facts”—statements about the business that are agreed upon as true within a specific context.

Core Components of the Fact-Based Modeling Approach

• Grounding in Examples: Unlike traditional modeling that looks at abstract entities and attributes, Fact-Based Modeling uses “data by example.” Instead of discussing a “Customer” entity, a modeler uses a statement like: “Customer 123 buys Product XYZ.”

• Binding Term and Fact: By combining the linguistic term with a concrete fact, modelers can identify misalignments quickly. For instance, seeing that one system identifies a customer by an email and another by a numeric ID reveals a transformation problem that abstract modeling might miss.

• Information Modeling vs. Data Modeling:

  • Information Modeling: Focuses on how humans communicate about data to reach alignment.

  • Data Modeling: Focuses on technical storage, structures, and optimization.

  • The Distinction: Information modeling is the “primary citizen,” while technical artifacts (SQL, schemas) are secondary outputs derived from it.

--------------------------------------------------------------------------------

Methodology: The Process of Fact-Based Modeling

Fact-Based Modeling follows a specific logical flow to ensure that complexity is simplified without losing essential nuances.

1. Scope the Domain: Identify the specific problem area (e.g., Sales, Emergency Room, Tax).

2. Gather Data Stories: Interview SMEs to collect verbalizations of how they describe their work.

3. Identify Business Constraints: Use interactive questioning to find the “rules” of the data. (e.g., “Can a citizen be registered in more than one municipality at once?”)

4. Identify Exceptions: Use the data examples to flush out the “edge cases” that SMEs often forget until they see a specific record.

5. Alignment through Generalized Objects: When different departments use different identifiers for the same concept (e.g., Name vs. Email), Fact-Based Modeling uses “generalized object types” to link these different views into a unified communication framework.

--------------------------------------------------------------------------------

Strategic Value and Modern Application

1. Automation and Efficiency

Fact-Based Modeling allows for a “context-first” implementation. Because the model is rich in semantics and constraints, tools can automatically generate:

• SQL for database creation.

• Data Vault or normalized models.

• Database views that represent the original user stories.

• Test data derived directly from the interviews.

2. Grounding AI and LLMs

LLMs are proficient at generating “fabricated stories” but lack organizational context. Fact-Based Modeling provides the “ground truth” needed to keep AI outputs accurate. By feeding an LLM the terms, definitions, facts, and business constraints from a fact-based model, the AI can perform tasks with a much higher degree of reliability.

3. Avoiding the “Generic Model” Trap

The document highlights the failure of massive, pre-built industry models (e.g., the IBM Banking Model). These models often fail because organizations do not know their own “edge” or specific context. Fact-Based Modeling allows a company to capture its unique business logic rather than trying to fit into a generic template that ignores their specific reality.

--------------------------------------------------------------------------------

Notable Insights and Quotes

On Complexity and Simplicity: “If the end product is presented and everybody goes: ‘Wow, is this it? Did it really take you that long... I could have done this,’ then I succeeded in making something very complicated very simple to understand.” — Marco Wobben.

On the collaborative nature of solving data ambiguity: “It is somehow a team effort to slay this beast of miscommunication until everybody agrees and understands each other. ... It’s all about, working together and trying to fight it. What are we not seeing? What are we missing? How do we tackle this? And a lot of that is just human interaction.” — Marco Wobben.

On the Definition of a Fact: “A fact is a piece of data that’s physically represented somewhere... I can point to it. It has been created. I’m not inferring it. It is something that is factually there.” — Shane Gibson.

On Party Entity Data Models: “The most expensive part of our systems is the humans and [understanding] that context... as soon as you design a system with ‘thing as a thing’ and that context lives nowhere else, I now have to spend a massive amount of expensive time trying to understand what the hell [it is].” — Shane Gibson.

On the concept of “business debt” created by rapid technological change: “This is the paradox where business wants to have changed faster. And it ruined the party by saying, we can deliver faster with this new latest tech, but neither party realized what they were losing along the way. So it’s technical debt, it’s business debt.” — Marco Wobben.

--------------------------------------------------------------------------------

Conclusion

Fact-Based Modeling serves as a bridge between the human understanding of business processes and the technical requirements of data storage. By prioritizing the “authentic story” of the business and grounding it in real-world data examples, organizations can mitigate the risks of technical and business debt, ensuring that their data remains a usable, understood asset even as technology and personnel change.

Transcript

Shane: Welcome to the Agile Data Podcast. I’m Shane Gibson.

Marco: And I’m Marco Wobben.

Shane: Hey, Marco. Thank you for coming on the show. Today we’re gonna talk about a thing called fact-based modeling, but before we do that, why don’t you give the audience a bit of background about yourself?

Marco: Ah, yes. Thank you. Thanks for having me. Yeah, background a lot of background there. I’ve been around a few decades. I first fell in love with computers when I was still in secondary school. That got me hooked when somebody showed me the break key on a keyboard and we could stop games mid play, change the code and resume.

And that was like, oh, this is magic. I need to figure out how to make this my toolbox. And I enjoyed making software, hacking software working with computers ever since. And. From getting a job onboarding people on Microsoft Windows and Office. Back in the day, I decided to just chase my own career, quit the job and started doing entrepreneur work, made custom software from design to end product for a number of startups. And then somewhere early two thousands, a professor knocked on our door and said, we have this source code of a modeling tool. And the kids that graduated on it, with, they took a different path and we need somebody to maintain it. And ever since early 2000, I’ve been working on fact-based modeling that I had to learn from the inside out.

So that’s where I am now. I’m being considered the expert currently. ‘Cause a lot of professors retired and the young people haven’t caught on for it yet, just and here I am talking about fact-based modeling or fact oriented modeling, if you will.

Shane: Yeah, it’s interesting. We before we started, we talked briefly about the fact that data modeling’s become a lost art. And actually, I think it’s coming back now with all the AI focus, it seems that data modeling is a term I’m seeing used a lot. But, if you think about your career path, that idea that you could start out with games as a way of introducing yourself to, to computers in the early days.

I was like, you I started out. We had a couple of computers at school. The old I think they went green. I think they were amber screens back then. And yeah, again, I got hooked by the games. And I wondered whether that’s the thing that’s been missing is gamifying data modeling.

Like actually making it exciting is, it’s probably the missing thing, right? Is if there was a game that you just happened to date, a model when you did it. Maybe that’s what we needed to make what we do sticky with with people that are coming up in their career.

Marco: that’s an interesting take on it. it’s interesting looking at my users, I’ve been maintaining this information modeling tool for years now, is that there seems to be more. Interest from people that are curious and like doing things right and talk about things. And this is, I can see that in, in a lot of gaming communities where, you know, in the old days it was like you play your game single player with this computer, right?

But gamification nowadays is so much more you can team up and you none of that stuff was there in the early years. Even graphics were not there yet. The communication aspects right nowadays is more and more prevalent and important and it’s. If you look back, what data modeling really is to get the technical requirements of what people actually needed the computer to do in store and how to manage it.

Seeing that come back a little I dunno if gamification would help, but there’s definitely more openness to let’s talk about it. And you can see it a little bit in the agile phenomena where there’s a lot of standups stakeholders, product owners, and everybody starts talking and communicating with each other.

So that’s definitely on the rise. I’ve seen a few data modeling efforts that actually try to gamify it and it’s like a, they have this data modeling tool and it has flashing text and animated tables and, I’m not sure if that’s really the end goal, but I can see how it might help.

Shane: Yeah. I was thinking more about gamification as in the adrenaline rush when you are successful in solving a problem not the flashing things. Because earlier in my career, I worked for a software startup in the accounting space called X. One of the reasons that they became successful was they had this gamification of bank reconciliation.

So pretty much you had bank transactions on the left and your invoices and all that on the right. And whenever a transaction came up on your bank statement, you pretty much dragged it and matched it to the right. And then that road disappeared and it was a very simple gamification process.

But there’s this adrenaline rush going in and seeing your bank rec with 50 transactions you haven’t reconciled, and just going bing, bang Bosch, and it becomes clear. And, I don’t, it didn’t have confetti. Actually that was one of the big arguments at the time was should it have confetti con?

Marco: more like a Tetris road disappearing from your screen.

Shane: Yeah. Yeah, exactly. And so you sit there going, maybe that’s it. So as I think about it more, when I’m conceptually modeling the adrenaline rush is when I create a map that I can show to a stakeholder and they just nod and go, yeah, that’s business reality. I get it. Yeah. That’s how we work.

When I logically model, it’s this idea of yeah, I can take that conceptual model and I can slam it into the modeling Pattern that I use and it makes sense. And then when I take that logical model and I make a physical model and the cloud database actually takes it and I can actually query it fast and it doesn’t cost me a fortune, and any question that I get asked can be answered with that data.

There’s that adrenaline rush, right? That gamification of each of those steps adding value to my life or somebody else’s. I don’t dunno, I just, I haven’t thought about it that way until you mentioned that you started out hacking games.

Marco: That is, it definitely is true for me because, as I add features that support user functionality and all that, and it’s like there’s definitely a rush when I see people use it and they go oh, this is handy. This is practical. This, makes my work easier. Then I get the adrenaline rush. Definitely on the modeling part itself. It usually goes through long cycles of deep talks about the subject matter at hand, which, there’s definitely an adrenaline rush, but not necessarily always the good ones. As a, as an example that I used a lot and some other authors put it in their book too, is I had a long interview with the subject matter expert and it took us about two hours to figure out one requirement, and there was a lot of talk on the type level that created confusion because, yeah, what is a customer really? So I have to get through that. And in the end there was a little bit of an euphoria, which is, the adrenaline in rush, if you will, where we finally nailed it. The subject matter expert re replied, somewhat baffled and in disbelief, and he says, I had no idea my work was that complicated. And then being able to write it down in a way that everybody suddenly understands. That’s definitely a moment of adrenaline and rush, if you will. It takes two hours of hard work. It takes a lot of interviews, a lot of digging. And then finally when you reach that point, and it’s like one of, I remember a quote in a book I forgot which one it was, but I the quote really spoke to me and he this man was speaking about information modeling on a data modeling level.

And he said if the end product is presented and everybody goes. Wow, is this it? Did it really take you that long and work so hard to just present this? I could have done this, and then his reply was then I succeeded in making something very complicated. Very simple to understand.

Shane: And that’s the key is to take that complexity and describe it in a simple way where everybody nods and either agrees or disagrees. I remember one where we literally spent three months with an organization trying to get the definition of active subscription. And the problem was we had three business units.

Effectively three domains that all had different definitions, but couldn’t agree. They actually had different definitions. It wasn’t around the model itself, it was around the plain language description of that term where either we added a term in front of it, marketing, active subscription, finance, active description.

So we were clear they two different definitions or they actually agreed active description was described in this way. And that human engagement was where the time was spent not creating a map with nodes and links on it. Not creating the database, but without actually getting to a stage that we ever agreed or disagreed with that term.

There was minimal value carrying on because we would present a number that people don’t agree with because the definition’s wrong, not the number.

Marco: in the years of information modeling that I, I call it information modeling, but it’s really just a fact-based modeling underneath as well. But I’ve encountered so many synonyms or harmoniums or it’s just, people just completely get confused and it becomes almost like a battle.

But here’s the thing, and I think you explained that as well. It is somehow a team effort to slay this beast of miscommunication until everybody agrees and understands each other. I just recently started watching the latest series on Stranger Things that it’s we have to team up to fight this monster from the underworld, which is, we can’t see it. We know it’s there. And it’s like, how do you fight it? And and it’s all about, working together and trying to figure it out. What are we not seeing? What are we missing? How do we tackle this? And a lot of that is just human interaction.

Shane: And finding ways of taking that expertise that we have that ability to take complexity in a business organization and try and create a map that has simplicity that we can all share. That is a skill, And it’s how do we take other people on the journey without going into a room for six months on our own or creating really complex ERD diagrams with many to many crow’s feet that, few people understand, And terminology is really important. And so it’s interesting that you talk about information modeling and then you also talk about fact-based modeling. ‘cause as soon as I heard of fact-based modeling, I naturally go to dimensional modeling and star schemas because that is where I first heard a definition of the term fact.

And my understanding is you are talking about information modeling rather than anything to do with dimensional or star schemas. Is that correct?

Marco: Yes. That’s funny ‘cause your first take on the word fact is how I met my wife at a data modeling conference in Portland in the USA. She was like, oh, fact-based modeling I’m doing something with data warehousing. I should go to that class to listen what this man has to say.

And it was just not the same fact. So even there, even in it, we have confusion of words, but the word fact really boils down to something that is maybe a little older even where database records really store facts as they happen in our administrative reality as I call it nowadays.

Because there’s a lot of the single point of truth. We need to get the truth out there, the reality and all of those things. But there’s something seriously flawed with that, is that we all perceive from our own bias and subjective reality, the world out there. So there is no such thing as truth. But when you store data and you consider that to be true in your world, then you can state that as effect. Effect as in, I’m writing this down and me and my colleagues agree on it. And so that’s where the terminology came from. Nowadays we trying to figure out, maybe we should call those claims instead because we all say something and we all think it’s true.

And sometimes, especially in data warehousing you collect data from different source systems. Yeah, but you can’t just say that they all state facts because some might be alternate facts. So let’s put it as all the source systems claim a certain statement about this is what happened.

Shane: it’s interesting, ‘cause we’re talking again beforehand about how we’ve both been in the domain for quite a while, but we’ve never really crossed paths. And I’ve heard your name a lot, but I’ve never really read a lot of your stuff. And one of the things I did do, I had to train a new team moving to the data space.

And so I was trying to describe the difference between facts, measures, and metrics. And what he ended up coming up with is a language definitions that I used and the definition I used was a fact is a piece of data that’s physically represented somewhere. So if I go into a database and I see a number.

That is a fact. If I go into a spreadsheet and I see a number or a piece of text, that is a fact because I can prove it existed. I can say, that factually it’s there. I can point to it. It has been created. I’m not inferring it. It is something that is factually there for me. And that’s kinda why I use fact.

And the reason I raised information modeling versus dimensional modeling is ‘cause as soon as I use that term fact, anybody in the data domain goes, oh, you’re talking about a fact table. And I’m like, no. And then for me, I defined, measure as a formula, some of this, that kind of thing.

And then a metric is a complex formula. So this over this based on that at this point in time. And so for me, I didn’t mind whether people disagreed with my definitions as long as they gave me a different definition. But that’s the three terms that I used that seemed to get clarity and understanding when I was talking to people who weren’t data experts.

So yeah, I go back to the true definition of that term. Fact is not a fact table and a dimensional model. And, maybe yeah, should we move to claim or should we just bring back the true definition of that term, that’s one of the problems we have in our domain is we have what we call pedantic semantic arguments about the the most non-interesting things for.

Marco: We are not gonna solve that because there’s so much when I do actually information modeling and we can come up with the word, I’ve used it in, in, in different environments, but we can all agree upon what the definition is for the word inventory. It’s the amount of things that we have and offer certain article, but then you go into different departments.

Sale comes up with three because they already sold a few. Purchasing says, we have eight because they already ordered a few. And then you go into the warehouse and the guy goes but I only have one on the shelf. The, what the hell is going on? No, even though everybody agrees they have different data. And as soon as you, you and I would speak about facts, then, I could say that the customer buys a product and we agree and we call that a fact, but it has nothing to do with the data at all. So the word fact itself is like inventory, is location is like customers that you can apply it to anything and it doesn’t mean anything. So getting too hung up on it. Is very tricky because then you will start tribal wars because no. This is what the definition for a fact is. But the reality is that the word fact the linguistical part of it, the term of it is used in different contexts. So if you wanna use it within your dimensional world, it’s fine. I’m not gonna argue with that. Is similar to calling something red, we will find it in different environments. I’m looking at a book that has a red cover. You look at the fire truck is also red. It’s, oh, we’re all good.

Shane: Unless you’re in a country where the firetrucks green. And yeah, it’s interesting ‘cause Remco Broekmans talks about the example he has of definition of a bed in a hospital. And where one group basically said, it is the metal frame that the patient sleeps on. And another part of the organization said, nah, hold on.

It’s the room where the patient’s located. And so if I looked at the data, I would see one was probably two meters by two and a half meters and the other one was probably five meters by five meters. And then I could say the fact that this beard is five meters by five meters and has no wheels confuses me because I’d expect it to be two by three with wheels.

Maybe that data’s gonna gimme a hint that the definitions are different. So getting into that. can you just gimme a helicopter view of how fact-based modeling works.

Marco: You already gave me some beautiful examples is that to distinguish the things. You also have to look at the data and what I see happening in the data modeling space is usually. The data is not that relevant. We’re all looking at tables, entities, classes and what kind of attributes they have, what columns need to go in and what are the relationships or foreign keys and all.

So there’s a lot of technical views on it, and some of it may be guided by the data at hand. But the data itself is a secondary citizen. And as I just stated with the example of inventory, is it’s only by looking at the data that you start realizing, wait a minute, we’re all calling this inventory, but we’re seeing different things.

So there must be a distinction. And instead of having, this, I call it these tribal wars, these linguistical fights about no, that’s not what I said. This is what it means. And all of that. I call that. Type level arguments, type level discussions. It’s abstract in a way. So what factory ended modeling does is two things, is first, how do I talk about my data? And I use the data in the expressions, so I’m grounding it. I’m not talking about customer buys product. I’m saying customer with customer number 1, 2, 3 buys the product X, Y, z. Where both 1, 2, 3, and x, YZ is actual data, is real examples to ground the discussion.

Because, if I use inventory and I don’t specify that, I look at it in a certain way by giving the data, we will not discover that we might have a difference of perception there. So why the factor oriented modeling or fact-based modeling is that really we need to figure out how. I talk about it, how you talk about it and how we can talk about it and ground it with actual facts, this actual data it’s not enough to say the customer buys the product because you and I would agree, but it’s only by giving the actual data examples that we might start realizing, wait a minute, I have numbers to identify my customers, and you might have email addresses to identify your customers, so there’s something else going on. So it is really digging that one level deeper into it, and not letting that go because in the fact-based modeling, we keep tying the examples data with the language, with the fact types. So it’s always that package so that we can at any point, transform our information models into any kind of artifact, but still show the example data to accompany the definitions, to accompany the structures and to illustrate that this comes with a specific data use. And I think that’s the biggest difference from what you look at data modeling or dimensional modeling or data vault modeling is that it’s all on a type level and it’s usually geared towards how do we structure the storage of data. And it’s less about how do you and I figure out how we talk about the data and how do we quantify with data that we’re talking the same thing?

Shane: What I found interesting was so I’ve used the who does what Pattern a lot, And I think, I probably got it from Lawrence Coors Beam stuff. ‘cause that was some of the stuff I read earlier in my career. And I’ll often talk about customer orders, product from employee in store.

And then I’ve also used data by example a lot. And again, I can’t remember where I found that patent, but I found it valuable, so Bob purchased three scoops of chocolate ice cream from Jane at 1 0 6 High Street in Swindon. And what interests me about the fact-based modeling when I had a quick look at some of your presentations, is it seems like you are combining both those patterns.

So you are combining both the term and the fact into that data story. So you are saying, customer Bob purchased product, chocolate, ice cream from employee Jane at store 1 0 6 High Street.

Is that what you’re doing? You are combining the term and the fact together to give much more context around that business reality to then help you in the rest of the work.

Marco: yeah, definitely. It’s very much that approach and of course tools give you different functions to, to do it in a different way. But the theory that was even developed in, mid seventies and continue to develop all the way up to the end of the nineties was really about how can we write it down in a way that non-technical people can actually read what it means.

And in the interviewing phase, in the workshops, in the explorative areas, is that, as I’ve given the examples earlier is that it’s not enough to just say, okay, we have a customer and we need a data system for that. It’s like we needed to also figure out how you talk about it.

And I think what is an increasingly more difficult problem is that where in the seventies, eighties, even nineties a lot of systems were built for a specific purpose, for a specific department, for a specific system that everybody knew the context of. So now if I’m doing a customer registration, then I’m just doing just that for my department within my group of employees.

And we all know the context. So everybody understands that if I put customer there, they all know what customer is. Now with the increase of it is that. It added a computer on every department, in every system, But they never integrated. So this is where data warehousing came in. But because it was very high context, implied the data warehousing, the BI team now had to figure out, okay, what if we pull the data?

What does it even mean? But we’re really trying to rediscover the context of where it is used. The human aspects of understanding need to be reverse engineered on top of the data structure. And this especially goes for what we’re currently seeing with all the LLMs. Yes, it can give great semantics or storylines, but does it really understand the context?

So this is another gap. And what the fact oriented modeling approaches is that we try to interview and capture the knowledge of the domain and subject matter experts. With all the context, with their language, with their examples and try to line that up with the data, the language and the meaning of some other department that uses a different system. And not just to be able to talk about that specific system, but to also align the communication across departments, across contexts. That’s really what got lost. If you have individual silos that you want to bring together into a data warehouse solution, that the missing piece is the original and authentic communication about their systems in the first place. And then the factor oriented approach allows those subject matter experts to talk about it in a way that they understand and can be verified by colleagues and can be transferred across. Context. And I think that’s the real added value that when systems were built within a specific context, the value of that was not really seen because it served the context.

Everybody knew that. where you see now an increasing in interest in, okay, we need to get back to our ontology, we need to get back to our semantics. We need to get back to meaning, we have to get back to information. So all of that was there and it’s still there if you don’t throw away the baby with the bath water.

Shane: I agree. I, we go through waves, so we see a wave right now outside the AI wave. We see a wave of bi me metric layers. This idea of having a layer where you can define a metric and then regardless of which system, the data’s coming from or you are hitting that same metric’s been applied.

And I go well, you know, years ago in the old BI tools we had end user layer or a universe. We’ve had that Pattern and then we lost it, and then we get it back. I’m also with you in terms of the way systems have evolved. We’ve gone from a mainframe where we had one system where there was a term called customer and there was one fact and that was it.

And then we went to client Server N Tier, and we ended up with seven. And then software as a service, 50 to a hundred. And with the AI wave now and Gen ai, we’re gonna see these one shot apps. We’re gonna see thousands of things that have created for one use case get stood up really quickly, and then probably disposed of.

And so this ability to have a language where regardless of the system and how many we have, we get shared context is really important.

And that example that you used, I just wanna come back to that because I hadn’t thought about it that way. So if I said customer orders, product. I have a thousand systems that involve a customer, an order, and a product.

I have absolutely no context whether they’re defining the same things the same way.

But if I use this data by example, if I say customer, Bob, ordered product, and then in another system I see customer bob@gmail.com ordered product, and in another system I see customer 1, 2, 3, ordered product, and in another system, customer A, B, C, ordered product.

What I know now based on Pattern recognition, is that I have a problem with the unique identifier of customer.

Like I can tell that in 10 seconds by seeing those four data examples and it’s something I know I need to solve.

Because to mash that all up, I need some form of conformity, right?

Some form of either shared identifier or a way of mapping it. And so I can see by this combining this term and this fact together, it gives me the richness to understand the problems I need to solve much quicker than any of the other techniques where they’re separated.

Marco: That’s true and it’s, you’re still some somewhat catching an optimal path because you already assumed in this example that all those systems had something called customer. And in one system it’s probably CST. Then you have to combine it with SAP, where it has X 3 0 4 as a table name and then you have a very abstract table called persons. So you can see where this is going because all of those systems, the storage and the structuring of the data in that system served only one purpose. Optimizing the IT part of it, the IT end product presented the data in a way that the business wants to work with it and wants to see it, but it doesn’t represent how it is stored. The storage and the management of the physical data is technically optimized. It is not with the business representation in mind necessarily. And this is what a lot of people in the data space have encountered too. It’s okay, now I have 200 source systems and I need to figure out what is what.

So is this email address, does that indeed correspond with my customer? In the other system where it’s identified with 1, 2, 3, is that even the same customer? Is it a customer in the first place? So there is so much not just technical debt for systems that are undocumented or not well documented or behind in documentation, but also a, what I increasingly call the business debt is that nobody knows what that meant to the business anymore. And this is the paradox where business wants to have changed faster. And it ruined the party by saying, we can deliver faster with this new latest tech, but neither party realized what they were losing along the way. So it’s technical debt, it’s business debt. In governments it’s even worse because there’s a massive gap between the legal articles made up by politicians full of compromises and loopholes to the actual systems used by government bodies.

And, now we changed the law. Which system did we need to change or vice versa? We’re looking at data here but we have no idea if we’re even allowed to have this data or even be able to look at it because we don’t know the legal articles with it. So everything in it scaled up in the past 30 years so quickly. It totally got outta control in a way where the next tool’s not gonna solve it. An LLM is wonderful, magical stuff, but it’s not gonna solve the real thinking issues. We can do data profiling because we were still not sure if we got the context right. We can do LLM, but we are still not sure if the relationships are correct. So there’s still that gap of knowledge that we lost and somehow need to reintroduce to make things really work.

Shane: And actually that’s interesting around that organizational context and losing that knowledge. Because if I think about it, if I was walking into a new organization and I wanted to understand the context of that organization, my natural technique was to find somebody who’d been there for a long time as a subject matter expert, that person that’d been there for 20 years because they’ve got the stories of how that context happened and why it happened.

And so moving to the uk I had to set up bank accounts. And so I went, and there’s a whole reason that I needed to do a UK domicile bank rather than one of the newer, easier to deal with banks. And so I went to one of the main banks, I created a personal bank account.

Took a while. All good. And then I went back to that. ‘cause now I’m a customer of being identified. In theory. Everything is easy. And I tried to create a business bank account and it forced me to create a new identity. I had to go through the whole identity process, even though I used the same email address, which was my form of identification, my identifier, I had to go and revalidate myself that I have a same address, same passport number.

And you sit back as a customer and you go, that’s just crazy bollocks. And then I talked to somebody that had worked for that bank, that subject matter expert who’d been there for a while. And they said, yeah, but you gotta understand that was two different banks, two different systems, and they haven’t been merged.

And therefore you may think you’ll be dealing with the same organization, but you’re not really. And I’m like, yeah, actually. Okay, that makes sense. Now why? Why did I not understand that working in data so often, but if we go back to that example of term, in fact, so yes, if the term changes, so now I see a customer and I see person and I see prospect and I see X 2 0 3.

If the fact is the same, if every one of those is customer Bob at Gmail X 2 0 3, Bob at Gmail, prospect Bob at Gmail, again, I’m getting more context, I’m not getting an answer, but I’m getting more information that can help me understand a problem to be solved. And as we know with data, there’s so many problems to be solved.

It’s such a complex space. When we hit reality of organizations, the way they work, the terms they use, the systems they use, the way they create and store data. But this idea of binding term and facts. Gives me some more hints because now I can say they’re all the same email address, are they the same term?

And then somebody will say actually no, when you see prospect and customer, it is a different rule and then somebody else will go. But you do know that they can change their email address whenever they feel like it. And you’re like, yep, seen that Pattern before. Okay, so it’s an identifier, it’s a unique identifier, but it’s not a consistent or persistent identifier.

There’s all these patterns and data that we know are gonna hit us, but by binding that term, in that fact, I get some hints at the beginning across multiple subject matter experts. So I can see real value in, in that part of the process early. And so talk me through that, You drop into a new organization.

You wanna start off with fact-based modeling. How do you do it? What do you actually do?

Marco: It’s a funny question. I, a lot of people ask me that, how do you start it? If you open up the books about this topic that either are written as a university. Proof of concept all the way to, self study. It always starts with gather your sources, figure out what is the domain about in the first place.

So there’s very much a almost top down approach where you go okay, we got sales, we’ve got production. so it’s the general area. You can’t just jump into the jungle and start describing all the little insects on the jungle floor.

It’s that’s not how it works. But usually there is a problem domain, there is an integration problem. and. So what needs to happen is that you need to be able to carve out some time with at least some business domain expert subject matter expert to sit down and say, okay what’s the issue here?

What do you, what are you doing? What does it do? And start writing it down. so far, not that much different from actual data modeling, if you will. But the distinction starts with the nitty gritty where you need to get things right. And it’s usually getting things right, not just to verify if you understood correctly, because that’s already a hard part as my two hour example earlier showed to, just get one line of requirements, correct. But that back and forth with language and examples is of great help in getting to actual understanding. But the major part is usually you need to work with a colleague that also needs to understand it. And with the current short-lived career jumps, if you will, is that you may find an expert, but he might be gone in four hours or as is happening currently as well. A lot of seniors that actually know the organization, they’re getting in a pension range and they just leaving the company. And what is left behind is usually short lived career steps, short lived managers that move on. There’s a lot of tempo where at the average job years is four to six years. So the knowledge actually evaporates while we’re looking at it, while we’re trying to document it. and that’s where the difference starts to rise between traditional data modeling that creates diagrams and type level schemas into a, if you compare that with factory and modeling, you have real user stories that you can, with a click of a mouse can pull up and you can read how it’s being used in language in the organization and how it’s being transferred to other departments. And I think that securing that knowledge, that semantic rich document, if you will that is something that I see that gets lost with the more traditional, more technical data modeling. It’s, yes, you have the traditional layers of conceptual, logical, physical. But still, there’s not really a story there. And the people that I talked with and explained this kind of stuff is that they all recognize what I’m describing and a lot of the architects and a lot of the data models will reply with, yeah, that’s what I do in my head. And then my question is, my obvious question is but does it leave your head, does it get written down somewhere so that if you leave, your colleague can take over? And then the answer is usually no. There might be a document somewhere, describes the use case, but that very quickly gets evaporated as well, because everybody starts looking at the artifacts and the technical schemas. So by having an environment where you tie it all together, that you cannot do one without the other. That was really the grounding for the discussions the proof of the pudding, if if you will. But it also gives you the anchors going forward to technical artifacts. So by capturing that from the domain experts, putting that in an information model and being able to generate technical artifacts, even SQL to generate database, it will still comment all that SQL with all the written language and examples. It will generate a database with test data that came from the interview in the first place. It will add database views representing the full user stories on top of the production data, representing the interview. So it really is that don’t throw it away, don’t throw it away. Keep it as long as possible so that everybody is able to understand and read what the actual data is and what it means and how it’s communicated.

Shane: that’s interesting because that is a form of context first implementation. And what I mean by that is with our product, we define the context first and then the technical implementation is hydrated. So I’ll create the context of a business rule, change rule, and then our system will hydrate that into physical tables and SQL transformations.

But that context that I create is the key thing we, we care about, right? That is our pet. The way we deploy it and run it is our cattle. And what we’re seeing in the new Gen ai, LLM world is that context is actually far more important for the LMS and the physical implementation of that context.

And then as I said, we’re I’m working with Juha Coer at the moment trying to write a book around how to concept model. And one of the things we came up with is one of the steps is define your domain scope. So like you said, find a subject matter expert who understands it.

The next one is get data stories. And then after that, identify events, then concepts, and then connections or relationships. And the key thing is, as you said, is when you talk to experts in this process, if they can articulate the steps they take, those steps are often common. They might use different terms, and they might do them in slightly different order in slightly different ways, but we all do it the same.

If we think about it consciously, it’s where we bring in patent templates, where we bring in artifacts that we use repeatably, that we get that repeatability, that ability for that context to be stored in a way that another person can use it.

So in my view, yes, it’s great if we have a system that does all that for us, but we don’t have to, if we just have templates that are reusable, that’s valuable on its own.

If we have a repeatable process that’s not a methodology, right? It’s not fixed, but it’s just a way of working that is valuable because it’s bringing that knowledge back. So I just wanna take you back to something that you said, So if I take this idea of term, in fact having massive value and that we get that with a subject matter expert, and by documenting it in that format.

That context is able to be seen and understood by many people other than us. You then said that it’s also a way of getting alignment. So if I get that term, in fact, if I get that fact-based model for inventory from three different domains,

How do you deal with the alignment problem?

Marco: I can illustrate it by an example. And let’s stay close to the examples that we already mentioned, but it’s more powerful, more generic and more diverse than that. But I think we would need a podcast of another two days that would explain all of the ins and outs, but, so really to just show you how that would work. And I have to for the listeners doing this in an audio only podcast, I’ll try to make it as visual as I can. So when I say Marco Ban lives in Urich, which is the city of, where I live, is that would be effect that you and I can agree on and, me and my wife, we definitely agree upon that.

So let’s consider that effect and the statement that is to hold some truth value there. But there’s something else going on because Marco Warban lives, INTA is a state statement that, my wife lives in nut, my kids live in, so there’s a bunch of statements.

They all express something that we can classify as the city of residence. So by stating multiple examples like that, we can type those kind of statements as city of residence as the fact type. Now within me saying Mark of wo lives inre, I actually embed knowledge in there, even though you and I can agree upon the actual fact. I’m also saying, wait, Marco Ban is the citizen, is the city. But it goes a little deeper because my citizen is not really a citizen. It is just how I identify a citizen. And in that identification I can see, wait a minute, there’s a first name and there’s a surname. Similar with the city of Urich. It’s not really a city, it’s we’re representing the city by storing the name of the city. So you can see that if you can visualize that there’s knowledge, almost like a graph in there where I started with the city of residence. I gave it the semantics lives in, but it also has structure. It has the citizen, it has the city, it has the first name, the surname, the city name, and all of that ties together into this single fact statement. Now I can populate that with different examples. I can give my wife a position in there and my kids. And so it is populated with all kinds of example data and then the subject matter experts is then post with a series of questions. Could it happen at the same time that Marco lives in re as ma Marco lives in New York?

And then he would probably say, no, there’s something wrong with that. By going through these interactive sessions, which is almost like a gamification of the interview if you will, is you discover the business constraints and then by discovering the business constraints, that will lead to a certain structure.

When it comes to data modeling. If I can live in only one city, then the city is probably an attribute in the citizen’s table. If I can have multiple cities where I can live because that’s allowed in our register, then I would probably need a linking table in the end in physical model. So these constraints steer how the data is structured. And this is the interesting byproduct and I’m not sure if I’m still on track of their question, is that. In the interviews with subject matter experts, we can find ourselves very easily in hours long sessions about, what is a citizen. But as soon as they cannot give me a proper example to illustrate what they’re talking about, I’m talking to the people that are working outside of their scope of expertise. So that helps steering that. So there’s an organizational alignment in my efforts to find the data, illustrating the information. Now, the alignment on the other hand is now I start identifying Marco at some Gmail address lives in. So now I’m identifying the same citizen, but I’m using an email address. Now, obviously, in, in official organizations and registers that would never do, but, let’s suppose we have a small tennis club and it’s fine, so what I find now is that I still speak of city of residence. I still speak of citizen and city name, but I don’t have first name and surname.

So suddenly that citizen, where it used to be first name and surname now is an email address, but it still identifies a citizen. So what happens in the information grammar, if you will, is that there is something introduced called a generalized object type. My citizen can either be identified by first name and surname or by email address. So in the modeling part, it’s very easy to find statements that either generalize, which means that it allows different ways of identification for it, or different ways to talk about it. I now, I am presented with a data challenge because which combination of first name and surname goes with which email address. So again, I need a different fact statement that now would introduce, Marco W has an email address called Marco such and such at Gmail, which links the two keys. So nothing in my communication has to be altered to support alignment of different ways of identifying it, doing data mapping, just as part of the verbalization. I have a department here that only works with names. I have a department there that only works with Gmail addresses. I can make them talk to each other ‘ cause that needs to happen sooner or later. And the way that they talk to each other is say yeah, you’re right. This market woman corresponds with that email address because I have proof of that.

So then suddenly you have a mechanism introduced in the communication on how to identify certain entities in your data administration in a unified way. So this is different examples on how these alignments work on both semantic level, on a data level, on an identification level. In short.

Shane: I was thinking about it slightly differently, but it’s exactly the same process, I think. So let me play it back to you where I think I got to. you are talking there about, okay, we have this term and the term has the same context definition but the facts that identify that term are slightly different, right?

And we can then do some alignment where we see these different identifiers.

Marco: Yep.

Shane: The one that I was thinking about is where a domain or a business unit has a completely different definition for that term. And another one does. And we identify that. So let me give you an example that’s real for me right at the moment around citizenship or residency.

Marco: Oh.

Shane: So I think I know what made me a resident in the uk. It’s when I got a certain kind of visa and I entered the country. And as soon as I did that, that’s the rule that tricked the tick in the box that I am a resident of the uk but when I deal with tax residency, it’s a different set of rules.

And so they’re slightly different, I have to live here, but then also I have to make sure that there’s certain things I don’t do anymore back in my original country of tax residency. And so they’re both residencies, but they’re slightly different. And then how would I articulate that using term and fact, How would I fact based modeling it. And the thing that you talked about is this idea of business constraints. If I can describe the business constraint of what our residency is versus the business constraint of a tax residency, again, now I’ve got two patterns and I can look at those two patterns, those two constraints and say they’re the same or similar or they’re very different.

And in this case I’d say they’re different, One is around how my financial transactions work and one is where my ass sits when I have breakfast for the majority of the year, ‘cause I can spend a certain amount of time outside the country and I am still a resident of this country.

And then I think the other thing that this Pattern gives us, which we know happens a lot because we see humans do it, is as soon as I give somebody those terms and facts. And a set of business constraint, a set of descriptions of how they behave. Humans love to point out exceptions, especially subject matter experts.

It’s yeah, you could be a tax resident in the UK if you do that, but actually if you do this one other thing that nobody ever tells you about, actually you are not That’s the exception. Oh. And by the way, system A knows about that, but System B doesn’t. And humans are really good at pulling out that.

I dunno, what would you call it? The knowledge that only they have. And like you said, when we used to work for organizations for 25 years, that knowledge state in the organization, now you’re lucky if it’s five. So that context, that knowledge of those exceptions disappears. I can see how this idea of defining or articulating business constraints, identifying exceptions, finding them to that term, in fact Pattern in an artifact that’s repeatable.

We can now shortcut, like you said, the need for a data expert and a subject matter expert to spend a week in a room going through this magical process that only two people can do. And make it a little bit more, not democratized, but a little bit more accessible and repeatable.

Marco: it’s an interesting example, but again, it’s, it touches the exceptions. Again, your example in itself is an exception. And it’s, it is too much to say about exceptions. But even talking about it as you just did from a tax context, Shane, the citizen dot. So even in our semantic and in our language use, oh, we already start distinguishing that. And then it comes back to do we identify the citizen in a text context the same as the citizen in a legal context. And to tie it back to my previous example, are those two the same citizen? And all three of them are different domains. There is the text domain, there’s the legal domain, and there is the domain where we might want to match if this is the same person, for whatever anti-terrorist rule organization or whatever. So there’s different domains and every domain has different rule set. So blankly calling them all citizen because in mentally we can say, yeah, it’s the same person, it’s the same citizen. But data administration wise, the data administration of identifiers for a citizen are not the same as the data administration for identifiers for a citizen in a different context. And I think again, that’s the big separator between and it causes a lot of confusion in interviews and workshops, is that. We humans unify that because we see that the reality, it is about one person. But what distinguishes that is that we need to talk about how we talk about the data. And that is not the same thing as the reality. And by separating those out, it makes it, in a way, it makes it easier to say, okay, but are we talking about the same data administration?

No. Then we’re separating those out and if we are talking about three different domains that need to come together in one data administration, then we need more semantics. Distinguishing the three.

Shane: a couple of things I want to loop around there. So again if we go back to the term actually, there is a citizen and a , physical residency and a tax residency because I’m actually a citizen of New Zealand still, but I’m resident physically in the UK and my tax residencies coming with me. But again, it’s not until we, we start using term and facts together that I can articulate that those are three different things, and it’s the business constraints and the exceptions that tell me there are different things. one of the things that I find interesting is this idea of focusing on the complexity and modeling that, is the area that’s gonna cause you a problem.

So if you ever do data bulk training with Hans and WinCo Brockman, they’ll, and I dunno if they do it anymore, but they used to talk about Peter the fly. And so the example would be if I am sitting in this ice cream store and I see Bob come in and order that ice cream and there’s an order id, So I can say when I put term, in fact, there’s a an ID in there of 1, 2, 3, 4, which is the order number. And if I want to know whether they actually got the ice cream. Is a separate part of that process. And on Peter the fly, I can see the ice cream being handed to Bob. So I know it happens if I’m in the room, but if I’m looking purely at the data, at the facts, I see nothing because there is no handover,

there’s no delivery ID that they got the ice cream. So I’m gonna have to infer that if there was an order and no refund turned up, that I’m inferring that happened, but it’s not a fact. And so I think, again, those terms and facts tell our stories and then as data modelers we can look at the exceptions that we know we need to worry about.

But I wanna go back to this idea of domain because it’s one that I struggled with a lot and I still do is a domain boundary can be anything. It can be a business unit, it can be a series of core business events. It can be a process, it can be a use case, it can be a team topology it, a domain is just a boundary where you say it’s in that boundary or it’s not right.

It’s in this domain. It’s in that domain. And I hadn’t thought about using the term fact and business constraints as a way of defining boundaries. For domains. So if I see, a term, in fact that’s all around my citizenship and I see another term in series of facts and business constraints around my residency and I see another one around tax residency and they are different, then I can use those as domain boundaries.

They might be too granular for the intent that I want to use it for. But what we’re saying is they are different. You write them down as words and numbers you can tell they’re different. So that gives us a form of boundary. And so that’s really interesting because it gives us a Pattern,

it gives us a formula of how we can say that this sits in boundary A domain A, and this sits in domain B because of these rules. And I find that really intriguing and valuable ‘cause it’s solving a problem that I’ve been trying to solve for many years. ‘cause it’s annoying me.

Marco: Yeah. It’s, it is. So there’s two you’re right there’s two big areas where you can see there’s a separation of domain. And so rules is one of them. It’s in a hospital systems for example, you might wanna need somebody’s birthday to be able to put it into the computer.

As we had this kind of operations, these treatments, it needs to go this to his insurance company and all of that. But if you’re brought into the emergency room unconscious, you have no idea on you. You’re still being registered as a patient. So the rule is entirely different, but yet we have a patient number somehow. And the definition of the patient is clear across the whole hospital. So rules determine some sort of context. Or, if you come in with an appointment, they probably know your birthdate. If you wanna send it to the insurance company, they have to have your birthdate. The emergency room doesn’t really care. So how do you align all of that? So then you see, again it’s we’re unified in the language, we’re differentiating on the rules, and somehow we have to integrate systems. The most important part is that independent of which system is being built, we need to figure out how we communicate first.

So that language always goes number one. And that’s basically the what fact-based modeling does. It allows people to talk about it from their context, and then given that specific context, you add specific rules for that context. The example where can I live in multiple cities, municipality wise, no, you cannot. You have to be registered in one municipality. Now, in a more generic way, you can have people registered to different places. Sure. I have an office here, I live there, I have a vacation home there. And it’s yeah, different places. But on a municipality level where it becomes more context specific, there is a rule that says you can only register in one municipality. So there’s that too. There’s, there is a way to generalize in a more abstract way. If Ft has been doing that for years, where they introduced the party model, is it a person, is it an organization? Is it, we don’t know, is let’s call it party. We’ll give it an artificial key and we’re good. That’s just the way to keep the IT system running. It has nothing to do with semantic or integration or alignment or whatsoever. It’s just a technical solution because we didn’t get the business meaning in the first place, or we are not specific enough to a specific context. So these problems will always occur. And I think that separates the data modeling from information modeling is where the data modeling is. Do we find alignment in how we need to build the system? Whereas information modeling is, do we find alignment on how we communicate about the data?

Shane: so again, you are making a context differentiation

between the audience. That’s the consumer of what we are creating. And you are saying if it’s stakeholders who aren’t data experts or IT experts, then we’re information modeling. If it’s data people or IT people then we are data modeling,

And the ability to have both languages, but then a mapping and sharing across those languages of where the real value is. And so like you, I’m not a fan of thing as a thing. I don’t agree we should ever use that term in our information models. We should never use party entity thing as a thing.

A thing is associated with a thing as a generic way of describing context to a stakeholder. I also don’t think we should put that in our technical systems. Years ago when we had mainframes and we had no memory and we had no disc and the infrastructure was expensive yes, it had value to us. Right now, the most expensive part of our systems is the humans and understand that context.

And as soon as you design a system with thing, as a thing, and that context lives nowhere else, I now have to spend a massive amount of expensive time trying to understand what the hell, how many things you have, how many things they are related to, and how many things those relationships are.

And that is an expensive piece of work. And I just can’t justify doing that anymore. But as you can tell, I’m slightly opinionated on that one. And it’s also, if I come back to the way we do that information modeling the grain of it, The detail we go to is interesting

because if I do it just based on terms customer orders product, it’s very different doing it based on terms and facts customer bob orders ice cream

Marco: Yes.

Shane: And so what you are saying is actually you are bringing more detail, a higher level of grain into that information modeling process earlier because it has value, so you’re doing more work upfront because you’ve found ways of taking that work that’s done early and automating some of the downstream work in terms of the technical implementations.

And so bringing, from an agile point of view, you are doing work upfront work in advance, which could be waste, but you’re found a way of taking that work and reducing the waste further down the value stream.

Marco: Yeah. Correct. It’s really that, and for those who are interested in this a lot of the information can be found on the website called casetalk.com. What it shows, and this was developed in in, like I said in the seventies, all the way up to 2000 is it’s not just, oh, how can we capture the language?

But it was really, how do we talk to the domain expert to come up with the appropriate IT system? And I think what happened really is that it is such a rich and detailed environment where people modeling traditionally with business knowledge in their heads doing the IT itself, which was very close bound to, organization. When the first computers came in, it’s like it was usually the domain expert that got an IT training. They knew the context, but nowadays it scales up so much is that you are an IT professional. You have no idea about any business. You just, your business is it. So getting back on that and trying to find that alignment with business is, it is of increasing interest, but it’s not the majority.

A lot of new young professionals base their efforts on the tools and the ability of tools and not as much as are we doing the right thing for the business because they don’t really care. They were not educated as such. There’s a massive gap there that where business looks at it is like you’re, you are the expert, so you tell me and then there’s this rift and. You mentioned data Vault a couple of times where, traditionally the hub is the natural business key, right? It’s not a technical key in the source system. So what is the business key? Then you have to talk to the business. And the rich information modeling does with all the fact base is that already mentioned Marco Ledge and Nutra.

So I already have the natural business keys right there. And with a push of a button, it can then be generated into a citizen table and a city table and even have artificial keys with minor annotations like city of residence. We might want to keep a log in time, we might wanna have history in there, but also when is he planning to move to the next city so it becomes bitemporal. And these are very simple flags in the information model that can be generated to a data model that says, did you want a data volt model or a normalized model, or did you want a adjacent schema? So the physical parts become automatable. The data model becomes automatable and having all the semantics and verbiage and examples, it makes it verifiable and readable by the business. And so that really what it ties in and some experiments show that precisely that combination is the real power to keep LLMs grounded as well.

Shane: I definitely agree on the LM front. We know that if we pass at the terms definitions of the terms, the facts, so data examples around those terms and their relationships and natural language business constraints into those lms, we get a much better response when we want to do a task.

Marco: Yep.

Shane: So that context is really important.

And that’s why, it’s interesting watching all the, bi semantic layer vendors try to say they’re context engines and you’re sitting there going, but you’re just at the end of the chain, you’ve just got metrics and maybe some definitions, but the richness is all sitting on the left of our value stream.

It’s all around problem ideation, discovery design. It’s not physical implementation of our consume layer for your cube. Yes, it’s got some value And so I’m in really interested to watch the reinvention of the market yet again. ‘cause as you said technology is often what people get taught.

A number of times I’ll talk to a data science student who’s been taught Python

And you go back and you go, but. How do you understand the business problem? How do you understand the value you’re gonna deliver if you build that ML model? And we’ve seen, as you said, we’ve seen over time data teams that don’t add value don’t survive.

Marco: True. And I remember an interview that was told that we’re, and not to talk them down, but it really points at the lack of education as well, or, the business debt and the technical debt. It’s just people are being trained in tools and in technical approaches where some of the analysts really said, what is your highlight of the day?

Is that when they discover insight, it’s like insight. Should it not have started with the insight as documentation, or what are we really doing instead of let’s discover it. That is, like you said it’s the end of the chain while everybody shouts. Shift left shift left. It’s yeah, why?

Because that’s where it started and that’s what we lost. From my perspective, it just starts with can we talk to each other and are we writing that down? How we do that? Because that’s what we need to do in the end. And whatever it system we built, whatever dashboard is being built, it actually serves to better communicate about what we are really doing. So it all ties back to that. And yeah, there’s a never ending story because the LLM is the next silver bullet. It’s gonna solve all our problems. And it’s not, it’s helpful, but it’s not the silver bullet. ‘cause we still need to get the authentic story, not the fabricated story.

Shane: Yeah it’ll help us understand where the context differs but it won’t help us actually define what the context is.

Marco: Yep.

Shane: At the moment. from the stuff I’ve done with it. Oh maybe the, the next generation a GI version will, or, maybe we all end up with standardized context that every business follows.

But we know that’s not true either, right? We saw that with tools like SAP, where it’s implemented vanilla and then $50 million and five years later there’s a bastardized customized version because that organization is different with air quotes.

Marco: and I can illustrate that by fairly simple examples is that, I mentioned that somewhere else. It’s like there’s not a bank in the world that I didn’t purchase the IBM banking model at the same time I dare any bank in the world to actually have implemented it. and that points to a massive gap is that, to be able to implement it, you first have to know what you already do. Which is a massive dilemma ‘cause nobody has the information models and then you are presented with a technical model, like the banking model that says, everything will fit in here.

It’s okay, but what do we have that actually fits? So there’s a massive dilemma there. And then in the end every bank will try to do it a little bit different. So they have an advantage over the competition. So they don’t really want to confirm to one standard, which points back at. There is no real universal Pattern because everybody tries to give it their own little edge, which means you have to capture that edge, not just build something that you think will fit.

You have to be very specific and in that I’ve seen systems database was designed with rigor and the software was developed three times over, but the database didn’t change. What happened is. A new wave of tech came, it used to be mainframe, then it became Windows, then it became internet, then it became mobile Data did not change the way the technology work, changed the organization, changed the business processes because, we’re not having a physical storefront for the bank, but we now we have a mobile app. So the process of working with the data changed, but the data itself didn’t change. So there’s a lot of dynamics going on and it really shows the importance of getting the data right and don’t throw away the story that came with it so that you can actually reinvent all the processes and all the software on top of it without losing meaning. But it also points at the fast changing world where we perceive everything changes all the time. So we cannot sit down to do a proper information model and have a data model set up, et cetera, et cetera, because we’re in a hurry. We need to deliver what? Nobody realizes that if you sit down long enough to have that information correct, you have the data model correct.

And don’t throw away the story that it will definitely give you a return of investment.

Shane: I’ve gotta say that the IBM data model was a masterclass in sales, The fact that you could sell a diagram, a picture for millions of dollars that nobody ever used, apart from putting it on their wall. That is a masterclass. But I go back just to close it out, I go back to that idea of terms and facts.

So if I have a retail bank, had a store. The terms and facts might have been customer Shane has an account. 1, 2, 3.

If I became a mobile only customer when we changed technology, I might see the term in fact change slightly, where it’s customer oh seven five, yada y has an account 1, 2, 3.

Now I can look at those two lines and I can see something’s changed.

The fact that relates to that term has changed and now I can have a conversation about is that just what you are showing me? And in the background it still says Shane, or did it never say Shane in the real data? It was always customer id. 1, 2, 3, 9, 2, 4. There’s a whole lot of conversations I have, but I know that something’s different and now I know what to have a conversation about with that subject matter expert.

As you said in the past, the subject matter expert, the data person and the IT person was the same person.

Then it became the same team. Then we became a business and IT team, and now we are a business subject matter expert team, a data team and an IT team. We are, we’ve team topologies have got more matrix E, more bigger, and that separation causes some problems.

So by using patents and patent templates and artifacts and shared language, we can close that gap again. And for me, this idea of fact-based modeling, this idea of binding terms and facts with a set of rules, with a set of exceptions as context is really valuable. We can use it in so many ways.

So just to close it out, if people wanted to hear more about this, read more about this, find out more around fact-based modeling, where do they go?

Marco: I think the quickest way into it is to just go to I personally wrote a little book which is published by techniques publications called Just the Facts. It tells the story and names a few examples that I mentioned in this podcast too. It gives you an overview from management to architecture, modelers and developers, and how communication and effects weave through all the disciplines. Obviously CaseTalk is the software tool to support all of that. But you will find links for actual books if you happen to have a copy of the DMBOK by DAMA . The older edition has a crippled article about it. The version two latest edition has a slightly improved article about it. It’s on Wikipedia sources enough, but the good starting point would be casetalk.com.

Shane: Excellent. Alright, thank you for that. I’ve got a new set of patterns and templates I need to go and read a lot more about. It’s gonna be me over the next few months again. But thank you for that

Marco: good. Thanks for having me and talk to you soon.

Shane: I hope everybody has a simply magical day.

«oo»

Struggling to gather data requirements and constantly hearing the conversation above?

Want to learn how to capture data and information requirements in a repeatable way so stakeholders love them and data teams can build from them, by using the Information Product Canvas.

Have I got the book for you!

Start your journey to a new Agile Data Way of Working.

Buy the Agile Data Guide now!

«oo»

Just under seven years ago Nigel Vining and I onboarded our first customer, The Good Registry, to give our MVP AgileData.Cloud platform a run for its money, to test what survived and what didn’t.

One of the first requirements was to collect data from Shopify.

We had a principle back then (a principle we still have today) to use the Google Cloud Data Transfer Services as our first cab off the rank for data collection.

Alas at that time there was no Google Cloud data collection service for Shopify, so Nigel crafted a set of patterns using the Meltano open source framework to automate the collection of this data.

Part of this work was to refactor Meltano so we could operate it as a “serverless” pattern on Google Cloud, rather than it operating under a 24/7 always on container pattern, that would cost more money.

We have reused this data collection pattern many many times over the last 7 years, another great example of our DORO (Define Once, Rese Often) principle.

I saw last month an announcement that Google Cloud have added Shopify and Mailchimp to their ever expanding list of data collection services or what they call “Data Transfer Services”.

If you haven’t noticed Google have been quietly expanding out the list of systems of capture that this service can collect data from.

To me it looks like they are starting to ramp up in this space and will become a serious competitor to tools like Fivetran.

Time will tell.

One of the many things I have learnt about Google Cloud over the last 7 years, is they may not be first to market, but when they go after a market, they have the firepower to do it at scale.

Google Gemini showed us that yet again.

You can check out the doco for the new Data Transfer Services here:

https://docs.cloud.google.com/bigquery/docs/shopify-transfer

https://docs.cloud.google.com/bigquery/docs/mailchimp-transfer