In this episode, Shane Gibson sits down with anchor modeling co-creator Lars Rönnbäck and data modeling practitioner Christian Kaul.
The conversation explores the foundational principles of anchor modeling, an agile, sixth normal form (6NF) ensemble modeling technique developed in 2004 to achieve data warehouse longevity and cope with rapid business changes without degrading into unmaintainable “just a bunch of tables”. The group examines the architecture’s core constructs—anchors (representing immutable entity identities), attributes (isolated in separate tables per field), ties (relationships with flexible arity), and knots (shared finite value sets)—as well as the newly introduced nexus, which translates activity schema concepts into 6NF. They contrast these constructs with other ensemble methods such as Data Vault and Focal, emphasizing how strict separation of immutable surrogate identities from natural business keys provides a consistent, opinionated framework suited for automated schema generation.
The discussion also dives deep into anchor modeling’s advanced temporal capabilities, contrasting standard unitemporal progression with bitemporal architectures that use annex tables to track orthogonal historical corrections, as well as complex concurrent reliance temporal models that record conflicting source statements alongside confidence ratings. Lars shares practical deployment insights, including Nikolay Golov’s 500-terabyte implementation for an online marketplace in Russia, and the panel reflects on why ensemble techniques have gained stronger traction in European and Australasian Inmon-aligned environments compared to US Kimball patterns. Looking forward, the panel explores the concept of the model-driven organisation, evaluating how pre-joined perspective views, hybrid operational/analytical databases, and open-source anchor modeling LLM skills enable AI agents to safely generate code and navigate multi-table schemas.
Whether you are curious about sixth normal form ensemble modeling, strategies for handling bitemporal and multi-source historical corrections, or how AI agents and LLM skills can interact with highly decomposed data architectures, this episode is packed with valuable insights for data architects and modeling practitioners.
Listen
Listen on all good podcast hosts or over at:
Subscribe: Apple Podcast | Spotify | Google Podcast | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser | Deezer | Podcast Addict |
Tired of vague data requests and endless requirement meetings? The Information Product Canvas helps you get clarity in 30 minutes or less?
Google NotebookLM Mindmap
Google NoteBookLM Briefing
Executive Summary
Anchor Modeling is an ensemble data modeling technique developed in 2004 by Lars Rönnbäck and Olle Regardt to resolve the longevity and maintainability issues plagued by traditional enterprise data warehouses. Built upon Bill Inmon’s foundational data warehousing principles—time-variance and non-volatility—Anchor Modeling incorporates high structural and value agility, preventing systems from degrading into unmaintainable collections of tables (referred to as a “JBOT” or “Just a Bunch of Tables”).
The methodology decomposes data structures into Sixth Normal Form (6NF), separating changing elements from immutable ones. Anchor Modeling consists of five core constructs: Anchors, Attributes, Ties, Knots, and Nexuses. It supports sophisticated temporal capability models, ranging from unitemporal tracking to complex concurrent reliance temporal frameworks that assess source conflict and confidence metrics.
While market adoption remains niche (estimated at under 100 major global implementations, including a 500-terabyte data warehouse in Russia and operational logistics systems in the UK), Anchor Modeling demonstrates high user retention due to its rigid naming conventions, automated code generation capabilities, and resilience to source system volatility. Furthermore, its strict adherence to formal patterns makes it exceptionally compatible with modern High-Performance Hybrid Transactional/Analytical Processing (HTAP) databases, automated Large Language Model (LLM) query engines, and emerging “Model-Driven Organization” frameworks.
1. Background, Origins, and Conceptual Foundations
The Genesis of Anchor Modeling
Anchor Modeling was created in 2004 during an enterprise data warehouse initiative for an insurance business with four primary branches. Previous data warehouse attempts at the company had repeatedly failed within a few years due to an inability to cope with rapid changes in source systems and evolving business demands.
Core Objective: Design a modeling methodology adhering to Inmon-style principles (non-volatile, time-variant) while remaining agile enough to accommodate structural and value changes without requiring destructive migrations.
First Deployment: The initial production release occurred in 2005. Core parts of that original system remain active in production over 20 years later.
Etymology: The name “Anchor Modeling” was derived from the visual appearance of the graphical models, where central entities act as “anchors” holding surrounding data structures.
The Ensemble Modeling Context
Anchor Modeling falls under the broader classification of ensemble modeling, alongside techniques such as Data Vault, Focal, and Head Version.
Shared Principles: Ensemble techniques normalize data beyond standard third normal form (3NF) by explicitly separating static identities from elements that change over time or structure.
Agility Types: These patterns resolve challenges associated with both micro-agility (changes in data values over time) and macro-agility (changes in business structures over time).
2. Structural Constructs of Anchor Modeling
Anchor Modeling relies on five distinct graphical and physical constructs. Fundamentally derived from entity-relationship modeling concepts, these constructs enforce a strict Sixth Normal Form (6NF) implementation where every attribute resides in its own database table.
+---------------------------------------------------------------------------------+
| ANCHOR MODELING BUILDING BLOCKS |
+---------------+-----------------------------------------------------------------+
| Construct | Functional Description & Database Mechanics |
+---------------+-----------------------------------------------------------------+
| Anchor | Stores entity identity (surrogate key sequence only). |
| | Completely immutable. |
+---------------+-----------------------------------------------------------------+
| Attribute | Stores properties of an anchor in 6NF (one table per attribute).|
| | Manages temporal value changes. |
+---------------+-----------------------------------------------------------------+
| Tie | Captures relationships between anchors. Supports n-ary |
| | relationships and temporal states. |
+---------------+-----------------------------------------------------------------+
| Knot | Constrains attributes to closed value domains / enumerations. |
| | Acts as a shared reference lookup. |
+---------------+-----------------------------------------------------------------+
| Nexus | Merges a tie and an anchor to model 6NF activity schemas |
| | optimized for time-series and event data. |
+---------------+-----------------------------------------------------------------+
Detailed Breakdown of Constructs
Anchors
Corresponds to traditional entities or Data Vault “hubs.”
Holds only an immutable surrogate identity (typically an integer sequence; hash keys are avoided).
Natural/business keys are not stored within the anchor table itself; they are stored as separate attributes attached to the anchor.
Attributes
Holds individual properties or descriptive data (equivalent to Data Vault “satellites,” but split into single-column 6NF tables).
Each attribute is implemented as a standalone physical table attached to its parent anchor.
Ties
Models relationships among anchors (equivalent to Data Vault “links” or relational foreign keys/fact relationships).
Unlike binary links, ties can naturally represent n-ary relationships (connecting multiple anchors simultaneously without artificially breaking them into binary components).
Ties contain temporality when relationships evolve over time.
Knots
Formed by combining an anchor and an attribute into a single logical construct.
Used to model closed domains, fixed value sets, or lookup enumerations (equivalent to reference tables).
Enforces referential integrity via database foreign keys; values outside the knot domain are rejected upon insertion.
Nexuses
Added to the methodology around 2022 following a request by practitioner Nikolay Golov.
Combines a tie and an anchor to incorporate the concept of an activity schema in 6NF directly into an anchor model.
Provides performance optimizations for queries executing across time-series or event-driven data workloads.
3. Key Technical Mechanics: Keys, Loading, and Temporality
Identity vs. Identifier Mechanics
Anchor Modeling strictly distinguishes between an entity’s internal identity and its external identifiers:
Identity: Managed entirely by the Anchor table using simple surrogate integer sequences.
Identifiers: Business keys or natural keys are modeled as attributes or directional paths connecting multiple anchors. This allows an entity to maintain multiple alternate business keys simultaneously or track changes in business keys over time without altering the underlying entity identity.
Temporal Modeling Tiers
Anchor Modeling defines three progressive tiers of temporal handling, selected at the code-generation level:
Unitemporal Anchor Modeling:
Tracks a single history timeline representing natural, real-world value progression over time (e.g., aging, status updates).
Effective dates are included directly in the primary key of the attribute table.
Bitemporal Anchor Modeling:
Captures two timelines: real-world historical evolution and the timeline of database corrections/state changes.
Implemented by adding a separate annex table to each attribute table. The annex table stores orthogonal metadata detailing when corrections to historical statements were made.
Concurrent Reliance Temporal Modeling:
Extends bitemporal capabilities by adding two additional tracking columns to the annex table:
Source Assertion: Tracks who or which system made the statement, allowing coexisting conflicting assertions from different source systems.
Reliability Score: Captures a percentage score (e.g., 85% vs. 100%) representing the degree of belief or confidence in a given assertion.
Trade-off: High query complexity and processing overhead, which directly spurred Lars Rönnbäck’s research into Transitional Modeling to handle multi-perspective temporal data more efficiently.
Data Warehouse Ingestion & Query Layer Abstraction
[ Source Systems ]
│
▼
[ Staging Area (Raw ELT Data) ]
│
├───────────────────────────────┐ (Parallel Load Pass 1)
▼ ▼
[ Anchors (Identities) ] [ Knots (Domain Enumerations) ]
│ │
├───────────────────────────────┘ (Parallel Load Pass 2)
▼
[ Attributes (6NF Tables) ] ─── AND ─── [ Ties (Relationships) ]
│
▼
[ Pre-Joined View Layer (Point-in-Time / Latest Perspectives) ]
│
▼
[ Analytics, BI Tools, & AI Query Engines ]
Load Sequencing: Loading requires an ELT (Extract, Load, Transform) approach. Ingestion occurs in two distinct parallel phases:
Phase 1: Anchors and Knots are loaded in parallel.
Phase 2: Attributes and Ties are loaded in parallel once Phase 1 identities and reference values are established.
View Layer Abstraction: Because query execution against dozens or hundreds of normalized 6NF tables requires complex SQL joins and temporal logic, Anchor Modeling relies on an automated view layer. Generated pre-joined views present flattened, point-in-time, or “latest state” perspectives directly to reporting tools and analysts.
4. Market Adoption, Ecosystem, and Global Modeling Trends
Adoption Profile
Scale: Total global implementations are estimated at under 100 organizations, but user satisfaction and retention among adopters are exceptionally high.
Prominent Use Cases:
Enterprise Data Warehousing: Implemented at a major Russian classifieds/goods platform (led by Nikolay Golov), handling over 500 terabytes of rapidly evolving source data.
Operational Systems: Utilized as an operational system database for a major logistics provider in the UK.
Regional Modeling Preferences:
North America: Heavily dominated by two-layer Kimball dimensional modeling formats.
Europe & Australasia: Stronger historical alignment with Inmon three-layer architectures, driving higher adoption of ensemble approaches (Data Vault, Anchor, Focal).
Tooling and Automation Ecosystem
Because hand-coding 6NF anchor implementations is impractical, automation is critical:
Online Modeler (
anchormodeling.com): An open-source, web-based graphical modeling tool. It converts visual models directly into target DDL and view execution code for platforms including Snowflake, BigQuery, Postgres, SQL Server, Oracle, and Vertica.AI Integration: An open-source skill repository available on GitHub enables modern LLMs to interpret anchor patterns, automatically construct valid anchor models from raw source layouts, and write complex queries against 6NF schemas.
5. Future Directions: AI and the Model-Driven Organization
The Model-Driven Organization Paradigm
Anchor Modeling principles intersect directly with the concept of the Model-Driven Organization:
Centralized Conceptual Core: Rather than building isolated databases inside disparate vendor applications, an enterprise defines its core business concepts and relationships centrally.
HTAP Convergence: Modern Hybrid Transactional/Analytical Processing (HTAP) engines (such as Snowflake) enable a single anchor-modeled database to serve simultaneously as both the operational database and the analytical data warehouse.
Lightweight Applications: Operational systems become lightweight read/write clients built directly on top of the centralized model-driven database.
Synergy with Artificial Intelligence and Code Generation
Overcoming Human Cognitive Limits: Humans often struggle to manually manage 6NF schemas due to the cognitive load of handling hundreds of isolated tables. AI systems and LLMs do not suffer from these context limitations; they easily process structural metadata and navigate 6NF join paths.
Rapid Client Generation (”Vibe Coding”): Historically, building custom lightweight applications around a central database required prohibitively large software development teams. The emergence of AI-assisted code generation dramatically reduces application development cycles, making the deployment of light applications over centralized anchor models commercially viable.
Tired of vague data requests and endless requirement meetings? The Information Product Canvas helps you get clarity in 30 minutes or less?
Transcript
(00:00:01.370):
Welcome to another Agile Data and Info podcast and live stream.
(00:00:05.391):
Hey Lars, hey Christian, thank you for coming on the show.
(00:00:08.732):
Today we’re going to talk about anchor modeling and no doubt we’ll go
(00:00:12.274):
into automation and that whole space because we always tend to do that.
(00:00:16.695):
Before we kick off though, why don’t we get each of you kind of introduce yourself
(00:00:20.536):
for a couple of minutes for the audience.
(00:00:21.837):
So Lars, do you want to go first?
(00:00:24.164):
Sure.
(00:00:25.285):
Lars Rambeck, I live in Sweden, run a small business called Up2Change,
(00:00:31.008):
where we work with data warehousing, business intelligence, analytics,
(00:00:36.571):
financial forecasting, lots of different things.
(00:00:41.374):
And I’ve been working in the industry for almost 25 years now.
(00:00:48.397):
And I’m one of the founding fathers of the anchor modeling technique.
(00:00:53.981):
And I’ve been doing collaborative research with Stockholm University.
(00:00:58.702):
So we’ve also developed something that came after Anchor Modeling called Transitional Modeling.
(00:01:05.123):
So some people may know me from that as well.
(00:01:08.944):
Excellent. Christian?
(00:01:11.105):
Yeah. My name is Christian Kaul.
(00:01:13.705):
I live and work in southern Germany, Bavaria.
(00:01:18.285):
I’m a data modeler, data biasing person with a little bit less experience than last,
(00:01:22.730):
but also in the space for around 12 years now, I would say.
(00:01:27.636):
And I’m currently working for a company called Virtual7, who does IT consulting,
(00:01:32.261):
data consulting for government clients, basically.
(00:01:36.621):
And my sidekick is writing
(00:01:39.823):
and organizing meetups and events on data modeling.
(00:01:43.885):
So you might know me from the Data Modeling Meetup Munich
(00:01:46.286):
or from the Knowledge Gap Conference I used to do.
(00:01:49.039):
Yeah, I remember presenting to it once where I had to get up it.
(00:01:53.089):
I think you gave me a 2 a.m. New Zealand
(00:01:58.036):
I think there was another one where you were really kind and I got a 11 p.m.
(00:02:00.598):
one, but I’ve got to say, it’s better being on this side of the world to be able
(00:02:04.782):
to do these sessions at times when I’m not meant to be asleep.
(00:02:08.465):
All right, so let’s kick it off, right?
(00:02:11.767):
How do we explain the core foundational patterns that are anchor?
(00:02:16.451):
And I don’t know, Lars, if you want to do anchor and then maybe go into transitional
(00:02:20.974):
and see whether...
(00:02:22.155):
What the differences are or we can just go through Anchor first and see if we have time.
(00:02:26.418):
Up to you kind of how this way you explain it to people in the data practitioners
(00:02:31.862):
who haven’t found or understood those patterns yet.
(00:02:36.626):
Yeah, so Anchor.
(00:02:39.555):
Sort of came from the needs of building long lasting data warehouses.
(00:02:46.100):
It was back in 2004 at an insurance company
(00:02:50.504):
where we were tasked with building
(00:02:53.766):
an enterprise data warehouse.
(00:02:55.127):
They had four different branches within their business
(00:03:00.411):
and they wanted something
(00:03:03.233):
that It would last a long time.
(00:03:06.713):
They had had previous data warehouse projects and they failed after a few years.
(00:03:14.879):
So we had to come up with something new.
(00:03:18.541):
And I and a colleague called Ole Regert,
(00:03:22.244):
we were looking at different ways of doing this.
(00:03:26.007):
And of course, we wanted to live up to the Bill Inman style.
(00:03:31.460):
Like time variant, non-volatile.
(00:03:36.484):
So we wanted something that was along those principles,
(00:03:42.189):
but also agile enough to be able
(00:03:45.592):
to Cope with changes because that was
(00:03:49.320):
the major pain point in the previous attempts that they had with data warehouses,
(00:03:53.384):
that the rate of change in sources and business demands was so high
(00:03:59.571):
that the data warehouse couldn’t just keep up with it and they had to scrap them
(00:04:03.535):
after a while. They became unmaintainable.
(00:04:07.545):
I sort of coined a term somewhere later called the J bot, just a bunch of tables.
(00:04:12.889):
So they didn’t really have any methodology in their previous data warehouses
(00:04:17.953):
or if they had one, it sort of degraded at some point
(00:04:21.056):
and they eventually became just a bunch of tables.
(00:04:24.538):
They happen to reside in the same database and they call it a data warehouse.
(00:04:29.923):
So we also wanted to have something where the guiding principles were sort of simple
(00:04:34.346):
and quite rigid.
(00:04:35.965):
So when you follow anchor modeling,
(00:04:38.627):
you stick to a certain set of principles and constructs that you have.
(00:04:44.450):
So basically, anchor modeled data warehouses around
(00:04:49.413):
the world would look quite similar and not deviate much
(00:04:55.316):
in methodology or even in naming.
(00:04:59.939):
We came up with naming conventions that we have stuck to actually since then.
(00:05:06.576):
So that was sort of the premise.
(00:05:07.876):
And we came up with something and at the time we didn’t really know what it was.
(00:05:13.698):
We called it anchor modeling because when you looked at graphical representation
(00:05:19.300):
of this, it looked like you had these things that anchored a lot of other things.
(00:05:24.262):
So that’s why we call it an anchor model.
(00:05:28.684):
And yeah, that’s basically how it started.
(00:05:33.077):
We implemented this enterprise data warehouse at that client
(00:05:37.901):
and parts of it are still alive.
(00:05:42.321):
It was put into production in 2005,
(00:05:46.263):
so it’s over 20 years in production.
(00:05:50.665):
Like the core part is still there and running and using anchor modeling still today.
(00:05:55.147):
So that’s I think is a testament to that we achieved what we wanted when it came to longevity.
(00:06:02.412):
Then, of course, anchor modeling has evolved a little bit since,
(00:06:06.214):
but basically the core of anchor modeling has remained the same ever since that day.
(00:06:13.739):
So I can go into constructs and such later, but that’s the backstory anyway of anchor modeling.
(00:06:20.363):
Yeah, and my understanding is, you know,
(00:06:21.804):
we have a bunch of Physical modeling techniques or patterns that are part
(00:06:26.277):
of ensembles, right?
(00:06:27.798):
Yeah, kind of a group we call ensembles.
(00:06:30.740):
So, as most people know, I tend to have more of a background in Data Vault.
(00:06:35.323):
But Christian, is that your understanding, right?
(00:06:36.764):
That, you know, we’ve got Data Vault, Anchor, Focal,
(00:06:40.086):
there’s probably a few more that are really, I don’t know, what would you call them?
(00:06:44.349):
You know, they’re in the same environment.
(00:06:46.150):
I think this is a good umbrella term.
(00:06:50.091):
Ensemble modeling, we have Data Vault, we have Anchor, we have Head Version,
(00:06:55.839):
and we have some more obscure patterns, I would say,
(00:06:59.322):
but they are all similar in that they go beyond regular normalization and they split
(00:07:05.448):
the things that change from the things that don’t change.
(00:07:08.891):
I have an article about this somewhere,
(00:07:10.672):
but basically You have the problem that the naive approach doing like three
(00:07:16.136):
and a half plus time or whatever, it doesn’t work very well with change,
(00:07:20.759):
both like change of the values over time and change of the structures over time.
(00:07:24.721):
And so approaches like Anchor in particular,
(00:07:29.643):
they help you cope with this kind of change,
(00:07:31.704):
both what
(00:07:36.467):
the focal people call micro and micro agility.
(00:07:41.973):
Yeah, I just want to confirm that we’re talking about agility, not Scrum, right?
(00:07:46.776):
I just heard somebody today going, you know,
(00:07:49.177):
that they don’t think this patent would work with Agile because you couldn’t do it
(00:07:52.358):
in a two-week iteration.
(00:07:53.519):
And I’m sitting there, you know, hitting myself going, you know, Scrum, Agile is not Scrum.
(00:07:59.622):
Scrum can be Agile.
(00:08:01.303):
But anyway, we’ll get off that rant.
(00:08:03.324):
Okay, so Lars, do you want to...
(00:08:05.085):
Start off with maybe the kind of core components of Anchor because I suppose
(00:08:10.231):
when I think about modeling techniques,
(00:08:11.873):
I naturally want to understand the physical modeling structure first.
(00:08:15.336):
Don’t know why, but I always do.
(00:08:16.858):
And then from there, we’ll kind of go out and talk about the fact
(00:08:19.441):
that it’s probably more than just a bunch of tables with types.
(00:08:23.125):
So let’s go through some of the basics first.
(00:08:25.587):
Yep, absolutely.
(00:08:29.202):
At the time, we weren’t aware that there were other modeling techniques
(00:08:33.164):
that were sort of siblings to anchor modeling.
(00:08:36.725):
I mean, Data Vault predates anchor modeling by quite a few years,
(00:08:40.567):
but we had never heard of Data Vault.
(00:08:43.869):
It was something that we heard of afterwards.
(00:08:46.330):
But there are, of course, many similarities.
(00:08:48.791):
So basically, in anchor modeling,
(00:08:51.693):
we have We have three core components and those come
(00:08:57.088):
from something called entity relationship modeling,
(00:09:01.251):
which was invented by Cod a long time ago.
(00:09:08.496):
I think so, at least.
(00:09:09.556):
And those are the core that he called entities,
(00:09:16.521):
we call anchors.
(00:09:19.162):
So that’s where you store like a customer, an order, things like that.
(00:09:25.126):
Then you have the attributes, which are attributes of an entity.
(00:09:30.490):
We call them attributes as well in anchor modeling.
(00:09:32.711):
So that would be like the order value, customer name, things like that.
(00:09:39.436):
And then finally, we need to represent relationships as well.
(00:09:44.419):
And we call them ties.
(00:09:48.717):
Ties in anchor models can be more than binary.
(00:09:54.019):
So they can be an area, you can have any arity you want for a tie.
(00:09:58.701):
So you can represent a relationship
(00:10:02.382):
that relates many different anchors in a single tie.
(00:10:08.224):
And of course, you can break them down if you want to.
(00:10:10.505):
That’s a modeling choice.
(00:10:11.945):
If you want to make them binary, you can always break down relationships into the binary.
(00:10:19.241):
It’s not something I recommend.
(00:10:21.623):
I think you represent more naturally what the relationship actually is
(00:10:26.466):
if you keep them at the arity that it is in reality.
(00:10:32.369):
So those are the three core components that we have.
(00:10:35.832):
But we realized also
(00:10:37.133):
that it serves two purposes.
(00:10:42.916):
But if you combine some of these entities, the core components we have,
(00:10:48.443):
You can create new components.
(00:10:50.484):
And if you take one anchor and an attribute and combine that into one single entity,
(00:10:57.146):
you get a knot.
(00:10:58.767):
We call them a knot, like this thing you tie.
(00:11:04.069):
And you can represent using a knot a small
(00:11:09.945):
Enumeration or a value set or a value domain like you have a small set of values
(00:11:15.491):
and those are the only values that you are allowed to use in an attribute.
(00:11:23.398):
We also have that construct and we had that one from the beginning.
(00:11:26.779):
But theoretically, you don’t need it,
(00:11:31.361):
but it simplifies both understanding and it also scopes values
(00:11:38.084):
to a specific value set that you can use.
(00:11:42.446):
I think they’re called reference tables or something like that in Data Vault, for example.
(00:11:46.807):
So we have that.
(00:11:50.572):
Our anchors are called hubs in Data Vault and we have attributes
(00:11:56.195):
that would be satellites but it’s not a one-to-one because our attributes are
(00:12:02.640):
As we later learned in sixth normal form,
(00:12:05.762):
which actually means that you have one table per attribute,
(00:12:09.964):
whereas in a satellite you can actually group attributes
(00:12:14.487):
into a satellite based
(00:12:17.188):
on some heuristic rules.
(00:12:20.770):
We don’t have that.
(00:12:23.572):
So those were the four components that we had for a very, very long time.
(00:12:28.394):
But actually quite recently we added a fifth Component which is again something
(00:12:35.051):
that you can theoretically decompose into those three core components I mentioned
(00:12:39.533):
from the beginning but if you actually merge a tie
(00:12:43.216):
and an anchor you can also get something new and we call that a nexus
(00:12:50.059):
and the nexus is like just a couple of years old in anchor modeling so it’s quite
(00:12:54.262):
a new concept it was actually requested by
(00:13:00.650):
One person out there who has done a lot of anchor modeling, his name is Nikolai Golov.
(00:13:07.572):
And what he realized was that there was a technique at the time that was quite popular.
(00:13:15.553):
I don’t know if you remember it, was called Activity Schema.
(00:13:20.850):
Yeah, we still use it.
(00:13:23.503):
We use it for very specific use cases.
(00:13:25.884):
It’s great.
(00:13:27.125):
Exactly. So this is an activity schema in six normal form in an anchor model.
(00:13:34.167):
So basically, I looked at what an activity schema does and how would you incorporate
(00:13:40.169):
that into an anchor model and make it six normal form.
(00:13:43.111):
And I promised myself I would only add this construct if it was decomposable
(00:13:48.713):
into those three core components.
(00:13:50.413):
And it turns out it is.
(00:13:52.929):
And I was also able to prove that it is sixth normal form, almost always.
(00:13:58.933):
It’s the same thing with certain ties.
(00:14:00.894):
If you break some rules, you can make it fifth normal form, and so goes with the Nexus as well.
(00:14:07.099):
It can make it fifth normal form.
(00:14:09.100):
But anyway, it’s a concept that actually enables performance gains
(00:14:15.584):
if you have specific types of queries over time series or event-like data.
(00:14:21.808):
So that’s a useful contribution as well to anchor modeling.
(00:14:24.971):
So now we have five constructs like the anchor, attribute, tie, knot, and nexus.
(00:14:33.176):
Okay. Well, we might come back.
(00:14:35.318):
Yeah, we’ll kind of go through those.
(00:14:36.959):
I mean, I had an experiment with anchor modeling years ago,
(00:14:43.309):
and I always struggled with the fact that an attribute was effectively a table.
(00:14:48.608):
So, you know, I’d end up with 50 to 100 tables of attributes and back then,
(00:14:54.836):
you know, performance of the cloud computing wasn’t that great
(00:14:58.221):
and it was damn expensive and so at reasonable volumes it was Interesting, you know,
(00:15:04.709):
especially where we’re using BigQuery and BigQuery wasn’t a great fan of joins back
(00:15:08.212):
then.
(00:15:09.373):
Definitely changed now.
(00:15:10.494):
So let’s go back though and just kind of restate those objects and do some comparisons.
(00:15:15.299):
And Christian, you jump in as I get them wrong.
(00:15:19.417):
So if I think about a dimension, dimensional modeling,
(00:15:23.398):
it holds a key and it holds attributes of a key.
(00:15:26.819):
If I think about Data Vault,
(00:15:28.179):
we hold the key as a hub and we hold the attributes into one
(00:15:31.640):
or many satellite tables.
(00:15:33.620):
So what you’re saying is effectively an anchor is a form of a key
(00:15:40.442):
and then the attributes are individual tables per field.
(00:15:45.863):
Did I hear that right?
(00:15:48.443):
Close, but not exactly.
(00:15:51.464):
Let me clarify this.
(00:15:54.245):
We have more of a focal approach when it comes to keys.
(00:15:59.446):
The only thing that exists in the anchor table is actually a surrogate key.
(00:16:04.268):
The natural key or business key is spread out
(00:16:08.449):
into one or more attributes in the model.
(00:16:13.424):
So a key in anchor model is actually a sort of a path in a model that connects,
(00:16:19.726):
if it’s a composed key anyway,
(00:16:21.007):
it’s a path that connects different attributes in the model with each other.
(00:16:25.428):
So that’s a key in anchor, which means that you can have several different keys
(00:16:31.350):
for the same anchor simultaneously.
(00:16:35.492):
And you can also historize keys if necessary.
(00:16:40.893):
In Focal, I think they make the distinction between identity and identifier
(00:16:44.576):
and basically what you have as well, yeah.
(00:16:47.378):
Okay.
(00:16:47.558):
Yeah, I think there are some approaches even in Data Vault where you call the bag
(00:16:52.342):
of keys or something like that, where you have alternative keys outside the hub.
(00:16:58.727):
And in Focal, you have, I forgot the name of it,
(00:17:02.129):
but there’s a specific lookup table for keys,
(00:17:05.872):
also not on the actual Focal table itself.
(00:17:09.320):
Which corresponds to our anchor.
(00:17:11.621):
So if I have a concept of a customer and a concept of a product,
(00:17:15.544):
and they either have a natural business key or a surrogate key,
(00:17:19.706):
so they have an identifier, and I go and look at the physical data structures,
(00:17:25.209):
I’m going to see a dimension called customer, badly named no doubt,
(00:17:30.072):
and a dimension called product.
(00:17:31.272):
If I go into a data vault, I’ll typically see a hub customer or customer hub product.
(00:17:36.655):
Are you saying that with an anchor,
(00:17:38.055):
I’d see a table called anchor and it would hold all the...
(00:17:42.496):
No, so would I get a table per customer and a table per product?
(00:17:47.258):
Yes, you would have a customer anchor and a product anchor.
(00:17:53.139):
Exactly like you would in any of the other Data Vaults or Focal as well.
(00:17:59.189):
But for the customer anchor, let’s say you have a customer number,
(00:18:03.877):
that would be a single attribute on the anchor, keeping the customer number,
(00:18:08.986):
which also happens to be the key.
(00:18:13.186):
Right. Okay. And then you use a surrogate,
(00:18:15.248):
which is the unique identifier for the row in that customer anchor table.
(00:18:20.013):
Yeah. And the surrogates that we prefer anyway are just sequences,
(00:18:27.019):
integral sequences.
(00:18:28.380):
That’s usually what we use.
(00:18:29.822):
No hashing.
(00:18:31.023):
Yeah. Well, we can go down the religious hashing argument right about now.
(00:18:35.852):
But then we can go and argue what a data product is as well if you want to get into that.
(00:18:41.435):
Okay. And then ties are relationships, right? Yes.
(00:18:45.837):
They are a form of a data vault link.
(00:18:49.339):
They are a form of a dimensional fact,
(00:18:52.501):
although I’m going to assume they don’t hold any attributes against them directly.
(00:18:57.115):
And then what you said was you tended to prefer multi-part ties.
(00:19:02.838):
So when I’m modeling, we have an object called an event,
(00:19:07.140):
which is our relationship table effectively.
(00:19:10.122):
And it’s kind of interesting because my co-founder, Nigel,
(00:19:12.803):
comes from a dimensional relational background.
(00:19:16.245):
And so he kind of likes short events, events with few keys.
(00:19:22.222):
I don’t know why I think about it this way.
(00:19:24.523):
Maybe it’s coming from a lot of Lawrence Kuhler who does what kind of stuff that I used to use.
(00:19:29.545):
But my event tables always have lots of keys, right?
(00:19:33.006):
You know, it’s almost a natural language of customer orders product from store
(00:19:37.468):
and employee, that kind of thing.
(00:19:39.949):
And that’s just naturally how I model.
(00:19:43.930):
So is that multi-part?
(00:19:47.186):
Ties, is that like prescriptive or is that just a kind of modeling pattern
(00:19:54.048):
or flavor opinion that you tend to have?
(00:19:58.409):
I think it’s a modeling pattern
(00:20:01.850):
that I think I tend to have and I would like to say
(00:20:05.471):
that it’s prescriptive as well.
(00:20:07.411):
We don’t enforce it.
(00:20:09.092):
We built a modeling tool to help Make anchor models and generate the code
(00:20:14.701):
to implement it in various databases.
(00:20:18.301):
But that tool doesn’t enforce anything when it comes to ties.
(00:20:22.462):
So you’re free to create binary ties if you’d like.
(00:20:26.143):
But the risk with using multipath ties is
(00:20:32.184):
that you could end up in a situation
(00:20:34.405):
where it’s no longer six normal form.
(00:20:37.708):
For example, if you have two paths that could potentially change over time,
(00:20:44.092):
then the tie is no longer in six normal form and that’s like
(00:20:49.254):
when we have this Heuristic rule that says that you should split this tie
(00:20:54.758):
so it becomes obvious which of the path is like the path that is changing over time
(00:21:00.639):
because you keep only one time point,
(00:21:03.260):
a changing time point in the tie for changing relationships that change over time.
(00:21:09.421):
So in order to resolve and to have like a clear picture of what is going on,
(00:21:14.022):
you should split that tie so you have just one thing that changes over time.
(00:21:18.325):
So you can deduce which thing it is that is actually connected to the tying point
(00:21:22.528):
that you’re storing.
(00:21:24.209):
And if you’re using the tool,
(00:21:25.709):
it will actually warn you if you’re trying to create a tie
(00:21:28.791):
that is breaking this pattern, but it’s just a warning.
(00:21:32.313):
So, yeah.
(00:21:33.994):
Okay. Maybe we should emphasize a key difference between the data board links
(00:21:39.077):
that I guess many people out there know that the tie is temporary,
(00:21:43.920):
while the link is not, the data board link is not temporary.
(00:21:47.673):
As in data vault links don’t have an effectivity date directly on the link table,
(00:21:53.758):
whereas ties do. Is that what you’re saying?
(00:21:56.560):
Basically, yes. Yeah.
(00:21:58.501):
Although in theory, depending which data vault religion you believe in,
(00:22:03.305):
you might have a satellite on a link table with effectivity or you might not, right?
(00:22:09.028):
There’s different flavors within that.
(00:22:11.210):
Out of interest, Christian,
(00:22:13.212):
are you a two-way relationship modeler or a seven-way by default?
(00:22:18.738):
Which way do you model?
(00:22:22.962):
I come from a data world corner,
(00:22:26.726):
definitely, and there are different schools there as well.
(00:22:35.248):
Usually you have some kind of temporality or other attributes,
(00:22:40.273):
a relationship that you have to put somewhere in data world.
(00:22:42.575):
And I’m a relationship describing hub kind of guy.
(00:22:45.577):
And so then you can basically do easily the two way links because you have like
(00:22:50.182):
the relationship describing hub plus the other thing.
(00:22:54.866):
But it’s different flavors, different tastes, especially in data world.
(00:22:59.210):
We had this discussion linked in a few weeks ago.
(00:23:02.152):
There are many different schools in many different flavors.
(00:23:06.253):
For example, Data World Builder, the tool, they do two-way links by default.
(00:23:11.775):
Other people are more on the dimensional-like links,
(00:23:16.977):
especially if you go to what Denlin said at least a few years back,
(00:23:21.299):
it’s closer to the dimensional effect tables
(00:23:26.040):
than what is done mostly today
(00:23:29.181):
and especially with the Data World Builder does.
(00:23:32.340):
Yeah, and I think one of the things I picked up, Lars, was you said that, you know,
(00:23:35.462):
anchor is quite opinionated in potentially a good way.
(00:23:38.624):
So, for example, naming conventions are baked into the patterns, whereas, you know,
(00:23:43.847):
if I look at dimensional or vault, You can name it any way you want.
(00:23:47.661):
You can be hub-cust, cust-hub, customer-hub,
(00:23:53.802):
h-customer, dim, d, all those kind of things.
(00:23:59.004):
So I think, for me, part of picking up a set of patterns is
(00:24:02.805):
when they’re opinionated, they’ve been proven.
(00:24:07.352):
Those problems have been hit before and that opinionation is coming
(00:24:10.976):
through because it’s solving it there.
(00:24:13.258):
I think the other thing that I find interesting is when people think
(00:24:15.741):
about data modeling patterns or physical data modeling patterns at least.
(00:24:21.227):
They always think about hand coding, right?
(00:24:23.329):
And I know, Christian, you and I have had this conversation before that a lot
(00:24:26.773):
of the times adopting a pattern that comes with automation
(00:24:33.598):
Is how you actually get a lot of the benefits and especially
(00:24:36.160):
when there’s some complexity in the pattern.
(00:24:38.482):
If I think about focal and anchored to a degree with the number
(00:24:41.225):
of physical tables you have to create, you don’t want to be doing that by hand,
(00:24:44.648):
right? You want automation to come with it where you define some context
(00:24:48.812):
and then effectively everything else kind of gets built for you based
(00:24:52.976):
on those rules.
(00:24:54.657):
Christian, is that what you’re seeing as well?
(00:24:55.957):
You’re seeing that, you know,
(00:24:57.278):
adopting a set of patterns and automation that comes with those patterns
(00:25:01.959):
where you get your best bang for buck with these types of things?
(00:25:05.199):
Yeah, you definitely have to go with something that allows automation.
(00:25:08.960):
So either you pick something that’s already very opinionated itself,
(00:25:12.441):
like Anchor or Focal, or you basically define your flavor of Data Vault
(00:25:18.182):
and then stick to that particular flavor so that you can automate
(00:25:22.443):
and generate basically everything.
(00:25:25.512):
Just a few days ago, I was doing some research on what people think on data water,
(00:25:31.375):
and I stumbled upon some blog posts where basically people said data water is bad
(00:25:36.437):
because you can’t hand code it because there are too many people.
(00:25:39.698):
So hand coding is better than generation.
(00:25:42.059):
And this is something I definitely disagree with.
(00:25:45.045):
Yeah, I think DBT with their models, right?
(00:25:48.767):
They spawned a whole industry there.
(00:25:51.088):
I mean, it’s kind of interesting, also, if you think about modeling patterns and its popularity.
(00:25:57.291):
I mean, when DBT came out, Dimensional Modeling, in my view,
(00:26:00.793):
kind of went straight back up again, right?
(00:26:03.747):
I saw a hell of a lot more customers using dementia modeling patterns with DPT.
(00:26:09.710):
And I think, you know, one of the things people say
(00:26:13.172):
about things like Anchor is really who is adopting it.
(00:26:17.995):
And again, if I look at Data Vault, you know,
(00:26:19.536):
it’s really weird that it’s got a stronghold in certain European countries like
(00:26:24.078):
the Netherlands, and yet it was created in America and, you know,
(00:26:28.200):
doesn’t really seem to have any mind share or market share there to a degree.
(00:26:32.263):
So, Lars, you know, Somebody did actually ask on LinkedIn when I said we’re going
(00:26:35.674):
to do this somewhere, you know,
(00:26:38.917):
what’s the kind of adoption level of anchor modeling around the world?
(00:26:44.002):
And what’s your view on that? Is it large, small?
(00:26:47.306):
No, it’s small.
(00:26:49.968):
I mean, it’s much smaller than Data Vault.
(00:26:54.393):
But what I can say is that the ones Who have actually converted and started
(00:26:59.956):
to use anchor modeling, they are still using it today and are very satisfied.
(00:27:04.278):
So that brings me a lot of joy.
(00:27:08.860):
I haven’t really aimed for anchor modeling to become super large or anything.
(00:27:14.383):
I haven’t done much marketing.
(00:27:15.923):
I’ve just appeared and talked about it in different venues.
(00:27:22.666):
And it sort of gained traction in the Netherlands to begin with.
(00:27:27.497):
But then it spread out and actually
(00:27:30.580):
at one point it was very large
(00:27:35.405):
and maybe now I haven’t checked in Russia.
(00:27:39.133):
I think that is partially because Nikolai Golov was there and he built
(00:27:43.917):
a very large anchor model data warehouse
(00:27:47.139):
at sort of the Craigslist of Russia.
(00:27:51.462):
It’s one of the biggest sites in Russia where you sell used goods.
(00:27:56.586):
And the data warehouse there was like 500 terabytes or something like
(00:28:01.970):
that of anchor model data and it was also Rapidly changing both when it came
(00:28:08.318):
to sources and business demands.
(00:28:12.900):
So it was an exciting story and he was promoting that quite a lot in Russia.
(00:28:20.925):
So I think that partly was why it gained a lot of traction there.
(00:28:25.188):
And I also want to say that anchor modeling isn’t
(00:28:28.250):
a specifically Tailored for data warehousing.
(00:28:33.901):
It just happens to be suitable for data warehousing.
(00:28:36.923):
It is a generic modeling technique.
(00:28:39.665):
And there are also places where anchor modeling is used in operational systems.
(00:28:45.648):
Like there’s a logistics system built
(00:28:49.171):
in England that is using anchor modeling
(00:28:53.453):
for its systems database.
(00:28:55.174):
So there are also some examples of that.
(00:28:58.016):
But I would say Probably less
(00:29:01.783):
than 100 installations of Anker models around the world.
(00:29:07.349):
Maybe I’m wrong, but I think it’s that small actually.
(00:29:11.694):
Maybe it will take off. I think it has some nice qualities with respect
(00:29:20.433):
AI assistance and AI modeling.
(00:29:22.694):
So yeah, we’ll see about that.
(00:29:25.996):
Maybe we can touch upon that later.
(00:29:28.197):
Yeah, maybe two aspects there.
(00:29:30.558):
One is if you look at the anchor model,
(00:29:33.759):
you have the six and F and you’re closing the circle back to knowledge graphs
(00:29:39.042):
and the stuff. It’s very, very, very similar to these approaches.
(00:29:44.244):
And the other thing is I think the distribution or the hotspots
(00:29:47.826):
of Ensemble modeling techniques like anchor data,
(00:29:50.941):
what it goes back to the Kimball versus Inman boss in a sense that it seems
(00:29:56.722):
that in Europe and also in like Australia and Australia and these places,
(00:30:00.984):
it seems that Inman has prevailed while in the U.S. Kimball has prevailed.
(00:30:04.805):
Because if I look at American data people like Kimball two layer dimensional,
(00:30:11.247):
it seems to be the default.
(00:30:13.187):
And if I look at European Data warehouse for people in European companies and also
(00:30:19.070):
from what I can gather from here at Australia and New Zealand,
(00:30:25.755):
it seems that three layers and Inman is more standard and this naturally leads
(00:30:30.438):
to ensemble modeling techniques.
(00:30:34.080):
Because if you don’t have a middle layer, then where do you put your data?
(00:30:39.504):
Yeah, I think, yeah.
(00:30:41.327):
I was in New Zealand when we were having the Inman versus Kimball wars,
(00:30:45.308):
which I think we associated with Inman and Kimball themselves,
(00:30:49.590):
but actually in hindsight it was us arguing amongst ourselves in New Zealand
(00:30:53.411):
and pretending it was them.
(00:30:55.291):
I don’t think they ever did it right.
(00:30:58.972):
Yeah, there was an interesting, like Lars,
(00:31:01.254):
it’s interesting you say that like one person can kind of spawn off almost
(00:31:04.878):
an industry. I think in Australasia it was,
(00:31:09.543):
Brisbane in Australia was like a hotspot for Data Vault and it just kind of came out
(00:31:13.887):
of there and then it got picked up.
(00:31:16.830):
And then, I don’t know about you Christian, but I saw It’s where the money’s made, right?
(00:31:21.401):
So either training or consulting companies or vendors building products
(00:31:26.042):
and when they enter into backing a set of patterns and those patterns became more
(00:31:35.763):
Seen, I suppose, and taught and used because there’s kind of somebody really vested
(00:31:40.307):
in training you on it or selling you their product on it or using it
(00:31:44.630):
when they consult as their secret weapons.
(00:31:46.732):
So I think that’s the key.
(00:31:49.294):
And probably nowadays, what is it?
(00:31:51.376):
It’s writing lots of content and make sure it’s available to LLM
(00:31:54.178):
so they can suck it up into their foundational models and you have AEO.
(00:31:57.281):
And then when somebody asks the question,
(00:32:00.083):
it’s going to come back and pick those patterns as Peace.
(00:32:05.504):
The best.
(00:32:13.923):
I think the other thing I’m intrigued
(00:32:17.285):
by is I often model in a very opinionated way
(00:32:22.568):
because I have a proven set of patterns and I know how they work and I can test them
(00:32:26.771):
and I know they don’t change.
(00:32:29.332):
Going back to Gen AI tooling and those foundational models and the idea of nodes
(00:32:35.115):
and links being All over the place in the LLM finding the safe path
(00:32:41.319):
through to put it together without me having to precompile it effectively.
(00:32:45.501):
I’m seeing that as an interesting place right now.
(00:32:48.143):
But like you said, Lars, I can never tell how,
(00:32:51.185):
like I can never guarantee it’s going to go through the right path, right?
(00:32:55.247):
And that’s the thing that scares me the most looking at it now.
(00:33:00.966):
So, yeah, that’s me just talking for a while,
(00:33:03.267):
and now I’ve lost the question I was going to have.
(00:33:04.867):
So...
(00:33:06.428):
So you said you
(00:33:18.630):
So you said your thoughts Notts was like,
(00:33:23.613):
well, now we’re in a sailing kind of language.
(00:33:28.095):
We’ll just carry on and Notts came out of that.
(00:33:30.357):
Is that a good guess?
(00:33:32.658):
It is. Yeah. Yeah.
(00:33:33.999):
And it sort of represented what they were also.
(00:33:36.181):
You sort of put some things together.
(00:33:38.742):
It was like these two components that you
(00:33:52.373):
And so let’s go back.
(00:33:53.774):
So that is where you have a reference set of data that that anchor has to comply
(00:33:59.918):
with, like an approved list effectively.
(00:34:02.880):
You’re binding those two things together to say that those two objects having
(00:34:06.963):
a relationship has value.
(00:34:08.584):
So it’s almost like a data contract to a degree, right, in kind of the new world,
(00:34:12.767):
which is if I’m going to put something into that table,
(00:34:15.128):
it has to meet this contract,
(00:34:16.349):
and that contract’s actually describing a set of values that are acceptable into it.
(00:34:21.207):
Yeah, exactly.
(00:34:22.728):
That’s precisely it.
(00:34:26.349):
So, yes.
(00:34:29.891):
I haven’t seen that one before, right?
(00:34:33.272):
I’m sorry.
(00:34:36.194):
He distinguishes open and closed domains,
(00:34:39.495):
and I think the notes are for the closed domains where you have like a fixed set
(00:34:43.497):
of values that are allowed.
(00:34:45.898):
Okay, but normally that most other patterns that sits outside the modeling pattern.
(00:34:51.697):
Doesn’t it? Yeah.
(00:34:52.978):
The kind of enforcement of rules against what data goes into it,
(00:34:59.685):
apart from things like keys maybe, key types.
(00:35:06.630):
Yeah, is that true? No.
(00:35:08.473):
Yeah.
(00:35:09.540):
Yeah, well we…yeah.
(00:35:11.086):
In an act of modeling L -A -L -E -S -T -O -N -F -R -I -G.
(00:35:17.680):
foreign keys as well so That ensures also
(00:35:22.427):
that we stick to that set of values.
(00:35:25.887):
You can’t enter a value that’s outside.
(00:35:28.008):
Even if you try to reference it with an ID that is not in the NOT table,
(00:35:33.509):
you would get an error in the databases that support foreign keys.
(00:35:38.980):
So that means from a load sequencing,
(00:35:40.821):
you now have to load the knots before you load the anchors?
(00:35:43.981):
Would that be?
(00:35:46.242):
Before the attributes and ties. Ah, okay.
(00:35:48.403):
Yeah.
(00:35:49.963):
So you have to load.
(00:35:51.184):
Usually it’s anchors and knots first and then attributes and ties in parallel.
(00:35:56.605):
So you can load anchors and knots in parallel and you can load all the attributes
(00:36:00.226):
and ties in parallel.
(00:36:03.265):
Do most people tend to use a layered architecture when they’re using Anchor?
(00:36:07.067):
So will they have a layer before with, you know, kind of raw, historical, immutable data?
(00:36:12.851):
Then treat the Anchor as the designed layer and then potentially a consumer layer on top?
(00:36:21.708):
Yeah, some sort of staging area is almost a must.
(00:36:25.932):
You have somewhere the data needs to land in the database.
(00:36:29.275):
I usually and always almost prescribe that you use some ELT workflow.
(00:36:35.340):
So you extract from the source load as quickly as possible into the database
(00:36:39.464):
and then do the transformations there.
(00:36:42.607):
So there’s a staging area and then there’s transformation going on on its way
(00:36:47.631):
into the anchor model.
(00:36:49.913):
Then what happens afterwards is a little bit different depending on what
(00:36:56.558):
the customer believes in or what tooling they have.
(00:37:01.521):
Some tooling require almost a dimensional model and then of course they have
(00:37:07.765):
an extra layer with some sort of dimensional models on top of the anchor model.
(00:37:13.069):
But there are also I would say half of the places use the anchor model directly from the tools.
(00:37:20.864):
So the tools are going and
(00:37:22.985):
doing joins, like massive number of joins. Yeah.
(00:37:26.486):
Yeah. Yeah.
(00:37:27.526):
But they usually don’t write the joins themselves because we provide views on top
(00:37:32.608):
of the model that pre-join the model according to different perspectives,
(00:37:38.770):
we call them, depending on if you need the latest perspective or like
(00:37:43.691):
a temporal perspective or point in time perspectives, things like that.
(00:37:51.230):
So you have to sort of flatten the model or look at the model
(00:37:56.854):
through a particular time slice or something like that.
(00:38:00.297):
Because the underlying model is like a time machine.
(00:38:04.941):
You can go to any point in time you want and see what the information looked like.
(00:38:10.643):
So that becomes quite complex if you want to write all the SQL queries yourself
(00:38:15.125):
with the joins and all the temporal reduction logic.
(00:38:18.566):
So we have a lot of views and functionality that help you do that so you don’t have
(00:38:24.068):
to do that by hand.
(00:38:26.349):
And I’m assuming if you’re using a GenAI LLM-based tool,
(00:38:30.415):
Once it understands the patterns,
(00:38:31.936):
it actually know how to write those queries itself,
(00:38:34.737):
even if it did want to handcraft it, because it’s a set of patterns, right?
(00:38:37.438):
You’ve got effectively a set of temporal flags.
(00:38:39.759):
You want to know, do you want to report as it now? Is it then?
(00:38:42.761):
Is it some time in between?
(00:38:45.662):
So it knows the patterns to how to write those queries versus a human that would struggle.
(00:38:50.784):
Yeah.
(00:38:51.785):
Thankfully, AI models of today seem to be aware of what anchor modeling is and how it works.
(00:39:00.085):
I’m not sure that it would prefer them if you would just tell them to build me
(00:39:04.708):
a data warehouse.
(00:39:05.969):
I’m not sure if the model would recommend anchor modeling yet,
(00:39:11.412):
but they’ll know about anchor modeling and they know about the patterns.
(00:39:16.474):
So basically you can take any of the modern models today and say, here’s some data,
(00:39:22.418):
make me an anchor model of this and it would know what to do.
(00:39:27.379):
It would be interesting to go and try that.
(00:39:30.860):
And, you know, I suppose the problem I always have is because when I’m using them,
(00:39:35.282):
there’s a whole set of memory with bias already built in.
(00:39:37.803):
So I’d have to actually figure out how to create a whole new clean session
(00:39:40.984):
with nothing that I’ve talked about before and get it to give me
(00:39:44.286):
an honest answer rather than, you know,
(00:39:47.087):
the sycophant answer where it knows kind of what my flavoring and bias is.
(00:39:50.728):
So it always kind of does what I want as much as it pretends not to.
(00:39:55.531):
Yeah, yeah, yeah. That was one of the first things I turned off.
(00:39:58.513):
No memory of what I’m doing.
(00:40:04.600):
All right. I think this is interesting about, Anke, you already alluded to it.
(00:40:08.181):
It’s like the handling of different timelines and different perspectives.
(00:40:13.545):
You have something built in there that, for example,
(00:40:15.887):
data water or other techniques don’t have to the same extent.
(00:40:18.489):
Maybe you can elaborate on that a bit as well.
(00:40:21.912):
Yes, well, we have.
(00:40:23.711):
Different flavors when it comes to how much timeline management you want.
(00:40:29.135):
So basically the anchor models that we built first back in 2005,
(00:40:34.659):
we call unitemporal today.
(00:40:36.841):
They have one history.
(00:40:40.223):
So that history is normally used
(00:40:43.686):
to represent like traditional
(00:40:47.329):
evolving changes like my hair going gray.
(00:40:52.064):
Things that change naturally over time.
(00:40:54.505):
So you can say that there was one value that was valid at this point in time
(00:40:59.347):
and now it’s a different value.
(00:41:01.168):
But both are valid when you look at it at different time points.
(00:41:05.330):
So like a natural progression of different values evolving over time for something.
(00:41:11.633):
But then we also have bi-temporal anchor modeling.
(00:41:16.222):
So you can generate code that includes two timelines.
(00:41:20.965):
And in that case, you’re also able to represent corrections.
(00:41:27.808):
So you can have a value.
(00:41:30.189):
Let’s say someone says that my hair is red
(00:41:37.553):
at this time point,
(00:41:38.733):
but someone later discovers, well, that was actually wrong.
(00:41:42.775):
It’s brown.
(00:41:44.421):
So then they need to correct the value,
(00:41:46.962):
but then they want to correct the value that was at this time point.
(00:41:51.043):
So they want to say that, well, it was brown actually from today’s date, not something else.
(00:41:58.606):
So that sort of change goes orthogonally to the other type of change.
(00:42:04.068):
So you can actually change, you can even change the time point
(00:42:06.929):
of the When you recorded that the hair was red or brown as well.
(00:42:10.968):
So corrections go like orthogonally towards those naturally evolving changes.
(00:42:17.654):
So that’s the bi-temporal anchor modeling.
(00:42:19.997):
Then we have yet another flavor which is quite complicated.
(00:42:25.159):
It’s a tongue twister.
(00:42:27.284):
We call it Concurrents Reliance Temporal.
(00:42:33.481):
Then we also add on top of those bi-temporal two timelines, we add two more things.
(00:42:40.283):
It’s also who is making sort of a statement about something.
(00:42:45.925):
So then I can keep a record
(00:42:49.667):
of like source system A is saying that the hair is red
(00:42:54.288):
and source system B is saying that it’s brown.
(00:42:57.529):
So I can record a conflict between these two sources.
(00:43:02.594):
And the final thing that we add in that is also a reliability.
(00:43:07.579):
That’s why we call it concurrent reliance temporal.
(00:43:11.482):
So we add a reliability.
(00:43:12.803):
So you can say that, well, the trustworthiness of source system A is 85 percent
(00:43:19.569):
and source system B is 100.
(00:43:21.911):
So you can sort of distinguish or manage also
(00:43:27.903):
The degree of belief that you have
(00:43:30.744):
in these statements that you’re making in the data warehouse.
(00:43:35.126):
Which is that final thing I mentioned now.
(00:43:38.888):
That adds a lot of complexity to the anchor model layer, of course.
(00:43:44.370):
So running queries against the bitemporal anchor model is slower than running
(00:43:50.553):
on a unitemporal model and running it on a concurrent reliance temporal model.
(00:43:55.418):
It’s slower than running on both bitemporal and unitemporal.
(00:43:58.940):
So for each level of functionality that you add,
(00:44:03.043):
you get more complexity and it slows down querying,
(00:44:05.864):
but you gain functionality instead.
(00:44:07.845):
And this raising the complexity was what actually got me
(00:44:14.389):
into that other thing I was talking about, transitional modeling.
(00:44:18.094):
Sort of a new paradigm.
(00:44:19.875):
Can you represent information differently so you can actually create something
(00:44:25.440):
that is performant for that last thing I said?
(00:44:29.262):
Like if you have concurrent reliance temporal anchor modeling,
(00:44:32.185):
instead of trying to squeeze that into a relational database,
(00:44:36.348):
is it possible theoretically to represent it differently
(00:44:39.670):
so you can get something that’s performant to query?
(00:44:44.098):
Even though you still maintain and gain all that functionality.
(00:44:49.609):
Okay, a lot to unpack there.
(00:44:52.510):
So let me start at what I think is the simple part and ask a few questions
(00:44:56.821):
and I’m not sure I’ll get to the complex part.
(00:45:01.803):
So we effectively To begin with,
(00:45:04.141):
we’re storing multiple effectivity dates based on what point of time or what event,
(00:45:10.884):
what behavior we want to report on.
(00:45:13.785):
So what does it look like? Who changed it?
(00:45:17.266):
And if they changed it, when are we saying we’re going to state when that change was made?
(00:45:21.187):
And I’m assuming we hold those dates all separately.
(00:45:24.388):
We don’t overwrite them, so we can always kind of bounce between the different ways.
(00:45:28.050):
So that’s true, right? Okay.
(00:45:30.096):
And those date fields, where are they held?
(00:45:32.938):
Are they held on the tie?
(00:45:34.619):
Are they held on an anchor?
(00:45:37.340):
Are they held on an attribute of an anchor?
(00:45:40.602):
Where do we store those dates physically?
(00:45:44.264):
Yeah, that’s a good question.
(00:45:46.405):
So the anchor itself, it’s sort of
(00:45:51.635):
It’s immutable and it’s always immutable.
(00:45:54.636):
So the anchor itself only stores the surrogate identity.
(00:45:59.257):
So that’s the identity, it never goes away.
(00:46:02.418):
Even if a thing would disappear in reality, we can still remember that thing,
(00:46:08.640):
which means that the identity remains even if the thing itself doesn’t remain.
(00:46:14.741):
So we said from the beginning that anchors are immutable.
(00:46:18.955):
So attributes on the other hand can change like hair color.
(00:46:23.919):
So that can change over time and if you have a unit temporal model
(00:46:27.662):
the effective date would be in the attribute itself.
(00:46:33.707):
So it’s part of the primary key in the attribute.
(00:46:37.750):
Whereas if you have a bitemporal anchor model you actually have a separate table
(00:46:45.763):
That we call an annex table to the attribute in which you store
(00:46:51.611):
when corrections are made.
(00:46:53.313):
So you have a different table with a different primary key there storing sort
(00:46:59.382):
of opinions about this.
(00:47:02.983):
Attribute table.
(00:47:04.764):
So that’s how we solve it.
(00:47:06.785):
It was not a good idea to cram it all into a single attribute table.
(00:47:12.627):
So it made more sense to keep those two timelines in two different tables.
(00:47:17.429):
So you get even more tables if you use bitemporal anchor modeling.
(00:47:21.730):
And then I’m assuming you can mix and match.
(00:47:25.032):
So you can say one attribute is unitemporal and one attribute is bitemporal for an anchor?
(00:47:32.667):
Actually, not right now.
(00:47:34.188):
You select that on a model level.
(00:47:37.229):
But there’s actually no theoretical reason why not.
(00:47:40.391):
It just happened to be a choice that we made in the modeling tool.
(00:47:46.013):
So theoretically, there’s nothing stopping you if you’re doing an anchor model by hand.
(00:47:52.116):
And probably we could open up the tool so you can actually do it also in the tool.
(00:47:57.038):
But the risk is that You miss it on an attribute and, yeah,
(00:48:02.881):
you would have wanted it to be bitemporal and it happened to become unitemporal.
(00:48:07.083):
So right now it’s on the model level.
(00:48:09.004):
You select when you generate the code.
(00:48:11.485):
I mean, the model graphically looks exactly the same.
(00:48:15.107):
So it’s only when you generate the code for the implementation that you select,
(00:48:19.049):
I want a unitemporal, bitemporal implementation,
(00:48:22.230):
or I want a concurrent reliance temporal implementation.
(00:48:25.212):
So the actual model itself doesn’t change.
(00:48:27.533):
It’s just a selection when you generate the code.
(00:48:30.238):
Okay. I like that. I mean, I like simplicity.
(00:48:32.220):
So, you know, I always say, make everything
(00:48:33.962):
a slowly changing dimension type two these days, because why wouldn’t you?
(00:48:38.780):
Why even do a Type 1?
(00:48:40.541):
Because storage and computers are relatively cheap in those models unless you have
(00:48:44.782):
a reason not to.
(00:48:46.502):
Again, I remember back in the old days having massive arguments
(00:48:49.603):
about which of our DIMMs were SED Type 1 and Type 2 because we’re running
(00:48:54.204):
on expensive Oracle boxes that cost a fortune.
(00:48:57.705):
All right.
(00:48:58.966):
So then that third one.
(00:49:01.640):
So I’m assuming the way you hold that third temporality, the reliance one, is that,
(00:49:08.930):
again, you’re physically storing something different if you turn that option on?
(00:49:14.172):
What happens is in that table I call the annex table that you have
(00:49:19.398):
for every attribute, you get some extra columns in that table that also stores
(00:49:25.446):
who is making this, who is having this opinion and also
(00:49:30.912):
with which certainty are they stating that opinion.
(00:49:36.857):
So you get two additional columns,
(00:49:38.619):
but otherwise it looks exactly like the bitemporal implementation from a number
(00:49:44.106):
of tables thing.
(00:49:46.048):
Okay, and the next question obviously is how is that opinion calculated?
(00:49:50.834):
Yeah, that’s a good question.
(00:49:54.910):
I mean, you could have, like I said,
(00:49:57.211):
like every source has a certain trustworthiness, but then you’re sort of,
(00:50:03.612):
it’s quite a rough estimate then.
(00:50:06.912):
I mean, you could potentially have for every different value in every attribute,
(00:50:11.953):
have a different certainty.
(00:50:15.454):
And I mean,
(00:50:17.234):
if you’re working with like maybe Machine learning models or something
(00:50:22.540):
that are spitting out predictions and values,
(00:50:24.781):
then they usually come with some sort of certainty.
(00:50:28.963):
So probably only in different,
(00:50:32.525):
very specific use cases that you could actually use that with specific certainties
(00:50:38.868):
for every value.
(00:50:41.310):
Okay.
(00:50:42.490):
And I was going to go next with that one.
(00:50:46.339):
So the Anchor keys are immutable, right?
(00:50:50.120):
We see them once, we do an insert, we never delete them.
(00:50:53.421):
Same as a Data Vault Hub, effectively.
(00:50:56.042):
How do you deal with conforming concepts?
(00:50:59.063):
You know, the golden record,
(00:51:00.284):
the customer IDs in five different systems with different keys,
(00:51:04.545):
and we want to say it’s the same customer.
(00:51:06.246):
How’s that dealt with within Anchor?
(00:51:09.167):
Yeah.
(00:51:11.456):
An anchor model, we also said that early on,
(00:51:14.377):
since we wanted to have longevity in the model,
(00:51:17.277):
we said that it’s best if the model represents what the business is doing rather
(00:51:23.078):
than looking at the source systems and figuring out what they are storing.
(00:51:28.200):
So we’re very close to the business when we build the anchor models
(00:51:33.681):
and that also gives them Longer longevity,
(00:51:37.429):
because it’s more rare that the business changes what they’re doing
(00:51:42.011):
than source systems changing.
(00:51:45.412):
And that also means that we want to reconciliate identities
(00:51:50.334):
on the way into the anchor model.
(00:51:55.435):
So I’d say that if you use an anchor model,
(00:51:59.277):
you would have a bit more hefty ETL than you would to, say,
(00:52:07.122):
A raw data vault or whatever you want to call them that are source oriented.
(00:52:13.123):
Since we take the step directly from the staging area into the anchor model.
(00:52:18.844):
But of course you can end up in a situation where you’ve made a mistake.
(00:52:23.025):
So you thought to begin
(00:52:25.025):
with that these two different customer numbers were actually two different
(00:52:29.746):
individuals and later it turns out that they were indeed the same individual.
(00:52:35.637):
And then there are different ways to do
(00:52:38.978):
that reconciliation depending on if you’re
(00:52:43.819):
in unit temporal modeling, for example, then, well, maybe in the worst case,
(00:52:49.861):
you actually migrate some of the relationships to point to like the new one
(00:52:57.223):
or consolidate attribute to actually deactivate
(00:53:01.725):
with a flag or Delete the one
(00:53:05.262):
that came in the wrong way.
(00:53:07.763):
If you’re bitemporal or in the concurrent reliance temporal, you have more options.
(00:53:13.144):
You can actually say that, okay, this other, the wrong,
(00:53:18.926):
you did select one and say that this one is now logically deleted
(00:53:22.446):
and the other one is the one that is active and that gains the information
(00:53:26.727):
that was connected to the other one.
(00:53:30.101):
So it depends a little bit on what implementation you have.
(00:53:33.302):
But in unit temporal, it’s a little bit more work and potentially destructive.
(00:53:41.425):
And in bitemporal or concurrent reliance temporal,
(00:53:43.986):
you can manage it and keep the history of what you made wrong and that you had
(00:53:48.747):
an other idea back in the day about those two numbers.
(00:53:52.909):
So we usually don’t use this same-ass link
(00:53:57.250):
or whatever it’s called in Data Vault, but yeah.
(00:54:02.453):
Okay. And then derived attributes, so business logic,
(00:54:07.735):
do you tend to push that left as well so it’s done before it comes
(00:54:11.437):
into the anchor model or do you tend to create derived attributes
(00:54:18.781):
against the anchors?
(00:54:20.229):
Yeah, we tend to create them in a separate pass after.
(00:54:24.170):
Let’s say you have the source system loading.
(00:54:26.831):
Once the source system loading is done,
(00:54:29.872):
that’s usually when we have a second pass that starts and takes
(00:54:34.814):
and derives these calculated attributes and derived relationships and stuff
(00:54:40.676):
that you might have.
(00:54:42.756):
And we usually keep that in the same model.
(00:54:46.898):
So We don’t create a separate layer
(00:54:51.995):
with another anchor model or something that contains those.
(00:54:55.140):
They are in the same anchor model and the same as everything else.
(00:55:00.908):
Good. So nice and opinionated.
(00:55:02.229):
So no more raw vault versus business vault. What the hell is it?
(00:55:05.411):
Where does it live?
(00:55:07.152):
Religious arguments that I seem to remember spending far too long having many years ago.
(00:55:12.075):
All right, we’ve got about five minutes left.
(00:55:13.656):
So Christian, anything else we should cover? What do you reckon?
(00:55:16.898):
There’s a topic that’s close to my heart that’s related to all this last already mentioned.
(00:55:21.500):
We have the way the source systems work, how the source systems represent things,
(00:55:26.263):
and the way the business sees things.
(00:55:29.265):
What we have proposed a few years back,
(00:55:30.866):
and I think we have to revisit it at some point, is starting with the model.
(00:55:35.147):
So what’s called the model-driven organization.
(00:55:37.468):
So basically you say, okay, these are my core business concepts,
(00:55:41.169):
my natural business relationships, and basically I generate everything from it.
(00:55:46.150):
So basically I build my own source systems, I build my data,
(00:55:51.132):
build my org chart from this so that everything is basically Align so that we have,
(00:55:57.567):
okay, we have the concepts and data is grouped around the concepts.
(00:56:01.392):
The people are working for a certain concept.
(00:56:03.955):
And this is something that we have proposed a few as your base back.
(00:56:06.919):
And last, maybe you want to chime in as well, which seems more feasible now in the,
(00:56:11.865):
in the, in the age of AI.
(00:56:14.976):
It certainly does.
(00:56:16.137):
So we sort of turned our concept around.
(00:56:18.919):
I mean, we go from centralization to decentralization to centralization again over time.
(00:56:25.543):
And sort of this is like the ultimate centralization theme
(00:56:32.656):
where you basically have one database in your organization.
(00:56:35.969):
And it’s both the operational database and the analytical database at the same time.
(00:56:41.873):
And it could be an agri-model.
(00:56:44.532):
And you sort of organize the whole organization around this and you build,
(00:56:49.034):
instead of having applications with hefty large databases within them,
(00:56:54.415):
you build light clients or light applications that write and read
(00:57:00.057):
from this central application, central database that you have.
(00:57:05.498):
And when we were looking at this, the technology wasn’t really there,
(00:57:10.240):
but right now there’s You know OLAP and OLTP, that distinction.
(00:57:16.091):
Now there’s actually something called HTAP,
(00:57:19.934):
hybrid transactional analytical processing databases.
(00:57:23.877):
Snowflake is one, for example,
(00:57:25.518):
and they haven’t really pushed this much from a marketing perspective,
(00:57:30.001):
but it can actually do this now.
(00:57:31.562):
It can act both as an operational database and as data warehouse analytical
(00:57:38.067):
and can do both types of workloads simultaneously.
(00:57:43.313):
So I think, and one thing that was like a big blocker, I think,
(00:57:47.714):
when we talked about this question was that who’s going
(00:57:51.995):
to build those light applications against this central database?
(00:57:57.796):
Because when they can buy something off the shelf, I mean,
(00:58:01.357):
that would require large development departments within each organization.
(00:58:06.878):
So that seemed quite far-fetched, but it was like, A really nice thought experiment.
(00:58:13.736):
And now we have AI.
(00:58:15.257):
So it’s not as far-fetched anymore.
(00:58:17.318):
I mean, you can do so much in so little time with AI.
(00:58:20.659):
So, yeah, I think we’re going to see some variant of this over time developing.
(00:58:28.223):
Yeah, I think there’s going to be another our conversation, right?
(00:58:30.520):
It kind of aligns with my thinking where, you know,
(00:58:33.886):
Johar and Coppola and I are writing a book around concept modeling.
(00:58:37.583):
Because we’re coming back to that age-old pattern of define
(00:58:41.289):
the business language first and the relationships as the core pet that you care
(00:58:45.555):
about and then hydrate everything else.
(00:58:47.918):
So you can hydrate your warehouse, you can hydrate your operational systems.
(00:58:52.545):
Okay, everybody’s Vibe coding now,
(00:58:54.667):
but if they were Vibe coding against a set of immutable rules
(00:58:58.110):
that they couldn’t break, then who cares if it’s been deployed on Snowflake,
(00:59:04.592):
Databricks, SQLite, DuckDV, operational, analytical,
(00:59:08.118):
it’s the hydration is fit for purpose for who’s going to consume it
(00:59:11.541):
and can be disposable, right?
(00:59:13.062):
It can be built to a degree.
(00:59:14.825):
And rebuilt time after time.
(00:59:16.506):
So I think, yeah, I think you’re both right that those models,
(00:59:19.187):
those patterns are going to come back to the fore,
(00:59:21.568):
but we’re really early pioneering stage.
(00:59:24.290):
So yeah, we’d spend a good hour going through potential patterns around that.
(00:59:29.873):
So if you’re both keen, maybe get your back and we’ll deep dive on that one
(00:59:33.815):
because I think that’s a topical subject right now and incredibly interesting,
(00:59:38.357):
for me at least, right?
(00:59:39.317):
So yeah. Absolutely.
(00:59:42.032):
It sounds like we’ve just gotten started here.
(00:59:45.054):
So yeah, we can definitely do another hour or two.
(00:59:49.298):
Yeah, we should probably start publishing old lengths for new in a good way, right?
(00:59:55.695):
You see that thing over here that’s called this, you know, Medellin Architecture?
(00:59:59.056):
Well, actually, that’s just a layered data architecture that’s been around
(01:00:01.617):
for 30 years and it has value now again.
(01:00:04.538):
Yeah, maybe that’s that we just keep thinking grumpy old me in that point.
(01:00:07.939):
Not in a good way, right?
(01:00:08.939):
It’s like, well, there’s some patterns over there, just read up on them.
(01:00:11.380):
You’re not reinventing a new thing,
(01:00:13.120):
but you’re just reusing things that are valuable because, as you said,
(01:00:16.501):
the technology has unlocked a lot of the effort that used to be big and expensive
(01:00:21.843):
and long.
(01:00:23.323):
Excellent. All right. No, hey, thank you. Thank you, team.
(01:00:25.465):
I think I’ve got a much better understanding of Anchor.
(01:00:29.728):
I’m going to go play with my favorite LLM and see what it will do for me this time.
(01:00:35.633):
If I tell it I’m an Anchor specialist and it needs to create the models based
(01:00:41.718):
on that, not based on the flavors I normally give it,
(01:00:44.820):
and then tell it to delete the memory so it doesn’t corrupt every bit
(01:00:48.062):
of work I do straight after it.
(01:00:53.229):
I can say that I actually developed an anchor modeling skill for LLMs.
(01:01:00.097):
It’s available on GitHub, so that makes it easier if you use that skill.
(01:01:05.384):
Okay, actually a question I was going to say,
(01:01:07.184):
so the application you built that builds Anchor Modeling,
(01:01:12.025):
is that a commercial or an open source?
(01:01:14.726):
It’s open source, yeah.
(01:01:16.286):
And if you go to anchormodeling.com, you can click on the left-hand side,
(01:01:22.427):
click on the online modeler, and you probably want the test version.
(01:01:26.868):
We’ve used that in production already, so it’s quite safe to use.
(01:01:30.888):
But use that one because that has better support for different database targets.
(01:01:35.500):
Like, yeah, BigQuery, Snowflake, Vertica,
(01:01:38.881):
SQL Server, Oracle, Postgres, yeah, stuff like that.
(01:01:43.063):
Thank you for saying BigQuery first.
(01:01:44.764):
So the modeling pattern itself is open and free.
(01:01:48.565):
The tools that you’ve built are open source and therefore able to use,
(01:01:52.027):
and you have skills that can be put into your favorite LLM to help somebody start
(01:01:57.409):
to understand how to use it. Yes.
(01:01:59.410):
Yeah, makes perfect sense.
(01:02:02.271):
And yeah, I look forward to playing with it more.
(01:02:05.510):
Excellent. All right. Well, thanks.
(01:02:06.832):
Thanks, guys, for coming on.
(01:02:07.913):
And I hope everybody has a simply magical day.
(01:02:11.917):
Thanks for having us. Thank you very much.
(01:02:13.979):
It was great fun.
(01:02:18.925):
Okay.
«oo»
Information Product Canvas Workshop
“That’s not what I wanted.”
“But that’s what you asked for.”
Shane runs a 6 hour workshop on the Information Product Canvas, a repeatable way to gather data requirements in 30 minutes. The approach he actually uses with stakeholders, not a textbook version that falls apart on contact with messy data and vague requirements.
He runs it regularly.
Latest dates : Gather data requirements in 30 minutes using a shared language
Workshop details : https://maven.com/agiledataguides/info-prod
