As we worked with an Agile Data Network partner this week to help them onboard a new Customer, we decided to use an ADI first approach for all the data work in the AgileData App and Platform and see what would happen.

And of course part of all data work is data modeling.

So we got ADI to look at the source system data that had been collected into History, it was data we had never modeled before, and got her to take us through the data modeling process.

Our Agile Data Network partner had already worked with the customer to understand their required Information Products and so had a good understanding of the Core Business Concepts, Core Business Processes, Facts, Measures and Metrics that would potentially meet their organisational needs.

This also stopped us letting ADI define a source system specific data model that would break on the first engagement with a Stakeholders chnaging requirements.

ADI did ok, but we always know we can do better, so time to McSpikey.

Agile Data Disco

Last year we did a raft of work for a Customer, where we reverse engineered 100’s of their Cognos Report definitions to help document their legacy data platform before they moved to a greenfield’s Modern Data Stack. This work removed the need for a team of Business / Data Analysts to spend months documenting the legacy system.

We ended up building this in public and semi-productising it, calling it Agile Data Disco.

You can read about that journey here:

We saw a potential market solving that data problem of understanding what a legacy data platform actually contains.

https://agiledata.team/data-problems/use-case/legacy-data-platform-discovery/

You can see an interactive demo of the final Agile Data Disco product we built to help us do that Fractional Data Work here:

https://agiledata.cloud/disco/#demo

One of the things we did as part of Disco was some interesting prompt engineering to take a single input (say a blob of SQL, a log from SQL execution or the defintion of a report) and from that create a series of useful populated Pattern Templates as the output.

The outputs we generate are:

• Information Product Canvas

• Event Model

• Conceptual Model

• Physical Model

• Reporting Model

• Business Glossary

• Data Dictionary

• Metric Definitions

• Bus Matrix

• Source Mapping

And we also did another McSpikey with Disco where I uploaded an image of a completed Information Product Canvas and had Disco generate all those object, from that one image.

You can read about that one or see the Interactive Demo for it here:

Can you guess where we went with this

Yup we took the core parts of Disco and did a McSpikey to see how much of the Disco prompt and reinforcement logic we could reuse to improve ADI.

Results of the McSpikey still to come

I still need to take the time to write up the results of the McSpikey properly, but the TL:DR is:

• Reusing the Disco prompt and reinforcement logic had massive value “time to value” wise;

• We ended up experimenting with breaking out a ADI sub agent - ADI the Agile Data Modeller, which had some real benefits in improvement in the ADI responses.

A word from our sponsor

One of the things we got asked in the early days of Disco was rather than just document what was in the legacy Data Platform, could we automagically migrate it to a new data platform.

When we thought about it that looked like a very complex problem, that would take a few years of development to make feasible and we would end up with a many to many problem before it was viable, we would need to read from many tools and technologies and also need to write to many tools and technologies. Until you had critical mass at both the read and writes ends the product wouldn’t be viable.

And we also saw data platform modernisation as a great catalyst for organisations to rethink the way their data teams worked, and to rearchitect a lot more than just their database, ETL tool and BI tool. So we weren’t fans of supporting a better “like for like” modernisation pattern.

But we also had in the back of our mind the idea that if we could take a Customers legacy data platform and automagically migrate it to the AgileData Platform with minimal human effort, that would be very valuable in removing one of the key points of friction for working with potential Fractional Data Service customers who had already had invested a shit ton of money in a data platform (legacy or modern).

Because our AgileData Platform is based on our very opinionated Ways of Working, Data Engineering and DataOps patterns, this wouldn’t be a “like for like” but an automated rebuild from new.

As we experiment with ADI the Agile Data Modeler we are wondering if that Agent on its own has some value. Upload your current data model, and get back a bunch of candidate data models to review.

This wouldn’t be just a LLM going text to text, we would extend the reinforcement model we used for Disco, and bring in our opinionated Business, Concept and Physical data modeling patterns to it as well.

If I have to place a bet (and I do) I am going to carry on with the Context Plane bet.

All the experimentation and development we do in that space is also immediately available in the AgileData App and Platform, so another form of DORO.

But if you think there is value in an ADI the Agile Data Modeler on her own, reach out and lets have a chat. I would be keen to understand the use case you have in mind.

An incoherent stream of Context

You can find all the previous articles with my train of thought listed in this thread:

https://agiledata.substack.com/t/context-plane

We are building the Context Plane while flying it, so always looking for early adopters to help us decide the final destination.

If you want a virtual chat grab a slot here:

https://contextplane.ai/contact-us/#bookemdanno

Different personas and use cases demand different ways of interacting with Context.

“Context” of this post

I often find writing helps me coalesce and refine my thoughts when new patterns start to emerge, but aren’t very clear yet.

So this article is a brain dump / train of thought continuation of the architecture needed to have one Context Plane to rule them all, as part of a proposed “AI Data Stack”.

This article provides an overview of the specific persona types / use cases I have identified so far that need to access the Context Plane and the typical UX patterns for some of those.

Updated Context Plane Architecture

This iteration in my thinking has a resulted in an update to the architecture diagram for the Context Plane:

The four personas / use cases

Here are the 4 persona types / use cases I have discovered so far:

• Human, GUI centric
  A person who wants to use a web based App, chat or Graphical User Interface to access/discover/explore/update the Context.

• Human, Code Centric
  Wants to use a Command Line Interface (CLI) or Code based App to access/discover/explore/update the Context.

• System
  Systems can access the Context directly, either querying the Context or creating to it programatically.

• Agent to Agent
  AI Agents can access/discover/explore/update Context autonomously, collaborating with other agents without a human in the loop.

The UX patterns

Here are the UX patterns I have experimented with that seem to make sense:

• Human, GUI centric

  • GUI centric Data Catalog

  • GUI centric Chatbot

• Human, Code Centric

  • GenAI App

  • CLI tool

• System

  • API’s

GUI centric Data Catalog

Your typical browser based Data Catalog interface.

Prefer a manual click through version?

https://guides.agiledata.io/demo/cmflomypg048l170irpj5zf6h

Click through demo

GUI centric Chatbot

Your typical browser Chatbot and Text to SQL interface.

GenAI App

Typical Claude “AI Agent” app tool interface.

CLI Tool

Typical Gemini CLI command line interface.

[Screenshot TBA]

API’s

Typical API endpoints.

Missing UX patterns

The key UX pattern I have yet to discover is how the Agent to Agent UX works.

I think we will need to do a McSpikey with the Google A2A to understand the options in that space a little more.

The Technology patterns

We already had a lot of the technology patterns in place before we started experimenting with the Context Plane.

Things like the browser based Data Catalog capability.

We also had a lot of ADI already built based on “AI Assisted” features we have experimented with over the last 6 odd years.

Our App and Platform architecture has always been based on API’s in the middle:

App > API > Context > Code / Data

The main iteration technology wise has been the addition of a MCP server. This has allowed the use of tools like Claude and Gemini CLI.

We have iterated ADI to use the MCP server to access the Context (well we actually use a hybrid access model but ill leave the diagram as simple as this for now.)

So many new questions

User?

UX stands for User Experience, but some of these persona types and use cases are machines not humans, should they still be referred to as Users?

BI Semantic Layer?

Where does the typical BI Tools and the “BI Semantic Layer” pattern fit into this?

For Context Plane we are only holding an Organisations Context not their Data, so we can’t execute any queries, like we can in the AgileData App and Platform.

Or do we want to look at generating the query the human can cut and paste into the data platform.

Or do we want a Context Agent to push the Query to a BI agent inside the Organisations agent ecosystem?

We don’t provide a caching layer or query rewrite patterns which is what the BI Semantic Layers / Metric Layers are doing these days. Im pretty sure we don’t want to go there.

When will BI Tools move to using MCP servers as a way of querying the data?

When will they all put Agents in from of their BI Semantic Layers"?

One step forward, but a raft of new uncertainties

So many new questions, so few answers.

Looks like there even more McSpikeys to add to the list!

Wood from the Trees

Still a way to go before I have a coherent set of Patterns that I can Coach / Mentor / Teach somebody else for the “Context Plane”, and the “AI Data Stack” or present as a robust Architecture map.

But as I have already said, writing my half formed ideas helps me think.

An incoherent stream of Context

You can find all the previous articles with my train of thought listed in this thread:

https://agiledata.substack.com/t/context-plane

We are building the Context Plane while flying it, so always looking for early adopters to help us decide the final destination

If you want a virtual chat grab a slot here:

https://contextplane.ai/contact-us/#bookemdanno

“Context” of this post

I often find writing helps me coalesce and refine my thoughts when new patterns start to emerge, but aren’t very clear yet.

So this article is a brain dump / train of thought continuation of the architecture needed to have one Context Plane to rule them all, as part of a proposed “AI Data Stack”.

This article provides an overview of a way to describe both the Object Types and the Object Categories that will hold the metadata I think should be included in the Context Plane as part of a new “AI Data Stack”.

The Context Plane Architecture

I have written some initial thoughts about the architecture of the Context Plane and the things that shoud be stored or federated in it before.

As a quick reminder my current thinking is the architecture diagram for the Context Plane should look something like this:

Object Types stored in the Context Plane

In that previous article I did a bit of a stocktake of all the Patterns and Pattern Templates I see in the Data Domain that hold what I would term as Context.

I have been mulling those over more as we do each McSpikey (experiment) to learn more by doing and to reduce some of the many uncertainties we have identified.

I am currently sitting with this list of Object Types that seem to have value being stored or federated in the Context Plane:

• Outcomes

• Actions

• Business Questions

• Business Glossary of terms, aliases and descriptions (use tags for Aliases)

• Conceptual Data Model

• Logical Data Model

• Physical Data Model

• Data Dictionary (with schema, fields types, field descriptions etc, also flags)

• Facts

• Transformation code

• Data Quality Rules

• Data Contract (Boundary of other Objects))

• Measures, and their formulas

• Metrics and their formulas

• Information Applications (Reports, dashboards, AI Agents etc)

• Information Products (Boundary of other Objects)

• Data Quality Scores

• Notifications

• Usage Statistic

• Data Sync Statistics

• Number of rows in tiles

• Principles

• Policies

• Patterns

• Personas

• Previous effort of change

Semantic Language is important

I often rail against the lack of clear Semantic definition in the Data Domain, but I am often as loose with my definitions as anybody else.

I had original used the following terms:

• Context Object

• Context Object Types

But as is the LinkedIn way Gabriel Tanase kindly reviewed my crap semantic definitions and provided a much better version.

https://www.linkedin.com/feed/update/urn:li:activity:7366194247613501441?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A7366194247613501441%2C7366400089965129728%29&dashCommentUrn=urn%3Ali%3Afsd_comment%3A%287366400089965129728%2Curn%3Ali%3Aactivity%3A7366194247613501441%29

So based on that I have ended up with:

• Context Object
  A single instance of something in the Context Plane (e.g. the action “Approve Loan”).

• Context Object Type
  The class or category that defines what kind of object it is (e.g. Actions).

• Context Object Type Category
  A higher-level grouping of related object types that share a domain (e.g. Business Context).

Not reinventing the Wheel

Gabriel also pointed me at some excellent documentation from Collibra in that thread, for example:

https://productresources.collibra.com/docs/collibra/latest/Content/Settings/OperatingModel/to_operating-model-settings.htm

Im always keen to reuse Patterns from others, rather than reinventing the wheel so to speak, and so I went down a rabbit hole of the excellent Collibra documentation.

I came away with a feeling of an architecture that is founded on the Pattern of “thing is a thing of a thing”.

Which of course is an infinitely flexible Pattern, but to me has always been an Anti-Pattern and so not a Pattern I want to adopt.

I will need to spend more time looking into bth the Data Catalog and Information Science domains as I am pretty confident that the Patterns I am looking for already exist.

But there is also the value of trying to defining these patterns myself, as I learn by doing (and struggling). Plus when I finally think i have nutted something else and I spot that same pattern articulated in lots of other places then I knwo I am on to something repeatable and valuable.

Context Object Type Categories

Back on task, these are the Context Object Type Categories I have ended up with.

• Business Context
  Captures the intent, language, and needs that connect data work to business language.

• Structural Context
  Describes the technical metadata that defines how data is stored, shaped, and connected.

• Operational Context
  Provides the live signals and guardrails (trust scores, usage stats, policies, and access rules etc) that keep data reliable and governed in practice.

• Agent Context
  Provides the prompts and guardrails that guide AI agents in applying context.
  

Yes Structural Context is the same as what is commonly called “Technical Metadata”.

My definition for Operational Context has got examples, which means I do not have a clear semantic defintion for it yet, so I need to keep iterating that one.

Mapping Context Object Types to Categories

I stress tested my categorisation by trying to assign the Context Object Types to one and only one category.

Moving to the AssistedAI pattern and letting my ChatGPT friend expand my bullet point list to become richer text and write a more detailed story, ever so slightly edited by me ….

When people talk about data, they often focus on the raw tables, pipelines, and dashboards. But the real power comes from the Context that surrounds them, the language, the intent, the rules, and the patterns that let both Humans and AI agents understand what data means and how it should be used.

In the Context Plane, we capture that knowledge as a set of Object Types. These are building blocks that represent everything from business questions to transformation code, from policies to personas. Together, they create a shared layer of context that connects business, data, operational and agentic worlds.

To make sense of them, we group these Objects Types into four categories: Business Context, Structural Context, Operational Context, and Agent Context.

Business Context

Object Types in the Business Context category capture the why behind the data: the intent, the questions, and the people who care.

• Business Questions – The driving questions that stakeholders ask and want answered.

• Actions – What people (or processes) actually do with data once they have it.

• Outcomes – The results or changes in the business that happen because of those actions.

• Business Glossary – Shared terms, their aliases, and agreed definitions, so everyone speaks the same language.

• Conceptual Data Model – The high-level map of business concepts (customers, products, orders) and how they relate.

• Logical Data Model – A more detailed structure that shows how those concepts can be represented in data.

• Personas – Representations of the different types of people who use or consume data, each with their own needs.

• Information Products – The boundary objects that package up context into a consumable unit, such as a unified customer view, a churn model, or a financial performance pack.

Structural Context

Object Types in the Structural Context category describe the what of data: the way it’s stored, shaped, and transformed to become useful.

• Physical Data Model – The actual schema and structures in the database.

• Data Dictionary – A catalog of fields, data types, and descriptions, along with useful flags.

• Facts – Core, measurable events or counts in the data (e.g. sales transactions, logins).

• Transformation Code – The SQL, scripts, or pipelines that turn raw data into shaped, ready-to-use structures.

• Data Quality Rules – Checks and guardrails that ensure the data stays trustworthy.

• Measures – Defined calculations, such as revenue or average handling time.

• Metrics – Business-relevant measures with formulas and thresholds (e.g. Net Promoter Score, Churn Rate).

• Information Applications – The outputs where information is applied and consumed: reports, dashboards, AI agents, apps.

Operational Context

Object Types in the Operational Context category describe the how well and at what cost dimensions of data. These objects let us monitor and manage the reliability and change effort of the system.

• Usage Statistics – Insights into who is using which data, when, and how often.

• Data Quality Scores – Aggregated indicators that show how healthy the data is across rules and checks.

• Data Sync Statistics – Performance and freshness indicators of data movement.

• Number of Rows in Tiles – A quick proxy for data size and growth in a given slice of the platform.

• Notifications – Alerts that something requires attention, whether it’s a failed job or a threshold breach.

• Principles, Policies, Patterns – The guiding rules and repeatable approaches that shape how data is managed.

• Previous Effort of Change – A signal of how much time, cost, and disruption a past change required, helping estimate future impact.

Agent Context

Finally, Object Types in the Agent Context category deals with the prompts and instructions that guide AI agents in using all this context. These Objects define the boundaries and cues for generative systems to behave predictably and helpfully.

• Prompts – The structured inputs, templates, and guardrails that tell an AI agent how to use context objects to answer questions, generate insights, or perform actions.

Why it Matters

Each Object Type might seem less than valuable in isolation, but together they form a unified layer of meaning, that can be used by Humans and AI Agents alike to gain understanding.

By capturing both the why (business intent) and the what (structural detail), as well as the how well (operational health) and the how to guide agents (AI context), the Context Plane ensures that every stakeholder—whether a business leader, a data engineer, or an AI agent—works from the same playbook.

That’s how you move from disconnected metadata and data assets to a coherent ecosystem where context is always present, always available, and always shared.

And back to all me again ….

And once again with a map

Visual maps often give a better quick view than plain text, so here you go:

Far from Done Done

Based in my experience to date I will be iterating the Object Types and their categorisation for a while yet, but I think I have the the four Context Object Type Categories pretty stable.

Famous last words?

Wood from the Trees

Still a way to go before I have a coherent set of Patterns that I can Coach / Mentor / Teach somebody else for the “Context Plane”, and the “AI Data Stack” or present as a robust Architecture map.

But as I have already said, writing my half formed ideas helps me think.

An incoherent stream of Context

You can find all the previous articles with my train of thought listed in this thread:

https://agiledata.substack.com/t/context-plane

We are building the Context Plane while flying it, so always looking for early adopters to help us decide the final destination.

If you want a virtual chat grab a slot here:

https://contextplane.ai/contact-us/#bookemdanno

“Context” of this post

I often find writing helps me coalesce and refine my thoughts when new patterns start to emerge, but aren’t very clear yet.

So this article is a brain dump / train of thought continuation of the product and architecture needed to have one Context Plane to rule them all, as part of a proposed “AI Data Stack”.

This article provides an overview of adding the AI Agent capability to the AgileData Web App.

Not starting from scratch

We have been experimenting with an in App “Chatbot” helper in the the AgileData App for multiple years so I was lucky we already had the core UX needed for us to experiment with this feature.

And we had already integrated Gemini with the AgileData App a while ago. We already used it to assist or automate a few of the more complex bits of data work.

So it was ”just” a case of changing the Gemini Model we were using to switch over to use Gemini 2.5 Pro to get the reasoning capability and also adding the connector to expose it to our MCP Server.

So from a AgileData App UX point of view nothing changed.

Same Prompt

We have a simple set of prompts already defined in the AgileData Platform for ADI.

You are ADI, the AI assistant inside the AgileData.io product. Your purpose is to guide a data team of one—a data analyst—in configuring a data warehouse using the AgileData App and Platform.

Context & Capabilities:

You have access to:
	•	The AgileData user guides
	•	The screens and workflows within the AgileData app
	•	The APIs that power the AgileData app and platform
	•	The data stored in the AgileData platform

The Challenge:
	•	Every set of data has unique challenges.
	•	There is no single, linear process for working with data—users must “pick a path” or “choose their own adventure.”
	•	Each user will dynamically use the AgileData app in different ways based on their specific needs.

Who You Are Helping:
	•	The user is data-savvy but not a data professional, not a highly technical user, not a data expert.
	•	They may not always know exactly what data task they need to do—but they know the outcome they are aiming for.

How You Should Respond:
	•	Be clear and practical: Guide the user toward the best approach to complete their data task efficiently.
	•	Adapt to their needs: If they are unclear on what to do, ask clarifying questions and suggest possible paths forward.
	•	Use all available information: Leverage guides, app screens, APIs, and platform data to provide the most relevant answers.
	•	Encourage exploration: Offer multiple options for how they could proceed, allowing them to discover the best way forward.
	•	Think like a mentor, not a manual: Rather than just explaining how the platform works, help the user make decisions that will get them to their goal.
	•	Present them with the relevant url to the relevant screen if that will help them complete the task

When you have multiple choices:
	•	Pick the one you think is the most suitable and do that.
	•	Append the other options you had to your response and tell the Data Analyst which choice you selected, let them know they can tell you to pick another option.


Your goal is not just to answer questions but to help the user successfully build their data warehouse in the most effective way possible—no matter what challenges they face.

As you can see they are based on previous experiments where we were focussed on helping a Data Analyst build a Data Warehouse as a “team of one”.

I left this prompt in place for this experiment, but I should do another McSpikey where I tailor them for the Context Plane use cases.

In previous experiments where I used Gemini CLI and Claude to access the Context in the Context Plane, there was no “system” prompts in place at all so this should mean I get slightly different responses from Gemini 2.5 Pro in the AgileData App vs Gemini 2.5 Pro in the Gemini CLI.

ADI also has access to our Doco and also examples screens from the App, so again it has more Agent Context than the previous experiments with Gemini CLI had, as that only had access to the API’s via the MCP Server.

MCP Services

We limited the services available in the AgileData App to a subset of the available APIs.

When I use my typical question to get a list of Tiles and a list of Fields in a specific Tile then it pretty get the response I am expecting, no surprises there.

Except ……

When I use Claude to get the Blast Radius for a field change it gives me very rich Context back.

ADI in the AgileData App however…

She ain’t being that helpful!

Over to Gemini CLI to see if its her or Gemini.

Ah nope

We made a fundamental change to the MCP Server we are using since the last McSpikey. The previous version only had access to a subset of our API’s as tools. We changed it to have access to all API’s.

For Claude this didn’t really make much of a difference, but for Gemini CLI it seemed to have a major impact and not in a good way.

Listing the tiles always seems to work ok, simple task.

Listing the fields in a tile has worked well up until we made the MCP Server Tool change.

That is the LLM version of go F$#@ yourself lol.

I had a quick go to see if I could limit the Tools that Gemini CLI could access on the client side, as we limited them in both the AgileData App for ADI and I limited them in the Claude client.

So new entries in the Gemini CLI settings.json it is:

  "securityPolicy": {
    "mode": "configured",
    "allowedTools": [
      "get_business_glossary",
      "get_catalog_tiles",
      "get_ensemble_config",
      "get_change_rules"
    ]
  }

But that did nada, when I listed the tools in Gemini CLI using /mcp it could still see them all and there was no behaviour change.

If I was very specific with my language it would sometimes work:

But then it would go back to be a real dumb arse again.

So back to very specific language:

So theories from that rabbit hole:

• To many MCP Server Tools makes the AI Agent Tools job harder

• The previous Tools we exposed had some additional hints in the naming of the Tool that probably helped the AI Agent Tools

• Gemini 2.5 Pro is not as good as Claude Sonnet 4 for this use case

One more thing

I ran Gemini CLI against a different AgileData Tenancy, Newcastle, compared to the ADI version which was running in the Kapiti Tenancy (hence the change in Tile name I was asking questions about).

Lets go try ADI in the Newcastle Tenancy and see what happens.

Nope same problem.

So Im going to guess the problem could be:

• the Gemini 2.5 Pro model

• and / or the extra MCP Server Tools

• and / or the Prompts in the AgileData Platform

• and / or all of the above

that is causing the problems.

Out out time for this McSpikey so thats is an experiment for another day.

So in Summary

We can provide a web based interface in the AgileData App easily enough by reusing the current ADI capability.

But need to work out if it actually has any value.

And also validate the assumption that it would be a different personas using it compared to a AI Agent Tool and should that impact the type of response they receive or not. Gut feel is no, it should be “any Human, any AI tool, same response”

And I need to experiment with the combination of LLM models, MCP Server Tools and Prompts a lot more.

Claude Sonnet 4 seems more compotent than Gemini 2.5 Pro

Well for this use case anyway. I just seemed to get back better “Context” from my Context when I use Claude over Gemini.

As slow as a wet pig

One of the surprises was ADI was sooooo slow in the Kapiti Tenancy vs the Newcastle Tenancy.

Check it out.

Newcastle

Kapiti

Almost twice as long in Kapiti vs Newcastle.

One of these things is not like the other

After chatting with Nigel it seems were running Gemini 2.5 Flash in Newcastle and Gemini 2.5 Pro in Kapiti, hence the difference in response times.

So another McSpikey needed to see how different the responses are between the two models for our key use cases.

Its all about the reducing uncertainity

The key to a McSpikey is to reduce uncertainty and for this McSpikey I think we have increased the uncertainty.

But thats ok, better to know that now than later when we are much further down the Context Plane path.

Wood from the Trees

Still a way to go before I have a coherent set of Patterns that I can Coach / Mentor / Teach somebody else for the “Context Plane”, and the “AI Data Stack” or present as a robust Product Overview and / or Architecture map.

But as I have already said, writing my half formed ideas helps me think.

Context Plane Use Cases

You can find all the use cases where I think the Context Plane may have sone value over at:

ContextPlane.ai

An incoherent stream of Context

You can find all the previous articles with my train of thought listed in this thread:

https://agiledata.substack.com/t/context-plane

We are building the Context Plane while flying it, so always looking for early adopters to help us decide the final destination.

If you want a virtual chat grab a slot here:

https://contextplane.ai/contact-us/#bookemdanno

“Context” of this post

I often find writing helps me coalesce and refine my thoughts when new patterns start to emerge, but aren’t very clear yet.

So this article is a brain dump / train of thought continuation of the architecture needed to have one Context Plane to rule them all, as part of a proposed “AI Data Stack”.

This article provides an overview of creating and updating Context via a AI Agent Tools such as Claude and Gemini CLI within the “Context Plane” and via the use of a MCP Server.

Are they really AI Agents?

First thing first are Claude and Gemini CLI’s really “AI Agents” or are they just “Tools” that you can use to build agents?

I think of them as tools, other people I have asked think of them as Agents, so for now lets call them “AI Agent Tools”, to cover both Semantic versions.

Architecture Reminder

As a reminder of what we are currently working with we ended up with an architecture that looks like:

• Claude, Gemini CLI etc talks to MCP Service

• MCP Service calls applicable API (exposed as MCP tools)

• API talks to “Context Plane”
  

Connecting Claude to MCP server

Nigel uses Claude for his McSpikeys on the “Context Plane” and so I would join the fun.

First thing was download the Claude App for my Mac, and then find the option to set it to dark mode so I can pretend to look like a proper engineer.

Second thing is you need to pay for Claude if you want to use it with MCP Servers, so NZD $40 a month added to the ever growing AgileData McSpikey toolkit cost bucket.

Next thing hook it up to our “Context Plane” MCP Server via the custom connector:

A little bit of cheating on the auth side for this McSpikey but Claude makes it all pretty easy.

Update, Update baby

(Sung to the tune of Ice, Ice baby)

Nigel has done his plumbing magic and exposed the API that allows the creation or update of a Catalog Tile (think of Tiles as Conceptual or Physical tables).

New tool is enabled for Claude.

And its available for Gemini CLI.

Any AI Agent Tool

I can see a lot of value in the concept of a MCP Service and tools being exposed via that service.

People can bring what ever AI Agent Tool they prefer to do the work, and we can expose new tools at will and they all just talk nicely to each other.

Imagine if all our other data, analytics and BI tools had that level of interoperability!

Lets update a Tile

List Tiles

First lets list the Tiles that are available in the Context Plane.

Gemini CLI was way slower at processing this task than Claude!

Lets try an update

Its still easier for me to quickly check the details for a Tile in the AgileData App:

Claude first.

Then Gemini.

Ok simple example but proves it all works.

Lets create a new Tile

Claude first.

Hell no I ain’t spending time giving you all that “Context” (well not for this McSpikey anyway).

So minimal details you shall have.

Gemini’s turn.

Mmmmm well I know what I need to provide as the minimal amount of detail to create a Draft Tile, but most won’t, so Claude deffo wins this one so far.

But Gemini did do the job.

Update Tags

In the AgileData Platform we allow the same Tile Alias to be used (of course each Tile gets a unique GUID), so lets see if we can differentiate them using Tags.

Claude first

We hold Topics as an array for a Tile. Claude worked out that Tag was the same as Topic, but failed at adding one to the Tile. It did try a few different way to try and achieve its task.

Over to Gemini to shine (or not as the case maybe)

Mmmm but have you really Gemini?

Nope in fact you have not, you may call it Hallucination I call it damn lying!

So in Summary

I am not sure this is the fastest way for me to create and edit Tiles in the “Context Plane”. But that is because I know how to quickly do it via the AgileData App.

I can see for people that prefer code, this would be much faster for them.

Claude is deffo more helpful with its feedback loops as it does the work, and is also much faster in doing those tasks.

But I think the “use any tool” pattern has massive value, what I am thinking of as:

As does my other favourite saying:

The fact we can expose our API’s as Tools to the MCP Service and any “AI Agent Tool” gets access to it straight away, is one step closer to the data technology “Interoperability” that we have been striving to achieve for over the 3 decades I have being working in the data domain.

The “Golden Path” problem

This is a major problem for us in the AgileData App.

Nigel and I know how to use the AgileData App and Platform to do data work that takes hours in minutes. But nobody else does.

As Data Professionals already know, there is no “Golden Path” to doing data work. No linear steps of this then that, which apply everytime.

Every new set of data kicks your arse the first time you touch it.

So we designed the AgileData App and Platform so that it allowed you to do the work in any order.

But that made it confusing for people who have never used it before, or done data work very often.

We tried a bunch of things over the last 6 1/2 years, user guides, video examples, click through demos, and even a ChatBot, nothing really nailed it.

I can see how “AI Agent Tools” like Claude and Gemini CLI solve this “Golden Path” problem.

But as we experiment with the “Context Plane” I think we have stumbled across another bigger complex problem, at potentially the right time, that we can solve with simplicity.

So of course I had to create a website just in case I was right.

https://ContextPlane.ai

Time to design a new T-Shirt.

Actually if your reading this Nigel, of course i’m not! (um well …..)

Wood from the Trees

Still a way to go before I have a coherent set of Patterns that I can Coach / Mentor / Teach somebody else for the “Context Plane”, and the “AI Data Stack” or present as a robust Architecture map.

But as I have already said, writing my half formed ideas helps me think.

An incoherent stream of Context

You can find all the previous articles with my train of thought listed in this thread:

https://agiledata.substack.com/t/context-plane

We are building the Context Plane while flying it, so always looking for early adopters to help us decide the final destination.

If you want a virtual chat grab a slot here:

https://contextplane.ai/contact-us/#bookemdanno

“Context” of this post

I often find writing helps me coalesce and refine my thoughts when new patterns start to emerge, but aren’t very clear yet.

So this article is a brain dump / train of thought continuation of the architecture needed to have one Context Plane to rule them all, as part of a proposed “AI Data Stack”.

This article provides an overview of integrating and using Claude and Google Gemini CLI with the “Context Plane” via a MCP service.

MCP (Model Context Protocol)

The next thing I wanted to understand was how a MCP would work with the current version of the “Context Plane” we have in the current AgileData Platform.

McSpikey

So time to do a McSpikey (experiment) with Nigel. Success for this experiment was to be able to query the AgileData “Context Plane” with tools such as Claude and Gemini CLI.

MCP Server

We are a Google Cloud only platform so we thought that using this MCP server would make sense

https://github.com/googleapis/genai-toolbox

But after some investigation Nigel worked out it would mean a lot of rework, as we would have had to define all the tools and sql queries needed to query Google Spanner, where we store our Context.

Since we already have all those queries defined as part of the AgileData Platform API’s. Nigel discovered that we could just wrap our existing FastAPI endpoints with FastApiMCP and so we implemented that.

In the words of Nigel:

We always like minutes over hours at AgileData and we love the DORO principle (Define Once, Reuse Often)

There were some challenges deploying FastAPIMcCP via Google App Engine, if you want more details on that just ask and Nigel can write up a more technical post on how that went.

Architecture

So we ended up with an architecture that looks like:

• Gemini CLI talks to MCP Service

• MCP Service calls applicable API (exposed as tools)

• API talks to “Context Plane”
  

Connecting Gemini CLI to MCP server

Its always funny what takes the most time when doing a McSpikey.

When Nigel setup the MCP server he connected to it using Claude, next learning you need to have a paid version of Claude to provide MCP connectivity.

So I though I would use Google Gemini CLI for me.

Next learning, the paid version of Claude makes it very easy to define the MCP connection, you pretty much paste the url for it in.

Gemini CLI not so much, you need to frig around with JSON strings in a settings.json file.

And of course finding the exact structure for that JSON is a pain.

After a bit of trial and error got it working with this:

{
  “selectedAuthType”: “oauth-personal”,
  “theme”: “Dracula”,
“mcpServers”: {
  “MCPServer1": {
    “url”: "https://mcpserver1/mcp
  }
}
}

Lets Get Ready to Rumble

Does it connect?

So first test does it connect to our “Context Plane”?

/mcp list

Yup it can see the MCP server and it can see four API end points that we have exposed.

Does it see Context objects?

Next lets test if it can see objects stored in the “Context Plane”

list tile names

Its worked out the correct API to hit to get back a list of Tiles from the Catalog.

And its given me a list of Tile names.

Does it understand Context for those objects?

We run a layered data architecture and the Data Layer is held as Context for the Tiles, not in the tile name etc. So that why im getting multiple Tiles back with the same name.

Lets try to get some Context.

list tile names and their data layer

All good, it worked out what Data Layer the Tiles belong in.

Interestingly we never use the term “Data Layer” in the “Context Plane”, im guessing its picked up the Data Layer from the table prefix or the object type we hold against each tile.

I had assumed we would need to add extensive aliasing of terms in the “Context Plane” to make it find answers to people natural language questions, but given Gemini CLI is backed by the Gemini LLM maybe we don’t.

(This is the value of doing a McSpikey, you discover things you didn’t know, and of course end up with more questions to answer).

Real Life Use Cases

I have been asking people at the coal face for use cases they constantly hit where they think accessing the “Context Plane” may reduce the complexity of the data work they do.

Im looking for use cases that are more than the usual “text to SQL” use case everybody now has as table stakes in their product.

Get Data Catalog Tiles

We have already proven we can connect and get Catalog Tiles using Gemini CLI.

But I am a GUI by default kinda guy and so for that use case I would just use the Catalog search screen in the AgileData App.

A couple of clicks and i’m done, no need to type lots of words.

Ask a question, get an answer, Context provided for me

The challenge with the App approach is I know that I need to search for Consume Tiles if I want to use the data on a Last Mile report. A new user would not.

So lets try that scenario by asking:

I need to use Netflix data in the report I have to create for the boss

Again its using the concept of our Data Layers to understand that data in the Consume layer should be used in a report.

Given we have provided no RAG or Prompt reinforcement on our Data Layer Architecture I wonder if its because we use clear terms like “Consume” rather than “Gold” that is helping the model.

Its also come back with a follow up question, lets answer it and see what happens.

details about cast and directors

Looks like its checking the Data Dictionary for the Tiles to see what fields are available.

As an aside there is now finally a reason for us to populate a Description for each Field in the “Context Plane”.

And its come back with the Fields I would need to use for my report.

Its also asking if I want it to help me build that report.

show me the data

It recognises that it has no MCP service to access data or run a query.

One of the features we built in the AgileData App ages ago was the ability to click on a Catalog Tile and open the data directly in Looker Studio, as a very quick way to explore the data via a GUI.

Gemini CLI has understood that from the “Context Plane” and look like it found the relevant Looker Studio url to open the data I need in Looker Studio.

Lets cut and paste that url ….

Yup thats exactly what it has done that was unexpected but magical.

Wonder if I could get it to rewrite that url to only include the Fields I am after, that is a question for another day.

Show me the impact of a change

The second real life use case I got given was to see if it would help streamline the process of understanding the impact of the changing a Field somewhere in the data pipeline flow.

i need to update the field type for the director for the netflix tile what do I need to worry about impact wise

It did a pretty good job of understanding the flow of the data and transformation logic for that Field.

Because we store the Information Apps that are accessed by Information Consumers in the “Context Plane” and we link tose to the Consume Tiles they consume it also picked up the Last Mile objects that may break and impact our Information Consumers.

One question I have is how it determined the lineage for this data from the four API services we gave it access to, as we did not give it access to the Data Map (lineage) API service.

Im guessing it stitched it together from the Change Rules.

Another question to be answered. I am going to have to figure out how to get Gemini CLI to show me what MCP services it using to understand the process more. Task for another day.

So in Summary

Colour me impressed!

Wood from the Trees

Still a way to go before I have a coherent set of Patterns that I can Coach / Mentor / Teach somebody else for the “Context Plane”, and the “AI Data Stack” or present as a robust Architecture map.

But as I have already said, writing my half formed ideas helps me think.

An incoherent stream of Context

You can find all the previous articles with my train of thought listed in this thread:

https://agiledata.substack.com/t/context-plane

We are building the Context Plane while flying it, so always looking for early adopters to help us decide the final destination.

If you want a virtual chat grab a slot here:

https://contextplane.ai/contact-us/#bookemdanno

“Context” of this post

I often find writing helps me coalesce and refine my thoughts when new patterns start to emerge, but aren’t very clear yet.

So this article is a brain dump / train of thought continuation of the architecture needed to have one Context Plane to rule them all, as part of a proposed “AI Data Stack”.

This article provides an overview of the things or metadata I think should be included in the Context Plane as part of a new “AI Data Stack”.

The “Context Plane”

My current thinking is the architecture diagram for the “Context Plane” should look something like this:

Things stored in the Context Plane

So with that context for how I currently think of the “Context Plane” lets get into the meat of this article, of what I think should be stored in that plane.

I have worked in the data domain for over three decades. The data patterns and the data terms I regularly use I have learnt over those decades. I find them very difficult to unlearn.

As somebody mentioned to me lately I am haunted by “the Ghost of Data Past”.

But also there is a recurring pattern where people seem to invent a new term for a pattern that has been around for decades (medallion data layered architecture anyone), so the patterns of the past do have value.

(and yes I realise the dichotomy of me talking about “Context” not “Semantics” in the same article where I point the finger at the “Medallion Architecture”)

So with all that said, let me use the data components of the past to frame what we need to hold and surface via a “Context Plane” to support what a “AI Data Stack” future might look like.

Business Glossary

The Business Glossary stores a list of agreed-upon definitions for organisational terms.

This holds a list of plain-language descriptions and aliases that standardise terminology across stakeholders, ensuring consistent interpretation and usage of key terms such as ‘Customer’, ‘Order’, or ‘Churn’.

Core Business Concepts

Core Business Concepts stores the primary things (‘entities’) an organisation manages or counts. This includes concepts such as ‘Customer’, ‘Product’, ‘Order’, and ‘Payment’.

This holds a list of the definitions and identifiers for the Core Business Concepts.

It contains a subset of the terms that exist in the Business Glossary and forms the foundation of the organisations data design.

This describes a list of “things”.

Core Business Processes

Core Business Processes stores the major operational workflows, events or life cycles (‘relationships’) within an organisation.

This holds a list of the interactions between Core Business Concepts, representing relationships such as “Customer Places Order” and “Customer Pays for Order”

This also holds the sequence of activities or states that Core Business Concepts transition through, such as invoice approval or support resolution (which I call “Administration Processes”).

This describes ‘relationships’ between “things”.

Conceptual Data Model

The Conceptual Data Model stores a high-level map or diagram that provides an abstraction of the organisation’s data design.

This holds information that outlines the Core Business Concepts, Core Business Processes the other relationships between Core Business Concepts in a platform-independent format and without specifying implementation details.

The Conceptual Data Model can also hold the objects that generate the Diagram, so overlaps with the Core Business Concepts and Core Business Processes.

This describes “things” and the ‘relationships’ between “things”

Logical Data Model

The Logical Data Model stores a more detailed map or diagram that provides Details (‘attributes’) that are part of the organisation’s data design.

This holds information that outlines the Core Business Concepts, Core Business Processes / Relationship’s in a platform-independent format.

It also holds the Detail (‘attributes’), that are related to a Core Business Concept, in a platform-independent format

This describes the Details of “things”.

Physical Data Model

The Physical Data Model stores a technical implementation view of how the organisation’s data is stored and accessed within a specific system.

This holds information on the the physical structures such as tables, columns, data types, indexes, partitions, and naming conventions used in the data platform.

It is derived from the Logical Data Model but adds system-specific optimisations and constraints.

It overlaps with the Logical Data Model by representing the same Core Business Concepts and their Details but also includes physical performance and deployment details that are specific to the technology used.

This describes how to implement the “things”, their ‘relationships’, and their Details.

Data Contracts

Data Contracts store the expected structure, quality, and behaviour of data exchanged between systems, or components. They describe formal agreements between a data producer and a data consumer.

This holds information such as schema definitions, required fields, data types, validation rules, and guarantees such as update frequency, timeliness, or completeness.

It overlaps with the Logical Data Model and Physical Data Model by referencing the same attributes and structures, but focuses on expectations and enforcement rather than storage or modelling.

This describes the Rules that govern the movement of the data.

Data Dictionary

The Data Dictionary lists and describes the metadata for data fields used across the organisation’s data assets.

This holds a list and information for data fields, such as field names, data types, descriptions, allowable values, formats and default values.

It overlaps with the Business Glossary. Logical and Physical Data Models by documenting the same ‘attributes’.

This describes how “things” and their Details were implemented.

Data Profiles

Data Profiles provide statistical summaries and characteristics of the data physically stored.

This holds data such as minimum and maximum values, distribution ranges, cardinality, null counts, uniqueness, and data type conformity.

They overlap with the Data Dictionary (by describing the same fields) and with Facts (by profiling the raw values captured).

This describes the observed characteristics of the Details of “things” and ‘relationships’ between “things”.

Transformation Logic

Transformation Logic stores how raw or source data is converted into a trusted, usable form.

This holds the business rules, mappings, calculations, filters, joins and aggregation logic or code applied to reshape data.

It may reference fields defined in the Logical or Physical Data Models and rely on terms from the Business Glossary and Core Business Concepts to apply meaning.

It overlaps with Measures, Metrics, Actions, and Information Products by forming part of the logic used to answer questions or trigger downstream outcomes.

This describes how the raw “things”, Details of “things” and the ‘relationships’ between “things” were reshaped into something more useful.

Facts

Facts are raw numerical values stored or sourced directly from systems.

This holds a list of data such as quantities, amounts, durations.

Facts are the foundational inputs used in downstream calculations. They are not aggregated or transformed, but instead represent the atomic, immutable data from systems where that data is first captured.

They overlap with the Business Glossary, Physical Data Model and Data Dictionary where they are described and with Measures and Metrics, which are derived from them.

This describes the raw numerical values about “things”.

Measures

Measures are standardised aggregations of raw Facts.

This holds the logic or code for calculations such as “total revenue”, “count of orders”, or “average order amount”, typically derived using functions like sum, count, min, max, or average based on a defined Fact.

They overlap with Facts (as their source), Transformation Logic (which defines how they are calculated), and Metrics (which may use them as components).

This describes aggregated values about the Details of “things”.

Metrics

Metrics are calculated formulas that combine Measures, Facts, or other Metrics to express business performance or operational efficiency.

This holds the logic or code for calculations such as “average revenue per customer”, “conversion rate”, or “churn percentage”, typically involving arithmetic, ratios, or conditional logic.

They overlap with Measures (which they reference), Transformation Logic (which defines their formulas), and Business Questions (which they help answer).

This describes derived insights from aggregations of the Details of “things”.

Business Questions

Business Questions are predefined or commonly asked queries that reflect specific decision-making needs within the organisation.

This holds a list of the questions that have been asked before such as “What is our monthly churn rate?”, “Which products are underperforming?”, or “How many new customers joined last quarter?”

They overlap with Metrics (which may be the answer), Transformation Logic (which defines how to calculate the answer), and Information Apps (where answers are delivered).

This describes the repeatable questions asked about the “things”, Details of “things” and ‘relationships’ between “things” to support decisions to take action.

Actions

Actions define the decisions, tasks, or system behaviours triggered as a result of answering a Business Question.

This holds a list of actions that have been taken in the past or need to be taken int he future.

It also holds information of the mappings between specific Business Questions and the Actions they enable, such as “Identify outstanding support tickets” and “resolve outstanding support tickets,” or “Identify Customers who havent placed an order in the last 6 months” and “Offer discount to Customers to reduce the rate of customer churn.”

They overlap with Business Questions (which initiate them), Information Apps (where they may be launched), and Core Business Processes (which they may automate or influence).

This describes the decisions and actions we take based on what we know about the “things”, Details of “things” and ‘relationships’ between “things”

Information Apps

Information Apps are curated outputs that present data and information in a usable form to answer Business Questions and support Actions.

This holds lists and information about dashboards, reports, visualisations, datasets, APIs, data servies, and user interfaces that package answers to Business Questions and trigger related Actions.

They overlap with Business Questions (which they answer), Actions (which they enable), and Facts, Metrics and Measures (which they visualise or expose).

This describes the delivery mechanism that provides access to what we know about the “things”, Details of “things” and ‘relationships’ between “things”

Fragmentation and Overlaps

I realised as I wrote this one of the major problems is deffo “the Ghost of Data Past” and specifically the fragmented set of data technology and tools that I have used over the years. These set of data technology and tools helps define the architecture and language I use, and also influences the patterns I applied to list the things that should be stored in the ”Context Plane”.

I have ended up using these technologies and tools to define the things that are needed in the “Context Plane” and also to define boxes or boundary around things.

The problem I hit is when I use these standard languages / boundaries from the data domain for those things then I get overlaps, there are things that are in within / across multiple boundaries.

For example Core Business Concepts are held/described within both a Business Glossary and a Conceptual Model.

An example using a Metric

Metrics are a part of a Business Question, but also defined in both a Metrics Layer / Tool, Business Glossary and potentially a Data Dictionary.

Lets look at an example of the overlaps in more detail using a specific Metric ‘Active Users’.

Metrics:

'Active users' is the number of people who engaged with our site or app in the specified date range.

An active user is any user who has an engaged session or when Analytics collects:

* the first_visit event or engagement_time_msec parameter from a website
* the first_open event or engagement_time_msec parameter from an Android app
* the first_open or user_engagement event from an iOS app

The user is considered an active user as soon as the user_engagement event is detected within a second.

(Source: https://support.google.com/analytics/answer/12253918?hl=en)

Business Questions:

"The Number of Active Users who accessed our Website last month."

"The Number of Active Users who accessed our App last month."

The Metrics Layer often holds the relationship the Metric has with the Core Business Processes, the Business Glossary often does not.

Yaml Metric Definition in a Metrics Layer tool:

views:
  - name: active_users
    description: "14 days rolling count of active users"
    includes:
      # Measure
      - users.rolling_count
      # Dimensions
      - users.is_paying
      - users.signup_date
      - company.name

(Source: https://cube.dev/blog/introducing-views)

Business Glossary:

Active user

Active users are the people who currently use your product. A user who becomes inactive may have churned.

(Source: https://posthog.com/docs/glossary)

As you can see there are overlaps in what we define for the Metric of “Active Users”, depending on which traditional data component we define it in.

Removing the Boundaries and Overlaps

When I get stuck with a problem like this I find the key is to keep breaking it down into smaller and smaller things until I get to a list of atomic things that are unique.

Here is where I have ended up with two different languages I can use so far:

(compiled with some help from my ChatGPT friend using the above as the input, and then the human iterated it)

The list of Context we store

Or put another way the things we store.

1. Business Terms

   Agreed-upon definitions and aliases used across the organisation (from the Business Glossary).

2. Core Business Concepts (entities)

   The primary entities the organisation manages or counts (e.g. Customer, Product, Order).

3. Core Business Processes

   The workflows, events, or life cycles that define how Core Business Concepts interact and change over time.

4. Conceptual Relationships

   High-level, platform-independent relationships between Core Business Concepts.

5. Details (attributes)

   Descriptive details of Core Business Concepts, such as name, status, date of birth (from the Logical Model).

6. Field Metadata

   Technical descriptions of fields, such as data types, formats, default values, and allowable values (from Logical Data Model and Data Dictionary).

7. Physical Structures

   Tables, columns, data types, and indexes used to implement data in a specific platform (from the Physical Model and Data Dictionary).

8. Data Rules and Expectations

   Contractual schema definitions and guarantees that govern how data is exchanged and validated (from Data Contracts).

9. Statistical Profiles

   Observed characteristics of data, including distributions, cardinality, and null rates (from Data Profiles).

10. Transformation Logic

    Business rules and operations that convert raw data into usable outputs (e.g. mappings, joins, calculations).

11. Raw Numerical Values

    Source data captured as facts, such as quantity, amount, or duration (from Facts).

12. Aggregated Values

    Summarised calculations based on raw numerical values (from Measures).

13. Derived Metrics

    Formulas combining multiple measures or facts to express performance or ratios (from Metrics).

14. Business Questions

    Repeatable questions that drive insight and inform decisions (from Business Questions).

15. Actions

    Operational steps or decisions taken based on answers to Business Questions.

16. Delivery Interfaces

    Mechanisms that present data and enable interactions, such as dashboards, reports, datasets, and APIs (from Information Apps).

The types of Context we store

Or put it another way, how it is stored.

1. Lists

   Repeating sets of labelled items.
   

   Examples:

   – List of business terms (Business Glossary)

   – List of fields (Data Dictionary)

   – List of metrics, measures, concepts, questions, or actions

   – List of attributes, rules, or processes
   

2. Definitions

   Text-based descriptions that explain meaning, purpose, or intent.
   

   Examples:

   – Term definitions

   – Field descriptions

   – Action explanations

   – Business question phrasing
   

3. Identifiers

   Unique keys, codes, or labels used to reference or join data.
   

   Examples:

   – Concept identifiers

   – Field names

   – Relationship keys
   

4. Relationships

   Structured mappings between two or more things.
   

   Examples:

   – Concept A interacts with Concept B
   – Concept A, Concept B and Concept C are involved in Process A

   – Detail belongs to Concept (attribute belongs to entity)

   – Measure uses Fact
   – Metric uses Measure

   – Question → Metric → Action

   – Term → Concept → Field
   

5. Rules / Logic

   Outputs produced from raw data using logic or formulas or expressions that define behaviour or constraints.
   

   Examples:

   – Validation rules

   – Data contracts

   – Calculation logic

   – Transformation pipelines

   – Measures (aggregated)

   – Metrics (calculated)
   

6. Structures / Schemas

   Models that define how things are organised or composed.
   

   Examples:

   – Logical/physical data models

   – API schemas

   – Table definitions
   

7. Raw Data / Values

   Captured data points or measurements.
   

   Examples:

   – Facts (quantities, amounts)

   – Timestamps

   – Event logs

   – Data Profile results

   – Data Quality results

8. Interfaces / Outputs

   Representations used for delivery or consumption.
   

   Examples:

   – Information Apps
   – Dashboards

   – Reports

   – APIs

   – Data services

Time to Cross Domains

As I mentioned at the start, my experience and expertise is founded in the data domain.

I think its time to look for help from the other domains, and for this one specifically the Library and Information Sciences domain.

Juan Sequeda posted an LinkedIn article on how he is thinking about metadata/ontologies/knowledge graph/semantic layers. In that article he uses this framing:

I can probably simplify “the List of Context we store” and “the Type of Context we store” above to the Business/Technical/Mapping metadata categories in that diagram.

But Im not sure simplifying it will get me any closer to achieving my goal, I think I need to go down into the weeds a little more to get the clarity I need.

One option to do this safely I think I need to identify a number of use cases where “AI” will use the “Context Plane” and see what I need to provide it to be successful.

Another option is to try and collaborate with an expert from the Library and Information Sciences domain to help add their views and language to see if it gets me to the next step.

The other option is to follow a suggestion Joe Reis made on the Practical Data Modeling Discord,

“As a thought experiment, ask any AI what it needs. It’s very different from what we’ve devised so far”.

Time to have a little think about what the next step in my train of thought for the “AI Data Stack” will be.

Wood from the Trees

Still a way to go before I have a coherent set of Patterns that I can Coach / Mentor / Teach somebody else for the “Context Plane”, and the “AI Data Stack” or present as a robust Architecture map.

But as I have already said, writing my half formed ideas helps me think.

An incoherent stream of Context

You can find all the previous articles with my train of thought listed in this thread:

https://agiledata.substack.com/t/context-plane

We are building the Context Plane while flying it, so always looking for early adopters to help us decide the final destination.

If you want a virtual chat grab a slot here:

https://contextplane.ai/contact-us/#bookemdanno

“Context” of this post

I often find writing helps me coalesce and refine my thoughts when new patterns start to emerge, but aren’t very clear yet.

So this article is a brain dump / train of thought continuation of the architecture needed to have one Context Plane to rule them all, as part of a proposed “AI Data Stack”.

This article provides an overview of how I think of the Context Plane in our AgileData Product as that anchors a lot of my thinking for what the Context Plane in a new “AI Data Stack” would look like.

First defining our current Context Plane pattern

When we started building the AgileData Platform and AgileData App, Nigel and I agreed on a set of core principles and patterns that we would align to as much as was practical.

One of these was the use of what we called Config (what I now refer to as Context) which should be at the center of everything we would build.

Context is a Pet not Cattle

The Context Plane is where we hold everything we care about that makes our AgileData Platform and AgileData App work, apart from our customers actual data.

We treat both the Context we hold and the Customers data we hold as Pets, and we treat everything else in our Platform as Cattle.

We generate, deploy, execute and destroy code at will. We automate the generation of all code from the Context we hold.

Context drives our App and our Platform

For example everything displayed on our Data Catalog screen is stored in our Context Plane, and then rendered in the AgileData App as needed.

The Concept, Conceptual, Logical and Physical data models are stored in the Context Plane and again rendered in the AgileData App, or used as part of our DataOps processes as and when required.

We define and hold Core Business Concepts, Detail for those Concepts and Core Business Events as the core “entities” in our Context Model.

Concepts, Details and Events are the semantic language we use as part of our Context Language.

We hold the relationships of how Core Business Concepts relate to Core Business Events in the Context Plane.

Which allows us to render and explore a “Graph” view of these relationships in our App.

For those old school of us, we also render these relationships as Bus Matrix in our App. Out Context Plane allows us to render the same Context in any visualisation or Map format that helps simplify complex data tasks.

Our version of Data Quality (Trust Rules) and Observability (Notifications) are again stored in the Context Plane and rendered / used as needed.

Context drives the generation of our code

Our version of Data Transformation logic (Change Rules) is you guessed it, stored in the Context Plane and then used as and when required.

If you perused that last series of screenshots you will see we actually hold surface the Change Rule logic in multiple languages, we surface it via the UI notebook style, we surface it as form of a Gherkin script and we surface the SQL that it is generating to load or transform data in Google BigQuery.

These are all dynamically generated from the Context stored in the Context Plane for each Change Rule we hold. We have defined it once and we can render it in many different ways, many different languages and for many different use cases.

Some people call this a table driven pattern, a metadata driven pattern or a model driven pattern etc.

Orchestrate via dynamic manifests based on Context

To orchestrate the execution of the Data Transformations (Change Rules) we of course build out a Directed Graph to run them in the order they need to be run, and to honour the dependencies they need to honour.

But this Directed Graph is dynamically generated.

If I go and add another Change Rule, it is stored in Context Plane and the next time I come into this screen it will be dynamically rendered including that rule.

When a new record turns up in a table on the left, it will trigger a bunch of steps.

• The Context Plane will be queried to see all the Change Rules that are dependent on that data.

• The Context Plane will be queried to find the dependencies between the Change Rules that will run, and tables they will load.

• A “Manifest” is created to describe what Change Rules need to run.

• The SQL code for each relevant Change Rule will be generated from the Context stored in the Context Plane about that Change Rule

• The code will be submitted to run using a Pub/Sub model, in parallel.

• When all the code has been successfully run, the code and the manifest will be destroyed. (We do hold a copy of the code that was executed in our Audit Vault, so we can always view what was run when).

• The latest Observability and Trust data will be stored in the Context Plane.

As we hold the Context of how every physical table (Tile in our language) relates to every other table (Tile), via these Data Models / Change Rules, we can easily show all the dependencies (Related Tiles) for a specific table (Tile), when viewing the Context of that table (Tile).

This includes the relationships for both the Data Model. i.e for this Concept of “customer” we hold these Details, and that Concept of “customer” is part of these four Events, as well as the relationships for all the Change and Trust and Consume rules that the “customer” Concept is part of, i.e this Detail tile is loaded by this Change Rule and validated by these Trust Rules.

Multiple languages to create Context

You can create new Context in the Context Plane via the AgileData App UI, or you can use the API’s to create the Context.

It would be relatively simple to allow the creation of the Context in yet another language, for example by uploading YAML files.

(Fun fact the first interface we had to create Context was Google Sheets!)

One set of Context to rule them all

So you can see how the Context we hold on in our Context Plane is the thing that drives everything our AgileData Platform and AgileData App does.

Context is first and powers everything that happens

I know what you thinking, this sounds a little bit like a data catalog, or a metadata repository.

But heres the thing, data catalogs run by sucking on the Data and Metadata exhaust.

We treat Context as the fuel that powers everything we do.

You cannot run anything if Context does not exist for that thing.

I cannot create a data catalog object, I cannot create a table, I cannot create code or run it, without it existing in the Context Plane.

I first create the Context and then the AgileData Platform then does all the other horrible and complex data work for me.

Its about more than just data and code

As you would expect of any mature data platform, we have added “Data Management” and “Data Governance” capabilities over the years where we found them useful.

One of those is the ability to store the Context needed to render a “Data Dictionary”.

But we are still missing a lot of useful Context

We only build things in our AgileData Platform and AgileData App, when they save us time or automate something we hate doing.

The Data Dictionary feature was built so we could find fields we needed to find by searching all the Context of all the data we hold for a customer.

There is some other Context we know will give us value that we have yet to add.

Things like Context typically related with Business Glossaries, Metric Layers/Engines and Semantic BI Layers to name three.

I believe they are all part of the Context that is needed in a Context Plane to make it useful for AI Agents.

Which will be the next article I write to help me with my thinking.

Wood from the Trees

Still a way to go before I have a coherent set of Patterns that I can Coach / Mentor / Teach somebody else for the “Context Plane”, and the “AI Data Stack” or present as a robust Architecture map.

But as I have already said, writing my half formed ideas helps me think.

An incoherent stream of Context

You can find all the previous articles with my train of thought listed in this thread:

https://agiledata.substack.com/t/context-plane

We are building the Context Plane while flying it, so always looking for early adopters to help us decide the final destination.

If you want a virtual chat grab a slot here:

https://contextplane.ai/contact-us/#bookemdanno

So the hypothesis was:

To test out the hypothesis I uploaded one of the canvas that I collaborated on to is being used as an example in the upcoming “an Agile Data Guide to Information Product Canvas” book.

As this canvas was based on a real world example of an Information Product somebody had already created for one of their customers, and I only facilitated the population of the cavas, this would reduce the bias in the experiment compared to me creating one from scratch just for this purpose.

Here is an interactive demo / guide showing the results of the experiment:

https://guides.agiledata.io/demo/cm7c9gj7h0jr49ujfhaymm0um

It is a quick and dirty interactive demo, where I test the process and then make notes on what I think worked and what we need to iterate.

An aside on the value of interactive demos

I have started using this interactive demo format a lot, much better than making disconnected notes, and also much better to see what each note relates to.

I can quickly document something and then share it with Nigel Vining and he can view it whenever he has time, great for our remote only, work from anywhere in the world way of working.

I have also been using this interactive demo pattern to log support calls with other vendors when I have a problem with their product.

As they say seeing is believing.

Back to the result.

Overall the Disco outputs were not good enough to be used as a data design (not sure it ever will be perfect), so some more iteration to be done.

But the output it did generate was an encouraging start.

Observe everything that is going on in the AgileData Platform, as it happens.

Stay connected and observe every important change within the AgileData Platform with real-time notifications. Designed to keep you informed, this feature ensures you never miss critical updates or changes to your data.

Whether it’s a completed data refresh, a new data Tile becoming available, or an alert about an issue with Trust Rules, notifications provide instant insights into every key data flow in the platform.

Integrated seamlessly into the AgileData App, notifications empower you to act quickly, maintain data trust, and stay constantly informed about your data work.

Feature Requirement

The system must provide real-time notifications to data users, keeping them informed about significant changes, updates, or alerts within the AgileData Platform that directly impact their data tasks.

Requirement Rationale

Notifications ensure data users stay connected and informed about critical events, such as data refreshes, data quality alerts, or the availability of new data assets. This empowers them to act promptly, maintain trust in their data, and complete their data tasks efficiently without missing key updates or changes.

How

View Interactive Demo

Why

This was another scale the machine not the Nigel, driven feature.

In the early days of the AgileData App, when something didn’t seem to work properly I had to go into the Google Cloud Console and search the logs to see what had happened.

As you can imagine I didn’t do this, I just slacked Nigel and asked him to have a look

As you can also imagine Nigel wasn’t a fan of this approach so he built the Notifications feature.

The key to this feature is it shows everything that happens in the AgileData Platform.

Again as you can imagine this causes a signal vs noise problem, which has been a major focus of iterating this feature over the years.

Real-time notifications are visible at a glance at the top menu of the app. This means when I do a data task, for example uploading a manual csv file, or manually running a data flow, I can quickly observe it happening at a glance.

This removes the need for me to go to the Notifications screen to watch things run, I can stay in the screen that does the data task I need to do, while still observing the process.

To provide this real-time view of the Notifications was a major piece of platform engineering, as it required the use of a Websockets component to provide the pub/sub pattern.

Real-time notifications are great, but you have to have the AgileData App open in a browser to see them.

Often I want to check all the Notifications for a Customers Tenancy that I am not currently logged into.

So we built the Notifications screen.

This screen shows all the Notifications for the Tenancy.

The initial versions of this screen hit the signal vs noise problem, in multiple ways.

First was the fact that there are lots of notifications and loading them for all time was slow. So we defaulted it to only show the last 24 hours, and then provided an easy way to extend the time window to show older notifications.

It’s interesting how rarely I need to extend the time window, most times a snapshot of notifications over the last 24 hours is enough for me to get the data task done.

As I mentioned earlier the Notifications screen shows everything that is happening on the AgileData Platform.

We display when Tiles are created, new data is loaded, Change Rules are iterated, data fails Trust Rule validation.

Nigel is constantly adding more events to the Notifications as we find more things we want to be able to easily observe and investigate.

We also show every successful event and every event that is treated as a failure.

This amplifies the signal vs noise problem,

First thing we did was to apply a DataOps scorecard pattern at the top of the screen that allows you to see the number of Notifications within a specific category.

The key categories we found valuable.

• Success vs Failure

• System identified Anomalies

• Trust Rule identified anomalies

• Data Freshness anomalies

This scorecard is interactive, click on Anomalies indicator and the Notification list will filter to only those Notifications.

Saves a few clicks and as I have mentioned in previous articles, a few clicks and seconds saved in a data task you do repeatably all add up to minutes and hours saved overtime.

We added search to enable you to filter the Notifications based on any freeform text.

The next thing we found useful was adding Context to a Notification type.

For example in the Tile load notification we show a small bar graph showing the number of records loaded and a red line which shows the average load volume for that Tile. This allows me to quickly confirm at a glance that the load volume is within the expected threshold, without needing to click the Notification to see more details.

These glance widgets are context sensitive to the Notification type.

At any time I can click on the Menu Anywhere (3 dots) for a Notification and drill down to more detail.

For example drill down to the Trust Rule that has generated an anomaly, or the Change Rule that was used to load data into the Tile.

I can also see the detail behind any Notification.

Again we provide context sensitive details based on the Notification Type.

For Tile Load Notifications we will show the load stats for the historical loads for that Tile.

For an error Notification we used to show the BigQuery error.

As you can imagine, as soon as I saw one of those I would just message Nigel to have a look at it and tell me what I needed to change.

And again as you can imagine Nigel automated that task, so he doesn’t have to do it for me.

We now pass the error to the Google Gemini LLM and get it to return both a description of what the error is likely to mean, and a suggestion on how to resolve it.

Unsurprisingly this has resulted in me resolving a lot of the issues myself.

We also broadcast some of the Notifications to Private slack channels for a Tenancy. This allows us to view Notifications from multiple AgleData tenancies in one place, without the need to login to each Tenancy and check the Notifications screen.

I have a backlogged feature to create a new cross tenancy capability where we can see Notifications for multiple tenancies in an AgileData App screen, a cross tenancy portal so to speak.

As we have AgileData Network Partners starting to support multiple customers in multiple tenancies, this feature will become important in reducing the time they spend monitors and managing their customers data, and also remove the need for us to use Slack for this purpose.

Search your data as effortlessly as using Google Search, finding the exact data you need in seconds.

Simply type in any data value you want to find in a Tile (table) and the AgileData App will automagically search every row and every column in that Tile and it returns the rows that have that value.

Data Search enables Data Persona to look beyond metadata and dive into the actual content of datasets, finding the specific data values or patterns directly within the cataloged data.

This makes finding and using relevant data faster, removing the need to write query code or the need to use a third party query tools to find the data they need to complete their current data task.

Feature Requirement

The system must allow users to perform data search within the catalog, enabling them to search not only metadata but also the actual content of datasets for specific values or patterns.

Requirement Rationale

Catalog data search enhances the ability to find relevant information quickly, supports deeper data exploration, and improves the efficiency of data analysis by allowing users to locate specific data points directly within the cataloged datasets, and minimising the need to write code or use third party query tools.

How

View Interactive Demo

Why

This feature was built out of Nigel’s frustration with me asking him to write and run queries for me all the time.

Like Track N Trace, it is another one of those small micro features that we built to remove a frustration that ended up being one of the surprise features I use a lot.

As I often joke “I can't code, won’t code, don’t code”.

This statement is not quite true, while I can’t sit at a blank screen and write code from scratch, I can read and edit other peoples code, Im ok at googling and reusing code snippets and with the advent of LLM co-pilots I am even more proficient with cobbling code examples together to get a data job done.

But one of our core principles is if I have to write code in our AgileData App, or heaven forbid I have to write and execute code in BigQuery Studio in the Google Cloud Console, then we have some work to do to make the no/low code experience magical.

To be clear, a magical no code solution is nirvana for us.

The analogy I often use is the “Magic Sorting Hat”, stealing a pattern from Harry Potter. If we could build a magic sorting hat that would allow us to dump any of our customers data, of any complexity, into that hat, and it then did all the data work we have to do manually, automagically, then we are Done Done. I don’t believe that I will see that in my working lifetime, but then again, I wouldn’t have believed a tool like ChatGPT would have existed in my lifetime either.

As we are bootstrapping, so don’t have billions of VC money to build the magic sorting hat, and are not young enough anymore to couch surf and subsist on 2 minutes noodles, we are taking a softly softly catchy monkey approach to this, implementing AskAI, AssistedAI and AutomatedAI patterns where they add value.

We are iteratively moving towards the magic sorting hat, one small step at a time, knowing we are unlikely to get to the final end we desire, but still getting massive value and reducing the complexity of managing data on every step of the journey.

An interesting point from the three AI patterns lens is that we decided to use a Search pattern to solve this problem, not an AskAI chat pattern.

Back to Data Search.

I use this feature in a few different ways, often in combination with Track N Trace.

One use case is when I want to see the Details that relate to a specific Concept.

For example I have a Concept of Customer and I have some Details about the Customer, attributes such as name, date of birth, phone number etc. For a specific customer record there is a problem with the data, maybe the phone number isn’t what is expected.

So I will go into the Catalog Detail screen for the relevant Tile and search for that Customer, maybe based on their customer id, maybe on their customer name. I will get back a row (or rows) for any row/column that has that value, and I can then peruse all the values on that row.

Or maybe I want to search for a specific phone number to find which Customer(s) it is stored against.

Nothing that can’t be done using a SQL query tool, but remember I won’t code and I never want to have to leave the AgileData App to get the data work done.

And Nigel doesn’t want to write and run queries for me, hence this feature appearing.

Before this feature it was either Ask Nigel, use the BigQuery Studio Console or push the data to a Consume Tile and use Looker Studio on it.

The Data Search feature reduces the time, my effort and my cognition to find the data I need, which is what we are all about.

After using this feature for a while, there are a few things we need to iterate.

It would be good to be able to have more control on the columns that are searched and to be able to search for two different values at the same time. Of course those iterations will start us down the journey of building a query tool in the app, not sure that is something we want to do.

And I would love to be able to search every Tile for a data value, a bit like Track N Trace.

One of the secrets of Data Search is we are leveraging a BigQuery SEARCH() function for this, we are not writing and running a custom SQL query.

For now I will keep saving time when I need to find a data value by using it, and sleep well knowing I haven’t had to Ask Nigel to “write me another quick query”.

Track n Trace provides end-to-end visibility of your data journey, allowing you to trace every step from source to consumption with ease, as if you were tracking the delivery of a physical package.

This enables Data Personas to understand where data originates, how it transforms, and where it is used, ensuring full transparency and a reliable audit trail. Whether for troubleshooting, compliance, or optimisation, Track n Trace delivers the clarity needed to manage your data flows effectively.

With its simply magical interface, Track n Trace empowers data teams to maintain trust in the data and streamline their data operations with ease.

Feature Requirement

The system must provide capabilities to track and trace data lineage for a specific data value, enabling users to follow the flow of data through various transformations, sources, and destinations within the platform.

Requirement Rationale

Tracking and tracing data ensures transparency and accountability in data processing, supports compliance with regulatory requirements, and enhances  in data by allowing users to understand the origin, transformation, and usage of their data.

How

View Interactive Demo

Why

This feature was built out of my frustration with the typical pattern for tracking and understanding lineage of data.

One of the benefits of working in a dedicated data team for an organisation, or only working for one customer, is over time you get tacit knowledge of all the data flows.  If you want to understand how the data is or isn’t flowing through the data platform, you already have an idea and you typically just want to validate your understanding.

If you are a Fractional Data Team, you are often switching Customer context on a regular basis and that increases the level of data cognition you need.  You have to tune into that Customers data flows so to speak, or “see the matrix”.

Often part of that cognition is the need to trace the lineage of a particular data flow for a Customer.

It might be for the dreaded statement from a Stakeholder, “that number doesn’t look right” which requires you to trace it from the Consume Tile the Last Mile tool is using, back to the Change Rules that are applying business logic to the data and potentially all the way back to the History Tile and System of Capture that generated that data.

It might be to trace the impact of a Systems of Capture change that you have been told is happening in the future, or more likely has happened and broken the current Data Contract. So you need to trace forward to see all the transformation logic that relies on that data and the Consume Tiles that use it.

Or you need to investigate the effort to “just make a small tweak to the business logic on how we calculate xxx”  and you need to start in the middle of your data flow and move both left and right at the same time.

One of the first features we built in the AgileData App, was the Data Map, our version of the typical lineage graph.  You can of course click on a Tile in the map and see the lineage both forward and back to visualise the flow of data.  From there you can simply view the Change Rules that transform the data or view Tiles that hold the data as a result of each transformation step.

But if you are trying to trace an individual column this process is far from magical. 

We could have implemented a column level view of the Data Map, but that would have resulted in a complexity of lines and flows I like to call “mad persons knitting”.  And I would still have to click on each node or line to understand what was actually going on within that Data Flow.

One of the things we often do to understand the flow of the data is to try and trace a single data value.  For example, pick the Id of an order in the System of Capture, or the name of a Supplier and trace that value through the Data Flows.

And that led us to the development of Track N Trace.

You enter a data value for a specific Tile and it will return with every Tile in the Data Map that is related to that Tile and has that specific value in it.  And it will tell you the number of times that value appears in each dependent Tile.

It seems like a simple process, but as always creating simplicity from complexity is a bit like a duck floating on top of the water while peddling madly under the water.  

Here is what the AgileData Platform is doing under the covers to achieve this magic:

• You start with a Tile in the AgileData App and you search for a specific data value.  

• It automagically traverses the Data Map in the background to harvest each transformation rule that is dependent on the Tile you started with.

• It identifies each target Tile that each of the identified rules populate.

• It then searches all the data in each of those Tiles for that data value.

• It returns the name of the Tile and the number of times the data value appears as a simple list

• You can click on any of the Tile names to drill through to the Change Rule that populates that Tile.  

With our Track N Trace feature we adopted the same pattern you use when you are tracking the delivery of a physical parcel.  It’s just a little faster and a lot more reliable.

Again this feature saves me minutes every time I need to understand the lineage of the data.  And when you do that data task multiple times a day, those minutes saved all eventually add up to hours.

Data Persona’s can use the Catalog to quickly search, find and access the Data Assets (Tiles) they need to complete a data task, such as supporting the development of Information Products or using the data for analysis.

Topic Tags and Dynamic Filtering allows Data Assets to be easily found using the Organisations business terminology.

With the Menu Anywhere capability they can drill out to any other screen in the AgileData App, retaining the context of the Tile, removing the need to go to the screen and manually searching for that Data Asset again.

Any time a data user performs a data task in the AgileData App, or an automated data task is executed on the AgileData Platform, the Catalog is automagically updated.  The Catalog data is treated as a first class citizen, not as data exhaust.

The Catalog is part of a wider ensemble of features that let them search or browse to locate essential Data Assets quickly, view comprehensive details for each set of data, and track the flow of data similarly to parcel tracking.

Feature Requirement

The system must include a comprehensive data catalog that allows users to organise, search, and manage metadata for all data assets within the platform.

Requirement Rationale

A data catalog enhances data discoverability, improves governance by providing clear documentation of data assets, and supports efficient data management by enabling users to easily locate and understand the available datasets.

How

View Interactive Demo

Why

This is the one feature I use the most, I use it everyday when I am doing data work, because it is typically where I start from to do that data work.

It’s also one of the first features we built when we started developing the AgileData App.

We designed the Catalog based on the patterns for browsing and searching within Netflix.   We treat a table or view of data as if it was a movie, we present each as a Tile.

We had three choices when we designed the Catalog feature:

1. Use the typical data Catalog pattern that is adopted by most data catalog tools, a row per Data Asset and lots of detail about that asset in each row, what I think of as a library pattern.  For me this was always orientated at hard core data people.

2. Use a Google search pattern, a blank screen with search, and then present a row per data Asset with a subset of detail as a result of that search.

3. Netflix browse or search pattern, which is what we went with.

One of the problems with the search only patterns for Data Catalogs is it assumes you know what you are looking for.  

We had a hypothesis that users would want to browse the Data Assets that were available.  

I am a great fan of looking at consumer products and patterns and seeing if applying them to the data domain reduces complexity (or increases it).  Netflix seemed to have the best pattern for browsing large volumes of movies to find what might interest you, so we implemented this as the primary UX in the Catalog to browse across large volumes of data sets.

This browse hypothesis was proven wrong, not because Information Consumers don’t want to browse, but because they just want to access the answer. They want to find and access the Information Product that quickly answers their Business Question and lets them get on with the Action they need to take once armed with that answer.  They don’t want to browse for sets of data that might or might not answer their Business Question and then have to do a whole bunch of data work themselves to get that answer.

We added the Marketplace feature to solve this problem, the Marketplace feature reuses all the browse and search capabilities of the Catalog, another great example of our Define Once, Reuse Often (DORO) principle.

After using the Catalog for quite a while, we found immense value in the browse pattern ourselves.  One of the interesting things about being a Fractional Data Team is you switch Customer context on a regular basis.  The ability to be able to land into the Catalog for a Customers Tenancy and quickly browse the data Tiles that are available, helps with reorienting oneself quickly.

If I know what I am looking for I often find browsing is still the quickest way to get to that data.

But not always.

So thats why we added search.

We started off with the typical search capability, text box at the top of the Catalog screen, type some text, hit enter and wait till the search results, in our case the Tiles, were returned.

It worked but it didn’t seem very magical. 

One of the useful micro interaction patterns common in web based applications these days is the idea of type-ahead.  You start typing text and the machine starts to makes suggestions for completing those words for you.

We applied an iteration of this pattern to provide Dynamic Filtering in the AgileData App.  As you start typing in the search box we are automagically filtering the Tiles in the Catalog screen based on your search criteria.   Small interaction pattern, but one that is magical.

One of the problems we have experienced with previous generation of Data Catalogs is they treat data and metadata as exhaust.

By this I mean other tools create the data and the metadata and the Data Catalog then “harvests” this on a scheduled basis.

We use a completely different pattern at AgileData.  

When you do the data work in the AgileData App, or it’s automagically done for you by the machine in the AgileData Platform, the data work is immediately available in the Catalog.

This is because we flip the standard pattern for a Data Platform.

Instead of the pattern being:

• create Code > create Metadata > populate Catalog

Our pattern is:

• Create Metadata > store in Catalog > create/execute Code

We call this our Config Driven pattern, others call it Active Metadata.

When you use the AgileData App, or the machine does something automatically in the AgileData Platform, every step in the data tasks is stored in Config.  The Catalog screens are then just surfacing this Config.

With this pattern there is no way the Catalog can get out of sync with the Data and/or Metadata.

We then use that Config time and time again to generate and execute the code we need, before deleting the code. We treat the Code as Cattle and the Config as Pets in DevOps parlance.

From day one we also allowed the creation and management of Topic Tags in the Catalog.

These Tags are freeform, you can create a Tag with any name for any use.  They are purposely not stored in Config as a set of restricted or regulated “lookups”.  The reason is you often need to quickly iterate the use of those Topic Tags.

When you go into the Catalog you will see the data Tiles in the relevant row for their Data Design role, ie History, Concept, Detail, Event, Activity, Consume.

You will also see a row for every Topic Tag that has been applied.

Want to see a row with data Tiles that relate to “Sales” add a Topic Tags called “Sales” to one or more Tiles and that row will automagically appear in the Catalog. 

The Tags are also “active”, click on a Tag that is shown on the Tile and the Catalog screen will Dynamically Filter to only show Tiles with that Tag.

This way of navigating to quickly find Tiles has saved many hours over the years.

The other part of the Catalog feature I use every day is the Menu Anywhere.

On each Tile in the Catalog there is the Menu Anywhere option, 3 dots (…).  Click on this and it will provide the list of things you can do that relates to this Tile, for example show the Change Rule that populates it, or the Trust Rules that validate it.

This means you can punch out to the Change Rule Step screen in one click, without needing to go to the Rules List screen, search for that Tile and then click to open the rule.

Interaction patterns like Menu Anywhere might save a few clicks and just a minute or two, but those clicks and minutes soon add up when you do them day in and day out.

We also present the Catalog screen from different parts of the top menu that are again context aware.  If you click on the History Tiles menu under Collect, you will be taken to the Catalog screen already Dynamically Filtered by Data Design type of History.

Again it’s only a few clicks to open the Catalog screen, type History in the search box, and then the Catalog is dynamically filtered.  But saving those clicks, by just clicking on a single menu option, just makes sense.  Especially when you do that interaction multiple times everyday.

The Catalog feature started its life as a feature built for Information Consumers, because that was what everybody else did, they brought and implemented a Data Catalog for their end users (and then wondered why they never came).

Over years of iteration we have made it a tool for ourselves to help make the data work we do less complex.

Remember we are We are building a product that is focused on:

1. taking a data task that takes me an hour and making it 10 minutes.

2. taking a data task that I have to think about how I can do it, and getting the machine to recommend what I should do, reducing the need for cognition.

3. automating the data task so we don't have to do it, getting the machine to do the work for us.

We have added a small bit of personalisation into the AgileData App as well.

The Catalog screen is where every Data Persona starts when they login to an AgileData Tenancy, 

A Consumer Personas automagically starts with the Marketplace screen.

With Single Sign-On (SSO), users gain streamlined access to the AgileData App, eliminating the need for repeatedly logging in and reducing friction in their data day. 

This feature leverages Google Identity and Access Management (IAM) to provide seamless and secure authentication, ensuring that users can easily connect to the platform using their existing credentials. 

AgileData's SSO is not only secure but also scalable, designed to meet the needs of growing teams while maintaining the highest standards of data protection and access control.

Feature Requirement

The system must support Single Sign-On (SSO) integration, allowing users to authenticate through a centralised identity provider to access the platform without needing separate credentials.

Requirement Rationale

SSO enhances security by centralising authentication management, improves user convenience by reducing the need for multiple logins, and streamlines access control across different systems and applications.

How

View Interactive Demo

Why

When we started building the AgileData Platform, we used Google Identity and Access Management (IAM) by default from day one, it just made sense.

We also use Google IAM to provide both Authentication capability and Single Sign On (SSO) capability for the AgileData App.

A quick definition of Authentication vs SSO:

• Authentication verifies a user’s identity to grant access to a specific system. 

• SSO enables a user to log in once and access multiple systems without re-entering credentials.

There were a few reasons why using Google IAM made sense to us.

We used GSuite for our internal systems, email, Google Docs, Google Sheets etc.  As a result Nigel and I already had Google Accounts.

When you start using the Google Cloud platform you typically start by accessing it via a Google Account to use the cloud services, which is what we did.

Our initial interface for Change Rules leveraged Google Sheets, so again we could easily access this via our Google Accounts.

When we started to build out the AgileData App on top of the AgileData Platform, we needed to decide how we would authenticate users to the App and how we would support SSO.

We had a few choices:

• Keep using Google IAM

• Introduce the capability for us to store usernames and passwords in the AgileData Platform

• Use a third party OAuth provider, for example Okta or Microsoft Active Directory

For us creating and storing usernames and passwords directly in the AgileData Platform was not a pattern we wanted to adopt, we didn't want to store Passwords in the AgileData Platform.

It introduces a massive amount of risk when you directly hold those passwords, a risk we did not want to incur.

That left as with the option of using Google IAM or a third party provider such as Okta or Microsoft Active Directory (or whatever Active Directory is called by Microsoft these days)

Both of those products are aimed at Enterprise Customers and at that time we had not identified what our ideal customer type was.  Plus there was an additional and quite substantial cost to both licence those products and to integrate them within our product.

One of our core Principles we operate at AgileData is to leverage serverless capabilities available within the Google Cloud Platform wherever possible.  So naturally that led us to keep using Google IAM as our primary authentication method.

We also started using Google Groups as a foundational service for managing Authorisation and role level security, i.e controlling who could access what.  That felt like a pretty weird pattern, but it is what the Google Cloud Platform uses itself to control role based access, so was the logical pattern for us to adopt.

One of the downsides of using Google IAM is each AgileData User has to have a Google Account to be able to login.  Luckily Google provides an easy way to link a Google Account to an external email address, which meant our customers could log into the AgileData App using their standard corporate email address and didn’t need a gmail address.

A downside of this linked Google Account pattern has been the fact they can’t use the password they use for their corporate email (well they can, but it is not automagically sync’d) to login. We always thought we would eventually integrate Google IAM with our customers' Microsoft Active Directory environments when needed, a pattern that is well documented.  But we have yet to have a customer where that integration was required.

We did think about opening up the SSO capability to allow users to login into the AgileData App via their social media account, i.e Meta (well Facebook back then) X, (well Twitter back then) etc.  But we felt that most customers would not be comfortable with their users accessing their organisational data via these personal social media accounts.

One of the other benefits of using Google IAM as both our SSO and Authentication service is we automatically inherit the scale and might of Google to identify and mitigate threats.  Something we couldn’t afford to do well ourselves.

Information Consumers can quickly search, find and access the Information Products they need to answer their business questions.

This feature enables Information Consumers to search and find all the available  dashboards, reports and analytical models in the AgileData App, regardless of what third party Last Mile tool they are created in.

It also enables the Information Consumer to quickly open that report or dashboard directly from the Marketplace, when those reports are accessible by a web URL

Feature Requirement

The system must include an Information Product Marketplace, where users can browse, discover, and access a curated selection of Information Products.

Requirement Rationale

An Information Product Marketplace facilitates the sharing of information assets, enhances collaboration across teams, and provides users with easy access to valuable insights and tools, driving data-driven decision-making and innovation.

How

View Interactive demo

Why

One of the core principles we decided on when we started building AgileData, was we wouldn‘t focus on building the last mile reporting visualisation tools.  The Visualisation market was very crowded and the cost of developing the table stake features were too high, there were more valuable areas of complexity we could focus our development effort on making simpler.

We also had a bet that most organisations would already have a Visualisation tool of choice and wouldn’t want to use the basic version we would create.  This bet turned out to be wrong as we started to focus on helping organisations who didn’t already have a data team, and typically they also didn't have a visualisation tool.

In this scenario we started using Google Looker Studio to deliver these visualisations as it was a good tool for the basic stuff, it had no licence cost, it ran as a serverless / SaaS product and lastly it was tightly integrated with Google BigQuery which we use to store the data in AgileData.

The Marketplace feature came as a result of watching how our Customers used, or in this case did not use the AgileData App.

Over the years we have spent a bit of time extending out the Catalog capability in the Agiledata App.  We use the Catalog capability extensively ourselves, as the launching point for most of the Data Work we need to do.

But when an Information Consumer from a  Customer went to use the AgileData App for the first time, they got confused.  They could see a bunch of Tiles that related to the Data Work we were doing for them.  

There were the History Tiles in the Catalog where we stored an immutable and histories copy of the data we had collected from their Systems of Capture.

There were Concept, Detail and Events Tiles in the Catalog which represented tha data in a way that matched was Designed to match their Business Language and make it fit for purpose

There were Consume Tiles in the Catalog, these were the sets of data that could be used to create the visualisations, reports etc.

But the final thing that they actually consumed, the Information Products, were not there and the Information Consumers found that confusing.

We also had a problem were there was no real portal capability in Looker Studio, so it was difficult for Information Consumers to find and use the Looker Studio reports they needed.  We found ourselves constantly emailing URL links to the reports to them.

To solve this our first bet was to enable third party URL’s to be created In the Catalog, as Info App Tiles.  This allowed you to paste a URL to a report and give it a name and a Tile would appear for it in the Catalog.

This resolved some of the issues for the Information Consumer, but they still found the Catalog confusing as Info App Tiles were just one of the many Tiles they could see.

We had a choice where we could have used personalisation to default a filter for the Information Consumer user types when they viewed the Catalog, and only showed them Info App Tiles by default, but instead we decided to break it out to its own top menu, which we named Marketplace.

This was based on a bet that we will extend out the Information Product sharing capabilities in the AgileData App in the future.

When a user who is an Information Consumer logs into the AgileData App, they are now taken directly to the Marketplace screen and are presented with the Information Products they can consume.

From there they can search, find and access any Information Product they need to answer their business question with data and take the action they need based on that data.

All the other user persona’s still get shown the Catalog when they login to the AgileData App.

We are literally building the product we have always wanted and needed to do data work, faster, easier and cheaper.

We are building a product that is focused on:

1. taking a data task that takes me an hour and making it 10 minutes.

2. taking a data task that I have to think about how I can do it, and getting the machine to recommend what I should do, reducing the need for cognition.

3. automating the data task so we don't have to do it, getting the machine to do the work for us.

Nigel and I have a specific set of complimentary skills which means we are pretty good at building the product we need.

The thing that neither of us are good at is Product Marketing. We have fallen into the old start-up trap of building it, rather than selling it.

Thats not such a bad thing, our current business model is to provide our customers with a Fractional Data Service. For $5k a month, they are buying a data team, that comes with a data platform and continuous data outcomes, not just our AgileData App & Platform.

Our product is designed to enable us to scale the Fractional Data Service. The goal is a single data person should be able to sustainably support at least 10 different organisation's at the same time.

As part of trying to achieve this scale we have built a pretty amazing product, when I show people they often comment at the breadth and depth of what we have built.

So I am going to try an experiment.

I am going to create 42 posts, one that outlines each of my favourite AgileData App feature we have built and why we built it.

At the end I will post a Google Sheet with the description of those 42 features (and a few more) as a recommended list of requirements for a data platform.

Each post will have the same structure:

• The What, short overview of the the feature;

• Feature Requirement, if you were asking a vendor for that feature what would you ask for;

• Requirement Rationale, if you had to justify to the powers to be why you needed this feature, you would use this;

• The How, a short interactive click through video of how it works in AgileData (seeing is believing);

• The Why, the story behind why we built that feature and what we learnt.

We have built these features because we needed them, so if you are looking to build or buy your own data platform, those are the features I would recommend you look for.

At the end of the 42 posts I will give you a shopping list of features, with the Feature Requirement and the Requirement Rationale which you can use as a requirements document or evaluation whenever you need it. I will also aim to add the features into that list that we know we need but haven’t built yet.

I will update this post as I go, to create a table of contents for the features so to speak, or you can click on the “42 Features” menu above to get a list of features/articles I have already posted.

Still to come:

1. Catalog Details

2. Data Map

3. Design Map

4. Disco

5. Answer Quick Question

6. Ask ADI

7. Trust Rules Roll Up

8. Custom Trust Rules

9. File Upload

10. API’s

11. Change Rule

12. Explore

13. Catalog Review

14. Menu Anywhere

15. Opinionated Data Architecture

16. Topic Canvas

17. Shared Tiles

18. Trust Rules Results

19. ADI assist

20. Events

21. Explore in Looker Studio

22. Share Google Sheets

23. Data Preview - Profile

24. Catalog - Business Context

25. Rule Data Diff

26. Rule version

27. Concept / Event Matrix

28. Renamed consume fields

29. Rule Step Wizard

30. Data sync

31. Natural Language

32. PII detection

33. Orchestration of dependent code

34. Data Contract

35. Export / Import rules

36. Data Match

One of the main changes was moving from a menu structure that had a subset of the Outputs as sub menus under the Outputs category:

To a single Outputs menu that had all the outputs listed.

A few other changes:

• We also moved the Upload and Notifications menus to the the top menu, to make them easier to find.

• We extended the Output version dropdown from only showing the date/time to include a combination of the Input used and the date/time, to make it easier to find the output version you want.

• We provide a link from each notification to the Outputs screen.

• We broke the Physical and Reporting models to include options for model patterns, i.e Dimensional vs Data Vault.

• We automated the creation of a Zip file with all the Outputs, so you can easily download them in one go.

All these changes were all based on feedback from our first user.

I knew which Output was under which Output Type, as I defined them, our first user didn’t. So they couldn’t easily find the Output they needed.

A simple change of menu solves that.

When I started out my journey into the product domain, I was that typical product novice.

I always thought our product wasn’t easy enough for others to use ( I was right, but thats not the point), and so I held off letting others use it.

This wasn’t that much of a problem for us, as our initial goal was to reduce the effort of doing the data work for our customers.

We were building a product that focused on:

1. taking a data task that takes me an hour and making it 10 minutes.

2. taking a data task that I have to think about how I can do it, and getting the machine to recommend what I should do, reducing the need for cognition.

3. automating the data task so we don't have to do it, getting the machine to do the work for us.

There were a bunch of table stake features we needed to deliver the basic data tasks required to collect, combine and consume data.

Every new customer we onboarded gave us new data complexity, and we needed to create the features that made that complexity simple. Each time we added these features it made it quicker to do the work next time.

As we were bootstrapping the product, we would often use ugly things to deliver the first cut of the features, initially our Change Rule logic was typed into a Google Sheet cell, with no validation.

And so I convinced myself that the product was never ready to let somebody else lose on it.

That was a mistake.

If I had of done that I would have got feedback earlier that would have been useful.

Would those initial users have continued to use the product, probably not. But that would have been ok, as our goal was and still is to use our Product to provide a Service in a way nobody else can.

But that feedback would have helped us design a better product.

A lesson learn’t, and one I am applying to the AgileData Discovery App.

So if you are interested in using the initial version and providing some feedback, let me know.

Get in touch

Having AI embedded in your product have become table stakes it seems.

I have been thinking about our approach to AI in our product for a while and landed on 3 patterns that I use as a reference.

1. Ask AI

2. Assisted AI

3. Automated Ai

Ask AI

Ask AI is where a human asks a question and the machine replies, this is repeated until the human gets what they need, then the human goes and undertakes the data work.

This is the now typical “ChatGPT” pattern.  The user starts a conversation, asks a question, gets a reply and asks the next question.

Lets use two examples, answering business questions and understanding data to highlight this pattern.

For the answering business questions , the typical example is the Text to SQL pattern, where the Information Consumer asks how many customers there were last month using natural language and the machine creates some SQL, runs it and returns the answer.  Then the Information Consumer asks their next question and gets the next response.

Then they move on to take the actions required to create an outcome and hopefully that outcome results in business value.

For understanding data , an example would be a Data Engineer collecting the data from a System of Capture, the Data Engineer passes the schema for all the tables to the LLM and asks it to indicate which system the data comes from and the LLM responds with something like Shopify.  The Data Engineer can then ask what is the likely Ontology for this data and then what is the likely Taxonomy all based on that schema.

They then move onto the manual data work of Designing the Data Model, creating the code to transform the data to load it into the data model.

The key to this pattern is the human asks a question, gets some information, asks another question, gets more information and finally the human does the data work themselves.

The value of the pattern is it reduces the cognition required to do the data work.

Assisted AI

Assisted AI, is where the machine helps the human do the data task that is required, without the need for the human to ask.

Lets use the same two examples, answering business questions and understanding data.

For answering business questions, an example is an Information Consumer using a Viz tool to graph data.  The tool looks at the shape of the data and suggests the best graph type that should be used to visualise data of that shape.  The Information Consumer can accept that recommendation and use that graph type or override it and create whatever graph type they want.

For understanding data, an example is a Data Engineer who needs to create transformation code to define new Concepts.  The machine looks at the schema and data examples from the Systems of Capture and suggests that the Data Engineer creates Concepts for Customers, Orders, Products, Order Lines and Payments.  It also suggests the unique business keys that can be used to define each of those Concepts.  The Data Engineer can accept those recommendations and the transformation code is created for them to review and publish.

The key to this pattern is the machine is making a recommendation which the human can accept or ignore during the process of doing the data work.

The value of the pattern is it reduces the time and the cognition required to do the data work.

Automated AI

Automated AI, is where the machine does the work, the human is no longer involved.

For answering business questions, an example would be where the machine observes all the business questions that are asked and answered overtime, determines what the intent and outcome of those questions are, and then generates and sends answers to new questions directly to the Information Consumer or an action directly to the System of Action.

For example it might determine that Information Consumers are constantly looking for # of abandoned carts and so it will send a daily alert with the number of abandoned carts on the ecommerce store to the relevant Information Consumer. Or it might go one step further and push the abandoned carts to the System of Action that will send the customer an email reminding them that they have left something in the cart.

For understanding data, an example would be the first time data has been collected from a new System of Capture the machine will determine the relevant Concepts that should be created, automatically create the Data Model and Transformation code to create them, publish that code and execute it so the data is ready for consumption.

The key to this pattern is the machine is doing the data work, the human is not involved at any step in the process.

The value of the pattern is it removes the need for the human to spend time doing the data work and removes the need for the human to have the experience and the cognition required to do the data work.

An AgileData Platform and App view of these three patterns.

I often joked in the old waves of Data Mining and then Data Science that 90% of all analytical models were "Group By’s", 5% were linear regression and then a small percentage were advanced techniques like neural networks etc.

I still think the same is true in the AI wave.

We did some work a while ago in the AgileData Platform using KMEANS to determine load anomalies.  We found that using some Group By patterns we were able to detect the same load anomalies, but at a much cheaper compute cost.  Obviously we need to keep experimenting and testing this and at some stage we will find the use case where we need to move to more advanced patterns.

So here is where my thinking is at with the 3 patterns of AI in the AgileData Product and App.

AgileData Ask AI

We have experimented multiple times with this pattern over the five years we have been building out AgileData.

Initially using some NLP patterns, then leveraging Google Clouds alpha Q&A service.  Each time the level of effort and cost to build this was higher than we wanted to spend and so it was deprioritised.

Like most companies who provide data tooling and technology we are now of course experimenting with LLM’s to see how we can embed automated text to sql capabilities, and more importantly experimenting to see how we can ensure it does not hallucinate and give the Information Consumers invalid answers to their Business Questions.

The emergence of LLM’s and Generative AI tools and patterns means we can finally implement this capability in the AgileData App, at a lower level of effort and cost.  We built the first version leveraging OpenAI and ChatGPT, and as we are closely bound to Google Cloud we have implemented the latest experiments using Vertex AI and Bison.

We are also experimenting with how we can use Ask AI to help understand where to go in the AgileData App to get a data task done, and to reduce the cognition needed to use the App.

We will keep iterating these to improve the usability, expand its use cases and also reduce the chances of the dread hallucinations (aka when the machines makes shit up and provides answers the user cannot trust)

AgileData Assisted AI

We have had ADI, our version of a chat bot, in the AgileData App, since day one.  Originally she was your typical chatbot “phone menu”, where she presented a bunch of menu options.

Then she became a feedback loop, when you are doing a task in the AgileData App, she explains what will happen next and asks you to confirm.  If the thing you are about to do is semi destructive she will explain the impact in more details and ask you to confirm again.

We will move her to provide more recommendations and assistance over time, so when you are doing a data task in the AgileData App, she will recommend things that will reduce the time or cognition you need to complete that task.

The balance will be making her useful and not useless and annoying like Microsoft Clippy was.

AgileData Automated AI

Again we have been automating data tasks in the AgileData Platform and App since day one.

We do automated anomaly detection to notify you when the data collection looks out of whack.

We automatically check the Data Design is still valid whenever new data is loaded into it.  For example we check that the unique business keys for all Concepts are still unique.  Originally you used to have to manually add a Trust Rule for this each time you define a new business Concept, now we do it automagically in the Platform, so you don’t need to do that effort (or remember to do it).

You can also ask for the Concepts and Details to be automatically designed, generated and loaded from a System of Capture table, and the AgileData Platform will automatically do that Data Work with no input required from you.  We will extend this out to enable it to do the same automated work across multiple tables from a single System of Capture and then across multiple systems.

Group By vs Deep Learning

Often the automation we have built is based on simple patterns, simple code, simple group by’s, not large complex deep learning models.

But in my view that's not the point, who cares what the kitchen looks like as long as I get the meal and experience I wanted.

And that is why we focus on the three patterns of AgileData AI:

1. Ask AI

2. Assisted AI

3. Automated Ai

Each of these are bound by the way a user interacts with the machine, asking questions, getting recommendations, having the work done for them (aka no interaction needed).

How we build the magic in the backend is our problem.

https://agiledata.io/blog/agiledata-product/the-3-patterns-of-agiledata-ai/

My doesn’t 30 days fly by when your having fun #AgileDataDisco

30 days ago Nigel Vining and I decided to do a 30 day experiment.

We wanted to extend our McSpikey to see if using a LLM to do discovery on a legacy data warehouse was a feasible, viable and valuable product idea.

And we decided to do it in public, posting daily about our journey.

Time is up!

And the answer is ……. yes.

We confirmed it was feasible to build, we confirmed it was viable to build with just the two of us.

And we have just got our first paying customer and a verbal yes from a second, so I think we can say it’s valuable.

There is still a lot to do, but I will revert to my usual posting process of writing whenever something comes to the top of my mind. No doubt some of that will be about AgileData Disco. So keep following along if you are interested.

And we have sponsored a stand at Big Data London in September, so if you want to see the product, pop along to stand Y760 and say hi.

Those that know me well, know I have a thing for t-shirts.

And given we have a new product, I thought it was only fitting that we got a new ADI for this product.

I give you …..

AgileData Disco!

2024/07/05

Product Market Fit is wow

#AgileDataDisco

As in wow that is frickin cool, not as in Way or Working.

For AgileData we have a simple sales message.  It will cost you at least $600k per annum in salaries for a data team.  We (and our AgileData Network Partners) will deliver that as a Fractional Data Team for $60k a year.

You will also need to pay for a bunch of data technology, data collection, data storage, transformation tool, catalog etc, and then all the ongoing storage and compute costs to run that technology.  We provide all that in the $60k (yes you read that right, that includes the storage and the compute costs, we pay for it not you)

You will be amazed at the lack of wow we get from this sales message.

I have been socialising the AgileData Disco prototype and outputs a lot over the last 30 days.

I am constantly hearing people say wow.

And then they do one of two things.

They say, the prototype didn't automagically create it, did it? You did some manual turking to produce that. To which I demo uploading a query log from a BI tool, hitting generate and a few minutes later showing them the output, seeing is believing.

Or they stop and think, and then they come back with a use case they could apply it to right now.

I think that wow is what Product Market Fit looks like.

Next let’s close the 30 days experiment out #AgileDataDisco

Follow #AgileDataDiscover if you want to follow along as we build in public.

2024/07/04

Where is my moat?

#AgileDataDisco

TL;DR

There is no moat!

You know when you buy a car. You think oh that's a little different, and then after you buy it you notice hundreds of them on the road.

This is different, everybody is building an AI product, you don’t even need to look around to find them.

It’s so bad I wouldn’t be surprised if when I visit my 74 year old mother next I find she is building an AI product on her new laptop.

Every data vendor is embedding AI into their current products.

Every data consultancy I talk to is using bench time to build an AI powered consulting product.

VC founded AI startups are emerging from “stealth” on a daily basis.

The big LLM providers are releasing capabilities that eat these start-up ideas daily and then provide them as part of their fixed monthly fee or pay per token offering.

The cost of access to these LLM capabilities is so low there is little if any barrier to entry.

In fact you can use a LLM to help you build your LLM based AI product, reducing the barrier to entry even more.

I talk to lots of people and I am finding AI products being built for so many different use cases over and above data.

Our AgileData Disco experiment is based on a language, patterns and pattern templates I have been iterating on for over a decade.

But that is no moat.

I reckon I can use somebody's LLM powered product for a little while and then reverse engineer the majority of their prompts and RAG patterns to have a clone quickly created.

We are using a lot of the AgileData capabilities we have already already built for AgileData Disco, things like private tenancy’s, file upload, Google Gemini API’s integration etc.

But lots of other data vendors have these things.

Our AgileData Network is growing, but it’s not big enough to deliver access to a market that nobody else has, it’s not big enough to give us (and them) a flywheel motion (yet).

There is no moat.

Not for us, and not for most other companies building something using GenAI and LLM’s.

And I am ok with that.

I see enough potential value in what we are doing, to keep investing our time in experimenting with AgileData Disco.

I see ways we can help customers reduce the complexity of their data in a simply magical way.

I can see how it will help us achieve our vision of working with people we respect, who can work from anywhere in the world (we think of it as working from any beach), who get paid well for their expertise and experience, to support the lifestyle they want.

Next my second to last post in the #AgileDataDiscover series.

My doesn’t 30 days fast when you are having fun!

Follow #AgileDataDiscover if you want to follow along as we build in public.

2024/07/03

Let’s think about the pricing options

#AgileDataDisco

Here is when the chicken and the egg pattern hits again, without defining the GTM channel its going to be difficult to define a pricing strategy.

Time to brain dump via writing to collate the jumble of thoughts in my head.

A key Product Management pattern for pricing is to focus on the value delivered to the customer not the cost it takes to deliver it.

The problem I have with this is we end up with that ridiculous behaviour I outlined where I would get “custom” pricing for a bottle of milk in a supermarket.

While being profitable is important, we are boostrapping and so need to be profitable as we don’t have VC funding to spend money while bleeding money, Nigel Vining and I want to build a sustainable business we can run for 20 years not a “growth engine” we can flick quickly and move on to the next shiny thing.

Our core values and our vision leads us to a pricing model that means we make good money, but fair money.

We will probably leave money on the table with this approach but I am ok with that.

With that in mind lets list the options we could use to charge for AgileData Disco.

• Pay per token used

• Pay for a set of tokens in advance

• Pay per input uploaded

• Pay per input used

• Pay per output created

• Pay each time its run to produce an output

• Pay a fixed monthly fee

I tested these with the leader of a data team asking what their preferences was for paying for the outcomes AgileData Disco delivers, they responded with:

1. Pay per output

2. Pay for a bunch of tokens in advance

3. Pay per token used

4. Pay each time it is run to produce an output

One problem is I am not doing robust research by asking that, I gave a predefined list of options and asked for an answer, rather than explored the problem and value space.

I am clear research skills is a small T for me. But we decided that part of this experiment is a Way of Working that means it's only Nigel or me working on this, so we use the T Skills we have.

If I link this answer back to the GTM channels I thought of in the last post.

#1 Pay per output & #4 Pay each time it is run to produce an output.

Equal some risk to us, we set the price and we wear the overs and unders. We can partially mitigate this by limiting the size of the Inputs, we know the size of the Prompts and the likely Output size so we can estimate the potential cost per Output or cost per Run.

#2 Pay for a bunch of tokens in advance & #3 Pay per token used.

Much easier and less risky for us, we already provide a guesstimate of the number of tokens to be used in the prototype , so we can just times this by a unit rate and wallah.

#1 & #4 seem to align in my head with our current Channel led GTM and #2 & #3 seem to align with a PLG GTM. But I think I am just making shite up with those mappings.

So no closer to a decision.

Next let’s think about the dreaded moat #AgileDataDisco

Follow #AgileDataDiscover if you want to follow along as we build in public.

2024/07/02

Next let’s review our GTM channel options for

#AgileDataDiscover

I was going to call this post “lets pick our GTM channel option” but I am not at the stage where I have enough certainty on which option to pick, so I am going to hold off on this decision yet again.

I would love to hear from any Product Leaders who have any suggestions on the patterns they use to make this call.

One of the benefits of building in public and these posts, is the writing forces me to clarify my thoughts. It helps me take the jumble and try and make some sense of it. So let's get that jumble written down.

When I think about the channel options I think about three patterns.

First, we are great at building, ok at selling, crap at marketing.

So I naturally levitate towards Product Led Growth and Growth Marketing, it seems like the nirvana option.

This would require us to make #AgileDataDisco a no touch SaaS product.

We would auto provision, which is ok, we have a pattern for that already. Sign-up is ok, we use Google Identity Manager for this already.

Onboarding would be a lot less complex than the full AgileData capability, we would have a simple flow of uploading a file (query log etc), pick your output types (Information Product Canvas, Conceptual Model, Data Dictionary etc) and then hit generate and wallah!

We would need to build out the subscription management, but we would use a SaaS subscription management solution for that and integrate it, rather than build our own.

That would still leave the marketing problem (unless we hit the magic viral nirvana from listing on Product Hunt which is unlikely). As we are bootstrapping we don’t have the dolleros to spend big on ads and outbound email spam (which we wouldnt do anyway)

Second, we are already building out an AgileData Network of Consulting partners around the world.

We have learnt that providing them with a Swiss Army knife selection of things they can offer their customers, over and above the Fractional Data Service has value. For example the ability to offer the AgileData Courses to their customers.

We could add AgileData Disco to this toolkit so they would be the only people who can use it.

When I talk to new potential partners one of the most common questions (after how much) is who finds the new customers.

This GTM pattern means that effort still sits with them.

Which takes me onto the third pattern.

We offer Agiledata Disco using a freemium pattern.

Same work to be done as the PLG pattern, but we limit how much of the product a customer can use and if they want to carry on, we pass them over to an AgileData Network partner.

This approach would be good for us and the partner, not sure how a customer would feel about it. I need to test that.

Of course it doesn’t solve our fiscal marketing constraint.

So I am no closer to a decision on this one.

Next let’s think about the pricing options #AgileDataDisco

Follow #AgileDataDiscover if you want to follow along as we build in public.

2024/07/01

There must be patterns I can use to help us decide our GTM for

#AgileDataDiscover

I spent a short amount of time searching Google and browsing the Reforge templates, but in the end I reverted back to the help of the LLM, this time ChatGpt.

It’s interesting how much easier I find the ubiquitous ChatGPT app which can I access on both my iPhone, IPad and MacBook compared to the browser based interface of Google Gemini when I want to use an AskAI capability for something.

On the subject of GTM, after a bit of prompting ChatGPT replied with:

>o<

Top 7 Steps to Define a SaaS GTM Strategy

1. Identify Target Market and Segments

Conduct market research and create detailed buyer personas.

2. Define Unique Value Proposition (UVP)

Clearly articulate what makes your product unique and develop compelling messaging.

3. Choose the Right Channels

Select and utilise the most effective channels for reaching your target audience.

4. Develop a Pricing Strategy

Offer tiered pricing and consider a freemium model or free trial.

5. Create a Demand Generation Plan

Produce valuable content, optimise for SEO and SEM, and engage through email and social media marketing.

6. Implement Customer Onboarding and Success Programs

Develop a seamless onboarding process and provide robust customer support.

7. Measure, Analyse, and Iterate

Track key metrics, collect feedback, and continuously improve your product and strategy.

>o<

Target market is any organisation that has a legacy or current data platform and no adequate documentation for it, which is a bit broad. Will need to refine this market a lot more as we get deeper into validation using the MVP.

For persona’s I have identified three main data personas that I think the AgileData Disco product will help, a Head of Data, a Data Governance Manager and a Data Consultant. All of these of course have various different roles and titles depending on the organisation they work for.

Funnily enough only the Data Governance Manager is a new persona for us, so there is a risk that I have brought in a bunch of bias for these, time will tell.

For the Unique Value Proposition current draft is:

>oo>

For the Head of Data (or Data Governance Manager or Data Consultant)

Who needs to discover, document and understand their current Data Platform so they can rebuild it or augment it.

The AgileData Disco is a Product

Which automagically discovers and documents a data platform, producing actionable outputs in less than one hour.

Unlike the current manual task that takes multiple people multiple months.

The AgileData Disco Product will:

- Accept multiple input files

- Output multiple pattern templates that are actionable

- Not require a direct connection to the data platform

- Provide outputs that can be stored offline

>oo>

Next let’s pick the GTM channel option for #AgileDataDisco

Follow #AgileDataDiscover if you want to follow along as we build in public.

2024/06/29

GTM, Houston we have a problem

#AgileDataDiscover

While I have worked in the Data Domain for over 3 decades I am fairly new to the Product Domain. There are a bunch of product patterns I have not found or used.

I feel like defining a Go to Market approach is one of those.

I have a myriad of things buzzing around in my head.

Do we go SLG, PLG, CLG, ELG? Are there other LG’s I haven’t found. Are these LG’s strategies, patterns or tactics (and do I need to care about the distinction?)

How do tactics like Freemium fit into it, I am assuming you can only apply a Freemium tactic when you are using a PLG motion, is this true? I have done free consulting in the past as a loss leader, isn’t that freemium?

Is Freemium a tactic?

I think I know why they call PLG a motion, it’s all about the flywheel, do I need to care about the definitions?

I have strong opinions on what I prefer and what I don’t prefer, I have strong opinions on how AgileData will behave and how it won’t behave.

There are patterns that although they are known to be effective, I am adamant AgileData won’t use them.

For example the whole GTM SLG pattern using SDR’s and AE’s, making pricing opaque, so they can make a price up to extract the most value out of the customer.

I dislike that pattern.

I posted this on another LinkedIn thread the other day that was discussing software vendors pricing tactics.

>oo>

I went into the Supermarket the other day to buy some bread and milk #SnarkyMcSnarky

I perused the aisles and found a bottle of milk I liked, I went to grab it but one of the supermarket staff told me to stop.

They took me to a counter where I was asked why I wanted to buy milk, when I was likely to buy the milk, how many people at home were going to drink the milk, how much everybody in the house earned, who’s name was on the credit card I was going to use and had they signed off on the credit card expenditure.

After I had answered those questions I asked how much the bottle of milk was.

I was told to take a seat and wait.

A second person eventually came along with the bottle of milk I wanted.

They brought out a glass, poured some milk into it, swished it under my nose, they took a drink of it and told me how good the milk tasted.

They asked would I like to buy some milk.

I said yes I would, how much is the bottle of milk.

They told me to sit down again and wait.

Eventually the second person came out with a sealed envelope. In it was the price of the bottle of milk.

The price was $10,000

I didn’t buy the milk.

This is the process some vendors take a potential customer through.

Does that sound ridiculous to you?

>oo>

While I have strong opinions, can make a decision and can self justify how I made it, this build in public process is as much about the learning as the outcome.

There must be patterns I can use to help me decide our GTM.

Let’s explore patterns that are available.

Follow #AgileDataDiscover if you want to follow along as we build in public.

2024/06/28

Current AgileData Go to Market

#AgileDataDiscover

I was talking to somebody the other day and they made a comment about startup founders who build the best tool in the world that nobody ever knows about.

When you are building a product you have to care as much about how you will sell it as you do about building it.

I much prefer building over selling.

For AgileData we did a bunch of experiments with our Go To Market (GTM) last year to see where we would place our bet for this year.

The experiments were based on the usual GTM patterns:

Sales Led Growth (SLG)

We have a team of people who find and sell direct to our customers.

Product Led Growth (PLG)

We let our customers serve themselves, they sign up, subscribe and use the product with minimal assistance.

Partner/Channel Led Growth (CLG)

We work with other organisations who sell our product to their customers.

EcoSystem Led Growth ( ELG)

We piggyback on a large technology vendors ecosystem/marketplace and customers find and buy our product via that.

Each of these GTM requires some sort of investment:

For SLG we hire a team of Sales Development Representatives (SDR’s) who do outreach and qualify the opportunity’s before handing it to an Account Executive (AE) who closes the deal.

For PLG we need to spend money advertising so people can find the product, we need to ensure the product is so easy to use that they don’t need any human assistance to use it and they get value out of it quickly so they don’t cancel.

For CLG we need to figure out how to find potential partners, convince them there is value for them if they partner with us, upskill them in our product and then wait until they find and onboard customers.

For ELG we need to pick a large vendor, integrate our product with theirs, validate our product so we can be listed on their marketplace, then wait for the deal to rolling in, or not as the case maybe.

All of these GTM pattens need a way to fill the top of the sales funnel, which is another set of patterns.

Most companies use a combination of these GTM patterns and investments, which becomes complex.

We are always focussed on simplicity over complexity.

So we decided that for 2024 we would place a bet on Partner / Channel Led Growth.

The logic and reasons behind placing this bet will require a much longer post, but one of the key reasons comes back to the vision Nigel and I have for AgileData:

“working with people they respect, who can work from anywhere in the world (we think of it as working from any beach), who get paid well for their expertise and experience, to support the lifestyle they want.”

Now we have the context for the current AgileData GTM bet, lets think about the GTM options for #AgileDataDiscover

Follow #AgileDataDiscover if you want to follow along as we build in public.

2024/06/27

Whats happening in the Data & GenAI / LLM space, an #AgileData #Podcast with Joe Reis 🤓

#AgileDataDiscover

The latest AgileData Podcast has hit the airwaves:

https://agiledata.io/podcast/agiledata-podcast/ai-data-agents-with-joe-reis/

As part of the 30 day experiment we are doing I wanted to understand whats going on in the wider Data / GenAI / LLM space.

A great way to understand this is chatting to Joe Reis to hear what he is seeing and hearing.

Then I thought why not record it as a podcast so we can share his insights.

So thats what we did.

Thank you as always to Joe Reis 🤓 for being so generous with his time!

2024/06/26

Opps where did that pesky data come from?

#AgileDataDiscover

We are hyper paranoid about our customers' data, as any data platform provider should be.

We run a private data plane for our customer tenancy’s as part of our agile-tecture design.

So we of course adopted this pattern for the inputs we used in the Discover prototype to keep them secure.

Even though the input files are secured in a private tenancy we made sure that we only used Metadata as inputs into the Prototype not actual data records.

We believe we can get the outputs that are needed without having to pass the LLM actual data and we know from experience the barrier requiring data has on a product’s adoption.

One of the inputs we tested was using query logs from the BI tools querying the data.

Imagine our surprise when we started seeing actual data values in the LLM’s outputs!

Lots of thoughts surging through the brain, until we realised where the LLM had got them from.

In the Consume view the reports were using we had a case statement. The case statement was being used to map some reference data.

It was an old pattern we used before we built the ability to dynamically collect and manage reference data via Google Sheets.

So of course this reference data mapping was in clear text in the query logs and that is where the LLM was getting it from.

We will have to think about a pattern for scanning the LLM output for data values. Not sure how far we should take that.

The platform is secure, we are using Google Gemini within a private tenancy, rather than passing the inputs over the internet to OpenAi’s API like most other products so we know any data values are secure.

But should we surface them in the output or not.

We will need to get some feedback when we test the MVP for this one.

Feel free to post a comment if you have an opinion.

Next let’s think about our ideal go to market #AgileDataDiscover

Follow #AgileDataDiscover if you want to follow along as we build in public.

2024/06/25

Whats happening in the Data & GenAI / LLM space, an #AgileData #Podcast with Joe Reis 🤓

#AgileDataDiscover

The latest AgileData Podcast has hit the airwaves:

https://agiledata.io/podcast/agiledata-podcast/ai-data-agents-with-joe-reis/

As part of the 30 day experiment we are doing I wanted to understand whats going on in the wider Data / GenAI / LLM space.

A great way to to understand this is chatting to Joe Reis to hear what he is seeing and hearing.

Then I thought why not record it as a podcast so we can share his insights.

So thats what we did.

Thank you as always to Joe Reis 🤓 for being so generous with his time!

2024/06/24

Let’s look at the new patterns we need to move from Prototype to MVP

#AgileDataDiscover

We have been able to use a lot of the patterns we have built into the AgileData Platform and App as part of the Discover prototype.

To move it from Prototype to MVP and add the initial list of features we need, a number of new patterns need to be implemented under the covers, including:

1. Logging of prompt tokens so we can keep track of how many are being used each time we send a prompt off to the LLM

2. Logging of responses from the LLM so we don’t need to make unnecessary requests, we can just show the latest response if its is still applicable

3. Uploading multiple input files and including them in the prompt sent to the LLM, this turns out to have the most complexity as the documentation is very light, and is currently skewed towards processing multiple image and video files

4. Lightly modelling the most extensible pattern to hold configuration data, ie bite size prompts that can be combined to produce different outcomes

5. Parsing and storing the raw markdown responses into separate buckets so they can be surfaced separately in the app.

Most of this activity happens in our API layer as thats where the AgileData magic happens. We add new endpoints when required or augment existing endpoints to deliver additional functionality for the web application.

In this case we re-use our existing file upload pattern but send discovery files to a separate bucket where we can access them from the new interfaces.

We re-use our existing LLM wrapper, but extend it slightly to provide the option to choose which model is called . We use different models depending on their strengths and weaknesses, some are better at ‘texty’ stuff, while others are better at ‘codey’ stuff.

So a few bits of behind the scenes plumbing we need to do to make the product scalable. It’s tempting to bypass some of these for the MVP, but we already know after spending some time using the prototype, that without them we would end up in a world of manual effort hurt.

Scale the machine, don’t scale the team!

Next we will talk about a little surprise we encountered with the prototype.

Follow #AgileDataDiscover if you want to follow along as we build in public.

2024/06/23

Let’s review the feedback so far and the use cases that have started to emerge

#AgileDataDiscover

After talking through the product idea with a few people and showing them the prototype the feedback has been positive.

The uses cases that have come out of the conversations look something like:

Understanding a legacy data platform

An internal data team or a data consultant(cy) wants to understand the current data platform and the human effort to do so is large and time consuming.

This might be supporting a green fields data platform, or making a change in the current data platform.

All the people who have built it have left and there is no upto date documentation.

An interesting variation is the senior leaders in the organisation have left and the new leaders have no idea what was built or why.

In a number of conversations I was asked if we could produce data lineage back to source systems.

The typical approach to do this would be to parse the ETL code or execution logs for the code. I can potentially see how the LLM might be able to create the lineage if we gave it a bunch of the inputs, but I am worried about the level of hallucination it will give back.

Data Governance

This is effectively the same pattern of documenting the data platform, but with slightly different use cases.

A lot of Data Governance teams have been reduced in size and the technical BA team members have been laid off, meaning the Data Governance Managers no longer have people available to manually document the data.

Or the Data Governance Managers have relied on the technical BA skills in the data teams, those teams have been downsized and can no longer do this work for them.

A lot have tried to do the work using a Data Catalog and that work has failed to deliver any results.

An interesting pattern emerged in this space.

Can we upload regulations or policies and determine if the data platform is complying with them? I am going to have to experiment with this one, as I can’t think of a pattern we can use to deliver this use case yet.

Automated Data Migration

As with all things AI there are a myriad of startups experimenting in this space.

As part of the conversations I have stumbled across a bunch doing similar things outside of the data domain. And talked to people who know people who are doing similar things in the data domain.

A use case that came up a couple of times was automating Data Migrations from one data platform / technology to another data platform / technology.

Overall there seems to be a bunch of use cases that people have said will have value to them, let’s carry on and invest some more time and effort into it.

Next let’s look at the new patterns we need to move from Prototype to MVP.

Follow #AgileDataDiscover if you want to follow along as we build in public.

2024/06/22

One response to rule them all, not! 

#AgileDataDiscover

As I started showing people the prototype one bit of feedback I consistently got was that having all the outputs in one screen was confusing.

There are a few other problems with the single output model we have initially built into the prototype.

The number of output tokens the LLM will generate is limited, which means the LLM reduces the number of words used as it moves through the output types. 

We want the definition of the Data Dictionary to be as verbose as the Information Product, even when the LLM produces it as the last part of the output.

We want to be able to regenerate just one of the output types in isolation, not all the output types.  A single output response type makes this difficult and costly.

We want to make it easy to copy and paste one of the output types to another medium, right now I have to copy the entire output and then remove the bits I don’t want.

The single output pattern we have built into the prototype won't allow these things to happen easily.

And also scrolling down a long list of output text to find what you are looking for is just a bit of a pain in the arse.

So we need to break the various output types into their own unique objects.

Luckily its an easy enough change to experiment with (well Nigel made it look easy enough ;-). 

We have already decided to break the Discover menu for the prototype out into its own menu for now, so it’s “just” a case of adding new menu options that only display the response for that output type.

We have split the outputs into the following screens / menus / output types:

Information Product

Models

• Business Model

• Conceptual Model

• Physical Model

• Reporting Model

Glossaries

• Business Glossary

• Data Dictionary

Metrics

• Facts

• Measures

• Metrics

Matrix

• Bus Matrix

• Event Matrix

• Concept Matrix

One of the interesting benefits of this change is it makes it much clearer to explain to people what we are producing and what they can be used for.

I have spent the last decade finding and iterating pattern templates that have value in the data domain.  That knowledge and content has been instrumental in defining these output types.

And that clarity has resulted in getting much clearer feedback on what output types we are missing.

Next let’s review the feedback we have had so far from surfacing the prototype.

Follow #AgileDataDiscover if you want to follow along as we build in public.

2024/06/21

Let's sort out that menu problem, for now anyway  #AgileDataDiscover

It’s not a big thing, and I know I shouldn’t be sweating the small stuff but I also know that if I don’t it's going to really needle me until I scratch this itch.

And this is where I would normally hand the problem off to our UI designer to use their experience and expertise to solve the problem by using their dark design arts.  But with our experiment in our new Way of Working it’s either Nigel or me, or both of us.

A problem shared and all that, so a Slack huddle with Nigel it is.

When talking through the problem Nigel mentioned what I had done to the header area in our marketing website.  I have iterated it to use each of our core colours depending on what part of the AgileData product we are talking about, Network, Wow, Platform or App.  Why can’t we do that for the App header?

Intriguing thought, we use those core colours heavily in our course material as well to indicate a transaction from one course module to another.

We could spin on this decision for months if we let ourselves, but one of the key things a Product Leader has to do is make decisions quickly if they can, and I can.

So we will introduce a pattern where the App header changes colour when you are in the Discover area.

Now how to get to it?

Well we always talk about Personas, and again we teach data and analytics teams how to apply the persona pattern in the data domain (hell it might even become the second AgileData Guide book ;-) 

Let's add the ability to quickly change personas in the app and as a result change the menus that are presented to us so that we can get the job that personas needs to do, done.

And speaking of jobs, on the menu options, job done (for now)

Now let’s move on and sort out the usability of the one big prompt response.

Follow #AgileDataDiscover if you want to follow along as we build in public.

2024/06/20

Protect my top menu at all costs, or not?

#AgileDataDiscover

One of the patterns I have been focussed on since the first day we started building the AgileData product is trying to retain simplicity in the top menu of the AgileData App.

I have been in the software world long enough to remember the days when we had text only screens.  I remember when the Graphical User Interface was introduced and we got the little icon instead of text for menus, and then funnily enough we got text as a mouse-over on the icons so we could know what they did.  It still makes me giggle when a UI has the diskette icon as the save icon.

I have seen the complexity of menuing systems that come with enterprise products such as SAP and Oracle.

From day one I wanted to try and retain a simple top menu structure, again influenced by Xero.  Originally I thought we would need to move the Manage menu to a left menu option at some stage to allow a myriad of options but we haven’t been forced to do that yet.  We also haven’t needed to extend the top menu to more than a single level of sub-menu.  I love this simplicity. 

However we have extended the number of top menu items as we expanded the capabilities of the app.  We introduced the Design menu option a while ago, we moved the Collect menu option out from Manage to its own top level position.  And we are about to introduce Marketplace as a new top level menu.  This will result in 7 menu options at the top and this in my view is the limit. 

As part of the introduction of the Markekplace option I am rejigging the order of the menu’s as well.

Until now the order was:

Catalog | Consume | Design | Rules | Collect | Manage

It will become 

Marketplace | Catalog | Collect | Design | Rules | Consume | Manage

Previously I had the Catalog and Consume on the left as those were the two menus designed to help Information Consumers.

The introduction of Marketplace is designed to help Information Consumers find the Information Products they need, while the Catalog will continue to help Data Magicians find the Data Assets they need.  

As part of this change I have reordered the other menu options to follow the right hand side of the Information Value Stream flow that we teach, i.e Design > Build > Deploy > [Value] > Maintain.

So what to do about the AgileData Discover menu option? 

The Discover option takes us into the left hand side of the Information Value Stream flow, i.e Problem > Idea > Discover > Prioritise.

I can predict we may need to break the Discover option out into a series of sub menus, so that means it can not become a sub level menu under Design. Plus Discover is in the left hand side of the Information Value Stream and Design is in the right hand side.

That leaves it becoming a top level menu and that would make 8 and that’s a no. 

Ok we need to find a different solution for my menu problem.

Follow #AgileDataDiscover if you want to follow along as we build in public.

2024/06/19

Let’s show a few people the initial prototype and see what feedback we get.

#AgileDataDiscover

I am lucky to have a network of talented data peeps across the world, people I have connected with over the years, people whose opinion I respect.

As part of the No Nonsense Agile Podcast we have a series of awesome guests on the show, and one of those was Teresa Torres.  One of the key takeaways I got from that chat was the pattern of continuous discovery, my key takeaway was talk to 3 people a week, and ask them about one thing that has popped up for me that week.  Something I have been trying to do for a while now.

I combined those two things and started asking people about their data problems and showing them the prototype to elicit some feedback.

The conversations were enlightening, I can see a few problems we might be able to solve with this thing.  Ill explore those in more depth later.

One bit of feedback I got as I started showing people the prototype was that having all the outputs in one screen was confusing, we need to iterate that.

Another thing that frustrated me from day one with the prototype was trying to find the url to demo it.  Especially when we have a couple of different demo Tenancy’s.  It’s a small thing, but it’s one thing that has caused me frustration and a surprising amount of wasted effort and time in the past, so i'm very conscious of it.

We were always going to have a bunch of decisions to make about where this fits in our current product Information Architecture aka our menuing system.

Let’s deal with that menu problem next.

Follow #AgileDataDiscover if you want to follow along as we build in public.

2024/06/18

Let’s build an initial prototype.

#AgileDataDiscover

We have already done some prototyping as part of the initial McSpikey.

We can provide specific inputs into the LLM, we can give it a specific language to use, we can get the outputs we think have value.

But in the McSpikey we did that manually.

Next step is building out a prototype app that allows customers to do that with or without us.

Here is the first cut of what we built:

https://guides.agiledata.io/demo/clx2hlkru0ibi35xebw371v0f

In this demo example I used a file that contains the Consume query log for my demo AgileData Tenancy.

As part of this step we learnt a lot about the feasibility of building the potential product.

Application Framework:

We need to build, deploy and run a browser based application, this capability is a core part of the AgileData App.

Secure Sign-in:

We need to allow customers to securely sign in, this capability is a core part of the AgileData Platform.

File Upload:

We need to upload input files and store them for use by the LLM, this capability is a core part of the AgileData Platform and App.

Data Security / Privacy:

Even though we wouldn’t be using Customers actual data for the prototype we would be using their logs etc at the MVP stage, we need to make sure whatever we use is stored securely. The AgileData Platform architecture provides a Private data plane for every customer tenancy, we can reuse this for the prototype and deploy a new AgileData Tenancy for each MVP customer in the future.

LLM:

We need to access an LLM via API so we can run it from within the protoype app. AgileData is built on Google Cloud, Google Gemini is the obvious choice of LLM service for this.

We have already used Google Gemini as part of the Customer Trust Rules feature we recently released, we have a lot of the plumbing (engineering) patterns in place for it already.

Prompt Storage:

We need a way to store the prompts we are using so they can be used across all prototype Tenancy’s. The AgileData App stores everything in Google Spanner which is then accessed by the AgileData Platform, so we can use this to store the prompts.

We were presently surprised how many reusable patterns we had built as part of the AgileData, that we could reuse for this protoype and the fact they are all production ready.

Let’s show a few people the initial prototype and see what feedback we get.

Follow #AgileDataDiscover if you want to follow along as we build in public.

2024/06/17

Let’s experiment with our Way of Working while we do this product experiment.

#AgileDataDiscover

In response to my last post on Research led vs Solution led Chris Tabb replied with:

“Do both 🤷‍♂️
Bet on two horses 🐎
Increase your chance of winning.
Similar to the difference of

👉Logical vs creative”

To which my responses was:

“funnily enough we have been using a hybrid pattern of the two for a while. We didn't end up with the worst of both worlds, but I'm not sure we ended up with the best of both worlds. So it’s time to try something new and see what happens.”

Until now we have been building something to solve a series of data complexities that we have experienced as data professionals over the last 3 decades.

In the beginning the target user for the product was me. If I could do the data work on my own, without having to ask Nigel Vining to write code, then that was the initial goal achieved.

As we achieved that goal we started focussing on three measures of success:

• If we could reduce something that takes me 1 hour of effort to 10 minutes of effort, that would be a good thing

• If we could reduce the cognition, expertise and experience required to do the data work, that would be a great thing

• If we could automate the data work so the machine did it, not a human, that would be a magical thing

As an engineer Nigel Vining doesn’t really care how our product looks and behaves, but I do.

Having worked at Xero briefly in its early days I got to see the benefit of great UX and UI design. So early on we started working with an awesome product designer in Kris Slazinski ☮️

Being a fractional team working in the different timezones, we found the process of great design upfront started to slow us down and we got impatient.

Over the years we had built out a design system with a lot of reusable components and we found we could experiment by cobbling those components together and see if it helped us achieve one of our measures of success.

We started moving more and more towards the MVP pattern for how we built and tested new product features ideas.

So which pattern should we lean into for this product experiment.

Let’s look at the summary from ChatGPT again:

• User Research vs. Market Testing:
  UX research perfects user experience; MVP tests market demand.

• Development Time:

  UX takes longer; MVP is faster.

• Risk Management:

  UX aligns with user needs; MVP quickly validates the market.

• Iteration Approach:

  UX iterates on design; MVP on product-market fit.

Based on our journey todate we naturally lean towards MVP.

We are boostrapping which means we are cash constrained, and we have put a 30 day limit on this experiment so we are also time constrained.

Which naturally leads us to lean into our MVP bias with all the work being done by Nigel and Myself.

Let’s build a prototype to learn what we don’t know and have something to test with potential customers.

Follow #AgileDataDiscover if you want to follow along as we build in public.

2024/06/15

Research led vs solution led

#AgileDataDiscover

Being a co-host on the No Nonsense Agile Podcast I have been lucky to speak to some of the worlds best product people.

I have come up with a hypothesis that there are two different Product Value Streams an organisation can adopt.

One is being research led the other is MVP or solution led.

Research led is where the organisation starts by researching the problems that exist in the world, identify problems that could be solved, ideate how they might solve those problems, and then discover whether those ideas do indeed solve it. Only after all that do they build something.

Solution led is where you start with a problem you know or believe exists, you build a minimal viable product that you think solves that problem, then you test the MVP. You are testing both problem space and the solution space and the value space all in one go.

Given #AgileDataDiscover is based on using an LLM I thought I would ask ChatGPT to given me more details on the differences between research and MVP led.

ChatGPT Prompt:

Whats the difference between a SaaS product company that takes a UX research approach vs a mvp lean startup approach?

Reduce it to 1000 characters

ChatGPT Response:

UX Research Approach

1. Focus on User Experience:

Goal: Excellent user experience.

User-Centric Design: Extensive research on user needs and behaviours.

2. Research Methods:

Qualitative & Quantitative: Interviews, usability testing, surveys, and analytics.

3. Design Process:

Iterative Design: Continuous refinement based on feedback.

Prototyping: Detailed prototypes for validation.

4. Development Cycle:

Longer Initial Development: Extensive research and testing.

High Fidelity: Aligns closely with user needs from the start.

MVP Lean Startup Approach

1. Focus on Speed and Learning:

Goal: Quickly validate business ideas.

Build-Measure-Learn: Rapid development, market measurement, and learning.

2. Development Methods:

Rapid Prototyping: Functional versions to attract early adopters.

Lean Testing: Experiments and metrics for viability assessment.

3. Product Features:

Minimum Feature Set: Core functionalities for quick market entry.

Incremental Improvements: Based on feedback and market response.

4. Development Cycle:

Shorter Initial Development: Quick launch.

Pivot or Persevere: Adjust based on feedback.

Key Differences

User Research vs. Market Testing: UX research perfects user experience; MVP tests market demand.

Development Time: UX takes longer; MVP is faster.

Risk Management: UX aligns with user needs; MVP quickly validates the market.

Iteration Approach: UX iterates on design; MVP on product-market fit.

So which approach are we going to take for #AgileDataDiscover

Follow #AgileDataDiscover if you want to follow along as we build in public.

2024/06/14

Should we experiment with our Way of Working while we do this product experiment.

#AgileDataDiscover

When I coach data and analytics teams, I coach them to reduce uncertainty as much as possible, make things simple and don’t boil the ocean.

This is going to be a case of do as I say not do as I do.

We have a massive amount of uncertainty in what this product is or will be, so the last thing we should be doing is introducing additional uncertainty by changing the way we work while experimenting with the product.

When we designed our product development Way of Working a few years ago we had to design it based on a fractional development team that worked in different timezones.  That led us down to a value stream flow of work that was asynchronous and chunked down into UI Design, Backend Engineering and UI Build steps.

We were careful to not create a waterfall process, a process based on large requirements upfront, no discovery and a high cost of change.

But given the need to work asynchronously we naturally adopted a flow based way of working.  And the initial flow was UI Design, then Backend build and then UI build.

Overtime we realised this linear flow wasn’t working for us.  We were over investing in UI Design upfront, which we then discovered wasn’t feasible to build in the backend or the UI.

We ended up with multiple value stream flows depending on the level of uncertainty in what we were planning to build.

Sometimes we will do a very light UI design, build a MVP backend service, then build the UI, use it for a while to see if it had value and identify what we want to iterate,

Sometimes we will do a light UI design, then stubb a backend service, then loop back what we have learnt into the UI design process.

Sometimes we will build the backend service to reduce some uncertainty in what data structures we need and how we can model it, and then move onto the UI design process, then into UI build.

Sometimes we will build a backend service and cobble together a UI with our current components and no UI design work, to help us reduce some uncertainty on the problem we actually want to solve.

The key to all of this is Nigel and I were the key users, we were the customer for our product.

We knew the data problems we wanted to solve, we had lived with them for over 3 decades.

We were focussed on reducing the time it took us to do data work, reducing the cognition it took us to do it, or automating it completely so the work was done without us.  We could measure how well we were delivering these.

For this product we are building it for somebody else, we have a high level of uncertainty, we want to do a high level of experimentation, so we might as well experiment with changing the way we work.

Let’s aggressively iterate the way we work while we do this.

Follow #AgileDataDiscover if you want to follow along as we build in public.

2024/06/13

If this new capability becomes both viable and feasible, where does it fit with our current product strategy.

#AgileDataDiscover

We have a few simple choices:

Treat it as a completely separate product, give it a new name, a separate pricing model and separate Go To Market strategy.

Treat it as a separate module in our current product, give it a new main menu name, add a separate pricing model (unbundling the current pricing strategy).  Retain the current GTM  strategy for this module or create a new one.

Treat it as a bunch of features in our current product, make it part of the current pricing, make it a feature in our current GTM patterns.

There are a few other drivers I haven’t thought about as well.  I immediately start diving down a rabbit hole into a complex multidimensional decision model, and want to reach out to Google Sheets to create a pivot table to manage it.

Sure sign that ugly complexity has appeared, so we need to move back to simplicity.

Lets ignore the Pricing and Go To Market options, we can deal with those later.

Still leaves us with a choice between a separate product, new module or a set of features.

As I have mentioned, we are technologists at heart, we have a small team, don’t plan to ever have a massive team, and as a result of these two things we are great at building, we are ok at selling, but we really struggle at marketing.

Introducing a whole new product would mean we would need to split our limited time and skills across two products.

For that reason alone the separate product option is out, for now.  

This is not a decision that is set in stone, this is just a decision made quickly, so we can move onto the many decisions we need to make next.  We can always change this decision as we learn more, but the cost and consequences of that change will be greater as we progress further into this process.

Also there is a massive amount of bias embedded in this decision.

I don’t want to have to design and build yet another website, which I would need to do if it was a separate product.

I have seen dual product strategies before, Tableau & Tableau DataPrep, Wherescape Red & Wherescape 3D, and I was never a fan of those strategies.

There are probably a bunch of other unconscious bias reasons why I have discounted this option.

That leaves a decision between a separate module, or just a bunch of features.

For some reason I am reluctant to make this call at the moment, I feel like I need some more certainty before we make the decision.

And that’s ok, there is so much uncertainty in everything we are doing for #AgileDataDiscover right now, we can afford to delay some decisions.

So before we make the decision on seperate module or a bunch of features, I want to explore where does building this fit in our current AgileData Way of Working?

Follow #AgileDataDiscover if you want to follow along as we build in public.

2024/06/12

Next we need to see if we should adjust our current product strategy, or is this just product business as usual.

#AgileDataDiscover

We are lucky at AgileData that we are purely a product company (even though we bundle services as part of our Fractional Data Team offering) and so our business strategy and our product strategy are closely aligned if not one and the same.

We made a conscious decision when we started out to reduce the complexity of our business as well as providing a product that reduces the complexity of managing data.

This aligns with Nigel and my personal goals of building a profitable and sustainable business, which we can work within from anywhere in the world, and which we can be part of for the next 20 years.

Part of our business strategy is to bootstrap our company and keep our team, processes and costs ridiculously lean.

As a result we have always gone for simplicity over complexity, and as a result have a single Product Strategy, and a single pricing model.

We know that most SaaS product companies choose to unbundle their products so they can “upsell” their customers to the next “tier” or by adding access to an “advanced” set of features. To these companies revenue growth within their current Customer base is as important to their reporting metrics as new customer acquisition metrics.

Our view has always been that if the product capability makes the data work less complex, why wouldn’t we make it accessible to our partners, after all that's what we are promising them.

We don”t charge extra for more users consuming the data, we don’t charge more to use the Catalog features etc.

Each time we decide to experiment with something new, we need to review our current business and product patterns and see if we want to change them.

So yes, it’s a good time to review our product strategy and see where this might fit within it, or if we need to iterate it.

Next up is where does it fit in our current AgileData product strategy?

Follow #AgileDataDiscover if you want to follow along as we build in public.

2024/06/11

Next we need to explore the initial viability of the product idea a little bit more. 

#AgileDataDiscover

I am lucky I have a network of people I’ve met over the 3 decades I’ve been working in the data domain.  They span a breadth of backgrounds, skills, roles, organisations and locations.

Even more valuable is they are very giving of their time, something I am very conscious of and try not to waste.

We really want to explore the viability of the migration to the AgileData platform use case, but we have a hypothesis that Data Governance / Enablement might be the most valuable use case. 

There is a growing trend for Data Governance teams to move towards being Data Enablement teams, a trend that was reinforced by an excellent presentation at the Wellington Data Governance Meetup I attended recently where somebody was playing back the key themes from the DGIQ conference in Washington DC last year.

The next step is to see if we can get some quick market validation on the value of the product supporting Data Enablement.

I reached out to a couple of people I know in the Data Governance space to “pick their brains”. 

I know I should have taken a more structured product research approach to this rather than just reaching out to people I know.

We are a small team, we don’t have a researcher in the team, I don’t have those research skills, I make do with what I have and what I know.  Then I live with an  imposter syndrome most founders live with everyday, when I do something that is not a strong T-skill.

Here is what I think I heard:

Data Governance Managers are losing their BA team members as a result of the fiscal downturn, or they are losing access to those skills in other teams as a result of downsizing.  They need to do the discovery work themselves.  

How can they create this info as a team of one?

Data Governance Managers who get a new role in a new Organisation, are faced with a standing start. If the outputs they need do not already exist, they are blocked until they can get them created.  

How can they quickly document what already exists?

How can they create insights into what exists, without directly accessing their source systems or internal data networks?

Starting a conversation with a blank piece of paper is always harder than starting with some known state.

How can they generate insights into the Org and its data, as research for themselves, before starting a conversation with stakeholders?

Data Policies and Procedures often exist but there is a lack of visibility if they are being complied with.  

How can they compare what is actually happening with what should be happening?

These seem like problems we can help with.

We have quickly validated the feasibility of the product idea and we think we have quickly validated there may be a market and demand for the product.  

Next up is where does it fit in our current AgileData product strategy?

Follow #AgileDataDiscover if you want to follow along as we build in public.

2024/06/10

Next we want to explore the initial feasibility of the product idea a little bit more. 

#AgileDataDiscover

As technologists at our core Nigel Vining and I always err towards understanding the feasibility of building something, before we work on the viability of building it.  

There is joy in building something that works, compared to getting a constant “no”, or even worse a shoulder shrug, to what you think is a great idea.

To stop us boiling the ocean and spending too much time validating the feasibility of the wrong thing, we have a process where we do internal research spikes, which we call McSpikey’s, to time box the feasibility research and force us to quickly swap back to the viability. 

As part of the Greenfields Data Warehouse Rebuild research spike we did for a customer, we tacked on an internal McSpikey we had been thinking about for a while.

For this McSpikey we tested the LLM based on using these inputs:

• Screenshots of reports

• Exports of report definitions

• Query logs from the BI tools

• Query logs from the ETL loads

• The ETL code

• DDL for the data warehouse schemas

• DDL for the source system

• 550 page word document that contained a data warehouse specification

• Sets of gherkin templates for transformation/business rules

• Sets of data by example templates

Then we created a series of tailored prompts, based on the shared language we have crafted and use when we are teaching via our AgileData Courses, coaching Data and Analytics teams or helping our AgileDataNetwork partners with their Way of Working.

Last we used the LLM to produce these artefacts:

• Information Product Canvas

• Business Model

• Conceptual Model

• Physical model

• Reporting model

• Business Glossary

• Data Dictionary

• Definitions for Facts

• Definitions for Measures

• Definitions for Metrics

The outputs the LLM produced were much better than we expected, the key seemed to be the quality of the shared language we used, which we have been crafting and refining for over a decade.

Another interesting discovery was the report screenshots cost us twice as much to process compared to the text based inputs.  That's the value of McSpikeys, you always learn something new.

This product idea looks feasible, we should explore the viability of it some more.

Or put it another way, now we think we can build it, will anybody buy it or use it?

Follow #AgileDataDiscover if you want to follow along as we build in public.

2024/06/09

Some background on what led us to the decision to do this 30 day bet on #AgileDataDiscover

#3 Greenfields Data Warehouse Rebuild

One of the things we have found our customers value, is the speed and cost which we can execute Data Research Spikes using AgileData.

We fix the delivery time period to 3 months and fix the cost for the work.

Often customers already have a data team and data platform, but the team is under the pump with higher priority data work and cannot do the research work anytime soon.

The goal of the research work is to reduce uncertainty, or show the art of the possible.  

Often a business stakeholder has an idea and wants to know what is feasible, before they decide to invest in it fully.  

The Head of Data wants to deliver the research work, but the data team has no spare capacity to do it.  The Head of Data prefers the business stakeholder does not independently engage yet another data consulting company with whom the organisation does not already have a relationship and that often treats the work as a black box delivery where they only provide the answer but not the workings, so they engage us.

We have just finished one of these Research Spikes.

The organisation's data team is building a new greenfields data platform to replace their legacy data warehouse.

The legacy data warehouse currently provides over 1,000 separate reports all built in a legacy BI tool.  The reports and data warehouse all have variable levels of documentation.  

The team knows that a lot of the reports are slight variations on each other, where they have been copied and a filter added, or a different time period hard coded.

But the team is stuck with manually reviewing and comparing each report to see what it does, if it can be consolidated or decommissioned.

The research spike we did was to answer this question:

“Can we use a LLM to compare exported report definitions to reduce the manual comparison work required?”

We received a subset of the report defintions as XML.  We received a cluster where the data team had already reviewed them and they were slight variations on each other.  We received a second cluster that was based on a seperate set of core business processes.

We then used Google Gemini to compare the report definitions.

The answer to the research question was yes, the LLM approach could reduce the manual work required but could not automate it fully.

When we asked the LLM to merge some of the XML report definitions it politely responded with:

“I can't do that. The reports are complex XML documents that define the data source, layout, and other properties of a report. Simply combining three separate report XMLs will result in an invalid structure.”

There must be a better way to automate this data discovery work.

Follow #AgileDataDiscover if you want to follow along as we build in public.

2024/06/08

Some background on what led us to the decision to do this 30 day bet on #AgileDataDiscover

#2 From Data Governance to Data Enablement

As we are bootstrapping AgileData, every now and again I do a side hustle consulting gig to refill the AgileData funding / runway bucket.

I did a side hustle earlier this year that involved a lot of collaboration with an enterprise organisation’s Data Governance team, well they were called a team by name, but in reality it was a team of one. That one person was frickin awesome, but that is a different story for a different time.

The organisation was migrating from a legacy Data Warehouse to a “Modern” Data Platform.

The Data Governance lead was attempting to enable this to happen as quickly as possible, while putting in place the data Principles, Policies and Patterns that were lacking when the legacy Data Warehouse was created.

As you can imagine being a team of one meant that they could not actually do a lot of the work themselves, and by having no team members they had to ask for other data teams to help with the work. Those data teams were head down, bum up trying to rebuild everything on the new platform so the old one could be decommissioned. They didn’t have a lot of time to put into the Governance data work, and they treated it as not being their work to be done, but work the Data Governance team should do.

We decided to flip the model and move from being a Data Governance team of one trying to get the work done, to being a Data Enabler of one, helping the other data teams collaborate together to share and leverage the data Principles, Policies and Patterns they had each created independently.

We also focussed on helping to unblock those teams when they were stuck with something outside their control and also started a series of capabilities to increase the data fluency across the entire organisation, again to improve the ability for the data teams to have conversations with their stakeholders.

One area that was a real blocker for the data teams was the ability to document the current state of the Data Warehouse, especially based on a pattern of Information Products, identifying what was already available, what was being used, what was adding business value, what could be decommissioned and the effort / feasibility to rebuild those that had ongoing value.

As you would expect, the organisation tried to implement a data catalog solution, which had failed to produce these useful outputs.

The typical pattern of getting five business analysts to spend six months reverse engineering the legacy platform and engaging with business stakeholders to discover this information wasn’t a viable option.

There must be a better way to automate this data discovery work.

Follow #AgileDataDiscover if you want to follow along as we build in public.

2024/06/07

Some background on what led us to the decision to do this 30 day bet on #AgileDataDiscover

#1 Legacy Data Platform Migration to AgileData

One of the blockers our #AgileDataNetwork partners hit when talking to organisations about becoming their Fractional Data Team, is the theory of sunk costs.

The organisation may have already invested in a data platform and maybe a permanent data team to build that platform and deliver data assets.  Or they may have already paid a data consultancy to build a new modern data platform for them, and then continue to pay the consultancy for the ongoing data work.

Even when our partners offer to reduce the organisations annual data costs from $600k a year to $60k, the theory of sunk costs still seems to remain.

One of the barriers is the cost / time to rebuild the current data assets and Information Products in the AgileData Platform.

So we have always been keen on finding a way to remove or markedly reduce the time and that cost for that migration data work.

The way we think about it is this:

How do we automate that migration so it is automagic, so that it removes all human effort to do that data work.

We know that 100% automation is probably not achievable but it’s the vision we wanted to strive for.

In reality we would be happy with 80% of it being automated and human effort for the last 20%.

The obvious answer is we just migrate the organisation’s code and we run it on our platform.

The problem with this pattern is we lose all the secret sauce of what we have built in AgileData that allows our partners to reduce an organisation's data cost from $600k to $60k.

So we need to automate the process of discovering the core data patterns in the organisation's current data platform, map those to the core data patterns we have built into AgileData, then automagically generate the AgileData config to apply those patterns.

There must be a way to automate this data work.

2024/06/06

We are working on something new at AgileData, follow us as we build it in public

As part of our constant experimentation we have stumbled on an interesting use case in the data domain.  

It's a data use case that has got Nigel Vining and myself so excited that we have decided to dedicate the next 30 days to experimenting with it.

It might turn out to be nothing valuable.

It might turn out to be a set of features in AgileData, or a new module.

It might turn out to be a completely new product.

We have no idea, but we are interested enough in it to bet 30 days effort to find out the answer.

As part of this we are also going to completely change the way we do our product build, for this idea we are going to change the way we work.

As part of this process we are also going to post daily as we decide and execute each step of the product process.

If you want to follow along as we build in public, checkout this post on a regular basis, we will add each days update to it so you can watch as we build and learn.

Time for an upgrade

As the AgileData journey continues, we find ourselves facing an interesting challenge – an email from Google urging us to upgrade our Python version. With the current version’s support ending next year, it’s time to embrace the latest and greatest, …. yeah, upgrade time !

If you’ve been following our previous posts, you know the AgileData Platform is built on the Google stack, relying heavily on Google Cloud Spanner as our backend config database and Google App Engine / Python for our API layer, Web-Sockets Service and AgileData App.

Cloud Functions serve as the glue holding everything together, providing a pay-as-you-go and scalable and cost-effective solution. We love this Functions as a Service (FaaS) pattern, it lets us run the AgileData App and API’s for a fraction of the cost of using dedicated servers or containers.

However, with the Python upgrade on the horizon, it’s time to ensure that our system remains robust and reliable.

Upgrading Python: The Plumbing Essentials

The AgileData Platforms backbone is built on Python 3.7, but it was time to transition to the newer Python 3.11 version.

The process is not without its challenges, as we must ensure that all our client libraries and dependencies are compatible with the latest Python version.

One critical aspect we knew we needed to address is a change in the client library’s handling of JSON data, which could potentially break one of our core functions.

So we decided we needed to bring the spanner library up to the latest version at the same time we did the python upgrade. 

# before the python upgradegoogle-cloud-spanner==3.11.1

# after the after upgradegoogle-cloud-spanner==3.36.0

The Query Spanner Function: Unveiling the Core

One of the most crucial functions in our backend is query_spanner(). It allows us to query data from Spanner using a simple SQL query and returns a JSON result payload.

This function serves as a pivotal connector, providing access to our data and ensuring smooth operations. 

query = “select table_id,topics,state from catalog where project_id = @project_id”results = query_spanner(query)

[    {         “table_id”: “consume.cars”,         “topics”: [         “Cars”,         “Vehicles”,         ],          “state”: “active”    }]

# ========================== query_spanner ========================

def query_spanner(query):

     “””    Simple function that takes a sql query and executes it     “””

     schema = []

     output = []

     # remove line breaks (we may have formatted for readability)

     query = query.replace(“\n”, ” “)

     try:

          with spanner_database.snapshot() as snapshot:

               results = snapshot.execute_sql(                    query,                   params={“project_id”: project_id},                   param_types={“project_id”: spanner.param_types.STRING},              )

               json_columns = []

               for row in results:

                    # first read – pickup the schema for the results

                    if not schema:

                         for field in results.fields:

                                   schema.append(field.name)

                                   # create a list of json columns we need to parse

                                   if “JSON” in str(field.type_):

                                            json_columns.append(field.name)

                    # combine the two lists (schema and result row)

                    output.append(dict(zip(schema, row)))

     # load json payloads

     if json_columns:

          for rec in output:

               for json_column in json_columns:

                    json_payload = rec.get(json_column)

                    if isinstance(json_payload, JsonObject):

                         # Updated 15.07.2023                         # Spanner library now returns json columns as a JsonObject

                         json_string = json_payload.serialize()                         rec[json_column] = json.loads(json_string)

                    else:

                         rec[json_column] = []

except Exception as e:

     logging.error(query, e)

return output

Adapting to the Change

Fortunately, the necessary code changes are minimal. The only modification required is to check if we are reading a JSON object from Spanner and then serialise it before passing it through json.loads() for our response payload. A smooth adjustment that keeps our plumbing intact!

Deployment and Beyond

With the code tested and validated, it’s time to deploy the upgrades.

For App Engine, a simple update to the app.yaml file, specifying the latest Python version, does the trick. 

runtime: python311

Likewise, for Cloud Functions, we update the deployment template, ensuring the right Python version is set for each service. 

gcloud functions deploy bigquery_job_completed –runtime=python311 —source=cloud_functions –trigger-topic=biquery_job_completed –memory=512MB —timeout=540s –entry-point=bigquery_job_completed

Our system is now upgraded and ready to take on the future with confidence!

Finally, the critical service we NEVER need to upgrade

Finally, there’s one service that stands tall, immune to the need for upgrades, the databases in our Customers private tenancies!

We rely on the mighty Google BigQuery, a server-less and cost-effective enterprise cloud analytics database that effortlessly scales with our customers data.

Put the data in, query the data out – it’s as simple as that.

No upgrading, no migrations, no worrying about disk space or temp space or log space or memory or release version or any of the hundred and one things my DBA friends spend all their time worrying about, BigQuery proves to be the indispensable backbone of our plumbing infrastructure.

Why wouldn’t you use a service like that ?

Flowing Ahead with Upgraded Plumbing

By adapting our code, upgrading the necessary libraries, and deploying with precision, we ensure that our customers data continues to flow seamlessly.

Leveraging the power of Google’s infrastructure, we maintain our momentum, confident that our data plumbing will stand the test of time.

And as we look to the future, we rest easy knowing that BigQuery will always have our back, ensuring a smooth and cost-effective journey through the data waters.

Onward and upward!