Raw capture — raw/beyond-the-warehouse-data-stacks-that-actually-work.md
url: https://gist.github.com/4213ae0471f35801e28267cbf058c76f date_fetched: 2026-09-13
You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Beyond the Warehouse: Data Stacks That Actually Work — Thomas in 't Veld (Tasman Analytics)
The 85% failure statistic is a hook, not the story. Citing VentureBeat's "85% of data projects fail," in 't Veld diagnoses three recurring failure modes from ~60–70 client engagements over six years: over-designing the stack, neglecting data modeling, and failing at activation. His three takeaways: prioritize stack choices by business value; don't underestimate data modeling; build semantic layers because they're "critical to do anything with AI in analytics."
An analytics stack is not a production engineering stack. Production systems can tolerate duplicate events (better to send a pro-subscription event twice than miss it); analytics systems can't, because duplicates double-count revenue. Event streams are not logs — logs absorb duplication and unstructured data; analytics pipelines must be unique, complete, and objective.
Prioritize ingestion by business output, not completeness. List the insights stakeholders need (campaign performance, revenue reporting, A/B analysis, user journeys), rank them, and load sources in that order. "In most subscription startups... revenue reporting trumps everything else" — so RevenueCat subscription data goes in first. Otherwise you spend three months building ingestion before thinking about use cases.
Shift left: fix data at source. Raw third-party data is messy and "probably wrong when you load it." Understand Facebook's pivots and schemas, app store limitations, attribution quirks — before building models on top. Don't try to repair source problems in the data model.
The domain model is the talk's centerpiece: a "narrow waist" and a static target. Query-driven development — bolting on a new query each time a business question appears — is "the most surefire way to get your data stack and your data team into trouble." Instead, recognize the business is a relatively static target (users, transactions, app events, campaigns persist through growth), and model entities once in the middle of the stack. He combines Inmon (entity modeling in the warehouse) with Kimball (dimensional/one-big-table modeling in the data marts). The superpower: swap HubSpot for Salesforce and only the extraction logic changes — downstream models and dashboards survive.
Infrastructure as code, all the way down. GUI-heavy modern data tools are his stated frustration. Data engineering is "25, 30% actually building the pipelines and it's 70, 75% building the testing suite," run through CI/CD on every change.
Activation is the #1 problem even in successful builds. Companies spend heavily on stacks and consulting, "and then still end up with the marketing team downloading raw data into Excel to do their own pivots." His four pillars: presentational models linked to decisions (not teams); very short feedback loops; treating every insight as a data product with scope and acceptance criteria; and internal analytics on your own dashboards ("is my CEO actually looking at it every single day. And the answer is probably no").
The semantic layer: defined once, used everywhere. It carries not just metrics but entities, dimensions, and join rules, is version-controlled, and is enforced in CI/CD — "the only possible way that you can get safe automation and... safe AI agentic applications." Three build paths: homegrown text files, dbt's semantic layer, or Omni. In Q&A he added pragmatism over purity (Databricks' native metrics are fine), and: "If you end up with two different types of semantic layers, you've done it wrong."
Security through modeling. The main cyber risk is querying raw data — it exposes PII and opens the door to prompt injection and SQL injection. A well-designed data model that surfaces only safe, business-actionable data is the security control.
"The world is littered with the corpses of companies that did everything right. They spent €200,000 on the best possible data stack. Still couldn't make a good decision." — His framing of why tool choice alone doesn't save you.
"Your data stack is not a production engineering stack." — The core principle behind event uniqueness and tracking plans.
"Don't treat the data source as canonical gospel. It will probably be wrong when you load it." — On why shift-left beats cleanup in the model.
Treating event streams as logs "makes data modeling really, really, really shitty down the line." — His blunt case for tracking plans.
On conference-goers proudly displaying sprawling dbt DAGs: "Turns out people are really proud of this stuff... I get the hair on my arms starting to stand up... This is not a good data model." Data modeling, he adds, "is a bit of a lost art" — more preached than practiced.
"At that point you're just building vanity projects rather than something that helps your business forward." — On building dashboards or agents in a vacuum.
"I've never met a stakeholder who's able to describe in detail exactly the type of dashboard reporting that they need from scratch." — Why a dashboard is the start of a discussion, not an endpoint.
"If you end up with two different types of semantic layers, you've done it wrong." — His zero-tolerance line on metric inconsistency.
On legacy, inconsistent event data: "Burn it all, start from scratch... burn it with petrol and fire and just move on." People "overestimate the importance of old event data" — it's written in stone and can't be regenerated anyway.
"Don't use MySQL is probably my main recommendation there." — His database advice for a non-data person: just put your credit card into Databricks or Snowflake.
"The worst thing you can do is try to do Agile while still trying to pre plan everything six months ahead of time." — The only certainty he claims about methodology.
"Keep receipts." — Keep copies of the emails inviting stakeholders to definition sessions, so you can point to when they had their chance to define the logic.
"Always assume your stakeholder does not know the data as well as you do." — And if they still won't use the dashboard: "maybe they're not as data driven as they want to be, which is not your problem, it's their problem."
Tools
dlt (DLTHub) — Python-based ingestion-as-code for third-party data. Testable, configurable, integrated into CI/CD so you build "additive functionality rather than accidentally messing something up." A "massive difference from the old world of ingestion tools like Stitch or even Fivetran."
Rudderstack / Snowplow — Event data pipelines with strong auditing of the event architecture; both have built-in tracking-plan support.
Avo — Tracking-plan tool (Icelandic team) for event governance: consensus, scalability, versioning, design-first. Build plans in a UI, collaborate with product/engineering/data teams, version-control and document every event spec, then feed it into your systems. Ensures an iOS button click generates the same event as an Android one.
RevenueCat — Subscription data platform for apps. "If you're not using it to use subscriptions, go use RevenueCat." In subscription startups its data is the priority source because revenue reporting trumps everything.
Databricks / Snowflake (vs. BigQuery, Redshift, MotherDuck) — Cloud warehouses, "pretty equivalent" in functionality; differentiated in unstructured data and AI/ML support. Their access-permission structures are "probably their biggest moat" versus newcomers. For beginners: "just put your credit card into Databricks or Snowflake."
Prefect — Orchestration.
dbt — Transformation, used "almost exclusively still" — though he's "looking at alternatives" given how dbt is evolving. The dbt semantic layer (cloud/enterprise plans only) exposes metrics as a robust, performant API for building applications at scale.
Omni — BI built by the old Looker team after Google's acquisition; metrics-first, infrastructure-as-code everywhere, version-controlled, with a good browser interface. At small fast-growing teams it doubles as the semantic layer with no extra subscription. Prefers dimensional models.
Metabase, Hex, notebooks — Consumption tools for different use cases, all drinking from the same source of truth.
Looker — Pioneered the YAML-configured semantic layer 5–8 years ago; integrates with Snowflake RBAC so C-level-only reports enforce warehouse-level access too.
Sigma — BI tool more comfortable with one-big-table models than dimensional ones — pick modeling style to match consumption patterns.
Elementary (plus a second testing tool, garbled in the transcript) — dbt-native testing/monitoring to keep the stack accurate.
Homegrown semantic layer — Literally "a bunch of text files" specifying dimensions, base queries, key join rules, and canonical metric definitions, fed into an agentic engine. Got him "correct SQL consistent every single time." Integrates with agents but not easily with BI tools.
Practices
Shift left — Fix data issues at source, not in the data model; also applies to data retention, which he says should be owned by data engineering.
Business analysis via Scrum — Sprint 1 analyzes tasks, sprint 2 delivers; shared roadmap and backlog. Prioritization of data sources comes from ranking the insights stakeholders need.
Domain model as a paper exercise — Half a day, stakeholders in one room, define central logic "once and for all": what is a user, does a user need an ID or just an email. Do it before writing anything.
The narrow waist — Entity modeling in the warehouse (users, transactions, subscriptions), fanning out into dimensional or one-big-table data marts for consumption. Vendor-proof and query-change-proof.
Data products, not dashboards — Apply product thinking (scope, acceptance criteria, who/what/why) to every piece of insight, documented up front.
Growth loops / internal analytics — Measure dashboard usage via query logs (Snowflake, Databricks, MotherDuck) and BI usage analytics; learn, adjust, iterate — weekly.
Buy tools, build the glue — Never build what's off-the-shelf (e.g., text-to-SQL engines); do build your own CI/CD layer and tests. Enterprise with extreme event volumes may be the exception.
Never let agents query raw data — The data model is the security boundary against PII exposure, prompt injection, and SQL injection.
Data mesh — Name-dropped as the philosophy for aligning metric definitions across multiple data teams in large orgs; details deferred to a hallway conversation.
The 85% stat is never interrogated. It's the hook, but he offers no evidence his approach moves the number — no failure post-mortems, no before/after from his own 60–70 clients, no success metrics for a stack that "actually works" beyond dashboard-usage analytics.
Enterprise is waved off. "Enterprise, very different beast altogether" — and when data mesh came up as the multi-team answer, he deflected: "Come talk to me afterwards. Long discussion." The hardest organizational problem in the talk is a hallway conversation.
The dbt elephant in the room. He name-checks the dbt–Fivetran merger, says he's "looking at alternatives... given how DBT as a tool is evolving," and never says what's wrong with dbt or what the alternatives are. Meanwhile the dbt semantic layer sits behind paid cloud/enterprise plans — flagged as "sometimes a blocker" for exactly the startups he serves, and left unresolved.
What happens when the business does change fundamentally? The whole architecture rests on "as long as your business doesn't change fundamentally" — pivots, M&A, new product lines are unaddressed. And there's an unresolved tension with his own claim that "you almost never have the time to try and go back and fix a broken data model," versus his later "the best time to fix it is a year ago and the second best time is today."
Thin AI content for an AI & Data Summit. MCP is dismissed in half a sentence ("That's not just building an MCP server"), agents appear mainly as a consumption pattern for the semantic layer, and there's nothing on agent evaluation, hallucination risk, or how to actually operationalize agentic analytics beyond a reference to yesterday's demo.
The Power BI Copilot question was dodged. Asked how to harmonize Databricks and Power BI Copilot, he answered about vendor lock-in and "SQL is a commodity language" — never the harmonization steps.
Definition politics are sidestepped. "Have a marketing lead and a sales qualified lead if you need to" is a workaround, not an answer to who owns the semantic layer when stakeholders genuinely disagree.
"Burn it with petrol" vs. retention obligations. Telling teams to torch untrustworthy event data sits awkwardly next to his own point that financial data must be kept 10–15 years — and he admits the audit-and-repair path is "a very expensive migration job, which actually we don't really like to do. Come talk to us about anything else, but not that part."
The "people problem" never arrives. He opens by saying scaling data teams is "primarily a people problem" and promises his "one opinion" on running data teams — the talk then stays almost entirely technical, and the opinion is never really delivered.
Missing topics you'd expect: data contracts, streaming/real-time requirements, unstructured-data modeling for AI workloads (despite citing it as the warehouses' differentiator), and any depth on data quality tooling beyond a passing mention.
Speaker A: So our next speaker, who is Thomas Intweld and he is, as you can hear from the name, he is a guy who was born in Belgium, lives in the Netherlands, close to Amsterdam, in Haarlem, and he also worked a lot in London. So this was the idea which brought together his company Tasman analytics and he's going to talk about from dashboards to decisions, how building agenting marketing analytics with Gen AI helps us. No, that was the previous one. Okay, so beyond warehouse, these data stacks that actually work. So that, that is the one which is coming up and Thomas is going to talk about that deeply because his team, which is based in the Netherlands and in London, working together, provides insights for their clients. This way. So let us welcome Thomas.
Speaker B: Hello, good morning again. For a minute I thought, oh damn, I have the wrong presentation for this morning. Let's talk about warehouses and data stacks. I hope all of you had a great first day yesterday. Really good talks, really nice to be back here to talk about the more technical part of the, of the data stack. So it's good to be here yesterday. For those of you who were here yesterday, we kind of looked at agentic implementations. Now failing data projects is nothing new and this statistic is about 2 years old, but it's still quite relevant I think from VentureBeats as 85% of data projects fail one way or the other. Methodology issues, I'm sure, but I'm just trying to use it as a really good hook to try and understand a little bit why that's the case. And then we go technically into the data stack and see what the best way is to get your data stack built in such a way that it doesn't fail and that your data projects generate all the insights and business outcomes that make everybody happy. And I've got a working clicker today as well, which is great. So this is the sort of data stack we'll be discussing. We go sort of silo by silo by silo, from ingestion to trans to sort of staging to data modeling, monitoring and QA to activation, single source of truth, semantic layer and then how to consume your data from there. Of course. There we go. I was afraid for one second we'll go through the sort of simplified version of this. Do it, do it stack by stack. Quick recap, because you see a lot of very interesting speakers yesterday already. My own background and where it comes from. So I run Tasman. This is us, as I said yesterday, we run data analytics and insights for fast growing startups and scale ups. That's Important, because I'm going to tell you about what I know from experience. Works for scale ups, right? Enterprise, very different beast altogether. Same principles might apply, but my experience is not as large in enterprise organizations, but we have done this for about 60, 70 clients over the last six years. And that means that hopefully we got some lessons, mistakes, pitfalls that we can tell you about. All right, so for whom is this talk? And it's quite interesting to sort of see who identifies with what position in data. There's still broadly three big roles, right? There's the data engineering role, there's the analytics engineering role, and there is the data analyst role. So who here raise your hand if you think of yourself as a data engineer. All right, analytics engineer. Okay. And then data analyst. All right, so we're quite an engineering heavy crowd. Just out of interest, who here thinks of themselves as a software engineer instead? Great, fantastic. Hardware engineers. No, hardware engineers. You're at the wrong conference, but that's great. This talk is a bit more for analytics engineering. So what does that mean, data engineering? And particularly, I think if you come from a software background, we'll talk a little bit about infrastructure as code because it's one of the things where we can start shifting left a little bit into software engineering principles with our analytics platforms. We're not really diving into data analysis yet either. We talked a bit about that yesterday, how we could usentix systems for that. There's plenty more that we can do when it comes to data analysis. But not today. Today we're going to talk about the center part of the stack, which is sort of for us, the analytics engineer responsibilities. All right, so there's sort of three big trends that are still very true to this point. And I've talked about them a couple of times over the last couple of years. So this might be a repetition for some of you, but there's three big things that I think are very true still. First of all, startup analytics tools have incredibly evolved incredibly fast over the last 10 years. But the last, say two or three years, they've matured. They're now consolidating, whether that's DBT and 5J merging or other sort of things that are happening on Horizon. And of course they're getting ready for the AI revolution. That's not just building an MCP server. That's also about making sure data quality is of primary importance. Second, starting and scaling data teams in new organizations or scale ups that all of a sudden have a need for a data team. That's still really a complex problem and primarily a people problem. So let's hope to give you some insights there as well, how to solve. And there are still many different ways to run a data team. I have got one opinion. I'm sure all of you running within data teams have got slightly different versions of that, so let's use it as an opportunity to have a bit of a discussion about that. The three takeaways that I need you to take home with you today are first of all, your data stack choices. You need to prioritize those based on business value critical parts, as I call them. Second, that you need to not underestimate the importance of data data modeling in your stack. There's ways of doing this properly. Three, that semantic layers are critical to maintain a source of truth through all of your data consumption patterns. And this is where the talk today links up with what we talked about yesterday. Semantic layers are critical to do anything with AI in analytics. So we'll, we'll learn a bit about these three takeaways. We'll get back to those at the end. Startups. And again, this is my comfort zone, right. Actually, who here works for a store would consider themselves working for a startup or a scale up? Raise your hand. All right, big company, enterprise. All right, whew. You might recognize the problems but you're sort of more empathetic maybe, hopefully to them. All right, so first of all, three problems. Complexity of landscape is changing. There's a huge amount of choice. How do you pick the best data stack for what you need to do? Second, rapid growth of startup scale ups hopefully means that requirements are dynamic. You're building a platform today and in six months time the questions from your business might be completely different. So how do you build for that? How do you design for that? Third, it's easy to spend a lot on data. The world is littered with the corpses of companies that did everything right. They spent €200,000 on the best possible data stack. Still couldn't make a good decision. So external insight, good decision making is very hard to do. We're going to go through the mistakes. Three big mistakes. The first one is over designing your data stack. Big, big, big pitfall. I've got a scar tissue for that as well. Takes you six months to sort of build a perfect data stack and then that's it. Now this is quite a natural mistake to first make when you're starting up a new team or when you're joining a new team for the first time and sort of trying to get your data stack up and running. This is Just a blueprint of the data, the modern data landscape that a 16Z built about a year and a half, two years ago. It is complex. They added on top of that with of course the sort of merging large language model stack which is even more complex. And the idea here is that it's just too much choice. If you don't have the experience of knowing what works for you and what you need to build your data platforms, it's very hard to know exactly what stack you need to prioritize and how to go from there. So what I'll do is I'll just quickly take you through the type of data stack that we'd like to build for fast growing clients. And just as a reminder, what we do is we come into a fast growing scale up that is going from intuition driven decision making to now realizing that in order to maintain scaling they need to make their decision making a bit more sustainable. So they realize they need a good data team and a good data stack to do that. And then we build them as lean a stack as possible. And what it looks like typically is something like this here. So we forgot the sources typically for startup or scale up. That is a range of user generated event streams and third party data like meta advertising data, Snapchat Data, maybe some HubSpot Salesforce CRM data as well. Ingestion typically happens on the event side in tools like Rudderstack or Snowplow that are really good event data pipelines with strong auditing of the event architecture beneath them. We'll talk about that. We are massive fans of DLTHub and using that in our third party data ingestion. We'll get back to that in a second. AVO and Hydraulic I'll talk about later. The warehouses were quite agnostic, but we really like working with databricks on modern at this point. Snowflake of course as well. There's some bigquery redshift projects still hanging about in the background there, orchestrated with tools like Prefect and Transformation happening almost exclusively still in DBT though we are for understanding understandable reasons given how DBT as a tool is evolving, we're looking at alternatives as well. Just to make sure that we get some experience under our belt there the analysis and output. Again, less about the analysis today, but tools like Omni, Metabase, Hex or just Deep notebooks. Very different use cases, very different tools, but all consuming from the same source of truth. All right, and then don't forget about the testing Cat or elementary kind of nice tools to look at over there. So if you then look at the typical data stack and how to differentiate there and how to make all of this work together. So I've just taken you through the tools that we choose, but maybe just to give you some inspiration, all the parts here. First of all, the sort of ingestion part of the pipeline which is downloading into the first layer of your data warehouse. Just two bits here. One is your data stack is not a production engineering stack. And this is a really important principle to take into account because a production engineering stack will optimize for different set of principles than an analytics stack. Simple example, if your production data, if your production stack that's optimized for understanding which of your users is a paying pro subscriber, you might err on the side of sending a pro subscription event a couple of times just to make sure that the pipeline is recognizing it and offering the pro subscription environment to those users. You don't want to do that in analytics stack because then all of a sudden you're starting to double count revenue. So the sort of, how should I say, uniqueness of events are more important. Capturing 100% of events because you don't have any space for sending events two or three times because they're lost forever if you don't send them the first time around. Different principles at stake here, the source and the staging layer. This is also where infrastructure as code comes in as a really important requirement. And luckily most of you are engineers one way or the other. So you understand my frustration when I talk about a lot of the modern data tools being quite GUI heavy, meaning that there's a lot of buttons to click and not that much infrastructure as code principles applying there. Whereas what we want to do is we want to go in and in our own repository, so we have a Tasman kit that we deploy to every client. We're able to just build an entire pipeline as a virtual version of as a set of scripts that just run a bunch of things straight away and are easily deployable, easily templatable, and easily testable as well. And that is the type of additive data engineering that we really like. The last things to test. Other examples of where infrastructure is code becomes really important is dlt, as you just heard from the last speaker, a really important way of making sure that your data ingestion pipelines are testable, easily configurable, integrated in your CI CD pipeline. So you can make sure that you're always building additive functionality rather than all of a sudden accidentally messing something up. A massive difference from the old world of ingestion tools like Stytch or even fivetran in many ways as well. Come talk to me afterwards if you disagree. And then of course, infrastructure as code also means that when you're building this into your CI CD systems, you can make sure that you just run your entire suite of tests every single time. So for US data engineering, it's 25, 30% actually building the pipelines and it's 70, 75% building the testing suite to make sure that you get that you kind of retain your excellent state of data accuracy and stability in there as well. All right, the middle part of your data stack, this is where the data modeling comes in. And I want to talk a bit about data modeling in a second, but consider this to be massively under appreciated part of how important it is to build a good, scalable, accurate, not failing data stack. And probably the main reason why even in cases where people are kind of sort of, they just send me a message that I can keep talking apparently on the screen over here. So I got momentarily distracted. Even in cases where you build a very, very, very good pipeline, if you do not take your data modeling steps seriously, you're really going to get failing results and failing insights at the end of that. And then the last step of course, is when you've got everything surfaced in a presentational layer which is your source of truth, then you can make sure it gets consumed by your semantic layer for your dashboards, your self serve or your agents. As we know that the same data gets consumed by any deep analysis you're doing or any ETL that feeding back into HubSpot, Salesforce and so on and so on. All right, so when you look at all of these data sources that you are connecting into your big nice new data stack that's loading everything into databricks. If you are a scaler, one of the biggest constraints you have is time. And one of the things that means that you need to prioritize which of your data sources you're going to load first. And that's instructive, as in obviously you're going to try and get all of it in there straight away, but if you could try to sort of. And typically in our projects we have four or five months to work with our clients to get their data stack set up and running. If we do everything all at once, we're going to spend first three months building the ingestion layer and only then thinking about the use cases. So for us we have to be able to prioritize our data sources. Properly and maybe there's some useful tidbits for you there as well. So say you're in a E commerce or sorry, say you're in a subscription app startup and a typical set of data sources you have. There are things like your Facebook ad spend, data app events, how users navigate through the app, how they install it, and so on and so on. Subscription data from revenuecat for instance, which if you're not using it to use subscriptions, go use revenuecat. It's a great tool, attribution information to understand how people install the app and where they come from. And then an AB test engine because you got good product managers who want to test hypotheses and so on and so on. So if those are your data sources, how are you going to prioritize as a data team what you're going to load first? Well, the answer is always in function of the output, reporting and insight that you need to provide. So the first thing that you have to do, even if you're a data engineer, because you're going to make the prioritization decisions, typically is to list up the different types of insights that your business stakeholders need, whether that's campaign performance reporting, monthly revenue code reporting, a B, test analysis, user journeys and start ranking them. And in most subscription startups, at least in my experience, revenue reporting trumps everything else. And that means that in this case the revenue cat subscription reporting becomes the most important data source straight away. There is a lot of mechanics around talking to business stakeholders to try and understand what they actually want. We use quite an agile process for that, which we call business analysis, which for all those of you that use Scrum in any way whatsoever, you know the drill of in Sprint one, trying to sort of analyze all the tasks if you want to realize in Sprint two, and then having a nice shared roadmap and backlog to work on as well. So that's how you prioritize your data sources and go from there. Now, if you load your data sources, one of the things you're going to find out really quickly if it's not your own data, is that it is really messy if you just load the raw data straight away. So how do you do your data cleaning? Let's take the same mobile subscription company that we talked about just now, look at the data sources and the reporting that you have on the right side. These five data sources over here, how do you make sure that they're accurate and that you can actually understand what's going on there? And the answer there is to really use A sort of methodology that we call shift left, which means that rather than try and find fix any issues in these data sets in your data model, if at all possible, you got to fix this at source. Sometimes it's a bit easier said than done, but for instance, for Facebook, you got to understand the different pivots. You got to understand the schemas that are available there and how they work and why. The splits between geos or mobile device types is quite a different beast in Facebook than it is in say, Snapchat data. At recent information, you got to understand your schemas as well and your available pivots wherever you get data. For instance, I'm just giving examples here of the considerations of different sources. You need to understand the app store limitations, which of course has equivalent functionality or equivalent considerations in Shopify E commerce data. Trying to match up anonymous purchases with identified purchases, for instance. And then a B test engines. They're so different that I can't even start anywhere with with how all of that works. The point I'm making is data sources are really complex. Don't treat the data source as canonical gospel. It will probably be wrong when you load it. There's probably a lot of testing or edge case data and that you don't understand. So do your business and technical analysis first before you start building data models on top of this. All right, AppEvents, this is a different beast. Who here manages sort of an app events type pipeline or something in Snow Plow? Okay, raise your hands. Yeah, there's not that much enthusiasm. Okay, so I'm gonna hopefully help you out a little bit over here. App events, really the two bits I want you to think about here are around tracking plans and around objective data. So now tracking plans have become a lot more popular last couple of years, luckily. But the point is that if you're sending an event stream from your mobile app or your website or backend system that you have, don't treat this as a log because as I said earlier, a log is a production engineering system. An analytics system works differently. A log can deal with duplication, A log can deal with unstructured data in there and so on and so on. You do not want any of that in an event stream that you're sending into your warehouse because it makes data modeling really, really, really shitty down the line. So what you do instead is you got to think about the type of events that you that you're sending into your systems. We really like tracking plans. They keep the data quite objective and also quite easily comparable between iOS apps and Android apps, making sure that when you have a button click event on iOS, that it's the same type of generation of event as a button click in Android and the tool we typically use for that. And there's a few different tools out there. Rudderstack has one built in, for instance, Snowplow has one built in as well. But with the sort of criticality of event governance that we have around consensus, scalability, versioning and design. First, we really like a tool called Avvo. So avo, there's a bunch of Icelandic people who have been building this over the last four or five years and it allows you to build tracking plans in a really nice ui, collaborate with your product teams, your engineering teams, your data analysis teams too. For every event that you're designing, you've got a nice spec that you can agree on, version control nicely, document nicely, and feed it into your systems. We really like the way of thinking. It really works well. It's also a really good way to start identifying issues. So here, think about this as infrastructure as code. Again, have your testing framework so that when you are talking about event analytics, that you've got test cases that you can run in your data models, for instance, to make sure that your screen view events are coming through correctly or that your purchase events are coming through correctly as well. All right, so the tip number one that I have is that business analysis, the sort of understanding your business, understanding how your data is generated and what the requirements are, that leads to a leaner, better data stack. Very useful for us at the very least. Pitfall number two is when you actually have to start modeling the data. The sort of things that I want you to understand here is that particularly when you build AI, as you saw yesterday, but also when it's just plain reporting, you need the single source of truth there. You want to make sure that when you're building a report for marketing, that it uses the same definition for lead as when you're building a report for sales. This may be a bit more of an analysis problem, but I can guarantee you that it starts in the data engineering layer. The second reflection is that I'm afraid it is a bit of a lost art. And I've been saying this for a few years, it's still a lost art, alas, it is more preached and practiced. So let's tell you a bit more about data modeling and how we, how we run this. How do you build a data model in a business that is changing, in a business that has dynamic business requirements? So this is a Typical data model, you got business questions, you start to answer them in your data model based on the available sources. And typically what happens is that when there's a new business question that arises, you sort of say, okay, okay, now I need some finance data. So you connect into your finance data using DLT Hub and you just extend the data model with like one more query in there to kind of have it all work out. Now this is called query driven development. And it is, it is probably the most surefire way to get your data stack and your data team into trouble quite quickly down the line. Maybe not immediately, but for sure. 6 months, 12 months, 18 months down the line. And if you're working in the same sort of companies that I work in, you almost never have the time to try and go back and fix a broken data model or a data model that's grown organically. So let's help you fix all of this creative development bad thing. Avoid as much as possible. The sad thing is that I go to DBT conferences or data modeling conferences or whatever and I see people showing their dags. So I think most of you will know what the DAG looks like. This is essentially representation of all the tables that someone has built in dbt. This is a real one. Turns out people are really proud of this stuff. And I look at this and I get the hair on my arms is starting to stand up a little bit on this. This is not a good data model. Okay folks, let's talk about what a good data model looks like instead. You got to realize that the business that you're in, even if it's a fast scaling startup, it's still a relatively static target. Meaning that your business right now and in six months time and in 12 months time will have users transactions for users subscribing to your pro environment. It will have app events for installing onboarding gameplays, whatever it is, it will have a marketing set of campaigns, marketing, ad set, marketing creatives. And it all fits together quite nicely in the same way, even if you launch into a new Geo or even if you add a few new games into your gaming suite in a mobile app. So why is it then that we're building our data models not as a static target, but as a set of expanding queries to answer business questions. And that's because we're not really thinking through making the data model a static target as well. So what we want to do is want to make sure that the data model is the statistics that the data model doesn't change. As long as your Business doesn't change fundamentally right now. Luckily, software engineering has figured this out 30 years ago as most difficult things in data. So thank you software engineers. And there's a way to start modeling your data inside of your data warehouse to make the static target and we call it the domain model. You can call it entity modeling, you can call it object different modeling, whatever carries your fancy. But the problems that we're describing here, they're not new. And I don't know if you're a data modeling sort of veteran. You might know these two guys, Bill Inman and Ralph Kimball, because both of them have what people consider to be conflicting methodologies around data modeling. Mr. Kimball, Mr. Inman had an idea of modeling the entire data warehouse as entities where you model users, where you model transactions, where he models subscriptions and make those static targets. But then that turns out to be a very expensive operation in the old data warehouse world. So Ralph Kimball came along and built a star schema type approach where you build dimensional tables and factual tables that allow you to pre build for the type of reporting that you want to do in reality. As with most nice things in life, you got to combine the two. So what we do at Tasman is we think about our business use cases, our application, loading that into the data warehouse and then seeing the output of the data warehouse. The output of the data modeling is a set of data marts and in practice it means that we do entity modeling in the warehouse. This is where we define our user objects, our transaction objects, our gameplay objects, and so on and so on. And then in the data mart we do some amount of dimensional modeling or one big table type modeling to make everything ready for consumption by our BI tools or your other patterns. This entity modeling. Because I'm sure most of you are familiar with dimensional modeling one way or the other, this entity modeling, it sits within here kind of here in the middle of your data stack. So after the source and staging data has been loaded at that point we create this narrow waste in your data model that models your users, financial transactions, subscription objects and so on and so on and then fans out again in to your presentational layer things. There's a massive superpower here for scale ups. The superpower is that because you've built a static target in the middle of your data stack, if you swap out HubSpot for Salesforce in your third party data or in your CRM systems, you kind of want to avoid having to rebuild all of your dashboards that you built on the other side of your of your data Stack, Right. A static target allows you to do that because it essentially allows you to say, hey, look, I've modeled my central objects. They don't change because we're changing from HubSpot to Salesforce. So my downstream data models don't change as well, and so my downstream dashboards don't change. All I have to do when I swap out HubSpot for Salesforce is change the extraction logic and make sure that I still load into the same users and say, customer support ticket tables. All right, perfect. Let's go. I'm going to skip through this a little bit, but what I want you to understand is static targets are important in designing your data models. Scalability is important when designing your data models. And the business logic consensus of trying to align with everybody on exactly what a user is, exactly what a financial transaction is, exactly what a lead is. For instance, okay, cool. Sits in here in the data stack. And now we've gone from ingestion to data collection to data storage and transformation. And let's now talk a bit more about the last part of the data stack that's important, which is arguing stakeholders. And if you do your job really well in parts one and two of the stack, how can you avoid your stakeholders being unhappy? And the sort of, shall I say, dichotomy that you typically get here is the you always bifurcate between no one is looking at my dashboards, or on the other hand, dammit, my backlog for dashboard creation is about a mile long. It will take me a year to go through that. So what's the middle ground here? How do you deal with this? First of all, it's around inside activation that's worth talking about for a second. Activation is probably the number one problem that we see in even successful data capability builds. And activation means are people actually using the reports? There's too many businesses that spend a lot of money on data stacks, a lot of money on consulting and so on and so on, and then still end up with the marketing team downloading raw data into Excel to do their own pivots. This happens every single day, even in the most successful companies. And the reason for that is that no one has really thought about, well, what do we do to make sure the dashboards are connecting with people's day to day needs? Do not build your dashboards or your reporting in a vacuum. Do not build your agents in a vacuum either. Right, because at that point you're just building vanity projects rather than something that helps your business forward. So data needs to earn its keep and I think the sort of four pillars that we see work well for data creation and four things that I think you can try out in your own data team. One is those presentational models that I talked about. So remember the third layer of the data stack where we building the data marts on top of that or downstream from those domain models, you gotta link those into decisions. You don't want to link those necessarily into teams or into available data sets, but you gotta link those to decision making. So you'd have a presentational model for marketing campaign decisions, you'd have a presentational model for product feature design decisions, you'd have a presentational model for financial reporting and so on and so on. And that helps you design a model around how it's actually going to be used, which is typically cross functional rather than just one particular team. Second, growth loops, insights back into those growth loops. It's a fancy way of saying build very short feedback loops. So do not build a dashboard from scratch. Isolate it away from the people who are going to use it and then after two months come back and say here's your nice dashboard v1.0 and then come back six months later to see whether people are using it because they won't be using it because it needs probably two or three iterations before it happens. That means you got to measure, you got to know which dashboards are being used and how they're being used. You got to learn, adjust and iterate from there. And there's tools to do that, which I'll tell you about in a second. And then of course also avoid the idea of a dashboard as the endpoint. Think of data products instead. So for the last year and a half, two years, we've moved everything from just thinking about charts or reporting into thinking about every single piece of insight as a data product where we can apply product thinking around scope acceptance criteria, who's using this for what are they using it, why are they using it? Document and think about all of them front so that when you make your design decisions around your dashboard production or your agentic production, that you can make sure to kind of check boxes against all of those scope and acceptance criteria decision straight away. Use your internal analytics, okay? Practice what you preach almost. If you're an internal data team, you got to gather analytics on whether people are using your reporting, your dashboards. And for data models, that means having a good query log in snowflake or databricks or model duck. For dashboards it means using looker or omnium build analytics tools to try and Understand, hey, this dashboard that I built three months ago is my CEO actually looking at it every single day. And the answer is probably no. And then you have a data point you can go to them with and say, hey look, what can we do to make this more useful for you? This should probably be quite an important thing for you to do on a weekly basis to try and understand whether what you're building is being used by the business or not. All right, Last question. How do you make sure that all the tools agree? And we've been talking a bit about semantics already yesterday. I know that Adrian talked a bit about it as well. I think the main answer to making tools agree is around the central source of truth. Now we've talked about the source of truth already a little bit in that presentational layer in your data warehouse where you're building data marts that sit on top of the domain model. Thin waste. The domain model contains most of the business logic around what is revenue, how do we pre calculate a whole bunch of other things that we need to surface to the end users. But it doesn't contain all of the logic and it doesn't contain the aggregation logic. And so what has become really clear over the last few years is that once you start surfacing different types of tabular data into your data consuming products, whether that's a BI dashboard or an agent or your own notebooks, you gotta also, apart from giving the right data sets, give and agree on the logic to join data sets together to aggregate revenue in it, to to aggregate user counts on them, and so on and so on. And that's what we call the semantic layer. I'm going to take you through three ways to build semantic layers, if that's all right. So first of all, maybe the design bits that you need in order to make this work. So first of all, semantic layers should be defined once and used everywhere. So no discussion about that lead definition. If you do have a discussion, have a marketing lead and a sales qualified lead if you need to. That's one way to solve the particular problem. Center truth for all the metrics version controlled nicely. Making sure that not just metrics are defined, but also entities, dimensions, joins and so on and so on. Enforcing everything, making sure that things are not passing your CI CD testing if it doesn't meet the semantic constraints that you put on top of it. This is engineering, right? This is again about applying engineering philosophy to what used to be kind of more software data analysis, things, which is better for everybody, I think. And we think it's the Only possible way that you can get safe automation and of course safe AI agentic applications on top of this as well. So the three examples I'll tell you about is homegrown, which is possible particularly if you build your agentic systems like we did yesterday. Second one is using DBT for this and the third one is using Omni, which is one of the BI tools that we really like at this point. So, so first of all, the homegrown one, honestly this is as simple as a bunch of text files that you feed into an agentic engine. For instance, to build your SQL queries from there that tell you about the dimensions you have, the base queries you can actually write there, where you actually in your files would specify the base queries that you then can compose together. You derive metrics from there. And the key join rules, I just literally grabbed the ones that I built for the demo I ran you through yesterday because that got me correct SQL consistent every single time I ran it. Things like canonical metric definitions, having those agreed as well. And then the join rules in there. And any other semantic layer system is going to be a variety of these different rules, which is why I thought it was instructive to show you the homegrown variation. First, this doesn't integrate easily with BI tools or notebooks and so on and so on, but it does integrate with Gentec applications that you're building, which is why I, I wanted to include it here. The second big philosophical sort of thing is doing a DBT led. And the DBT semantic layer is something that's only really available in the DBT cloud and enterprise plans, but it is worth looking at if you have a relatively large data platform where you gotta make sure that your semantic layer is available as an API that's robust, that is performance, and that you can actually build a lot of applications on top of. That's where the DBT LED approach really works. They will have as part of your big DBT project, not just your definitions for your columns in your tables. If you're using dbt, you're quite familiar with the way that the models are set up. In that way you also have definitions around the measures you can build on top of that dimensions, how they join together and how they sort of aggregate up and how they roll up. Now this is not that different from those of you that might have used tools like Looker and Co in the past. Who here has used Looker or sort of a code based BI tool? Raise your hands. I see Luker has some work to do. Still Here, but this language, this sort of way of using YAML files to configure your semantic layer was probably pioneered by Luker about five, six, seven, eight years ago. And so therefore it still looks like this a little bit. The difference between the way that DBT does it is DBT says we're not going to do this as part of the BI toolkit. We're going to do this in the middle of the data stack to make sure that it's accessible by everything that consumes from it. Useful if you're interested in scale. But you got to pay for the DBT cloud environments, which is sometimes a blocker for people. Another way to do it is to, and this is particularly interesting in small, fast growing teams where we deploy it is use a tool like Omni. Omni is actually the old Looker. People who build, built that first semantic understanding of Looker and then built a new BI tool after Looker got acquired by Google a few years ago. And Omni is not just a BI tool, it's a platform that you can build on top of. One of the things that it does really well is it's a metrics first metrics of this infrastructure BI tool. So everything that you do in Omni is version controlled. They apply this infrastructure as code philosophy everywhere. We can build our own sort of, this is our internal measurement strategy for our internal utilization and who's working on what task for which client and so on and so on. And there we build our measures and dimensions in the same way. So we build it in the Omni tool, the BI tool, rather than the central data model layer. The main benefit of that is that you don't need to get an extra subscription in, you don't need to get an extra tool in and you can just make Omni the sort of main platform that you're going to query from there. Omni, by the way, has quite a good browser interface with which DBT doesn't really have for the semantic layer here just to make sure that you can query it, you can always, even if you are say a sales leader, know exactly what metric means what and go from there. So whether you put your semantic layer within the presentation layer or whether you say, hey look, it actually is part of the BI toolkit and we're going to do it over there, consider it, it's a very important step in getting ready for AI and making sure that your BI is, is properly centralized. All right? And that's tip number three, is build your semantic layers because they're a key part of the activation journey and the activation journeys around those very short feedback loops there. And now we've come to the very end, but I just wanted to sort of repeat again the three takeaways I want you to remember. So first of all your data stack choices, consider those in functional for the business needs and then build them as lean as possible. Second, model your business in a domain model because you want a static target to orchestrate your entire data model around. Do this from scratch, don't wait, build this. As soon as you start to build a data model, really, it really pays off. And then three is your semantic layer because that centralization source of truth and is a really important part of your activation journey. And with all of that, thank you so much for coming again. It's a big pleasure to be here. Thank you.
Speaker A: Thank you, Thomas. And we have a few questions all through your presentation. The semantic layer seemed to be the concept which came back all the time. So we have a few questions related to that. And this is related to. How about the databricks metrics as a semantic layer?
Speaker B: Yeah, absolutely, I think so. What you see and this is part of the consolidation of the landscape a little bit. Tools like Snowflake or Databricks, platforms like Snowflake or Databricks, I should say they will come with their own version of these semantic layers. I think they are very valuable. Do not confuse intellectual purity around where the semantic layer should be in a sort of abstract vacuum with what's really useful for you. Pragmatically, I can immediately think of use cases where, where it makes total sense to build your data models in DataBricks, use the DBT integration of DataBricks, use the metrics integration in databricks to get your semantic layer and there and immediately accessible. If that works for you, go do it. It's great. They will follow the same principles as the ones I've just discussed. Absolutely.
Speaker A: And if the case is that we have semantic layers across different platforms and we need to have this consistence across those platforms, so how is this the best approach to get that done?
Speaker B: So consistency across platforms is the point of the semantic layer. So if you end up with two different types of semantic layers, you've done it wrong. Right. I think if you do find yourself struggling with different definitions for the same type of metrics of dimensions in your semantic layer, you got to go back to the drawing board that we designed from scratch. That being said, if you are in a big organization with a few different data teams or a few different data functions, then there are Philosophies around how data and business work together that allow you to align on the same definitions across the team. And one of those is data mesh. We really like data mesh as a way of thinking around data teams. Come talk to me afterwards. Long discussion. But it involves the product thinking. It involves the sort of federated ownership of data stacks, getting people to own data creation at source, all of those different things that are important over there. But there are methodologies to make this easy and make this happen and consistent and centralized. Yeah.
Speaker A: And any insights into, if we harmonize it between DataBricks and Power BI copilot, what are the best steps?
Speaker B: So honestly, it's a great time at this point to build data stacks because while we see some consolidation into tools like databricks and snowflake and so on and so on, building platforms that lock you in a little bit, there's still so many different choices out there for every single layer in your data stack. I was thinking at one point to just give you a bit of an overview of the different choices you have, but there are just too many. It's part of the problem as well. As I described earlier, you need to be very opinionated about all of this to sort of be convinced that one tool is obviously better than the other tool, but it is the best possible tool time to sort of build all of this from scratch for sure. When it comes to agentic implementations, we are still at the point where SQL is a commodity language, meaning that if you build your data stack well, it can be consumed by any agentic tool, any downstream business intelligence consumption reporting tool or whatever it is. So you don't really need to make that choice up front. The only choice you have to make up front is your data warehouse and how you're going to model your data. So do that wisely, I suggest.
Speaker A: Thank you. An interesting mention because you talked about briefly about Agile, which was a few years ago, the keyword everywhere. And now it's less in the talking because AI seems to be the one which takes over. So how do you see that? How Agile is still there within an AI and data first environment.
Speaker B: So there are different things, of course, But I think the interesting bit about Agile and for us why it works is, well, we designed a company around sprints. We run sprints for our clients, so we're biased when it comes to this. But agile means, hey, what is the best way to get to the leanest set of deliverables that still meet your user's requirements? And in order to do that, it turns out that you do need to have a really good idea of what your user requirements are before you start working on something. And for us that's what Agile means. It means that we know what the roadmap looks like, we know what the users need in terms of functionality, whether it's from reporting or from data engineering or modeling, and so on and so on. And so we can actually start mapping out those requirements to what's technically necessary to get there and then try to run this in the most efficient way. And for us that means sprints, it means story points, it means big Jira or Monday boards, and so on and so on. If you're an internal team, you might not need it. If you're an internal team, you might be able to run longer projects that are really preplanned because there's not that much uncertainty anymore, I think in this part of the stack. Or you can run on a more day to day Kanban style way. It's sort of really up to you. I don't think the level of discussion around Agile is as intellectually dogmatic as it was for or five years ago. Do whatever works for you. The worst thing you can do is try to do Agile while still trying to pre plan everything six months ahead of time. That doesn't work. But that's the only certainty I have about all of that. All the rest is up to you.
Speaker A: Yeah. Use it if it serves you. Absolutely, yes. What are the suggestions regarding whether to develop something in house or open source free tool or pay for essays?
Speaker B: Yeah, different for enterprise versus startups and scale ups. For startups and scale ups, I'd say never really build it yourself unless you're absolutely sure that this is something that only you can build. I see too many teams spend six months on building a text to SQL engine, for instance, where I know you can just get those off the shelf and they do a better job. You have better product development and better customer support than anything you can do yourself. Where the sort of build does come in is I'd say buy your modeler tools, but then build the glue that glues them together. Right? So build the CI CD layer yourself because you want to control that quite extensively. Build your test yourself. Don't rely on the testing that comes out of the box. But that's a slightly different layer of structure from the model tools itself. If you're enterprise, I know your requirements might be very different. You might not be able to run the same SaaS tools that were run in startups and scale ups and that's quite different. As a sort of scenario. So there you might not have any other option than to build it yourself, particularly if your volumes of events for instance, are very high. But luckily the days that people were home coding their own event data pipelines are long past. Excellent solutions for all of those things out there. Go and explore them and then make an argument as a, an exception to build it yourself rather than as a default. Happy to advise as well. So come talk to me and what
Speaker A: do you think about the cyber security considerations? Because I think that's closely related here.
Speaker B: Yeah, I think so too. So maybe two things. When we do projects for clients, we never work in our systems, we always work within their systems. Basic cybersecurity. I wouldn't say it's a solved problem, but Snowflake databricks have excellent access permission structures and so on and so on that allow you to really lock down systems in ways that you couldn't do five, six years ago. And it's probably their biggest moat in many ways compared to new kids on the block like mother duck and so on. When it comes to security in the different tools. Well, CI CD is quite an important part here because you can test for security and for particular access drift or something along those lines. But also most of these tools that we use because they're configurable with code all the way down, all the way from ingestion to the reporting that you built. It's the same with access control. You can control access all the way down as well. And so tools like Omni Looker integrate really neatly with for instance your Snowflake rpac. I'm not sure exactly how it works for databricks, but I imagine it's very similar where you can, hey, there's looker reports that only C level execs can access and that then works together with the roles based access control that you have in Snowflake to ensure that only they can indeed query the data as well. All those cybersecurity things, standard practices apply like all of you know, around cloud security, accessing data only via VPN and that sort of stuff. But that's, I mean those are logical principles that still apply here of course.
Speaker A: And what do you see as the biggest cybersecurity risk here? Because with AI, the cyber actors who are the bad actors there can be also making steps more quickly than before.
Speaker B: Yeah, I think that's fair. But I think that maybe the main way I'd spin that is to say that the main risk is querying raw data. So one of the reasons that the data model exists is to remove raw data access as much as possible from downstream consumption patterns. It's another reason why you do not want your agentic implementation to access your raw data to query your raw data. Because at that point you do open yourself up for raising PII where you would where you don't want to be able to query pii. It's one way where you can all of a sudden open yourself up for prompt injection or SQL injection injection and so on and so on. So we think that a really good, well designed data model that can answer any type of business question as long as the business doesn't change, remember is the key philosophical principle to ensure that you've got good cybersecurity all the way through. Because you should be able to filter out any of the dangerous information or PII and only surface in your reporting that which is safe and relevant and business actionable.
Speaker A: Cool. Clarification question. So why we should avoid dashboards as endpoints?
Speaker B: Yeah, so what I mean with an endpoint is not in technical way. What I mean is a dashboard is where the sort of discussion with your team start. If as a data team you just consider your job to be delivering dashboards to your decision makers, I'm not sure you're going to get very far at this point because you got to sit next to them and try and understand, understand what it is, what decisions they are making and help guide them. And that is a conversation that only starts when you've got that first version of the dashboards in their hands. I've never met a stakeholder who's able to describe in detail exactly the type of dashboard reporting that they need from scratch. They always need this sort of conversation iteration that takes place. So you give them a first version, you say, hey look, I looked at the data and these are the first ideas I have around visualizing your customer acquisition cost against your customer lifetime value. Does this look good? Oh, you want to split up by geo. Okay, let's do this as well. And so on and so on. So it's the start of discussion. That's what I mean by dashboards not being the endpoints.
Speaker A: We can be tempted to hurt data. And what happens to that part which doesn't deserve to be kept?
Speaker B: Well, gdpr, for all of the faults around implementation and case law, I think GDPR has really changed the culture around data retention and quite rightly so. You can only really retain personal data or usage based data in my view if there's a really good business reason for it. And that really Forces you towards the business analysis side of things, I think data retention for the rest. Yet there are data points that you just need to gather into the sort of 10, 15 year time horizon, particularly if it's around financial transactions or anything. That's really core business logic. But I think people are much more mature around figuring out why do we need to keep something rather than keep everything as a default. And again, the data model there becomes important as well, because if you do your job well in the data model, you actually can forget about the raw data or you can at least start deleting PII parts of the raw data that you just want to have forgotten after two years. Absolutely. So data retention is a core part and in my view it should be owned by the data engineering team because it's a shift left type of discussion as well.
Speaker A: The domain and presentation layer in Data Warehouse, is it a requirement or alternative to the semantic layer?
Speaker B: It's a requirement, I think. But there are semantic layers you can build on top of raw data as well. It just depends on how complex you want to make it. For me, the presentation layer is about surfacing central source of truth data. The semantic layer is about telling you how you can aggregate that data. So semantic layer is a set of rules. Presentational layer is a set of tables with data in it. That's a big difference. You can build semantic layers again on top of raw data, but I think you're going to struggle to scale this, or at least to make it performant over 6, 12, 18 months, particularly in a scale of type environment where your questions will change.
Speaker A: It seems that some have some frustration about leadership not using the dashboard, the nice outcome of their work. So what is to be done here?
Speaker B: Well, that's the whole problem is you build, you spend two months on a great dashboard and then your CMO isn't using it. And I know that often they will have a particular set of stubborn requirements that you then build for. Always assume your stakeholder does not know the data as well as you do. So build the first version that you kind of build, understanding their requirements, understanding what's possible in data, and then have the conversation about what it is that they need and how to iterate from there. If that still gets you into trouble, or if it still means that they're not using the dashboard two months, three months down the line, then maybe they're not as data driven as they want to be, which is not your problem, it's their problem. But that's the sort of advice that I give is sit next to them and talk to them and make them understand.
Speaker A: And another one related to the semantic layer and that's the one between the warehouse and BI tools and shooting. Should we consider using the semantic layer in the notebooks and reverse etl?
Speaker B: I knew someone was going to ask that because I put the semantic layer in between the presentation layer and the bi. Yes, I think you should use in reverse ETL and notebooks as well. I think there you are typically querying some more raw data that's already pre processed. So I would say that maybe it's less critical. But yeah, ideally you want to be able to have every single tool consume the exact same metrics, layers to clear presentational layers, all the rest. Absolutely, yeah. It's an interesting question.
Speaker A: Legacy applications so if the event tracking is inconsistent over the time and it's difficult to maintain, what is the best approach to audit that?
Speaker B: Burn it all, start from scratch? No, I think people overestimate the importance importance of old event data. Right. If you can't guarantee the one thing about event data is that once it's generated, it's done. You can't come back and regenerate the event or redefine the event. It's there, it's always there. It's sort of written in stone. Whenever we've had this issue in the past, we just went back and redesigned it from scratch and say, look, you just gotta forget about the last two, three years, let's look at your review data instead or some other third party data that you can aggregate better. But your event data, if you don't trust your events from a year ago, just ignore them. I also think data because at the end of the day data is meant to inform forward looking decisions by your business. Turns out that event data from a year ago is just not that relevant as people think in current business decision making as well. So don't be afraid to cut your losses, burn it with petrol and fire and so on and so on and just move on from there. If you do need to audit and repair, yes, you need to do it in data model. And this is a very expensive migration job, which actually we don't really like to do. Come talk to us about anything else, but not that part. Talk to other people about that. I think it's a difficult thing to get right, particularly if it hasn't really been maintained well. But you know what they say, the best time to fix it is a year ago and the second best time is today. So if you have messy event architectures, then please come talk to us at that point and we can help you get it better for the future or join us. We have open data engineering and analytics engineering positions so come talk to me as well about those. Sorry.
Speaker A: Yeah, we have just a few more questions to cover quickly. So if you have any suggestions on general data model that fits all sizes.
Speaker B: Well, that's the domain model in many ways. Model. Do it as a paper exercise first. We do that. It's half a day. Get your stakeholders in the same room and say hey folks, we're going to define your central logic and we're going to have disagreements about definitions but we're going to do it once and for all and then we've got to sort it around. What is a user? Does a user need a user ID to be a user and counted as a user or do they just need an email address? Those different questions are really important. If you get those right then I don't care whether you use an immanent style entity model or go straight to dimensional modeling. I've got my preference. But there are different really good modeling approaches there. We are really pragmatic. We like the narrow waist, the main layer because it allows for rapid scale and dynamic answers in your data model. But I can immediately see why one big table might work for you as well. And we do those in a presentational layers. Sometimes it's one big table, sometimes it's fact and dimension, sometimes it's very complex. Snowflake kind of star schemas. Do whatever works best for you. Different tools have different requirements as well. A tool like Sigma in BI is much more comfortable with one big table than a tool like Omni which is more comfortable with dimensional models. So it really depends on your consumption patterns.
Speaker A: A non data person asks what is your best recommendation for database and how to store it?
Speaker B: Use a cloud data warehouse for sure. Don't use MySQL is probably my main recommendation there. I hear some nervous laughter. Sorry. We like to sort of be cloud warehouses for relational data storage. They're pretty equivalent when it comes to functionality. Whether you use Databricks, Snowflake even more new kits on a block like Motorduck and so on and so on. They all have their strengths and weaknesses but. But broadly you don't really make a wrong choice in there and they're competing nicely in a nice arms race where they are differentiated are in what you can do with unstructured data in them. Databricks has great support for AI, ML and Genai applications straight out of the box. So does Snowflake in slightly different ways I would look at that when you know what your plans are. But when you're first getting started, just put your credit card into databricks snowflake and kind of choose databricks, whatever works for you to kind of go from there.
Speaker A: Sometimes it's easy to get lost in understanding where data comes from. So the question was that how to map the possible relationship logic when you don't really get support to understand the sources of your data?
Speaker B: Yeah, that's a problem. Keep receipts, right? Keep copies of, of the emails you sent out to your stakeholders to invite them to the sessions where you can point to and say you had a chance to define the logic here. This is probably where I'd say you're in a much better position now than say four years ago because you got to remember that however complex your stakeholders think the business is, you're probably not that different from other similar sized businesses. So if you're a subscription app business, you've probably got a fairly trivial. Sorry, you've got a fairly similar, similar layout to other subscription app businesses and you're probably going to find some good information online about, hey, here's the work we did. We've got some cases on our website about subscription business logic that we built. So go have a look at that and see if it inspires you to kind of at least get a V1 in place. Also, don't forget that the good current set of large language model chatbots are pretty good at surfacing these type of relationships as well. So have conversations, get, get them as a sort of V1 inspiration to say, hey, what are typical event definitions in a mobile subscription startup or an E commerce company? And then let them lead you from there. Yeah, there's a lot of information out there online that can get you to V1 as long as you realize that that is the start of a discussion then where you come in with the first definition and go from there.
Speaker A: Yeah, always asking our LLM friends,
Speaker B: they might talk back to you.
Speaker A: Yeah, yeah, absolutely. Thank you very much, Thomas.