Raw capture — raw/dr-egor-kraev-on-structured-objects-no-escape-no-leak.md

url: https://gist.github.com/3ef64cb852dadd4e90234c69b82252bb date_fetched: 2026-09-13

You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert

Dr. Egor Kraev on Structured Objects: "No Escape, No Leak"

Key Points

The most impactful production use of GenAI is making LLMs emit structured objects — reliably, every time. Kraev, who has done ML "since the last millennium," frames the whole talk around a single claim: LLMs only add value when their output plugs into old-school systems, and those systems "expect objects of a very defined shape," not essays. The "agent AI nirvana... I don't think will ever happen." Everything else follows from this.

Schema validation is not enough; you need guaranteed validation of complex, real-world constraints. Pydantic-backed structured output guarantees the shape. It does not guarantee that names and amounts match the input, that a SQL query returns a non-empty result, or that a string contains valid markdown. "Reality is rarely so simple that just enforcing a schema is enough."

Every standard approach leaks. He walks an escalation ladder — prompt + parser + hope ("probably will not work"), JSON mode (valid JSON, wrong schema), function calling/with_structured_output (valid schema, unvalidated semantics), retries ("try again, try again, you might get lucky"), validator-as-a-tool agentic loop (the LLM gives up after 3–4 tries and returns something anyway), and return_direct (still no guarantee the returned object is the validated one). The recurring failure: "you cannot trust the LLM to reliably relay its input to the tool. You just can't, because it's not what it does."

The fix: make the validator the only exit. In his framework, an agent flag forces return through the validation tool — "there is no escape, there is no leak." If validation fails after X iterations, you get None: "at least you get a clear failure. You don't get something that the LLM thinks is valid, but it's not."

Don't let the LLM touch objects; give it tools that touch objects. For modifying complex artifacts (chart configs, etc.), the "ship in a bottle pattern" keeps the object living outside the LLM, with the LLM choosing among semantically natural actions (add a series, change chart type). Otherwise, "if you just say, here's the original thing, change X, it will change X, it will also change Y, Z, A, B and C all over the place." Same principle, he says, should govern MCP servers: "You don't have atomic transactions as tools. You have meaningful semantically natural actions."

Don't generate SQL; generate semantic-layer queries. He "laugh[s] evilly whenever I see anybody declare reliable security SQL generation." Instead, expose a semantic layer of metrics and dimensions, have the LLM produce a small pydantic query against it, and convert to SQL with "fully deterministic machinery" that is "stupidly easy to make work."

The philosophy: LLM as a small box among boxes, not a conductor. "Contrary to most of the agentic hype," give the LLM "as little leeway as possible" and let it do only "the one little thing that nothing but an LLM can do." Everything deterministic — including data orchestration — stays old school. "This way it's cheap, it's simple and it works."

Second production pattern: keep LLMs off the critical path. Have the LLM produce a validated complex object once, then reuse it. This gives you "squishy inputs, but hard outputs" — faster, more reliable.

Pithy and Provocative Quotes

"I don't care about building what's sexy, I care about building what works. What works every time?" — his opening framing, from someone who has watched multiple hype cycles.

"AI is just a marketing term." — deployed to justify treating LLMs as one tool among many.

"If you send information to a regulator that's wrong, your company may not survive it." — why the Wise suspicious-activity-report use case demands guarantees, unlike customer care, where "you get shitty customer care, but your company can survive it."

"You could probably write a SQL query that could compose the works of Shakespeare if you tried hard enough." — why SQL's expressiveness makes LLM-generated SQL unverifiable.

"I laugh evilly whenever I see anybody declare reliable security SQL generation." — his bluntest dismissal in the talk.

"The LLM is an unreliable messenger... you cannot trust the LLM to reliably relay its input to the tool. You just can't, because it's not what it does." — the recurring theme, stated as a law of nature.

"There is no escape, there is no leak." — describing the validation pattern where the agent can only return through the validator.

"It's pretty good at obeying instructions. It's just not good at not doing other stuff as well." — his answer on why LLMs call tools fine but mangle everything else; the tools enforce the "not doing other stuff."

"The poor LLM has no choice but to return valid output. So as long as it's something released this year, it's probably going to work." — the side effect of using validation everywhere: model choice stops mattering.

"Writing your own coding agent... is really not a good hobby this year. Last year it might have been worth it." — on using headless Claude Code instead of building your own.

"Then you get back into the stochastic country." — his rejection of LLM-as-judge validators for multimodal output.

"Using LLMs to remove friction so you can do the same things you used to before, but now the tool has common sense." — his prediction for the next wave, as the "bigger LLMs" wave saturates.

Tools, Practices, and Methodologies

Pydantic — Python data-model library. His baseline requirement: "I would never ever try to do anything simpler than this when producing structured objects." Use it to define the schema for structured output.

LangChain with_structured_output (and equivalents in every framework) — schema-guaranteed LLM output via function calling. Guarantees schema validity only, not custom validators — but "in a lot of applications that's already enough."

with_retries — the "dumbest approach": retry until lucky. Treated as a floor, not a solution.

Validator-as-a-tool agentic loop — the classic pattern where the agent calls a validation tool before returning. "For me, this kind of feedback loop is exactly what an AI agent is." Insufficient because the LLM can give up and return anyway.

return_direct — tool attribute that returns the validated result directly, bypassing the LLM. Eliminates one leak; leaves the give-up leak.

Motley Crew — his open-source agentic framework (written a year ago to mix and match agentic frameworks). The key features: an agent flag forcing return only through the tool (the validation pattern), tools with direct access to the agent's original input, and a key-value store attached to the agent that tools can use to exchange tokens/blobs without the LLM as messenger. He admits you could build all this in custom Python or LangGraph — "here it comes built in."

Semantic layer — an abstraction exposing only the metrics and dimensions users might want. The LLM generates a small pydantic query against it; deterministic code converts to SQL. His recommended replacement for direct LLM SQL generation.

Ship in a bottle pattern — the object lives outside the LLM; the LLM manipulates it only via semantically natural action tools (add/remove measure, add dimension, change chart color). Also his stated pattern for MCP servers.

Intermediate stripped-down pydantic object — even when the final output is complex, have the LLM produce a minimal object containing only the fields it should decide, then have the validation function convert it to the real thing. Used to turn a loose wordy prompt into full SQLAlchemy schemas with foreign keys.

Regex + spaCy entity extraction — "old school" verification that names and amounts in a suspicious activity report exactly match the raw input.

LLM log-likelihoods as classifier features — for LLM-as-classifier, return log likelihoods (a structured output) and feed them into an old-school model alongside other features.

Synthetic Data Vault — library he recommends (but didn't use) for reproducing the structure of a real dataset.

Headless Claude Code — for stage-two synthetic data population: describe the data-generating process in words, let Claude Code write the Python, wrap it in validation (code must run, must touch every table). For code generation generally: "I wouldn't use raw LLMs, I would use something like Claude Code or Cursor."

Restart the loop — when the LLM gets stuck in a wrong pattern, "just restarting the loop has always worked" so far.

Off-the-critical-path pattern — LLM generates a validated complex object once; the object, not the LLM, serves production traffic. Faster, reusable, reliable.

Unanswered Questions and Omissions

Zero numbers. No iteration counts, failure rates, latency, token costs, or before/after reliability metrics — from a talk whose entire thesis is reliability. "After X iterations" — what is X, in practice? What does a validation loop cost in wall-clock time and tokens?

The termination caveat is hand-waved. He concedes "there is no guarantee that the validation loop will ever conclude" and covers it with "so far, restarting the loop has always worked." That's an anecdote, not a guarantee — from a talk titled around guarantees.

Validation guarantees shape, not quality. A schema-valid, validator-passing object can still be semantically wrong in ways the validator doesn't check. No discussion of evaluating output goodness, only output validity.

Guardrails and DSPY: explicitly dodged. "To be honest, I don't know. I haven't tried guardrails." His novelty claim ("I haven't really seen it advertised elsewhere") is untested against a ecosystem he admits he can't track — "there's been an agentic framework a month at least lately."

Security is absent. He mocks "reliable secure SQL generation" yet never addresses prompt injection, adversarial inputs, or what happens when the data feeding the semantic layer is attacker-influenced. For a talk about giving LLMs database-adjacent access, that's a hole.

No human-in-the-loop discussion. For regulator-facing outputs (the SAR example), is there a human reviewer on top of the validation loop? Never said. Also unclear: was the SAR system actually deployed at Wise, or is it a motivating hypothetical?

Conflict of interest unexamined. The talk's central solution is his own framework. LangGraph is acknowledged as capable of the same custom flows only in passing during Q&A.

Multimodal output: punted. "I'm not sure how you would do this" — the entire validation edifice depends on deterministic validators, and he concedes LLM judges return you to "the stochastic country." This is a real boundary of the approach, noted but not explored.

The "model doesn't matter" claim is asserted, not shown. "As long as it's something released this year, it's probably going to work" — no evidence, and it sits in tension with widespread experience that weaker models struggle with tool-calling discipline.

Big-picture claims without argument. "Agent AI nirvana... I don't think will ever happen" and "the current wave of just making the LLMs bigger is saturating" are both delivered as verdicts, not defended.

State and concurrency ignored. The key-value store attached to the agent is sketched in one breath — no word on persistence, multi-user access to a shared object, or what happens when the "object hanging around somewhere" is needed by two sessions at once.

Speaker A: And before I introduce our next speaker, some technical announcements again. So the WhatsApp application which is needed for the drawing for the LEGO robot, not the robot, but the LEGO set is available outside on the QR code printing. So you can see the QR code there and check it and you will get into the WhatsApp group and then when you interact with the chatbot which is behind the WhatsApp group, then you are into the drawing. So please make sure that this happens if you would like to be in the drawing and who would not like to be there. And I noticed myself that the camera coupons which are for the evening party alcoholic drinks, they can easily fall from your badges. So if you notice that it's already trying to leave the main part of the badge, then just put it in your pockets because that's the safe way to keep it and use it later on. A question just for the audience. I think that was really interesting to hear about the human robots and how they are evolving. So I would like to ask a yes or no type of question and I'm interested in who is more for the human robots and who is seeing rather risks. So I would ask to raise your hand if you are more pro robot. Okay, who is seeing the risk rather? Almost 50 50. Okay, so this is really interesting. So I'm announcing Dr. Jaeger Krav who is talking about the structured objects which are most impactful application for Gen AI production. And he has a background in investment banks and after more than 10 years he walked out from the investment banks and then went to work for Wise where he was the headed head of the AI team which he built from scratch to 30 people. And then he decided to move to his own company which is Motley AI. He is the CTO there and he wants to help us in storytelling and he is, you will see that he is also having lots of libraries which are open source and you can check their useful knowledge. Those libraries include Vice, Pizza and casualtune. And I guess you will see the QR codes to there or the routes. So let's welcome Jagor Krav and hear about those.

Speaker B: Hi everyone. So I will tell you about the awesome power of structured objects. Question number one, why should you care? Why do structured objects matter? Why do I care about structured objects? Well, I've been doing machine learning or what they call AI these days since the last millennium and I've seen a couple of hype cycles come and go. And so I don't care about building what's sexy, I care about building what works? What works every time? And to squeeze the maximum value out of LLMs, because AI is just a marketing term, you really have to treat them as one tool among many and then you have to make it play nicely with the other tools. Now most of these other tools expect not freeform squishy output like an essay or even code. They expect objects of a very defined shape. And so the key to adding value with LLMs until we reach this agent AI nirvana, which I don't think will ever happen, the key right now to adding value with LLMs is making LLMs produce structured objects and do so reliably. And this is what this talk is about. So let's go through a couple of examples. One is from wise, where we wanted to generate suspicious activity reports. That's actually a technical term, Wise, like every financial company is required by many different regulators to report anything that is considered suspicious activity. And of course this is a very boring but very precise task. And when you do this, you want to be very certain that if you use an LLM to help you produce this, that the names and amounts, et cetera, that appear in the summary are exactly the ones that appeared in the raw data that the LLM was fed. Which is not something the LLMs are generally famous for. And maybe when you use LLMs to do customer care, you get shitty customer care, but your company can survive it. If you send information to a regulator that's wrong, your company may not survive it. So it's important to be sure. Another one is data queries. So you want to query a database. Now anybody who has worked with SQL knows how wonderfully expressive it is. You could probably have write a SQL query that could generate the compose works of Shakespeare if you tried hard enough. But for that same reason, even validating a SQL query like does it parse? Is one thing, but after it has parsed is it even close to what you wanted? Is a very hard task. So LLMs can write SQL? Sure, they can write anything, but how do you constrain the LLM to write a query which at least comes from the very small set of queries that you actually care about and not something strange and wonderful and creative? Another one, which is actually something we've been doing a lot in Motley, is you have a chart, just a picture of a chart, and you have a database given somewhere and you want to generate a chart config to recreate that image. This means both the data query as well as this series as a bar chart of color is pink. The axis labeled the Legend is vertical, the legend is in the top right corner, the whole thing. And this config, as you can imagine, is quite complex and it has many interrelationships and also it might depend on some external things. For example, the data query that you generate must return a non zero result, which is not something the LLM can evaluate on its own. And so enforcing a JSON schema is helpful, but that alone doesn't get you there, even if you could be bothered to write a sufficiently precise schema. Another one. Suppose you've created that object, you've created the chart config, you've created the database query, whatever. Typically, if you have any kind of human facing application, you want to interact with it, you want to modify it, add a line, change this bar chart to color green, et cetera. And in itself, getting an LLM to do this is not very hard. What is hard to get it to do? To stick to just the change you wanted. Because if you just say, okay, here's the original thing, change X, it will change X, it will also change Y, Z, A, B and C all over the place. And how do you make it not do that? So if you wanted to do all these things reliably, not maybe not 99% of the time, but every time, how would you do it? So there is a naive approach, right? This is like two years ago, you put what you want into a prompt, the whole prompt engineering thing. Then you send the output of the LLM to a parser, just call the RAWLM API, send the output to a parser and hope and, well, probably will not work. You can try fancy parsers, prostibulate will break randomly. You can do slightly better. Most APIs these days have some kind of a JSON mode flag, which at least guarantees the result will be valid JSON. Well, okay, but will it have the right schema? Probably not. Especially if you think about what complex thing like a chart config, complete with data query and all the chart properties, et cetera. All right, next one is, which I'm sure people will, you know, people have been using here is use function calling or equivalent. And personally I'm a big fan of pydantic for representing data structures in general in Python. And then if you already have, if you have a pydantic schema that you want to have as a result, like LangChain has the with structured output method, but every kind of LLM framework has some kind of equivalent and so you can effectively specify your schema through a pydantic Object data model and then at least you're guaranteed to get back that kind of valid Pythagoras. But again, valid only so far as the schema is concerned. So for example, if your object has additional validators that they will not be satisfied necessarily. But in lot of applications that's already enough if you do. And certainly I would never ever try to do anything simpler than this when producing structured objects. But what if it's not enough? What if you need complex validation? So for example, they have this monster object with lots of fields. This field is text, this field is a string which must contain valid markdown or this database query must return non empty result or the kind of things, the set of names in the end result must be exactly the same as the set of names in the input and the set of amounts. So any kind of after the fact complex validation logic. And of course that's what the real world is like. The reality is rarely so simple that just enforcing a schema is enough. And so this is what this talk is about really is guaranteeing validation. And the dumbest approach, which is the with retries argument in the with structured output in LangChain, is just to try again. If you fail, never mind try again, try again, try again, you might get lucky, you might not. The next one, which is if you ever played with agents, the next logical step. And by the way, you know, I'll be sharing the slides anyway and putting them on my LinkedIn as well. I've shared them with organizers as well. So this is like the classic agentic pattern. You have a user query coming in and then your validator is just a tool that the agent can call. So before returning the result, the agent can call the validation tool. The validation tool says, okay, is this valid? No matter how complex tells back to the LLM, okay, no, fix this, this is bad, fix this. And then once the validation tool is happy, hopefully the LLM, the agent will return. The agent will return a good result. And for me, this kind of feedback loop is exactly what an AI agent is. But that's not really enough because the LLMs, they are inherently unreliable. So first of all, you cannot guarantee that the LLM will keep calling the validation tool until the validation tool is happy. And even if it does, because sometimes what typically happens is it tries like three, four times and then it just gives up and returns something. And even if it does, you have no guarantee that the thing it returns to you will, will be the same thing as the thing that was just successfully validated. It hopefully will be but you have no guarantees, and this talk is all about guarantees. So, next step. And so far we're still within the realm of classic agentic frameworks. So I call them frameworks like LangChain. You have this return direct attribute which lets the tool to return, which lets the tool return the result directly instead of asking the LLM to do do so. And so this is already. So at least you've eliminated one problem. So basically, the way it works now, LLM sends the candidates the draft output to the validator. If the validator is unhappy, it tells the LLM why. If the validator is happy, it just returns the result. Agent terminates, great. But you see, there is a leak there. There is still no guarantee that if the LLM doesn't succeed in making the validator happy that it wouldn't just return something. And in fact, it almost certainly will. Because the reason you bother with this validation logic is because it's not super easy to satisfy. And therefore the LLM will try once, they'll try again, and then it'll just give up. So that's one problem. The other problem with this classic pattern is that very often, like in the Suspicious Activity Report example, the tool needs access to the input of the agent. So you need to know what exactly were the names and amounts specified in the raw input. And again, you cannot trust. Like this is a recurring theme. By now I'm sure you've seen you cannot trust the LLM to reliably relay its input to the tool. You just can't, because it's not what it does. And that's the point where I can't help beating my own drum a little bit. So we. So that's an open source agentic framework that we wrote a year ago to address these things. And it also, we also meant. Well, it also does lots of other things, but this is the one that turned out to be truly useful. And you, by the way, caveat. Of course, you could do all these things in custom Python code, align graph, or whatever your custom thing is, but here it comes built in. And so here. So one of the things we've built is an agent. You can set a flag on an agent to say that you'll only return through the tool. And so you'll get a pattern like this. And that's one thing that I haven't really seen advertised elsewhere, which is almost the only way I generate outputs these days, which is the validation pattern. So the agent cannot return directly, ever. The agent sends stuff to the validator and if the letter is happy, the validator returns the result. If the validator is unhappy, it tells the LLM why. The LLM tries again, but there is no escape, there is no leak. And then as another thing, the tool objects have direct access in a simple way to access the original input of the agent. All very simple wiring. I think it should come as standard. I'm not surprised it's not more talked about. So before we go into complex objects, let's go back to those examples and show quickly how you'd solve them. Suspicious activity report. Very simple, you just have old school regex verifying that. What's in here? What's in there? Maybe use spacey for entity extraction and then do that. Hold on, where are we? Sorry, I am a little lost. Okay, so yes, no guaranteed validated queries. Here's another one where I laugh evilly whenever I see anybody declare reliable security SQL generation. I think even if reliable SQL generation directly by LLMs is feasible, it's so hard that it's not worth the trouble. So really what I would recommend here is to use or maybe create something like a semantic layer. Because all the semantic layer really is, is just an abstraction that just exposes the options that you think might be useful. You can aggregate by these things. You can, these are the numbers, like these are the metrics, these are the dimensions. And that's a much, much more limited interface. And then that interface, the query for the semantic layer is what the LLM generates and that you can, it's very easy to write pedantic objects for that. And then there is fully deterministic machinery converting that into SQL and that is stupidly easy to make work. Chartconfigs from images, much the same thing. So you have a standard validation, you have a pydantic validator, and you might also want to query the database. The beauty of it is you don't have to worry about the complexity of your validation because all you need to do is if something goes wrong, the validator just sends an exception back to the LLM. Okay, manipulation of domain objects is a somewhat different story. And that's the one that we can now get. And that will now get to. Right. Modifying a complex object as now, as I'm sure you're all sick of hearing me say by now, the LLM is an unreliable messenger. You cannot ever trust it to pass things through correctly. And in the standard tool calling agent pattern, what most of the agentic frameworks call agents, each tool only sees the inputs that LLM passes to it and thereby we trust the LLM to not mangle stuff. And that never goes well. So instead what we want to do is you want to have tools first. You want to have your object outside of the LLM, so you have your object living as an object somewhere. And then what you give to the LLM is not access to the object directly, but, but just tools for natural actions. And by the way, coming back to, I think that's something that's mentioned in the previous talk, that's also the pattern you should be using for MCP servers. You don't have atomic transactions as tools. You have meaningful semantically natural actions. And so there is no MCP required here. Although of course you can use one if you want to, but that's the same principle, and I call that ship in a bottle pattern. So when you build a ship in a bottle, you don't actually touch the ship when you assemble it. You put it in pieces in a bottle, and then you use little tools to go through the bottleneck and manipulate it indirectly. And by the way, if you want to stress test your favorite image generation model, try generating this. Try generating a human manipulating an object inside a bottle with a stick. I'll guarantee you your favorite image generation model will have a very hard time. But that's by the by. So the pattern is just like this. You have a query and then the LLM gets the query, and it has a selection of tools. And so instead of returning the object directly, as we did before, the object just hangs around somewhere and the LLM chooses to modify it in semantically meaningful fashion, like change the color of the chart, change chart type, or whatever it is, add a series, and the object just hangs around there somewhere and your surrounding code can access it. In this way, the LLM changes just what each tool allows it to do and things are good. And the LLM can still return conversational output for user interaction. And the final. But the question is then, where does this object live? Of course you can always write custom code, but the other thing we added in Motley Crue is a key value store that's kind of attached to the object, to the agent object, but it's not. But it's not visible to the LLM necessarily, but the tools can access it. And so the tools, for example, tools can exchange information. They can exchange tokens, binary blobs, whatever it is they need without relying on the LLM to be the messenger. And this, of course, yes. So this has users beyond the ship in the bottle pattern. So if you want to try it out in Motley Crue, you just have a simple function for the validation pattern. Because we use it so much, we just wrap the age into simple function and give it a schema. And another thing for this, which I found very, very useful, is that even if the thing you ultimately want to produce is already a pydantic object, it still pays to produce an intermediate one, which is like super stripped down, which only contains the options that you want the LLM to determine and nothing else. And then you have a validation, and then your validation function, along with validating before returning, it converts the simpler thing that the LLM produces into the thing you actually want. And so for. So one example of this that I've written lately, I think two weeks ago, was when we wanted to create synthetic data for our demo environment, for example, with lots of tables, complex schema, lots of foreign keys, et cetera. Then I use this very simple wrap around this function to convert a prompt, just like human language, blah, blah, blah. Give me a schema into a valid collection of SQL alchemy schemas with foreign keys and with all the good stuff that you can just create. And by the way, if you want me to open source that specific example, just drop me a line. Because I'm a big believer in open sourcing everything except customer data and core business logic. But of course, it's only worth the trouble if somebody cares, Right? I am ahead of schedule, which I'm sure you will not hate me for. And so the moral of this talk is that with a bit of effort, you can actually generate guaranteed valid objects, no matter how complex your validation rules are using an LLM. There's a tiny caveat. In principle, there is no guarantee that the validation loop will ever conclude. But then the worst that's going to happen is that after X iterations, if the validation doesn't pass, you just get none. But at least you get a clear failure. You don't get something that the LLM thinks is valid, but it's not. And even then, just if the LLM just gets stuck in a wrong pattern, so far, just restarting the loop has always worked. And this gives you reliability, because now you know for sure whether it works or not. And generally, a philosophical thing is that instead of contrary to most of the agentic hype, the key to make LLMs work, or to add value, rather, is not to treat them as this genius conductor in the middle of everything, but rather as this little box surrounded by other boxes, and you give it as little leeway as possible and just have it do the one little thing that nothing but an LLM can do. And everything that is deterministic, including data orchestration between the other parts you do in an old school way. So that, and this way it's cheap, it's simple and it works. And at Motley we use this to create a smooth UX experience, to generate complex synthetic data and to play lots of other fun games. Because ultimately, and that's actually the other thing for using a LEMS in production. So one is this. So one pattern is this kind of validation loops. The other pattern for making LLMs work reliably is to not actually have the LLMs on the critical path. So what you do instead is you have the LLM produce a complex object that you validate and then this complex object is what you actually use. And this way it's faster and more reliable and usually you can reuse those complex objects many times over. And this way you can get the power of the LLMs of having squishy inputs, but have hard outputs. All right, that's it for me,

Speaker A: Thank you. And let me ask the questions. First of all, how the name came. So how did you find that your company is going to be called Mertley?

Speaker B: Well, it was. Well, we started with Motley Crue because the original idea of Motley Crue is to be able to mix and match all the different agentic frameworks out there. And so Motley Crue came naturally and then Motley was a natural derivative from that.

Speaker A: Okay, so how do you use these patterns and various. It used in practice

Speaker B: everywhere really, that you use LLMs. So because LLMs have to work with old school systems, so right now we generate these schemas for synthetic data. We generate chart configs and generally complex object configs for the presentations that we're doing monthly. This kind of. Another one was for classifiers, also for using LLMs as a text classifier. Actually the way to do so is to have the LLM return the log likelihoods, which is again a structured output, and then use this along with other features in an old school model. So really, if you want to make LLMs work in the real world for anything apart from writing essays, then I think this is the way to go.

Speaker A: And how does it compare with the guardrails and can they be used together, which is the best way to do that?

Speaker B: To be honest, I don't know. I haven't tried guardrails. Over the last year, year and a half, there's been such an explosion of frameworks that it's hard to keep track. I've heard of DSPY also, but which I think does something somewhat similar, but I've never had a chance to try.

Speaker A: And how is it comparing with the LLM cores and similar available tools? Sorry, could you repeat that the question. I was also thinking what the question entails, but. So how does it make sure that the LLM calls the right tools they're available to us?

Speaker B: I think in the validation, even the validation pattern, you only give the LLM the one tool. So that makes it easy. And if you if for the object modification, I think if you have half the trick is to keep the number of tools down. But if you have like less than a dozen tools with intuitive, with intuitive descriptions, LLM is pretty good at calling them like it's pretty good at obeying instructions. It's just not good at not doing other stuff as well. And the tools enforce that.

Speaker A: Okay. Do you have multimodal experience with music and audio?

Speaker B: No, I never had to generate any of these things in my professional work. As inputs we use a lot of images like charts, but not as outputs. And for outputs, for multimodal outputs, this would be. I'm not sure how you would do this because the point about the validation tool is that it's deterministic and it would be. Of course, you could have some kind of an LLM judge as well there, but then you get back into the stochastic country.

Speaker A: And how would you give a validation tool to an LLM prompt? So how would you use it and define what is valid? So within the prompt,

Speaker B: so the validation tool, you have to write some code and just write whatever validation logic you want. And then if you just want to use it as a tool. Any framework allows you to write custom tools now. LangChain, Lang Graph, palamaindex, you name them. If you want specifically for validation pattern, you need to have custom flows and then, well, motley query does it obviously langgraph can do it. You can just try to custom graph literally that I drew there. For other frameworks, I'm not so sure really. There has been an agenda framework a month at least lately. So it's hard to keep track.

Speaker A: And what are the tools for the semantic layer?

Speaker B: So the tools would be just modify. So the natural actions like add a measure, remove a measure, add dimension, remove a dimension. The nice thing about the semantic layer is that it's such a simpler attack surface than full SQL complexity that. Well, you look at your business problem, you see what your users might want to do and those things are what the tools should be doing.

Speaker A: And what about the image generation, is it applicable? Is it similar?

Speaker B: I haven't tried, but I think it would be again sort of the same problem. How do you validate your result? So you could have an LLM judge as your validator, maybe a different LLM, but that still would be, but that still would be hard. And by the way, one other side effect of using validation pattern everywhere is I don't really have an opinion on different LLMs because it really doesn't matter. The poor LLM has no choice but to return valid output. So as long as it's something released this year, it's probably going to work.

Speaker A: Synthetic data generation this is the next question. How close was the result to the real data set and what is a good methodology to use for the data generation?

Speaker B: So if I. So I did not try there to reproduce the real data set because for the real data set like the last thing I looked at was synthetic data Vault was a great library for that if you want to actually reproduce the structure of something real. What I wanted is more narrative based generation. So I have, so I want to have a synthetic data set which represents an imaginary company and then there is two stages. Stage one I write a prompt describing database schema in a very loose wordy fashion and then I use this pattern to generate the actual tables, the schema with the foreign keys and everything. Stage 2 I want to populate it and for this actually I was using headless cloud code just because I could write my own coding agent probably. But it's really not a good hobby this year. Last year it might have been worth it. This year cloud code API is good enough, but again so I just give the cloud, I just give cloud code, I just give cloud code the description of what the code should be doing. Like this is a plus, this is plus one distributed and then every user time the user joins and start generating reports and so on and so forth and then the cloud code generates the Python code. But then again there is a validation wrap around it like this Python code must run correctly, must populate, must touch every single table and so on. But it's important to use the right tool for the task. And I think for code generation I wouldn't use royal oms, I would use something like Claude code or Ador.

Speaker A: Headless people are asking for your slides.

Speaker B: The slides, well I've given them to the organizers. I will also post them on my LinkedIn and my email is very simple egortly AI so drop me a line as well if you want. In fact if you want to discuss any of this or try using it. I'd love to talk about it.

Speaker A: Thank you. And as a closing question, you said that the past one and a half years were really busy and lots of things happened. So how do you see what would happen within a year? And what is the outlook for you for the next 10 years, if you can judge anything about it?

Speaker B: Well, outlook for 10 of the next 10 years is anybody is, of course, anybody's guess. It does look like the current wave of just making the LLMs bigger is saturating. So they're getting more reliably, they're getting more reliable, but it's incremental, which is also why we founded Motley, because it's not about doing the latest, greatest bleeding edge thing. It's about using LLMs to remove friction so you can do the same things you used to before, but now the tool has common sense. And if you tell it, you know, if you just tell me, just tell it. Add a slide about X, it generates the slide with the right templates, the right configs and so on. And the slide itself is deterministic, but the LLM helps you get there. So I think it's this kind of convenience aspect of LLMs, where they are used to generate the same things like now, but. But more intuitively, that is the next big wave of applications.

Speaker A: Thank you very much, Igor, and a huge round of applause.