Raw capture — raw/ytx-linus-lee-context-engineering.md
url: https://www.youtube.com/watch?v=JMA9D8J9EyI date_fetched: 2026-09-22
Speaker A: So Linus, before we get started, I just wanted to ask you what's the motivation to go into vc? Your head of AI at Thrive now and you come from a product company?
Speaker B: Yeah, I've gotten this question a few times. I don't really think of it as moving from product to vc. I think of it as I've always operated from trying to find and answer questions that I'm personally interested in. And the question that I've been obsessed with for a long time is people's relationship to information, how people retrieve information, understand information, make sense of it, data visualizations, interpreting data sets, trying to figure out new and interesting questions to ask. In some ways I spent time at Notion trying to do this both in the product and then also in how we try to understand how people use the product. And when I came to Thrive, I was still interested in this in particular looking at it through the lens of model interpretability. Um, one way to look at models is as a way as a function that takes inputs and generates outputs. But another way to look at it as, as, as kind of a lens into the training data set of models. And so by applying models or training models in interesting data sets and then studying them, it's an interesting way to try to study trends and data sets that the models are trained on. And so I was thinking about these questions and getting more and more interested in how models can help us understand more and more complex data sets. And an interesting place where there's a lot of impact and value from understanding complex semi structured data is an investing. And so that combined with my general love of working with people who are building new and interesting creative things in the world, in the world of AI kind of led me here. But a lot of the work that I do today I still think of as fundamentally about helping people understand and make sense of more and more interesting kinds of data and datasets.
Speaker A: That's great. And what is it that you found most interesting in terms of the types of data sets that you have a chance to work with now that you might not have explored before?
Speaker B: I think there are a couple of different maybe lessons that I've learned or ideas that I've encountered. One that I learned fairly quickly coming from a product like notion to. Some of the things that I thought about more recently is the distinction and the pros and cons of unstructured versus structured data. Notion was very unstructured data pill in some ways where you kind of imagine everything that happens inside a company or inside the product as everything is at the End of the day, just a bunch of text on pages. And then you try to build up a world, a model of the world from that, where in the real world it turns out there's a lot of structured data already. And a lot of the questions people want to answer are not just looking for facts or looking for knowledge or stories, but also aggregating across time, aggregate across different dimensions, looking at a lot of quantitative data, trying to answer questions like how has the importance of this category versus that category shifted over time? What is the frequency of these events versus those events? And so you know, in the real world there's, you kind of can never get away from. There's, there's a level of elegance that I used to find in purely thinking of the world. It's just a bunch of text. And I think one of the things I came to realize is actually that is unrealistic in many ways. And if you want to answer really important questions that people actually care about, you have to be able to shift fluidly between working with structured data and unstructured data. And I think the way that language models can be harnessed for these two different worlds are slightly different. So that's one kind of dichotomy that I've thought about a lot of recently. Another interesting idea that's been on my mind, which I don't know if this will make the cut or not, but I'll just kind of talk about it, is the role that, how you index the data plays and what you can do with the data. This is something I also want to talk about more in my main talk. But one of the cool things that you can do once you have vertically integrated control of not just what data you have to begin with, but how you, how you ingest that data is you can do a lot of the, like massaging of the data to pre structure it when you're indexing it at indexing time or at right time. And then that allows you to make certain kinds of queries that you would never be able to make if all you had was just a collection of pages or record index. And so I, I've analogized it to many of my friends to trying to work with satellite imagery or web indexes or these very, very, very low level data sets where a lot of the make these datasets useful is pre structuring it, preparing out these interesting patterns or semantic units. And then you can make more and more interesting queries on top of it. And so how much of the work should you do at write time versus Read time is another kind of big question that I've been exploring.
Speaker A: And how do you think about that? Do you work backwards from what you want the output of the use case or the output of the query to be, and then figure out sort of what the data needs to look like the optimal data structure on the ingest, or do you have a mental model for actually affecting this in a system?
Speaker B: I think it's hard to answer these kinds of questions decoupled from a particular use case. And it also depends on the technology that's available, right? In most specific use cases, there's already something that humans are doing. Oftentimes if you go in the real world, people will be already doing a lot of the work of pre structuring the data into a spreadsheet, into a project tracker, into a table in Microsoft Excel or something. And so you can already look at kind of organic structures that have emerged, like the course of people doing the work manually, and then start with that as the base, and then think about the work that needs to happen as kind of indexing into that structure and then reading out of that structure. And so that's one good place to start. One of the more kind of academically interesting, conceptually interesting threads related to this is sometimes you can mess with the representation that you use to store the data. And so I haven't spent too much time recently thinking about this, but one of the ideas that I was really, really obsessed with a couple of years ago was whether you could use sparse vectors derived from interpretability techniques as a way to index data. So instead of just dense vectors, which are just, you know, you chop the head off a model and then whatever is inside the model as the data passes through the model is the embedding. But instead of just using those raw embeddings where individual dimensions don't mean anything, if you could apply mechanistic interpretive related techniques like sparse auto encoders to get a sparse vector where each dimension actually means something legible to a human, maybe you can use those sparse vectors where a dimension may mean a particular sentiment or dimension may mean the presence of some idea. And you could using a totally different representation of a document or of an idea or of a concept, you could index into that data and make kinds of queries that you otherwise couldn't make. And so I guess this brings us back to my answer to the previous question again, which is, in addition to thinking about how you retrieve the data, there's a lot of design space that's wide open in how you index the data and store the data that can totally open up new kinds of queries that you can make over the data set itself.
Speaker A: Yeah, I totally agree. Well, I want to acknowledge your talk coming up at AI Council, which sounds awesome. Context engineering on the frontier. I noticed that you comment on how context windows are an expensive and incomplete workaround in your abstract. What is the trap that you find most engineers fall into when they embrace this method of speaking to the model?
Speaker B: Everything is a trade off, right? I think even internally, oftentimes the first prototype that you build, you just try to throw everything into model context and it's a good baseline, but then you want to try to improve upon it for latency reasons, for cost reasons. Often I think an underrated reason to do these kinds of design interference is also for engineering flexibility reasons. What I mean by that is if you have a single agent or a single inference doing a single kind of monolithic, you know, inference stat, doing the full task and then you want to change some part of it. If it's a single agent, then changing any part of it could technically influence how the model performs in any other part of it. And so if you change something with a system prompt or change the description of a particular tool, you have to run all of the evals again of anything that touches this particular generation step because it's one monolithic system where I think one of the under discussed benefits of techniques like retrieval subagents is that not only is it saving costs and potentially saving latency and improving performance, but it also breaks the total system down into composable sub pieces where you can, they have better defined interfaces so you can iterate on just the retrieval subagent and make it really, really good. And then that doesn't impact the rest of how the top level agent interacts with all of his other tools. So that's kind of an under discussed reason to try to build bigger systems out of these more composable units. And I think that leads me to, I think one of the high level ideas I want to talk about in the talk, which is retrieval isn't just, you know, I have a tool or I have a collection of tools that calls into a data set. It's also what goes in building the data set and then also all the different gradations of steps of how the, how the data ends up in the final models context. You can use tools, you can kind of think of a sub agent as just a very expensive tool. So you could use a sub agent, which is a very expensive tool call. You can Also have a lighter weight tool that does heavier re ranking, but it's quite a full agent. You can, you could have a sub agent that writes structured queries over a structured data set of relational database, for example. And so I think the design space is less. Just what do we put in the context and how long is the context? And more, okay, we can spend the expensive tokens in the final models context and that has some trade offs, or we can push some of that retrieval work down into more and more specific tools that are less and less and less costly, but require maybe a little more design thinking. And so I think that kind of provided my answer. But the question on context is about how do you trade off between the benefits of putting everything in one monolithic context versus spreading that work out over smaller models or more dedicated tools to make sure that at the end of the day your metrics, which may be precision or maybe the completeness of the answer, or maybe the speed, how do you fulfill the final metrics that you want to do to fulfill performance using all of these, the full bag of techniques that are available to you from getting the information for the data set into the final models context?
Speaker A: Well, I've heard you say that, or reframe context engineering as a search problem. What exactly did you mean by that? Is that related?
Speaker B: Yes, totally. One of the things that I'm prone to doing is kind of take very modern problems and then try to call out how age old computer science problems relate to it. And actually earlier today I had this image which I'm probably gonna use in the talk, which is this comic From I think 1968 that was I think, made originally for IBM, which is about information retrieval. And it's kind of funny because it takes a very complex problem and tries to boil it down into these simple terms. But the comic basically says if mission retrieval is all about structuring the information you have into terms that the computers can understand, where you want to bring in just enough information to answer the questions that the user wants, but not so much that you overwhelm the computer. And that's really what all information retrieval is at the end of the day. It's like trading off between precision and recall by trying to translate the data into ways that the computer can understand. And it's just such a profound way to think about even everything that agents are doing. So information retrieval is one classic idea you can apply to think about agents. And then all of the literature from the last 50, 60 years of research into this domain you can bring in to think about agents, another classic idea is search, which is to say you define some space of options or space of documents that you are trying to optimize some set of values over. And then you have some way of either narrowing down your search base or enumerating the space so that you can iteratively search through it. And you can apply this, I think in a bunch of different ways. I think one of the classic techniques, very high level techniques in search, is just iterative refinement. In a lot of production search stacks, even unrelated to language models, you will see you start with a very large dataset, you have a very fast retrieval re ranker that kind of over retrieves and then you have stages of progressive refinement and narrowing where initially you're optimizing for recall and then over time you're optimizing for precision to get to the final result. And relating to what I said a moment ago of it's not just what's in the tool, it's actually all these different techniques that you could apply. I think an idea that I'm starting to use a lot more in my work and I'm starting to see elsewhere in the ecosystem is in the process of getting the data from the originating dataset into the final top level agents models context. You can apply the same mindset of iterative refinement where you over retrieve initially and then you use whether it's re rankers or subagents, or maybe you run multiple methods in parallel and try to union them together or try to merge their rankings. All these different techniques you can combine to have this kind of iterative pipeline before the data makes it into the final model. And it's one of the many kind of classic ideas from search we can bring into thinking about elements and context.
Speaker A: Very cool. Well, before I let you go, I just wanted to ask you one more thing. You mentioned that you've used tens of billions of tokens at thrive in your work there so far. I'm just curious to understand more about what that really means at the end of the day. And is there a failure mode at that scale that you feel like folks are not talking about publicly yet?
Speaker B: That figure came from when I spoke about some of what we built last year. I think now we are a bit or quite a bit beyond it, both in magnitude but also in terms of how we think about measuring progress. A lot more of what we think about these days is more about adoption and results and so are people using it, what are people doing with it? But regardless of whether you're measuring usage and outcomes or measuring inputs like tokens. I think one failure mode is as you get more scale, it starts to become harder and harder to actually know what's going on, how these things are failing, whether they're failing. It's easy to end up building these systems in a way where you don't even have a day to day sense of what percent of the runs that you're running in production are achieving successful outcomes as defined by the user. And so understanding observability monitoring, I think gets really difficult. And observability, I think a lot of folks in this domain used to refer to the health of the infrastructure and the software system, which is certainly one way to think about it, is like how many runs are completing successfully at the infrastructure level. But I think an even more important one that's even harder to stay on top of just intuitively is, are these runs or are these pipelines delivering results in a way that is sort of semantically successful in the way that the user would define it? Like when they want to generate a deck, is there a deck at the end of it? Or is the model kind of stumbling through and not delivering a text answer instead? And so one kind of domain in this area, one kind of work that I'm excited about in this area is can you build tools to help people as you scale up, stay on top of, and kind of get a retain like a hand feel for how people are using the tools that they're building? And are the models struggling? What tools are models struggling with? Like when you're, when you're using coding agents on your own laptop, you know intuitively what tools are hard and where models bump into problems, but you just lose track of that when you're running this at higher scale. And I think you can build tools to help pull that feeling back into your hands, recover that.
Speaker A: Well, that's great, Linus. Yeah. Very excited to hear your talk, Context Engineering at the Frontier. And thanks for chatting about it in advance of the conference and I look forward to seeing you there.
Speaker B: Me as well. I'm very excited.