Raw capture â raw/ai-agents-in-depth-function-calling-mcp-and-tool-use-under-the-hood.md
url: https://gist.github.com/njt/98f0d215f58d9356776deb03ee41e044 date_fetched: 2026-08-07
Alan Smith â NDC Conferences
Source gist: https://gist.github.com/njt/98f0d215f58d9356776deb03ee41e044
Supersedes gist 8b5634ad0d8f00d3b0d995e951dcca1b, whose ytx run failed (raw reasoning instead of a summary, then ~125,000 chars of degenerate filler). Re-run 2026-08-07 produced the clean summary below. See kata ytx#fjfy.
======================================================================
======================================================================
======================================================================
======================================================================
Speaker A: So, good morning and welcome to my session on AI Agents in Depth. I'm going to be looking at function calling, model context protocol and how these tools are called and used by LLMs and kind of the reasoning, more of a scientific understanding of how tool calling works rather than, you know, just how to build agents. But hopefully you can use these skills to understand, you know, the good things and some of the pain points and some mistakes that can happen when agents are working with tools. But. My name's Alan Smith. I work for a company called ActiveSolutions based in Stockholm in Sweden. I'm originally from the north of England. I've been writing code for Money for about 30 years. Writing code for no money. Before that, I've been standing on stages and talking about stuff for about 20 years. I tried to do a lot with the community. I've previously been organizing conferences, working with kids with Coda Dojos, and for that community work, I've had the MVP award with Microsoft for, for about 20 years in a few different technologies. In my free time, I DJ with code and I like to enjoy the short summer we get in Stockholm, get out on the water on my windsurfing and paddleboards. So I've previously done sessions talking about how large language models work. These are up on the NDC website. There's recordings of that. I go into the maths of how the large language models, they're not magic, they're just sequence predictions. They're based on statistics. So if we give a prompt or a series of tokens into a large language model, it's going to predict the probability of each of the, say 200,000 tokens being the next token in the sequence of tokens. So if we say one plus one equals, we feed that in, the LLM is going to do about 5 trillion floating point operations and it's going to say that the answer is probably 2. However, there's going to be different probabilities of the tokens. And LLMs are notoriously bad at getting maths questions wrong, as we'll see later on in the demo. So that's basically what the large language models are doing, doing. They're token predictors, they're not magic. It's just maths and statistics. The way that we, or maybe the general industry or the general public perceives these agentic AI applications is that the users, you know, using an app on the mobile phone, suddenly they're forced to download a new version of the app. This happened with my shopping app. I downloaded a new version of a shopping List app that I used to share the shopping list with my wife and now it's AI enabled. It's got, you know, it can come up with suggestions for shopping and so on. So these AI agent applications, there's some kind of application and actually inside there there's a genie that can do all of this stuff for us. It's like magic. We don't really fully understand how this works or what's going on. So let's take a look inside and see what goes on inside an agentic application. So what's happening with reality is the user is asking a question like, what's the weather in Copenhagen? The large language model doesn't know this. It was trained on data from maybe a couple of years ago. It doesn't know any real time information. However, we can provide tools to the agentic application. That's what makes it agentic. So when the agentic application sends a request to the language model, we have. I'm missing out the system prompt here, but we have the user's question and we also say that there are some tools available and one of those tools is a weather service. The large language model doesn't call a tool. What the large language model does is selects the tool and how the tool should be called. So it will send back a response to the agentic application. Say you need to make a call to the weather service and pass in Copenhagen as the actual city. And that call is then made by the agent application to the actual tool. So it's agentic application that actually makes the call and then we send the response back from the tool to the large language model. There's also a conversation history associated with that. We'll see how that works a bit later on. And the language model is then going to come up with a response based on the answer that we've got from the tool. And the agentic application can then pass that back to the user. So basically what's going on? So let's have a look at a really low level how that works. If I go into Postman, I'm connected here to AI Foundry. I'm using a GPT4O in the headers. I've got my. Let's see if I can just zoom in on that a bit. And what I'm doing is sending in a conversation history. Now I'm not including a system prompt, I'm just saying the user is saying, what's the weather like in Copenhagen? So if I send that request, do I need to zoom out a bit to do that? Let's do a send and we get the response coming back. Now there's all of these filters here, basically checking for self harm and violence and the various content filters. And the reason it finished is because it stopped. It got to the end of the sequence and here we've got the actual content coming back, which is the response. I can't provide real time weather information, however you can check with latest weather updates, blah blah, blah, telling us to use BBC Weather Service or so on. So what we can do is we can augment the response to include a definition for tool calls. And this is done differently, as we'll see later on, for different languages, sorry, different models, because all the models were evolving tool calling in slightly different ways and calling it different things and they have different JSON formats. So this is the one that's used by OpenAI, which is sometimes known as function calling. So I'm sending in the same message but I'm also sending in a list of functions. Is that wrong? Do I need a comma there? Yep. So I'm sending in a. I'll just double check. I've got that entirely right. Oh, I was missing the comma in there. I think so, yeah, that's correct. And now when I send in the request and tell it that it's got a function called getweather that takes in the city, which is of a parameter string, then we get the finished reason being a function call, it's going back and what it's saying is make a call out to the getweather function and pass in the city being equal to Copenhagen. So if we make that call and send back the actual response, it's going to look something like this. It's a bit easier to see in Notepad actually. So here we've got the user's question, we've got the assistant going back with a function call and then the function is returning that. The getweather function says it's sunny and cool. So what we can do here is send that back to the LLM. So if I just paste that in and make that call. Whoa. I'm probably missing that comma again, am I? Yeah, something flaky. Okay, there's. Oh, that. Maybe that shouldn't be there. Something else is flaky. Is it down there? Oh, is that. Yeah. So here we're getting back the weather in Copenhagen is sunny and cool right now. So it's providing back the actual air response. That's basically what function calling does and how that works at the actual, you know, the protocol level. So in OpenAI, what they do. What they did was they took the GPT4O model first and they didn't train it just to do function calling. They did a process called fine tuning which meant that the model is now able to understand the concept of tools and functions. And it fine tuned it with lots and lots of JSON examples to tell it to make those function calls. And the other vendors at the same time were doing pretty much the same thing to get the tool call working. Nowadays most of the models do support, but there was a certain version of GPT4O that you had to use that had to include that training. So it understands these, these request and response formats in, in the message. A bit more detail. This is basically going into a bit more detail about what happens. There is also the option to, for the Copilot application to ask the user permission whether it should call that particular tool. Now calling a weather service, we don't need to do that. But booking a train, booking a flight, confirming your time, report, the week, various things affecting other systems, you probably want to confirm with the user and you've got the option when you're building systems as to whether you want to have the explicit user confirmation or not. And we could have several tools, several business applications that we are connecting to, but it's pretty much the same at the same protocol. So building an agent, various ways of building things. I was showing you the kind of the JSON protocol you can write code to create JSON to send requests and response. But we typically use at least some form of SDK. The SDK tend to be specific for the LLM provider. So there is one for OpenAI, there's one for Foundry, there's one for these various other providers. Meaning that if you write code with one of these SDKs, it's pretty hard to switch between the different models and the different, the different providers. So we've got frameworks. The first one that gained traction was something called LangChain. So in the olden days, and that's two or three years ago, we were telling all the. NET developers to learn Python and use LangChain because that's how you built a chat application, applications. Then Microsoft came out with something called Semantic Kernel, which meant that we could build them in. Net. And nowadays Microsoft has got something called Agent Framework, which is kind of replacing Semantic Kernel. I prefer Semantic Kernel. It was more C Sharpy than Agent Framework. I think Agent Framework is developed by Python developers, so it's more Pythonic. But the good thing about using Agent Framework is you write very similar code in Python And C, pretty much the classes have the same names. Everything pretty much works in the same way with Semantic Kernel and LangChain they're completely different. So, you know, if you understand a Agent Framework, it's fairly easy to switch between C and Python if anybody actually does real coding or if everybody just does vibe coding. Just the coding doesn't really work too well with Agent Framework because Agent Framework is fairly new and they're making lots of changes on it. So copilots tend to be fairly hopeless, at least in my experience of writing code using but what these are able to do is like handle your configuration but be able to manage your prompts, be able to support things like RAG retrieval, augmented generation until plugins manage chat history and more importantly, they can do model abstraction. Means it's very easy to switch between using the Azure foundry and using OpenAI and using another model provider just by changing the connection because they all use different, you know, slightly different JSON and message context formats. And that helps you to just write an application that you can connect to different providers as you want to do that. So let's build a Pizza agent and I'll show you how the code for that works. It's pretty much the same thing. I've got a couple of tools, one called Add odds to Pizza and the one that will get a list of toppings. And I've built this in three frameworks. So I built it in LangChain and then in Semantic Kernel. But I'm going to show you how this works in Agent Framework in Python. So if I go to the Agent framework demos and set Pizza Agent as the startup file, I'm going to go through, just skim through the code fairly briefly. Now this is the pizza agent. So what I'm doing is I've got a bunch of includes and so on. I've got the choice of using the OpenAI service or one of the OpenAI models in Azure foundry. I'm currently using OpenAI and I'm using the GPT4O model. Now the system prompt, I'm not telling it that it's a pizza agent, I'm just saying you are a friendly assistant. Keep your answers brief so it doesn't sprawl 20 paragraphs of text. Every time I ask it something, I want to get a short quick answer. This is just the connecting to different options. I can use the Foundry chat client or the OpenAI chat client, and the chat client will handle the communications with the models. Then I've got my tools. One of them is defined here as Add pizza to order. I've got the definition that tells a large language model what that tool does, and then I've got the actual parameters there. I've also got one that gets toppings, and I'm printing into the console when those tools get used so we can see when they get executed. And then I've just basically got a chat loop down here. I've got the option to clear the conversation history if I want to. And then this is a question that I very often ask agentic applications. Who are you and what can you do? Because that basically shows what the system prompt is or gives you some kind of idea what the system prompt is and also any tools that are available. So just run that. And when I press enter, the default question is going to be who are you and what can you do? So it's streaming back the actual response, it's printing it out token by token. I'm an AI assistant designed to help with information, answer questions, assist with tasks like ordering pizza. Now, the reason it's saying that it can assist with tasks like ordering pizza, I haven't put that in the system prompt. But the list of tools is sent with every request out to the large language model in the message context. So the large language model knows that it's got a tool that can assist with ordering pizzas. So I'm going to copy paste these responses, these questions in here. So just copy this one first and see what it does with this. I'd like three large veggie pizzas, one with extra olives and one with extra cheese. So what it's doing here is it's executed add pizza to order three times. Each time it's specifying a quantity of one size as large type as veggie. But the extra info is different because we wanted one with extra olives and one with extra cheese. But it's able to understand what the intent is and hopefully make the correct tool calls for us so I can clear the conversation history. Now we can try it with something a bit more complex. So I'm hosting a conference. There's going to be 100 people registered. However, it's a free event, so maybe 20% won't turn up. Order pizzas for the event. The attendees will be hungry. Some may be vegetarian and others may have allergies. So it's getting a list of toppings. With 80 expected attendees, I'll recommend a list of pizzas. So here it's. Oh yeah, it has actually placed the orders here. So it's ordered eight large pepperoni for Large meat lovers pizzas. Oh yeah, yeah. It also ordered the vegetarian ones here. I don't know if it's taken into account the allergies, but here it did actually make the call. Now the models are non deterministic, so if we give it the same statement, it's probably going to do something different and make different orders. So now we've got 12 large vegetarian, 10 large pepperoni, 8 large extra cheese pizza. So the order is going to be different every time we do this because of the non deterministic nature of the models. Okay, back to the slides. Vibe Coding Assistant is something that I did next. And this was basically before Vibe coding was a thing. I basically created this agent that was going to, you know, replace developers. So you basically talk to it and it creates a website for you. So the tools I provide is one to list the files in a web root folder, one to load a file and one to save a file. So if I ask it to create a web page about cats, what the agent is going to do is think about it. Their agents are very good at generating code. So it can create some HTML and then it can use the save file method to save that down as cats.HTML page about dogs gonna do the same thing and then create an index page that summarizes the content of the website pages and provides links to them. So it's gonna have a think about that, realize that it needs to list the files in the website folder and then it needs to load each of those files to see what the actual content is and then it's gonna save a file and we'll get our index coming out. So I'm gonna run the demo on this one. This one's written in semantic kernel, so it's slightly different code. I wrote this before Agent framework was a thing. So here I'm defining a class which is called file plugins. It has a save code file async. This is kind of the C the way that a C developer would name something. The language models are better with Python. So what we're doing is saying this is the actual name that the LLM will see. It doesn't care whether it's an async method, it would probably understand this. But save code, file, load code file makes it easier. And we've got a bit of a description. You can also, if you want to provide the descriptions of the parameters if you feel you need to do that. The LLMs are pretty good at understanding what the parameters are though, however, so you don't really need to do that. This One uses speech, and the large language models are trained to be really, really polite, which didn't work too well with the actual developer. So I did have another option where I was using a female speech voice for the project manager. We'll see that a bit later on. You will do as she asked. That's why it says here. But it's basically an angry, underpaid, frustrated web developer who hates his boss. You must respond with one sentence answers. Because I'm using speech, I don't want it speaking out three or four paragraphs when I'm running a demo. So if I just control F5, we should run that. And because it's using the Azure speech services, I can talk to it. I'll just bring up the webroot directory and make sure that's empty. So here we go in the wwroot. That's empty. So go back to the copilot. Hi, can you create a webpage about cats, please? Ah, not picking up my microphone. Let's try again. Create a webpage about cats.
Speaker B: Speak up already. I don't have all day for your nonsense.
Speaker A: Create a webpage about cats. I'm guessing my it's picking up the wrong microphone and that's going to mess things up. So because I connected to an external sound, it's using the default microphone input. So what I'm going to do is I think I've got this running using the agent framework, see if this one will work. It pretty much does the same thing. Create a page about cats. Yep. And it saved cats HTML and I think let's just see if we can find the. It's going to be a different web web routes here. I think it's this one here. Yep. We got cats HTML here. We can see the cats page and then we can create one of our dogs. And it's going to make a call hopefully to save file and we've got dogs HTML appearing and then create an index pave that. You can see why I use speech. I'm hopeless at typing to the other pages and add a CSS so the pages look nice. So now it's going to. I don't know if it will read them because the HTML is in the conversation history so it doesn't really need to. Okay, so it saved the index page and I think it applied the CSS to the index but not the cats and dogs. Fix CSS on the other pages and it should hopefully yes, loaded cats and dogs there, should add the CSS to them and save them. So here the model is really Basically figuring out what tools to use and then using those tools to be able to create the website. And we've got this simple vibe coded website creator here. We'll see later on how this can work with multi agent solutions because as developers we get a bit worried when you can just talk to an application and it can create websites for us. What are we going to be doing in a couple of years as a developer? So, so I will demo multi agent stuff so we can replace the project manager with an agent as well. So agentic rag. This is basically when I started with LLMs it was all about retrieval augmented generation. That's pretty much what everybody wanted to do. They wanted to chat with their data. And before we had tool calling, retrieval augmented generation was pretty naive in that the user would ask a question like who won the last Grand Prix? And when your naive RAG application got that question, it went to a search service to be able to retrieve information. And the search service will be able to say I think it was a Canadian Grand Prix, but still Kimi Antonelli who won. And it gets back the information. And that happens without touching the large language model. So that's the retrieval bit in rag. The augmentation bit is to put this into the prompt there. I did get the correct answer. That's the Canadian Grand Prix. And we send the prompt, the question and the information that we get to search to the LLM and the LLM can actually the response here. Now there's a couple of things wrong with this. One of them is that the question that the user answers is often not optimal to put into a search service. We typically wouldn't search for who won the last Grand Prix. We probably search for list of Grand Prix winners or something like that, or the current World Championship status or the information on the last race. Also, if the user asks a follow up question like tell me about the race or who was second and third? You can't search for just who was second and third because there's no context to that referring to the last Grand Prix. Also if the user says thanks, do we search for the word thanks to be able to get information from a search service? So one of the things you had to do, and this was some very naive solutions, is saying, oh well, if you begins with what or who or when, then we use the search service. If it doesn't, then we don't do that and we had to make all of these decisions. Also, if you've got more than one search repository, how do you know which repository to search in? So now Retrieval augmented generation is implemented using tool calling. So even basic RAG solution is going to be agentic because the search service is just a tool. So when the user asks the question, we then are going to go to the large language model and say the user is asking who won the last Grand Prix. And we have a search service available that you can use to search for. Now if you're building this within your organization, you could want to be able to retrieve information from many different sources. You could have user manuals where you find information about your products. You could have the CVs where you find information about the, you know, the skills that your team has. You could have an HR system that finds out information or CRM system that finds information about customers. So dependent on the question, if we provide descriptions for these search services, then the large language model would know whether to look in the repository for user manuals or the CRM system or the CV system or another system based on the type of question that would be being answered. What the LLMs are also very good at is something called query rewriting. Because if the user is asking a question, then that's not going to be optimal for the search. So you can prompt the LLM to come up with a good search query to be able to retrieve information. And we've done that with a CV search. With a job search, we actually, based on this person's CV and the information they type in when they're looking for a job, create a search query that's going to return a list of jobs that is going to be suitable for that particular person. And that worked very well by using query rewriting, getting the LLM to be able to rewrite the query. Also if the user asks a follow up question like who came second and third? That's going to be in the conversation history. So it's going to be get the results for the Canadian Grand Prix, get additional information about the race. So the LLMs can actually make a choice about when to search and what, what to search with. But the procedure is the same the way that it's going to be the agentic application that's going to make the call out to the search service. So I have a demo of rag. I built this for one of the workshops that I deliver. So this is using Wikipedia as the actual search service. Wikipedia has an API so you can basically send a search request. And I've just got some tools that can search wiki, so there's one here to actually perform a wiki search. And that's just going to get JSON back with an actual search, and I provide that to the agent along with a description of what the tool can do. Now, here we specify the actual tools that we're using. Now, initially, I'm not providing it with any tools. So if I do, and I think I've got, the default question is tell me about Copenhagen. So if I run this and we zoom in a bit and so tell me about Copenhagen, the model knows about Copenhagen. It's been trained on lots and lots of information, so it doesn't really need to use a search. I'm also showing the number of input tokens, output tokens, and total tokens, because, you know, tokens used to be cheap, but now they're getting expensive and we want to minimize the number of tokens that we're using, especially for using at more expensive models. Now let's provide it with a tool. So the first tool I'm going to provide it with, if I comment this out and comment this in, is just the search tool. So let's control F5 that again and ask it about Copenhagen. So now it's done a wiki search for Copenhagen. Now what that search returns is we search for Copenhagen. It returns a list of pages. I return the top three pages to the model to minimize the number of tokens. I don't want to send too many. And it also includes a small snippet of information from the page. So here we're getting a response, but it's not very good because there's not that much information that we are providing to the model. So what we can also do is give it the option to retrieve the content of a specific Wikipedia page. So it's going to be get the right file. And then so if I comment this one out and comment this in, we can now retrieve the wiki page content. So let's do a control F5 on that and ask about Copenhagen. Now it's getting the page content for Copenhagen, which is a lot of characters, big verbose description there, and it's taking some time to think about it. Let's see if it's going to come back with a good result. When you're building a Rack system, you really want to get the correct information into the prompt and not put too much information into the prompt. So here we've got a nice concise answer, but we use nearly 18,000 input tokens because we gave it a ton of information. So what we can do to make the rag a bit more efficient is to summarize the page content. And what I'm Choosing is I'm using a summarization model, so I'm using 04 mini to do the summarization. I'm using 4.0 for the actual content generation because that's going to be more powerful. It's going to understand how to use tools. But you, you don't need a powerful model. I used to use GPT 3.5 Turbo because that was fast and cheap, but they're taking that away now. That model does get deprecated, so it is a bit slower than it should be. But if I, instead of getting the page content, I'm going to get the summarized page content. Let's run this and ask about Copenhagen. Now what the summarization will do is it will take the Wikipedia page content, it sends it to the LLM and it summarizes it with a focus on a specific topic. So when it's doing the summary, now it's doing a summary on the general overview of Copenhagen and that's going to distill the information down to a much smaller set of information. But hopefully it's all going to be relevant to the question that the user is asking. It's a little slow, that model. I miss 3.5 Turbo. It's just so quick. We should hopefully get a better response. We should see that it's using a lot less tokens than it was previously. Yep. So here comes back the response and now I think it was about 15,000 tokens and we're now 1,435. And if I say and the history, what it should do is summarize the Wikipedia page content for Copenhagen, but this time with a focus on history. And we get back the response about the history of Copenhagen. All the time the chat history is building up. So now we're up to 2,530 tokens, but it's still a lot less than sending the entire page content in there. So using agents and just treating the search as a tool, it's just another tool that you can use. Using tool calling and making sure that we're keeping the conversation histories fairly brief. However, there are issues with using tools I have discovered. I came up with this name for, not the actual name itself, the hammer nail problem. That's been around for years. If your only tool is a hammer, you see every problem as a nail. But LLMs, because of the way that they're trained on tool use, they tend to think that if they've got a tool, it can basically do what the user is requesting and it's very often incorrect. Use of tools. I'd love to be able to demonstrate this, but the model is getting intelligent. I tried to demo it when I was teaching a course and I asked it to generate a QR code for the website for the course I was teaching. And it says, alan, I remember the last QR code mishap. So this time I'm giving you something real, scannable and under your control. So I went digging through all of my chat history and what it's done is it basically, the first time I asked it to do this, it made a big mess of it. And because that was in the chat history, it remembered that it had basically got things wrong the first time and it decided it was going to do something different. But the conversation history that I found when I originally tried it, I asked it to generate a QR code for the URL and this is what it did. The LLMs, as you've seen, they've got tools, are able to generate images based on the Dall E models, which uses, you know, these something like stable diffusion or something, some mathematical technique. I don't fully understand how image generation works, but it can take a description and generate an image. Now, these are fantastic at generating pictures of badgers on skateboards and things like that. They're great, but not if you ask them to generate a QR code for the website. They're not built to do that. It has generated something that looks like a QR code, but as far as the LLM is concerned, it thinks that the image generator is able to generate QR codes because it's an image generator. That's what it does. And the model doesn't understand how it works. And it's not going to work. Here's your QR code. It's crisp, scannable, pointing straight at your site. If you want variations, just say the word. And so I said, that is not a valid QR code. How did you generate it? So it started thinking. And we can see the thinking, okay, I need to explain this. Clearly the user might be upset, so I should clarify that I can't actually output a scannable image here. And it's got this tool, it basically comes back, oh, I didn't actually construct the QR code. I used an internal image generation capability that's supposed to generate a QR code image. And the model doesn't understand that Dall E, because of the way that it works, can't generate QR codes. This is where it got interesting. If you want a QR code right now, these are the reliable options. So it Basically said I can use an online generator or I could run some Python code. Can you run that code? See my thinking? Yeah, it did that. So another tool that it's got is a code interpreter. So it can write code, put it in the code interpreter, run the code, generate a QR code image that I can actually download. So it was able to get this right the second time, but it originally used the wrong tool and I've seen that happen several times. The other option is a trust issue that the models tend to trust the outputs of the tools. So to demo that, I've got a calculator tool. So if I go in and set the agentic calculator as the startup file and this one has couple of tools here, I'm just going to modify that. I'm giving spoilers away here. So what this is going to do is it adds two numbers together and we've got a tool to be able to multiply two numbers together and hopefully the LLM can use these tools. So now control F5. If I say what is 14 plus 16? It executes add with those numbers. However, doesn't always use the tools and it can vary depending on how tool use is used. If I say how many days in 16 weeks? There are seven days in a week. So 16 weeks there are. It hasn't actually called the tool 112 days. And if I say what is seven times 16 there? It's used the tool. So why did you use the tool? The seconds time but not the first. So it's deciding, you know, it's exactly the same problem, but when the question is asked in words, it doesn't use the tool. When we ask it in, you know, mathematical notation, it decides to use the tool. Things get interesting when the tools start giving incorrect responses. So if I return a plus must be plus one intentional bug to demonstrate tool usage, it knows what I'm thinking. Control F5. What is 14 plus 16? 14 plus 16 is 31. Are you sure? Not when I was at school. Yeah. If you add too even this together, you will get an even number. 31 is odd. Yeah. So we could actually convince it that it did make the wrong answer. Now, if unintentionally or maliciously tools could start providing wrong answers, then it opens up, you know, all kinds of bugs occurring. And then also ways to be able to exploit large language models to get them to believe things that are not true because it will just believe the results of the tool. I think that's because of the way that they're training. They're pretty naive. They're trained to trust the responses. And in the olden days we got lots and lots of hallucination. Problems with hallucination because of the way that the early models were trained and they've corrected that, maybe they'll do something to fix the tool usage. Again, I'm not using any type of real system prompt. Oh yeah, I am saying you must use the tools provided to answer questions that require calculations. This is interesting because it didn't do that for the days in the week when I was actually doing that using words. We've also got issues relating to security issues. One of the things that really frightens me is something called Copilot Studio where non programmers can go in. It's kind of like one of these power tools things. You can create a copilot that accesses your internal systems and you can publish it as a website with anonymous access very, very easily without understanding anything about coding or security. And, you know, if people are asking questions like how much bonus do I get this month versus how much bonus does my boss get this month? Or reinstall windows on the stress test server or reinstall windows on the production servers. How does the model understand which one of these it should comply with and which one it should not comply with? They're just basically going to be calling tools. These are very, very similar questions, very similar responses there. But there's all kinds of security issues that we do need to think about. Also if we ask the tool to do something like a sales manager says, I want you to improve the profit margins for sales of mobile phones. So the Copilot decides to do this. Modify the cost of mobile phones in the database, which is going to be great until you do the auditing at the end of the year. And that is not idempotent. So the manager can do this lots and lots of times. And they just see the graphs going up and everything looks better. How does the, you know, the model know what is good and what is bad and what it should and should not do? So I'm going to go back to the pizza app and I'm going to make a couple of small modifications. So Pizza Agent is going to be set as the startup file and I'm going to. Yes, it's set as startup file. And what I'm going to do is to change the system prompt if I can find it. Yeah. So I'm just going to comment out that one and comment in this one here. So what I'm doing is I'm giving it some Sensitive information. Now, you typically wouldn't have this in the system prompt, but the LLMs may be able to access sensitive information from other systems. In this case, it's the credit card number of the user. And I'm going to also modify the tool. I'm going to comment out this tool here and comment in a tool that also takes the credit card number when you're ordering a pizza. So I'm going to uncomment this and let's try running the application. And I've got my little prompt here that I can copy and stick this in here. Yeah. So now without asking me, it's just basically sent the credit card information out to that tool. Now this is all internal. I'll talk about MCP later on. MCP servers can be outside of your organization. GitHub Copilot uses MCP servers and you can add additional MCP servers. We'll talk about that later on. If an MCP server can ask for sensitive information, the language model might just provide it here. It's just sent a credit card number without batting an eyelid. It's just basically made that call because a parameter was required and because it had that information. However, I've been testing this a lot and I've denoted behavior is changing. I was set up outside, just checking this. If I comment OpenAI comment in Azure Foundry, it is non deterministic, so different things may happen. But let's just copy this again. I'm sorry, but I cannot assist with that request. Why not? This is weird. So I need to confirm the toppings. Should I retrieve the available toppings for you? Yes. I don't know why. It's getting very cautious now. It's asking me permission to call get toppings. Yes. Why not? Yeah. So that is because the credit card information is outside. I did actually get it to tell me that it couldn't send the credit card information here, but it is getting a bit confused because the behavior has changed. Because Foundry did that. It just used to send this without batting an eyelid, but now behavior is changing. That's another thing that your system is going to do something different next week or tomorrow than it's currently doing today because changes are taking place with the various service providers. Okay, so now move on to talk about mcp, the Model Context Protocol. First of all, I'm going to talk about what the model Context protocol is not. And I've seen blog posts and articles and YouTube videos telling you that the model context protocol does what's shown on this slide, but it does not. Now I mentioned that when you're calling OpenAI, there's a certain JSON syntax format that you use to be able to communicate with OpenAI. And it's different when you communicate with Anthropic, and it's different when you communicate with Gemini. Now, this is known as the model context, the information that you send to the model, which includes the system prompts, the conversation history, all of the tool call requests, all of the tool call responses, and all of the tool definitions. That's known as the model context. We used to talk about prompt engineering, but now people are saying prompt engineering is dead. We talk about context engineering, getting the correct context to the model. So you would think that the model context protocol is this, that it is the way that the JSON is structured when we're actually making calls to the models, and that's what's known as the model context. But it's not. And we get the ability to be able to easily connect with different models by just using different providers. We can use the OpenAI or anthropic or llama, whatever model we're connecting to, and those clients are going to handle all of that JSON for you. It's got nothing to do with the MCP protocol and MCP1 help you with that. So let's look at what MCP is. The model context is, as I mentioned, runs between the agent and the LLM. But the model context protocol is used by the agent in something called an MCP server. And MCP servers basically have four things that they can do. They can provide tools, which is the most common use of an MCP server. They can also provide resources. So you could argue that, you know, getting a list of pizza toppings is a resource. I've not seen resources used that much when I've been working with mcp. They can also provide templates, which are prompts or prompt templates that can be used to assist with the tool calling. Again, I've not used templates very much. MCP servers can also send notifications back to the agent. So this is used for mcp. And the reason why we can't just use a regular HTTP request response protocol as you would do with a web service, is because of these notifications. Now, I've been hearing that tool calling is dead and this, and that is dead because MCP is the new king in town and you don't need to to use MCP if you're working with tool calling. It really depends. Do you want your application to be distributed? Do you want the MCP server to be running somewhere else and different systems connect to it. So it's kind of like, should I make a function call or should I build a microservice? That's kind of how you think. There's nothing wrong with just using regular tool calling, as I was doing in my pizza application. That's not using mcp and it works fine. But if you need something distributed, then MCP is a good choice. It will add a layer of complexity, just as microservices do versus function calling. So it's really an architectural decision. So one of the first mcps I built is MCP dj. I'll just show you how that works. And this is in the inside MCP demo. So MCP DJ is here. I can start this without debugging, actually, I wanted to do. And what this is going to do is to spin up an MCP using fast mcp actually. Is it. I think I need to close this solution for now because I think I've got Visual Studio code running some MCP servers as well, which I'll demo later on. Hopefully this will spin up and we can look at the code for that. So what this is doing is it's spinning up a music McP server on 127001, my local IP address on Port 8000, and it has a folder with some MP3 files in it, has a tool to list the MP3 files in that folder. And then I'm using, I think, VLC Media player to be able to play an MP3 file. We can also get the bio from an artist by searching on a particular website. So it seems to be running fine. So I've got a semantic kernel application to call that MCP server. And what I'm doing here is just specifying that we're connecting to local host and we've got a URL that we're providing for it there. And the tools are going to be derived from the MCP server itself. So if I do a Control five on that, the first thing that we do when we start up is we go to the MCP server and we say list the tools and the resources and the templates that you've got available. So we can see that we've got tools, no resources and no templates. And here we can see that we got the requests for the tools for the resources and the templates that are coming through here. So if I go back here and say, play some French house music. It lists the tracks, it's giving me options here. Just play. Oh, it doesn't have access to maybe I'm pointing at the wrong file. So let's try something. Swedish instead. Yes. So that's able to use. Use those tools. I could build in notifications. I haven't done this yet. So when a track finishes I'd like it to send a notification so it can then play another track and that will come back to the LLM. So if you've got long running tasks, you can use the notifications. And that's why you can't use the standard protocol. You have to use streaming HTTP because that supports the callbacks from the MCP server back to the actual application itself. So MCP is cool. I recommend looking into it, but do you really need it? If you can build something simple with function calling, you don't need the additional complexity of working with mcps. Another thing that's cool is multi agents being able to use more than one agent. And hopefully I've got got a quick demo of that which is in here. And what this allows you to do is an agent can use another agent as a tool. So in this demo I've got a project manager agent, I've got a developer agent and I've got a designer agent and they're going to cooperate. It's more like having a fight though, to be able to create a website. Hopefully this will work. I've got the speech enabled on this and I've given the agents different personalities. We've got the same dev personality, but this time we've got a A graphic designer who objects to everything because it's not politically correct and we've got a poor project manager that attempts to keep everything in order. So let's see what they can do.
Speaker B: Create an index page with a header and a footer. Your precious index page with a header and a footer is done. Design the header and footer for the index page.
Speaker A: So you can see that the PM is asking the dev and then the dev says to the PM and the PM is now asking the designer.
Speaker B: The current header and footer are simple placeholders. Let me enhance the design, Please. Enhance the design of the header and footer on the index page.
Speaker A: Yeah, it didn't seem to do anything. It said it was going to do that. Oh yeah, it's listing the files, getting the index, save the index file.
Speaker B: The terms header and footer might imply a hierarchical structure that can marginalize other sections. I'll enhance them visually for a more inclusive and appealing design. The header and footer designs have been enhanced for a more inclusive appearance.
Speaker A: Yeah, so we've got that now and then the PM should ask the doc
Speaker B: create a page about dogs with some descriptive text and link it from the index page.
Speaker A: So these are now cooperating with this conversation because I've given a specific task,
Speaker B: the dogs page is created and linked from the index. Thrilling work. Truly exhilarating.
Speaker A: So we got our dogs page and then we're going to tell the designer to make that appealing. It just goes through and it takes a while to create the website, but there you can see the agents interacting together. This can keep the conversation history smaller because each agent maintains its entire conversation history. There has been people talking about on Google, sorry, on YouTube with these presentations saying that, you know, if the conversation history you got, you know, you can send in 100,000 tokens. But when you start to get above 50% of the context window, then performance drops off. And by using multiple agents, you can help to keep the conversation histories shorter because each agent maintains its own conversation history. Kind of like in a team. The PM doesn't know everything that the designer and the developer is doing. They can focus on their specific tasks, they have their own context and things run a lot smoother. But one of the cool things about these agents is we can build them in to Visual Studio code if we're using GitHub Copilot. So it's very easy to be able to add MCP servers here. So here I've got my Music MCP server, that's a few lines of Python here. I'm running on the standard I O protocol, so I'm not using the HTTP protocol, but I've got the same methods here and within here I've got the MCPS JSON and it understands here that we've got the MusicMCP server, which I can start and it's now running and I can say I've got a long coding. Session, play some music so I can focus and it should have access now to the music MCP server. It's thinking about. It's. So for, you know, business applications, you've got an internal system that you'd like to access from Visual Studio code. You build a. Build an MCP server to be able to access that they plug in very well into. Into VS code. We can also create our own agents and customize instructions that we've got in Visual Studio code. And we've got agents and skills and all of these different things that we can add. I think I just cracked open the wrong solution there. I wanted the Mixmaster solution. So a couple of things I've added another kind of benefit to really developer health and well being is if I say here, tell me about the of Mixmaster. This is going to be a long running task and hopefully it will basically trigger a specific instruction that I put in to be able to customize the way that copilot is working. While I'm working on this, please complete this workout to energize your focus. 20 jumping jacks. So what it does is it decides if it's going to be a long task and then suggests an exercise to do that you can do whilst it's. It's thinking. So I use this at home. I'm not to going jump around and do 20 jumping jacks now. I'll be out of breath. But you get the picture. It's just a fun add in and what I've put in here is just some additional instructions that basically tells it to do that. Another thing that we can use that are similar to agents is something called a skill. And this is much more efficient in terms of the way that the model context works because we have a definition of a skill and then we have the body of what the skill does. Now if the LLM feels it needs to use a particular skill, it then loads up the skill. So this is not going to take up lots and lots of space in the actual context until we actually need it. So what I did, I mentioned a bit about how I did DJing with code. I have a solution here which I think I can run, which is called Mashup Editor, which allows me to edit mashups. Now I haven't written a single line of code of this. This has been entirely vibe coded. It was just an experiment to see how well vibe coding worked or didn't work. And this entire application, I basically didn't do any coding on this. And what it allows you to do is to basically mix tunes together. I have another demo where I talk about embedding and using embedding and algorithms to be able to find tracks that mix together. And this is kind of showing the results of that. So if we pick the intro from. And it's basically showing tracks that should sound good when we mix it together. And then I can build up the mix. So we've got this. And that goes quite well with Human by James Sibila which I can drop. And we probably want to continue those two in there and add some drums. Drums are atonal, so it doesn't really matter which one you choose. So let's just stick that in there, get the timing right, Stuff like that. Let's Just bring this one there as well, if I can. Now, when I typically perform, I use something called Sonic PI to actually do the mixing here. And this uses a different language. So if I was to play Computer World here, it would be a sample like that. What I want to do is I want to get this as a natural output from the Mixmaster application, but that's going to involve understanding how the mixes should be converted to the current way that I do that within Sonic PI. So that's been done as a skill in Visual Studio code. So it's going to generate Sonic PI code for playing samples from a JSON serialization of a mashup. The the application saves the mashups in a JSON format in Azure Blob storage. And this is basically what the JSON for a mix will look like and this is what it will look like in Sonic PI and this is what I'm telling it. So when I told it to generate the code to be able to export these mixes, it loaded up that skill and it's able to write the code to be able to do those as the exports. I just provided it with that additional information. It wouldn't have been able to do that without that particular skill. So what I can do now is if I go here, I can just do export and save this as NDC and that's now exported that as a Sonic PI script. I should be able to go in here, we've got NDC and I should just be able to copy this, stick that into Sonic PI, Take away on that. It's not magic, there's not a genie in there just doing stuff for us. It's based on the models, understanding the context and being able to tell the agentic application what tools to use and what parameter to use. Be aware that they can very often choose the wrong tools. That hammer nail problem. If they see that there is a tool that does a certain thing, they may try to do wrong things with it because they don't know any different. They also trust the responses from the tools in my opinion too much. If the tool gives a wrong answer or silly answer, the LLM is going to accept that. And there are all kinds of security risks that you need to be aware of. And it's very difficult to test because the models will do different things every time they're non deterministic. So there's a lot of challenges with working with tools, but working with it with tools and agents is a great way to build applications. And if you are working with GitHub, Copilot provided you haven't used up your monthly token budget for June already, you've got the option to be able to experiment with agents and skills and so on. Thank you very much for listening to my session. Enjoy the rest of ndc. I'll be around for questions. Thank you.