Raw capture — raw/continuous-ai-with-don-syme.md
url: https://gist.github.com/njt/022cc46bcfca6d72131b198e9075ba0c date_fetched: 2026-08-26
You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Continuous AI is a distinct category alongside CI and CD. CI establishes that software works (deterministic). CD establishes that software is correctly deployed (deterministic). Continuous AI covers the subjective activities — bug triage, documentation, performance improvement, duplicate detection, issue labeling — that need to run on a permanent, operational basis within a bounded context, but don't have binary pass/fail outcomes. Don is explicit: "We don't say we're putting AI into your CI. No, no, no, no, no, no. We don't do that at all. It's CI CD, they stay as they are."
The shift that matters is from individual productivity to situated, collaborative automation. The AI industry has a blind spot: massive focus on chat modality and individual productivity, but not enough on automation that lives in a shared, bounded context (the repo) where teams collaborate. Continuous AI is about folding AI into the change/event-driven rhythm that already governs how software teams work.
Bounding context is non-negotiable. AI "goes off the rails unless you're situated inside some bounded context." The repo is the natural boundary — it defines what inputs the automation gets, what outputs it can produce (a PR, an issue), and has a firewall around it. This is why CI/CD is the right analogy: people already understand bounded automation in that form.
The repo is a site of production, not just a factory. Don pushes back on the factory framing as too narrow — a film studio is also a site of value production. Factories imply identical repeating tasks on a pipeline; AI use is broader. But the factory analogy works when there are repeating tasks coming through.
CI/CD's organizational model is the template for AI automation. When an organization adopts CI/CD, it tells developers: "here's a place called the repo, where we give you permission to automate within that bound." Developers get limited cloud resources (compute, network, storage) without asking the cloud team for a machine or a budget. Continuous AI should go the same way — empower maintainers closest to the software to decide how to use AI, with cost centers and policies flowing up to the organization.
Guardrails determine speed. "The stronger the train tracks, the faster you can run." In automated mode, you must be able to sleep at night. Information flow integrity matters: don't make private repos accessible to agents in public repos, don't triage issues from untrusted contributors by default, force opt-in for risky actions.
Human review is the bottleneck — so the factory creator's job is to equip the reviewer. Don's framing: "My job as a factory creator is to deliver high quality pull requests where the reviewer is equipped with all the information they need." This means evidence (before/after CI timings), risk assessment, trade-offs, and quality gates that throw away trash before it reaches the human.
Cost control is the missing feature in the harness discussion. GitHub Agentic Workflows has explicit cost controls — per-run cost visibility, budgeting, and a model comparison feature ("examination testing") that runs the same work with multiple models simultaneously to see which performs best. Don: "It's the thing that's missing from the harness discussion."
One workflow that does multiple things beats an agent zoo for production use. Don's collaborator Pelle de Hallio runs "Pelle's Agent Zoo" with dozens of workflows to explore the space. But for maintaining real repos, Don prefers a single broad-spectrum workflow (his is called RepoAssist) for conceptual simplicity, cost control, and attention management across multiple repos.
"We don't say we're putting AI into your CI. No, no, no, no, no, no. We don't do that at all. It's CI CD, they stay as they are. And continuous AI is a separate bucket of things. You can do none of it if you like, you can do a lot of it if you like."
Context: Don drawing a hard line between deterministic CI/CD and subjective continuous AI, addressing the fear that AI would corrupt the "place where the grown ups are."
"The stronger the train tracks, the faster you can run."
Context: On why guardrails in agentic automation aren't a speed limit — they're an enabler of speed. You can only run autonomous agents while you sleep if you trust the constraints.
"If it produces rubbish, you throw it away. Right. You know, the can, it's got a crush on it, you know, throw it away."
Context: On the factory reality that much AI-generated work will be low quality — and that's fine because the work was cheap. The job is quality gates that catch trash before it reaches the human reviewer.
"My job as a factory creator is to deliver high quality pull requests where the reviewer is equipped, and I love that word equipped. My job is to equip the reviewer with all the information they need."
Context: Don's core design principle for agentic workflows applied to GitHub's own CI performance improvement factory — PRs must come with evidence, risk assessment, and trade-off analysis.
"A lot of people listening on this might say CI CD, I don't want AI anywhere near that. Right. That's my determinism. And there's a reason for that, which is one of my friends kind of says it's the place where the grown ups are."
Context: Acknowledging the visceral resistance to mixing AI with CI/CD, and validating it as the reason continuous AI is positioned as a separate category.
"You've got to be able to sleep at night. You've got to be able to have those things running when you're not there."
Context: On why guardrails and information flow integrity are existential for automated agentic workflows, not nice-to-haves.
Continuous AI (concept) The category of subjective, AI-driven activities (triage, labeling, documentation, performance improvement, duplicate detection) that need to run on a permanent, operational basis within a bounded context. Use it to frame AI automation as a discipline parallel to CI/CD — not embedded in it.
GitHub Agentic Workflows GitHub's implementation of continuous AI — intelligent automation running coding agents (Copilot CLI, Claude Code, Gemini CLI) with strong guardrails inside GitHub Actions. Now in public preview. Use it as the default in-repo automation platform; bring your own model and key. Open source.
RepoAssist (pattern/workflow) Don's personal broad-spectrum repository assistant workflow — labels issues, does first-response research, optionally implements PR fixes (never auto-merges). Use it as a model for consolidating multiple automation tasks into one workflow for simpler cost control and attention management.
Pelle's Agent Zoo A documented collection of 20-40 agentic workflow patterns (code improvement, performance, styling, etc.) published on GitHub's workflow site. Use it as a pattern library to explore what's possible — but Don cautions that maintaining a zoo of separate workflows is a maintenance burden for production use.
Side repo pattern Running automation in a separate "side repo" that targets a monorepo or other repos. Use it to control who can write and run the automation (separate auth boundary) while keeping the target repo clean.
Examination testing A feature in GitHub Agentic Workflows that runs the same work with multiple models at the same time, picks the first model's result, but logs comparative performance. Use it to evaluate which models work best for your specific workflow. Critical detail: must run simultaneously because models will find each other's answers in the shared ledger if run sequentially.
Goal chasers (continuous goal chasers) A workflow pattern that runs once daily, making incremental progress toward a defined goal (e.g., accessibility bugs), producing one PR per day. Use it for enterprise-scale backlogs where a slow, cooperative cadence is preferable to dumping 1,000 PRs at once.
Skill libraries Reusable markdown-based skill definitions that can be imported into agentic workflows, either by reference or by checking them into the repo. Use them to share automation logic across repos — the execution stays in the workflow, the skill spans repos.
GitHub CLI extension (GHAW) The CLI extension for GitHub Agentic Workflows, which itself has agentic workflows checking its terminal output styling. Example of dogfooding: the tool improves itself.
Quality gate stacking The practice of building as many automated quality gates as possible before the human review point — CI green, evidence attached, risk documented. Use it to reduce human review fatigue; the factory's job is to never send trash to the reviewer.
Automated pushes to PR branches GitHub allows automated pushes to pull request branches (especially those created by the automation itself). Use it for loops that fix failing CI — find red PRs, make extra pushes to get them green — without auto-merging.
What happens when human review can't scale? Don insists the human is always in the loop at the PR merge point. Guy pushes back hard: "That's not a sort of, that's an untenable sort of destination, right?" Don acknowledges the pressure toward auto-merge but doesn't offer a path beyond "we're being responsible." The talk doesn't address how teams already drowning in PR review should handle a step-change increase in PR volume.
How do you evaluate agentic workflow quality itself? Guy asks directly: how do you know your automation isn't degrading over time, especially with self-improving loops? Don deflects to "the human is in the loop" and mentions GitHub Actions' built-in limitations on self-modifying workflows. He doesn't address evals, metrics, or how to define "correct behavior" for subjective automation beyond vibes.
Cross-repo and org-level context is acknowledged but unsolved. Don admits "we have work to do on making sure that the repo is not the only notion of context that works effectively." Multi-repo collections, org-level automation, and enterprise-spanning workflows are raised as real needs but left as future GitHub work. The monorepo coalescing pressure — driven by agents needing cross-repo context — is identified as a tension but not resolved.
Security model is sketched, not specified. Don mentions information flow integrity, not exposing private repos to public-repo agents, and defaulting to ignoring untrusted contributor issues. But there's no discussion of prompt injection via issue bodies, secret exfiltration through agent tool calls, or what happens when an agent with write access to a repo is fed malicious content. These are the actual attack surfaces for agentic workflows.
No discussion of what happens when agents produce subtly wrong work that passes quality gates. The factory analogy assumes trash is obviously trash — "the can's got a crush on it." But AI's failure mode is plausible-looking-but-wrong work that passes CI, passes style checks, and gets merged. This is the hardest review problem and it's not addressed.
No data on merge rates, cost per PR, or time savings. Don mentions "70-80% PR merge rates" for some zoo workflows and "cents to a dollar" per run, but there's no systematic data on whether continuous AI actually improves outcomes, what the ROI is, or how it compares to human-only workflows at scale.
The relationship between continuous AI and loop engineering is left fuzzy. Guy explicitly asks whether these are different words for the same thing. Don says "some of those conversations are orthogonal" and that loop engineering concepts "can flow very freely" into continuous AI — but doesn't draw a clear boundary or explain when you'd use one framing vs. the other.
No mention of testing, test generation, or test quality. For a talk about software factories, the absence of any discussion about how AI-generated changes are tested — beyond "CI green" — is notable. Who writes the tests for the agent's output?
Speaker A: I mean, cheap or not, it's the nature of the factory. If it produces rubbish, you throw it away. Right. I do understand, of course, that human review is a bottleneck. When you're creating a factory, you know, a lot of the work is on the quality gates. You know, there's just no point creating a pull request for the human to review when that thing is low quality. My job as a factory creator is to deliver high quality pull requests where the reviewer is equipped, and I love that word equipped with. My job is to equip the reviewer with all the information they need.
Speaker B: The AI Native Dev is a podcast for developers and engineering leads at the cutting edge of AI and agentic coding. Join your hosts, Guy Pijani and me, Simon Maple, every week as we chat with the most exciting voices in AI and tackle the biggest questions facing developers. Today, this is the AI Native Dev. Yesterday you were writing code, today you're prompting agents. Tomorrow you'll be running a software factory. Agentic coding is moving fast enough to give anyone whiplash. This June, AI Devcon London sold out. Hundreds of developers, engineering leads and VPs packed in to hear speakers from Anthropic, OpenAI, Netlify and GitHub. Cut through the hype and get to what actually works. This November, we're bringing it back to New York, bigger, bolder and full of people actually building the AgentIQ future. Three days of real talks, real tools, real answers. Whether you're directing the agents, building the loops themselves, leading the teams, or setting the strategy. Our early bird pricing ends on September 30th, so make sure you lock in your place before prices increase. As a podcast listener, you can get 15% off your ticket using promo code POD15. We have so much planned and and we can't wait to have you join us in New York.
Speaker C: Hello everyone. Welcome back to the AI Native Dev. Today we're going to talk about continuous AI. We'll talk about repos and what is the sort of the container for factories or whatever we call them on that front. And to dig into all of that, we have with us Don Sime, who is a long term kind of industry contributor when it comes to programming languages and how we develop big open source contributor. Worked a lot on F on various other projects you would have known. And today a principal researcher at GitHub working on many things, AI development, but specifically also kind of continuous AI and agentic workflows which we'll cover more. So Don, thanks for coming onto the show. I should also point out you've given an amazing Talk at our latest AI native DevCon. And you're a big part here of the London community of, of development as a whole, but also specifically now with as we transform into AI. So love having you as part of the community Guy.
Speaker A: It's a pleasure to be here and it's been a pleasure to get to know you over the last couple of years as you've created tesol and what a wonderful community here in London and also worldwide AI native dev community. I've used your resources a lot, I've learned a lot from what you've done and you know, really kudos to take that community ethos that was so strong in the developer world and continue it through the really big shifts that have been happening in the developer world. And yeah, I love what you're doing.
Speaker C: Cool, appreciate that. And it's all about gathering brilliant people and giving them a stage to share their learnings. So precisely what we're doing here. So we're going to talk a lot about how do you sort of systemize, if you will, software development in the sort of new era? And you use the term, you kind of individually and also at GitHub use the term continuous AI when you think about that type of development. So maybe for starters, just define for us a little bit, what is continuous AI?
Speaker A: Yeah, so the thinking is that there's been a bit of a blind spot in the AI industry, a really big focus on the kind of use of AI, basically coming from the chat modality, the individual productivity modality. Of course, we see that with ChatGPT just exploding in its initial and then all the kind of productivity enhancements that come around the chat modality. But we think there's kind of a missing point of view which is about collaboration and which is about automation and which is about situated automation. And we were thinking about this a lot last year at GitHub. We did some kind of thinking about, thinking around, about it. Then we kind of looked, well, you know, we already do a lot of situated automation in what in cicd.
Speaker C: Right.
Speaker A: We have continuous integration, we have continuous deployment. Then there are this whole range of activities that people want to do in this continuous modality. And the continuous modality is just so central to how we work in the, in the software industry that change is happening in this kind of context, usually the repository, but there can be other kind of contexts, the organization or multi repo. And that means continuous means you're kind of continually folding in that kind of sense of change and eventing and difference and sort of incrementality that's kind of happening and always re establishing this crucial invariance. Like the software actually works in continuous integration or the software is correctly deployed in continuous deployment. And a lot of these things are very, they're central to how software collaboration and deployment works. But equally we want something new which is also these subjective activities like continuous documentation. Now writing documentation is kind of a subjective activity. It's not the sort of thing where those CICD kind of concerns are about like determinism. We want everything to be exactly right and double check absolutely everything.
Speaker C: Repetition and sort of insistence that it always happens. Like a lack of, yeah, less creativity and more consistency.
Speaker A: Yeah, that's right. But now we have this whole new bucket of things. Like we want to continuously improve the performance of our software. We want to continuously fix bugs, we want to continually triage bugs or have every issue labeled correctly within some kind of schema. We want to continuously look for duplicates in our bugs. And of course you can kind of do this as these things, each as a one off, but in the end you're going to have to put them on some kind of operational, permanent kind of basis in the context of change. And it's so easy for people to kind of blank on that and kind of say, oh look, I can find the duplicate bugs in my repo, but you got to make that continuous, right? You got to get that on an operational, permanently running kind of basis for that to really be a sustained activity. So we looked at this kind of collection of things and we said, well what really characterizes this is they're different to CI, they're different to cd, but they're also similar. Okay? And it's that. And that said, well, let's partner that CI, CD and continuous AI. And the key things is continuous, but also kind of subjective and implemented with kind of AI. And that's where we land with that term. When we put that out there and it kind of seemed to resonate with people, it kind of captured something that other people weren't talking about. And that's our.
Speaker C: So I like, let me kind of echo back a little bit and the things that resonate with me. So it starts from the sort of the single player to multiplayer move on it and then I guess naturally over there weaves into automation, right? Because to one extent we talk about continuous, but I think what happens with AI is that it has this jagged edge and sometimes it works amazing and sometimes not. And we have this kind of a hard to determine points in time in which we want a human to intervene and Apply their judgment, which is really a blocker for scaling. And in that sense it's similar to continuous integration because before continuous integration it was non continuous integration, like someone manually running it and sort of running the sequence.
Speaker A: There was a lot of homebrew continuous automation as well, sort of gauntlet systems or something like your PR would run the gauntlet of your organization. And that kind of all came together into things like GitHub Actions and other CI cities kind of systems. And we see plenty of homebrew for people kind of automating parts of AI kind of workflows. Well, and it's just like, yeah, we should consolidate that into systematic ways of thinking about it and systematic ways of actually kind of doing it in practice performing.
Speaker C: So it has these maybe like two different routes, one going from single player to multiplayer and the collaboration element to it. The other is the automation. And so do this repeatedly maybe starts with sort of homebrew automation and doing it and then consolidating those into the sort of infrastructure that is CICD usage style, but extended and expanded into the type of actions that AI can now do.
Speaker A: It's exactly
Speaker C: naturally less deterministic in nature. And so that really leans into not just the automation but also the definition of how would you track this activity? What is a failed build, you know, versus a successful one? How do you continue that does that? So all of those kind of go into that term of continuous AI.
Speaker A: Yeah, yeah, absolutely. And there's another key aspect here which is about bounding the context. I think we're going to kind of talk bounding the context of automation and bounding the possibilities of what happens in the automation people conversations about some kind of automated. I often go AI go off the rails unless you're actually situated inside actually some bounded kind of context. We're going to do this in the repository with these particular outcomes. Like it can create a single pull request, it can create a single issue. Whatever the job of the automation is on each round it gets these particular kinds of inputs and doesn't get anything else. It has a firewall around it and so on. So to have these kind of bounded, easy to understand, simple, kind of simplified view of what, what automation are we actually doing here? And because people are already really familiar with CI CD automation, a lot of people use GitHub Actions or some other kind of CID systems. All sorts of people can successfully kind of use those. It's really a natural starting point for saying, hey, let's automate, let's do this together. Because we're all doing it already. And now let's kind of automate some AI driven subjective kind of elements of this as well. And so situating it there does make a lot of sense. It's also counterintuitive to a lot of people. I gotta say. A lot of people listening on this might say cicd. I don't want AI anywhere near that. Right. That's my determinism. And there's a reason for that, which is one of my friends kind of says it's the place where the grown ups are. It's the place that really establishes that your system actually works as it is. And a lot of people have plenty of problems with SEICDC systems. It's not like it's so important. That's a major focus for team activity, is just kind of get this invariance established there, get the green ticks and get everything kind of checked there. They use it for everything because it's so useful. And the thought of adding kind of more subjective activities into that like labeling or issue triage or bug fixing, that can be disturbing for people. And it's one reason that we don't say we're putting AI into your CI. No, no, no, no, no, no. We don't do that at all. It's CI cd, they stay as they are. And continuous AI is a separate bucket of things. You can do none of it if you like, you can do a lot of it if you like. You can dial that as you like. That's really, really important that we're not mucking with the core ethos of what CI CD does and how crucial that is in the industry. We want people to double down on that because that's where we really establish software equality in the industry, I think.
Speaker C: So I relate to that. I think on the context of context, of defining context and defining it, I do think that there's a lot of analogies to the world of CI CD in that environments are actually a core element of CI CD as well. They had us move up from works on my machine to something that is defined. Continuous deployment is about the environment of the production. And so defining the context is just, just as probably, I'm saying it in quotes because it might actually get fairly complex, just a new type of environment variable for it. But it's a good tee up though to a question I had, which is you use the term continuous AI. I feel like there's a lot of analogies in the continuous AI narrative to the world of sort of factories and factory lines in my, in my own kind of a Native devcon talk. I talked about sort of the software stack. And in one of those paths I was a little bit stuck with saying, okay, you have models, on top of which you have tools, on top of which you have skills, on top of which you have harnesses. They all kind of compose down and then the harness. It was hard to know what's above a harness. And you know, I wrote their factory line. The term loop became popular like the next week or something. So maybe that's the term it now you have this continuous AI. But I guess, do you feel those are just different words or is like a system harness or a factory line or even a loop? A loop has a little bit of that optimization piece to it, the same as continuous AI, I guess how much is just a different lens to arrive at the same destination versus you're defining something different.
Speaker A: I think some of those conversations are orthogonal because the shift that I'm most interested in is this one from the kind of individual productivity point of view to something which is more continuous and kind of team and kind of integrated into some particular kind of context. And so a lot of the things that are kind of getting developed, like the notion of kind of loop engineering, you can apply that all in a kind of continuous AI setting. The ledger of record might be different because we typically use the repository and issues pull requests as the actual kind of ledger of record of what happens and maybe some augmenting kind of things as part of that. Whereas maybe if you're kind of loop engineering on your local dev machine or something like that, you might be using a different kind of ledger and so on, but the concepts can actually flow very freely between the two. And I really like that. It's like there's a lot of conceptual kind of competition in that kind of like what comes next kind of space in the individual productivity space. And I kind of like saying, hey, that's not what we're doing. We're over here and yeah, do all of that and bring it over. Bring over all the good ideas and rethink, rethink them, how they kind of work, what the loops kind of mean. It might be in lots of steps of a loop, like you get to see, you set the continuous AI set up. Like we have goal chasers, for example, continuous goal chasers, which once a day try and make progress towards some particular defined goal. And each day you get a pull request towards that goal and you kind of integrate those. But maybe it won't work on the next pull request until you've done the previous One So is that a loop or is it not a loop? It's kind of a loop because it's kind of repeating those pull requests towards the goal. But it's a cooperative loop and it's actually quite slow in its kind of action. That might be what an enterprise exactly wants to chip away at. All the big backlog of accessibility bugs or something like that that a product has. Those kind of loops are actually really satisfying for teams where they have this cadence of daily improvement towards some kind of goal that is measurable. You can see you're making progr and managers can kind of understand what's kind of going on. Whereas, you know, if it's in the kind of loop engineering world where the individual developer is like, yeah, I fixed a thousand kind of accessibility bugs and here's 1000 pull requests for the team to review. Let's you all have to spend or drop everything you're doing and kind of review those. You know, that's kind of a little bit like a little bit disruptive to how teams work. So we kind of get to shape things a bit differently. When you slow things down a bit and kind of move in different kind of pace.
Speaker C: I feel like I relate to the distinction between the single player and the multiplayer. I think that is very important. There are still sort of two things here. One is the pipeline. So the continuous AI element of the pipeline, this is an aspiration to get to a point in which you can get a recurring or a repeatable motion of getting a type of input to come into the pipeline and process it and get a certain caliber output out of it in a repeatable fashion. From the. Maybe it's even two other aspects. What I would say is a loop, which is more look at how that was done again and again for it and improve it. So a self improving pipeline because you can apply a loop cicd. I mean, in fact we do it and we do it for ourselves. We do it with the Tesla agent for others, which is fix flaky tests. So flaky tests are an example of something that isn't necessarily a pipeline in its own right. Maybe it is, but it is about identifying failures in the CICD and fixing them. But I think that's the loop. And then I think debatable a little bit on the term factory where there is proactive initiation of a thing, of something that is like an event driven. A bug was reported that's probably not its own term. It's probably like either. We're talking about two things, the pipeline and the loop. Is actually the initiator. So do you agree with that sort of separation? We have the pipeline and maybe the loop and the initiation of it. And how does that relate to the GitHub agentic workflows? Like the sort of the core capability that is from GitHub? I mean, what is the scope of that product? Is it both of them? Is it more, is it others? Maybe tell us a little bit about the product side.
Speaker A: Yeah, yeah. Because we've talked conceptually about continuous AI, and I'm very excited to say that GitHub Agentic Workflows, which is our implementation of continuous AI system for GitHub Actions, is now in public preview and it is part of the GitHub product lineup. And so get out there and kind of use that. So what is GitHub agentic workflows? Think of it as sort of intelligent automation for GitHub at the high level. And that concretely that means you're running the coding agents you know and love, the ones you're very familiar with, you know, your Claude codes or your copilot Clis or your Gemini Clis and the like, and you're running those with Strong guardrails in GitHub Actions on. So that's on an automated kind of basis. And all of those elements are kind of crucial to us. First of all, it is intelligent automation, continuous AI like we've kind of been talking about. You get a choice of coding agent. We're not tying you into kind of copilot, although absolutely, that's kind of the default one. It's a wonderful, wonderful coding agent. And you get the choices of models and you bring your own key and lots of kind of things about that. It's all kind of open source as well. You can kind of look at it, look at, look at it all. How it works and with strong guardrails is absolutely key. And I'm kind of. We're going to be talking more about security and guardrails later on, I think. But, you know, the better the guardrails are, the faster you can run. With automation, it's like train tracks. You know, the stronger the train tracks, the faster you can run. This is so crucial and so many people are kind of missing this in that, in the automated modality, in the continuous AI modality, you've got to be able to sleep at night. You've got to be able to have those things running when you're not there, and responding to things and trusting those kinds of things and making sure all the kind of integrity and information flows are kind of right and reviewable to check you're not kind of leaking information.
Speaker C: Had Orey from Sarah on the podcast here and he pointed out like if you allow the agent to do something and you run it enough times, it's almost inevitable that it will eventually do that thing. So you have to set up.
Speaker A: That's right. You gotta set the guardrails, you gotta set the constraints really strongly and information flow integrity. And you do need to think about what you're doing. You shouldn't make your private repositories kind of accessible to agents running in public repositories. That's just asking for trouble. You've got to know your basics of agentic security. An agentic kind of information flow that people bringing in issues. If you're going to triage those issues, you've got to think twice about triaging issues from third parties. And we do that mostly through forcing you to opt in when you do certain actions. Like the default is to kind of ignore issues coming from untrusted contributors. For example, you got to opt in if you write an agentic workflow that's going to process that kind of information. So that's what GitHub agentic workflows are. Intelligent automation for GitHub running the coding agents you know and love with strong guardrails in GitHub action. And it's a starting point. We're going to see a lot of different implementations of continuous AI. We're going to see we have a thing called GitHub automations as well which are more for individual automations for individual productivity. There's going to be a lot of different kind of automated kind of systems. There's a whole spectrum of kind of things. This is how we're cracking open the space to say hey, that DevOps mentality of CI CD, bring it along to the AI.
Speaker C: It sounds like GitHub agentic workflows are in my analogy before maybe more the pipeline than the loop. So they are the means of interacting it they are an event driven system. An issue comes along or something, it might sort of trigger that. But they are not. It doesn't implicitly drive you to create a loop or not. It's sort of your choice. Now you can create another time driven schedule that will now do this on a daily basis. And so you can do that with the programmable system on it. But it is about an agentic pipeline.
Speaker A: Yeah. And you kind of think of it as one step of the loop as well. Like taking the issue and creating the pull request. Now taking the goal and doing one step of that, taking the backlog and doing a small part of that and implementing 1, 5, 10 kind of fixes in 1, 5, 10 PRS. It doesn't have to be one issue to one pull request. A single run might create multiple pull requests. It might process multiple issues, it might do other kinds of work. It might work off backlog as a source of input. It might also be a planning step, it might sort of decompose. That kind of. It might be a research step that's actually very common where it says, hey, here every day we're going to do some research on how we can improve performance in this repository. We're not going to do the work, we're going to do a lot of investigation of that. We're going to write a report on like what are your top five opportunities for improving performance. And then the human, the maintainer or the contributors can say let's break that down and actually take that to the next step and actually go and implement each of those kind of opportunities. Or actually just do numbers one and three. So there's kind of like, it allows you to create a lot of different interesting kind of back and forth assistive kind of flows fairly, they're not super fast kind of flows. It's like CI cd because you really are doing depth research that may involve building the product right, that may involve taking screenshots and doing research on the product. And that's where these things can be very different to other kinds of automation you see floating around the industry. Because we do have the expectation that a lot of the time these steps can do real software engineering. They can build the product, they can run, they can take, they can be working off production telemetry data, for example. They can be kind of getting the latest like 10 failures off your site and say let's prioritize those and write your research report on how's your site reliability going, surfacing all this kind of subjective information into useful reports and actionable kind of work to take things forward.
Speaker C: So that makes a lot of sense to me. And agents are amazingly powerful and you can run them. And cloud agents have also become, you know, much more sort of useful as we sort of run those.
Speaker A: So this is sort of situated cloud agents.
Speaker C: So, so that's a, that's kind of where I wanted, so I wanted to maybe shift and talk a little bit about the sort of the repo as the unit. So as I think about GitHub agentic workflows, which granted this is sort of a fairly new product on it. So I'm sure it'll expand many things. But one of the things that I feel if I was to explain why is it different, how is it different than any other cloud agent? You know, it feels like a lot of the core of it is just that sort of repo centricity. And one of the sentences kind of in the webpages there talks about turning every repo into a software factory. And so I guess it feels to me like GitHub agentic workflows are effectively cloud agents that are kind of repo native. So they get defined in the repo, they execute with an expectation that all of their activity and all of that which other agents can potentially also do, but it's just like a core element. Is that right? Is there something else that's sort of fundamentally different?
Speaker A: That's right. And that's absolutely a key core design decision about where we started with the CICD kind of analogies which are also highly repo centric kind of flows and automations. And there's a reason why in that there's just so much becomes simpler when you answer that question of what is the bound of operation? Where is this running? Where is the definition? Let's pair that kind of definition. There are pros and cons to that. And in the DevOps screen and of course that went back and forth for kind of many years. The definitions of the automation sit in the repo or not.
Speaker C: There's a whole world of GitHubs and such.
Speaker A: And on the terminology like the repo as a factory, I actually kind of prefer to kind of think of repo as a site of production, a site where things happen, a site where value is kind of produced that might turn out to be a factory, but we produce value in all sorts of. Of other kind of ways. You know, a film studio is not a, is not a factory, but it's also a place where value is hopefully kind of produced as well. So they're all. And the anal. The factory analogy is kind of most useful when there's kind of repeating tasks that are the same that are on the pipelines kind of coming through. But I think we should think quite broadly about since AI has such incredible breadth of use, we should look for other analogies and other kinds of inspiration. That user factory, sometimes that's great, but also use people. You do use words like studios or productions or workbenches and all sorts of other kind of things that kind of make most sense. So there's another crucial dynamic in the context of the Repo and the context of CI cd. And I want to kind of bring this out, which is that when an organization adopts CICD and adopts GitHub or some other similar platform, what they're really doing is saying to the developers in that organization, here's a place called the repo, where we as the organization give you the permission to automate within that bound you. The developers know best what automation you need for your repositories. You know best what CICD you need. Therefore, we're going to give you the permission and you can create these things called workflows, which claim some of those capabilities, claim the ability to create issues
Speaker C: like a security boundary and a decision authority boundary.
Speaker A: Yeah, but we give it to you. Go for it, do what you need to do. Okay, okay. Maybe we have some organizational guidelines, maybe they're reusable actions and so on. We kind of tame that a little bit to make sure we're kind of keeping a platform centricity in our organization. But this is an incredibly powerful decision because suddenly developers have access to a limited form of cloud resource. They have access to compute, they have access to network, and they have access to limited kinds of storage for caching and whatever else goes on. And that allows developers to be incredibly productive. And this is why CI CD is so successful, because there is a bounded context where developers are empowered to claim resources. They do not have to go and ask the cloud team in the company for a new machine. They do not have to go ask for a budget. The budget is somehow ultimately measured. But this is absolutely crucial organizational dynamic for why CI CD is so powerful and so successful. And we believe that continuous AI or that AI automation in context will go the same way. There will be a cost center where this stuff does cost money, of course, like cicd, there will be a cost center where that kind of flows up to. There can be accounting controls, there's visibility of where this goes. There's an authorization context that within that context, we let you use AI according to. We still have guidelines and policies and so on, but that allows incredibly creative aspects. And I think this really matters to me in terms of the kind of the jobs discussion in the industry. There's evidence that if you empower people close to the coal face in a sense to use AI, not, of course, not an actual coal face, metaphorical coalface, a little bit more. Because AI does sort of posh here than that on real coal faces. I get that. But then they find amazing productive uses for that, for these technologies. And So I love GitHub objective workflows because it is an empowering technology that allows the maintainers of the repos and the people close up, close to the software to work out how best to use AI. And you can have incredible conversations about this with people as you start to unwind, you know, talking to someone working on software for children and they say, look, the thing I struggle with is that my. The language used is getting written for 15 year olds, but I need it suitable for 10 year olds. And of course the AI can make those kind of judgments. The AI can read all your documentation strings and all your texts and kind of simplify the language down and just automate that and do that continuously day in, day out. It's just like incredibly productive.
Speaker C: I think. So I agree, but I want to challenge it on the two sides of it. So I think what we're describing here is we're saying a repo is a highly convenient unit. It is a space, a production site in which you can say, I consolidate a lot of my data on the repo and because of that I can define my automation over here, my code itself sits over there, or my other assets, potentially context can sit over there and such. I think there's a lot of merit to it. I think also the rise of GitHub Actions, which functionally have a lot in common with other CI systems which are successful in their own right. But the rise of GitHub Actions is probably because of that sort of convenience of being built into the repo. So a lot of value there. However, there are sort of two other counterfacts to it. One is the actions that happen cross repos. So we talk about CICD as though it's one thing, but actually CD oftentimes is not just sort of one repo. And so sometimes those things become a little bit tricky and they become another repo. So suddenly you have those paths and maybe even more. That one is maybe more of a side comment and we can get back to it after when we talk about factories as a whole. The other one though is like in the world of AI, there's been kind of this resurgence of the monorepo. It never went away, but I think it's fairly clear now that a lot of the agents, a lot of the models have been trained to work in the context of a single repo. But oftentimes they need information that comes from these other repos. And so the easiest solution to that is to consolidate everything into a single repo. But that starts breaking this kind of clarity of it's in the same spot. Because the monorepos can be beastly. I mean, they can be enormous, and they're no longer one thing. So using a repo now as a unit of access, a unit of security, a unit of empowerment, of who's allowed to do what, starts really kind of fraying at the edges. Especially what we're seeing. Like we're doing it here at tesol, which is field teams have access to sort of repos and do that because, again, it's so convenient in terms of agent access to pull them in. But you're kind of losing a lot of that identity of a repo as a project.
Speaker A: Yeah, I think you've hit the nail on the head. There are absolutely real tensions here. There's kind of a sweet spot somewhere in the middle where, like, for some kinds of projects, where just everything works really, really well in an individual repo, and we kind of recognize that when we walk up to a repo that's just the right size and it's perfectly set up for deployment, and everything is working really, really smoothly. And then there are these other tensions that are pulling things in various directions. I think at. At GitHub, we have work to do on making sure that the repo is not the only notion of context that actually kind of works effectively. That you could have a collection of repos as a kind of collaborative context, like friend repos or whatever kind of features you might add to that. That's one of the things that get discussed at GitHub, for sure. And then there's the question, the question of the organization as a kind of context, as a whole. But of course, organizations can be absolutely enormous and thousands and thousands of repos, and it's a huge question of kind of scale there. And even then, some you have enterprises which have multiple organizations and so on, which adds a new kind of question. And they're all kind of questions in all of this kind of work. And you're right, the coalescing effect into the mono repo, that was a trend. That was before AI. We see it in some projects I'm working on. I'm saying, hey, look, we should really consolidate this. Yeah, it's going to work a lot
Speaker C: better if we bring it together.
Speaker A: Bring it together. Like, there's just no point having these separate. So, yeah, the tensions you say are absolutely real. And you always see dynamics where people. People pull things apart and then they put them back together and they pull things apart. And I don't have a lot to kind of. I think there's absolutely work we can do at GitHub to tackle, to give more effective solutions to stop you being sort of forced to bring everything into one repo. At least give you more kind of choice and control about how that works. And there would also be the question of like auth within kind of mono repo. I'm not really an expert on that kind of thing, but I get that within those controls.
Speaker B: Hey everyone, hope you're enjoying the episode so far. Our team is working really hard behind the scenes to bring you the best guests so we can have the most informative conversations about agentic development. Whether that's talking about the latest tools, the most efficient workflows, or defining best practices. But for whatever reason, many of you have yet to subscribe to the channel. If you're enjoying the podcast and want us to continue to bring you the very best content, please do us a favor and hit that subscribe button. It really does make a difference and lets us continue to improve the quality of our guests and build an even better product for you. All right, back to the episode.
Speaker C: So I guess maybe concretely when we talk about agentic workflows, what type of automations would you say are sort of suitable to run within the context of a repo versus which ones are outside? I mean, you gave for instance the example of a repo pulling in production data example and running it. Clearly the production data is not naturally in the repo for I think many good reasons. And that system is elsewhere. So that's one example. Should that really be a repo level automation versus something that's outside? How would you suggest people start thinking about what automation should be done in repo versus which automations? I guess maybe you technically can do them in the repo, but there's no even shouldn't
Speaker A: we have on the GitHub workflow site some really interesting design patterns in this kind of space? And it's one of the things I think the team are doing really well, exploring the design patterns. So there are absolutely cases where we run a whole zoo of agents in a single repo and we've written that up as Peli's Agent Factory, Peli's Agent Zoo. And we've documented 20, 30, 40 agents there that are just really, really interesting repeated things that you want kind of happening in repositories. Some really very unexpected ones. A lot of them are about code improvement, a lot of them are about performance, they're ones about styling, for example, things that are maybe a bit really subjective and are bit unexpected, like GitHub, agentic workflows includes a GitHub CLI extension G H A W and it has terminal output and we want that well styled. And it kind of. So the workflow runs in the repository in this case and kind of runs the output and runs the tool and checks its outputs and say hey, you could kind of make this look better, organize it better. And that's actually really, really nice. And a lot of them have got like 70, 80% kind of PR merge rates. You can start to kind of track the causality of like of the repeating kind of action and get kind of statistics on it.
Speaker C: Can I pause on that a sec? On it. So I love those and you know there are many and I highly recommend people go check out. There's very, very useful automations on it. But that's a good example, like the styling one, which is, you know, the definition of that style. I mean it's could very well be something that you want to apply to multiple repos, is it not?
Speaker A: Oh yeah, absolutely. So what is the right kind of
Speaker C: pattern to think about for like where would you then store? Is that just a reusable action or reusable skills?
Speaker A: I mean, absolutely. So you import skill libraries. Right. So of course that's again where we. A lot of the things that are kind of emerging in the individual productivity kind of world, like whole libraries of skills. I know Tesla has kind of done work on these and yeah, bring them in to your agentic workflows and you can do that directly and just reference the markdown files or check the skills into the repo is another thing if you want to go to.
Speaker C: So it's an interesting boundary there of. So the execution might be defined in it and the skill that you sort of pull in. But if you want to create for instance shared access to system or such, you might do that through skills. The skills might sort of span repos. Yeah, and then there are, I guess mcps kind of serve as a similar type of shared access if you want to access this sort of gets managed in different ways, I guess all of those, the tools, the context and all those.
Speaker A: So to bring in production telemetry data,
Speaker C: they're all a part of the continuous AI narrative. They're just not. Because the continuous AI is broader than the agentic workflows.
Speaker A: Yeah, yeah, that's right. Good opportunity workflows can bring in improvements, of course, and a lot of those kind of things flow across. And you were asking like when to run things in repo and when to run things out. I mean GitHub is a very open platform. You can obviously kind of run automations outside thing outside as well. A lot of tools integrate with GitHub and that's absolutely great. I mean I think our point of view is to give, you know, there are for instance data integrity retention kind of policy kind of reasons why you might sort of convenient because you've copy the repo outside the GitHub platform, you've
Speaker C: already dealt with this, already approved it within your.
Speaker A: Org. Yeah, there may be legal whatever reasons, privacy reasons, whatever. You can and can't kind of take all that issue and pull request metadata and kind of take it out and then put it back in. So it's obviously we want a credible, powerful default kind of option to have in GitHub for these kind of workloads. We do of course recognize lots of kind of depth specialty security agents or penetration testing or all sorts of other kind of things that people do that will logically happen outside. Absolutely. We do see some people kind of doing a mix where their actual focus, an Israeli startup called HUD is one Mai was talking at AI native devcon where their focus is on producing production information. And they kind of need a way to kind of surface reports about what they're finding about bugs and problems in performance results through production probes. And for them GitHub genetic workflows is just a really easy way to do that kind of reporting kind of regular reporting into the repository kind of context. And so in that kind of sense the main intelligence, the main, the real kind of systems value add is actually over in the production probing that kind of gets done and maybe some AI processing that happens there. And GitHub Agentic workflows is kind of just a secondary role where it's kind of about using some intelligence to kind of surface information that's there. I think we'll see a lot of that kind of thing using it as a kind of auxiliary utility and other kind of flows to actually get that information regularly delivered into the repository.
Speaker C: Yeah, it sounds like there's sort of three tiers over there that you've got information that specifically relates to the code or other content that your repo holds. So those are the sweet spot. Like fine, you automated, you kind of, you bring the compute to the data, if you will, where it is, you have another sort of pair of things in which a repo might be a convenient way for you to run recurrence on it. And so the repo is just sort of. And you see that pattern often which is you create another repo like the whole world of GitOps and things like that are worlds in which the repo is basically just. It's purely. It's an automation kind of holder on it.
Speaker A: Yeah, that's kind of side repo pattern where you have an automation running in the side repo and the target repo actually some kind of mono something else.
Speaker C: And in that case you. And you get the benefit of the kind of enterprise whatever legals and security and such approvals. If you're already a GitHub user, it's kind of convenient then you might have a third bucket which are cases in which you need other sort of optimizations and specializations, whatever bursting to a lot of compute or things like that in which you might want sort of slightly different systems.
Speaker A: You might be applying some proprietary AI knowledge, our tool into your repository and that's kind of going to live off to the side. The side repo one is actually really useful for controlling the authorization about changing the automation as well. Like who can write the automation, who can run that, who's the factory runner in a sense. And so to have that side repo targeting a kind of control, a mono repo is a pretty effective pattern.
Speaker C: So should people. I want to talk a little bit about doing it right and maybe get into philosophy. But maybe one more tactical question. When people do the automation there are sort of these two again kind of push almost like monorepo sort of small repo style debate which is should you have one workflow that does multiple things in it? Because there's still a bunch of things you need to configure per workflow and all that. There's a bunch of convenience to it. To it. Or should you have. I think, I guess that's how I interpret the agent zoo to be which is, you know, I've got 100 different agentic workflows that are sort of running on it, I guess. What's your sort of bias over there? What's your guidance?
Speaker A: My particular bias is more towards one workflow which kind of can do multiple kind of things. One point of cost control. Like if you have a. One of the. Sorry, I should say my collaborator's bias, Pelle de Hallio is more to have lots of an agent zoo. That's why we call it Pele's agent zoo. And part of that's just exploring space like regular. You know, we don't. Pelly's amazing guy. We don't want to kind of constrain what he does and you know, we just like just go for it. Just create as Many workflows and let's see what comes out. Let's learn, learn, learn. That's that part of what we've been doing here. And but when it comes to actually putting automation, say into. I'm a repository maintainer for open source technologies and I don't, you know, you can create an agent zoo, but that's also a maintenance burden. Right. You know that you got to maintain those, you got to attend to that. And my attention like span is limited. Right. I don't want this to be sucking all my time sucking. I prefer to have kind of one point of attention, one point of. And I also maintain multiple repositories so I'm going to have to look at that across multiple repositories. Actually I've made myself a little IDE to kind of help with that to kind of attend to multiple automations running across multiple but in multiple automations in different repositories. So one example is a workflow I've put together called Repoassist, which is kind of a sort of broad spectrum kind of repository assistant. IT labels issues for you. It does first response to issues. So it sort of researches issues and responds to them. It can choose to implement a PR to kind of fix those issues. It won't merge the pr. So my regular daily working with Repoassist is to kind of look well what PRs did it create last night? Let's take a look at that.
Speaker C: But why do you prefer that for instance like, for instance you named there like labeling, researching and opening a pr. You could have just as well had three different workflows.
Speaker A: Yeah. So this is in sort of a trigger off it supervisor orchestrator pattern. And you can have workflows which effectively choose to delegate up to more powerful models. Is interesting kind of pattern or weaker models for kind of things that's important for cost control. So why do I prefer it? I think just conceptual simplicity. Just I just. It makes my life simpler and better
Speaker C: think about the one entity that you
Speaker A: just to have that cost control really does matter in this kind of thing overall complexity. I just find having five workflows considerably more complex than kind of having one.
Speaker C: How does cost manifest here?
Speaker A: Oh so these workflows are very often done in a scheduled kind of way. So you may not with issue triage you have a choice. You triage each issue as it kind of comes in and do depth research or you can do it in batch. We can say each day we're going to see what new issues have come in and we're Going to kind of catch up in a batch and actually maybe do kind of a batch of them in a single run, which can actually kind of give interesting kind of of results as well as it kind of collects information across. There are things in between that you could imagine having workflows which kind of have a service policy which says we're going to triage all issues within half an hour and do depth research on them within half an hour. That's kind of a policy. So then that sort of dictates how things kind of match. Yeah. So now if you repoassist is a scheduled workflow, so you choose the cadence that it kind of is triggered at and you hardwire that into the workflow and you have to go and modify the workflow to change that as things currently stand. And that's important because you dial up, you can dial the automation up, you can dial it back down if you're. If activity is very high in the repo, you kind of tend to dial it up. And if nothing is happening, well, probably you dial it down. The software is being maintained and there aren't many issues that remain to be fixed. Each of those cases, I can roughly estimate how much it's going to cost. You can see in the histories the average cost per kind of run. It might be in the cents, it might be in a dollar or something, depending on the models you're using. And so you've got a bound, a pretty good bound there. And I should add there are actually very strong cost control features in GitHub identic workflows. I think it's absolutely essential. It's the thing that's missing from the harness discussion. I think harnesses should have cost controls and kind of budgeting. That's an absolutely crucial kind of thing. And again, it's the kind of thing that becomes natural in the automation world, but it's a little bit unnatural in individual protocols.
Speaker C: It's also hard to know when you do the action again and again. Then you can kind of introduce an assessment, an eval, whether you can run it with metric.
Speaker A: Absolutely. We've added features that it will do exactly the same work at exactly the same time with multiple models. It will choose, say, the first one as its result, but you can turn that on for a day to get a sort of examination. Testing is what I call it, where you kind of exact. You're doing the models. A room full of models is doing exactly the same work at exactly the same time, setting them an exam. And that really matters when you're using a shared ledger. Because if you write anything to that ledger that hints at the answer to that, like a closed issue or something like that, those models will go find it, say, oh look, I saw another model kind of did it using this closed pull request or something like that. Now you've got to test those models at exactly the same time. So we've got a feature for exactly that which you can kind of turn on for a day and then you kind of get results about like hey, which models kind of work best, work
Speaker C: better in the exams here. So maybe let me veer us. We started talking about costs, which I would sort of qualify a little bit as like doing it right, you're sort of running it. And maybe I want to touch on two other topics. You know, one is maybe more around autonomy and sort of the risk of regressions. And as you rely more and more on these of continuous AIs and improvement, then the second is safety. Safety and autonomy are related a little bit. Just like what is it that it's allowed to do? So I guess I'll ask the sort of, the broader question is as I lean in, this is sort of beyond product, right? But as one leans into continuous AI and makes changes to the docs, to production, deployments, fixes, how do you know, especially when you think about loops and you think about systems that improve themselves over time, gets better and better on it, how do you sort of see or what do you see as a best practice for avoiding regressions here? Do you expect every one of these agentic workflows to generally default to want to have an eval? If not, when should they have evals to know that they are, they're not changing? When do you even define what's correct and how do you even define beyond vibes what correct behavior is?
Speaker A: Just to clarify, we're talking correctness for the software or correctness for the agentic kind of automation.
Speaker C: So let me kind of refer. So we're talking about automation. We talked a lot about how do you sort of systemize the automation of an AI workflow. But what we didn't talk about that much is around the fact that those workflows are non deterministic. And so you run them enough times and they might make mistakes, they might kind of cause problems. And so I guess what I'm asking around regressions is when we think about CIs, we have some indications, we know that tests will sort of run. The tests eventually could become kind of useless, but if they fail, we see it, we sort of start optimizing them. And again, it's also like humans have created them. I think in the world of AI you give much more sort of vague instruction. There's a lot, many more degrees of freedom. And especially if you have a self optimizing workflow that modifies itself and improves, how do you know or what should you have in place to keep degrading?
Speaker A: To be clear, in this world, the human is always in the loop at the point of the pull request in particular or the issue. So an issue gets created or a pull request gets created and no pull request gets merged without the human actually approving that. And that will. And it's also the case that there are limitations on what pull requests can sort of self modify. The automation that GitHub Actions has some built in kind of limitations about actions creating pull requests which themselves modify the actions. And those limitations are in there for good reasons. For you don't want sort of malicious actors or somehow kind of mucking with your kind of CI CD kind of pipelines. And some of that thinking does carry over. And so you can have self improvement kind of loops. You can have like them, they can write new skills or they can update new skills and so on, which are kind of separate to that kind of system. But I do want to kind of say that there are some important kind of controls about how much of that thinking we kind of bring into this world. And some of that is inherited from GitHub Actions.
Speaker C: But if you, I just want to dive a little bit. Like no pull request gets merged automatically. But that's not a sort of, that's an untenable sort of destination, right? Like as you sort of produce and you make more changes, can humans really review? Like is there a path for scaling this that doesn't require pull requests to be automerged?
Speaker A: So that is in some ways a separate discussion. I mean in the sense that for git hypogenic workflows we have a point of view on that, which is that the human is in the loop and the human is in the loop definitely at the point of the modification of the ledger, not the full ledger. Some of the ledger can be modified, the issues can be written, but the actual code, and it's always been that distinction, the code is kind of the golden truth of what's going on. And I do understand of course, that human review is a bottleneck. So of course you can have agentic workflows and other kinds of AI agentic automation and all sorts of other algorithmic automation which do review, review, review. And when you're, when you're creating a factory, a lot of the work is on the quality gates. There's just no point disturbing creating a pull request for the human to review when that thing is low quality. You've just got to create as many high quality gates as possible and automate those to reduce the amount of attention you're sucking from the humans involved. Ultimately, I think you do need some products you could almost like auto merge kind of things. You can of course use kind of stacked pull requests or you could just take one long running pull request. You can push to a pull request branch in an automated way to kind of create a larger and larger and larger feature. So that is okay and that is how some loops kind of work. And of course we have workflows which get the CI green, for example. So look at all the pull requests, find the ones that are red, work out what's going wrong, make the extra pushes. No, not propose actually make the extra push onto the branch because you're only pushing to a branch. So we do allow automated pushes to pull request branches and you can usually constrain that down to be pull requests that have been created by the automation itself. So you're really kind of carving down to a subset. So where the industry and different projects will land. I mean there's no one answer to that, but I do get that is there is more and more pressure towards sort of kind of automated, kind of full construction of software and automated absolutely everything. And we don't go in that direction. We're being responsible, you know. Yeah, there are, you know, there are guidelines about how and absolute human in the loop is absolutely a core guideline of what we're doing here of the path.
Speaker C: I mean, I think I absolutely understand the driver for it and it's safer. But I'm also very conscious of the fact that I guess when I think about a factory, I think about a higher level control plane and really everything that is in the current control plane that includes the modifications to the code, we should aspire to make it agentic. And so maybe it is the next batch of questions around saying okay, you have kind of continuous action that are sort of being done in a collaborative fashion on this sort of shared ledger that is the repo. And you propose those. And so you've created the automation of these activities at some point. And I think that point for many organizations is now they're at the point in which they already have more changes they can accept at sneak oftentimes actually very much so Today the problem is not finding issues, like I have a whole world of issues that I already know of vulnerabilities. The problem is fixing it. Hence the sort of the, you know, kind of focus from inception on fixing them. Similarly over here is like the, if you, if you just got it to the finish line, but you're not able to get it over the finish line and merged, then you just sort of create a review fatigue. And you know, to be blunt, like people are not necessarily well equipped anyway at that point in time. They haven't written.
Speaker A: No, but that's, they're not really able to make a sense of review, right? So some of my work is applying GitHub Agentic workflows to factories inside GitHub. Okay? So we have for example, a factory for CI performance improvement, okay? Now GitHub's a big product and CI takes quite a while to run and we want to improve that. The CI times, okay, we want to make incremental, steady sort of improvement by finding the long running tests and kind of working out. Maybe the matrix is a bit big or kind of set up. You can become a fixer or whatever. And. My job as a factory creator is to deliver high quality pull requests where the reviewer is equipped, and I love that word, equipped. My job is to equip the reviewer with all the information they need to understand why this is being done, what are the risks involved, what are the trade offs involved. Is this a minimal thing? As many quality gate checks on this thing as possible? Of course in this context you get the CI CD quality gates as a number one thing. Now in this particular example we're trying to improve CI perf. And actually in this setting you've got the evidence right there in front of you. You've got a run that was done with zero changes or the latest run on master or whatever on main. Then you've got the run that's done here. And you see, look, the test took 120 seconds there and it took 15 seconds there. We can bank that kind of improvement. We've got the evidence, so, so you give the evidence to the reviewer, you say I found it and maybe you didn't find the evidence. You say this thing should be closed or auto kind of closed. This kind of get rid of the trash off the production line because this was a dodgy one, throw it away. It's a lot of throwing stuff away in this kind of world, much more than before. You get trash.
Speaker C: The work has been cheap. It's easier to throw it away because it was just that sort of Ran along cheap in relative.
Speaker A: Yeah. Or it's just in the nature. I mean chief or not, it's the nature of the factory. If it produces rubbish, you throw it away. Right. You know, the can, it's got a crush on it, you know, throw it away. On our next off site for our, our team, I'm. I want to go visit some real factories with some real maybe, maybe film studios or something like that. How do you, how do people actually make stuff? How do they organize this? How do they tell what's good and what's not? How do they, you know, how many, you know, let's think about this differently.
Speaker C: I think I have a whole world more of questions on it, but I'm afraid we're sort of like a little
Speaker A: bit into some more of these things
Speaker C: a little bit out of time over here on it. So I love first of all Don, thanks for sort of coming on. I really, really relate to the sort of the term continuous AI and I think there's kind of a world to define around it. There's still very much around definitions
Speaker A: nascent world. I love that. It's always been my joy of working in the tech industry.
Speaker C: So super fun. Very much like powerful kind of new repo centered kind of cloud agent sort of platform in the agentic workflows and sounds like a lot over there that we didn't even sort of scratch the surface over here and then some sort of forward looking questions on the agentic workflows will kind of get you to the pr. What happens beyond how does it relate to the loops and lots and lots
Speaker A: to still explore exciting times. And yeah, we call it the agentic Repository Automation Revolution is I think the title I use for the talk and I do think it's going to be at the center of a lot of what practical teams are doing, especially in enterprise, but also open source context. It's a great time to be at GitHub and this kind of work. Thanks for having me on.
Speaker C: Yeah, thanks for joining us. Thanks everybody for listening in. I do very much encourage you to go watch Don's Talk, which is on our YouTube channel, the AI native dev channels. We'll put it in the show notes. It's a really excellent talk. It actually has a lot more content than what we discussed over here. It's very practical advice and I hope you tune in for the next one.
Speaker B: The AI native dev is brought to you by Tesol, the package manager for skills and context. Your hosts are Guy Pigiani and me, Simon Maple. Apogee, the producer is Tom Dowler, the AI native Dev is not just a podcast. It's a community. And we host monthly meetups at the Tesla offices in central London. Visit community to learn more, and I hope to see you there.