What It Actually Takes to Build a Software Factory — Tereza Tížková, Factory transcript

Uh hi everyone. Hey what's up? I'm Teresa. Uh and today I will be talking about software factory. uh everyone is talking about software factory but only few people are actually building one and even fewer people know what it actually takes to build one and how to define it. So I will be talking about what is a software factory, should I build your own or outsource it, what works in production already, what are the main challenges and how much is this all going to cost?

Uh my name is Theresa. I work at a company called factory.com and we've been building uh this concept of the software factory for quite a long time but finally uh the technology is catching up and it's possible to build this in production for enterprises like EY or Adobe uh I would define the software factory as the whole loop the whole life cycle of developing software with autonomy which doesn't mean just coding and generating code by that I mean collecting all the signals reacting to user feedback to logs, prioritizing what's important, then orchestrating it all, executing, validating, doing really good uh testing in production and then iterating on all this uh while also continuously improving in the process and gaining new knowledge and new skills.

So this is just our dashboard how our software factory looks if you if you look at it and it's catching the whole cycle there. uh this is just some history and by history I mean 2023 in AI. So the software factory wasn't really possible before even though we always had this idea from the beginning of uh CH GPT lounge and others we had this idea of the continuous loop there was autog baby agi already with the concepts of iterating on the software but it didn't work before because the LMS were hallucinating there was problem with context length uh problem with the reasoning quality uh different problems like this or we were missing good environments where the agents can actually work in the isolation.

So all this kind of led to why the software factory is the paradigm that's starting to be popular now and actually useful. I like to define things by what they are not. So I want to say software factory is not just coding agent and it's not even a swarm of coding agents even thousands of agents because generating code writing code that's the easy part compared to all the others. uh engineers don't even spend most of the time uh just writing code.

The challenges are the rest of it. Uh so this is what I want to enhance and it's also not just some consultancy or just some abstract strategy that someone will try to sell you and to your organization. Uh I think the approaches are of course different but we believe you should really rebuild your organization from ground up to be ready to become a software factory like you can't invite a consultancy and just throw something in the middle of your organization.

You should really be mindful and rebuild it from scratch. uh this is the approach I like uh showing to building the software factory and I think it's similar to building the real team of humans because there is a lot of agents as I said they will be testing validating iterating and it all can end up as a big chaos uh so you need to be mindful and I would say the three important things is being agnostic in the software factory so being independent of LLM choices uh of how you already work as the organization then you need to be autonomous.

So you need to give the agents the right trusts and all this permissions and governance and you need to trust it to run really for a long time because the predictions are that agents will run even a year or more years without the human in the loop. Of course that sounds ambitious but for example our missions already our long runninging sessions already build uh things and run for weeks. So it's not that crazy. And third thing always improving.

So same as human organization you need to onboard any new people in your team by giving them good uh code base understanding good structure documentation and also let them improve in the process and gain new knowledge and share it share it within the team. So these are the three points and now let's talk about the agnostic one. Uh I think the important thing to point out if you are a builder and building different primitives of the software factory you need to be ready for how everyone is already working you need to work across environments like Slack, GitHub and connected to everything and you need to also allow people to already bring subscriptions they are using because there is every every time there is like new technology, new frontier models, uh new benchmarks and it's so difficult to catch up but also to predict what we'll be winning.

So it's best to be agnostic for everyone. Uh then there's the topic of being agnostic to models. Uh this is a topic of a lot of discussions recently and this chart is from this chart is from CEO of Coinbase uh who tweeted about how they started uh started saving money in their organ or organization on AI but without reducing the token spend. So the black line is how much they spent on tokens. They continue growing and token maxing but they stopped spending so much money.

Uh how they did it uh different tricks like different default default models for people. So stop pushing people to use only the frontier as a default. Then caching of course uh to stop preferring the stuff every time. And then no limits on spending but just needing to see the results if you spend a lot. And the last thing that's really important topic is routing. So smart routing between LLMs and just a token optimizing between uh the LMS instead of just spending so much tokens.

Uh we built uh a thing in factory called uh automatic model routing and there's been a lot of questions around it but what it does is it automatically routes between different models in your process. The important thing is it decides first what model is most optimal and then uh it can switch to different model if it's failing the task or in case of any troubles but it usually doesn't happen and important thing is it doesn't just help uh spend money.

It also helps with reliability or with speed because open source models are often faster and uh if one LM provider fails you can just switch to another one automatically. So it really does more things than just saving the money. This uh is our benchmark which is very conservative. I prefer conservative benchmarks but I think the direction is clear. You can save for example 25% but even more probably. Uh how the routing works uh there are four parts.

First you just assign the task and you don't really need to do anything but as an organization you can for example give different permissions and different default models to different people which is very useful like marketing or sales or engineers can both uh have different default models and then you have the classification which is the important part. This is the magic of the routing. You need to really uh look at the structure of the prompt of the uh code base of how difficult the task is, what tools are being used, all these factors and make a classification of difficulty of the task.

Uh after that you basically do a threshold of what is enough to accomplish the task and you choose the cheapest model above the threshold. So cheapest model that is predicted to accomplish your task and then you go. Okay. So when we launched this we got a lot of questions like does it actually work? What if uh the model mis routes? Uh is it slower? Is it more expensive actually? What if the model can't accomplish the task or what if it needs to upgrade?

And how do you handle caching? So I think these are all valid questions and that's why it's so difficult to build a good router like you need to basically classify it very well and the challenge is to not not to need to switch too often but even if you're switching in the middle of the task to difficult to more difficult model you still overall are faster probably because it's still worth uh this is just some overview of caching uh lot of questions we got on this was also So how do we handle caching?

Do we do discounts for users? Uh because LLM LLM labs of course save a lot of money by caching and skipping all the context prefill all the time. Uh open models can do this as well. I think people sometimes forget this and you can just host open models as well uh on dedicated compute and you can take the same advantage of the caching. Uh so I want to enhance uh the final price for users. is just a pricing decision. It's not a technical challenge because everyone can do caching.

It's just what price you pass on the users and what deals you make with the API providers. So this is uh being agnostic and now the autonomy which is the core of the software factory. Uh everyone is talking about loops. Uh loops I think have been here the whole time already in different context. They are just now leveling up and moving to the context of agents. You probably heard about the Ralph loop and specifying the tasks and splitting to subtasks for agents.

Uh this is a known concept and uh the question is not the loop itself but the question is how how you define what it means to be done in the loop. This is one example from my colleague uh he he made a loop uh of agents building 3D printing of our logo. uh I think here you can see what is the challenge of the loops because uh before in programming loops had clear criteria of what it means to be done while now the criteria become open-ended because a lot of the tasks are very nondeterministic it's basically open world you can do even real stuff like printing something and it's really difficult to define how do you verify that the loop was done that the task was accomplished so that's the difficult part uh yeah they just need to be verifiable This I call the scary chart.

It basically shows that the tasks and how long they run autonomously with agents have been increasing. But still it's not that reliable. Even if you can run for a very long time, it doesn't mean that it will be reliable in production and you still probably need to need to iterate on it and uh provide feedback. So it's still not solved and there are problems like cheating. For example, if you write the what it means to be done in the wrong way, the agent can try to pass your tests but not really like not really verify what you need to do and accomplish but instead try to solve just passing your test.

So cheating by that uh we have something called factory missions and this is like long longunning sessions of the agents. We just call it missions because we send the agent to the mission and uh the missions work in the loop and iterating on task until it's done and they can do it even for weeks and uh the main agent is orchestrator and then it's uh assigns work to workers agents and then validators who review the task.

So this is one example uh real mission from our customers that run 16 hours and just just to see how important is the validation as well. it takes even 40% of the whole process. Uh to summarize, the orchestrator just decides and writes the conditions what it needs to be done. Then it gives it to the agents called workers. The agents work on it. And what is interesting or worth noting is that they work in a sequence. So they don't work in a swarm or parallel.

The agents work in sequence and everyone accomplishes something and passes it to the next one. uh we actually found that if you do this you end up with more fresh context and kind of fresh head. Same with when with humans have like other colleagues verifying their code. So similar thing we just let the agents pass to the next one. But still every worker of the sequence can have parallel agents doing smaller tasks. So I don't know researching on the web or building files during that.

So they are in sequence and everyone has sub agents as well. and validators they review the output and provide feedback and send it back to the beginning. Uh important thing is uh the validators judge code that they didn't write. Uh there's something from the orchestrator agent called validation contract uh that's written before any code is done. And first type is scrutiny validator. Second is user testing validator.

The scrutiny is the one who really verifies how the codebase looks. the liners types tests it's really rigorous check of the code but the second I think is really interesting is user testing validator and that's the one who is really in the arena trying the things so it doesn't care how it was made it just goes it works in its virtual computer and it clicks on the stuff and really checks if everything works I've seen one engineer migrating uh code bases with droid with our agent and uh it really needed the agent in the end to click through the stuff because with other products it just created created the product but it didn't work and was wasn't interactive.

It was just a dummy result. So our agent that really clicked on the stuff uh allowed to check that it's actually working and not just looking good in the code. I think this is one cool thing that was also allowed by the progress in computer use and the persistent environments virtual machines for agents which also wasn't here before or wasn't that great but now it's really great so it's allowing to go in the arena as an agent.

Uh this is just one example of real mission from our users. I I like to just learn what users are building and get feedback and this is just uh how it can look in your dashboard. And this is how I visualize the missions. What is interesting is that it's just a loop with smaller loops because every of the agents are running in loops as well, but overall it's like one big loop. Okay, maybe this is a bit weird, but I just like that it's all loops with smaller loops.

Uh, okay. And the third part, always improving, always learning. Uh there is this elephant in the room context with agents. A lot of questions which are valid are like how do you navigate the context blo because you work on these long missions in the software factory and how do you actually keep the context clean and that's very valid because enterprises especially they use hundreds tools they use Figma, notion, Gmail, drive, slack like hundreds tools on average.

So all these have specification in the code. all these have schema and parameters and log descriptions. So, uh it can cause agent to really bloat with that and pick wrong tools if there are two tools that sound similar or to actually lose context because they fill up the the context window and need to compress. So, this is really dangerous. Uh for this we have something within the software factory called uh deferred context engine and we build it such that we just progressively disclose what's in the context and what tools to use.

So we basically have a surprise tools uh that help later only if they are actually needed. So it first just has short list of the tools and uh only a short descriptions and when it's needed actually in the code uh they can call the tool and fully load it. So important thing is nothing is actually removed it's just hidden and not reachable until needed. So it's just for later and important thing is it actually saves a lot of tokens uh at scale once you use more and more tools. uh the more actually you save it really scales and you can save 50% of tokens or more.

Okay. Uh another thing which is tricky and was surprising for me for the first time is that when adopting AI you either succeed big or you can fail big the same way it's a bit of power law. So if your codebase is not ready, if you don't have structured codebase and all important things, then adopting or turning into software factory can actually make you end up worse and make your code degrading. And uh there is a big and growing gap in productivity between those who just adopted AI versus those who actually thought about it a bit more.

Uh there is another data from Stanford about yeah without structured uh code base and being ready and documenting well the AI can make your code worse and I think if you are engineers you probably can agree that sometimes AI really makes a mess and this can really compound more and more and it's difficult to go back. uh we have something as a part of the software factory uh we have something called agent readiness and it's a framework I don't like the word framework but uh take it as a hygiene check of your codebase and things you should do because there is actually nice correlation between how your codebase is looking and all the details there and how good you're going to adopt the AI and end up productive instead of uh with a messy code base.

So there are things like how how reproducible is your developer environment or if you have written good tests if you have all things well documented what's the style of your code all the tests and llinters uh everything that you would do also to keep codebased screen. So these factors actually show to be very useful and we have our bigger customers actually go through this agent readiness framework and do all checks and then they can follow up with recommended actions uh how to fix that.

Uh I think one more thing about the context and the continuous learning in the software factory is the most of the time when working with AI or at least from my experience is that you keep repeating to agents how how do you want the things done and you feel like the agent doesn't really get you and I think this is again nice parallel with human organizations and teams because when you join a new company there are a lot of rules that are not really codified anywhere like you just learn and observe how things are done and there are a lot of things between the lines that you can't really learn anywhere else than just observing.

I think this is big challenge for agents as well and one thing we launched for that is plugins which are like packaged reusable skills and context and the things behind the scenes that you can really codify or auto which is automatically updating our documentations and uh yeah and um reviewing and documenting what you have. So the question is what will happen to us humans if we build software factory like this? Uh will humans just lose jobs and go to permanent underclass?

I think they will not. And uh I want to be positive. So I'm thinking of this parallel of humans just moving up uh levels up to the uh to the cooler tasks than before. I think we actually had before the same same experience of us as humans abstracting some levels up and we actually started ourselves as human computers. We actually were the computers doing all the most detailed stuff at the beginning. Then we had programming languages codifying some of it, abstracting and then coding agents.

So outsourcing some of it but uh still being in the loop very closely uh and very monitoring everything. And now we are moving kind of up to software factories where we manage these agents as I said orchestrator agents and workers and validators and all these teams structured teams of agents that work really continuously in the loop and we just monitor and decide what to build. Uh so I want to really end this positively.

We should be as humans deciding what to build in the software not how to build it. uh because that's up for the agents and by this uh continuous cycle of autonomous software I think we can actually achieve that. Uh this is one warning chart or if someone thinks AI will take the cool cool stuff from us. I believe actually it will take the annoying stuff and in enterprises and organizations you already spend a lot on alignments on the meetings as I mentioned basically the thing behind between the lines that you really need you need to get context from everyone share your status sync on the things on meetings so all these things could be actually outsourced to your software factory and you could be just talking about the cool stuff okay so this is it thank you so much go touch some grass and let your agents uh built for you.

Thank you.