What Breaks When You Build AI Under Sovereignty Constraints - Bilge Yücel, deepset GmbH transcript

Hello everyone. Can you all hear me well? All right, amazing. Uh thank you for joining this session. In this one we're going to talk about sovereignty. Uh my name is Viguen. I work as a senior developer relations engineer at Deepset. But I want to start with a question. So how many of you are familiar with the term of sovereignty? Wow, okay, amazing. Maybe I don't need to do a lot of introduction for this one. And um just a little bit of context here about Deepset.

Uh Deepset is the company behind the open source orchestration framework called Haystack. We also have our enterprise platform. And with those products we are solving customer AI challenges for big organizations like Airbus, Bosch, Siemens, but also public sector uh organizations like European Commission, Federal Ministry of Research and Technology uh and space in Germany and other ministries as well. So you can imagine sovereignty is a very big and important topic for us.

So here is a like a policy definition you can say what sovereign sovereign AI is. Sovereign AI is the ability of an organization to design, deploy, and operate AI systems on its own terms. But we are all engineers here. We are not policy makers, we are not lawyers. So if you if you turn that into a technical uh definition, it's basically having explicit control over data flow, model choice, infrastructure, observability, and operations.

And um for us to understand it better, I want to split that into four pillars. So the first one is data sovereignty. So, it's about where you store your data, where does it get processed? The second one is model sovereignty, who controls the running models, the origin of the training data. The third one is infrastructure sovereignty. So, where does compute happen? And the last one is operational sovereignty. So, is your application, is your system traceable?

Who can update it? Who owns the incident response? So, let's go into the depths of this for a minute. Um data sovereignty. Data is the most important assets that we have in in an enterprise. And for us to have data sovereignty, data should be processed and stored within trusted jurisdictions to meet compliance requirements. What does it mean? So, basically, GDPR says that your the European citizen data should stay within Europe.

But, if you send that data to an embedding model, to an embedding API hosted in Virginia in the US, then you are already losing the control of your data. So, that's against data sovereignty. And the other aspect is access permissions. So, this not exactly about how the data is stored and processed, but if there are users in your organization, within your organization, that are that has access to data that are not that they are not supposed to see, there's also a breach of data sovereignty.

The other one is infrastructure sovereignty. So, where does compute happen? It means like we we have the AI application layer. So, we have our rag pipeline ingestion, we have agents, uh these the tools for the agents, and they all run somewhere. And where this application layer runs defines the infrastructure sovereignty of your system. Um and this and there's this whole spectrum from max control to max convenience. Maybe you run everything in an air-gapped environment, it gives you the EU AI Act safe.

Maybe you run on a uh private VPC, this gives you GDPR safety. There is also sovereign cloud, this kind of depends on the provider that you're going with. And there's also at the very end there's SaaS. So with SaaS, you kind of face the cloud act risk, meaning that if you are for example using US headquartered company, although you store, you run all your date all your application within Europe, uh they have ability to get access to your running data.

So that's against the sovereignty that you have over your system. The other pillar is modern sovereignty. So who controls the model and the origin of the training data? Um so the you should have the freedom to choose and switch models. But what it means that if your system can only work with one specific model, you are tightly coupled with that model provider. If they are if the API is down, then you lose access. If they increases the price, then you also have a cost issue.

Um this is kind completely against the model sovereignty idea. And the other one is swappability without architectural changes. So maybe you're not tightly coupled with that model provider, maybe you can use other models as well. But if your system, if your code doesn't allow that immediately, then you are although technically you are not tied to that provider, but you cannot just change the whole code base within one day.

And the last one is training data origin. So this is a bit controversial because we don't have a way to know like where the model was trained, which data was used to train this model. Uh but uh if a model provider is a European company, then they have a better advantage than other companies based in the US. The The last pillar is operational sovereignty. So, like building the whole AI system is one aspect, but monitoring So, basic basically maintaining this whole system by monitoring, evaluating, managing is also another.

Uh operational sovereignty is about monitoring how these systems behave in production, including model inputs and outputs. And in high-stake environments like HR or finance, it requires human in the loop. And managing versioning, updates to models and the application layer uh in a controlled auditable way also goes under this operational sovereignty. But the good news is sovereignty is a spectrum. So, not everyone needs like needs to be sovereign in all of these pillars.

Of course, like uh if you are in finance, in healthcare, in high-stake environment in high-stake domains, you might need an fully air-gapped solution. But if you are an enterprise or a startup working in a different domain, then maybe you don't need everything at all. You're You don't need to comply with every sovereignty pillar that I showed you. The important thing here is you need to know the level of control, so the the level of vendor lock-in you have with your system.

So, you are here in the conference this week, but next week on Monday you're going to go back to work and maybe your manager or CIO comes and says like, "Okay, now we need to make like we have this working system, but then now we need to make it sovereign." Uh and in this slide I want to show you what you do first usually and what you break in this existing system. So, probably intuitively the first thing that you do is the model and you replace the frontier API that you have with a self-hosted model.

And what happens? Then you need to translate this whole API logic to this new model architecture. Maybe you need to update the prompts. Basically, you need to evaluate the performance of the system from scratch. And you need to write a lot of code. The other one is you move private data into the required jurisdiction because you notice that okay, we have some information hosted in in the US. Now we need to take them somewhere in Europe and you do that, but you find yourself managing multiple databases and instances.

Then search becomes a problem. So how do you handle search? Do you do classification like query classification first or do you send the request to both of those databases and get information from those? That's just one challenge that comes with this change. And the other one is you replace manage infra with on prem and you immediately notice how much you had vendor lock-in in this area. You because now you also start thinking about okay, Kubernetes cluster management, how do I like deal with all these all these things that all these model pro all these cloud providers were handling for you?

And there are also hardware limitations because now you need to think about okay, I have this application layer running in GP CPU, but now I have my model run now running GPU. So I need to connect them in a way and this connection management, network management also becomes a problem. And the last one is you incorporate observability and tracing. I mean, if you if you didn't have observability until this point in your system, that's an important issue already.

But if you now you when you need to incorporate it, you notice that you have this fully black box thing. So you don't know actually what's happening in this AI application layer. So you need to understand now because you need to log it somewhere so that your system is auditable. And there's is version control aspect. How do you do version control for your application layer, for this whole system that you are managing?

And I want to hear shamelessly plug Haystack because a good orchestration framework I think solves some of these problems. I mean, good orchestration cannot solve your GPU limitation, but can help you with so many different stuff. Uh the the first thing is uh Haystack is has this consistent interface. So, basically, it when you want to transition your system from cloud to a self-hosted application, you can easily do so just changing couple lines of code and just focus on the hardware that you have and this hardware connection.

The second one is explicit data flow. So, every input and output in a Haystack application is typed and declared. So, you can read the pipeline definition all over the this whole application uh in a in a in a regular way and know exactly what data was where and even in less deterministic architectures like agents and uh like agents the whole data this tools tool calls and tool outputs are traceable. The third one is YAML.

So, the Haystack applications are serializable to YAML making them very easy to version. So, you can just after creating your application with Haystack, you can turn it into YAML, put it into your version control, and then when you need to go back to the history, you just need to go back to the the commit and you you can see the hash. And the last part is it's truly open source. So, there's no black box, no hidden assumptions.

And when you need to customize some code or extend one of the components, you can easily do so because you actually understand what's happening under the hood. And here is one Here's a sovereign architecture that I want to show you um today just as like a high-level overview. Uh imagine you're building this agent, but it needs to be some sort of sovereign. So, first thing that you do probably you add some guardrails before, and then because this guardrail needs to check if there's like a prompt injection coming from the user input coming through the user input, and it needs to also check some specific regulatory checks maybe because I this is like a very specific agent that needs to be used in a certain way, and you want to check if user has this intention.

And if it's unsafe, it just leaves the application layer immediately. Uh and if it's a safe uh request safe input, then it goes to the agent. And agent here is basically an LLM with a system prompt and lots of different tools. And these tools can be API calls, maybe they're connected to your knowledge base that you're you're doing some sort of search. Maybe you are using other agents if you're dealing with a um multi-agentic systems, and there are also MCV servers, of course, that are connected to your agent.

And agent creates an input, so does the uh does the work for you, and there is the last guardrail doing a compliance checks because you also don't want to leak sensitive information to your user, and then there becomes an output. And uh but of course, like how you design a system as a sovereign system is a challenge. And the the duty of Haystack here is to make sure that everything is swappable, traceable, and without vendor lock-in.

Um so, you know, you can work with the right level of sovereignty uh that you need. And here are some of the uh the tools that I picked from that I picked that you can work with Haystack. Um so, maybe you uh you decide you need like different models based on the task. Maybe you decide that for public data, you can steal some proprietary models from those providers, so you don't worry about those. But for some of the task, for guardrails, maybe for your knowledge base, for uh for your LLM, you think that you need local models running on on like self-hosted local models from other different providers like uh Mistral, Google, Nvidia, and Gina.

And then, of course, uh there's traceability aspect. And as I said, in Haystack, everything like every input coming to a component and every output, but as well as the traces in the agent is easily visible, so you can just connect those spans to your LLM observability tool. And you can since there is OpenTelemetry integration, you can also implement your own observability. And as the last part, there's the storage. I wanted to pick some of the providers who can give you cloud and also the open-source version of them, so you can easily host these ones uh locally uh on your prem.

And here is the um here is a guardrail um code. So, uh basically, how you connect a guardrail or how you define a guardrail is quite simple with Haystack. You just start with a model provider. For example, in this one, it's the Nvidia chat generator. You give the name of the model that you want to use. And by connecting it to your LLM message router, it helps you do the classification. So, it checks the input, if it's safe, it goes to the safe route, and if it's unsafe, it goes to the unsafe route.

And then, you add you start adding tools. So, you connect your MCP server that you host locally with uh with MCP tool set, and you give the names of the tools that you want to pick from that MCP server. Because probably you have lots of tools running on that on that MCP server. You don't want to get all of them. You don't want to have access to all of them with just one agent. And you pick, for example, knowledge base search and generating PDF report one.

And but of course like you can add the different tools. You can add just like you can start defining tools from a Python function or you can connect other components in Haystack like agent component into a tool or maybe there's some functionality that you define like data ingestion or rack pipeline and you can also convert those workflows into tools as well. And by after this defining all of these tools, you put them in a searchable tool set.

So this gives you a dynamic tool search with BM25 because you probably now have more than hundreds of tools to for your agent, but you don't want to fill in the whole context with just tool definitions. Then there's this agent component. So basically you define a system prompt. You say like you're a sovereign agent with access to multiple tools and you define the intention of this agent and then you put a brain to this agent.

Basically a chat generator. If you are running this model locally on prem, maybe you create your own custom component saying now this is my own prem chat generator connected to this internal company URL and getting the model inference from that. But of course if you have an open AI API compatible endpoint, you can already use an existing component in Haystack and then you give the tools and then you define confirmation strategies.

So basically you incorporate human in the loop saying that so if the user wants to submit a request, the agent should always ask for human approval, but if the agent wants to use the list payment request tool, maybe ask for a permission first from the human and then you can use this tool as much as possible, as much as you want in this whole cycle. Then you bring it all together. So, you start defining your pipeline.

You first add the component as a tracer that connects to your LLM observability. Then you put the input guardrail. You connect your agent, the output guardrail, and you you can just run this agent saying that pull the outstanding payment request for Q3 and generate a PDF for me. And by using the tools that this agent has, it creates the PDF for you and saves it under this directory under this name. So, with that, I want to give you just like when we are coming to the end of the presentation, I want to show you a checklist.

So, if you want to check if your system is sovereign, you can you can take a look at this question. So, you can think about can you swap models without changing the application logic? Do you have reproducible run logs stored in a complying way? And can your team respond to an incident without calling a vendor like one of those hyperscalers? Thank you for joining the session. And you can get the presentations and ask the questions to me by filling in the the form.

And you can also find me on social media if you want to connect there. I'm happy to talk about you about agents, Haystack, and especially in a sovereign setting. Thank you.