Building the Document Context Layer for AI Agents — Jerry Liu, LlamaIndex transcript

[music] >> I think we can get started. Hey everyone, I'm Jerry, co-founder and CEO of LlamaIndex, and today I'm excited to uh give a talk called building the document context layer for AI agents. Um really uh big shoutout to the AI Engineer World Fair for hosting. Um and if you've seen some of my earlier talks from the previous uh AI Engineer conferences, uh we've kind of traced through a lot of the evolution of how, you know, um agent advances have correlated with uh you know, how you inject context into evolving applications.

Um and so today, in 2026, we'll kind of talk about three main topics. One is, you know, what is RAG in 2026? Um two, uh basically that basically decomposes into an agent harness plus a context layer. So kind of going into the modern document context layer for agents, um and giving you a little bit of sense of kind of some of the core capabilities today, um especially uh unlocking, you know, the vast trove of document-based data today, and giving you a glimpse of what's next.

All right. RAG in 2026. But before that, just a little bit about the company, and I'll kind of get on with the main talk. Um you might have seen us as a RAG framework. We started in 2023, got pretty popular, um kind of created a lot of techniques around like advanced RAG, that type of stuff. Um today, we're basically the main document infrastructure for AI agents. Um we deliver the best platform for agents to actually read and operate over documents, and we see ourselves as unlocking basically the vast troves of document context even as agents and models themselves get better.

All right. So, what does RAG actually mean in 2026, and how are agents actually inhaling and addressing context today? This is a snapshot from um actually 3 years ago, um when I first gave a talk on this, and naive RAG was basically just building a simple chatbot over your private corpus of data. If you flash back all the way to January of like 2023, um you know, this technique basically consisted of you have some corpus of documents, um you chunk it up, embed it, and put it into a vector database.

You do some naive top K retrieval, and then you generate some stuff with an LLM. All the steps are fixed. You use kind of like a fixed set of techniques, and then with that you actually get some basic application results by just being able to chat over a corpus of documents. Of course, the agent landscape has changed quite a bit and quite dramatically since then. One is agent loops and tool use have gotten a lot a lot better, especially as both the models and agent harnesses have increased.

And there's basically a cleaner separation now between agent reasoning and how they actually interact with context. If you look at what I call kind of the modern generalized agents, which includes, you know, all your favorite applications and tools out there from Claude Code, Claude Code Work, Open Claw, Codex, and a few others, you know, the retrieval complexity has started to get baked into the agent layer. So, instead of coming up with a variety of hacks to really work around the limitations of naive top K retrieval, you can start getting the agent to really reason about, for instance, the best key best keyword to search for, best like, you know, search term to actually get back good results.

Even if the retrieval tools are basic, the agent can input the right queries to basically loop upon itself and help solve the task at hand. Number two, context is moving up the stack. There's been a lot of conversations, I think, for the first two and a half years of Gen AI of how do you actually manage the agent context window, make sure it doesn't overflow, that type of thing. But, I think as, you know, a lot of the evolving techniques, compaction, long contexts have evolved, I think more and more of the conversation is actually how do you just hook up the right MCP servers and skills and tasks to the agent to enable it to do various types of tasks.

And so, that also applies to agent orchestration, right? Coding agents have gotten better. Abstractions of actually defining what types of tasks and programs you want to build have moved a little bit upwards towards English as opposed to through code. And so, maybe in 2023 to 2025, you define programs and you still build stuff via like importing Python, using TypeScript through code. It's pretty clear these days more and more people are just building stuff using English, whether you're a software engineer or you're a non-technical function like go-to-market marketing.

And you're defining runbooks through English and kind of like defining the right goals and making sure the AI is aligned on the right task. And we see that carrying over going forward as well. So, in terms of just like general evolving knowledge work patterns, and this kind of forms a foundation for what we how we think about the context layer. You know, in the past maybe you could use AI to just like ask simple questions, get it resolved.

Today, you're starting to be able to define and solve more end-to-end like tasks just through English and also use English to start compiling repeatable programs to actually execute something at scale. Obviously, there's been a lot of discussion on how this will evolve even more, and we do see AI agents as behaving even more autonomously through solving long horizon tasks, looping upon themselves to be able to achieve a goal.

So, the future will move towards a state where you actually might not even have to define the task in English, but actually more of the goal and a scoring rubric, and then the agent will use all available context available to it to actually solve the task. So, with that overall framing, let's talk about what kind of what is a pretty core focus for us as a company, which is basically helping to unlock unstructured context to basically feed into these agents even as they get really, really good both in terms of the core model capabilities as well as the agent harness.

And we'll focus a lot on our core focus area, which is document context, but of course there's plenty of other sources of context as well. In fact, we do think context really is everything. In the end, you could have a infinitely smart agents, but your ability to actually get value out of this infinitely AI agent um, able to do more and more stuff end to end, is actually, you know, giving it the right things to do. Um, and the right things to do include the actual task, uh, the goal, um, but also access to the vast troves of organizational context available to it.

Whether that is web search, web context, whether that is connectors through, you know, like tools and skills and MCP servers, whether it's connectors through Snowflake or Databricks warehouses, um, or for us, whether it's access to the vast trove, like the 90% of documents that are stored within SharePoint, Box, Dropbox, S3, um, you know, all the unstructured, uh, data that's locked up within document containers. Um, and so for us, you know, we care a lot about how do you actually build the right tools and unlock the right context here.

We see documents as universal containers for unstructured context. Uh, there's over 10 trillion plus pages of human native knowledge locked up within PDFs, PowerPoints, Word documents, and Excel sheets. And at the same time, agents are also starting to generate exponentially more data in terms of, you know, more agent native formats in terms of markdown and HTML. For us, building an agent native document platform contains three main pieces.

Uh, it contains the document parsing layer, actually digitalizing, um, the vast troves of PDFs, PowerPoints, and Word documents into kind of accurate token efficient context, um, and feeding it, you know, either markdown, metadata, and other forms that actually enable agents to do stuff over your documents. The next is actually what we call the semantic and storage layer, um, which is kind of like this concept of document management for humans and agents.

Instead of humans opening up Microsoft Word, using your file system or file storage, how do you create some sort of agent interface that actually captures, stores all your documents, and enables agents to actually manage and operate over them in various ways. Um, and then there's still a room for actually kind of repeatable document workflows is agent layer they can actually encode within the uh within like this type of platform, too.

If there is a repeatable workflow, like invoice processing, KYC, or claims, instead of always offloading it to a generalized agent, how do you actually develop some sort of specialized workflow for it and really carefully tune cost and accuracy to make sure that you achieve your goal. So, when we talk about like the concepts today in in terms of these three layers, uh we'll talk about this concept of document OCR, which is one of the core concepts of this track, um to document extraction, document search, and document workflows.

There's still a variety of topics yet left unsolved from agent native document formats, document versioning, document editing, you know, hill climbing as a service, and just like a wide set of uh kind of remaining concepts to actually create a comprehensive piece of software where agents and humans can collaborate on documents and actually do work over them. So, we'll start with the first section um and maybe start kind of with uh the document OCR RP.

Um first off, uh you know, document OCR is hard. Um some of you might have seen a blog post that we put out a few months ago related to this topic. But, the reason this problem even exists is an agent cannot actually take the like the raw PDF like file binary and make sense of it. Um that's because the like the way PDFs are actually stored as a format is it's rendered for kind of like display purposes um and not really for uh you know, machine consumption in an interpretable manner.

Um they're designed for printing. So, basically text are represented as almost like individual glyphs with coordinates. Um tables are not represented as tables, they're represented as like typically line segments uh drawn with various types of borders, and also text drawn at certain cell positions. And so, if you're kind of an agent that's trying to make sense of this like document, you're going to have a really hard time actually trying to reason about what character and what shapes like map to what.

Um so the whole point of document OCR, um, you know, it's been around for like 20 plus years, is to really try to create some sort of, uh, digitalized well-interpretable, uh, representation um, that's both interpretable to humans, um, as well as AI agents. Also, reading order itself, if you have a multi-column layout, there is no, uh, guarantee that the way it's represented in the PDF, um, actually corresponds to the typical ways that humans would read it.

Cuz again, it's basically just an arbitrary sequence of characters drawn with uh, coordinate positions. Related to this, you know, even Word doc, uh, parsing is hard. Um, they're a little bit more structured than PDFs, but they're in kind of like this, uh, custom bespoke XML format, and this applies to PowerPoints as well. Um, it contains more structural information, but still there's a ton of like fluff. Like you don't actually need to ingest all the tags to actually have the agent make sense of the Word document.

It still needs to infer a bunch of structure from it. It needs like the right, uh, kind of to lift the right metadata around like formatting, semantics, that type of stuff. But also being able to ignore the tags and actually be able to render the document so that the agent can see the overall structure of the page. Um, it's still generally hard problem, and there's a lot of uplift you can get by actually parsing it into a more interpretable format instead of just using the native, uh, OXML and feeding that to an agent.

Um, if you're familiar with document understanding, um, you know, it's been around for quite a bit of time. Um, there's a lot of these like heuristic and pipeline-based approaches, which, uh, focused on kind of more, uh, I guess like human-driven, hand-handwritten like techniques to analyze like kind of various pieces of text, group them into clusters, and identify tables, paragraphs, and be able to kind of like, uh, generate some sort of output representation.

Um, a lot of basic techniques if you use open-source libraries like PyPDF, PyMuPDF, um, and of course like some of the, uh, more recent approaches, um, basically use this type of approach. There's also, of course, using a VLM to one-shot a document um, into text, uh, that's what we call like a vision-based approach. This works decently well. Obviously, you know, I think there's kind of a lot of uplift you can get by being able to read the visual structure, but there's a lot of sub-optimal pieces about it.

It can hallucinate on text-only pages. It costs a ton of money, and also it still lacks a lot of the semantics and grounding that you typically expect with some sort of document processing tool. And so for us, we really think about combining both the pipeline-based approaches of deeply understanding the file containers and binaries with the vision-based approaches to help generate, you know, kind of a hybrid approach that we think is at the Pareto frontier of cost and accuracy.

A quick note on this, I'll probably just like skim the high level, but the Pareto frontier for document OCR will always be like much more accurate and cheap compared to the Pareto frontier for, you know, wherever the frontier models are in terms of document understanding. It's because it's a very specific data type, and there's always ways to kind of like distill the latest visual understanding capabilities from, you know, Gemini, GPT, Opus into kind of a carefully tailored workflow that's able to process your documents at scale and in a highly accurate manner.

To some extent, that's exactly what we do. You know, we both optimize underlying like PDF engines plus like Word, PowerPoint, and others. We have an agentic harness that's like carefully tuned for auto routing between cheaper specialized models to frontier models, and also kind of specialized fine-tuned document VLMs that are parameter efficient and focus on specific classes of documents, elements like tables, charts, and others.

That's our commercial service called LlamaParse, which I'll talk about in a bit. But I think in general, you know, we're extremely committed to advancing the frontier of just document understanding cuz basically if you're within an enterprise organization and you have a massive long tail of documents across financial services, insurance, manufacturing, legal, government, and a bunch of others, there's just a lot of complexity in a lot of these document types.

And if you're actually trying to unlock context at scale, most of the models are not up for the task. And so, we've created this thing called ParseBench, which is a comprehensive enterprise document benchmark for agents. We we think about it as the most comprehensive enterprise document benchmark. You know, it contains 2,000 human-verified pages. It measures tables, charts, content faithfulness, semantic formatting. And it's optimized for just how AI agents are actually able to understand these documents instead of like syntactic correctness.

We've If you look at parsebench.ai, which is, you know, kind of the it's a fully public page on the internet. It's also available on Hugging Face and Kaggle. We benchmark probably like 50 different frontier models, open weight models, specialized OCR solutions. And you really want the Pareto curve to kind of be towards the left and up in terms of accuracy, like extremely high accuracy, but also extremely low cost. And there's just so many different types of documents, where some are a little bit simpler, maybe some are a little bit more complex.

And ideally, you want to cover all the points on the curve to deal with the dynamic distribution of various types of documents out there. It's pretty clear, even if you look at this graph, that it's like you can increase the complexity of the benchmark, and that document understanding is definitely not a 100% solved. But, you know, you fundamentally need to kind of advance a lot of the core capabilities to make sure they're able to process, unlock the vast trove of enterprise context out there.

There's kind of like a few different points on this accuracy cost latency Pareto curve. There's what I call like the high accuracy regime, where like, you know, some institutions basically need like 99 to almost 100% accuracy, because basically, the downside of an incorrect extraction is you completely mess up your financial model, you completely, you know, you basically get flagged for fraud or a bunch of other really really bad things.

In regulated industries like insurance and financial services, we see this a decent amount. This typically means you're willing to pay a little bit more money per page for like deeper agent tech reasoning to at least make sure that you get back the the information in the right format. There's also like the low cost regime. Let's say you're just trying to index, you know, the million plus documents per day within your that's you know, being continually updated within your SharePoint just for like rag knowledge base search.

You know, in these cases, you obviously want it to be not like terribly inaccurate, but even if it's a little bit messed up, it's okay, too. Because in the end if you have a sufficiently good agent, it can always dive deeper into the document and surface the right information with the right citations and grounding. So for these, you know, being able to create some sort of scalable offline indexing pipeline that has the best like cost constraints is something that is optimal.

One thing about VLMs based approaches though is that, you know, they're typically not very fast. And I think a lot of times if you have like real-time file uploads, let's say you're using Cloud Co-work, you upload a thousand documents and you need to process it within a minute a minute, like having a bunch of VLMs process that at scale is really tough for basically every single OCR service out there. And and to be fair, that includes ours, too.

I think in general, there's also some sort of need for an extremely low latency solution so that you can actually process stuff in real-time even if you have like deeper VLM enabled processing for kind of like deeper visual inspection and analysis. And so that's what I call kind of being in the agent loop. So besides Lama Parse, which is kind of our commercial service around like document processing and extraction, we also created this tool called Light Parse.

It is surprisingly really really good. I don't know if you've been following some of the Twitter threads, but it is Rust-based. It is the fastest open-source parser out there. It is completely free, um, and there is basically no strings attached. I think it's like MIT or Apache license. Um, and it basically is the most accurate like markdown parser out there that doesn't use a VLM or any sort of kind of like deeper model.

Um, and so this is kind of nice because you can use it as a default in the assistive agent loop. Um, let's say you're uploading a a bunch of documents to Quad Code, Quad Code Work, Codex, and you want it to process like a thousand PDFs extremely quickly. You can always do that, um, and then, you know, equip a VLM-based parser like Llama Parser or other frontier models as a tool. So, what these agents will do is they'll do like a fast pass over all the documents first, uh, uh, kind of like just scan through all the context extremely efficiently, um, and then if actually needs to dive into a page with like tables, with like charts, and actually needs to more deeply understand the values, um, it will use a VLM-based tool, slower processing, to actually make sure it reads the information correctly.

This is available as a one-click installable skill, um, complements kind of any other deeper VLM-based OCR tool you want to use, um, and we kind of designed to make it as fast as possible and also easy to plug in to your favorite AI agent. So, I kind of speed ran through a bunch of this stuff, but basically, you know, uh, we spent a bunch of time on the, uh, parsing layer. There's also other, um, general components around like the semantic and storage layer in terms of document extraction and search, and of course like document workflows.

Um, and due to time, I'll probably kind of just, uh, skip some of the, uh, unexplored areas like agent native document formats, hill climbing as a service, and others, but I'll share the full set of slides online. In terms of the semantic and storage layer, you know, besides document parsing, um, a lot of use cases also require actually getting back, uh, structured information at scale from documents. Whether you're processing, you know, a million invoices or expense reports or receipts or claims, you need to make sure that, you know, you want to actually get back structured outputs that you can put into a downstream database or system.

And so, a lot of these use cases basically revolve around the form of, you know, how do you automate a lot of workloads that humans typically do in scanning a lot of paperwork and doing data entry. Whether it is kind of, again, invoices, claims, contracts, receipts, or others. We kind of created these capabilities within Llama Parse as well. A lot of our capabilities are actually tuned towards like low cost while extremely high accuracy.

And you get back granular citations all the way back to the source document for every extracted output. And of course, you can run this in a pipeline at scale with confidence scores and also, you know, being able to actually flag whether or not we're certain about a certain value before deciding to put it into some sort of system. There's also document search, which is a basically expanded tool set as I mentioned around retrieval, BM25, grep, vector search, reading, and scrolling.

And so, all these capabilities are available within some of our commercial platforms as well as open source offerings. But I also just wanted to paint a picture of the general concepts out there today. So, I'll skip this section about kind of what's next and then maybe just go all the way to the end. I know I'm a little bit over time. So, really appreciate you all spending time today and then let me just how do I get to this part really quick?

Oh, right. I'm going to skip this piece. I'll put put this online. Our booth is at LG 47. If you guys are interested in stopping by and we're hosting a giant pickleball tournament today. So, thank you for your time. >> [applause]