From Ingestion to Agents: How AI Teams Build on Document Intelligence — Adit Abraham, Reducto transcript

[music] Hi. Can everyone hear me? Sweet. We have a short 20 minutes here and so I wanted to jump in and get right to it. Uh my name is De. I'm the co-founder and CEO of Redducto. Uh and today we wanted to talk about one of the I think really practical but maybe less sexy parts of building agents that actually work in the real world uh which is data. Um I'm sure you've seen plenty of talks about data today. We primarily have focused on building infrastructure for anybody working with some of the hardest sources of data which is unstructured images, PDFs, spreadsheets, everything that humans are used to using day-to-day.

Uh, have any of you used Reduct already or trial it? Cool. Um, I guess helpful context maybe to start is we're an agentic document processing platform. Um, so we help a lot of the world's leading AI teams build uh both AI applications and also workflows depending on what they're trying to do. Uh, that includes a lot of the AI natives that you've probably seen today. It includes the Harveys, the Lores, the Rogos of the world.

Uh, but also some of the largest enterprises in the world. And I think that's important context for what we're going to talk about. Um, we work with the largest tech companies, global financial institutions, insurance orgs, people that have decades worth of historical data that historically has been really, really hard to actually use outside of a demo context. And across all these companies at this point, we've processed many billions of documents for our customers.

I've gotten kind of lazy with that plus symbol at the end, uh, but the number keeps changing and so we'll let it sit there. Uh but the main thing that we've learned across those and what I wanted to focus on today is not actually a reduct product itself. It's the intricacies of what we've learned that hopefully you can actually take home and implement in the work that you're doing as well. Um and I think a lot of the work that we've done and a lot of the learnings we've had are in this sort of broader thematic context of the scope of AI applications has changed a lot recently.

Uh from just like information synthesis products to actual agents doing work. And with that we found that there are a few things that are worth talking through. One is framing the problem itself. Uh the bottleneck that people face. Uh why PDFs in particular are hard even though you've probably seen two dozen different PDF processing launches on Twitter. Uh where we find strengths and weaknesses with different tools. Uh we think there's a right place in time for traditional CV versus BLMs.

Um and then more interestingly I think the latter half of this talk will actually be focused on the next frontier. uh what we've seen be possible as a result of having agents in the loop uh things that we found as a result of being able to use harnesses for different types of tasks uh and our learnings around things like evaluations uh as you go from rag to building agent products but I'll start with the first thing which is something that I assume has been harped on a lot if you went to AI engineer a few years ago the word that you would have heard in every single talk would have been rag um and everybody was building some form of rag application for a while all applications ations were really some form of information synthesis, right?

You would pull in information from some context, whether that's a perfect prompt or a file that a user uploaded and you would have something like a search product. Um, you'd have enterprise search, you'd have a chatbot that would do simple question answer on top of the content and that was it. But today, the buzzword that you've probably heard a million times is agents. Uh, and for those of you that are engineers, you've probably had cloud code or similar tool do a lot of your endto-end work for many, many tasks.

And that same sort of shift is starting to happen for all sorts of white collar work. Um, whether you're in finance or insurance or healthcare, people are starting to make autonomous decisions. They're trying to create end-to-end work products, not just answer questions from a PDF, but generate and modify PDFs as well. And that's a very different framing. the tools that you need, the problems that you face change a lot versus just trying to build a retrieval uh platform.

And the common core for all of this is it actually ends up being an even more importance problem to solve for when you start having these multi-step pipelines. If you're just doing question answer, there is obviously a risk that comes with just answering the question. But when agents are making multiple decisions that compound over the course of a source of files, when they're pulling in multiple sources of data, the risk of bad inputs becomes really, really pervasive in your pipeline.

And that's what we focus on because at the end of the day, a lot of the value of language model tools in the real world only applies in the context to which you apply that intelligence. And for a lot of enterprises, data is unstructured, it is scattered, and it is multimodal. uh it's not this like cleanly organized repository. You have random teams and different organizations that go through and put things in Google Drive and Box and wherever else you might have things.

Um you're going to have data for formats that are unstructured by default. You don't necessarily know what is in your corpus of information. And you're going to have all sorts of downstream problems that come with that. Um some of the problems are going to be parsing extraction accuracy and that definitely matters. But it's also going to be about problems like are you retrieving the right context? It's going to be about how do you actually interact with that context and apply modifications to it.

And the thing that you've probably heard harp on and others in the space is PDFs are surprisingly still a very hard problem. Um I don't know if any of you follow Serge which is a data lab that works with a lot of the foundation model companies. uh they have this really great benchmark called GDP PDF where even Fable uh like current frontier of model intelligence is at about 30% on their benchmark um and it's entirely predicated on this idea of can we have models go through and actually make determinations off of contents that would be in documents like PDFs uh and the reason why they're hard I'll come back to that benchmark in a second is fundamentally PDFs as a file format are both very old but were designed in a very different context I've genuinely met people that have worked on PDF processing longer than I've been alive.

Uh I've met the people that worked on printer drivers for printers to print PDFs in the early 1990s. And that's what it was like. You wanted to be able to represent what was faithfully on the documents when you originally created it and have it be printable at the end of it. A lot of the considerations today are not around that. Um ultimately what we want is something like a markdown representation, something that agents will effectively reason on.

And in the real world, humans encode so much context visually. Like the average financial analyst that is a new grad in their IB role uh is not going through and thinking in terms of will agents consume this deck. They're creating these really creative slides. I'm sure you've seen the softbank slides with uh goose laying eggs. Those details matter, right? Like a lot of the data that you're going to reason on is going to be tabular structures that maybe don't have clean grid lines and separate out the merge cells.

You're going to have things like line charts and graphs. You're going to have messy handwriting that even I as a human would often struggle to read. And that's the sort of problem that you need to solve for if you're dealing with that long tail. And we started the company in 2023 because we felt that there was a sort of step change in what was possible here. uh for a while as people would think about any sort of PDF processing problem, it used to be some modified version of an NLP pipeline.

You do a simple OCR pass and then you try to post-process that text. And that worked when you would have really consistent layouts, right? Like if you knew that a W2 is always going to look like a W2, that's a constrained problem space that you can template your way around. But VLMs are interesting because they are fundamentally horizontal in nature. um for the first time you can have this premise of read the document the way that a human would have.

You can have this premise of we want to address the longtail. Uh and so we found that they were incredible for all sorts of things like handwritten text in a way that traditional OCR just never was. But the flip side is that we don't think that they're a one-sizefits-all solution. And if you're solving this problem at scale, if you're a company dealing with hundreds of millions of documents, there are all sorts of secondary considerations like determinism. you care a lot about the efficiency of your processing.

Uh, and so we find that there are some things where traditional CV is actually still really really strong. And this has been really underappreciated as autonomous vehicle research has gotten better. Techniques like object detection are more sophisticated than they were a decade ago. And so we find that subund million parameter models are really effective for things like detecting the layout of a document. Um, you can go really really far without even needing a large foundation model.

And these are models that can actually run on CPU. You can run them at scale. You can make sure that you understand on a region level what are the hard things for you to process. And VLMs introduce this notion of semantics. You can go through and actually identify and make correct the sorts of mistakes that you were likely to have in your pipeline. And off of that idea of semantics, uh, when you're deconstructing this problem and you have a clear sense of where the more nuanced things are, if you've segmented the text on your page and you understand where the handwriting is, you can introduce this notion of a sort of agent in the loop.

Whereas historically, you would have had a human review team go through and annotate and correct mistakes. Uh, VLMs can now present this sort of idea of what we call agentic OCR. Really for us what that looks like is if you've ever used you know a tool like cursor that's applying fast edits in your IDE there's this notion of speculative decoding where you're applying token level edits to your output. You can apply a similar sort of principle here uh that is not just you know sending OCR to Gemini writing a really pretty prompt asking it nicely to not deviate too much from the original because we find that when you're doing that sort of next token prediction you introduce net new loss cases where models that are really intelligent will start actually correcting things not faithfully to what was in the document.

They'll see the word total and if the human made a mistake in that table models will actually sometimes go through and add up the values in the table themselves. What you want is to correct the token level edits that you want. Maybe you messed up a period versus a comma, a zero versus an O. Those sorts of details really, really matter. And that's a question of how do we actually represent what a human would have seen if they had read that document.

So the way that we see this is agentic OCR is almost like that human loop analogy where you have the first inputs uh go through with a CD plus VLM parse but then you have a verification correction layer that ends up leading to a high confidence output. But I mentioned earlier that we don't see the range of problems as purely just parsing and extraction. And a lot of what this looks like is I think it's important to think through the details of your pipeline.

Even if you have a great documents to markdown pipeline and a great example of this is if you've built any sort of rag platform um you've probably had some consideration around things like tables. There are a lot of things that you can encode well in something like markdown. uh but things like this table where the merge cells actually encode a lot of meaning. It matters that you're preserving that sort of structure and it's not a model limitation.

LMS are incredible at reasoning through like an HTML structure of the same table, but you're also wasting a lot of tokens and that gets expensive quickly. And obviously on the other end of the spectrum, you probably don't want to go through and encode simple tables in HTML because then you have a lot of HTML tags that are erroneous. And so what we ended up doing was looking at this as sort of like a dynamic problem of when you have a simple table, great, we can approximate that data in markdown.

When you have a more complex table, you may want to use something like HTML. But it's not only language models that should be a consideration in your pipeline. If you're doing anything related to embedding, you're also going to have the secondary problem of retrieval of that context. Right? That same table that I showed you earlier, if you look at the HTML representation is really really messy. The vast majority of that snippets is just HTML tags.

It's just classifying the structure of the document. And the unfortunate thing is whereas in some blanket evals, you might have contents that is really trying to do the work for the model and say exactly what you're looking for. A real world person does not enumerate the values in the table. They just say how did revenue change over time and they assume that you're going to retrieve the right table when it's relevant.

And whereas language models can reason through that text effectively if you're pulling from a large corpus, we find that embedding models really struggle to correlate that natural language human prompt with this messy blob of HTML tags and numbers. And so a big thing that you can do that actually takes very little effort is creating a representation that's more so designed for the embedding model itself. taking the same table, we're creating a natural language representation of that table.

So you have the best of both worlds. When you're actually passing this into the model for reasoning, you're using the HTML table representation. And when you're trying to make sure that you retrieve the right snippets, you're using that natural language block. Off of that, uh there's also this idea of there's a lot to do that is not just parsing and extraction. Um, and I think a lot of the industry's focus has been on parsing and extraction historically because we do think that there's a massive uplift there.

And I talked earlier about the GDF GDP PDF benchmark which I think is a great illustrative example of what you can see uh as a result of improving your data pipeline. So what we found is that if you take the same exact benchmark that I mentioned earlier, uh, unfortunately we couldn't test it on Fable because our access was cut. uh but if you test on other models and you give it both the original PDF but also a structured representation of the PDF like the parse results here across models whether it's Gemini whether it's anthropic or openi um you find that you actually improve end LLM performance just from better inputs and it's to an extreme where models like GPT 5.5 and opus actually outperform something like Fable out of the box not just on an accuracy basis but as a result of giving better inputs, the models end up needing to use fewer reasoning tokens as well.

Um, they're focused less on representing the data and more on the actual outputs and as a result they end up driving down latency and also getting to the correct answer more quickly. But even once you have that sort of pipeline and you've gone through and you've actually inspected everything in your parsing layer, a lot of human work is going to require actually understanding the range of what you have in your corpus, routing it to the appropriate pipeline and sort of decomposing that problem or even at the end editing and modifying your document.

And so what we tried to do is look at this as this problem of how do you make sure that every interaction that a language model has with a document is as effective as if a human would have done it. Um if you're filling out a form, how do you make sure that you have precision in where you fill out fields? And a really good example of this uh on the orchestration side is I think classification splitting are a very underappreciated way to have an LM do its best work.

Obviously, you can just go through and dump as much context as you want. And if you're doing a sort of needle in the haststack test, that might be fine. But in practice, there is erosion that you find in quality outside of just the token economics as a result of passing in too much. And instead, what we find is you can get a lot of headroom by thinking through things like how do you classify the right documents to the right sort of pipeline?

And even for large documents, how do you make sure that you're passing in the snippets uh that are actually relevant? We see use cases where people will have things like paper mail. Uh and these paper mail packets can be hundreds of pages long. You don't necessarily know what is going to be contained within it. Um you might have issues like a person interle the content and having the model do that sort of work is almost like a distraction from the work that you're actually trying to achieve which might be extracting the data from the paper mail reasoning on it or making a decision.

Again, uh I mentioned earlier that the second half I think is the more interesting piece. Um which is once you have the sort of initial classification and splitting layer, you've figured out sorting. Uh I think the thing that we are really excited about as a team is agent harnesses have been this really really interesting frontier to push past what canonically used to be hard unsolved problems. Um one good example of this that I'll talk about in a second is things like line charts. uh we work with many of the largest hedge funds in the world and things like line charts historically have been really really difficult because one they're an imaged format but two there's a lot of pixel level granularity that if you're doing anything with a traditional vision encoder you're probably going to lose you're going to get a rough plot of how revenue trended but you're not going to get the individual data points and so we've been thinking through how do we give agents the ability to have the right tools to solve for the specific type of problem that you're looking at in the case of chart extraction the chart on the left encodes codes a massive table of data.

If you actually went through and tried to plot every single pixel, it would be really really difficult. U but it would also be hard for a model to even approximate the intricacies of the lines in between. And there's no model that out of the box can do this as a singleshot problem. What you're seeing on the right is a reconstruction of the markdown table that we're able to generate off of the initial line chart. And the only way that we were able to get there was to have an agent with all sorts of tools.

It has its own code interpreter. It has the ability to visualize the chart that it's generating. And it's iteratively going through. It's finding mistakes in the line chart again and again and again until it's able to get to the final output. That applies for problems like structured extraction as well where for a while we've had this documents to structured output feature. Uh but you can really take it a step further by having an agent harness around that same sort of task.

You can have a parent agent go through set validation criteria for sub agents to follow. Uh and this means that if you have something like a CBP form with tens of thousands of fields, that's the sort of problem where you end up finding a lot of issues that are silent in nature like you drop content, you drop rows and micro one actually released a really good benchmark in the space this morning where there's this like bifurcation in the market.

Uh Frontier models with Max Reasoning are really really precise. Like provided that they extracted a row, odds are it's not a hallucinated row like they actually got it correct. but they silently drop a lot of the contents across the benchmark. Uh recall really really struggles. On the flip side, a lot of dedicated document processing services are actually behind frontier models from a precision perspective, but close that gap on a recall perspective.

And so there's always been this sort of trade-off and it was only with an agent harness that we were able to find that sort of local maximum of both precision and also recall for this sort of task. The last thing and maybe the most important thing from this talk uh is that I think at the end of the day eval should underpin all of your decisions and it's been a big part of how we think about our product. Um that applies both to off-the-shelf data sets that you eval against but also to things like real-time production monitoring because your production data is going to differ from whatever else you have in your contrived set.

And I really think it's important to think of eval not as just this like macrolevel view, but also the best teams that we work with look at eval on a granular level for each step of their pipeline. Uh the first thing might be that you want to make sure that the inputs to your pipeline are great. And of course, you should eval things like your parsing pipeline. But even perfect parsing with a horrible retrieval pipeline is not going to help if you're not passing the right context.

And so it's important that you're thinking through details like your retrieval pipeline, your formatting at the end of the pipeline, and also ultimately the most important thing is are you able to improve end agent performance. I'll close off just with a a sense of where we are headed and where we've seen the industry head. Uh the most important thing I think is as agents get better and better, you can deviate from the sort of deterministic pipeline that you would have had a few years ago.

Um, a lot of our customers will actually create effectively a file system for their agent to go through and navigate and let the agent decide what sorts of tools it wants to use. So, we create a CLI where instead of people creating a endto-end pipeline where documents always follow one specific flow, um, the agent will decide if it needs to read a certain type of document and they'll split that into two sets. Uh, one is a content field which the agent can read as it needs to.

Uh, but the other is all the metadata that you would have wanted. If you're doing things like citations, you may want bounding boxes, so on and so forth. Um, I'll skip this part on editing. I think there's a lot of interesting work being done here. We've already released some of it, but in the next few months, you will see us look more and more towards things like document generation. But the recap for today, and I really appreciate your time, is one, I highly recommend that you decompose the parsing problem.

To the extent possible, you should think of it as the right tool for the right task so that you can hit that perimeter frontier of accuracy, cost, and latency. Two, I think agentic verification is the biggest step change that the industry has had for a while and it's a really good opportunity for you to make sure that you're building pipelines that work in production. Three, I think it takes very little effort, but there's a lot of headroom from details like formatting the data for its consumer. or similar vein, I think it's really really important to think about not just the data processing but the the data orchestration and so you should always think about tools like classify and split as a way to augment your pipeline.

Five uh make sure that you eval at every stage and six think through what that next frontier looks like for you because I think most successful companies in today's era have deviated a lot from what we used to do two three years ago. But if you have any questions uh please feel free to reach out at any point. My email is just first namered reductto.ai. Uh and you can also reach out on our website if we can be helpful for your use case.

Thank you.