# items

url: https://blogs.nvidia.com/blog/what-is-retrieval-augmented-generation/
title: What Is Retrieval-Augmented Generation aka RAG | NVIDIA Blogs
added_at: 2025-08-27 15:59:38
thumb_url: https://blogs.nvidia.com/wp-content/uploads/2023/11/Patrick-Lewis-RAG-lead-author-150x150.jpg
content_html:
  <p></p><p><em>Editor’s note: This article, originally published on Nov. 15, 2023, has been updated.</em></p>
<p>To understand the latest advancements in <a href="https://www.nvidia.com/en-us/glossary/data-science/generative-ai/" rel="noopener noreferrer" target="_blank">generative AI</a>, imagine a courtroom.</p>
<p>Judges hear and decide cases based on their general understanding of the law. Sometimes a case — like a malpractice suit or a labor dispute — requires special expertise, so judges send court clerks to a law library, looking for precedents and specific cases they can cite.</p>
<p>Like a good judge, large language models (<a href="https://www.nvidia.com/en-us/glossary/data-science/large-language-models/" rel="noopener noreferrer" target="_blank">LLMs</a>) can respond to a wide variety of human queries. But to deliver authoritative answers — grounded in specific court proceedings or similar ones  — the model needs to be provided that information.</p>
<p>The court clerk of AI is a process called <a href="https://www.nvidia.com/en-us/glossary/retrieval-augmented-generation/" rel="noopener noreferrer" target="_blank">retrieval-augmented generation</a>, or RAG for short.</p>
<h2><b>How It Got Named ‘RAG’</b></h2>
<p>Patrick Lewis, lead author of the <a href="https://arxiv.org/pdf/2005.11401.pdf" rel="noopener noreferrer" target="_blank">2020 paper that coined the term</a>, apologized for the unflattering acronym that now describes a growing family of methods across hundreds of papers and dozens of commercial services he believes represent the future of generative AI.</p>
<figure><a href="https://blogs.nvidia.com/wp-content/uploads/2023/11/Patrick-Lewis-RAG-lead-author.jpg" rel="noopener noreferrer" target="_blank"><img alt="Picture of Patrick Lewis, lead author of RAG paper" loading="lazy" src="https://blogs.nvidia.com/wp-content/uploads/2023/11/Patrick-Lewis-RAG-lead-author-150x150.jpg"/></a><figcaption>Patrick Lewis</figcaption></figure>
<p>“We definitely would have put more thought into the name had we known our work would become so widespread,” Lewis said in an interview from Singapore, where he was sharing his ideas with a regional conference of database developers.</p>
<p>“We always planned to have a nicer sounding name, but when it came time to write the paper, no one had a better idea,” said Lewis, who now leads a RAG team at AI startup Cohere.</p>
<h2><b>So, What Is Retrieval-Augmented Generation (RAG)?</b></h2>
<p><a href="https://www.nvidia.com/en-us/glossary/retrieval-augmented-generation/" rel="noopener noreferrer" target="_blank">Retrieval-augmented generation</a> is a technique for enhancing the accuracy and reliability of generative AI models with information fetched from specific and relevant data sources.</p>
<p>In other words, it fills a gap in how LLMs work. Under the hood, LLMs are neural networks, typically measured by how many parameters they contain. An LLM’s parameters essentially represent the general patterns of how humans use words to form sentences.</p>
<p>That deep understanding, sometimes called parameterized knowledge, makes LLMs useful in responding to general prompts. However, it doesn’t serve users who want a deeper dive into a specific type of information.</p>
<h2><b>Combining Internal, External Resources</b></h2>
<p>Lewis and colleagues developed retrieval-augmented generation to link generative AI services to external resources, especially ones rich in the latest technical details.</p>
<p>The paper, with coauthors from the former Facebook AI Research (now Meta AI), University College London and New York University, called RAG “a general-purpose fine-tuning recipe” because it can be used by nearly any LLM to connect with practically any external resource.</p>
<h2><b>Building User Trust</b></h2>
<p>Retrieval-augmented generation gives models sources they can cite, like footnotes in a research paper, so users can check any claims. That builds trust.</p>
<p>What’s more, the technique can help models clear up ambiguity in a user query. It also reduces the possibility that a model will give a very plausible but incorrect answer, a phenomenon called hallucination.</p>
<p>Another great advantage of RAG is it’s relatively easy. A <a href="https://ai.meta.com/blog/retrieval-augmented-generation-streamlining-the-creation-of-intelligent-natural-language-processing-models/" rel="noopener noreferrer" target="_blank">blog</a> by Lewis and three of the paper’s coauthors said developers can implement the process with as few as <a href="https://huggingface.co/facebook/rag-token-nq" rel="noopener noreferrer" target="_blank">five lines of code</a>.</p>
<p>That makes the method faster and less expensive than retraining a model with additional datasets. And it lets users hot-swap new sources on the fly.</p>
<h2><b>How People Are Using RAG</b></h2>
<p>With retrieval-augmented generation, users can essentially have conversations with data repositories, opening up new kinds of experiences. This means the applications for RAG could be multiple times the number of available datasets.</p>
<p>For example, a generative AI model supplemented with a medical index could be a great assistant for a doctor or nurse. Financial analysts would benefit from an assistant linked to market data.</p>
<p>In fact, almost any business can turn its technical or policy manuals, videos or logs into resources called knowledge bases that can enhance LLMs. These sources can enable use cases such as customer or field support, employee training and developer productivity.</p>
<p>The broad potential is why companies including <a href="https://aws.amazon.com/blogs/machine-learning/simplify-access-to-internal-information-using-retrieval-augmented-generation-and-langchain-agents/" rel="noopener noreferrer" target="_blank">AWS</a>, <a href="https://research.ibm.com/blog/retrieval-augmented-generation-RAG" rel="noopener noreferrer" target="_blank">IBM</a>, <a href="https://www.glean.com/resources/guides/what-is-retrieval-augmented-generation-rag" rel="noopener noreferrer" target="_blank">Glean</a>, <a href="https://cloud.google.com/use-cases/retrieval-augmented-generation?hl=en" rel="noopener noreferrer" target="_blank">Google</a>, <a href="https://learn.microsoft.com/en-us/azure/search/retrieval-augmented-generation-overview" rel="noopener noreferrer" target="_blank">Microsoft</a>, NVIDIA, <a href="https://www.oracle.com/artificial-intelligence/generative-ai/retrieval-augmented-generation-rag/" rel="noopener noreferrer" target="_blank">Oracle</a> and <a href="https://www.pinecone.io/learn/retrieval-augmented-generation/" rel="noopener noreferrer" target="_blank">Pinecone</a> are adopting RAG.</p>
<h2><b>Getting Started With Retrieval-Augmented Generation </b></h2>
<p>The <a href="https://build.nvidia.com/nvidia/build-an-enterprise-rag-pipeline" rel="noopener noreferrer" target="_blank">NVIDIA AI Blueprint for RAG</a> helps developers build pipelines to connect their AI applications to enterprise data using industry-leading technology. This reference architecture provides developers with a foundation for building scalable and customizable retrieval pipelines that deliver high accuracy and throughput.</p>
<p>The blueprint can be used as is, or combined with other <a href="https://build.nvidia.com/blueprints" rel="noopener noreferrer" target="_blank">NVIDIA Blueprints</a> for advanced use cases including <a href="https://developer.nvidia.com/blog/build-a-digital-human-interface-for-ai-apps-with-an-nvidia-nim-agent-blueprint/" rel="noopener noreferrer" target="_blank">digital humans</a> and <a href="https://developer.nvidia.com/blog/three-building-blocks-for-creating-ai-virtual-assistants-for-customer-service-with-an-nvidia-nim-agent-blueprint/" rel="noopener noreferrer" target="_blank">AI assistants</a>. For example, the <a href="https://build.nvidia.com/nvidia/ai-virtual-assistant-for-customer-service" rel="noopener noreferrer" target="_blank">b</a><a href="https://build.nvidia.com/nvidia/ai-virtual-assistant-for-customer-service" rel="noopener noreferrer" target="_blank">lueprint for AI assistants</a> empowers organizations to build AI agents that can quickly scale their customer service operations with generative AI and RAG.</p>
<p>In addition, developers and IT teams can try the free, hands-on <a href="https://www.nvidia.com/en-us/launchpad/ai/generative-ai-knowledge-base-chatbot/" rel="noopener noreferrer" target="_blank">NVIDIA LaunchPad lab</a> for building AI chatbots with RAG, enabling fast and accurate responses from enterprise data.</p>
<p>All of these resources use <a href="https://www.nvidia.com/en-us/ai-data-science/products/nemo/" rel="noopener noreferrer" target="_blank">NVIDIA NeMo Retriever</a>, which provides leading, large-scale retrieval accuracy and <a href="https://www.nvidia.com/en-us/ai/" rel="noopener noreferrer" target="_blank">NVIDIA NIM</a> microservices for simplifying secure, high-performance AI deployment across clouds, data centers and workstations. These are offered as part of the <a href="https://www.nvidia.com/en-us/data-center/products/ai-enterprise/" rel="noopener noreferrer" target="_blank">NVIDIA AI Enterprise</a> software platform for accelerating AI development and deployment.</p>
<p>Getting the best performance for RAG workflows requires massive amounts of memory and compute to move and process data. The <a href="https://nvidianews.nvidia.com/news/gh200-grace-hopper-superchip-with-hbm3e-memory" rel="noopener noreferrer" target="_blank">NVIDIA GH200 Grace Hopper Superchip</a>, with its 288GB of fast HBM3e memory and 8 petaflops of compute, is ideal — it can deliver a 150x speedup over using a CPU.</p>
<p>Once companies get familiar with RAG, they can combine a variety of off-the-shelf or custom LLMs with internal or external knowledge bases to create a wide range of assistants that help their employees and customers.</p>
<p>RAG doesn’t require a data center. <a href="https://developer.nvidia.com/blog/supercharging-llm-applications-on-windows-pcs-with-nvidia-rtx-systems/" rel="noopener noreferrer" target="_blank">LLMs are debuting on Windows PCs</a>, thanks to NVIDIA software that enables all sorts of applications users can access even on their laptops.</p>
<figure><a href="https://blogs.nvidia.com/wp-content/uploads/2023/11/Using-RAG-on-PCs.jpg" rel="noopener noreferrer" target="_blank"><img alt="Chart shows running RAG on a PC" loading="lazy" src="https://blogs.nvidia.com/wp-content/uploads/2023/11/Using-RAG-on-PCs.jpg"/></a><figcaption>An example application for RAG on a PC.</figcaption></figure>
<p>PCs equipped with NVIDIA RTX GPUs can now run some AI models locally. By using RAG on a PC, users can link to a private knowledge source – whether that be emails, notes or articles – to improve responses. The user can then feel confident that their data source, prompts and response all remain private and secure.</p>
<p>A<a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblogs.nvidia.com%2Fblog%2F2023%2F10%2F17%2Ftensorrt-llm-windows-stable-diffusion-rtx%2F&amp;data=05%7C01%7Crmerritt%40nvidia.com%7Cdfd2267cb8344597f73408dbd0e6fc36%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638333462716560839%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=V5FClNyBybzOnm3%2FKxiTT4i9aoT0GdLWzdLfnF8LBk0%3D&amp;reserved=0" rel="noopener noreferrer" target="_blank"> recent blog</a> provides an example of RAG accelerated by TensorRT-LLM for Windows to get better results fast.</p>
<h2><b>The History of RAG </b></h2>
<p>The roots of the technique go back at least to the early 1970s. That’s when researchers in information retrieval prototyped what they called question-answering systems, apps that use natural language processing (<a href="https://www.nvidia.com/en-us/glossary/natural-language-processing/" rel="noopener noreferrer" target="_blank">NLP</a>) to access text, initially in narrow topics such as baseball.</p>
<p>The concepts behind this kind of text mining have remained fairly constant over the years. But the machine learning engines driving them have grown significantly, increasing their usefulness and popularity.</p>
<p>In the mid-1990s, the Ask Jeeves service, now Ask.com, popularized question answering with its mascot of a well-dressed valet. IBM’s Watson became a TV celebrity in 2011 when it handily beat two human champions on the <i>Jeopardy!</i> game show.</p>
<p><a href="https://blogs.nvidia.com/wp-content/uploads/2023/11/Ask-Jeeves-2.jpg" rel="noopener noreferrer" target="_blank"><img alt="Picture of Ask Jeeves, an early RAG-like web service" loading="lazy" src="https://blogs.nvidia.com/wp-content/uploads/2023/11/Ask-Jeeves-2.jpg"/></a></p>
<p>Today, LLMs are taking question-answering systems to a whole new level.</p>
<h2><b>Insights From a London Lab</b></h2>
<p>The seminal 2020 paper arrived as Lewis was pursuing a doctorate in NLP at University College London and working for Meta at a new London AI lab. The team was searching for ways to pack more knowledge into an LLM’s parameters and using a benchmark it developed to measure its progress.</p>
<p>Building on earlier methods and inspired by <a href="https://arxiv.org/pdf/2002.08909.pdf" rel="noopener noreferrer" target="_blank">a paper</a> from Google researchers, the group “had this compelling vision of a trained system that had a retrieval index in the middle of it, so it could learn and generate any text output you wanted,” Lewis recalled.</p>
<figure><a href="https://blogs.nvidia.com/wp-content/uploads/2023/11/IBM-Watson-wins-Jeopardy-YT.jpg" rel="noopener noreferrer" target="_blank"><img alt='Picture of IBM Watson winning on "Jeopardy" TV show, popularizing a RAG-like AI service' loading="lazy" src="https://blogs.nvidia.com/wp-content/uploads/2023/11/IBM-Watson-wins-Jeopardy-YT.jpg"/></a><figcaption>The IBM Watson question-answering system became a celebrity when it won big on the TV game show Jeopardy!</figcaption></figure>
<p>When Lewis plugged into the work in progress a promising retrieval system from another Meta team, the first results were unexpectedly impressive.</p>
<p>“I showed my supervisor and he said, ‘Whoa, take the win. This sort of thing doesn’t happen very often,’ because these workflows can be hard to set up correctly the first time,” he said.</p>
<p>Lewis also credits major contributions from team members Ethan Perez and Douwe Kiela, then of New York University and Facebook AI Research, respectively.</p>
<p>When complete, the work, which ran on a cluster of NVIDIA GPUs, showed how to make generative AI models more authoritative and <a href="https://blogs.nvidia.com/blog/what-is-trustworthy-ai/" rel="noopener noreferrer" target="_blank">trustworthy</a>. It’s since been cited by hundreds of papers that amplified and extended the concepts in what continues to be an active area of research.</p>
<h2><b>How Retrieval-Augmented Generation Works</b></h2>
<p>At a high level, here’s <a href="https://developer.nvidia.com/topics/ai/retrieval-augmented-generation" rel="noopener noreferrer" target="_blank">how retrieval-augmented generation works</a>.</p>
<p>When users ask an LLM a question, the AI model sends the query to another model that converts it into a numeric format so machines can read it. The numeric version of the query is sometimes called an embedding or a vector.</p>

<figure><img alt="" loading="lazy" src="https://blogs.nvidia.com/wp-content/uploads/2024/11/ragexplainer131-960x1143.png"/><figcaption>In retrieval-augmented generation, LLMs are enhanced with embedding and reranking models, storing knowledge in a vector database for precise query retrieval.</figcaption></figure>
<p>The embedding model then compares these numeric values to vectors in a machine-readable index of an available knowledge base. When it finds a match or multiple matches, it retrieves the related data, converts it to human-readable words and passes it back to the LLM.</p>
<p>Finally, the LLM combines the retrieved words and its own response to the query into a final answer it presents to the user, potentially citing sources the embedding model found.</p>
<h2><b>Keeping Sources Current</b></h2>
<p>In the background, the embedding model continuously creates and updates machine-readable indices, sometimes called vector databases, for new and updated knowledge bases as they become available.</p>
<figure><a href="https://blogs.nvidia.com/wp-content/uploads/2023/11/LangChain-2-LLM-with-a-retriveal-process.jpg" rel="noopener noreferrer" target="_blank"><img alt="Chart of a RAG process described by LangChain" loading="lazy" src="https://blogs.nvidia.com/wp-content/uploads/2023/11/LangChain-2-LLM-with-a-retriveal-process-672x268.jpg"/></a><figcaption>Retrieval-augmented generation combines LLMs with embedding models and vector databases.</figcaption></figure>
<p>Many developers find LangChain, an open-source library, can be particularly useful in chaining together LLMs, embedding models and knowledge bases. NVIDIA uses LangChain in its reference architecture for retrieval-augmented generation.</p>
<p>The LangChain community provides its own <a href="https://blog.langchain.dev/tutorial-chatgpt-over-your-data/" rel="noopener noreferrer" target="_blank">description of a RAG process</a>.</p>
<p>The future of generative AI lies in <a href="https://blogs.nvidia.com/blog/what-is-agentic-ai/" rel="noopener noreferrer" target="_blank">agentic AI</a> — where LLMs and knowledge bases are dynamically orchestrated to create autonomous assistants. These AI-driven agents can enhance decision-making, adapt to complex tasks and deliver authoritative, verifiable results for users.</p>

<p></p>

content_text:
Editor’s note: This article, originally published on Nov. 15, 2023, has been updated.

To understand the latest advancements in

generative AI

, imagine a courtroom.

Judges hear and decide cases based on their general understanding of the law. Sometimes a case — like a malpractice suit or a labor dispute — requires special expertise, so judges send court clerks to a law library, looking for precedents and specific cases they can cite.

Like a good judge, large language models (

LLMs

) can respond to a wide variety of human queries. But to deliver authoritative answers — grounded in specific court proceedings or similar ones  — the model needs to be provided that information.

The court clerk of AI is a process called

retrieval-augmented generation

, or RAG for short.

How It Got Named ‘RAG’

Patrick Lewis, lead author of the

2020 paper that coined the term

, apologized for the unflattering acronym that now describes a growing family of methods across hundreds of papers and dozens of commercial services he believes represent the future of generative AI.

Patrick Lewis

“We definitely would have put more thought into the name had we known our work would become so widespread,” Lewis said in an interview from Singapore, where he was sharing his ideas with a regional conference of database developers.

“We always planned to have a nicer sounding name, but when it came time to write the paper, no one had a better idea,” said Lewis, who now leads a RAG team at AI startup Cohere.

So, What Is Retrieval-Augmented Generation (RAG)?

Retrieval-augmented generation

is a technique for enhancing the accuracy and reliability of generative AI models with information fetched from specific and relevant data sources.

In other words, it fills a gap in how LLMs work. Under the hood, LLMs are neural networks, typically measured by how many parameters they contain. An LLM’s parameters essentially represent the general patterns of how humans use words to form sentences.

That deep understanding, sometimes called parameterized knowledge, makes LLMs useful in responding to general prompts. However, it doesn’t serve users who want a deeper dive into a specific type of information.

Combining Internal, External Resources

Lewis and colleagues developed retrieval-augmented generation to link generative AI services to external resources, especially ones rich in the latest technical details.

The paper, with coauthors from the former Facebook AI Research (now Meta AI), University College London and New York University, called RAG “a general-purpose fine-tuning recipe” because it can be used by nearly any LLM to connect with practically any external resource.

Building User Trust

Retrieval-augmented generation gives models sources they can cite, like footnotes in a research paper, so users can check any claims. That builds trust.

What’s more, the technique can help models clear up ambiguity in a user query. It also reduces the possibility that a model will give a very plausible but incorrect answer, a phenomenon called hallucination.

Another great advantage of RAG is it’s relatively easy. A

blog

by Lewis and three of the paper’s coauthors said developers can implement the process with as few as

five lines of code

.

That makes the method faster and less expensive than retraining a model with additional datasets. And it lets users hot-swap new sources on the fly.

How People Are Using RAG

With retrieval-augmented generation, users can essentially have conversations with data repositories, opening up new kinds of experiences. This means the applications for RAG could be multiple times the number of available datasets.

For example, a generative AI model supplemented with a medical index could be a great assistant for a doctor or nurse. Financial analysts would benefit from an assistant linked to market data.

In fact, almost any business can turn its technical or policy manuals, videos or logs into resources called knowledge bases that can enhance LLMs. These sources can enable use cases such as customer or field support, employee training and developer productivity.

The broad potential is why companies including

AWS

,

IBM

,

Glean

,

Google

,

Microsoft

, NVIDIA,

Oracle

and

Pinecone

are adopting RAG.

Getting Started With Retrieval-Augmented Generation

The

NVIDIA AI Blueprint for RAG

helps developers build pipelines to connect their AI applications to enterprise data using industry-leading technology. This reference architecture provides developers with a foundation for building scalable and customizable retrieval pipelines that deliver high accuracy and throughput.

The blueprint can be used as is, or combined with other

NVIDIA Blueprints

for advanced use cases including

digital humans

and

AI assistants

. For example, the

b

lueprint for AI assistants

empowers organizations to build AI agents that can quickly scale their customer service operations with generative AI and RAG.

In addition, developers and IT teams can try the free, hands-on

NVIDIA LaunchPad lab

for building AI chatbots with RAG, enabling fast and accurate responses from enterprise data.

All of these resources use

NVIDIA NeMo Retriever

, which provides leading, large-scale retrieval accuracy and

NVIDIA NIM

microservices for simplifying secure, high-performance AI deployment across clouds, data centers and workstations. These are offered as part of the

NVIDIA AI Enterprise

software platform for accelerating AI development and deployment.

Getting the best performance for RAG workflows requires massive amounts of memory and compute to move and process data. The

NVIDIA GH200 Grace Hopper Superchip

, with its 288GB of fast HBM3e memory and 8 petaflops of compute, is ideal — it can deliver a 150x speedup over using a CPU.

Once companies get familiar with RAG, they can combine a variety of off-the-shelf or custom LLMs with internal or external knowledge bases to create a wide range of assistants that help their employees and customers.

RAG doesn’t require a data center.

LLMs are debuting on Windows PCs

, thanks to NVIDIA software that enables all sorts of applications users can access even on their laptops.

An example application for RAG on a PC.

PCs equipped with NVIDIA RTX GPUs can now run some AI models locally. By using RAG on a PC, users can link to a private knowledge source – whether that be emails, notes or articles – to improve responses. The user can then feel confident that their data source, prompts and response all remain private and secure.

A

recent blog

provides an example of RAG accelerated by TensorRT-LLM for Windows to get better results fast.

The History of RAG

The roots of the technique go back at least to the early 1970s. That’s when researchers in information retrieval prototyped what they called question-answering systems, apps that use natural language processing (

NLP

) to access text, initially in narrow topics such as baseball.

The concepts behind this kind of text mining have remained fairly constant over the years. But the machine learning engines driving them have grown significantly, increasing their usefulness and popularity.

In the mid-1990s, the Ask Jeeves service, now Ask.com, popularized question answering with its mascot of a well-dressed valet. IBM’s Watson became a TV celebrity in 2011 when it handily beat two human champions on the

Jeopardy!

game show.

Today, LLMs are taking question-answering systems to a whole new level.

Insights From a London Lab

The seminal 2020 paper arrived as Lewis was pursuing a doctorate in NLP at University College London and working for Meta at a new London AI lab. The team was searching for ways to pack more knowledge into an LLM’s parameters and using a benchmark it developed to measure its progress.

Building on earlier methods and inspired by

a paper

from Google researchers, the group “had this compelling vision of a trained system that had a retrieval index in the middle of it, so it could learn and generate any text output you wanted,” Lewis recalled.

The IBM Watson question-answering system became a celebrity when it won big on the TV game show Jeopardy!

When Lewis plugged into the work in progress a promising retrieval system from another Meta team, the first results were unexpectedly impressive.

“I showed my supervisor and he said, ‘Whoa, take the win. This sort of thing doesn’t happen very often,’ because these workflows can be hard to set up correctly the first time,” he said.

Lewis also credits major contributions from team members Ethan Perez and Douwe Kiela, then of New York University and Facebook AI Research, respectively.

When complete, the work, which ran on a cluster of NVIDIA GPUs, showed how to make generative AI models more authoritative and

trustworthy

. It’s since been cited by hundreds of papers that amplified and extended the concepts in what continues to be an active area of research.

How Retrieval-Augmented Generation Works

At a high level, here’s

how retrieval-augmented generation works

.

When users ask an LLM a question, the AI model sends the query to another model that converts it into a numeric format so machines can read it. The numeric version of the query is sometimes called an embedding or a vector.

In retrieval-augmented generation, LLMs are enhanced with embedding and reranking models, storing knowledge in a vector database for precise query retrieval.

The embedding model then compares these numeric values to vectors in a machine-readable index of an available knowledge base. When it finds a match or multiple matches, it retrieves the related data, converts it to human-readable words and passes it back to the LLM.

Finally, the LLM combines the retrieved words and its own response to the query into a final answer it presents to the user, potentially citing sources the embedding model found.

Keeping Sources Current

In the background, the embedding model continuously creates and updates machine-readable indices, sometimes called vector databases, for new and updated knowledge bases as they become available.

Retrieval-augmented generation combines LLMs with embedding models and vector databases.

Many developers find LangChain, an open-source library, can be particularly useful in chaining together LLMs, embedding models and knowledge bases. NVIDIA uses LangChain in its reference architecture for retrieval-augmented generation.

The LangChain community provides its own

description of a RAG process

.

The future of generative AI lies in

agentic AI

— where LLMs and knowledge bases are dynamically orchestrated to create autonomous assistants. These AI-driven agents can enhance decision-making, adapt to complex tasks and deliver authoritative, verifiable results for users.


url: https://docs.mistral.ai/guides/rag/
title: Basic RAG | Mistral AI
added_at: 2025-08-27 15:59:56
thumb_url: https://colab.research.google.com/assets/colab-badge.svg
content_html:
<p>Retrieval-augmented generation (RAG) is an AI framework that synergizes the capabilities of LLMs and information retrieval systems. It's useful to answer questions or generate content leveraging external knowledge. There are two main steps in RAG:</p>
<ol>
<li>retrieval: retrieve relevant information from a knowledge base or an external source, for example, using text embeddings stored in a vector store.</li>
<li>generation: insert the relevant information to the prompt for the LLM to generate information.</li>
</ol>
<p>In this guide, we will walk through a very basic example of RAG, you can find more in depth guides in our <a href="https://docs.mistral.ai/guides/rag/#rag-examples" rel="noopener noreferrer" target="_blank">cookbooks</a>.</p>
<a href="https://colab.research.google.com/github/mistralai/cookbook/blob/main/mistral/rag/basic_RAG.ipynb" rel="noopener noreferrer" target="_blank"><img alt="Open In Colab" loading="lazy" src="https://colab.research.google.com/assets/colab-badge.svg"/></a>
<h2>RAG from scratch<a href="https://docs.mistral.ai/guides/rag/#rag-from-scratch" rel="noopener noreferrer" target="_blank" title="Direct link to RAG from scratch">​</a></h2>
<p>This section aims to guide you through the process of building a basic RAG from scratch. We have two goals: firstly, to offer users a comprehensive understanding of the internal workings of RAG and demystify the underlying mechanisms; secondly, to empower you with the essential foundations needed to build an RAG using the minimum required dependencies.</p>
<h3>Import needed packages<a href="https://docs.mistral.ai/guides/rag/#import-needed-packages" rel="noopener noreferrer" target="_blank" title="Direct link to Import needed packages">​</a></h3>
<p>The first step is to install the packages <code>mistralai</code> and <code>faiss-cpu</code> and import the needed packages:</p>
<pre><code>from mistralai import Mistral<br/>import requests<br/>import numpy as np<br/>import faiss<br/>import os<br/>from getpass import getpass<br/><br/>api_key= getpass("Type your API Key")<br/>client = Mistral(api_key=api_key)<br/></code></pre>
<h3>Get data<a href="https://docs.mistral.ai/guides/rag/#get-data" rel="noopener noreferrer" target="_blank" title="Direct link to Get data">​</a></h3>
<p>In this very simple example, we are getting data from an essay written by Paul Graham:</p>
<pre><code>response = requests.get('https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt')<br/>text = response.text<br/></code></pre>
<p>We can also save the essay in a local file:</p>
<pre><code>f = open('essay.txt', 'w')<br/>f.write(text)<br/>f.close()<br/></code></pre>
<h3>Split document into chunks<a href="https://docs.mistral.ai/guides/rag/#split-document-into-chunks" rel="noopener noreferrer" target="_blank" title="Direct link to Split document into chunks">​</a></h3>
<p>In a RAG system, it is crucial to split the document into smaller chunks so that it's more effective to identify and retrieve the most relevant information in the retrieval process later. In this example, we simply split our text by character, combine 2048 characters into each chunk, and we get 37 chunks.</p>
<pre><code>chunk_size = 2048<br/>chunks = [text[i:i + chunk_size] for i in range(0, len(text), chunk_size)]<br/>len(chunks)<br/></code></pre>
<p><strong>Output</strong></p>
<h4>Considerations:<a href="https://docs.mistral.ai/guides/rag/#considerations" rel="noopener noreferrer" target="_blank" title="Direct link to Considerations:">​</a></h4>
<ul>
<li><strong>Chunk size</strong>: Depending on your specific use case, it may be necessary to customize or experiment with different chunk sizes and chunk overlap to achieve optimal performance in RAG. For example, smaller chunks can be more beneficial in retrieval processes, as larger text chunks often contain filler text that can obscure the semantic representation. As such, using smaller text chunks in the retrieval process can enable the RAG system to identify and extract relevant information more effectively and accurately. However, it's worth considering the trade-offs that come with using smaller chunks, such as increasing processing time and computational resources.</li>
<li><strong>How to split</strong>: While the simplest method is to split the text by character, there are other options depending on the use case and document structure. For example, to avoid exceeding token limits in API calls, it may be necessary to split the text by tokens. To maintain the cohesiveness of the chunks, it can be useful to split the text by sentences, paragraphs, or HTML headers. If working with code, it's often recommended to split by meaningful code chunks for example using an Abstract Syntax Tree (AST) parser.</li>
</ul>
<h3>Create embeddings for each text chunk<a href="https://docs.mistral.ai/guides/rag/#create-embeddings-for-each-text-chunk" rel="noopener noreferrer" target="_blank" title="Direct link to Create embeddings for each text chunk">​</a></h3>
<p>For each text chunk, we then need to create text embeddings, which are numeric representations of the text in the vector space. Words with similar meanings are expected to be in closer proximity or have a shorter distance in the vector space.
To create an embedding, use Mistral AI's embeddings API endpoint and the embedding model <code>mistral-embed</code>. We create a <code>get_text_embedding</code> to get the embedding from a single text chunk and then we use list comprehension to get text embeddings for all text chunks.</p>
<pre><code>def get_text_embedding(input):<br/>    embeddings_batch_response = client.embeddings.create(<br/>          model="mistral-embed",<br/>          inputs=input<br/>      )<br/>    return embeddings_batch_response.data[0].embedding<br/>text_embeddings = np.array([get_text_embedding(chunk) for chunk in chunks])<br/></code></pre>
<h3>Load into a vector database<a href="https://docs.mistral.ai/guides/rag/#load-into-a-vector-database" rel="noopener noreferrer" target="_blank" title="Direct link to Load into a vector database">​</a></h3>
<p>Once we get the text embeddings, a common practice is to store them in a vector database for efficient processing and retrieval. There are several vector database to choose from. In our simple example, we are using an open-source vector database Faiss, which allows for efficient similarity search.</p>
<p>With Faiss, we instantiate an instance of the Index class, which defines the indexing structure of the vector database. We then add the text embeddings to this indexing structure.</p>
<pre><code>import faiss<br/><br/>d = text_embeddings.shape[1]<br/>index = faiss.IndexFlatL2(d)<br/>index.add(text_embeddings)<br/></code></pre>
<h4>Considerations:<a href="https://docs.mistral.ai/guides/rag/#considerations-1" rel="noopener noreferrer" target="_blank" title="Direct link to Considerations:">​</a></h4>
<ul>
<li><strong>Vector database</strong>: When selecting a vector database, there are several factors to consider including speed, scalability, cloud management, advanced filtering, and open-source vs. closed-source.</li>
</ul>
<h3>Create embeddings for a question<a href="https://docs.mistral.ai/guides/rag/#create-embeddings-for-a-question" rel="noopener noreferrer" target="_blank" title="Direct link to Create embeddings for a question">​</a></h3>
<p>Whenever users ask a question, we also need to create embeddings for this question using the same embedding models as before.</p>
<pre><code>question = "What were the two main things the author worked on before college?"<br/>question_embeddings = np.array([get_text_embedding(question)])<br/></code></pre>
<h4>Considerations:<a href="https://docs.mistral.ai/guides/rag/#considerations-2" rel="noopener noreferrer" target="_blank" title="Direct link to Considerations:">​</a></h4>
<ul>
<li><strong>Hypothetical Document Embeddings (HyDE)</strong>: In some cases, the user's question might not be the most relevant query to use for identifying the relevant context. Instead, it maybe more effective to generate a hypothetical answer or a hypothetical document based on the user's query and use the embeddings of the generated text to retrieve similar text chunks.</li>
</ul>
<h3>Retrieve similar chunks from the vector database<a href="https://docs.mistral.ai/guides/rag/#retrieve-similar-chunks-from-the-vector-database" rel="noopener noreferrer" target="_blank" title="Direct link to Retrieve similar chunks from the vector database">​</a></h3>
<p>We can perform a search on the vector database with <code>index.search</code>, which takes two arguments: the first is the vector of the question embeddings, and the second is the number of similar vectors to retrieve. This function returns the distances and the indices of the most similar vectors to the question vector in the vector database. Then based on the returned indices, we can retrieve the actual relevant text chunks that correspond to those indices.</p>
<pre><code>D, I = index.search(question_embeddings, k=2) <br/>retrieved_chunk = [chunks[i] for i in I.tolist()[0]]<br/></code></pre>
<h4>Considerations:<a href="https://docs.mistral.ai/guides/rag/#considerations-3" rel="noopener noreferrer" target="_blank" title="Direct link to Considerations:">​</a></h4>
<ul>
<li><strong>Retrieval methods</strong>: There are a lot different retrieval strategies. In our example, we are showing a simple similarity search with embeddings. Sometimes when there is metadata available for the data, it's better to filter the data based on the metadata first before performing similarity search. There are also other statistical retrieval methods like TF-IDF and BM25 that use frequency and distribution of terms in the document to identify relevant text chunks.</li>
<li><strong>Retrieved document</strong>: Do we always retrieve individual text chunk as it is? Not always.
<ul>
<li>Sometime, we would like to include more context around the actual retrieved text chunk. We call the actual retrieved text chunk "child chunk" and our goal is to retrieve a larger "parent chunk" that the "child chunk" belongs to.</li>
<li>On occasion, we might also want to provide weights to our retrieve documents. For example, a time-weighted approach would help us retrieve the most recent document.</li>
<li>One common issue in the retrieval process is the "lost in the middle" problem where the information in the middle of a long context gets lost. Our models have tried to mitigate this issue. For example, in the passkey task, our models have demonstrated the ability to find a "needle in a haystack" by retrieving a randomly inserted passkey within a long prompt, up to 32k context length. However, it is worth considering experimenting with reordering the document to determine if placing the most relevant chunks at the beginning and end leads to improved results.</li>
</ul>
</li>
</ul>
<h3>Combine context and question in a prompt and generate response<a href="https://docs.mistral.ai/guides/rag/#combine-context-and-question-in-a-prompt-and-generate-response" rel="noopener noreferrer" target="_blank" title="Direct link to Combine context and question in a prompt and generate response">​</a></h3>
<p>Finally, we can offer the retrieved text chunks as the context information within the prompt. Here is a prompt template where we can include both the retrieved text and user question in the prompt.</p>
<pre><code>prompt = f"""<br/>Context information is below.<br/>---------------------<br/>{retrieved_chunk}<br/>---------------------<br/>Given the context information and not prior knowledge, answer the query.<br/>Query: {question}<br/>Answer:<br/>"""<br/></code></pre>
<p>Then we can use the Mistral chat completion API to chat with a Mistral model (e.g., mistral-medium-latest) and generate answers based on the user question and the context of the question.</p>
<pre><code>def run_mistral(user_message, model="mistral-large-latest"):<br/>    messages = [<br/>        {<br/>            "role": "user", "content": user_message<br/>        }<br/>    ]<br/>    chat_response = client.chat.complete(<br/>        model=model,<br/>        messages=messages<br/>    )<br/>    return (chat_response.choices[0].message.content)<br/><br/>run_mistral(prompt)<br/></code></pre>
<p><strong>Output:</strong></p>
<pre><code>'The two main things the author worked on before college were writing and programming. They wrote short stories and tried writing programs on an IBM 1401 in 9th grade.'<br/></code></pre>
<h4>Considerations:<a href="https://docs.mistral.ai/guides/rag/#considerations-4" rel="noopener noreferrer" target="_blank" title="Direct link to Considerations:">​</a></h4>
<ul>
<li><strong>Prompting techniques</strong>: Most of the prompting techniques can be used in developing a RAG system as well. For example, we can use few-shot learning to guide the model's answers by providing a few examples. Additionally, we can explicitly instruct the model to format answers in a certain way.</li>
</ul>
<h2>RAG Examples<a href="https://docs.mistral.ai/guides/rag/#rag-examples" rel="noopener noreferrer" target="_blank" title="Direct link to RAG Examples">​</a></h2>
<p>Find multiple RAG cookbooks exploring diverse topics and solutions in our community-driven <a href="https://github.com/mistralai/cookbook" rel="noopener noreferrer" target="_blank">cookbook</a>.</p>
<p>Among them you can find how to perform...</p>
<ul>
<li><strong>RAG with LangChain</strong>: Visit our <a href="https://github.com/mistralai/cookbook/tree/main/third_party/langchain" rel="noopener noreferrer" target="_blank">community cookbook examples</a> to discover how to use LangChain's LangGraph with the Mistral API. These cookbooks cover various implementations, including adaptive RAG, corrective RAG, and self-RAG, showcasing how to integrate LangChain's capabilities for enhanced retrieval-augmented generation.</li>
<li><strong>RAG with LlamaIndex</strong>: Visit our <a href="https://github.com/mistralai/cookbook/blob/main/third_party/LlamaIndex/llamaindex_agentic_rag.ipynb" rel="noopener noreferrer" target="_blank">community cookbook example</a> to learn how to use LlamaIndex with the Mistral API to perform complex queries over multiple documents using a ReAct agent, an autonomous LLM-powered agent capable of using tools.</li>
<li><strong>RAG with Haystack</strong>: Visit our <a href="https://github.com/mistralai/cookbook/blob/main/third_party/Haystack/haystack_chat_with_docs.ipynb" rel="noopener noreferrer" target="_blank">community cookbook example</a> to explore how to use Haystack with the Mistral API for chat functionalities with documents.</li>
</ul>

content_text:
Retrieval-augmented generation (RAG) is an AI framework that synergizes the capabilities of LLMs and information retrieval systems. It's useful to answer questions or generate content leveraging external knowledge. There are two main steps in RAG:

retrieval: retrieve relevant information from a knowledge base or an external source, for example, using text embeddings stored in a vector store.

generation: insert the relevant information to the prompt for the LLM to generate information.

In this guide, we will walk through a very basic example of RAG, you can find more in depth guides in our

cookbooks

.

RAG from scratch

​

This section aims to guide you through the process of building a basic RAG from scratch. We have two goals: firstly, to offer users a comprehensive understanding of the internal workings of RAG and demystify the underlying mechanisms; secondly, to empower you with the essential foundations needed to build an RAG using the minimum required dependencies.

Import needed packages

​

The first step is to install the packages

mistralai

and

faiss-cpu

and import the needed packages:

from mistralai import Mistral

import requests

import numpy as np

import faiss

import os

from getpass import getpass

api_key= getpass("Type your API Key")

client = Mistral(api_key=api_key)

Get data

​

In this very simple example, we are getting data from an essay written by Paul Graham:

response = requests.get('https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt')

text = response.text

We can also save the essay in a local file:

f = open('essay.txt', 'w')

f.write(text)

f.close()

Split document into chunks

​

In a RAG system, it is crucial to split the document into smaller chunks so that it's more effective to identify and retrieve the most relevant information in the retrieval process later. In this example, we simply split our text by character, combine 2048 characters into each chunk, and we get 37 chunks.

chunk_size = 2048

chunks = [text[i:i + chunk_size] for i in range(0, len(text), chunk_size)]

len(chunks)

Output

Considerations:

​

Chunk size

: Depending on your specific use case, it may be necessary to customize or experiment with different chunk sizes and chunk overlap to achieve optimal performance in RAG. For example, smaller chunks can be more beneficial in retrieval processes, as larger text chunks often contain filler text that can obscure the semantic representation. As such, using smaller text chunks in the retrieval process can enable the RAG system to identify and extract relevant information more effectively and accurately. However, it's worth considering the trade-offs that come with using smaller chunks, such as increasing processing time and computational resources.

How to split

: While the simplest method is to split the text by character, there are other options depending on the use case and document structure. For example, to avoid exceeding token limits in API calls, it may be necessary to split the text by tokens. To maintain the cohesiveness of the chunks, it can be useful to split the text by sentences, paragraphs, or HTML headers. If working with code, it's often recommended to split by meaningful code chunks for example using an Abstract Syntax Tree (AST) parser.

Create embeddings for each text chunk

​

For each text chunk, we then need to create text embeddings, which are numeric representations of the text in the vector space. Words with similar meanings are expected to be in closer proximity or have a shorter distance in the vector space.

To create an embedding, use Mistral AI's embeddings API endpoint and the embedding model

mistral-embed

. We create a

get_text_embedding

to get the embedding from a single text chunk and then we use list comprehension to get text embeddings for all text chunks.

def get_text_embedding(input):

embeddings_batch_response = client.embeddings.create(

model="mistral-embed",

inputs=input

)

return embeddings_batch_response.data[0].embedding

text_embeddings = np.array([get_text_embedding(chunk) for chunk in chunks])

Load into a vector database

​

Once we get the text embeddings, a common practice is to store them in a vector database for efficient processing and retrieval. There are several vector database to choose from. In our simple example, we are using an open-source vector database Faiss, which allows for efficient similarity search.

With Faiss, we instantiate an instance of the Index class, which defines the indexing structure of the vector database. We then add the text embeddings to this indexing structure.

import faiss

d = text_embeddings.shape[1]

index = faiss.IndexFlatL2(d)

index.add(text_embeddings)

Considerations:

​

Vector database

: When selecting a vector database, there are several factors to consider including speed, scalability, cloud management, advanced filtering, and open-source vs. closed-source.

Create embeddings for a question

​

Whenever users ask a question, we also need to create embeddings for this question using the same embedding models as before.

question = "What were the two main things the author worked on before college?"

question_embeddings = np.array([get_text_embedding(question)])

Considerations:

​

Hypothetical Document Embeddings (HyDE)

: In some cases, the user's question might not be the most relevant query to use for identifying the relevant context. Instead, it maybe more effective to generate a hypothetical answer or a hypothetical document based on the user's query and use the embeddings of the generated text to retrieve similar text chunks.

Retrieve similar chunks from the vector database

​

We can perform a search on the vector database with

index.search

, which takes two arguments: the first is the vector of the question embeddings, and the second is the number of similar vectors to retrieve. This function returns the distances and the indices of the most similar vectors to the question vector in the vector database. Then based on the returned indices, we can retrieve the actual relevant text chunks that correspond to those indices.

D, I = index.search(question_embeddings, k=2)

retrieved_chunk = [chunks[i] for i in I.tolist()[0]]

Considerations:

​

Retrieval methods

: There are a lot different retrieval strategies. In our example, we are showing a simple similarity search with embeddings. Sometimes when there is metadata available for the data, it's better to filter the data based on the metadata first before performing similarity search. There are also other statistical retrieval methods like TF-IDF and BM25 that use frequency and distribution of terms in the document to identify relevant text chunks.

Retrieved document

: Do we always retrieve individual text chunk as it is? Not always.

Sometime, we would like to include more context around the actual retrieved text chunk. We call the actual retrieved text chunk "child chunk" and our goal is to retrieve a larger "parent chunk" that the "child chunk" belongs to.

On occasion, we might also want to provide weights to our retrieve documents. For example, a time-weighted approach would help us retrieve the most recent document.

One common issue in the retrieval process is the "lost in the middle" problem where the information in the middle of a long context gets lost. Our models have tried to mitigate this issue. For example, in the passkey task, our models have demonstrated the ability to find a "needle in a haystack" by retrieving a randomly inserted passkey within a long prompt, up to 32k context length. However, it is worth considering experimenting with reordering the document to determine if placing the most relevant chunks at the beginning and end leads to improved results.

Combine context and question in a prompt and generate response

​

Finally, we can offer the retrieved text chunks as the context information within the prompt. Here is a prompt template where we can include both the retrieved text and user question in the prompt.

prompt = f"""

Context information is below.

---------------------

{retrieved_chunk}

---------------------

Given the context information and not prior knowledge, answer the query.

Query: {question}

Answer:

"""

Then we can use the Mistral chat completion API to chat with a Mistral model (e.g., mistral-medium-latest) and generate answers based on the user question and the context of the question.

def run_mistral(user_message, model="mistral-large-latest"):

messages = [

{

"role": "user", "content": user_message

}

]

chat_response = client.chat.complete(

model=model,

messages=messages

)

return (chat_response.choices[0].message.content)

run_mistral(prompt)

Output:

'The two main things the author worked on before college were writing and programming. They wrote short stories and tried writing programs on an IBM 1401 in 9th grade.'

Considerations:

​

Prompting techniques

: Most of the prompting techniques can be used in developing a RAG system as well. For example, we can use few-shot learning to guide the model's answers by providing a few examples. Additionally, we can explicitly instruct the model to format answers in a certain way.

RAG Examples

​

Find multiple RAG cookbooks exploring diverse topics and solutions in our community-driven

cookbook

.

Among them you can find how to perform...

RAG with LangChain

: Visit our

community cookbook examples

to discover how to use LangChain's LangGraph with the Mistral API. These cookbooks cover various implementations, including adaptive RAG, corrective RAG, and self-RAG, showcasing how to integrate LangChain's capabilities for enhanced retrieval-augmented generation.

RAG with LlamaIndex

: Visit our

community cookbook example

to learn how to use LlamaIndex with the Mistral API to perform complex queries over multiple documents using a ReAct agent, an autonomous LLM-powered agent capable of using tools.

RAG with Haystack

: Visit our

community cookbook example

to explore how to use Haystack with the Mistral API for chat functionalities with documents.


title: Native RAG vs. Agentic RAG: Which Approach Advances Enterprise AI Decision-Making? - MarkTechPost
added_at: 2025-08-28 00:50:48
thumb_url: data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7
url:
https://www.marktechpost.com/2025/08/22/native-rag-vs-agentic-rag-which-approach-advances-enterprise-ai-decision-making/#origin=https%3A%2F%2Fwww.google.com%2F&cap=swipe,education&webview=1&dialog=1&viewport=natural&visibilityState=prerender&prerenderSize=1&viewerUrl=https%3A%2F%2Fwww.google.com%2Famp%2Fs%2Fwww-marktechpost-com.cdn.ampproject.org%2Fc%2Fs%2Fwww.marktechpost.com%2F2025%2F08%2F22%2Fnative-rag-vs-agentic-rag-which-approach-advances-enterprise-ai-decision-making%3Fusqp=mq331AQGsAEggAID&amp_kit=1

content_html:
<p>Retrieval-Augmented Generation (RAG) has emerged as a cornerstone technique for enhancing Large Language Models (LLMs) with real-time, domain-specific knowledge. But the landscape is rapidly shifting—today, the most common implementations are “Native RAG” pipelines, and a new paradigm called “Agentic RAG” is redefining what’s possible in AI-powered information synthesis and decision support.</p>
<h2><strong>Native RAG: The Standard Pipeline</strong></h2>
<h3><strong>Architecture</strong></h3>
<p>A Native RAG pipeline harnesses retrieval and generation-based methods to answer complex queries while ensuring accuracy and relevance. The pipeline typically involves:</p>
<ul>
<li><strong>Query Processing &amp; Embedding:</strong> The user’s question is rewritten, if needed, embedded into a vector representation using an LLM or dedicated embedding model, and prepared for semantic search.</li>
<li><strong>Retrieval:</strong> The system searches a vector database or document store, identifying top-k relevant chunks using similarity metrics (cosine, Euclidean, dot product). Efficient ANN algorithms optimize this stage for speed and scalability.</li>
<li><strong>Reranking:</strong> Retrieved results are reranked based on relevance, recency, domain-specificity, or user preference. Reranking models—ranging from rule-based to fine-tuned <a href="https://www.marktechpost.com/2025/01/14/what-is-machine-learning-ml/" rel="noopener noreferrer" target="_blank">ML</a> systems—prioritize the highest-quality information.</li>
<li><strong>Synthesis &amp; Generation:</strong> The LLM synthesizes the reranked information to generate a coherent, context-aware response for the user.</li>
</ul>
<h3><strong>Common Optimizations</strong></h3>
<p>Recent advances include dynamic reranking (adjusting depth by query complexity), fusion-based strategies that aggregate rankings from multiple queries, and hybrid approaches that combine semantic partitioning with agent-based selection for optimal retrieval robustness and latency.</p>
<h2><strong>Agentic RAG: Autonomous, Multi-Agent Information Workflows</strong></h2>
<h3><strong>What Is Agentic RAG?</strong></h3>
<p>Agentic RAG is an agent-based approach to RAG, leveraging multiple autonomous agents to answer questions and process documents in a highly coordinated fashion. Rather than a single retrieval/generation pipeline, Agentic RAG structures its workflow for deep reasoning, multi-document comparison, planning, and real-time adaptability.</p>
<h3><strong>Key Components</strong></h3>
<figure>ComponentDescription<strong>Document Agent</strong>Each document is assigned its own agent, able to answer queries about the document and perform summary tasks, working independently within its scope.<strong>Meta-Agent</strong>Orchestrates all document agents, managing their interactions, integrating outputs, and synthesizing a comprehensive answer or action.</figure>
<h3><strong>Features and Benefits</strong></h3>
<ul>
<li><strong>Autonomy:</strong> Agents operate independently, retrieving, processing, and generating answers or actions for specific documents or tasks.</li>
<li><strong>Adaptability:</strong> The system dynamically adjusts its strategy (e.g., reranking depth, document prioritization, tool selection) based on new queries or changing data contexts.</li>
<li><strong>Proactivity:</strong> Agents anticipate needs, take preemptive steps towards goals (e.g., pulling additional sources or suggesting actions), and learn from previous interactions.</li>
</ul>
<h3><strong>Advanced Capabilities</strong></h3>
<p>Agentic RAG goes beyond “passive” retrieval—agents can compare documents, summarize or contrast specific sections, aggregate multi-source insights, and even invoke tools or APIs for enriched reasoning. This enables:</p>
<ul>
<li>Automated research and multi-database aggregation</li>
<li>Complex decision support (e.g., comparing technical features, summarizing key differences across product sheets)</li>
<li>Executive support tasks that require independent synthesis and real-time action recommendation.</li>
</ul>
<h2><strong>Applications</strong></h2>
<p>Agentic RAG is ideal for scenarios where nuanced information processing and decision-making are required:</p>
<ul>
<li><strong>Enterprise Knowledge Management:</strong> Coordinating answers across heterogeneous internal repositories</li>
<li><strong>AI-Driven Research Assistants:</strong> Cross-document synthesis for technical writers, analysts, or executives</li>
<li><strong>Automated Action Workflows:</strong> Triggering actions (e.g., responding to invitations, updating records) after multi-step reasoning over documents or databases.</li>
<li><strong>Complex Compliance and Security Audits:</strong> Aggregating and comparing evidence from varied sources in real time.</li>
</ul>
<figure><img alt="" loading="lazy" src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7"/></figure>
<h2><strong>Conclusion</strong></h2>
<p>Native RAG pipelines have standardized the process of embedding, retrieving, reranking, and synthesizing answers from external data, enabling LLMs to serve as dynamic knowledge engines. Agentic RAG pushes the boundaries even further—by introducing autonomous agents, orchestration layers, and proactive, adaptive workflows, it transforms RAG from a retrieval tool into a full-blown agentic framework for advanced reasoning and multi-document intelligence.</p>
<p>Organizations seeking to move beyond basic augmentation—and into realms of deep, flexible AI orchestration—will find in Agentic RAG the blueprint for the next generation of intelligent systems.</p>
<p></p><p></p><p>Michal Sutter is a data science professional with a Master of Science in Data Science from the University of Padova. With a solid foundation in statistical analysis, machine learning, and data engineering, Michal excels at transforming complex datasets into actionable insights.</p><p></p>

content_text:
Retrieval-Augmented Generation (RAG) has emerged as a cornerstone technique for enhancing Large Language Models (LLMs) with real-time, domain-specific knowledge. But the landscape is rapidly shifting—today, the most common implementations are “Native RAG” pipelines, and a new paradigm called “Agentic RAG” is redefining what’s possible in AI-powered information synthesis and decision support.

Native RAG: The Standard Pipeline

Architecture

A Native RAG pipeline harnesses retrieval and generation-based methods to answer complex queries while ensuring accuracy and relevance. The pipeline typically involves:

Query Processing & Embedding:

The user’s question is rewritten, if needed, embedded into a vector representation using an LLM or dedicated embedding model, and prepared for semantic search.

Retrieval:

The system searches a vector database or document store, identifying top-k relevant chunks using similarity metrics (cosine, Euclidean, dot product). Efficient ANN algorithms optimize this stage for speed and scalability.

Reranking:

Retrieved results are reranked based on relevance, recency, domain-specificity, or user preference. Reranking models—ranging from rule-based to fine-tuned

ML

systems—prioritize the highest-quality information.

Synthesis & Generation:

The LLM synthesizes the reranked information to generate a coherent, context-aware response for the user.

Common Optimizations

Recent advances include dynamic reranking (adjusting depth by query complexity), fusion-based strategies that aggregate rankings from multiple queries, and hybrid approaches that combine semantic partitioning with agent-based selection for optimal retrieval robustness and latency.

Agentic RAG: Autonomous, Multi-Agent Information Workflows

What Is Agentic RAG?

Agentic RAG is an agent-based approach to RAG, leveraging multiple autonomous agents to answer questions and process documents in a highly coordinated fashion. Rather than a single retrieval/generation pipeline, Agentic RAG structures its workflow for deep reasoning, multi-document comparison, planning, and real-time adaptability.

Key Components

ComponentDescription

Document Agent

Each document is assigned its own agent, able to answer queries about the document and perform summary tasks, working independently within its scope.

Meta-Agent

Orchestrates all document agents, managing their interactions, integrating outputs, and synthesizing a comprehensive answer or action.

Features and Benefits

Autonomy:

Agents operate independently, retrieving, processing, and generating answers or actions for specific documents or tasks.

Adaptability:

The system dynamically adjusts its strategy (e.g., reranking depth, document prioritization, tool selection) based on new queries or changing data contexts.

Proactivity:

Agents anticipate needs, take preemptive steps towards goals (e.g., pulling additional sources or suggesting actions), and learn from previous interactions.

Advanced Capabilities

Agentic RAG goes beyond “passive” retrieval—agents can compare documents, summarize or contrast specific sections, aggregate multi-source insights, and even invoke tools or APIs for enriched reasoning. This enables:

Automated research and multi-database aggregation

Complex decision support (e.g., comparing technical features, summarizing key differences across product sheets)

Executive support tasks that require independent synthesis and real-time action recommendation.

Applications

Agentic RAG is ideal for scenarios where nuanced information processing and decision-making are required:

Enterprise Knowledge Management:

Coordinating answers across heterogeneous internal repositories

AI-Driven Research Assistants:

Cross-document synthesis for technical writers, analysts, or executives

Automated Action Workflows:

Triggering actions (e.g., responding to invitations, updating records) after multi-step reasoning over documents or databases.

Complex Compliance and Security Audits:

Aggregating and comparing evidence from varied sources in real time.

Conclusion

Native RAG pipelines have standardized the process of embedding, retrieving, reranking, and synthesizing answers from external data, enabling LLMs to serve as dynamic knowledge engines. Agentic RAG pushes the boundaries even further—by introducing autonomous agents, orchestration layers, and proactive, adaptive workflows, it transforms RAG from a retrieval tool into a full-blown agentic framework for advanced reasoning and multi-document intelligence.

Organizations seeking to move beyond basic augmentation—and into realms of deep, flexible AI orchestration—will find in Agentic RAG the blueprint for the next generation of intelligent systems.

Michal Sutter is a data science professional with a Master of Science in Data Science from the University of Padova. With a solid foundation in statistical analysis, machine learning, and data engineering, Michal excels at transforming complex datasets into actionable insights.


url: https://www.xda-developers.com/how-isp-internet-know-using-vpn/
title: Here's how your ISP and the entire internet know that you're using a VPN
added_at: 2025-08-28 00:50:55
content_html:
<p>Virtual Private Networks (VPNs) have become ubiquitous tools on the internet of today. A VPN creates an encrypted tunnel between your device and a remote server, routing your traffic through that server to mask your real IP address and location. In a sense, it kind of shifts your ISP from your local provider to another, often offshore alternative. Although VPN technology began as a way for businesses to let employees securely access corporate networks in the 1990s, it's since evolved into a common privacy, anti-censorship, and geoblocking circumvention tool. In fact, one of the first widely used VPN protocols, Microsoft's PPTP, was introduced in 1996 as a method that enabled encrypted tunnels over the public internet. Fast forward to today, and     <a href="https://www.xda-developers.com/vpn-cheat-online-game-hackers-internet/" rel="noopener noreferrer" target="_blank">VPN usage</a> has surged globally.    </p>
<p>VPNs are essential for a variety of reasons, ranging from circumventing heavy internet censorship and surveillance in many regions around the world to accessing your favorite show on Netflix that's only available in one specific country. However, many countries engaged in internet censorship ban VPN usage for the purpose of circumventing those blocks, and Netflix itself actively tries to detect VPNs and may only show you globally available content or restrict your access entirely until you turn it off.</p> <p>Yet you may be wondering, given that a VPN is all about privacy, <em>how do they know?</em> There is a range of measures companies can undertake, from the simple to the sophisticated.</p> <h2>
                        IP address blacklists and geolocation checks
               </h2><h3>
            Your IP is in a database somewhere
    </h3>
<p> Every internet connection, whether a VPN or not, exposes an IP address to the destination service. Companies will often analyze the requesting IP address and its associated metadata to gather all kinds of information about the requesting user, and this can include a determination on whether it's coming from a VPN.     <a href="https://www.xda-developers.com/reasons-self-host-everything-own-email-server/" rel="noopener noreferrer" target="_blank">IP reputation and blacklists</a> are the first line of defense in this area, and services maintain or subscribe to databases of IP ranges known to belong to VPN providers, cloud hosting companies, or proxy services. If your current IP appears on one of these "VPN/proxy IP" lists, the service can assume you're using a VPN and block or restrict your access.    </p>
<p>Netflix is a company that will often make use of this approach. The simplest technique Netflix uses is checking if your IP is associated with a VPN or hosting provider. Companies like MaxMind compile lists of IP addresses tied to data centers or known VPN servers, and Netflix utilizes such data to blacklist those IPs. They may not <em>specifically</em> use MaxMind, but there are companies out there that exist to compile such lists and sell them to businesses for analytical and security reasons.</p> <p>Over the years, Netflix and other streaming services have become increasingly aggressive, even banning some cloud data center ranges (AWS, Azure, DigitalOcean, and more) outright since ordinary residential customers wouldn't typically stream from those networks. Years ago, while I couldn't stream on Netflix using a typical consumer VPN,     <a href="https://www.xda-developers.com/reasons-running-own-vpn-server-instead-vpn-provider/" rel="noopener noreferrer" target="_blank">I could still stream using a VPN hosted on a dedicated Kimsufi box in France</a>. Eventually, though, that was blocked too, and MaxMind identified it as an "OVHcloud" device.    </p>
<p>Banking and financial websites use similar tactics, but for different reasons; mainly fraud prevention. Many banks will flag or even block logins coming from IP addresses linked to VPNs or foreign locations not consistent with the customer's profile, and in the past, I remember needing to flag to my bank when I intended to travel and what country I would be in so that my account wouldn't be locked when I logged in. If you normally bank from, say, New York and suddenly appear with an IP in Europe (or one belonging to an Amazon AWS server), the bank's security system may suspect account takeover or fraud. In fact, some major banks (a notable example being <a href="https://www.privatebank.bankofamerica.com/articles/family-cyber-security-tips.html" rel="noopener noreferrer nofollow" target="_blank">Bank of America</a>) outright block most VPN connections, forcing users to disconnect the VPN to log in. The rationale is that criminals might abuse VPNs to mask their identities, so one way to cut down fraud is to disallow anonymizing networks altogether. The downside, of course, is that legitimate customers traveling abroad or securing their connection are inconvenienced.</p> <p>Looking past streaming and banking, advertising and analytics companies also monitor IP information to detect VPN usage. There are a couple of reasons for this: these companies are inherently interested in your true location and identity (or an idea of your identity) for both targeting and security purposes. On the one hand, they want to ensure that they're showing advertisements to relevant users (an advertisement for a European service being shown to an American is a waste of money on the side of the advertiser), and they also want to ensure that click fraud isn't taking place. Therefore, when you connect via an IP associated with a data center or known proxy, sophisticated ad networks will treat that as a signal that you might be a bot or hiding your real location.</p>
<p>In online advertising fraud detection, distinguishing data center traffic from residential user traffic is incredibly important. IP reputation services, such as IPQualityScore, IP2Location, and more, classify IPs by ISP type and anonymity status. Specifically, flagging an IP as a "hosting provider/VPN" address compared to an ISP-issued residential address. Many websites will respond to a high-risk IP by showing a CAPTCHA, limiting functionality, or simply logging that information for later risk analysis. Both Google and Reddit do this, with the former using a CAPTCHA when a request from a suspicious IP is logged, and the latter seemingly trying to crack down on spam and web-scraping abuse. In short, if your IP address comes from an unexpected source or location, modern web services are quick to notice.</p> <p>On that note of CAPTCHA, have you ever noticed that some sites, as they require you to solve more and more of these puzzles, seemingly make them harder as you complete them? Maybe they require you to do more of them, or the images get fuzzier, or the text gets harder to read. This is because the site strongly suspects that you're a bot, and so it throws harder problems at you to ensure that you are a real person.</p>
<p>All of these checks here heavily rely on the expected geolocation of the user or the information associated with the IP address making the connection. Services compare the geolocation of your IP (from GeoIP databases) to the expected location of you or the service you're accessing. If there's a mismatch, such as you claim to be an American user in your profile and have used an American payment card, but the IP address accessing the service is registered in another country, it raises a red flag. Even non-security-focused sites may use IP geolocation data to personalize content, and a sudden change can trigger a verification process.</p> <p></p><h2>
                        DNS and STUN requests
               </h2><h3>
            A sneaky way to check for configuration mistakes
    </h3>
<p> Sometimes, it's not just the IP address that flags you as a VPN user; it can be the surrounding network clues that give the game away. DNS (Domain Name System) queries are what translate website names to IP addresses. Normally, when using a VPN, your DNS requests are also sent to a DNS server provided by the VPN, keeping them consistent with your new IP location. However, misconfigurations or certain apps can leak DNS requests outside the VPN tunnel, and companies can exploit this. Netflix, for instance, checks for DNS leaks by seeing whether the location of the CDN you requested a video stream from matches your supposed location. If your IP says "London" but your DNS request resolved to a server in, say, Ohio, Netflix now knows that you're using a VPN.</p>
<p>Generally speaking, DNS leaks are something that any website can watch out for. All they need to do is trigger a DNS lookup (for example, loading an image from a unique subdomain) and see what DNS server resolves it; a site can then tell if that resolution happened through a DNS that doesn't make sense for your current connection. Your browser might use your ISP's DNS instead of the VPN's, and the site may learn your real ISP or region. While VPN providers aim to prevent DNS leaks, they still happen with an improperly configured VPN.</p> <p>Another sneaky tactic companies can use is WebRTC, though this applies more when your browser is what implements a proxy or VPN, rather than a VPN being handled by your operating system. WebRTC allows for peer-to-peer connections and can reveal a user's IP through STUN (Session Traversal Utilities for NAT) requests, as many browsers have WebRTC enabled by default. If a site then sees two IP addresses for you, one from your typical connection to the site and one from WebRTC, it's clear that you're using a VPN.</p>
<p>In other words, it's not just where you're connecting from that can flag your connection, but rather the inconsistencies that can be associated with your connection. DNS leaks and WebRTC are two common tactics, but even your system timezone can set off alarm bells, as a site can fetch this using JavaScript. You can even try this yourself: turn on your VPN, and type this into your browser console:</p>
<p></p><pre>Intl.DateTimeFormat().resolvedOptions().timeZone</pre><p></p> <p>You'll see that even though your VPN is switched on, the timezone doesn't update in your browser. A website can see this, and if it doesn't match the timezone of the region your IP address is connecting from, that's another potential leak vector. It's nowhere near as identifying as your ISP or even your actual IP address, but it can still flag the real country you're connecting from.</p> <p></p><h2>
                        Deep packet inspection
               </h2><h3>
            VPN usage has telltale signs
    </h3>
<p> This is where things get significantly more complicated, and it happens at the ISP or government level rather than individual websites. Since a VPN encrypts your traffic and hides its contents, you might assume an ISP wouldn't be able to tell that you're using a VPN, but this couldn't be further from the truth. What <em>is</em> true is that your ISP can't see what you're doing inside of the VPN tunnel, like the websites you visit or the data you send, but ISPs can certainly see that a lot of encrypted traffic is flowing to and from a single server, especially if that server is located outside of the country. Essentially, from an ISP's viewpoint, instead of your traffic going to many different servers and IP addresses with some plaintext traffic dotted through, practically everything is encapsulated and sent to one IP (the VPN server) in an encrypted form. This alone is a pretty strong indicator.</p>
<p>However, Deep Packet Inspection (DPI) is a significantly more intense method of detection, and can be employed in countries where VPN usage is restricted or even outright banned. It's expensive for an ISP to implement, but a nation-state with a vested interest in preventing VPN usage is likely to have the resources to commit to it. DPI equipment examines packet headers and payloads beyond just the basic IP info, looking for signatures of known protocols and flagging them as potential VPN usage. For example, VPN protocols such as WireGuard, OpenVPN, IPsec, PPTP, and others can potentially be identified by their handshake sequences or byte patterns. For example, OpenVPN uses TLS/SSL, but there are subtle differences in its handshake thanks to <a href="https://openvpn.net/community-docs/hardening-openvpn-security.html" rel="noopener noreferrer nofollow" target="_blank">its use of an HMAC signature and UDP packets</a> that will make it stand out when compared to regular TLS. A common method of hiding this is to use something like <a href="https://shadowsocks.org/" rel="noopener noreferrer nofollow" target="_blank">Shadowsocks</a>, essentially wrapping OpenVPN inside another tunnel.</p> <p>ISPs also have simpler ways to detect VPNs that don't require DPI. They can maintain their own lists of popular VPN server IPs (just like the websites we talked about do) and take note of whether traffic is going to those IPs. Corporate networks or public Wi-Fi providers often block known VPN port numbers (such as UDP 1194 for OpenVPN, or IPsec ports) or use DPI to disallow VPN protocols so that users can't bypass their firewalls. For instance, some workplaces or schools will deliberately prevent any traffic that looks like a VPN to ensure you can't tunnel out and visit disallowed sites.</p>
<p>Even without DPI, an ISP can recognize encryption patterns in packets streaming to and from your connection. Many VPN protocols add characteristic metadata, and examples of this include the IPsec protocol with its identifiable packet headers and older PPTP VPNs, which had easily spotted signatures.     <a href="https://www.xda-developers.com/benefits-using-wireguard-home-lab-openvpn/" rel="noopener noreferrer" target="_blank">WireGuard</a> is another protocol that can easily be detected, as while it focuses on being fast and secure, it makes no attempt at obfuscation.    </p> <p>However, just as websites and VPN providers play cat-and-mouse, so do VPN users and censoring ISPs. There are VPN obfuscation technologies, like Shadowsocks, designed to make VPN traffic look like regular web traffic. There's also <a href="https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyrebird/" rel="noopener noreferrer nofollow" target="_blank">obfs4</a> (which uses concepts and ideas from ScrambleSuit), and a supporting server can be used as a default bridge to the TOR network. However, packet sequence and size can still potentially flag circumvention, as while normal web browsing will produce varying packet sizes in more random sequences, a steadier sequence of packets from a VPN and consistently high-entropy encrypted traffic might raise suspicion.</p>
<p>In countries where VPNs aren't restricted, it's incredibly unlikely that your ISP will employ DPI, and it's still rudimentary to detect consistent VPN usage. While there are VPN services aimed at countries that restrict their usage specifically, most people don't need or care for that, and a more off-the-shelf VPN won't go to such lengths to hide its usage. However, if your ISP looks at your home connection's logs and sees an hours-long encrypted session to a single server belonging to a hosting provider, it's a pretty safe assumption that you were probably using a VPN.</p> <p></p><h2>
                        Most people don't need to worry about VPN detection
               </h2><h3>
            But it can be hard for those who do need to
    </h3>
<p> VPNs have undoubtedly become mainstream thanks to a combination of the internet privacy movement and online restrictions that have been put in place by some U.S. states and countries like the United Kingdom. However, the attributes that make VPNs useful, like anonymity and traffic encryption, are also somewhat paradoxically what make them stand out the most. Plenty of companies have a vested interest in detecting VPN usage, as do some governments, and it's important to be aware of what exactly can draw attention to your VPN usage.</p>
<p>It's a continuous cat-and-mouse game. Each time VPN providers introduce a new way to blend in, such as introducing new IP ranges, stealth protocols, and residential exit nodes, detection systems can step up their game through algorithm refinement and collecting additional data. No single method of detection is foolproof on its own, but combining them all can be enough evidence to flag a connection as coming from a VPN. For the average user, this means that while your VPN can hide what you do online, it's increasingly difficult to hide the fact that you are using a VPN in the first place. Services may not always care about it, but they are at least <em>aware</em> that it's happening. Knowing the detection methods is useful as it can help you to understand what information you might be sharing that you didn't realize.</p> <p>Companies know a VPN when they see one, by design, and as VPN usage grows, the continuous cat-and-mouse game will continue to play on. Privacy often conflicts with the commercial and regulatory interests of companies and governments alike, and it's important to know where the weaknesses are. <a href="https://www.xda-developers.com/more-important-ever-protect-yourself-vpn-not-good/" rel="noopener noreferrer" target="_blank">VPNs aren't a silver bullet to privacy</a>, and there are many other ways you can protect your privacy, too.</p> 

content_text:
Virtual Private Networks (VPNs) have become ubiquitous tools on the internet of today. A VPN creates an encrypted tunnel between your device and a remote server, routing your traffic through that server to mask your real IP address and location. In a sense, it kind of shifts your ISP from your local provider to another, often offshore alternative. Although VPN technology began as a way for businesses to let employees securely access corporate networks in the 1990s, it's since evolved into a common privacy, anti-censorship, and geoblocking circumvention tool. In fact, one of the first widely used VPN protocols, Microsoft's PPTP, was introduced in 1996 as a method that enabled encrypted tunnels over the public internet. Fast forward to today, and

VPN usage

has surged globally.

VPNs are essential for a variety of reasons, ranging from circumventing heavy internet censorship and surveillance in many regions around the world to accessing your favorite show on Netflix that's only available in one specific country. However, many countries engaged in internet censorship ban VPN usage for the purpose of circumventing those blocks, and Netflix itself actively tries to detect VPNs and may only show you globally available content or restrict your access entirely until you turn it off.

Yet you may be wondering, given that a VPN is all about privacy,

how do they know?

There is a range of measures companies can undertake, from the simple to the sophisticated.

IP address blacklists and geolocation checks

Your IP is in a database somewhere

Every internet connection, whether a VPN or not, exposes an IP address to the destination service. Companies will often analyze the requesting IP address and its associated metadata to gather all kinds of information about the requesting user, and this can include a determination on whether it's coming from a VPN.

IP reputation and blacklists

are the first line of defense in this area, and services maintain or subscribe to databases of IP ranges known to belong to VPN providers, cloud hosting companies, or proxy services. If your current IP appears on one of these "VPN/proxy IP" lists, the service can assume you're using a VPN and block or restrict your access.

Netflix is a company that will often make use of this approach. The simplest technique Netflix uses is checking if your IP is associated with a VPN or hosting provider. Companies like MaxMind compile lists of IP addresses tied to data centers or known VPN servers, and Netflix utilizes such data to blacklist those IPs. They may not

specifically

use MaxMind, but there are companies out there that exist to compile such lists and sell them to businesses for analytical and security reasons.

Over the years, Netflix and other streaming services have become increasingly aggressive, even banning some cloud data center ranges (AWS, Azure, DigitalOcean, and more) outright since ordinary residential customers wouldn't typically stream from those networks. Years ago, while I couldn't stream on Netflix using a typical consumer VPN,

I could still stream using a VPN hosted on a dedicated Kimsufi box in France

. Eventually, though, that was blocked too, and MaxMind identified it as an "OVHcloud" device.

Banking and financial websites use similar tactics, but for different reasons; mainly fraud prevention. Many banks will flag or even block logins coming from IP addresses linked to VPNs or foreign locations not consistent with the customer's profile, and in the past, I remember needing to flag to my bank when I intended to travel and what country I would be in so that my account wouldn't be locked when I logged in. If you normally bank from, say, New York and suddenly appear with an IP in Europe (or one belonging to an Amazon AWS server), the bank's security system may suspect account takeover or fraud. In fact, some major banks (a notable example being

Bank of America

) outright block most VPN connections, forcing users to disconnect the VPN to log in. The rationale is that criminals might abuse VPNs to mask their identities, so one way to cut down fraud is to disallow anonymizing networks altogether. The downside, of course, is that legitimate customers traveling abroad or securing their connection are inconvenienced.

Looking past streaming and banking, advertising and analytics companies also monitor IP information to detect VPN usage. There are a couple of reasons for this: these companies are inherently interested in your true location and identity (or an idea of your identity) for both targeting and security purposes. On the one hand, they want to ensure that they're showing advertisements to relevant users (an advertisement for a European service being shown to an American is a waste of money on the side of the advertiser), and they also want to ensure that click fraud isn't taking place. Therefore, when you connect via an IP associated with a data center or known proxy, sophisticated ad networks will treat that as a signal that you might be a bot or hiding your real location.

In online advertising fraud detection, distinguishing data center traffic from residential user traffic is incredibly important. IP reputation services, such as IPQualityScore, IP2Location, and more, classify IPs by ISP type and anonymity status. Specifically, flagging an IP as a "hosting provider/VPN" address compared to an ISP-issued residential address. Many websites will respond to a high-risk IP by showing a CAPTCHA, limiting functionality, or simply logging that information for later risk analysis. Both Google and Reddit do this, with the former using a CAPTCHA when a request from a suspicious IP is logged, and the latter seemingly trying to crack down on spam and web-scraping abuse. In short, if your IP address comes from an unexpected source or location, modern web services are quick to notice.

On that note of CAPTCHA, have you ever noticed that some sites, as they require you to solve more and more of these puzzles, seemingly make them harder as you complete them? Maybe they require you to do more of them, or the images get fuzzier, or the text gets harder to read. This is because the site strongly suspects that you're a bot, and so it throws harder problems at you to ensure that you are a real person.

All of these checks here heavily rely on the expected geolocation of the user or the information associated with the IP address making the connection. Services compare the geolocation of your IP (from GeoIP databases) to the expected location of you or the service you're accessing. If there's a mismatch, such as you claim to be an American user in your profile and have used an American payment card, but the IP address accessing the service is registered in another country, it raises a red flag. Even non-security-focused sites may use IP geolocation data to personalize content, and a sudden change can trigger a verification process.

DNS and STUN requests

A sneaky way to check for configuration mistakes

Sometimes, it's not just the IP address that flags you as a VPN user; it can be the surrounding network clues that give the game away. DNS (Domain Name System) queries are what translate website names to IP addresses. Normally, when using a VPN, your DNS requests are also sent to a DNS server provided by the VPN, keeping them consistent with your new IP location. However, misconfigurations or certain apps can leak DNS requests outside the VPN tunnel, and companies can exploit this. Netflix, for instance, checks for DNS leaks by seeing whether the location of the CDN you requested a video stream from matches your supposed location. If your IP says "London" but your DNS request resolved to a server in, say, Ohio, Netflix now knows that you're using a VPN.

Generally speaking, DNS leaks are something that any website can watch out for. All they need to do is trigger a DNS lookup (for example, loading an image from a unique subdomain) and see what DNS server resolves it; a site can then tell if that resolution happened through a DNS that doesn't make sense for your current connection. Your browser might use your ISP's DNS instead of the VPN's, and the site may learn your real ISP or region. While VPN providers aim to prevent DNS leaks, they still happen with an improperly configured VPN.

Another sneaky tactic companies can use is WebRTC, though this applies more when your browser is what implements a proxy or VPN, rather than a VPN being handled by your operating system. WebRTC allows for peer-to-peer connections and can reveal a user's IP through STUN (Session Traversal Utilities for NAT) requests, as many browsers have WebRTC enabled by default. If a site then sees two IP addresses for you, one from your typical connection to the site and one from WebRTC, it's clear that you're using a VPN.

In other words, it's not just where you're connecting from that can flag your connection, but rather the inconsistencies that can be associated with your connection. DNS leaks and WebRTC are two common tactics, but even your system timezone can set off alarm bells, as a site can fetch this using JavaScript. You can even try this yourself: turn on your VPN, and type this into your browser console:

Intl.DateTimeFormat().resolvedOptions().timeZone

You'll see that even though your VPN is switched on, the timezone doesn't update in your browser. A website can see this, and if it doesn't match the timezone of the region your IP address is connecting from, that's another potential leak vector. It's nowhere near as identifying as your ISP or even your actual IP address, but it can still flag the real country you're connecting from.

Deep packet inspection

VPN usage has telltale signs

This is where things get significantly more complicated, and it happens at the ISP or government level rather than individual websites. Since a VPN encrypts your traffic and hides its contents, you might assume an ISP wouldn't be able to tell that you're using a VPN, but this couldn't be further from the truth. What

is

true is that your ISP can't see what you're doing inside of the VPN tunnel, like the websites you visit or the data you send, but ISPs can certainly see that a lot of encrypted traffic is flowing to and from a single server, especially if that server is located outside of the country. Essentially, from an ISP's viewpoint, instead of your traffic going to many different servers and IP addresses with some plaintext traffic dotted through, practically everything is encapsulated and sent to one IP (the VPN server) in an encrypted form. This alone is a pretty strong indicator.

However, Deep Packet Inspection (DPI) is a significantly more intense method of detection, and can be employed in countries where VPN usage is restricted or even outright banned. It's expensive for an ISP to implement, but a nation-state with a vested interest in preventing VPN usage is likely to have the resources to commit to it. DPI equipment examines packet headers and payloads beyond just the basic IP info, looking for signatures of known protocols and flagging them as potential VPN usage. For example, VPN protocols such as WireGuard, OpenVPN, IPsec, PPTP, and others can potentially be identified by their handshake sequences or byte patterns. For example, OpenVPN uses TLS/SSL, but there are subtle differences in its handshake thanks to

its use of an HMAC signature and UDP packets

that will make it stand out when compared to regular TLS. A common method of hiding this is to use something like

Shadowsocks

, essentially wrapping OpenVPN inside another tunnel.

ISPs also have simpler ways to detect VPNs that don't require DPI. They can maintain their own lists of popular VPN server IPs (just like the websites we talked about do) and take note of whether traffic is going to those IPs. Corporate networks or public Wi-Fi providers often block known VPN port numbers (such as UDP 1194 for OpenVPN, or IPsec ports) or use DPI to disallow VPN protocols so that users can't bypass their firewalls. For instance, some workplaces or schools will deliberately prevent any traffic that looks like a VPN to ensure you can't tunnel out and visit disallowed sites.

Even without DPI, an ISP can recognize encryption patterns in packets streaming to and from your connection. Many VPN protocols add characteristic metadata, and examples of this include the IPsec protocol with its identifiable packet headers and older PPTP VPNs, which had easily spotted signatures.

WireGuard

is another protocol that can easily be detected, as while it focuses on being fast and secure, it makes no attempt at obfuscation.

However, just as websites and VPN providers play cat-and-mouse, so do VPN users and censoring ISPs. There are VPN obfuscation technologies, like Shadowsocks, designed to make VPN traffic look like regular web traffic. There's also

obfs4

(which uses concepts and ideas from ScrambleSuit), and a supporting server can be used as a default bridge to the TOR network. However, packet sequence and size can still potentially flag circumvention, as while normal web browsing will produce varying packet sizes in more random sequences, a steadier sequence of packets from a VPN and consistently high-entropy encrypted traffic might raise suspicion.

In countries where VPNs aren't restricted, it's incredibly unlikely that your ISP will employ DPI, and it's still rudimentary to detect consistent VPN usage. While there are VPN services aimed at countries that restrict their usage specifically, most people don't need or care for that, and a more off-the-shelf VPN won't go to such lengths to hide its usage. However, if your ISP looks at your home connection's logs and sees an hours-long encrypted session to a single server belonging to a hosting provider, it's a pretty safe assumption that you were probably using a VPN.

Most people don't need to worry about VPN detection

But it can be hard for those who do need to

VPNs have undoubtedly become mainstream thanks to a combination of the internet privacy movement and online restrictions that have been put in place by some U.S. states and countries like the United Kingdom. However, the attributes that make VPNs useful, like anonymity and traffic encryption, are also somewhat paradoxically what make them stand out the most. Plenty of companies have a vested interest in detecting VPN usage, as do some governments, and it's important to be aware of what exactly can draw attention to your VPN usage.

It's a continuous cat-and-mouse game. Each time VPN providers introduce a new way to blend in, such as introducing new IP ranges, stealth protocols, and residential exit nodes, detection systems can step up their game through algorithm refinement and collecting additional data. No single method of detection is foolproof on its own, but combining them all can be enough evidence to flag a connection as coming from a VPN. For the average user, this means that while your VPN can hide what you do online, it's increasingly difficult to hide the fact that you are using a VPN in the first place. Services may not always care about it, but they are at least

aware

that it's happening. Knowing the detection methods is useful as it can help you to understand what information you might be sharing that you didn't realize.

Companies know a VPN when they see one, by design, and as VPN usage grows, the continuous cat-and-mouse game will continue to play on. Privacy often conflicts with the commercial and regulatory interests of companies and governments alike, and it's important to know where the weaknesses are.

VPNs aren't a silver bullet to privacy

, and there are many other ways you can protect your privacy, too.


url: https://python.langchain.com/docs/tutorials/rag/
title: Build a Retrieval Augmented Generation (RAG) App: Part 1 | 🦜️🔗 LangChain
added_at: 2025-08-28 01:27:01
thumb_url: https://python.langchain.com/assets/images/rag_indexing-8160f90a90a33253d0154659cf7d453f.png
content_html:
<header><h1>Build a Retrieval Augmented Generation (RAG) App: Part 1</h1></header>
<p>One of the most powerful applications enabled by LLMs is sophisticated question-answering (Q&amp;A) chatbots. These are applications that can answer questions about specific source information. These applications use a technique known as Retrieval Augmented Generation, or <a href="https://python.langchain.com/docs/concepts/rag/" rel="noopener noreferrer" target="_blank">RAG</a>.</p>
<p>This is a multi-part tutorial:</p>
<ul>
<li><a href="https://python.langchain.com/docs/tutorials/rag/" rel="noopener noreferrer" target="_blank">Part 1</a> (this guide) introduces RAG and walks through a minimal implementation.</li>
<li><a href="https://python.langchain.com/docs/tutorials/qa_chat_history/" rel="noopener noreferrer" target="_blank">Part 2</a> extends the implementation to accommodate conversation-style interactions and multi-step retrieval processes.</li>
</ul>
<p>This tutorial will show how to build a simple Q&amp;A application
over a text data source. Along the way we’ll go over a typical Q&amp;A
architecture and highlight additional resources for more advanced Q&amp;A techniques. We’ll also see
how LangSmith can help us trace and understand our application.
LangSmith will become increasingly helpful as our application grows in
complexity.</p>
<p>If you're already familiar with basic retrieval, you might also be interested in
this <a href="https://python.langchain.com/docs/concepts/retrieval/" rel="noopener noreferrer" target="_blank">high-level overview of different retrieval techniques</a>.</p>
<p><strong>Note</strong>: Here we focus on Q&amp;A for unstructured data. If you are interested for RAG over structured data, check out our tutorial on doing <a href="https://python.langchain.com/docs/tutorials/sql_qa/" rel="noopener noreferrer" target="_blank">question/answering over SQL data</a>.</p>
<h2>Overview<a href="https://python.langchain.com/docs/tutorials/rag/#overview" rel="noopener noreferrer" target="_blank" title="Direct link to Overview">​</a></h2>
<p>A typical RAG application has two main components:</p>
<p><strong>Indexing</strong>: a pipeline for ingesting data from a source and indexing it. <em>This usually happens offline.</em></p>
<p><strong>Retrieval and generation</strong>: the actual RAG chain, which takes the user query at run time and retrieves the relevant data from the index, then passes that to the model.</p>
<p>Note: the indexing portion of this tutorial will largely follow the <a href="https://python.langchain.com/docs/tutorials/retrievers/" rel="noopener noreferrer" target="_blank">semantic search tutorial</a>.</p>
<p>The most common full sequence from raw data to answer looks like:</p>
<h3>Indexing<a href="https://python.langchain.com/docs/tutorials/rag/#indexing" rel="noopener noreferrer" target="_blank" title="Direct link to Indexing">​</a></h3>
<ol>
<li><strong>Load</strong>: First we need to load our data. This is done with <a href="https://python.langchain.com/docs/concepts/document_loaders/" rel="noopener noreferrer" target="_blank">Document Loaders</a>.</li>
<li><strong>Split</strong>: <a href="https://python.langchain.com/docs/concepts/text_splitters/" rel="noopener noreferrer" target="_blank">Text splitters</a> break large <code>Documents</code> into smaller chunks. This is useful both for indexing data and passing it into a model, as large chunks are harder to search over and won't fit in a model's finite context window.</li>
<li><strong>Store</strong>: We need somewhere to store and index our splits, so that they can be searched over later. This is often done using a <a href="https://python.langchain.com/docs/concepts/vectorstores/" rel="noopener noreferrer" target="_blank">VectorStore</a> and <a href="https://python.langchain.com/docs/concepts/embedding_models/" rel="noopener noreferrer" target="_blank">Embeddings</a> model.</li>
</ol>
<p><img alt="index_diagram" loading="lazy" src="https://python.langchain.com/assets/images/rag_indexing-8160f90a90a33253d0154659cf7d453f.png"/></p>
<h3>Retrieval and generation<a href="https://python.langchain.com/docs/tutorials/rag/#retrieval-and-generation" rel="noopener noreferrer" target="_blank" title="Direct link to Retrieval and generation">​</a></h3>
<ol>
<li><strong>Retrieve</strong>: Given a user input, relevant splits are retrieved from storage using a <a href="https://python.langchain.com/docs/concepts/retrievers/" rel="noopener noreferrer" target="_blank">Retriever</a>.</li>
<li><strong>Generate</strong>: A <a href="https://python.langchain.com/docs/concepts/chat_models/" rel="noopener noreferrer" target="_blank">ChatModel</a> / <a href="https://python.langchain.com/docs/concepts/text_llms/" rel="noopener noreferrer" target="_blank">LLM</a> produces an answer using a prompt that includes both the question with the retrieved data</li>
</ol>
<p><img alt="retrieval_diagram" loading="lazy" src="https://python.langchain.com/assets/images/rag_retrieval_generation-1046a4668d6bb08786ef73c56d4f228a.png"/></p>
<p>Once we've indexed our data, we will use <a href="https://langchain-ai.github.io/langgraph/" rel="noopener noreferrer" target="_blank">LangGraph</a> as our orchestration framework to implement the retrieval and generation steps.</p>
<h2>Setup<a href="https://python.langchain.com/docs/tutorials/rag/#setup" rel="noopener noreferrer" target="_blank" title="Direct link to Setup">​</a></h2>
<h3>Jupyter Notebook<a href="https://python.langchain.com/docs/tutorials/rag/#jupyter-notebook" rel="noopener noreferrer" target="_blank" title="Direct link to Jupyter Notebook">​</a></h3>
<p>This and other tutorials are perhaps most conveniently run in a <a href="https://jupyter.org/" rel="noopener noreferrer" target="_blank">Jupyter notebooks</a>. Going through guides in an interactive environment is a great way to better understand them. See <a href="https://jupyter.org/install" rel="noopener noreferrer" target="_blank">here</a> for instructions on how to install.</p>
<h3>Installation<a href="https://python.langchain.com/docs/tutorials/rag/#installation" rel="noopener noreferrer" target="_blank" title="Direct link to Installation">​</a></h3>
<p>This tutorial requires these langchain dependencies:</p>
<pre><code>%pip install --quiet --upgrade langchain-text-splitters langchain-community langgraph<br/></code></pre><pre><code>conda install langchain-text-splitters langchain-community langgraph -c conda-forge<br/></code></pre>
<p>For more details, see our <a href="https://python.langchain.com/docs/how_to/installation/" rel="noopener noreferrer" target="_blank">Installation guide</a>.</p>
<h3>LangSmith<a href="https://python.langchain.com/docs/tutorials/rag/#langsmith" rel="noopener noreferrer" target="_blank" title="Direct link to LangSmith">​</a></h3>
<p>Many of the applications you build with LangChain will contain multiple steps with multiple invocations of LLM calls.
As these applications get more complex, it becomes crucial to be able to inspect what exactly is going on inside your chain or agent.
The best way to do this is with <a href="https://smith.langchain.com" rel="noopener noreferrer" target="_blank">LangSmith</a>.</p>
<p>After you sign up at the link above, make sure to set your environment variables to start logging traces:</p>
<pre><code>export LANGSMITH_TRACING="true"<br/>export LANGSMITH_API_KEY="..."<br/></code></pre>
<p>Or, if in a notebook, you can set them with:</p>
<pre><code>import getpass<br/>import os<br/><br/>os.environ["LANGSMITH_TRACING"] = "true"<br/>os.environ["LANGSMITH_API_KEY"] = getpass.getpass()<br/></code></pre>
<h2>Components<a href="https://python.langchain.com/docs/tutorials/rag/#components" rel="noopener noreferrer" target="_blank" title="Direct link to Components">​</a></h2>
<p>We will need to select three components from LangChain's suite of integrations.</p>
<pre><code>pip install -qU "langchain[google-genai]"<br/></code></pre><pre><code>import getpass<br/>import os<br/><br/>if not os.environ.get("GOOGLE_API_KEY"):<br/>  os.environ["GOOGLE_API_KEY"] = getpass.getpass("Enter API key for Google Gemini: ")<br/><br/>from langchain.chat_models import init_chat_model<br/><br/>llm = init_chat_model("gemini-2.5-flash", model_provider="google_genai")<br/></code></pre>
<pre><code>pip install -qU langchain-openai<br/></code></pre><pre><code>import getpass<br/>import os<br/><br/>if not os.environ.get("OPENAI_API_KEY"):<br/>  os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter API key for OpenAI: ")<br/><br/>from langchain_openai import OpenAIEmbeddings<br/><br/>embeddings = OpenAIEmbeddings(model="text-embedding-3-large")<br/></code></pre>
<pre><code>pip install -qU langchain-core<br/></code></pre><pre><code>from langchain_core.vectorstores import InMemoryVectorStore<br/><br/>vector_store = InMemoryVectorStore(embeddings)<br/></code></pre>
<h2>Preview<a href="https://python.langchain.com/docs/tutorials/rag/#preview" rel="noopener noreferrer" target="_blank" title="Direct link to Preview">​</a></h2>
<p>In this guide we’ll build an app that answers questions about the website's content. The specific website we will use is the <a href="https://lilianweng.github.io/posts/2023-06-23-agent/" rel="noopener noreferrer" target="_blank">LLM Powered Autonomous
Agents</a> blog post
by Lilian Weng, which allows us to ask questions about the contents of
the post.</p>
<p>We can create a simple indexing pipeline and RAG chain to do this in ~50
lines of code.</p>
<pre><code>import bs4<br/>from langchain import hub<br/>from langchain_community.document_loaders import WebBaseLoader<br/>from langchain_core.documents import Document<br/>from langchain_text_splitters import RecursiveCharacterTextSplitter<br/>from langgraph.graph import START, StateGraph<br/>from typing_extensions import List, TypedDict<br/><br/><br/>loader = WebBaseLoader(<br/>    web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),<br/>    bs_kwargs=dict(<br/>        parse_only=bs4.SoupStrainer(<br/>            class_=("post-content", "post-title", "post-header")<br/>        )<br/>    ),<br/>)<br/>docs = loader.load()<br/><br/>text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)<br/>all_splits = text_splitter.split_documents(docs)<br/><br/><br/>_ = vector_store.add_documents(documents=all_splits)<br/><br/><br/><br/><br/>prompt = hub.pull("rlm/rag-prompt")<br/><br/><br/><br/>class State(TypedDict):<br/>    question: str<br/>    context: List[Document]<br/>    answer: str<br/><br/><br/><br/>def retrieve(state: State):<br/>    retrieved_docs = vector_store.similarity_search(state["question"])<br/>    return {"context": retrieved_docs}<br/><br/><br/>def generate(state: State):<br/>    docs_content = "\n\n".join(doc.page_content for doc in state["context"])<br/>    messages = prompt.invoke({"question": state["question"], "context": docs_content})<br/>    response = llm.invoke(messages)<br/>    return {"answer": response.content}<br/><br/><br/><br/>graph_builder = StateGraph(State).add_sequence([retrieve, generate])<br/>graph_builder.add_edge(START, "retrieve")<br/>graph = graph_builder.compile()<br/></code></pre>
<pre><code>response = graph.invoke({"question": "What is Task Decomposition?"})<br/>print(response["answer"])<br/></code></pre>
<pre><code>Task Decomposition is the process of breaking down a complicated task into smaller, manageable steps to facilitate easier execution and understanding. Techniques like Chain of Thought (CoT) and Tree of Thoughts (ToT) guide models to think step-by-step, allowing them to explore multiple reasoning possibilities. This method enhances performance on complex tasks and provides insight into the model's thinking process.<br/></code></pre>
<p>Check out the <a href="https://smith.langchain.com/public/65030797-7efa-4356-a7bd-b54b3dc70e17/r" rel="noopener noreferrer" target="_blank">LangSmith
trace</a>.</p>
<h2>Detailed walkthrough<a href="https://python.langchain.com/docs/tutorials/rag/#detailed-walkthrough" rel="noopener noreferrer" target="_blank" title="Direct link to Detailed walkthrough">​</a></h2>
<p>Let’s go through the above code step-by-step to really understand what’s
going on.</p>
<h2>1. Indexing<a href="https://python.langchain.com/docs/tutorials/rag/#indexing" rel="noopener noreferrer" target="_blank" title="Direct link to 1. Indexing">​</a></h2>
<h3>Loading documents<a href="https://python.langchain.com/docs/tutorials/rag/#loading-documents" rel="noopener noreferrer" target="_blank" title="Direct link to Loading documents">​</a></h3>
<p>We need to first load the blog post contents. We can use
<a href="https://python.langchain.com/docs/concepts/document_loaders/" rel="noopener noreferrer" target="_blank">DocumentLoaders</a>
for this, which are objects that load in data from a source and return a
list of
<a href="https://python.langchain.com/api_reference/core/documents/langchain_core.documents.base.Document.html" rel="noopener noreferrer" target="_blank">Document</a>
objects.</p>
<p>In this case we’ll use the
<a href="https://python.langchain.com/docs/integrations/document_loaders/web_base/" rel="noopener noreferrer" target="_blank">WebBaseLoader</a>,
which uses <code>urllib</code> to load HTML from web URLs and <code>BeautifulSoup</code> to
parse it to text. We can customize the HTML -&gt; text parsing by passing
in parameters into the <code>BeautifulSoup</code> parser via <code>bs_kwargs</code> (see
<a href="https://beautiful-soup-4.readthedocs.io/en/latest/#beautifulsoup" rel="noopener noreferrer" target="_blank">BeautifulSoup
docs</a>).
In this case only HTML tags with class “post-content”, “post-title”, or
“post-header” are relevant, so we’ll remove all others.</p>
<pre><code>import bs4<br/>from langchain_community.document_loaders import WebBaseLoader<br/><br/><br/>bs4_strainer = bs4.SoupStrainer(class_=("post-title", "post-header", "post-content"))<br/>loader = WebBaseLoader(<br/>    web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),<br/>    bs_kwargs={"parse_only": bs4_strainer},<br/>)<br/>docs = loader.load()<br/><br/>assert len(docs) == 1<br/>print(f"Total characters: {len(docs[0].page_content)}")<br/></code></pre>
<pre><code>print(docs[0].page_content[:500])<br/></code></pre>
<pre><code><br/><br/>      LLM Powered Autonomous Agents<br/>    <br/>Date: June 23, 2023  |  Estimated Reading Time: 31 min  |  Author: Lilian Weng<br/><br/><br/>Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.<br/>Agent System Overview#<br/>In<br/></code></pre>
<h4>Go deeper<a href="https://python.langchain.com/docs/tutorials/rag/#go-deeper" rel="noopener noreferrer" target="_blank" title="Direct link to Go deeper">​</a></h4>
<p><code>DocumentLoader</code>: Object that loads data from a source as list of <code>Documents</code>.</p>
<ul>
<li><a href="https://python.langchain.com/docs/how_to/#document-loaders" rel="noopener noreferrer" target="_blank">Docs</a>:
Detailed documentation on how to use <code>DocumentLoaders</code>.</li>
<li><a href="https://python.langchain.com/docs/integrations/document_loaders/" rel="noopener noreferrer" target="_blank">Integrations</a>: 160+
integrations to choose from.</li>
<li><a href="https://python.langchain.com/api_reference/core/document_loaders/langchain_core.document_loaders.base.BaseLoader.html" rel="noopener noreferrer" target="_blank">Interface</a>:
API reference for the base interface.</li>
</ul>
<h3>Splitting documents<a href="https://python.langchain.com/docs/tutorials/rag/#splitting-documents" rel="noopener noreferrer" target="_blank" title="Direct link to Splitting documents">​</a></h3>
<p>Our loaded document is over 42k characters which is too long to fit
into the context window of many models. Even for those models that could
fit the full post in their context window, models can struggle to find
information in very long inputs.</p>
<p>To handle this we’ll split the <code>Document</code> into chunks for embedding and
vector storage. This should help us retrieve only the most relevant parts
of the blog post at run time.</p>
<p>As in the <a href="https://python.langchain.com/docs/tutorials/retrievers/" rel="noopener noreferrer" target="_blank">semantic search tutorial</a>, we use a
<a href="https://python.langchain.com/docs/how_to/recursive_text_splitter/" rel="noopener noreferrer" target="_blank">RecursiveCharacterTextSplitter</a>,
which will recursively split the document using common separators like
new lines until each chunk is the appropriate size. This is the
recommended text splitter for generic text use cases.</p>
<pre><code>from langchain_text_splitters import RecursiveCharacterTextSplitter<br/><br/>text_splitter = RecursiveCharacterTextSplitter(<br/>    chunk_size=1000,  <br/>    chunk_overlap=200,  <br/>    add_start_index=True,  <br/>)<br/>all_splits = text_splitter.split_documents(docs)<br/><br/>print(f"Split blog post into {len(all_splits)} sub-documents.")<br/></code></pre>
<pre><code>Split blog post into 66 sub-documents.<br/></code></pre>
<h4>Go deeper<a href="https://python.langchain.com/docs/tutorials/rag/#go-deeper-1" rel="noopener noreferrer" target="_blank" title="Direct link to Go deeper">​</a></h4>
<p><code>TextSplitter</code>: Object that splits a list of <code>Document</code>s into smaller
chunks. Subclass of <code>DocumentTransformer</code>s.</p>
<p><code>DocumentTransformer</code>: Object that performs a transformation on a list
of <code>Document</code> objects.</p>
<ul>
<li><a href="https://python.langchain.com/docs/how_to/#text-splitters" rel="noopener noreferrer" target="_blank">Docs</a>: Detailed documentation on how to use <code>DocumentTransformers</code></li>
<li><a href="https://python.langchain.com/docs/integrations/document_transformers/" rel="noopener noreferrer" target="_blank">Integrations</a></li>
<li><a href="https://python.langchain.com/api_reference/core/documents/langchain_core.documents.transformers.BaseDocumentTransformer.html" rel="noopener noreferrer" target="_blank">Interface</a>: API reference for the base interface.</li>
</ul>
<h3>Storing documents<a href="https://python.langchain.com/docs/tutorials/rag/#storing-documents" rel="noopener noreferrer" target="_blank" title="Direct link to Storing documents">​</a></h3>
<p>Now we need to index our 66 text chunks so that we can search over them
at runtime. Following the <a href="https://python.langchain.com/docs/tutorials/retrievers/" rel="noopener noreferrer" target="_blank">semantic search tutorial</a>,
our approach is to <a href="https://python.langchain.com/docs/concepts/embedding_models/" rel="noopener noreferrer" target="_blank">embed</a> the contents of each document split and insert these embeddings
into a <a href="https://python.langchain.com/docs/concepts/vectorstores/" rel="noopener noreferrer" target="_blank">vector store</a>. Given an input query, we can then use
vector search to retrieve relevant documents.</p>
<p>We can embed and store all of our document splits in a single command
using the vector store and embeddings model selected at the <a href="https://python.langchain.com/docs/tutorials/rag/#components" rel="noopener noreferrer" target="_blank">start of the tutorial</a>.</p>
<pre><code>document_ids = vector_store.add_documents(documents=all_splits)<br/><br/>print(document_ids[:3])<br/></code></pre>
<pre><code>['07c18af6-ad58-479a-bfb1-d508033f9c64', '9000bf8e-1993-446f-8d4d-f4e507ba4b8f', 'ba3b5d14-bed9-4f5f-88be-44c88aedc2e6']<br/></code></pre>
<h4>Go deeper<a href="https://python.langchain.com/docs/tutorials/rag/#go-deeper-2" rel="noopener noreferrer" target="_blank" title="Direct link to Go deeper">​</a></h4>
<p><code>Embeddings</code>: Wrapper around a text embedding model, used for converting
text to embeddings.</p>
<ul>
<li><a href="https://python.langchain.com/docs/how_to/embed_text/" rel="noopener noreferrer" target="_blank">Docs</a>: Detailed documentation on how to use embeddings.</li>
<li><a href="https://python.langchain.com/docs/integrations/text_embedding/" rel="noopener noreferrer" target="_blank">Integrations</a>: 30+ integrations to choose from.</li>
<li><a href="https://python.langchain.com/api_reference/core/embeddings/langchain_core.embeddings.Embeddings.html" rel="noopener noreferrer" target="_blank">Interface</a>: API reference for the base interface.</li>
</ul>
<p><code>VectorStore</code>: Wrapper around a vector database, used for storing and
querying embeddings.</p>
<ul>
<li><a href="https://python.langchain.com/docs/how_to/vectorstores/" rel="noopener noreferrer" target="_blank">Docs</a>: Detailed documentation on how to use vector stores.</li>
<li><a href="https://python.langchain.com/docs/integrations/vectorstores/" rel="noopener noreferrer" target="_blank">Integrations</a>: 40+ integrations to choose from.</li>
<li><a href="https://python.langchain.com/api_reference/core/vectorstores/langchain_core.vectorstores.base.VectorStore.html" rel="noopener noreferrer" target="_blank">Interface</a>: API reference for the base interface.</li>
</ul>
<p>This completes the <strong>Indexing</strong> portion of the pipeline. At this point
we have a query-able vector store containing the chunked contents of our
blog post. Given a user question, we should ideally be able to return
the snippets of the blog post that answer the question.</p>
<h2>2. Retrieval and Generation<a href="https://python.langchain.com/docs/tutorials/rag/#orchestration" rel="noopener noreferrer" target="_blank" title="Direct link to 2. Retrieval and Generation">​</a></h2>
<p>Now let’s write the actual application logic. We want to create a simple
application that takes a user question, searches for documents relevant
to that question, passes the retrieved documents and initial question to
a model, and returns an answer.</p>
<p>For generation, we will use the chat model selected at the <a href="https://python.langchain.com/docs/tutorials/rag/#components" rel="noopener noreferrer" target="_blank">start of the tutorial</a>.</p>
<p>We’ll use a prompt for RAG that is checked into the LangChain prompt hub
(<a href="https://smith.langchain.com/hub/rlm/rag-prompt" rel="noopener noreferrer" target="_blank">here</a>).</p>
<pre><code>from langchain import hub<br/><br/><br/><br/>prompt = hub.pull("rlm/rag-prompt")<br/><br/>example_messages = prompt.invoke(<br/>    {"context": "(context goes here)", "question": "(question goes here)"}<br/>).to_messages()<br/><br/>assert len(example_messages) == 1<br/>print(example_messages[0].content)<br/></code></pre>
<pre><code>You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. Use three sentences maximum and keep the answer concise.<br/>Question: (question goes here) <br/>Context: (context goes here) <br/>Answer:<br/></code></pre>
<p>We'll use <a href="https://langchain-ai.github.io/langgraph/" rel="noopener noreferrer" target="_blank">LangGraph</a> to tie together the retrieval and generation steps into a single application. This will bring a number of benefits:</p>
<ul>
<li>We can define our application logic once and automatically support multiple invocation modes, including streaming, async, and batched calls.</li>
<li>We get streamlined deployments via <a href="https://langchain-ai.github.io/langgraph/concepts/langgraph_platform/" rel="noopener noreferrer" target="_blank">LangGraph Platform</a>.</li>
<li>LangSmith will automatically trace the steps of our application together.</li>
<li>We can easily add key features to our application, including <a href="https://langchain-ai.github.io/langgraph/concepts/persistence/" rel="noopener noreferrer" target="_blank">persistence</a> and <a href="https://langchain-ai.github.io/langgraph/concepts/human_in_the_loop/" rel="noopener noreferrer" target="_blank">human-in-the-loop approval</a>, with minimal code changes.</li>
</ul>
<p>To use LangGraph, we need to define three things:</p>
<ol>
<li>The state of our application;</li>
<li>The nodes of our application (i.e., application steps);</li>
<li>The "control flow" of our application (e.g., the ordering of the steps).</li>
</ol>
<h4>State:<a href="https://python.langchain.com/docs/tutorials/rag/#state" rel="noopener noreferrer" target="_blank" title="Direct link to State:">​</a></h4>
<p>The <a href="https://langchain-ai.github.io/langgraph/concepts/low_level/#state" rel="noopener noreferrer" target="_blank">state</a> of our application controls what data is input to the application, transferred between steps, and output by the application. It is typically a <code>TypedDict</code>, but can also be a <a href="https://langchain-ai.github.io/langgraph/how-tos/state-model/" rel="noopener noreferrer" target="_blank">Pydantic BaseModel</a>.</p>
<p>For a simple RAG application, we can just keep track of the input question, retrieved context, and generated answer:</p>
<pre><code>from langchain_core.documents import Document<br/>from typing_extensions import List, TypedDict<br/><br/><br/>class State(TypedDict):<br/>    question: str<br/>    context: List[Document]<br/>    answer: str<br/></code></pre>
<h4>Nodes (application steps)<a href="https://python.langchain.com/docs/tutorials/rag/#nodes-application-steps" rel="noopener noreferrer" target="_blank" title="Direct link to Nodes (application steps)">​</a></h4>
<p>Let's start with a simple sequence of two steps: retrieval and generation.</p>
<pre><code>def retrieve(state: State):<br/>    retrieved_docs = vector_store.similarity_search(state["question"])<br/>    return {"context": retrieved_docs}<br/><br/><br/>def generate(state: State):<br/>    docs_content = "\n\n".join(doc.page_content for doc in state["context"])<br/>    messages = prompt.invoke({"question": state["question"], "context": docs_content})<br/>    response = llm.invoke(messages)<br/>    return {"answer": response.content}<br/></code></pre>
<p>Our retrieval step simply runs a similarity search using the input question, and the generation step formats the retrieved context and original question into a prompt for the chat model.</p>
<h4>Control flow<a href="https://python.langchain.com/docs/tutorials/rag/#control-flow" rel="noopener noreferrer" target="_blank" title="Direct link to Control flow">​</a></h4>
<p>Finally, we compile our application into a single <code>graph</code> object. In this case, we are just connecting the retrieval and generation steps into a single sequence.</p>
<pre><code>from langgraph.graph import START, StateGraph<br/><br/>graph_builder = StateGraph(State).add_sequence([retrieve, generate])<br/>graph_builder.add_edge(START, "retrieve")<br/>graph = graph_builder.compile()<br/></code></pre>
<p>LangGraph also comes with built-in utilities for visualizing the control flow of your application:</p>
<pre><code>from IPython.display import Image, display<br/><br/>display(Image(graph.get_graph().draw_mermaid_png()))<br/></code></pre>
Do I need to use LangGraph?<p>LangGraph is not required to build a RAG application. Indeed, we can implement the same application logic through invocations of the individual components:</p><pre><code>question = "..."<br/><br/>retrieved_docs = vector_store.similarity_search(question)<br/>docs_content = "\n\n".join(doc.page_content for doc in retrieved_docs)<br/>prompt = prompt.invoke({"question": question, "context": docs_content})<br/>answer = llm.invoke(prompt)<br/></code></pre><p>The benefits of LangGraph include:</p><ul>
<li>Support for multiple invocation modes: this logic would need to be rewritten if we wanted to stream output tokens, or stream the results of individual steps;</li>
<li>Automatic support for tracing via <a href="https://docs.smith.langchain.com/" rel="noopener noreferrer" target="_blank">LangSmith</a> and deployments via <a href="https://langchain-ai.github.io/langgraph/concepts/langgraph_platform/" rel="noopener noreferrer" target="_blank">LangGraph Platform</a>;</li>
<li>Support for persistence, human-in-the-loop, and other features.</li>
</ul><p>Many use-cases demand RAG in a conversational experience, such that a user can receive context-informed answers via a stateful conversation. As we will see in <a href="https://python.langchain.com/docs/tutorials/qa_chat_history/" rel="noopener noreferrer" target="_blank">Part 2</a> of the tutorial, LangGraph's management and persistence of state simplifies these applications enormously.</p>
<h4>Usage<a href="https://python.langchain.com/docs/tutorials/rag/#usage" rel="noopener noreferrer" target="_blank" title="Direct link to Usage">​</a></h4>
<p>Let's test our application! LangGraph supports multiple invocation modes, including sync, async, and streaming.</p>
<p>Invoke:</p>
<pre><code>result = graph.invoke({"question": "What is Task Decomposition?"})<br/><br/>print(f"Context: {result['context']}\n\n")<br/>print(f"Answer: {result['answer']}")<br/></code></pre>
<pre><code>Context: [Document(id='a42dc78b-8f76-472a-9e25-180508af74f3', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'start_index': 1585}, page_content='Fig. 1. Overview of a LLM-powered autonomous agent system.\nComponent One: Planning#\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\nTask Decomposition#\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.'), Document(id='c0e45887-d0b0-483d-821a-bb5d8316d51d', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'start_index': 2192}, page_content='Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\nTask decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.'), Document(id='4cc7f318-35f5-440f-a4a4-145b5f0b918d', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'start_index': 29630}, page_content='Resources:\n1. Internet access for searches and information gathering.\n2. Long Term memory management.\n3. GPT-3.5 powered Agents for delegation of simple tasks.\n4. File output.\n\nPerformance Evaluation:\n1. Continuously review and analyze your actions to ensure you are performing to the best of your abilities.\n2. Constructively self-criticize your big-picture behavior constantly.\n3. Reflect on past decisions and strategies to refine your approach.\n4. Every command has a cost, so be smart and efficient. Aim to complete tasks in the least number of steps.'), Document(id='f621ade4-9b0d-471f-a522-44eb5feeba0c', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'start_index': 19373}, page_content="(3) Task execution: Expert models execute on the specific tasks and log results.\nInstruction:\n\nWith the input and the inference results, the AI assistant needs to describe the process and results. The previous stages can be formed as - User Input: {{ User Input }}, Task Planning: {{ Tasks }}, Model Selection: {{ Model Assignment }}, Task Execution: {{ Predictions }}. You must first answer the user's request in a straightforward manner. Then describe the task process and show your analysis and model inference results to the user in the first person. If inference results contain a file path, must tell the user the complete file path.")]<br/><br/><br/>Answer: Task decomposition is a technique used to break down complex tasks into smaller, manageable steps, allowing for more efficient problem-solving. This can be achieved through methods like chain of thought prompting or the tree of thoughts approach, which explores multiple reasoning possibilities at each step. It can be initiated through simple prompts, task-specific instructions, or human inputs.<br/></code></pre>
<p>Stream steps:</p>
<pre><code>for step in graph.stream(<br/>    {"question": "What is Task Decomposition?"}, stream_mode="updates"<br/>):<br/>    print(f"{step}\n\n----------------\n")<br/></code></pre>
<pre><code>{'retrieve': {'context': [Document(id='a42dc78b-8f76-472a-9e25-180508af74f3', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'start_index': 1585}, page_content='Fig. 1. Overview of a LLM-powered autonomous agent system.\nComponent One: Planning#\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\nTask Decomposition#\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.'), Document(id='c0e45887-d0b0-483d-821a-bb5d8316d51d', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'start_index': 2192}, page_content='Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\nTask decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.'), Document(id='4cc7f318-35f5-440f-a4a4-145b5f0b918d', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'start_index': 29630}, page_content='Resources:\n1. Internet access for searches and information gathering.\n2. Long Term memory management.\n3. GPT-3.5 powered Agents for delegation of simple tasks.\n4. File output.\n\nPerformance Evaluation:\n1. Continuously review and analyze your actions to ensure you are performing to the best of your abilities.\n2. Constructively self-criticize your big-picture behavior constantly.\n3. Reflect on past decisions and strategies to refine your approach.\n4. Every command has a cost, so be smart and efficient. Aim to complete tasks in the least number of steps.'), Document(id='f621ade4-9b0d-471f-a522-44eb5feeba0c', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'start_index': 19373}, page_content="(3) Task execution: Expert models execute on the specific tasks and log results.\nInstruction:\n\nWith the input and the inference results, the AI assistant needs to describe the process and results. The previous stages can be formed as - User Input: {{ User Input }}, Task Planning: {{ Tasks }}, Model Selection: {{ Model Assignment }}, Task Execution: {{ Predictions }}. You must first answer the user's request in a straightforward manner. Then describe the task process and show your analysis and model inference results to the user in the first person. If inference results contain a file path, must tell the user the complete file path.")]}}<br/><br/>----------------<br/><br/>{'generate': {'answer': 'Task decomposition is the process of breaking down a complex task into smaller, more manageable steps. This technique, often enhanced by methods like Chain of Thought (CoT) or Tree of Thoughts, allows models to reason through tasks systematically and improves performance by clarifying the thought process. It can be achieved through simple prompts, task-specific instructions, or human inputs.'}}<br/><br/>----------------<br/></code></pre>
<p>Stream <a href="https://python.langchain.com/docs/concepts/tokens/" rel="noopener noreferrer" target="_blank">tokens</a>:</p>
<pre><code>for message, metadata in graph.stream(<br/>    {"question": "What is Task Decomposition?"}, stream_mode="messages"<br/>):<br/>    print(message.content, end="|")<br/></code></pre>
<pre><code>|Task| decomposition| is| the| process| of| breaking| down| complex| tasks| into| smaller|,| more| manageable| steps|.| It| can| be| achieved| through| techniques| like| Chain| of| Thought| (|Co|T|)| prompting|,| which| encourages| the| model| to| think| step| by| step|,| or| through| more| structured| methods| like| the| Tree| of| Thoughts|.| This| approach| not| only| simplifies| task| execution| but| also| provides| insights| into| the| model|'s| reasoning| process|.||<br/></code></pre>
<p>For async invocations, use:</p><pre><code>result = await graph.ainvoke(...)<br/></code></pre><p>and</p><pre><code>async for step in graph.astream(...):<br/></code></pre>
<h4>Returning sources<a href="https://python.langchain.com/docs/tutorials/rag/#returning-sources" rel="noopener noreferrer" target="_blank" title="Direct link to Returning sources">​</a></h4>
<p>Note that by storing the retrieved context in the state of the graph, we recover sources for the model's generated answer in the <code>"context"</code> field of the state. See <a href="https://python.langchain.com/docs/how_to/qa_sources/" rel="noopener noreferrer" target="_blank">this guide</a> on returning sources for more detail.</p>
<h4>Go deeper<a href="https://python.langchain.com/docs/tutorials/rag/#go-deeper-3" rel="noopener noreferrer" target="_blank" title="Direct link to Go deeper">​</a></h4>
<p><a href="https://python.langchain.com/docs/concepts/chat_models/" rel="noopener noreferrer" target="_blank">Chat models</a> take in a sequence of messages and return a message.</p>
<p><strong>Customizing the prompt</strong></p>
<p>As shown above, we can load prompts (e.g., <a href="https://smith.langchain.com/hub/rlm/rag-prompt" rel="noopener noreferrer" target="_blank">this RAG
prompt</a>) from the prompt
hub. The prompt can also be easily customized. For example:</p>
<pre><code>from langchain_core.prompts import PromptTemplate<br/><br/>template = """Use the following pieces of context to answer the question at the end.<br/>If you don't know the answer, just say that you don't know, don't try to make up an answer.<br/>Use three sentences maximum and keep the answer as concise as possible.<br/>Always say "thanks for asking!" at the end of the answer.<br/><br/>{context}<br/><br/>Question: {question}<br/><br/>Helpful Answer:"""<br/>custom_rag_prompt = PromptTemplate.from_template(template)<br/></code></pre>
<h2>Query analysis<a href="https://python.langchain.com/docs/tutorials/rag/#query-analysis" rel="noopener noreferrer" target="_blank" title="Direct link to Query analysis">​</a></h2>
<p>So far, we are executing the retrieval using the raw input query. However, there are some advantages to allowing a model to generate the query for retrieval purposes. For example:</p>
<ul>
<li>In addition to semantic search, we can build in structured filters (e.g., "Find documents since the year 2020.");</li>
<li>The model can rewrite user queries, which may be multifaceted or include irrelevant language, into more effective search queries.</li>
</ul>
<p><a href="https://python.langchain.com/docs/concepts/retrieval/#query-analysis" rel="noopener noreferrer" target="_blank">Query analysis</a> employs models to transform or construct optimized search queries from raw user input. We can easily incorporate a query analysis step into our application. For illustrative purposes, let's add some metadata to the documents in our vector store. We will add some (contrived) sections to the document which we can filter on later.</p>
<pre><code>total_documents = len(all_splits)<br/>third = total_documents // 3<br/><br/>for i, document in enumerate(all_splits):<br/>    if i &lt; third:<br/>        document.metadata["section"] = "beginning"<br/>    elif i &lt; 2 * third:<br/>        document.metadata["section"] = "middle"<br/>    else:<br/>        document.metadata["section"] = "end"<br/><br/><br/>all_splits[0].metadata<br/></code></pre>
<pre><code>{'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/',<br/> 'start_index': 8,<br/> 'section': 'beginning'}<br/></code></pre>
<p>We will need to update the documents in our vector store. We will use a simple <a href="https://python.langchain.com/api_reference/core/vectorstores/langchain_core.vectorstores.in_memory.InMemoryVectorStore.html" rel="noopener noreferrer" target="_blank">InMemoryVectorStore</a> for this, as we will use some of its specific features (i.e., metadata filtering). Refer to the vector store <a href="https://python.langchain.com/docs/integrations/vectorstores/" rel="noopener noreferrer" target="_blank">integration documentation</a> for relevant features of your chosen vector store.</p>
<pre><code>from langchain_core.vectorstores import InMemoryVectorStore<br/><br/>vector_store = InMemoryVectorStore(embeddings)<br/>_ = vector_store.add_documents(all_splits)<br/></code></pre>
<p>Let's next define a schema for our search query. We will use <a href="https://python.langchain.com/docs/concepts/structured_outputs/" rel="noopener noreferrer" target="_blank">structured output</a> for this purpose. Here we define a query as containing a string query and a document section (either "beginning", "middle", or "end"), but this can be defined however you like.</p>
<pre><code>from typing import Literal<br/><br/>from typing_extensions import Annotated<br/><br/><br/>class Search(TypedDict):<br/>    """Search query."""<br/><br/>    query: Annotated[str, ..., "Search query to run."]<br/>    section: Annotated[<br/>        Literal["beginning", "middle", "end"],<br/>        ...,<br/>        "Section to query.",<br/>    ]<br/></code></pre>
<p>Finally, we add a step to our LangGraph application to generate a query from the user's raw input:</p>
<pre><code>class State(TypedDict):<br/>    question: str<br/>    query: Search<br/>    context: List[Document]<br/>    answer: str<br/><br/><br/>def analyze_query(state: State):<br/>    structured_llm = llm.with_structured_output(Search)<br/>    query = structured_llm.invoke(state["question"])<br/>    return {"query": query}<br/><br/><br/>def retrieve(state: State):<br/>    query = state["query"]<br/>    retrieved_docs = vector_store.similarity_search(<br/>        query["query"],<br/>        filter=lambda doc: doc.metadata.get("section") == query["section"],<br/>    )<br/>    return {"context": retrieved_docs}<br/><br/><br/>def generate(state: State):<br/>    docs_content = "\n\n".join(doc.page_content for doc in state["context"])<br/>    messages = prompt.invoke({"question": state["question"], "context": docs_content})<br/>    response = llm.invoke(messages)<br/>    return {"answer": response.content}<br/><br/><br/>graph_builder = StateGraph(State).add_sequence([analyze_query, retrieve, generate])<br/>graph_builder.add_edge(START, "analyze_query")<br/>graph = graph_builder.compile()<br/></code></pre>
Full Code:<pre><code>from typing import Literal<br/><br/>import bs4<br/>from langchain import hub<br/>from langchain_community.document_loaders import WebBaseLoader<br/>from langchain_core.documents import Document<br/>from langchain_core.vectorstores import InMemoryVectorStore<br/>from langchain_text_splitters import RecursiveCharacterTextSplitter<br/>from langgraph.graph import START, StateGraph<br/>from typing_extensions import Annotated, List, TypedDict<br/><br/><br/>loader = WebBaseLoader(<br/>    web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),<br/>    bs_kwargs=dict(<br/>        parse_only=bs4.SoupStrainer(<br/>            class_=("post-content", "post-title", "post-header")<br/>        )<br/>    ),<br/>)<br/>docs = loader.load()<br/><br/>text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)<br/>all_splits = text_splitter.split_documents(docs)<br/><br/><br/><br/>total_documents = len(all_splits)<br/>third = total_documents // 3<br/><br/>for i, document in enumerate(all_splits):<br/>    if i &lt; third:<br/>        document.metadata["section"] = "beginning"<br/>    elif i &lt; 2 * third:<br/>        document.metadata["section"] = "middle"<br/>    else:<br/>        document.metadata["section"] = "end"<br/><br/><br/><br/>vector_store = InMemoryVectorStore(embeddings)<br/>_ = vector_store.add_documents(all_splits)<br/><br/><br/><br/>class Search(TypedDict):<br/>    """Search query."""<br/><br/>    query: Annotated[str, ..., "Search query to run."]<br/>    section: Annotated[<br/>        Literal["beginning", "middle", "end"],<br/>        ...,<br/>        "Section to query.",<br/>    ]<br/><br/><br/>prompt = hub.pull("rlm/rag-prompt")<br/><br/><br/><br/>class State(TypedDict):<br/>    question: str<br/>    query: Search<br/>    context: List[Document]<br/>    answer: str<br/><br/><br/>def analyze_query(state: State):<br/>    structured_llm = llm.with_structured_output(Search)<br/>    query = structured_llm.invoke(state["question"])<br/>    return {"query": query}<br/><br/><br/>def retrieve(state: State):<br/>    query = state["query"]<br/>    retrieved_docs = vector_store.similarity_search(<br/>        query["query"],<br/>        filter=lambda doc: doc.metadata.get("section") == query["section"],<br/>    )<br/>    return {"context": retrieved_docs}<br/><br/><br/>def generate(state: State):<br/>    docs_content = "\n\n".join(doc.page_content for doc in state["context"])<br/>    messages = prompt.invoke({"question": state["question"], "context": docs_content})<br/>    response = llm.invoke(messages)<br/>    return {"answer": response.content}<br/><br/><br/>graph_builder = StateGraph(State).add_sequence([analyze_query, retrieve, generate])<br/>graph_builder.add_edge(START, "analyze_query")<br/>graph = graph_builder.compile()<br/></code></pre>
<pre><code>display(Image(graph.get_graph().draw_mermaid_png()))<br/></code></pre>
<p>We can test our implementation by specifically asking for context from the end of the post. Note that the model includes different information in its answer.</p>
<pre><code>for step in graph.stream(<br/>    {"question": "What does the end of the post say about Task Decomposition?"},<br/>    stream_mode="updates",<br/>):<br/>    print(f"{step}\n\n----------------\n")<br/></code></pre>
<pre><code>{'analyze_query': {'query': {'query': 'Task Decomposition', 'section': 'end'}}}<br/><br/>----------------<br/><br/>{'retrieve': {'context': [Document(id='d6cef137-e1e8-4ddc-91dc-b62bd33c6020', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'start_index': 39221, 'section': 'end'}, page_content='Finite context length: The restricted context capacity limits the inclusion of historical information, detailed instructions, API call context, and responses. The design of the system has to work with this limited communication bandwidth, while mechanisms like self-reflection to learn from past mistakes would benefit a lot from long or infinite context windows. Although vector stores and retrieval can provide access to a larger knowledge pool, their representation power is not as powerful as full attention.\n\n\nChallenges in long-term planning and task decomposition: Planning over a lengthy history and effectively exploring the solution space remain challenging. LLMs struggle to adjust plans when faced with unexpected errors, making them less robust compared to humans who learn from trial and error.'), Document(id='d1834ae1-eb6a-43d7-a023-08dfa5028799', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'start_index': 39086, 'section': 'end'}, page_content='}\n]\nChallenges#\nAfter going through key ideas and demos of building LLM-centered agents, I start to see a couple common limitations:'), Document(id='ca7f06e4-2c2e-4788-9a81-2418d82213d9', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'start_index': 32942, 'section': 'end'}, page_content='}\n]\nThen after these clarification, the agent moved into the code writing mode with a different system message.\nSystem message:'), Document(id='1fcc2736-30f4-4ef6-90f2-c64af92118cb', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'start_index': 35127, 'section': 'end'}, page_content='"content": "You will get instructions for code to write.\\nYou will write a very long answer. Make sure that every detail of the architecture is, in the end, implemented as code.\\nMake sure that every detail of the architecture is, in the end, implemented as code.\\n\\nThink step by step and reason yourself to the right decisions to make sure we get it right.\\nYou will first lay out the names of the core classes, functions, methods that will be necessary, as well as a quick comment on their purpose.\\n\\nThen you will output the content of each file including ALL code.\\nEach file must strictly follow a markdown code block format, where the following tokens must be replaced such that\\nFILENAME is the lowercase file name including the file extension,\\nLANG is the markup code block language for the code\'s language, and CODE is the code:\\n\\nFILENAME\\n\`\`\`LANG\\nCODE\\n\`\`\`\\n\\nYou will start with the \\"entrypoint\\" file, then go to the ones that are imported by that file, and so on.\\nPlease')]}}<br/><br/>----------------<br/><br/>{'generate': {'answer': 'The end of the post highlights that task decomposition faces challenges in long-term planning and adapting to unexpected errors. LLMs struggle with adjusting their plans, making them less robust compared to humans who learn from trial and error. This indicates a limitation in effectively exploring the solution space and handling complex tasks.'}}<br/><br/>----------------<br/></code></pre>
<p>In both the streamed steps and the <a href="https://smith.langchain.com/public/bdbaae61-130c-4338-8b59-9315dfee22a0/r" rel="noopener noreferrer" target="_blank">LangSmith trace</a>, we can now observe the structured query that was fed into the retrieval step.</p>
<p>Query Analysis is a rich problem with a wide range of approaches. Refer to the <a href="https://python.langchain.com/docs/how_to/#query-analysis" rel="noopener noreferrer" target="_blank">how-to guides</a> for more examples.</p>
<h2>Next steps<a href="https://python.langchain.com/docs/tutorials/rag/#next-steps" rel="noopener noreferrer" target="_blank" title="Direct link to Next steps">​</a></h2>
<p>We've covered the steps to build a basic Q&amp;A app over data:</p>
<p>In <a href="https://python.langchain.com/docs/tutorials/qa_chat_history/" rel="noopener noreferrer" target="_blank">Part 2</a> of the tutorial, we will extend the implementation here to accommodate conversation-style interactions and multi-step retrieval processes.</p>
<p>Further reading:</p>

content_text:
Build a Retrieval Augmented Generation (RAG) App: Part 1

One of the most powerful applications enabled by LLMs is sophisticated question-answering (Q&A) chatbots. These are applications that can answer questions about specific source information. These applications use a technique known as Retrieval Augmented Generation, or

RAG

.

This is a multi-part tutorial:

Part 1

(this guide) introduces RAG and walks through a minimal implementation.

Part 2

extends the implementation to accommodate conversation-style interactions and multi-step retrieval processes.

This tutorial will show how to build a simple Q&A application

over a text data source. Along the way we’ll go over a typical Q&A

architecture and highlight additional resources for more advanced Q&A techniques. We’ll also see

how LangSmith can help us trace and understand our application.

LangSmith will become increasingly helpful as our application grows in

complexity.

If you're already familiar with basic retrieval, you might also be interested in

this

high-level overview of different retrieval techniques

.

Note

: Here we focus on Q&A for unstructured data. If you are interested for RAG over structured data, check out our tutorial on doing

question/answering over SQL data

.

Overview

​

A typical RAG application has two main components:

Indexing

: a pipeline for ingesting data from a source and indexing it.

This usually happens offline.

Retrieval and generation

: the actual RAG chain, which takes the user query at run time and retrieves the relevant data from the index, then passes that to the model.

Note: the indexing portion of this tutorial will largely follow the

semantic search tutorial

.

The most common full sequence from raw data to answer looks like:

Indexing

​

Load

: First we need to load our data. This is done with

Document Loaders

.

Split

:

Text splitters

break large

Documents

into smaller chunks. This is useful both for indexing data and passing it into a model, as large chunks are harder to search over and won't fit in a model's finite context window.

Store

: We need somewhere to store and index our splits, so that they can be searched over later. This is often done using a

VectorStore

and

Embeddings

model.

Retrieval and generation

​

Retrieve

: Given a user input, relevant splits are retrieved from storage using a

Retriever

.

Generate

: A

ChatModel

/

LLM

produces an answer using a prompt that includes both the question with the retrieved data

Once we've indexed our data, we will use

LangGraph

as our orchestration framework to implement the retrieval and generation steps.

Setup

​

Jupyter Notebook

​

This and other tutorials are perhaps most conveniently run in a

Jupyter notebooks

. Going through guides in an interactive environment is a great way to better understand them. See

here

for instructions on how to install.

Installation

​

This tutorial requires these langchain dependencies:

%pip install --quiet --upgrade langchain-text-splitters langchain-community langgraph

conda install langchain-text-splitters langchain-community langgraph -c conda-forge

For more details, see our

Installation guide

.

LangSmith

​

Many of the applications you build with LangChain will contain multiple steps with multiple invocations of LLM calls.

As these applications get more complex, it becomes crucial to be able to inspect what exactly is going on inside your chain or agent.

The best way to do this is with

LangSmith

.

After you sign up at the link above, make sure to set your environment variables to start logging traces:

export LANGSMITH_TRACING="true"

export LANGSMITH_API_KEY="..."

Or, if in a notebook, you can set them with:

import getpass

import os

os.environ["LANGSMITH_TRACING"] = "true"

os.environ["LANGSMITH_API_KEY"] = getpass.getpass()

Components

​

We will need to select three components from LangChain's suite of integrations.

pip install -qU "langchain[google-genai]"

import getpass

import os

if not os.environ.get("GOOGLE_API_KEY"):

os.environ["GOOGLE_API_KEY"] = getpass.getpass("Enter API key for Google Gemini: ")

from langchain.chat_models import init_chat_model

llm = init_chat_model("gemini-2.5-flash", model_provider="google_genai")

pip install -qU langchain-openai

import getpass

import os

if not os.environ.get("OPENAI_API_KEY"):

os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter API key for OpenAI: ")

from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(model="text-embedding-3-large")

pip install -qU langchain-core

from langchain_core.vectorstores import InMemoryVectorStore

vector_store = InMemoryVectorStore(embeddings)

Preview

​

In this guide we’ll build an app that answers questions about the website's content. The specific website we will use is the

LLM Powered Autonomous

Agents

blog post

by Lilian Weng, which allows us to ask questions about the contents of

the post.

We can create a simple indexing pipeline and RAG chain to do this in ~50

lines of code.

import bs4

from langchain import hub

from langchain_community.document_loaders import WebBaseLoader

from langchain_core.documents import Document

from langchain_text_splitters import RecursiveCharacterTextSplitter

from langgraph.graph import START, StateGraph

from typing_extensions import List, TypedDict

loader = WebBaseLoader(

web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),

bs_kwargs=dict(

parse_only=bs4.SoupStrainer(

class_=("post-content", "post-title", "post-header")

)

),

)

docs = loader.load()

text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)

all_splits = text_splitter.split_documents(docs)

_ = vector_store.add_documents(documents=all_splits)

prompt = hub.pull("rlm/rag-prompt")

class State(TypedDict):

question: str

context: List[Document]

answer: str

def retrieve(state: State):

retrieved_docs = vector_store.similarity_search(state["question"])

return {"context": retrieved_docs}

def generate(state: State):

docs_content = "\n\n".join(doc.page_content for doc in state["context"])

messages = prompt.invoke({"question": state["question"], "context": docs_content})

response = llm.invoke(messages)

return {"answer": response.content}

graph_builder = StateGraph(State).add_sequence([retrieve, generate])

graph_builder.add_edge(START, "retrieve")

graph = graph_builder.compile()

response = graph.invoke({"question": "What is Task Decomposition?"})

print(response["answer"])

Task Decomposition is the process of breaking down a complicated task into smaller, manageable steps to facilitate easier execution and understanding. Techniques like Chain of Thought (CoT) and Tree of Thoughts (ToT) guide models to think step-by-step, allowing them to explore multiple reasoning possibilities. This method enhances performance on complex tasks and provides insight into the model's thinking process.

Check out the

LangSmith

trace

.

Detailed walkthrough

​

Let’s go through the above code step-by-step to really understand what’s

going on.

1. Indexing

​

Loading documents

​

We need to first load the blog post contents. We can use

DocumentLoaders

for this, which are objects that load in data from a source and return a

list of

Document

objects.

In this case we’ll use the

WebBaseLoader

,

which uses

urllib

to load HTML from web URLs and

BeautifulSoup

to

parse it to text. We can customize the HTML -> text parsing by passing

in parameters into the

BeautifulSoup

parser via

bs_kwargs

(see

BeautifulSoup

docs

).

In this case only HTML tags with class “post-content”, “post-title”, or

“post-header” are relevant, so we’ll remove all others.

import bs4

from langchain_community.document_loaders import WebBaseLoader

bs4_strainer = bs4.SoupStrainer(class_=("post-title", "post-header", "post-content"))

loader = WebBaseLoader(

web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),

bs_kwargs={"parse_only": bs4_strainer},

)

docs = loader.load()

assert len(docs) == 1

print(f"Total characters: {len(docs[0].page_content)}")

print(docs[0].page_content[:500])

LLM Powered Autonomous Agents

Date: June 23, 2023  |  Estimated Reading Time: 31 min  |  Author: Lilian Weng

Building agents with LLM (large language model) as its core controller is a cool concept. Several proof-of-concepts demos, such as AutoGPT, GPT-Engineer and BabyAGI, serve as inspiring examples. The potentiality of LLM extends beyond generating well-written copies, stories, essays and programs; it can be framed as a powerful general problem solver.

Agent System Overview#

In

Go deeper

​

DocumentLoader

: Object that loads data from a source as list of

Documents

.

Docs

:

Detailed documentation on how to use

DocumentLoaders

.

Integrations

: 160+

integrations to choose from.

Interface

:

API reference for the base interface.

Splitting documents

​

Our loaded document is over 42k characters which is too long to fit

into the context window of many models. Even for those models that could

fit the full post in their context window, models can struggle to find

information in very long inputs.

To handle this we’ll split the

Document

into chunks for embedding and

vector storage. This should help us retrieve only the most relevant parts

of the blog post at run time.

As in the

semantic search tutorial

, we use a

RecursiveCharacterTextSplitter

,

which will recursively split the document using common separators like

new lines until each chunk is the appropriate size. This is the

recommended text splitter for generic text use cases.

from langchain_text_splitters import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(

chunk_size=1000,

chunk_overlap=200,

add_start_index=True,

)

all_splits = text_splitter.split_documents(docs)

print(f"Split blog post into {len(all_splits)} sub-documents.")

Split blog post into 66 sub-documents.

Go deeper

​

TextSplitter

: Object that splits a list of

Document

s into smaller

chunks. Subclass of

DocumentTransformer

s.

DocumentTransformer

: Object that performs a transformation on a list

of

Document

objects.

Docs

: Detailed documentation on how to use

DocumentTransformers

Integrations

Interface

: API reference for the base interface.

Storing documents

​

Now we need to index our 66 text chunks so that we can search over them

at runtime. Following the

semantic search tutorial

,

our approach is to

embed

the contents of each document split and insert these embeddings

into a

vector store

. Given an input query, we can then use

vector search to retrieve relevant documents.

We can embed and store all of our document splits in a single command

using the vector store and embeddings model selected at the

start of the tutorial

.

document_ids = vector_store.add_documents(documents=all_splits)

print(document_ids[:3])

['07c18af6-ad58-479a-bfb1-d508033f9c64', '9000bf8e-1993-446f-8d4d-f4e507ba4b8f', 'ba3b5d14-bed9-4f5f-88be-44c88aedc2e6']

Go deeper

​

Embeddings

: Wrapper around a text embedding model, used for converting

text to embeddings.

Docs

: Detailed documentation on how to use embeddings.

Integrations

: 30+ integrations to choose from.

Interface

: API reference for the base interface.

VectorStore

: Wrapper around a vector database, used for storing and

querying embeddings.

Docs

: Detailed documentation on how to use vector stores.

Integrations

: 40+ integrations to choose from.

Interface

: API reference for the base interface.

This completes the

Indexing

portion of the pipeline. At this point

we have a query-able vector store containing the chunked contents of our

blog post. Given a user question, we should ideally be able to return

the snippets of the blog post that answer the question.

2. Retrieval and Generation

​

Now let’s write the actual application logic. We want to create a simple

application that takes a user question, searches for documents relevant

to that question, passes the retrieved documents and initial question to

a model, and returns an answer.

For generation, we will use the chat model selected at the

start of the tutorial

.

We’ll use a prompt for RAG that is checked into the LangChain prompt hub

(

here

).

from langchain import hub

prompt = hub.pull("rlm/rag-prompt")

example_messages = prompt.invoke(

{"context": "(context goes here)", "question": "(question goes here)"}

).to_messages()

assert len(example_messages) == 1

print(example_messages[0].content)

You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. Use three sentences maximum and keep the answer concise.

Question: (question goes here)

Context: (context goes here)

Answer:

We'll use

LangGraph

to tie together the retrieval and generation steps into a single application. This will bring a number of benefits:

We can define our application logic once and automatically support multiple invocation modes, including streaming, async, and batched calls.

We get streamlined deployments via

LangGraph Platform

.

LangSmith will automatically trace the steps of our application together.

We can easily add key features to our application, including

persistence

and

human-in-the-loop approval

, with minimal code changes.

To use LangGraph, we need to define three things:

The state of our application;

The nodes of our application (i.e., application steps);

The "control flow" of our application (e.g., the ordering of the steps).

State:

​

The

state

of our application controls what data is input to the application, transferred between steps, and output by the application. It is typically a

TypedDict

, but can also be a

Pydantic BaseModel

.

For a simple RAG application, we can just keep track of the input question, retrieved context, and generated answer:

from langchain_core.documents import Document

from typing_extensions import List, TypedDict

class State(TypedDict):

question: str

context: List[Document]

answer: str

Nodes (application steps)

​

Let's start with a simple sequence of two steps: retrieval and generation.

def retrieve(state: State):

retrieved_docs = vector_store.similarity_search(state["question"])

return {"context": retrieved_docs}

def generate(state: State):

docs_content = "\n\n".join(doc.page_content for doc in state["context"])

messages = prompt.invoke({"question": state["question"], "context": docs_content})

response = llm.invoke(messages)

return {"answer": response.content}

Our retrieval step simply runs a similarity search using the input question, and the generation step formats the retrieved context and original question into a prompt for the chat model.

Control flow

​

Finally, we compile our application into a single

graph

object. In this case, we are just connecting the retrieval and generation steps into a single sequence.

from langgraph.graph import START, StateGraph

graph_builder = StateGraph(State).add_sequence([retrieve, generate])

graph_builder.add_edge(START, "retrieve")

graph = graph_builder.compile()

LangGraph also comes with built-in utilities for visualizing the control flow of your application:

from IPython.display import Image, display

display(Image(graph.get_graph().draw_mermaid_png()))

Do I need to use LangGraph?

LangGraph is not required to build a RAG application. Indeed, we can implement the same application logic through invocations of the individual components:

question = "..."

retrieved_docs = vector_store.similarity_search(question)

docs_content = "\n\n".join(doc.page_content for doc in retrieved_docs)

prompt = prompt.invoke({"question": question, "context": docs_content})

answer = llm.invoke(prompt)

The benefits of LangGraph include:

Support for multiple invocation modes: this logic would need to be rewritten if we wanted to stream output tokens, or stream the results of individual steps;

Automatic support for tracing via

LangSmith

and deployments via

LangGraph Platform

;

Support for persistence, human-in-the-loop, and other features.

Many use-cases demand RAG in a conversational experience, such that a user can receive context-informed answers via a stateful conversation. As we will see in

Part 2

of the tutorial, LangGraph's management and persistence of state simplifies these applications enormously.

Usage

​

Let's test our application! LangGraph supports multiple invocation modes, including sync, async, and streaming.

Invoke:

result = graph.invoke({"question": "What is Task Decomposition?"})

print(f"Context: {result['context']}\n\n")

print(f"Answer: {result['answer']}")

Context: [Document(id='a42dc78b-8f76-472a-9e25-180508af74f3', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'start_index': 1585}, page_content='Fig. 1. Overview of a LLM-powered autonomous agent system.\nComponent One: Planning#\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\nTask Decomposition#\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.'), Document(id='c0e45887-d0b0-483d-821a-bb5d8316d51d', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'start_index': 2192}, page_content='Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\nTask decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.'), Document(id='4cc7f318-35f5-440f-a4a4-145b5f0b918d', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'start_index': 29630}, page_content='Resources:\n1. Internet access for searches and information gathering.\n2. Long Term memory management.\n3. GPT-3.5 powered Agents for delegation of simple tasks.\n4. File output.\n\nPerformance Evaluation:\n1. Continuously review and analyze your actions to ensure you are performing to the best of your abilities.\n2. Constructively self-criticize your big-picture behavior constantly.\n3. Reflect on past decisions and strategies to refine your approach.\n4. Every command has a cost, so be smart and efficient. Aim to complete tasks in the least number of steps.'), Document(id='f621ade4-9b0d-471f-a522-44eb5feeba0c', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'start_index': 19373}, page_content="(3) Task execution: Expert models execute on the specific tasks and log results.\nInstruction:\n\nWith the input and the inference results, the AI assistant needs to describe the process and results. The previous stages can be formed as - User Input: {{ User Input }}, Task Planning: {{ Tasks }}, Model Selection: {{ Model Assignment }}, Task Execution: {{ Predictions }}. You must first answer the user's request in a straightforward manner. Then describe the task process and show your analysis and model inference results to the user in the first person. If inference results contain a file path, must tell the user the complete file path.")]

Answer: Task decomposition is a technique used to break down complex tasks into smaller, manageable steps, allowing for more efficient problem-solving. This can be achieved through methods like chain of thought prompting or the tree of thoughts approach, which explores multiple reasoning possibilities at each step. It can be initiated through simple prompts, task-specific instructions, or human inputs.

Stream steps:

for step in graph.stream(

{"question": "What is Task Decomposition?"}, stream_mode="updates"

):

print(f"{step}\n\n----------------\n")

{'retrieve': {'context': [Document(id='a42dc78b-8f76-472a-9e25-180508af74f3', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'start_index': 1585}, page_content='Fig. 1. Overview of a LLM-powered autonomous agent system.\nComponent One: Planning#\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\nTask Decomposition#\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.'), Document(id='c0e45887-d0b0-483d-821a-bb5d8316d51d', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'start_index': 2192}, page_content='Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\nTask decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.'), Document(id='4cc7f318-35f5-440f-a4a4-145b5f0b918d', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'start_index': 29630}, page_content='Resources:\n1. Internet access for searches and information gathering.\n2. Long Term memory management.\n3. GPT-3.5 powered Agents for delegation of simple tasks.\n4. File output.\n\nPerformance Evaluation:\n1. Continuously review and analyze your actions to ensure you are performing to the best of your abilities.\n2. Constructively self-criticize your big-picture behavior constantly.\n3. Reflect on past decisions and strategies to refine your approach.\n4. Every command has a cost, so be smart and efficient. Aim to complete tasks in the least number of steps.'), Document(id='f621ade4-9b0d-471f-a522-44eb5feeba0c', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'start_index': 19373}, page_content="(3) Task execution: Expert models execute on the specific tasks and log results.\nInstruction:\n\nWith the input and the inference results, the AI assistant needs to describe the process and results. The previous stages can be formed as - User Input: {{ User Input }}, Task Planning: {{ Tasks }}, Model Selection: {{ Model Assignment }}, Task Execution: {{ Predictions }}. You must first answer the user's request in a straightforward manner. Then describe the task process and show your analysis and model inference results to the user in the first person. If inference results contain a file path, must tell the user the complete file path.")]}}

----------------

{'generate': {'answer': 'Task decomposition is the process of breaking down a complex task into smaller, more manageable steps. This technique, often enhanced by methods like Chain of Thought (CoT) or Tree of Thoughts, allows models to reason through tasks systematically and improves performance by clarifying the thought process. It can be achieved through simple prompts, task-specific instructions, or human inputs.'}}

----------------

Stream

tokens

:

for message, metadata in graph.stream(

{"question": "What is Task Decomposition?"}, stream_mode="messages"

):

print(message.content, end="|")

|Task| decomposition| is| the| process| of| breaking| down| complex| tasks| into| smaller|,| more| manageable| steps|.| It| can| be| achieved| through| techniques| like| Chain| of| Thought| (|Co|T|)| prompting|,| which| encourages| the| model| to| think| step| by| step|,| or| through| more| structured| methods| like| the| Tree| of| Thoughts|.| This| approach| not| only| simplifies| task| execution| but| also| provides| insights| into| the| model|'s| reasoning| process|.||

For async invocations, use:

result = await graph.ainvoke(...)

and

async for step in graph.astream(...):

Returning sources

​

Note that by storing the retrieved context in the state of the graph, we recover sources for the model's generated answer in the

"context"

field of the state. See

this guide

on returning sources for more detail.

Go deeper

​

Chat models

take in a sequence of messages and return a message.

Customizing the prompt

As shown above, we can load prompts (e.g.,

this RAG

prompt

) from the prompt

hub. The prompt can also be easily customized. For example:

from langchain_core.prompts import PromptTemplate

template = """Use the following pieces of context to answer the question at the end.

If you don't know the answer, just say that you don't know, don't try to make up an answer.

Use three sentences maximum and keep the answer as concise as possible.

Always say "thanks for asking!" at the end of the answer.

{context}

Question: {question}

Helpful Answer:"""

custom_rag_prompt = PromptTemplate.from_template(template)

Query analysis

​

So far, we are executing the retrieval using the raw input query. However, there are some advantages to allowing a model to generate the query for retrieval purposes. For example:

In addition to semantic search, we can build in structured filters (e.g., "Find documents since the year 2020.");

The model can rewrite user queries, which may be multifaceted or include irrelevant language, into more effective search queries.

Query analysis

employs models to transform or construct optimized search queries from raw user input. We can easily incorporate a query analysis step into our application. For illustrative purposes, let's add some metadata to the documents in our vector store. We will add some (contrived) sections to the document which we can filter on later.

total_documents = len(all_splits)

third = total_documents // 3

for i, document in enumerate(all_splits):

if i < third:

document.metadata["section"] = "beginning"

elif i < 2 * third:

document.metadata["section"] = "middle"

else:

document.metadata["section"] = "end"

all_splits[0].metadata

{'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/',

'start_index': 8,

'section': 'beginning'}

We will need to update the documents in our vector store. We will use a simple

InMemoryVectorStore

for this, as we will use some of its specific features (i.e., metadata filtering). Refer to the vector store

integration documentation

for relevant features of your chosen vector store.

from langchain_core.vectorstores import InMemoryVectorStore

vector_store = InMemoryVectorStore(embeddings)

_ = vector_store.add_documents(all_splits)

Let's next define a schema for our search query. We will use

structured output

for this purpose. Here we define a query as containing a string query and a document section (either "beginning", "middle", or "end"), but this can be defined however you like.

from typing import Literal

from typing_extensions import Annotated

class Search(TypedDict):

"""Search query."""

query: Annotated[str, ..., "Search query to run."]

section: Annotated[

Literal["beginning", "middle", "end"],

...,

"Section to query.",

]

Finally, we add a step to our LangGraph application to generate a query from the user's raw input:

class State(TypedDict):

question: str

query: Search

context: List[Document]

answer: str

def analyze_query(state: State):

structured_llm = llm.with_structured_output(Search)

query = structured_llm.invoke(state["question"])

return {"query": query}

def retrieve(state: State):

query = state["query"]

retrieved_docs = vector_store.similarity_search(

query["query"],

filter=lambda doc: doc.metadata.get("section") == query["section"],

)

return {"context": retrieved_docs}

def generate(state: State):

docs_content = "\n\n".join(doc.page_content for doc in state["context"])

messages = prompt.invoke({"question": state["question"], "context": docs_content})

response = llm.invoke(messages)

return {"answer": response.content}

graph_builder = StateGraph(State).add_sequence([analyze_query, retrieve, generate])

graph_builder.add_edge(START, "analyze_query")

graph = graph_builder.compile()

Full Code:

from typing import Literal

import bs4

from langchain import hub

from langchain_community.document_loaders import WebBaseLoader

from langchain_core.documents import Document

from langchain_core.vectorstores import InMemoryVectorStore

from langchain_text_splitters import RecursiveCharacterTextSplitter

from langgraph.graph import START, StateGraph

from typing_extensions import Annotated, List, TypedDict

loader = WebBaseLoader(

web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),

bs_kwargs=dict(

parse_only=bs4.SoupStrainer(

class_=("post-content", "post-title", "post-header")

)

),

)

docs = loader.load()

text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)

all_splits = text_splitter.split_documents(docs)

total_documents = len(all_splits)

third = total_documents // 3

for i, document in enumerate(all_splits):

if i < third:

document.metadata["section"] = "beginning"

elif i < 2 * third:

document.metadata["section"] = "middle"

else:

document.metadata["section"] = "end"

vector_store = InMemoryVectorStore(embeddings)

_ = vector_store.add_documents(all_splits)

class Search(TypedDict):

"""Search query."""

query: Annotated[str, ..., "Search query to run."]

section: Annotated[

Literal["beginning", "middle", "end"],

...,

"Section to query.",

]

prompt = hub.pull("rlm/rag-prompt")

class State(TypedDict):

question: str

query: Search

context: List[Document]

answer: str

def analyze_query(state: State):

structured_llm = llm.with_structured_output(Search)

query = structured_llm.invoke(state["question"])

return {"query": query}

def retrieve(state: State):

query = state["query"]

retrieved_docs = vector_store.similarity_search(

query["query"],

filter=lambda doc: doc.metadata.get("section") == query["section"],

)

return {"context": retrieved_docs}

def generate(state: State):

docs_content = "\n\n".join(doc.page_content for doc in state["context"])

messages = prompt.invoke({"question": state["question"], "context": docs_content})

response = llm.invoke(messages)

return {"answer": response.content}

graph_builder = StateGraph(State).add_sequence([analyze_query, retrieve, generate])

graph_builder.add_edge(START, "analyze_query")

graph = graph_builder.compile()

display(Image(graph.get_graph().draw_mermaid_png()))

We can test our implementation by specifically asking for context from the end of the post. Note that the model includes different information in its answer.

for step in graph.stream(

{"question": "What does the end of the post say about Task Decomposition?"},

stream_mode="updates",

):

print(f"{step}\n\n----------------\n")

{'analyze_query': {'query': {'query': 'Task Decomposition', 'section': 'end'}}}

----------------

{'retrieve': {'context': [Document(id='d6cef137-e1e8-4ddc-91dc-b62bd33c6020', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'start_index': 39221, 'section': 'end'}, page_content='Finite context length: The restricted context capacity limits the inclusion of historical information, detailed instructions, API call context, and responses. The design of the system has to work with this limited communication bandwidth, while mechanisms like self-reflection to learn from past mistakes would benefit a lot from long or infinite context windows. Although vector stores and retrieval can provide access to a larger knowledge pool, their representation power is not as powerful as full attention.\n\n\nChallenges in long-term planning and task decomposition: Planning over a lengthy history and effectively exploring the solution space remain challenging. LLMs struggle to adjust plans when faced with unexpected errors, making them less robust compared to humans who learn from trial and error.'), Document(id='d1834ae1-eb6a-43d7-a023-08dfa5028799', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'start_index': 39086, 'section': 'end'}, page_content='}\n]\nChallenges#\nAfter going through key ideas and demos of building LLM-centered agents, I start to see a couple common limitations:'), Document(id='ca7f06e4-2c2e-4788-9a81-2418d82213d9', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'start_index': 32942, 'section': 'end'}, page_content='}\n]\nThen after these clarification, the agent moved into the code writing mode with a different system message.\nSystem message:'), Document(id='1fcc2736-30f4-4ef6-90f2-c64af92118cb', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/', 'start_index': 35127, 'section': 'end'}, page_content='"content": "You will get instructions for code to write.\\nYou will write a very long answer. Make sure that every detail of the architecture is, in the end, implemented as code.\\nMake sure that every detail of the architecture is, in the end, implemented as code.\\n\\nThink step by step and reason yourself to the right decisions to make sure we get it right.\\nYou will first lay out the names of the core classes, functions, methods that will be necessary, as well as a quick comment on their purpose.\\n\\nThen you will output the content of each file including ALL code.\\nEach file must strictly follow a markdown code block format, where the following tokens must be replaced such that\\nFILENAME is the lowercase file name including the file extension,\\nLANG is the markup code block language for the code\'s language, and CODE is the code:\\n\\nFILENAME\\n\`\`\`LANG\\nCODE\\n\`\`\`\\n\\nYou will start with the \\"entrypoint\\" file, then go to the ones that are imported by that file, and so on.\\nPlease')]}}

----------------

{'generate': {'answer': 'The end of the post highlights that task decomposition faces challenges in long-term planning and adapting to unexpected errors. LLMs struggle with adjusting their plans, making them less robust compared to humans who learn from trial and error. This indicates a limitation in effectively exploring the solution space and handling complex tasks.'}}

----------------

In both the streamed steps and the

LangSmith trace

, we can now observe the structured query that was fed into the retrieval step.

Query Analysis is a rich problem with a wide range of approaches. Refer to the

how-to guides

for more examples.

Next steps

​

We've covered the steps to build a basic Q&A app over data:

In

Part 2

of the tutorial, we will extend the implementation here to accommodate conversation-style interactions and multi-step retrieval processes.

Further reading:


added_at: 2025-08-28 01:35:37
thumb_url: https://digiday.com/wp-content/uploads/sites/3/2025/08/grounding-ai-digiday.jpg
url:
https://digiday.com/media/wtf-is-ai-grounding-licensing-and-why-do-publishers-say-it-matters-over-training-deals/

title:
WTF is AI ‘grounding’ licensing, and why do publishers say it matters over training deals? - Digiday

content_html:
<p>AI training licensing deals are starting to feel like yesterday’s news as publishers and platforms focus on more dynamic, usage-based models.</p>
<p>Rather than the initial training deals that formed the backbone of AI licensing partnerships between AI platforms and news publishers, recent deals have forged around different parameters: what many in the industry refer to as “AI grounding.”</p>

<p>In fast-moving digital areas like AI, the terminology tends to splinter quickly. Vendors, publishers, platforms and analysts coin their own terms: for instance, “grounding,” “content inference compute”, and “retrieval augmented generation” (RAG) are all intertwined and refer more or less to the same thing. Those who can’t be bothered with jargon of any sort simply call grounding and RAG “web search.” </p>
<p>To AI engineers, there are subtle differences between them, but for publishers, RAG/grounding has changed how they get paid now given how the large language models (LLMs) now process information.</p>
<p>One-time lump sum payments are out; recurring, usage-based licensing agreements are in. “As we’ve moved more into RAG deals, the per-usage aspect of these pricing structures has become the preeminent piece of the pie when it comes to fees,” said Aaron G. Rubin, partner in the strategic transactions and licensing group for law firm Gunderson Dettmer. </p>
<p>Here’s a primer. </p>
<h2><strong>What is the difference between training versus grounding deals?</strong></h2>
<p>In a nutshell, payment terms of grounding or “RAG” deals are based on how AI systems fetch live content from publishers in real time. If a person searches for an update on some recent news like, “Show me an update on the meeting between Trump and Zelensky,” which happened over the last week, AI engines won’t have that stored in their training. “Training windows for AI engines tend to be up to six months old; they don’t know anything after the training date,” said Martin Alderson, co-founder of web performance consultancy Catch Metrics. That’s why they use RAG to pull the information from a multitude of publishers to provide the best response to the user. </p>
<p>That model should create opportunities for recurring licensing revenue, attributions and continued visibility. In contrast, training deals are typically one-time payments where publishers get an upfront lump sum, or have a fixed fee over years for content used to train a model. The New York Times agreed to a training deal with Amazon, to the tune of $20 million, while News Corp did similar for $50 million. Many of the agreements from the first wave of publisher-AI platform deals would have been for training.</p>
<h2><strong>Why is focus shifting to so-called grounding or RAG deals?</strong></h2>
<p>For starters, few publishers would have been able to negotiate to the same level as the NYT and News Corp. But also because the value of training data has receded for AI platforms. For publishers like DPG Media, training deals don’t warrant decent payouts, stressed Valerie de Naeyer, head of Gen AI transformation and operational excellence at DPG Media. “In terms of copyright law, publishers are not so keen on licensing content to train the model either — lots of questions on IP remain unresolved,” she said. “It’s possible that there is also a training component in some deals, in case of historic or less relevant content, but in case of real-time, content grounding is preferred,” she added. </p>
<p>On July 30, Gannett signed a licensing deal with Perplexity to allow it to license content from USA Today and the USA Network. As always, details on payment terms are scarce, but it’s an example of a RAG/grounding deal due to Perplexity’s approach, which centers on ad revenue sharing, not training content deals. </p>
<p></p>
<p>“Gannett has joined <a href="https://cts.businesswire.com/ct/CT?id=smartlink&amp;url=https%3A%2F%2Fwww.perplexity.ai%2Fhub%2Fblog%2Fintroducing-the-perplexity-publishers-program&amp;esheet=54299415&amp;newsitemid=20250730968765&amp;lan=en-US&amp;anchor=Perplexity+Publisher+Program&amp;index=3&amp;md5=ad9dee34b62920b38ee8cc1ed75a4aca" rel="noopener noreferrer" target="_blank">Perplexity’s Publisher Program</a>, which incorporates Retrieval Augmented Generation (RAG) as it relates to our trusted content being included as part of answers to Perplexity users question[s] through their consumer offerings,” confirmed a Gannett spokesperson in an email statement. </p>
<h2><strong>So if it’s not a flat fee, what is payment based on? </strong></h2>
<p>The umbrella term is usage-based payment structures. There are a plethora of examples already and which exact type of payment that will be agreed upon will vary depending on the AI company involved. Some examples are: pay per usage, pay per query, pay per crawl, and those based on ad revenue sharing, like Perplexity and Prorata.ai provide, which remunerate publishers when their content is used within RAG. The IAB Tech Lab is working with publishers and cloud edge companies to develop both pay-per-crawl and pay-per-query models for its standardized framework. </p>
<p>From a licensing standpoint, the key question is whether content is actually surfaced in the output — cited, attributed, and linked back. That’s what defines a RAG-style deal, stressed Rubin. In contrast, traditional training deals involve feeding content into a model so it can learn from it at scale, but without necessarily reflecting that specific content in the output, he added. </p>
<p>“I think a lot of these licensing deals have moved to…the grounding side of things, where if I want to cite and use News Corp articles in my output and link to them, I need to license that from them if I’m a tech company,” he said. “And so I think that’s another reason why we’re seeing these grounding deals become more prominent in the recent past, and going forward.”</p>
<h2><strong>Is there a preferred type of usage deal yet? </strong></h2>
<p>Too early to say. Deals will depend on the negotiating strength of each party, stressed Gary Kibel, partner at law firm Davis+Gilbert. “Both sides are learning and becoming more sophisticated in these deals,” he said. “Maybe publishers are starting to realize what additional controls they should push for in the agreements, and the AI platforms are starting to learn about maybe additional permitted uses they want to get into the agreement, he added. </p>
<p>A 2025 AI licensing deal already looks different from a 2024 one, thanks to lessons learned — and by 2026, deals will likely evolve again as new applications for content emerge, said Kibel.</p>
<p>“There is no one-size-fits-all with finance,” he added.  </p>
<h2><strong>But this evolution in the payment terms seems ultimately better for publishers, right? </strong></h2>
<p>Right. When the earliest version of ChatGPT first burst on the scene in November 2022, the picture looked very different. Publishers had a common fear: that the LLMs had stripped all their content. The models were built. It was game over. So it was, in a sense, a period of damage control on their part. “People negotiated deals and made some money, but none of the deals seemed particularly great, and they were all one-offs,” said Paul Bannister, chief strategy officer at Raptive. So it’s like, say you got a check for $20 million, that’s great, but it’s not going to save your business five years from now.” </p>
<p>For now, it’s all about usage. Publishers are reporting a surge in crawls, with the same piece of content sometimes scraped thousands of times a day by AI systems, stressed Bannister. The spike is tied to RAG and grounding techniques, which trigger fresh pulls of the same content for each new type of query. So, sure, there may be ways AI companies get more efficient at that in time, and a single pull will suffice, but for now, there is value in that for publishers, if they have a deal based on pay per crawl, for example. </p>
<p>“I do hear a lot from publishers these days that the type of training deals publishers were doing a year ago are not going to renew,” added Bannister. “Everyone is talking more and more about grounding being the right thing, and probably because, to some level, there is an easier business model behind it.” </p>
 

content_text:
AI training licensing deals are starting to feel like yesterday’s news as publishers and platforms focus on more dynamic, usage-based models.

Rather than the initial training deals that formed the backbone of AI licensing partnerships between AI platforms and news publishers, recent deals have forged around different parameters: what many in the industry refer to as “AI grounding.”

In fast-moving digital areas like AI, the terminology tends to splinter quickly. Vendors, publishers, platforms and analysts coin their own terms: for instance, “grounding,” “content inference compute”, and “retrieval augmented generation” (RAG) are all intertwined and refer more or less to the same thing. Those who can’t be bothered with jargon of any sort simply call grounding and RAG “web search.”

To AI engineers, there are subtle differences between them, but for publishers, RAG/grounding has changed how they get paid now given how the large language models (LLMs) now process information.

One-time lump sum payments are out; recurring, usage-based licensing agreements are in. “As we’ve moved more into RAG deals, the per-usage aspect of these pricing structures has become the preeminent piece of the pie when it comes to fees,” said Aaron G. Rubin, partner in the strategic transactions and licensing group for law firm Gunderson Dettmer.

Here’s a primer.

What is the difference between training versus grounding deals?

In a nutshell, payment terms of grounding or “RAG” deals are based on how AI systems fetch live content from publishers in real time. If a person searches for an update on some recent news like, “Show me an update on the meeting between Trump and Zelensky,” which happened over the last week, AI engines won’t have that stored in their training. “Training windows for AI engines tend to be up to six months old; they don’t know anything after the training date,” said Martin Alderson, co-founder of web performance consultancy Catch Metrics. That’s why they use RAG to pull the information from a multitude of publishers to provide the best response to the user.

That model should create opportunities for recurring licensing revenue, attributions and continued visibility. In contrast, training deals are typically one-time payments where publishers get an upfront lump sum, or have a fixed fee over years for content used to train a model. The New York Times agreed to a training deal with Amazon, to the tune of $20 million, while News Corp did similar for $50 million. Many of the agreements from the first wave of publisher-AI platform deals would have been for training.

Why is focus shifting to so-called grounding or RAG deals?

For starters, few publishers would have been able to negotiate to the same level as the NYT and News Corp. But also because the value of training data has receded for AI platforms. For publishers like DPG Media, training deals don’t warrant decent payouts, stressed Valerie de Naeyer, head of Gen AI transformation and operational excellence at DPG Media. “In terms of copyright law, publishers are not so keen on licensing content to train the model either — lots of questions on IP remain unresolved,” she said. “It’s possible that there is also a training component in some deals, in case of historic or less relevant content, but in case of real-time, content grounding is preferred,” she added.

On July 30, Gannett signed a licensing deal with Perplexity to allow it to license content from USA Today and the USA Network. As always, details on payment terms are scarce, but it’s an example of a RAG/grounding deal due to Perplexity’s approach, which centers on ad revenue sharing, not training content deals.

“Gannett has joined

Perplexity’s Publisher Program

, which incorporates Retrieval Augmented Generation (RAG) as it relates to our trusted content being included as part of answers to Perplexity users question[s] through their consumer offerings,” confirmed a Gannett spokesperson in an email statement.

So if it’s not a flat fee, what is payment based on?

The umbrella term is usage-based payment structures. There are a plethora of examples already and which exact type of payment that will be agreed upon will vary depending on the AI company involved. Some examples are: pay per usage, pay per query, pay per crawl, and those based on ad revenue sharing, like Perplexity and Prorata.ai provide, which remunerate publishers when their content is used within RAG. The IAB Tech Lab is working with publishers and cloud edge companies to develop both pay-per-crawl and pay-per-query models for its standardized framework.

From a licensing standpoint, the key question is whether content is actually surfaced in the output — cited, attributed, and linked back. That’s what defines a RAG-style deal, stressed Rubin. In contrast, traditional training deals involve feeding content into a model so it can learn from it at scale, but without necessarily reflecting that specific content in the output, he added.

“I think a lot of these licensing deals have moved to…the grounding side of things, where if I want to cite and use News Corp articles in my output and link to them, I need to license that from them if I’m a tech company,” he said. “And so I think that’s another reason why we’re seeing these grounding deals become more prominent in the recent past, and going forward.”

Is there a preferred type of usage deal yet?

Too early to say. Deals will depend on the negotiating strength of each party, stressed Gary Kibel, partner at law firm Davis+Gilbert. “Both sides are learning and becoming more sophisticated in these deals,” he said. “Maybe publishers are starting to realize what additional controls they should push for in the agreements, and the AI platforms are starting to learn about maybe additional permitted uses they want to get into the agreement, he added.

A 2025 AI licensing deal already looks different from a 2024 one, thanks to lessons learned — and by 2026, deals will likely evolve again as new applications for content emerge, said Kibel.

“There is no one-size-fits-all with finance,” he added.

But this evolution in the payment terms seems ultimately better for publishers, right?

Right. When the earliest version of ChatGPT first burst on the scene in November 2022, the picture looked very different. Publishers had a common fear: that the LLMs had stripped all their content. The models were built. It was game over. So it was, in a sense, a period of damage control on their part. “People negotiated deals and made some money, but none of the deals seemed particularly great, and they were all one-offs,” said Paul Bannister, chief strategy officer at Raptive. So it’s like, say you got a check for $20 million, that’s great, but it’s not going to save your business five years from now.”

For now, it’s all about usage. Publishers are reporting a surge in crawls, with the same piece of content sometimes scraped thousands of times a day by AI systems, stressed Bannister. The spike is tied to RAG and grounding techniques, which trigger fresh pulls of the same content for each new type of query. So, sure, there may be ways AI companies get more efficient at that in time, and a single pull will suffice, but for now, there is value in that for publishers, if they have a deal based on pay per crawl, for example.

“I do hear a lot from publishers these days that the type of training deals publishers were doing a year ago are not going to renew,” added Bannister. “Everyone is talking more and more about grounding being the right thing, and probably because, to some level, there is an easier business model behind it.”


url: https://www.techtarget.com/searchenterpriseai/tip/How-to-prepare-data-for-your-RAG-pipeline
title: How to prepare data for your RAG pipeline | TechTarget
added_at: 2025-08-28 01:36:59
thumb_url: https://www.techtarget.com/rms/onlineimages/code_g1127196618.jpg
content_html:
<section>
<p>Retrieval-augmented generation connects generative AI to enterprise data. However, without a proper RAG data pipeline, the RAG system might not find the most accurate data to answer user queries.</p>
<p>Data quality is paramount in <a href="https://www.techtarget.com/searchenterpriseai/definition/retrieval-augmented-generation" rel="noopener noreferrer" target="_blank">RAG</a> systems. Therefore, building a successful RAG application includes data preprocessing tasks, such as content filtering, text normalization, chunking, metadata tagging and embedding generation.</p>
<p>To ensure proper data preparation, RAG development teams must understand <a href="https://www.techtarget.com/searchenterpriseai/opinion/How-RAG-unlocks-the-power-of-enterprise-data" rel="noopener noreferrer" target="_blank">how RAG makes data searchable</a>, explore data preprocessing strategies such as chunking methods, and learn how to build a RAG data pipeline -- from selecting and cleaning data to embedding and storing it in a vector database.</p>
<section>
<h2><i></i>How RAG makes data searchable</h2>
<p>RAG helps generative AI services access external data by using a vector store or database with some search mechanisms.</p>
<p>First, to add unstructured data -- such as a document -- into a vector store, teams need to embed the data through an embedding model.</p>
<p>An embedding converts text into <a href="https://www.techtarget.com/searchenterpriseai/definition/vector-embeddings" rel="noopener noreferrer" target="_blank">numerical vectors</a> -- a list of decimals that represent text, such as words or sentences, in a way a computer can understand.</p>
<p>For example, consider the following paragraph from a document:</p>
<p>We launched Copilot last week, and adoption has been stellar. Early metrics show a 32% drop in ticket volume and a 27% jump in self-service resolution within the first 48 hours. Finance estimates an annual savings of $1.4 million if the trend holds.</p>
<p>Embedding this paragraph into vectors that represent the text looks like this:</p>
<p>[0.81, -0.23, 0.45, 0.12, -0.67, 0.94, 0.30, -0.55, 0.76, 0.10] </p>
<p>This is a highly simplified example; for instance, if teams used an embedding model such as OpenAI's text-embedding-3-small, it would output a 1536-dimensional vector. Multiple embedding models are available from different providers. Hugging Face has an embedding <a href="https://huggingface.co/spaces/mteb/leaderboard" rel="noopener" target="_blank">leaderboard</a> that shows the strengths and weaknesses of different embedding models.</p>
<p>These vectors and the original text are stored in a vector store such as Pinecone or Weaviate. The data in the vector store could look something like this:</p>
<p>{</p>
<p>  "id": "unique-id-123",</p>
<p>  "values": [0.81, -0.23, ..., 0.789],   // the vector (embedding)</p>
<p>  "metadata": {</p>
<p>    "text": "<i>We launched Copilot last week and adoption has been stellar…</i>.",</p>
<p>    "source": "Document A",</p>
<p>    "page": 3</p>
<p>  }</p>
<p>}</p>
<p>When a user wants to ask a question related to the data, it sends the prompt through an orchestrator -- such as LangGraph, <a href="https://www.techtarget.com/searchenterpriseai/tip/Compare-Semantic-Kernel-vs-LangChain-for-AI-development" rel="noopener noreferrer" target="_blank">LangChain or Semantic Kernel</a> -- that then sends a request to the vector store using a search method. The search method could return two or more chunks of relevant content and provide this information to the large language model (<a href="https://www.techtarget.com/whatis/definition/large-language-model-LLM" rel="noopener noreferrer" target="_blank">LLM</a>) that generates an answer to the user.</p>
</section>
<section>
<h2><i></i>The need for data chunking</h2>
<p>Chunking is a standard preprocessing method that turns a data set into smaller sections, called chunks. Since vector stores cannot store all the original data, it must be split into chunks.</p>
<p>For instance, the Copilot paragraph could be chunked into two sections:</p>
<p>Chunk 1:</p>
<p>We launched Copilot last week, and adoption has been stellar.</p>
<p>Chunk 2:</p>
<p>Early metrics show a 32% drop in ticket volume and a 27% jump in self-service resolution within the first 48 hours. Finance estimates an annual savings of $1.4 million if the trend holds.</p>
<p>The problem with this chunking approach is that some chunks might lose their original meaning. For example, if a user asked about Copilot in the RAG system, the search mechanism would only return the first chunk, not the second one, since there is no correlation between the two. The user doesn't get the added context and information about Copilot adoption without the second chunk.</p>
<p>This is where chunking strategies can help chunk<i> </i>data to fit into the vector store and retain its original meaning in a RAG engine.</p>
</section>
<section>
<h2><i></i>4 chunking strategies</h2>
<p>Without proper strategies, RAG systems can chunk data incorrectly. Different chunking methods are available, including fixed-size chunking, variable-size chunking based on content, rule-based chunking and sliding window chunking.</p>
<h3>1. Fixed-size chunking</h3>
<p>This method involves dividing text into equal-sized segments, such as 400 words or 800 characters per chunk, regardless of the content.</p>
<h3>2. Variable-size chunking based on content</h3>
<p>Here, the text is split according to natural boundaries, such as sentence-ending punctuation, line breaks or structural cues, identified by natural language processing (<a href="https://www.techtarget.com/searchenterpriseai/definition/natural-language-processing-NLP" rel="noopener noreferrer" target="_blank">NLP</a>) tools that analyze document layout and meaning.</p>
<h3>3. Rule-based chunking</h3>
<p>This approach relies on the document's intrinsic structure or linguistic boundaries. Typical methods include chunking by sentences or paragraphs using predefined rules.</p>
<h3>4. Sliding window chunking</h3>
<p>This technique produces overlapping chunks by sliding a fixed-size window through the text with a set size. For example, each chunk might contain 500 words, with the next chunk starting 300 words in to create a 200-word overlap between chunks.</p>
<p>For the Copilot paragraph, a sliding window or rule-based approach would ensure that the paragraph would not lose its meaning since both pieces of content would be available in the same chunk.</p>
</section>
<section>
<h2><i></i>6 RAG data pipeline steps</h2>
<p>Another consideration in RAG systems is keeping data up to date. For example, if the original data is a Word document before being embedded into a vector store, the data could quickly lose value if not regularly updated.</p>
<p>Also, documents that cannot be directly embedded, such as PDF files, need to be converted into a text-friendly format to push them into the vector store.</p>
<p>Therefore, it is crucial to build a data pipeline that can automate the process of continuously embedding the vector store with new data from different sources and file types.</p>
<p>A RAG data pipeline usually consists of the following six steps:</p>
<h3>1. Corpus selection and ingestion</h3>
<p>Choose the most relevant data sources and content depending on the use case.</p>
<h3>2. Data preprocessing and parsing</h3>
<p>Clean and standardize the raw data so it's ready for embedding and retrieval. A commonly used tool is <a href="https://github.com/microsoft/markitdown" rel="noopener" target="_blank">MarkItDown</a>. It supports PDF, PowerPoint and many other file formats and can convert the content to Markdown format, which makes it easier to structure when chunking.</p>
<h3>3. Enrichment</h3>
<p>Remove noisy content and add helpful metadata to the data, such as adding the document title on top so that it will be available during the chunking process.</p>
<h3>4. Filtering</h3>
<p>Remove any irrelevant or low-value documents that don't support the use case.</p>
<h3>5. Chunking</h3>
<p>Split the cleaned data into smaller, logical chunks for better retrieval performance. Frameworks such as LangChain have different chunking methods, such as LangChain CharacterTextSplitter.</p>
<h3>6. Embedding</h3>
<p>Convert each chunk into a vector and store it in a vector store.</p>
<p>Teams can automate much of this process with cloud services and frameworks. For instance, Azure AI Search, a Microsoft cloud-based vector store and search service, provides a schedule feature that automatically pulls data from a storage account, embeds it on a predefined schedule and puts it into Azure AI Search.</p>
<p><i>Marius Sandbu is a cloud evangelist for Sopra Steria in Norway who mainly focuses on end-user computing and cloud-native technology.</i></p>
</section></section>

content_text:
Retrieval-augmented generation connects generative AI to enterprise data. However, without a proper RAG data pipeline, the RAG system might not find the most accurate data to answer user queries.

Data quality is paramount in

RAG

systems. Therefore, building a successful RAG application includes data preprocessing tasks, such as content filtering, text normalization, chunking, metadata tagging and embedding generation.

To ensure proper data preparation, RAG development teams must understand

how RAG makes data searchable

, explore data preprocessing strategies such as chunking methods, and learn how to build a RAG data pipeline -- from selecting and cleaning data to embedding and storing it in a vector database.

How RAG makes data searchable

RAG helps generative AI services access external data by using a vector store or database with some search mechanisms.

First, to add unstructured data -- such as a document -- into a vector store, teams need to embed the data through an embedding model.

An embedding converts text into

numerical vectors

-- a list of decimals that represent text, such as words or sentences, in a way a computer can understand.

For example, consider the following paragraph from a document:

We launched Copilot last week, and adoption has been stellar. Early metrics show a 32% drop in ticket volume and a 27% jump in self-service resolution within the first 48 hours. Finance estimates an annual savings of $1.4 million if the trend holds.

Embedding this paragraph into vectors that represent the text looks like this:

[0.81, -0.23, 0.45, 0.12, -0.67, 0.94, 0.30, -0.55, 0.76, 0.10]

This is a highly simplified example; for instance, if teams used an embedding model such as OpenAI's text-embedding-3-small, it would output a 1536-dimensional vector. Multiple embedding models are available from different providers. Hugging Face has an embedding

leaderboard

that shows the strengths and weaknesses of different embedding models.

These vectors and the original text are stored in a vector store such as Pinecone or Weaviate. The data in the vector store could look something like this:

{

"id": "unique-id-123",

"values": [0.81, -0.23, ..., 0.789],   // the vector (embedding)

"metadata": {

"text": "

We launched Copilot last week and adoption has been stellar…

.",

"source": "Document A",

"page": 3

}

}

When a user wants to ask a question related to the data, it sends the prompt through an orchestrator -- such as LangGraph,

LangChain or Semantic Kernel

-- that then sends a request to the vector store using a search method. The search method could return two or more chunks of relevant content and provide this information to the large language model (

LLM

) that generates an answer to the user.

The need for data chunking

Chunking is a standard preprocessing method that turns a data set into smaller sections, called chunks. Since vector stores cannot store all the original data, it must be split into chunks.

For instance, the Copilot paragraph could be chunked into two sections:

Chunk 1:

We launched Copilot last week, and adoption has been stellar.

Chunk 2:

Early metrics show a 32% drop in ticket volume and a 27% jump in self-service resolution within the first 48 hours. Finance estimates an annual savings of $1.4 million if the trend holds.

The problem with this chunking approach is that some chunks might lose their original meaning. For example, if a user asked about Copilot in the RAG system, the search mechanism would only return the first chunk, not the second one, since there is no correlation between the two. The user doesn't get the added context and information about Copilot adoption without the second chunk.

This is where chunking strategies can help chunk

data to fit into the vector store and retain its original meaning in a RAG engine.

4 chunking strategies

Without proper strategies, RAG systems can chunk data incorrectly. Different chunking methods are available, including fixed-size chunking, variable-size chunking based on content, rule-based chunking and sliding window chunking.

1. Fixed-size chunking

This method involves dividing text into equal-sized segments, such as 400 words or 800 characters per chunk, regardless of the content.

2. Variable-size chunking based on content

Here, the text is split according to natural boundaries, such as sentence-ending punctuation, line breaks or structural cues, identified by natural language processing (

NLP

) tools that analyze document layout and meaning.

3. Rule-based chunking

This approach relies on the document's intrinsic structure or linguistic boundaries. Typical methods include chunking by sentences or paragraphs using predefined rules.

4. Sliding window chunking

This technique produces overlapping chunks by sliding a fixed-size window through the text with a set size. For example, each chunk might contain 500 words, with the next chunk starting 300 words in to create a 200-word overlap between chunks.

For the Copilot paragraph, a sliding window or rule-based approach would ensure that the paragraph would not lose its meaning since both pieces of content would be available in the same chunk.

6 RAG data pipeline steps

Another consideration in RAG systems is keeping data up to date. For example, if the original data is a Word document before being embedded into a vector store, the data could quickly lose value if not regularly updated.

Also, documents that cannot be directly embedded, such as PDF files, need to be converted into a text-friendly format to push them into the vector store.

Therefore, it is crucial to build a data pipeline that can automate the process of continuously embedding the vector store with new data from different sources and file types.

A RAG data pipeline usually consists of the following six steps:

1. Corpus selection and ingestion

Choose the most relevant data sources and content depending on the use case.

2. Data preprocessing and parsing

Clean and standardize the raw data so it's ready for embedding and retrieval. A commonly used tool is

MarkItDown

. It supports PDF, PowerPoint and many other file formats and can convert the content to Markdown format, which makes it easier to structure when chunking.

3. Enrichment

Remove noisy content and add helpful metadata to the data, such as adding the document title on top so that it will be available during the chunking process.

4. Filtering

Remove any irrelevant or low-value documents that don't support the use case.

5. Chunking

Split the cleaned data into smaller, logical chunks for better retrieval performance. Frameworks such as LangChain have different chunking methods, such as LangChain CharacterTextSplitter.

6. Embedding

Convert each chunk into a vector and store it in a vector store.

Teams can automate much of this process with cloud services and frameworks. For instance, Azure AI Search, a Microsoft cloud-based vector store and search service, provides a schedule feature that automatically pulls data from a storage account, embeds it on a predefined schedule and puts it into Azure AI Search.

Marius Sandbu is a cloud evangelist for Sopra Steria in Norway who mainly focuses on end-user computing and cloud-native technology.


url: https://developers.googleblog.com/en/vertex-ai-rag-engine-a-developers-tool/
title: Vertex AI RAG Engine: A developers tool - Google Developers Blog
added_at: 2025-08-28 01:55:38
content_html:
<p>Generative AI and Large Language Models (LLMs) are transforming industries, but two key challenges can hinder enterprise adoption: hallucinations (generating incorrect or nonsensical information) and limited knowledge beyond their training data. Retrieval Augmented Generation (RAG) and grounding offer solutions by connecting LLMs to external data sources, enabling them to access up-to-date information and generate more factual and relevant responses.</p><p>This post explores Vertex AI RAG Engine and how it empowers software and AI developers to build robust, grounded generative AI applications.</p><h3><b><br/>What is RAG and why do you need it?</b></h3><p>RAG retrieves relevant information from a knowledge base and feeds it to an LLM, allowing it to generate more accurate and informed responses. This contrasts with relying solely on the LLM's pre-trained knowledge, which can be outdated or incomplete. RAG is essential for building enterprise-grade Gen AI applications that require:</p><p></p><ul><li><b>Accuracy:</b> Minimizing hallucinations and ensuring responses are factually grounded.</li></ul><p></p><ul><li><b>Up-to-date Information:</b> Accessing the latest data and insights.</li></ul><p></p><ul><li><b>Domain Expertise:</b> Leveraging specialized knowledge bases for specific use cases.</li></ul><h2><br/>RAG vs Grounding vs Search</h2><p></p><ul><li><b>RAG</b>: a technique to retrieve and provide relevant information to LLMs to generate responses. The information can include fresh information, topic and context, or ground truth.</li></ul><p></p><ul><li><b>Grounding</b>: Ensure the reliability and trustworthiness of AI-generated content by anchoring it to verified sources of information. Grounding may use RAG as a technique.</li></ul><p></p><ul><li><b>Search</b>: an approach to quickly find and deliver relevant information from a data source based on text or multi-modal queries powered by advanced AI models.</li></ul><h2><br/>Introducing Vertex AI RAG Engine</h2><p>Vertex AI RAG Engine is a managed orchestration service, streamlining the complex process of retrieving relevant information and feeding it to an LLM. This allows developers to focus on building their applications rather than managing infrastructure.</p>


<p><b>Key Advantages of Vertex AI RAG Engine:</b></p><p></p><ul><li><b>Ease of Use:</b> Get started quickly with a simple API, enabling rapid prototyping and experimentation.</li></ul><p></p><ul><li><b>Managed Orchestration:</b> Handles the complexities of data retrieval and LLM integration, freeing developers from infrastructure management.</li></ul><p></p><ul><li><b>Customization and Open-Source Support:</b> Choose from a variety of parsing, chunking, annotation, embedding, vector storage, and open-source models, or customize your own components.</li></ul><p></p><ul><li><b>High-Quality Google Components:</b> Leverage Google's cutting-edge technology for optimal performance.</li></ul><p></p><ul><li><b>Integration Flexibility:</b> Connect to various vector databases like Pinecone and Weaviate, or use Vertex AI Vector Search.</li></ul><h3><b><br/>Vertex AI RAG: A Spectrum of Solutions</b></h3><p>Google Cloud offers a spectrum of RAG and grounding solutions, catering to varying levels of complexity and customization:</p><p></p><ul><li><b>Vertex AI Search:</b> A fully managed search engine and retriever API ideal for complex enterprise use cases requiring high out-of-the-box quality, scalability, and fine-grained access controls. It simplifies connecting to diverse enterprise data sources and enables searching across multiple sources.</li></ul><p></p><ul><li><b>Fully DIY RAG:</b> For developers seeking complete control, Vertex AI provides individual component APIs (e.g., Text Embedding API, Ranking API, Grounding on Vertex AI) to build custom RAG pipelines. This approach offers maximum flexibility but requires significant development effort. Use this if you need very specific customizations or want to integrate with existing RAG frameworks.</li></ul><p></p><ul><li><b>Vertex AI RAG Engine:</b> The sweet spot for developers seeking a balance between ease of use and customization. It empowers rapid prototyping and development without sacrificing flexibility.</li></ul><h3><b><br/>Common Industry use cases for RAG Engine:</b></h3><p></p><ol><li><b>Financial Services: Personalized Investment Advice &amp; Risk Assessment:</b></li></ol><p></p><p>Problem: Financial advisors need to quickly synthesize vast amounts of information – client profiles, market data, regulatory filings, and internal research – to provide tailored investment advice and accurate risk assessments. Manually reviewing all this information is time-consuming and prone to errors.</p><p></p><p>RAG Engine Solution: A RAG engine can ingest and index relevant data sources. Financial advisors can then query the system with a client's specific profile and investment goals. The RAG engine will provide a concise, evidence-based response drawing from the relevant documents, including citations to support the recommendations. This improves advisor efficiency, reduces risk of human error, and enhances the personalization of advice. The system could also flag potential conflicts of interest or regulatory violations based on information found in the ingested data.</p><p><br/><b>2. Healthcare: Accelerated Drug Discovery &amp; Personalized Treatment Plans:</b></p><p></p><p>Problem: Drug discovery and personalized medicine rely heavily on analyzing massive datasets of clinical trials, research papers, patient records, and genetic information. Sifting through this data to identify potential drug targets, predict patient responses to treatments, or generate personalized treatment plans is incredibly challenging.</p><p></p><p>RAG Engine Solution: With appropriate privacy and security measures, a RAG engine can ingest and index the vast biomedical literature and patient data . Researchers can then pose complex queries, like "What are the potential side effects of drug X in patients with genotype Y?" The RAG engine would synthesize relevant information from various sources, providing researchers with insights they might miss in a manual search. For clinicians, the engine could help generate suggested personalized treatment plans based on a patient's unique characteristics and medical history, supported by evidence from relevant research.</p><p><br/><b>3. Legal: Enhanced Due Diligence and Contract Review:</b></p><p></p><p>Problem: Legal professionals spend significant time reviewing documents during due diligence processes, contract negotiations, and litigation. Finding relevant clauses, identifying potential risks, and ensuring compliance with regulations is time-intensive and requires deep expertise.</p><p></p><p>RAG Engine Solution: A RAG engine can ingest and index legal documents, case law, and regulatory information. Legal professionals can query the system to find specific clauses within contracts, identify potential legal risks, and research relevant precedents. The engine can highlight inconsistencies, potential liabilities, and relevant case law, significantly speeding up the review process and improving accuracy. This leads to faster deal closures, reduced legal risks, and more efficient use of legal expertise.</p><h2><b><br/></b>Getting started with Vertex AI RAG Engine</h2><p>Google provides ample resources to help you get started, including:</p><p></p><ul><li><b>Getting Started Notebook:</b></li></ul><p></p><ul><li><b>Documentation:</b> Comprehensive documentation guides you through the setup and usage of RAG Engine.</li></ul><p></p><ul><li><b>Integrations:</b> Examples with Vertex AI Vector Search, Vertex AI Feature Store, Pinecone, and Weaviate</li></ul><p></p><ul><li><b>Evaluation Framework:</b> Learn how to evaluate and perform hyperparameter tuning for retrieval with RAG Engine:</li></ul><h3><br/><b>Build grounded generative AI</b></h3><p>Vertex AI's RAG Engine and suite of grounding solutions empower developers to build more reliable, factual, and insightful generative AI applications. By leveraging these tools, you can unlock the full potential of LLMs and overcome the challenges of hallucinations and limited knowledge, paving the way for wider enterprise adoption of generative AI. Choose the solution that best fits your needs and start building the next generation of intelligent applications.</p>

content_text:
Generative AI and Large Language Models (LLMs) are transforming industries, but two key challenges can hinder enterprise adoption: hallucinations (generating incorrect or nonsensical information) and limited knowledge beyond their training data. Retrieval Augmented Generation (RAG) and grounding offer solutions by connecting LLMs to external data sources, enabling them to access up-to-date information and generate more factual and relevant responses.

This post explores Vertex AI RAG Engine and how it empowers software and AI developers to build robust, grounded generative AI applications.

What is RAG and why do you need it?

RAG retrieves relevant information from a knowledge base and feeds it to an LLM, allowing it to generate more accurate and informed responses. This contrasts with relying solely on the LLM's pre-trained knowledge, which can be outdated or incomplete. RAG is essential for building enterprise-grade Gen AI applications that require:

Accuracy:

Minimizing hallucinations and ensuring responses are factually grounded.

Up-to-date Information:

Accessing the latest data and insights.

Domain Expertise:

Leveraging specialized knowledge bases for specific use cases.

RAG vs Grounding vs Search

RAG

: a technique to retrieve and provide relevant information to LLMs to generate responses. The information can include fresh information, topic and context, or ground truth.

Grounding

: Ensure the reliability and trustworthiness of AI-generated content by anchoring it to verified sources of information. Grounding may use RAG as a technique.

Search

: an approach to quickly find and deliver relevant information from a data source based on text or multi-modal queries powered by advanced AI models.

Introducing Vertex AI RAG Engine

Vertex AI RAG Engine is a managed orchestration service, streamlining the complex process of retrieving relevant information and feeding it to an LLM. This allows developers to focus on building their applications rather than managing infrastructure.

Key Advantages of Vertex AI RAG Engine:

Ease of Use:

Get started quickly with a simple API, enabling rapid prototyping and experimentation.

Managed Orchestration:

Handles the complexities of data retrieval and LLM integration, freeing developers from infrastructure management.

Customization and Open-Source Support:

Choose from a variety of parsing, chunking, annotation, embedding, vector storage, and open-source models, or customize your own components.

High-Quality Google Components:

Leverage Google's cutting-edge technology for optimal performance.

Integration Flexibility:

Connect to various vector databases like Pinecone and Weaviate, or use Vertex AI Vector Search.

Vertex AI RAG: A Spectrum of Solutions

Google Cloud offers a spectrum of RAG and grounding solutions, catering to varying levels of complexity and customization:

Vertex AI Search:

A fully managed search engine and retriever API ideal for complex enterprise use cases requiring high out-of-the-box quality, scalability, and fine-grained access controls. It simplifies connecting to diverse enterprise data sources and enables searching across multiple sources.

Fully DIY RAG:

For developers seeking complete control, Vertex AI provides individual component APIs (e.g., Text Embedding API, Ranking API, Grounding on Vertex AI) to build custom RAG pipelines. This approach offers maximum flexibility but requires significant development effort. Use this if you need very specific customizations or want to integrate with existing RAG frameworks.

Vertex AI RAG Engine:

The sweet spot for developers seeking a balance between ease of use and customization. It empowers rapid prototyping and development without sacrificing flexibility.

Common Industry use cases for RAG Engine:

Financial Services: Personalized Investment Advice & Risk Assessment:

Problem: Financial advisors need to quickly synthesize vast amounts of information – client profiles, market data, regulatory filings, and internal research – to provide tailored investment advice and accurate risk assessments. Manually reviewing all this information is time-consuming and prone to errors.

RAG Engine Solution: A RAG engine can ingest and index relevant data sources. Financial advisors can then query the system with a client's specific profile and investment goals. The RAG engine will provide a concise, evidence-based response drawing from the relevant documents, including citations to support the recommendations. This improves advisor efficiency, reduces risk of human error, and enhances the personalization of advice. The system could also flag potential conflicts of interest or regulatory violations based on information found in the ingested data.

2. Healthcare: Accelerated Drug Discovery & Personalized Treatment Plans:

Problem: Drug discovery and personalized medicine rely heavily on analyzing massive datasets of clinical trials, research papers, patient records, and genetic information. Sifting through this data to identify potential drug targets, predict patient responses to treatments, or generate personalized treatment plans is incredibly challenging.

RAG Engine Solution: With appropriate privacy and security measures, a RAG engine can ingest and index the vast biomedical literature and patient data . Researchers can then pose complex queries, like "What are the potential side effects of drug X in patients with genotype Y?" The RAG engine would synthesize relevant information from various sources, providing researchers with insights they might miss in a manual search. For clinicians, the engine could help generate suggested personalized treatment plans based on a patient's unique characteristics and medical history, supported by evidence from relevant research.

3. Legal: Enhanced Due Diligence and Contract Review:

Problem: Legal professionals spend significant time reviewing documents during due diligence processes, contract negotiations, and litigation. Finding relevant clauses, identifying potential risks, and ensuring compliance with regulations is time-intensive and requires deep expertise.

RAG Engine Solution: A RAG engine can ingest and index legal documents, case law, and regulatory information. Legal professionals can query the system to find specific clauses within contracts, identify potential legal risks, and research relevant precedents. The engine can highlight inconsistencies, potential liabilities, and relevant case law, significantly speeding up the review process and improving accuracy. This leads to faster deal closures, reduced legal risks, and more efficient use of legal expertise.

Getting started with Vertex AI RAG Engine

Google provides ample resources to help you get started, including:

Getting Started Notebook:

Documentation:

Comprehensive documentation guides you through the setup and usage of RAG Engine.

Integrations:

Examples with Vertex AI Vector Search, Vertex AI Feature Store, Pinecone, and Weaviate

Evaluation Framework:

Learn how to evaluate and perform hyperparameter tuning for retrieval with RAG Engine:

Build grounded generative AI

Vertex AI's RAG Engine and suite of grounding solutions empower developers to build more reliable, factual, and insightful generative AI applications. By leveraging these tools, you can unlock the full potential of LLMs and overcome the challenges of hallucinations and limited knowledge, paving the way for wider enterprise adoption of generative AI. Choose the solution that best fits your needs and start building the next generation of intelligent applications.

thumb_url:
https://storage.googleapis.com/gweb-developer-goog-blog-assets/images/Cloud-Vertex-AI-Sequence_5.2e16d0ba.fill-1200x600.png


url: https://www.youtube.com/watch?v=Y08Nn23o_mY
title: Intro to RAG for AI (Retrieval Augmented Generation) - YouTube
added_at: 2025-08-28 01:57:00
thumb_url: https://i.ytimg.com/vi/Y08Nn23o_mY/maxresdefault.jpg
content_html:
<p>Share</p>Include playlist<p></p><p>An error occurred while retrieving sharing information. Please try again later.</p>

content_text:
Share

Include playlist

An error occurred while retrieving sharing information. Please try again later.


url: https://www.zdnet.com/article/rag-can-make-ai-models-riskier-and-less-reliable-new-research-shows/
title: RAG can make AI models riskier and less reliable, new research shows | ZDNET
added_at: 2025-08-28 01:58:14
content_html:
<figure> <figcaption> zokara/iStock/Getty Images Plus via Getty Images</figcaption></figure><h3>ZDNET's key takeaways</h3><ul><li>Sometimes an AI can cause you or your company irreparable harm.</li><li>Sharing confidential data with an AI could have legal consequences.</li><li>Don't let an AI talk to customers without supervision.</li></ul><hr/><p>A few weeks ago, I shared with you "<a href="https://www.zdnet.com/article/9-programming-tasks-you-shouldnt-hand-off-to-ai-and-why/" rel="noopener noreferrer" target="_blank">9 programming tasks you shouldn't hand off to AI - and why.</a>" It's full of well-reasoned suggestions and recommendations for how to avoid having an AI produce code that could ruin your whole day.</p><p>Then, my editor and I got talking, and we realized the whole idea of "when not to use an AI" could apply to work in general. In this article, I present to you nine things you shouldn't use AI for while at work. This is far from a comprehensive list, but it should make you think.</p><p></p><p><strong>Also: <a href="https://www.zdnet.com/article/this-one-feature-will-make-gpt-5-a-game-changer-if-openai-gets-it-right/" rel="noopener noreferrer" target="_blank">This one feature could make GPT-5 a true game changer (if OpenAI gets it right)</a></strong></p><p>"Always keep in mind that AI isn't going to read you your Miranda Rights, wrap your personal information in legal protections like HIPAA, or hesitate to disclose your secrets," said LinkedIn Learning AI instructor <a href="https://cc.zdnet.com/v1/otc/00hQi47eqnEWQ6T9d4QLBUc?element=BODY&amp;element_label=Pam+Baker&amp;module=LINK&amp;object_type=text-link&amp;object_uuid=7a1d4360-4c28-4193-ab61-ecd79f0faffa&amp;position=1&amp;template=article&amp;track_code=__COM_CLICK_ID__&amp;url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Fpambaker&amp;view_instance_uuid=53a65cd4-222c-4125-8bec-6988b4f83974" rel="noopener nofollow sponsored" target="_blank">Pam Baker</a>, the bestselling author of <a href="https://cc.zdnet.com/v1/otc/00hQi47eqnEWQ6T9d4QLBUc?element=BODY&amp;element_label=ChatGPT+For+Dummies&amp;module=LINK&amp;object_type=text-link&amp;object_uuid=7a1d4360-4c28-4193-ab61-ecd79f0faffa&amp;position=1&amp;template=article&amp;track_code=__COM_CLICK_ID__&amp;url=https%3A%2F%2Fbuy.geni.us%2FProxy.ashx%3FTSID%3D368250%26GR_URL%3Dhttps%253A%252F%252Fwww.amazon.com%252FChatGPT-Dummies-Pam-Baker%252Fdp%252F1394314450%253Ftag%253Dzd-buy-button-20%2526ascsubtag%253D__COM_CLICK_ID__%257C53a65cd4-222c-4125-8bec-6988b4f83974%257Cdtp%26dtb%3D1&amp;view_instance_uuid=53a65cd4-222c-4125-8bec-6988b4f83974" rel="noopener nofollow sponsored" target="_blank">ChatGPT For Dummies</a> and <a href="https://cc.zdnet.com/v1/otc/00hQi47eqnEWQ6T9d4QLBUc?element=BODY&amp;element_label=Generative+AI+For+Dummies&amp;module=LINK&amp;object_type=text-link&amp;object_uuid=7a1d4360-4c28-4193-ab61-ecd79f0faffa&amp;position=1&amp;template=article&amp;track_code=__COM_CLICK_ID__&amp;url=https%3A%2F%2Fbuy.geni.us%2FProxy.ashx%3FTSID%3D368250%26GR_URL%3Dhttps%253A%252F%252Fwww.amazon.com%252FGenerative-Dummies-Business-Personal-Finance%252Fdp%252F1394270747%253Ftag%253Dzd-buy-button-20%2526ascsubtag%253D__COM_CLICK_ID__%257C53a65cd4-222c-4125-8bec-6988b4f83974%257Cdtp%26dtb%3D1&amp;view_instance_uuid=53a65cd4-222c-4125-8bec-6988b4f83974" rel="noopener nofollow sponsored" target="_blank">Generative AI For Dummies</a>.</p><p>"That goes double for work AI, which is monitored closely by your employer. Whatever you do or tell AI can and likely will be used against you at some point."</p><p></p><p>To keep things interesting, read on to the end. There, I share some fun and terrifying stories about how using AI at work can go terribly, horribly, and amusingly wrong.</p><p>Without further ado, here are nine things you shouldn't do with AI at work.</p><h2> 1. Handling confidential or sensitive data </h2><p>This is an easy one. Every time you give the AI some information, ask yourself how you would feel if it were posted to the company's public blog or wound up on the front page of your industry's trade journal. </p><p></p><p><strong>Also: <a href="https://www.zdnet.com/article/the-best-ai-for-coding-in-2025-including-a-new-winner-and-what-not-to-use/" rel="noopener noreferrer" target="_blank">The best AI for coding in 2025 (and what not to use)</a></strong></p><p>This concern also includes information that might be subject to disclosure regulations, such as HIPAA for health information or <a href="https://www.zdnet.com/article/gdpr-an-executive-guide-to-what-you-need-to-know/" rel="noopener noreferrer" target="_blank">GDPR for personal data for folks operating in the EU</a>.</p><p>Regardless of what the AI companies tell you, it's best to simply assume that everything you feed into an AI is now grist for the model-training mill. Anything you feed in could later wind up in a response to somebody's prompt, somewhere else.</p><p></p><h2> 2. Reviewing or writing contracts </h2><p>Contracts are designed to be detailed and specific agreements on how two parties will interact. They are considered governing documents, which means that writing a bad contract is like writing bad code. <em>Baaad</em> things will happen. </p><p>Do not ask AIs for help with contracts. They will make errors and omissions. They will make stuff up. Worse, they will do so while sounding authoritative, so you're more likely to use their advice. </p><p></p><p><strong>Also: <a href="https://www.zdnet.com/article/you-can-use-googles-math-olympiad-winning-deep-think-ai-model-now-for-a-price/" rel="noopener noreferrer" target="_blank">You can use Google's Math Olympiad-winning Deep Think AI model now - for a price</a></strong></p><p>Also, the terms of a contract are often governed by the contract. In other words, many contracts say that what's actually in the contract is confidential, and that if you share the particulars of your agreement with any outside party, there will be dire consequences. Sharing with an AI, as discussed above, is like publishing on the front page of a blog.</p><p></p><p>Let me be blunt. If you let an AI work on a contract and it makes a mistake, you (not it) will be paying the price for a long, long time. </p><h2> 3. Using an AI for legal advice </h2><p>You know the trope where what you share with your lawyer is protected information and can't be used against you? Yeah, your friendly neighborhood AI is not your lawyer. </p><p>As <a href="https://cc.zdnet.com/v1/otc/00hQi47eqnEWQ6T9d4QLBUc?element=BODY&amp;element_label=reported+in+Futurism&amp;module=LINK&amp;object_type=text-link&amp;object_uuid=7a1d4360-4c28-4193-ab61-ecd79f0faffa&amp;position=1&amp;template=article&amp;track_code=__COM_CLICK_ID__&amp;url=https%3A%2F%2Ffuturism.com%2Fchatgpt-legal-questions-court&amp;view_instance_uuid=53a65cd4-222c-4125-8bec-6988b4f83974" rel="noopener nofollow sponsored" target="_blank">reported in Futurism</a>, OpenAI CEO (and <a href="https://www.zdnet.com/article/what-is-chatgpt-how-the-worlds-most-popular-ai-chatbot-can-benefit-you/" rel="noopener noreferrer" target="_blank">ChatGPT</a>'s principal cheerleader) Sam Altman told podcaster Theo Von that there is no legal confidentiality when using ChatGPT for your legal concerns. </p><p></p><p><strong>Also: <a href="https://www.zdnet.com/article/even-openai-ceo-sam-altman-thinks-you-shouldnt-trust-ai-for-therapy/" rel="noopener noreferrer" target="_blank">Even OpenAI CEO Sam Altman thinks you shouldn't trust AI for therapy</a></strong></p><p>Earlier, I discussed how AI companies might use your data for training and embed that data in prompt responses. However, Altman took this assertion up a notch. He suggested OpenAI is obligated to share your conversations with ChatGPT if they are subpoenaed by a court. </p><p><a href="https://cc.zdnet.com/v1/otc/00hQi47eqnEWQ6T9d4QLBUc?element=BODY&amp;element_label=Jessee+Bundy&amp;module=LINK&amp;object_type=text-link&amp;object_uuid=7a1d4360-4c28-4193-ab61-ecd79f0faffa&amp;position=1&amp;template=article&amp;track_code=__COM_CLICK_ID__&amp;url=https%3A%2F%2Fwww.linkedin.com%2Fin%2Fjesseebundy%2F&amp;view_instance_uuid=53a65cd4-222c-4125-8bec-6988b4f83974" rel="noopener nofollow sponsored" target="_blank">Jessee Bundy</a>, a Knoxville-based attorney, amplified Altman's statement <a href="https://cc.zdnet.com/v1/otc/00hQi47eqnEWQ6T9d4QLBUc?element=BODY&amp;element_label=in+a+tweet&amp;module=LINK&amp;object_type=text-link&amp;object_uuid=7a1d4360-4c28-4193-ab61-ecd79f0faffa&amp;position=1&amp;template=article&amp;track_code=__COM_CLICK_ID__&amp;url=https%3A%2F%2Fx.com%2Fcreativeatlaw%2Fstatus%2F1949248338001334598&amp;view_instance_uuid=53a65cd4-222c-4125-8bec-6988b4f83974" rel="noopener nofollow sponsored" target="_blank">in a tweet</a>: "There's no legal privilege when you use ChatGPT. So if you're pasting in contracts, asking legal questions, or asking it for strategy, you're not getting legal advice. You're generating discoverable evidence. No attorney/client privilege. No confidentiality. No ethical duty. No one to protect you."</p><p></p><p>She summed up her observations with a particularly damning statement: "It might feel private, safe, and convenient. But lawyers are bound to protect you. ChatGPT isn't, and can be used against you." </p><h2> 4. Using an AI for health or financial advice </h2><p>While we're on the topic of guidance, let's hit two other categories where highly trained, licensed, and regulated professionals are available to provide advice: healthcare and finance. </p><p>Look, it's probably fine to ask ChatGPT to explain a medical or financial concept to you as if you were a five-year-old. But when it comes time to ask for real advice that you plan on considering as you make major decisions, just don't. </p><p></p><p>Let's step away from the liability risk issues and focus on common sense. First, if you're using something like ChatGPT for real advice, you have to know what to ask. If you're not trained in these professions, you might not know. </p><p><strong>Also: <a href="https://www.zdnet.com/article/metas-personal-superintelligence-dream-leaves-much-to-the-imagination/" rel="noopener noreferrer" target="_blank">What Zuckerberg's 'personal superintelligence' sales pitch leaves out</a></strong></p><p>Second, ChatGPT and other chatbots can be <a href="https://www.zdnet.com/article/how-chatgpt-actually-works-and-why-its-been-so-game-changing/" rel="noopener noreferrer" target="_blank">spectacularly, overwhelmingly, and almost unbelievably wrong</a>. They misconstrue questions, fabricate answers, conflate concepts, and generally provide questionable advice.</p><p></p><p>Ask yourself, are you willing to bet your life or your financial future on something that a people-pleasing robot made up because it thought that's what you wanted to hear? </p><h2> 5. Presenting AI-generated work as your own </h2><p>When you ask a chatbot to write something for you, do you claim it as your own? Some folks have told me that because they wrote the prompts, the resulting output is a result of their creativity. </p><p><strong>Also: <a href="https://www.zdnet.com/article/i-found-5-ai-content-detectors-that-can-correctly-identify-ai-text-100-of-the-time/" rel="noopener noreferrer" target="_blank">I found 5 AI content detectors that can correctly identify AI text 100% of the time</a></strong></p><p></p><p>Yeah? Not so much. Webster's defines "plagiarize" as "to steal and pass off (the ideas or words of another) as one's own," and to "use (another's production) without crediting the source." The dictionary also defines plagiarize as "to commit literary theft: present as new and original an idea or product derived from an existing source." </p><p>Does that not sound like what a chatbot does? It sure does "present as new and original an idea…derived from an existing source." Chatbots are trained on existing sources. They then parrot back those sources after adding a bit of spin.</p><p></p><p>Let's be clear. Using an AI and saying its output is yours could cost you your job.</p><h2> 6. Talking to customers without monitoring the chatter </h2><p>The other day, I had a technical question about <a href="https://www.zdnet.com/article/why-synologys-new-nas-drive-support-policy-isnt-as-bad-as-i-first-thought/" rel="noopener noreferrer" target="_blank">my Synology server</a>. I filed a support ticket after hours. A bit later, I got an email response from a self-identified support AI. The cool thing was that the answer was complete and just what I needed, so I didn't have to escalate my ticket to a human helper. </p><p><strong>Also: <a href="https://www.zdnet.com/article/is-ai-overhyped-or-underhyped-6-tips-to-separate-fact-from-fiction/" rel="noopener noreferrer" target="_blank">Is AI overhyped or underhyped? 6 tips to separate fact from fiction</a></strong></p><p></p><p>But not all AI interactions with customers go that well. Even a year and a half later, I'm still chuckling about <a href="https://www.zdnet.com/article/20-things-to-consider-before-rolling-out-an-ai-chatbot-to-customers/" rel="noopener noreferrer" target="_blank">the Chevy dealer chatbot</a> that offered a $55,000 Chevy Tahoe truck to a customer for a buck. </p><p>It's perfectly fine to provide a trained chatbot as one support option to customers. But don't assume it's always going to be right. Ensure customers have the option to talk with a human. And monitor the AI-enabled process. Otherwise, you could be giving away $1 trucks, too. </p><p></p><h2> 7. Making final hiring and firing solutions </h2><p>According to <a href="https://www.zdnet.com/article/60-of-managers-use-ai-to-make-decisions-now-including-whom-to-promote-and-fire-does-yours/" rel="noopener noreferrer" target="_blank">a survey by resume-making app Resume Builder</a>, a majority of managers are using AI "to determine raises (78%), promotions (77%), layoffs (66%), and even terminations (64%)." </p><p>"Why are you firing me?" </p><p>"It's not my fault. The AI made me do it." </p><p>Yeah, that. Worse, apparently at least 20% of managers, most of whom haven't been trained in the rights and wrongs of AI usage, are using AIs to make final employment decisions without even bothering to oversee the AI.</p><p></p><p><strong>Also: </strong><a href="https://www.zdnet.com/article/the-linux-foundation-finds-organizations-prioritize-upskilling-and-open-source-for-tomorrows-ai-driven-job-market/" rel="noopener noreferrer" target="_blank"><strong>Open-source skills can save your career when AI comes knocking</strong></a></p><p>But here's the rub. Jobs are often governed by labor laws. Despite the current anti-DEI push coming from Washington, bias can still lead to discrimination lawsuits. Even if you haven't technically done anything wrong, defending against a lawsuit can be expensive.</p><p>If you cause your company to be on the receiving end of a lawsuit because you couldn't be bothered to be human enough to double-check why your AI was canning Janice in accounting, you'll be the next one being handed a pink slip. Don't do it. Just say no.</p><p></p><h2> 8. Responding to journalists or media inquiries </h2><p>I'm going to tell you a little secret. Journalists and writers do not exist solely to promote your company. We'd like to help, certainly. It feels good knowing we're helping folks grow their businesses. But, and you'll need to sit down for this news, there are <em>other companies</em>. </p><p>We are also busy. I get thousands of emails every day. Hundreds of them are about the newest and by far most innovative AI company ever. Many of those pitches are AI-generated because the PR folks couldn't be bothered to take the time to focus their pitch. Some of them are so bad that I can't even tell what the PRs are trying to hawk. </p><p></p><p>But then, there's the other side. Sometimes, I'll reach out to a company, willing to use my most valuable resource -- time -- on their behalf. When I get back a response that's AI-driven, I'll either move on to the next company (or mock them on social media). </p><p><strong>Also: </strong><a href="https://www.zdnet.com/article/5-entry-level-tech-jobs-ai-is-already-augmenting-according-to-amazon/" rel="noopener noreferrer" target="_blank"><strong>5 entry-level tech jobs AI is already augmenting, according to Amazon</strong></a></p><p>Some of those AI-driven answers are really, really inappropriate. However, because the AI is representing the company instead of, you know, maybe a thinking human, an opportunity is lost. </p><p></p><p>Keep in mind that I don't like publishing things that will cost someone their job. But other writers are not necessarily similarly inclined. A properly run business will not only use a human to respond to the press, but will also limit the humans allowed to represent the company to those properly experienced in what to say. </p><p>Or go ahead and cut corners. I always need fun fodder for <a href="https://cc.zdnet.com/v1/otc/00hQi47eqnEWQ6T9d4QLBUc?element=BODY&amp;element_label=my+Facebook+feed&amp;module=LINK&amp;object_type=text-link&amp;object_uuid=7a1d4360-4c28-4193-ab61-ecd79f0faffa&amp;position=1&amp;template=article&amp;track_code=__COM_CLICK_ID__&amp;url=https%3A%2F%2Fwww.facebook.com%2Fdavidgewirtz&amp;view_instance_uuid=53a65cd4-222c-4125-8bec-6988b4f83974" rel="noopener nofollow sponsored" target="_blank">my Facebook feed</a>. </p><h2> 9. Using AI for coding without a backup </h2><p>Earlier, I wrote "<a href="https://www.zdnet.com/article/9-programming-tasks-you-shouldnt-hand-off-to-ai-and-why/" rel="noopener noreferrer" target="_blank">9 programming tasks you shouldn't hand off to AI</a>," which detailed programming tasks you should avoid passing along to an AI. I've long been nervous about ceding too much responsibility to an AI, and quite concerned about managing codebase maintenance. </p><p></p><p>But I didn't really understand how far stupid could go when it came to delegating coding responsibility to the AI. I mean, yes, I know AIs can be stupid. And I sure know humans can be stupid. But when AIs and humans work in tandem to advance the cause of their stupidity together, the results can be truly awe-inspiring. </p><p>In "<a href="https://www.zdnet.com/article/bad-vibes-how-an-ai-agent-coded-its-way-to-disaster/" rel="noopener noreferrer" target="_blank">Bad vibes: How an AI agent coded its way to disaster</a>," my ZDNET colleague Steven Vaughan-Nichols wrote about a developer who happily <a href="https://www.zdnet.com/article/what-is-ai-vibe-coding-its-all-the-rage-but-its-not-for-everyone-heres-why/" rel="noopener noreferrer" target="_blank">vibe-coded</a> himself to an almost-complete piece of software. First, the AI hard-coded lies about how unit tests performed. Then the AI deleted his entire codebase.</p><p></p><p>It's not necessarily wrong to use AI to help you code. But if you're using a tool that can't be backed up, or you don't bother to back up your code first, you're simply doing your best to earn a <a href="https://cc.zdnet.com/v1/otc/00hQi47eqnEWQ6T9d4QLBUc?element=BODY&amp;element_label=digital+Darwin+award&amp;module=LINK&amp;object_type=text-link&amp;object_uuid=7a1d4360-4c28-4193-ab61-ecd79f0faffa&amp;position=1&amp;template=article&amp;track_code=__COM_CLICK_ID__&amp;url=https%3A%2F%2Fdarwinawards.com%2F&amp;view_instance_uuid=53a65cd4-222c-4125-8bec-6988b4f83974" rel="noopener nofollow sponsored" target="_blank">digital Darwin award</a>. </p><h2> Bonus: Other examples of what not to do </h2><p>Here's a lightning round of boneheaded moves using AI. They're just too good (and by good, I mean bad) not to recount:</p><ul><li><strong>Letting a chatbot manage job applicant data:</strong> Remember how we told you not to use an AI for hiring and firing? McDonald's uses a chatbot to screen applicants. Apparently, the chatbot <a href="https://cc.zdnet.com/v1/otc/00hQi47eqnEWQ6T9d4QLBUc?element=BODY&amp;element_label=exposed+millions+of+applicants%27+personal+information&amp;module=LINK&amp;object_type=text-link&amp;object_uuid=7a1d4360-4c28-4193-ab61-ecd79f0faffa&amp;position=1&amp;template=article&amp;track_code=__COM_CLICK_ID__&amp;url=https%3A%2F%2Fwww.wired.com%2Fstory%2Fmcdonalds-ai-hiring-chat-bot-paradoxai%2F&amp;view_instance_uuid=53a65cd4-222c-4125-8bec-6988b4f83974" rel="noopener nofollow sponsored" target="_blank">exposed millions of applicants' personal information</a> to a hacker who used the password 123456.</li><li><strong>Replacing support staff with an AI, and gloating:</strong> A CEO of e-commerce platform Dukaan terminated 90% of his support staff and replaced them with an AI. Then he <a href="https://cc.zdnet.com/v1/otc/00hQi47eqnEWQ6T9d4QLBUc?element=BODY&amp;element_label=bragged+about+it&amp;module=LINK&amp;object_type=text-link&amp;object_uuid=7a1d4360-4c28-4193-ab61-ecd79f0faffa&amp;position=1&amp;template=article&amp;track_code=__COM_CLICK_ID__&amp;url=https%3A%2F%2Fx.com%2Fsuumitshah%2Fstatus%2F1678460567000850450&amp;view_instance_uuid=53a65cd4-222c-4125-8bec-6988b4f83974" rel="noopener nofollow sponsored" target="_blank">bragged about it</a>. On Twitter/X. The public response was less than positive. Way less.</li><li><strong>Produce a reading list consisting of all fake titles:</strong> The <a href="https://cc.zdnet.com/v1/otc/00hQi47eqnEWQ6T9d4QLBUc?element=BODY&amp;element_label=Chicago+Sun-Times&amp;module=LINK&amp;object_type=text-link&amp;object_uuid=7a1d4360-4c28-4193-ab61-ecd79f0faffa&amp;position=1&amp;template=article&amp;track_code=__COM_CLICK_ID__&amp;url=https%3A%2F%2Fchicago.suntimes.com%2Fnews%2F2025%2F05%2F20%2Fsyndicated-content-sunday-print-sun-times-ai-misinformation&amp;view_instance_uuid=53a65cd4-222c-4125-8bec-6988b4f83974" rel="noopener nofollow sponsored" target="_blank">Chicago Sun-Times</a>, normally a very well-respected paper, published a summer reading list generated by an AI. The gotcha? None of the books were real.</li><li><strong>Suggesting terminated employees turn to a chatbot for comfort:</strong> An Xbox producer (yes, that's Microsoft) <a href="https://cc.zdnet.com/v1/otc/00hQi47eqnEWQ6T9d4QLBUc?element=BODY&amp;element_label=suggested&amp;module=LINK&amp;object_type=text-link&amp;object_uuid=7a1d4360-4c28-4193-ab61-ecd79f0faffa&amp;position=1&amp;template=article&amp;track_code=__COM_CLICK_ID__&amp;url=https%3A%2F%2Fwww.bbc.com%2Fnews%2Farticles%2Fckglzxy389zo&amp;view_instance_uuid=53a65cd4-222c-4125-8bec-6988b4f83974" rel="noopener nofollow sponsored" target="_blank">suggested</a> that ChatGPT or Copilot could "help reduce the emotional and cognitive load that comes with job loss" after Microsoft terminated 9,000 employees. Achievement unlocked.</li></ul><p>What about you? Have you seen an AI go off the rails at work? Have you ever been tempted to delegate a task to a chatbot that, in hindsight, probably needed a human touch? Do you trust AI to handle sensitive data, communicate with customers, or make decisions that affect people's lives? Where do you draw the line in your work? Let us know in the comments below. </p><p></p><hr/><p><em>You can follow my day-to-day project updates on social media. Be sure to subscribe to <a href="https://cc.zdnet.com/v1/otc/00hQi47eqnEWQ6T9d4QLBUc?element=BODY&amp;element_label=my+weekly+update+newsletter&amp;module=LINK&amp;object_type=text-link&amp;object_uuid=7a1d4360-4c28-4193-ab61-ecd79f0faffa&amp;position=1&amp;template=article&amp;track_code=__COM_CLICK_ID__&amp;url=https%3A%2F%2Fadvancedgeekery.substack.com%2F&amp;view_instance_uuid=53a65cd4-222c-4125-8bec-6988b4f83974" rel="noopener nofollow sponsored" target="_blank">my weekly update newsletter</a>, and follow me on Twitter/X at <a href="https://cc.zdnet.com/v1/otc/00hQi47eqnEWQ6T9d4QLBUc?element=BODY&amp;element_label=%40DavidGewirtz&amp;module=LINK&amp;object_type=text-link&amp;object_uuid=7a1d4360-4c28-4193-ab61-ecd79f0faffa&amp;position=1&amp;template=article&amp;track_code=__COM_CLICK_ID__&amp;url=https%3A%2F%2Ftwitter.com%2Fdavidgewirtz&amp;view_instance_uuid=53a65cd4-222c-4125-8bec-6988b4f83974" rel="noopener nofollow sponsored" target="_blank">@DavidGewirtz</a>, on Facebook at <a href="https://cc.zdnet.com/v1/otc/00hQi47eqnEWQ6T9d4QLBUc?element=BODY&amp;element_label=Facebook.com%2FDavidGewirtz&amp;module=LINK&amp;object_type=text-link&amp;object_uuid=7a1d4360-4c28-4193-ab61-ecd79f0faffa&amp;position=1&amp;template=article&amp;track_code=__COM_CLICK_ID__&amp;url=https%3A%2F%2Fwww.facebook.com%2Fdavidgewirtz&amp;view_instance_uuid=53a65cd4-222c-4125-8bec-6988b4f83974" rel="noopener nofollow sponsored" target="_blank">Facebook.com/DavidGewirtz</a>, on Instagram at <a href="https://cc.zdnet.com/v1/otc/00hQi47eqnEWQ6T9d4QLBUc?element=BODY&amp;element_label=Instagram.com%2FDavidGewirtz&amp;module=LINK&amp;object_type=text-link&amp;object_uuid=7a1d4360-4c28-4193-ab61-ecd79f0faffa&amp;position=1&amp;template=article&amp;track_code=__COM_CLICK_ID__&amp;url=https%3A%2F%2Fwww.instagram.com%2FDavidGewirtz%2F&amp;view_instance_uuid=53a65cd4-222c-4125-8bec-6988b4f83974" rel="noopener nofollow sponsored" target="_blank">Instagram.com/DavidGewirtz</a>, on Bluesky at <a href="https://cc.zdnet.com/v1/otc/00hQi47eqnEWQ6T9d4QLBUc?element=BODY&amp;element_label=%40DavidGewirtz.com&amp;module=LINK&amp;object_type=text-link&amp;object_uuid=7a1d4360-4c28-4193-ab61-ecd79f0faffa&amp;position=1&amp;template=article&amp;track_code=__COM_CLICK_ID__&amp;url=https%3A%2F%2Fbsky.app%2Fprofile%2Fdavidgewirtz.com&amp;view_instance_uuid=53a65cd4-222c-4125-8bec-6988b4f83974" rel="noopener nofollow sponsored" target="_blank">@DavidGewirtz.com</a>, and on YouTube at <a href="https://cc.zdnet.com/v1/otc/00hQi47eqnEWQ6T9d4QLBUc?element=BODY&amp;element_label=YouTube.com%2FDavidGewirtzTV&amp;module=LINK&amp;object_type=text-link&amp;object_uuid=7a1d4360-4c28-4193-ab61-ecd79f0faffa&amp;position=1&amp;template=article&amp;track_code=__COM_CLICK_ID__&amp;url=https%3A%2F%2Fwww.youtube.com%2Fuser%2FDavidGewirtzTV&amp;view_instance_uuid=53a65cd4-222c-4125-8bec-6988b4f83974" rel="noopener nofollow sponsored" target="_blank">YouTube.com/DavidGewirtzTV</a>.</em></p>

content_text:
zokara/iStock/Getty Images Plus via Getty Images

ZDNET's key takeaways

Sometimes an AI can cause you or your company irreparable harm.

Sharing confidential data with an AI could have legal consequences.

Don't let an AI talk to customers without supervision.

A few weeks ago, I shared with you "

9 programming tasks you shouldn't hand off to AI - and why.

" It's full of well-reasoned suggestions and recommendations for how to avoid having an AI produce code that could ruin your whole day.

Then, my editor and I got talking, and we realized the whole idea of "when not to use an AI" could apply to work in general. In this article, I present to you nine things you shouldn't use AI for while at work. This is far from a comprehensive list, but it should make you think.

Also:

This one feature could make GPT-5 a true game changer (if OpenAI gets it right)

"Always keep in mind that AI isn't going to read you your Miranda Rights, wrap your personal information in legal protections like HIPAA, or hesitate to disclose your secrets," said LinkedIn Learning AI instructor

Pam Baker

, the bestselling author of

ChatGPT For Dummies

and

Generative AI For Dummies

.

"That goes double for work AI, which is monitored closely by your employer. Whatever you do or tell AI can and likely will be used against you at some point."

To keep things interesting, read on to the end. There, I share some fun and terrifying stories about how using AI at work can go terribly, horribly, and amusingly wrong.

Without further ado, here are nine things you shouldn't do with AI at work.

1. Handling confidential or sensitive data

This is an easy one. Every time you give the AI some information, ask yourself how you would feel if it were posted to the company's public blog or wound up on the front page of your industry's trade journal.

Also:

The best AI for coding in 2025 (and what not to use)

This concern also includes information that might be subject to disclosure regulations, such as HIPAA for health information or

GDPR for personal data for folks operating in the EU

.

Regardless of what the AI companies tell you, it's best to simply assume that everything you feed into an AI is now grist for the model-training mill. Anything you feed in could later wind up in a response to somebody's prompt, somewhere else.

2. Reviewing or writing contracts

Contracts are designed to be detailed and specific agreements on how two parties will interact. They are considered governing documents, which means that writing a bad contract is like writing bad code.

Baaad

things will happen.

Do not ask AIs for help with contracts. They will make errors and omissions. They will make stuff up. Worse, they will do so while sounding authoritative, so you're more likely to use their advice.

Also:

You can use Google's Math Olympiad-winning Deep Think AI model now - for a price

Also, the terms of a contract are often governed by the contract. In other words, many contracts say that what's actually in the contract is confidential, and that if you share the particulars of your agreement with any outside party, there will be dire consequences. Sharing with an AI, as discussed above, is like publishing on the front page of a blog.

Let me be blunt. If you let an AI work on a contract and it makes a mistake, you (not it) will be paying the price for a long, long time.

3. Using an AI for legal advice

You know the trope where what you share with your lawyer is protected information and can't be used against you? Yeah, your friendly neighborhood AI is not your lawyer.

As

reported in Futurism

, OpenAI CEO (and

ChatGPT

's principal cheerleader) Sam Altman told podcaster Theo Von that there is no legal confidentiality when using ChatGPT for your legal concerns.

Also:

Even OpenAI CEO Sam Altman thinks you shouldn't trust AI for therapy

Earlier, I discussed how AI companies might use your data for training and embed that data in prompt responses. However, Altman took this assertion up a notch. He suggested OpenAI is obligated to share your conversations with ChatGPT if they are subpoenaed by a court.

Jessee Bundy

, a Knoxville-based attorney, amplified Altman's statement

in a tweet

: "There's no legal privilege when you use ChatGPT. So if you're pasting in contracts, asking legal questions, or asking it for strategy, you're not getting legal advice. You're generating discoverable evidence. No attorney/client privilege. No confidentiality. No ethical duty. No one to protect you."

She summed up her observations with a particularly damning statement: "It might feel private, safe, and convenient. But lawyers are bound to protect you. ChatGPT isn't, and can be used against you."

4. Using an AI for health or financial advice

While we're on the topic of guidance, let's hit two other categories where highly trained, licensed, and regulated professionals are available to provide advice: healthcare and finance.

Look, it's probably fine to ask ChatGPT to explain a medical or financial concept to you as if you were a five-year-old. But when it comes time to ask for real advice that you plan on considering as you make major decisions, just don't.

Let's step away from the liability risk issues and focus on common sense. First, if you're using something like ChatGPT for real advice, you have to know what to ask. If you're not trained in these professions, you might not know.

Also:

What Zuckerberg's 'personal superintelligence' sales pitch leaves out

Second, ChatGPT and other chatbots can be

spectacularly, overwhelmingly, and almost unbelievably wrong

. They misconstrue questions, fabricate answers, conflate concepts, and generally provide questionable advice.

Ask yourself, are you willing to bet your life or your financial future on something that a people-pleasing robot made up because it thought that's what you wanted to hear?

5. Presenting AI-generated work as your own

When you ask a chatbot to write something for you, do you claim it as your own? Some folks have told me that because they wrote the prompts, the resulting output is a result of their creativity.

Also:

I found 5 AI content detectors that can correctly identify AI text 100% of the time

Yeah? Not so much. Webster's defines "plagiarize" as "to steal and pass off (the ideas or words of another) as one's own," and to "use (another's production) without crediting the source." The dictionary also defines plagiarize as "to commit literary theft: present as new and original an idea or product derived from an existing source."

Does that not sound like what a chatbot does? It sure does "present as new and original an idea…derived from an existing source." Chatbots are trained on existing sources. They then parrot back those sources after adding a bit of spin.

Let's be clear. Using an AI and saying its output is yours could cost you your job.

6. Talking to customers without monitoring the chatter

The other day, I had a technical question about

my Synology server

. I filed a support ticket after hours. A bit later, I got an email response from a self-identified support AI. The cool thing was that the answer was complete and just what I needed, so I didn't have to escalate my ticket to a human helper.

Also:

Is AI overhyped or underhyped? 6 tips to separate fact from fiction

But not all AI interactions with customers go that well. Even a year and a half later, I'm still chuckling about

the Chevy dealer chatbot

that offered a $55,000 Chevy Tahoe truck to a customer for a buck.

It's perfectly fine to provide a trained chatbot as one support option to customers. But don't assume it's always going to be right. Ensure customers have the option to talk with a human. And monitor the AI-enabled process. Otherwise, you could be giving away $1 trucks, too.

7. Making final hiring and firing solutions

According to

a survey by resume-making app Resume Builder

, a majority of managers are using AI "to determine raises (78%), promotions (77%), layoffs (66%), and even terminations (64%)."

"Why are you firing me?"

"It's not my fault. The AI made me do it."

Yeah, that. Worse, apparently at least 20% of managers, most of whom haven't been trained in the rights and wrongs of AI usage, are using AIs to make final employment decisions without even bothering to oversee the AI.

Also:

Open-source skills can save your career when AI comes knocking

But here's the rub. Jobs are often governed by labor laws. Despite the current anti-DEI push coming from Washington, bias can still lead to discrimination lawsuits. Even if you haven't technically done anything wrong, defending against a lawsuit can be expensive.

If you cause your company to be on the receiving end of a lawsuit because you couldn't be bothered to be human enough to double-check why your AI was canning Janice in accounting, you'll be the next one being handed a pink slip. Don't do it. Just say no.

8. Responding to journalists or media inquiries

I'm going to tell you a little secret. Journalists and writers do not exist solely to promote your company. We'd like to help, certainly. It feels good knowing we're helping folks grow their businesses. But, and you'll need to sit down for this news, there are

other companies

.

We are also busy. I get thousands of emails every day. Hundreds of them are about the newest and by far most innovative AI company ever. Many of those pitches are AI-generated because the PR folks couldn't be bothered to take the time to focus their pitch. Some of them are so bad that I can't even tell what the PRs are trying to hawk.

But then, there's the other side. Sometimes, I'll reach out to a company, willing to use my most valuable resource -- time -- on their behalf. When I get back a response that's AI-driven, I'll either move on to the next company (or mock them on social media).

Also:

5 entry-level tech jobs AI is already augmenting, according to Amazon

Some of those AI-driven answers are really, really inappropriate. However, because the AI is representing the company instead of, you know, maybe a thinking human, an opportunity is lost.

Keep in mind that I don't like publishing things that will cost someone their job. But other writers are not necessarily similarly inclined. A properly run business will not only use a human to respond to the press, but will also limit the humans allowed to represent the company to those properly experienced in what to say.

Or go ahead and cut corners. I always need fun fodder for

my Facebook feed

.

9. Using AI for coding without a backup

Earlier, I wrote "

9 programming tasks you shouldn't hand off to AI

," which detailed programming tasks you should avoid passing along to an AI. I've long been nervous about ceding too much responsibility to an AI, and quite concerned about managing codebase maintenance.

But I didn't really understand how far stupid could go when it came to delegating coding responsibility to the AI. I mean, yes, I know AIs can be stupid. And I sure know humans can be stupid. But when AIs and humans work in tandem to advance the cause of their stupidity together, the results can be truly awe-inspiring.

In "

Bad vibes: How an AI agent coded its way to disaster

," my ZDNET colleague Steven Vaughan-Nichols wrote about a developer who happily

vibe-coded

himself to an almost-complete piece of software. First, the AI hard-coded lies about how unit tests performed. Then the AI deleted his entire codebase.

It's not necessarily wrong to use AI to help you code. But if you're using a tool that can't be backed up, or you don't bother to back up your code first, you're simply doing your best to earn a

digital Darwin award

.

Bonus: Other examples of what not to do

Here's a lightning round of boneheaded moves using AI. They're just too good (and by good, I mean bad) not to recount:

Letting a chatbot manage job applicant data:

Remember how we told you not to use an AI for hiring and firing? McDonald's uses a chatbot to screen applicants. Apparently, the chatbot

exposed millions of applicants' personal information

to a hacker who used the password 123456.

Replacing support staff with an AI, and gloating:

A CEO of e-commerce platform Dukaan terminated 90% of his support staff and replaced them with an AI. Then he

bragged about it

. On Twitter/X. The public response was less than positive. Way less.

Produce a reading list consisting of all fake titles:

The

Chicago Sun-Times

, normally a very well-respected paper, published a summer reading list generated by an AI. The gotcha? None of the books were real.

Suggesting terminated employees turn to a chatbot for comfort:

An Xbox producer (yes, that's Microsoft)

suggested

that ChatGPT or Copilot could "help reduce the emotional and cognitive load that comes with job loss" after Microsoft terminated 9,000 employees. Achievement unlocked.

What about you? Have you seen an AI go off the rails at work? Have you ever been tempted to delegate a task to a chatbot that, in hindsight, probably needed a human touch? Do you trust AI to handle sensitive data, communicate with customers, or make decisions that affect people's lives? Where do you draw the line in your work? Let us know in the comments below.

You can follow my day-to-day project updates on social media. Be sure to subscribe to

my weekly update newsletter

, and follow me on Twitter/X at

@DavidGewirtz

, on Facebook at

Facebook.com/DavidGewirtz

, on Instagram at

Instagram.com/DavidGewirtz

, on Bluesky at

@DavidGewirtz.com

, and on YouTube at

YouTube.com/DavidGewirtzTV

.

thumb_url:
https://www.zdnet.com/a/img/resize/1f2cf68e1120d59dd66d54062a4b4844ee244af7/2025/04/30/7dd9b7b4-e6bc-4a3e-9aab-722a96a07281/brokenplasticgettyimages-988065734.jpg?auto=webp&fit=crop&height=675&width=1200


url: https://bytevagabond.com/post/how-to-build-enterprise-ai-rag/
added_at: 2025-08-28 02:13:58
thumb_url: https://bytevagabond.com/post/how-to-build-enterprise-ai-rag/images/rag_ai_architecture.png
title:
Building Enterprise AI: Hard-Won Lessons from 1200+ Hours of RAG Development | ByteVagabond – Digital Tinkering & Real-World Adventures

content_html:
<p>The following blog post is an opinionated result of hundreds of hours of intense research, implementation and evaluation while developing an enterprise AI chat system (source code can be found </p><a href="https://codeberg.org/maxcodefaster/teammind" rel="noopener noreferrer" target="_blank">here</a><p>). While I do not remember every source and arxiv paper, most will be linked directly in the text. Now please fasten your seatbelts and enjoy your deep dive into modern AI RAG architecture.</p><h2>AI Apps Are Really Just RAG</h2><p>AI is being crammed into every application. This guide will help developers understand what you actually need to build an AI app. Without debating whether the current state of cramming AI features into every existing app makes sense, we first need to understand what is meant by a text-based AI app.</p><p>In almost every commercial AI app, there’s no training of custom AI models. Instead, they use base models from big providers like OpenAI, Google, Anthropic, xAI, or open-source alternatives like Llama or Mistral. This is because training a model is highly resource-intensive, and state-of-the-art models have become commoditized - the gaps between models regarding intelligence, capabilities, and price are closing rapidly.</p><p>So if developing AI apps doesn’t typically involve building your own models, what do you need to do as a developer? The keyword is <strong>RAG</strong> (Retrieval Augmented Generation). Essentially, it means feeding base models the correct data and tuning requests to get desired answers. RAG isn’t new - it’s been around in data science for quite some time - but it has become a buzzword in the recent AI craze.</p><p>At its core, RAG involves two steps: <strong>Ingestion</strong> and <strong>Retrieval</strong>. This might seem trivial at first, but as with anything in data science, it’s always garbage in, garbage out. The AI space is evolving drastically - I can’t remember the last time I’ve seen any space evolve this fast. I’ve done the heavy lifting, scanning through dozens of papers and repositories while developing my own clone of enterprise applications like GleanAI, Zive, ChatGPT Enterprise, and Google Agentspace.</p><p>While there’s a lot of hocus pocus from snake oil vendors claiming their one magical solution will deliver enterprise RAG nirvana, the reality is far more nuanced. From my experience building production systems, it’s not any single technique that makes RAG work - it’s the thoughtful combination of proven methods that delivers results. The recommendations below sit on the Pareto frontier, balancing integration difficulty, performance, and cost. Ignore the shiny new papers promising 50% improvements on cherry-picked benchmarks; focus on techniques that actually work in messy enterprise environments with real users asking terrible questions.</p><figure><img alt="Enterprise RAG architecture diagram showing complete pipeline from data sources through ingestion, storage, and retrieval to user response" loading="lazy" referrerpolicy="no-referrer" src="https://bytevagabond.com/post/how-to-build-enterprise-ai-rag/images/rag_ai_architecture.png"/><figcaption><header><b>Enterprise RAG Architecture: A comprehensive view of production-ready RAG systems, from heterogeneous data sources through sophisticated ingestion pipelines to hierarchical retrieval and intelligent response generation. Note the two-stage approach that reduces search scope while maintaining quality.</b></header></figcaption></figure><p>Here’s what you need to know to become an AI developer:</p><h2>Ingestion</h2><h3>Streamlining Your Data</h3><p>You want your AI apps to know about your data, but that data comes in different forms and shapes. Let’s say you have Microsoft SharePoint, Notion, or Confluence spaces. Text generation AI works best when fed cleaned-up text content, so what do you do with all the PDFs, Office docs, or custom spec files?</p><p>You need a pipeline that converts heterogeneous data into a homogeneous format so your retrieval algorithm always finds the most relevant (top K) results. Base models work exceptionally well with markdown content - a format that’s plain but has enough rich text elements and strong hierarchical features. There are different markdown flavors, but the most popular in terms of features is <strong>GFM (GitHub Flavored Markdown)</strong>.</p><p>In my enterprise app, I have several converters to convert most file types (PDFs, Office docs, images) and custom enterprise apps (Notion, Confluence) in a plug-and-play way to markdown. The code includes integrations for <a href="https://codeberg.org/maxcodefaster/teammind/src/branch/main/packages/ai/src/integrations" rel="noopener noreferrer" target="_blank">SharePoint, Notion, Confluence, Jira &amp; Google Drive via OAuth</a>, <a href="https://codeberg.org/maxcodefaster/teammind/src/branch/main/packages/ai/src/file-converters/pdf-to-md-converter.ts" rel="noopener noreferrer" target="_blank">PDF to GFM Markdown via Gemini 2.5 Flash</a>, and <a href="https://codeberg.org/maxcodefaster/teammind/src/branch/main/packages/ai/src/file-converters/gotenberg-converter.ts" rel="noopener noreferrer" target="_blank">Office Files (PowerPoint, Word, etc.) to GFM Markdown via Gotenberg</a>.</p><h3>Chunking</h3><p>Once you have your cleaned data in markdown format, you need to tackle the next challenge: LLMs have limited context window sizes, meaning you typically can’t cram all your data into the API call. While context windows have grown enormous, it still doesn’t make sense to push everything in there for two main reasons:</p><ol><li><strong>Cost factor</strong> - Even with token caching, caching makes reflecting data changes non-viable</li><li><strong>Retrieval performance</strong> - Performance degrades with bigger contexts, leading to more hallucinations</li></ol><p>Therefore, we need to “chunk” our data and optimize it for retrieval to feed the LLM with relevant information. Chunking is a meta-science in itself, with different approaches varying in complexity, cost, and performance implications:</p><ul><li><strong>Fixed Size Chunking</strong> - Splits text into chunks of specified character count, regardless of structure</li><li><strong>Recursive Chunking</strong> - Divides text hierarchically using separators, recursively splitting until reaching desired sizes</li><li><strong>Document-Based Chunking</strong> - Splits content based on document structure (markdown headers, code functions, table rows)</li><li><strong>Semantic Chunking</strong> - Uses embeddings to group semantically related content together</li><li><strong>Agentic Chunking</strong> - Employs LLMs to intelligently determine chunk boundaries</li></ul><p>Moving from top to bottom, we see increasing sophistication and performance, while Document-Based Chunking sits at the Pareto frontier. In my testing, this approach is perfect, especially since our data is already in GFM format.</p><p><strong>The Context Problem</strong></p><p>There’s one significant issue: keeping context. Imagine this scenario: We have text about Berlin and chunk it into different paragraphs, but lose context to the overall topic. When I say “it’s more than 3.85 million…” what is “it” referring to? We lose context in this chunk.</p><figure><img alt="Context loss illustration showing how chunked text loses reference to the broader document topic" loading="lazy" referrerpolicy="no-referrer" src="https://bytevagabond.com/post/how-to-build-enterprise-ai-rag/images/context_illustration.png"/><figcaption><header><b>Context Loss in Traditional Chunking: Individual chunks lose connection to the document's main subject, making 'it's more than 3.85 million' ambiguous without knowing the document discusses Berlin's population [https://jina.ai/news/late-chunking-in-long-context-embedding-models/]</b></header></figcaption></figure><p><strong>The Solution: Context Path Breadcrumbs</strong></p><p>Here’s where we can apply a clever “hack” utilizing our markdown conversion. We create the <strong>abstract syntax tree (AST)</strong> from the markdown document hierarchy and prepend it as breadcrumbs to every chunk.</p><pre><code>// Example: If our paragraph is in "Prehistory of Berlin" (H3) 
// under "History" (H2) in document "Berlin" (H1)
const contextPath = "Berlin &gt; History &gt; Prehistory of Berlin";
const chunkWithContext = `${contextPath}\n\n${chunkContent}`;
</code></pre><p>This isn’t completely solved yet, as our document could live in a folder structure. To overcome this, we apply the same solution at the folder/file level: build a hierarchical path based on folder structure and prepend it to the markdown AST.</p><p>The complete GFM context path chunker with all its logic, including finding optimal chunk size and truncating context paths when they get too long, can be found <a href="https://codeberg.org/maxcodefaster/teammind/src/branch/main/packages/ai/src/embeddings/gfm-context-path-chunker.ts" rel="noopener noreferrer" target="_blank">here</a>. This includes <a href="https://codeberg.org/maxcodefaster/teammind/src/branch/main/packages/ai/src/integrations/shared/hierarchy-manager.ts" rel="noopener noreferrer" target="_blank">incremental updates</a> and hierarchy management.</p><h3>Embeddings</h3><p>So far, we’ve cleaned and split our content while preserving context. We could use a simple text search algorithm to find relevant chunks. If I search for “inhabitants Berlin,” we’d probably get relevant results. But what about searching “inhabitants capital Germany”? With basic text search, we’d get no results.</p><p>That’s where <strong>embeddings</strong> come into play - one of the key concepts of AI RAG data handling.</p><p><strong>What are embeddings?</strong></p><p>Embeddings are numerical representations of text that capture semantic meaning in high-dimensional vector space. Instead of matching exact words, embeddings allow us to find conceptually similar content - so “inhabitants capital Germany” would match chunks about “Berlin population” because the AI understands these concepts are related.</p><p><strong>Choosing the Right Embedding Model</strong></p><p>The quality of your embeddings depends heavily on the model you choose. The <a href="https://huggingface.co/spaces/mteb/leaderboard" rel="noopener noreferrer" target="_blank">MTEB (Massive Text Embedding Benchmark) leaderboard</a> is your go-to resource for comparing embedding models across different tasks. It evaluates models across 8 different task types including retrieval, classification, clustering, and semantic textual similarity.</p><p><strong>The Chunk Size Dilemma</strong></p><p>Chunk size significantly impacts retrieval quality - include too much and the vector loses specificity, include too little and you lose context. There’s no “one size fits all” solution. Most developers start with chunk sizes between 100-1000 tokens, but the <a href="https://jina.ai/news/on-the-size-bias-of-text-embeddings-and-its-impact-in-search" rel="noopener noreferrer" target="_blank">recommended maximum</a> is around 512 tokens for optimal performance.</p><p><strong>The Size Bias Problem</strong></p><p>Longer texts generally show higher similarity scores when compared to other embeddings, regardless of actual semantic relevance. This means you can’t use cosine similarity thresholds to determine if matches are actually relevant.</p><p><strong>Late Chunking: The Game Changer</strong></p><p>This is where <a href="https://arxiv.org/abs/2409.04701" rel="noopener noreferrer" target="_blank">Late Chunking</a> comes in as a breakthrough approach. Instead of chunking first then embedding, late chunking flips this process:</p><pre><code>// Traditional approach
const chunks = chunkDocument(fullDocument);
const embeddings = chunks.map(chunk =&gt; embed(chunk)); // Individual embeddings

// Late chunking approach  
const chunks = chunkDocument(fullDocument);
const contextualEmbeddings = embed(chunks); // Embed all chunks together as array
// This creates embeddings that consider inter-chunk relationships and context
</code></pre><p>This preserves broader document context within each chunk since the embeddings were created considering the full text, not just isolated segments. When you split text like “Its more than 3.85 million inhabitants make it the European Union’s most populous city” from a Berlin article, traditional chunking loses the connection to “Berlin” mentioned earlier, but late chunking preserves this context.</p><p>For an implementation of late chunking, take a look <a href="https://codeberg.org/maxcodefaster/teammind/src/branch/main/packages/ai/src/embeddings/jina-embeddings.ts" rel="noopener noreferrer" target="_blank">here</a>.</p><h3>Hybrid and Hierarchical Indexed Database Schema</h3><p>We’ve prepared our data to be stored in the database and later retrieved. AI RAG databases usually need the embedding field type. There are many paid embedding databases, but usually the database you already have is the best database. If you’re working with PostgreSQL, you can use an extension called <a href="https://github.com/pgvector/pgvector" rel="noopener noreferrer" target="_blank"><code>pg_vector</code></a>.</p><p><strong>Dedicated Vector Databases:</strong></p><ul><li><strong><a href="https://www.pinecone.io/" rel="noopener noreferrer" target="_blank">Pinecone</a></strong> - Managed vector database with excellent performance and scaling</li><li><strong><a href="https://weaviate.io/" rel="noopener noreferrer" target="_blank">Weaviate</a></strong> - Open-source vector database with GraphQL APIs</li><li><strong><a href="https://qdrant.tech/" rel="noopener noreferrer" target="_blank">Qdrant</a></strong> - Rust-based vector search engine with filtering capabilities</li><li><strong><a href="https://www.trychroma.com/" rel="noopener noreferrer" target="_blank">Chroma</a></strong> - Developer-friendly open-source embedding database</li><li><strong><a href="https://milvus.io/" rel="noopener noreferrer" target="_blank">Milvus</a></strong> - Cloud-native vector database built for scalable similarity search</li></ul><p><strong>Traditional Databases with Vector Support:</strong></p><ul><li><strong><a href="https://supabase.com/docs/guides/ai" rel="noopener noreferrer" target="_blank">Supabase</a></strong> - Hosted PostgreSQL with built-in pgvector support</li><li><strong><a href="https://redis.io/docs/latest/develop/ai/search-and-query/vectors/" rel="noopener noreferrer" target="_blank">Redis</a></strong> - In-memory database with vector search capabilities</li><li><strong><a href="https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/dense-vector" rel="noopener noreferrer" target="_blank">Elasticsearch</a></strong> - Search engine with dense vector field support</li><li><strong><a href="https://www.mongodb.com/products/platform/atlas-vector-search" rel="noopener noreferrer" target="_blank">MongoDB Atlas</a></strong> - Document database with vector search functionality</li></ul><p>While dedicated vector databases offer optimized performance, I’ve found that <strong>extending your existing database with vector capabilities often provides the best balance of simplicity, cost, and performance</strong> for most enterprise applications. You avoid data synchronization issues, leverage existing backup and security infrastructure, and can perform complex queries that combine vector similarity with traditional filters.</p><p>Enterprises can have vast amounts of documents in different data silos. This can be challenging for a RAG system to always retrieve the most relevant chunks. <strong>Hierarchical Indexing</strong> is the approach of structuring your database schema to have multiple levels for your documents.</p><pre><code>-- Parent level: document metadata and summary
CREATE TABLE documents (
    id bigserial PRIMARY KEY,
    title text NOT NULL,
    summary text NOT NULL,
    summary_embedding vector(1024) NOT NULL,
    -- ... other fields
);

-- Child level: document chunks with foreign key
CREATE TABLE document_chunks (
    id bigserial PRIMARY KEY,
    content text NOT NULL,
    embedding vector(1024) NOT NULL,
    document_id bigint REFERENCES documents(id) ON DELETE CASCADE,
    -- ... other fields
);
</code></pre><p>This helps us retrieve in a first step all possibly relevant documents, then only search over the chunks of those documents. The summary can be generated via a low-cost model such as <a href="https://ai.google.dev/gemini-api/docs/pricing?hl=en#gemini-2.5-flash-lite" rel="noopener noreferrer" target="_blank">Gemini Flash Lite</a>. A prompt for generating such a summary can be found <a href="https://codeberg.org/maxcodefaster/teammind/src/branch/main/packages/ai/src/prompts/doc-summary-prompt.ts" rel="noopener noreferrer" target="_blank">here</a>.</p><p>While embeddings are great, multiple <a href="https://ieeexplore.ieee.org/abstract/document/10707868" rel="noopener noreferrer" target="_blank">papers</a> have proven that a mixture of classic text corpus search such as <a href="https://en.wikipedia.org/wiki/Okapi_BM25" rel="noopener noreferrer" target="_blank">BM25</a> or even simpler <a href="https://en.wikipedia.org/wiki/N-gram" rel="noopener noreferrer" target="_blank">n-gram</a> search yield significantly better results. That’s why on our hierarchical indexed database structure we also create a TokenNgram index.</p><h2>Retrieval</h2><p>We’ve finally reached the last puzzle piece: the retrieval step. We have cleaned and saved our data, and now it’s time to create a sophisticated search algorithm. This is where the magic happens - turning a user’s messy query into precise, relevant results.</p><h3>HyDE - Making Queries Smarter</h3><p>The first technique that’s absolutely game-changing is <strong>HyDE (Hypothetical Document Embeddings)</strong>. Instead of just embedding the user’s query directly, we generate a hypothetical answer to what the user is asking, then use that answer’s embedding for retrieval.</p><p>Why? Because user queries are often short and ambiguous (“quarterly results”), while the documents they’re looking for contain full, detailed content.</p><pre><code>// Traditional approach
const queryEmbedding = await embed("quarterly results");

// HyDE approach
const hydeAnswer = await llm.generate("What would quarterly results contain?");
// -&gt; "The quarterly results include revenue of $2.3M, profit margins of 15%..."
const hydeEmbedding = await embed(hydeAnswer);
</code></pre><p><a href="https://arxiv.org/abs/2212.10496" rel="noopener noreferrer" target="_blank">HyDE was introduced by researchers at CMU</a> and consistently outperforms standard query embedding across most domains. In my implementation, I generate the HyDE response using a lightweight model like Gemini Flash, then embed both the original query and the hypothetical answer.</p><h3>Hierarchical Document Retrieval</h3><p>Here’s where our database design pays off. Instead of throwing embeddings at the wall and hoping for the best, we use a <strong>two-stage hierarchical hybrid search</strong>:</p><pre><code>// Stage 1: Document-level candidate filtering (hybrid search)
const documentCandidates = await client.rpc('match_documents_hierarchical', {
  query_embedding: queryEmbedding,        // For document summary similarity
  hyde_embedding: hydeEmbedding,          // For chunk-level scoring
  query_text: queryText,                  // For full-text search
  // ... other params
});

// The SQL function performs sophisticated two-stage filtering:
// 1. Document filtering with OR condition:
//    - Vector similarity: (1 - (summary_embedding &lt;=&gt; query_embedding)) &gt; threshold
//    - Full-text search: document.content &amp;@~ query_text
// 2. Chunk scoring from candidate documents with hybrid scoring:
//    - Semantic: (1 - (chunk.embedding &lt;=&gt; hyde_embedding))
//    - Keyword: pgroonga_score when chunk.content &amp;@~ query_text
</code></pre><p>The actual implementation is far more nuanced:</p><p><strong>Stage 1 - Document Candidate Selection:</strong></p><ul><li>Uses <strong>query embedding</strong> against <strong>document summary embeddings</strong> for semantic similarity</li><li>Simultaneously runs <strong>full-text search</strong> against the <strong>entire document content</strong> using PGroonga</li><li>Documents qualify if they match EITHER condition (OR logic)</li><li>This dramatically reduces the search space while maintaining high recall</li></ul><p><strong>Stage 2 - Chunk-Level Hybrid Scoring:</strong></p><ul><li>Uses <strong>HyDE embedding</strong> against <strong>individual chunk embeddings</strong> for precise semantic matching</li><li>Runs <strong>full-text search</strong> against <strong>chunk content</strong> for exact keyword matches</li><li>Combines both scores with proper normalization and weighting</li><li>Only processes chunks from documents that passed Stage 1 filtering</li></ul><pre><code>-- Simplified view of the actual SQL implementation
WITH document_candidates AS (
  SELECT d.id AS doc_id
  FROM documents d
  JOIN document_user_access ua ON d.id = ua.document_id
  WHERE ua.user_id = target_user_id
    AND (
      (1 - (d.summary_embedding &lt;=&gt; query_embedding)) &gt; similarity_threshold
      OR d.content &amp;@~ query_text  -- Full-text search on document content
    )
  LIMIT doc_search_limit
),
chunk_scores AS (
  SELECT c.*, d.*,
    -- Semantic score using HyDE embedding
    (1 - (c.embedding &lt;=&gt; hyde_embedding)) AS semantic_score,
    -- Keyword score using PGroonga
    CASE WHEN c.content &amp;@~ query_text 
      THEN pgroonga_score(c.tableoid, c.ctid) 
      ELSE 0 END AS keyword_score
  FROM document_chunks c
  JOIN documents d ON c.document_id = d.id
  WHERE c.document_id IN (SELECT doc_id FROM document_candidates)
)
-- Score normalization and weighting happens here...
</code></pre><p>This approach is inspired by hierarchical passage retrieval methods but optimized for real-world enterprise scenarios. The key insight is that <strong>most queries are looking for information from a small subset of documents</strong>, so we can massively reduce the search space without sacrificing quality. The hybrid approach at both levels ensures we capture both semantic similarity and exact keyword matches, which is crucial for enterprise search where users might search for specific terms, project names, or concepts.</p><h3>Query Expansion and Self-Reflective RAG</h3><p>Raw user queries are often inadequate. “Meeting notes from last week” could mean anything. That’s where <strong>query expansion</strong> comes in. Before hitting the database, I use a lightweight LLM to:</p><ul><li>Expand the query with related terms and synonyms</li><li>Extract time filters (“last week” → specific date range)</li><li>Identify the user’s intent and adjust search weights accordingly</li><li>Generate better search keywords</li></ul><p>But here’s where it gets interesting - after the initial search, we don’t just stop. We implement a <strong>self-reflective RAG approach</strong> where the system evaluates its own search results:</p><pre><code>// The complete orchestrated search pipeline
async function createOrchestratedStream(query: string, userId: string) {
  // 1. Query Expansion with HyDE
  const expansion = await generateObject({
    model: tracedExpansionModel,
    schema: queryExpansionSchema,
    messages: [{ 
      role: 'user', 
      content: `Expand query: ${query}` 
    }]
  });

  // 2. Initial hierarchical search
  const initialResults = await hierarchicalRetriever.hierarchicalSearch(
    expansion.expandedQuery,
    expansion.hydeAnswer, // Key: using HyDE for better retrieval
    {
      target_user_id: userId,
      embedding_weight: 0.7,
      fulltext_weight: 0.3,
      start_date: expansion.timeFilter?.startDate,
      end_date: expansion.timeFilter?.endDate
    }
  );

  // 3. Self-reflective gap analysis
  const evaluation = await generateObject({
    model: tracedGapAnalysisModel,
    schema: gapAnalysisSchema,
    messages: [{
      role: 'user',
      content: gapAnalysisUserPrompt.format({
        query: expansion.expandedQuery,
        initialSearchResults: JSON.stringify(initialResults)
      })
    }]
  });

  // 4. Follow-up searches if gaps identified
  let allResults = [...initialResults];
  if (evaluation.needsAdditionalSearches) {
    const followupPromises = evaluation.informationGaps
      .filter(gap =&gt; gap.importance &gt;= 7) // Only high-importance gaps
      .slice(0, 2) // Max 2 follow-ups
      .map(gap =&gt; 
        hierarchicalRetriever.hierarchicalSearch(
          gap.followupQuery,
          gap.followUpHydeAnswer,
          searchParams
        )
      );
    
    const followupResults = await Promise.all(followupPromises);
    allResults = HierarchicalRetriever.combineResults([
      initialResults, 
      ...followupResults
    ]);
  }

  return allResults;
}
</code></pre><p>This is similar to the <a href="https://arxiv.org/abs/2210.03629" rel="noopener noreferrer" target="_blank">ReAct pattern</a> but applied specifically to information retrieval. The magic number I’ve found is limiting this to 2 follow-up searches maximum - beyond that, you get diminishing returns and increased latency.</p><h3>Hybrid Search That Actually Works</h3><p>Everyone talks about “hybrid search,” but most implementations are inadequate. Here’s what actually works:</p><p><strong>Adaptive weighting based on query intent</strong>: By default, we bias toward semantic search (70% embeddings, 30% keyword) to capture documents with relevant meaning. However, during query expansion, we analyze the query to detect specific search patterns and dynamically adjust weights.</p><pre><code>// Query expansion determines optimal search strategy
const expansion = await expandQuery(query);

const weights = expansion.weights || {
  semanticWeight: expansion.queryType === 'specific_terms' ? 0.3 : 0.7,
  keywordWeight: expansion.queryType === 'specific_terms' ? 0.7 : 0.3
};

const combinedScore = (
  weights.semanticWeight * normalizedSemanticScore +
  weights.keywordWeight * normalizedKeywordScore
);
</code></pre><p><strong>Query type detection</strong>:</p><ul><li><strong>Conceptual queries</strong> (“project status”, “team performance”) → favor semantic search</li><li><strong>Specific terms</strong> (“Project Phoenix”, “JIRA-1234”) → favor keyword search</li><li><strong>Mixed queries</strong> → balanced weighting</li></ul><p><strong>Proper score normalization</strong>: You can’t just add cosine similarity and BM25 / PGroonga scores—they have completely different distributions. I normalize both to 0-1 ranges before combining.</p><p><strong>Two-phase scoring</strong>: Embedding similarity for document-level filtering, then detailed hybrid scoring only for chunks from candidate documents. This keeps it fast while maintaining quality.</p><p>This adaptive approach ensures we get the <strong>conceptual relevance</strong> that embeddings excel at, while not missing <strong>exact matches</strong> for specific terminology that enterprises rely on.</p><p>Enterprise data isn’t just text - it has structure, permissions, timestamps, and context. My retrieval system handles:</p><ul><li><strong>Temporal filtering</strong>: Queries like “recent sales reports” automatically extract time ranges and filter documents by <code>source_updated_at</code></li><li><strong>Permission-aware search</strong>: The <code>document_user_access</code> table ensures users only see results they’re authorized to access, all handled at the database level for performance</li><li><strong>Smart metadata filtering</strong>: Instead of rigid JSON matching, I implemented flexible metadata filters that handle type mismatches gracefully</li><li><strong>Source-aware retrieval</strong>: Different document sources can have different retrieval strategies and weights applied automatically</li></ul><h3>Reranking - The Diminishing Returns</h3><p>After all this sophisticated retrieval, we still have one more potential trick: <strong>reranking</strong>. In earlier versions of my system, I used Jina’s reranking models to take top candidates and reorder them based on deeper semantic understanding.</p><p>Here’s the thing though - <strong>I’ve disabled reranking in the current version</strong>. While academic papers show impressive improvements, in practice with a well-tuned hierarchical search, the quality gains are marginal while the latency cost is significant. The hierarchical approach with proper hybrid scoring and query expansion gets you 90% of the way there, and reranking that extra 5-10% isn’t worth doubling your response time.</p><p>The key insight is that rerankers work best when initial retrieval is poor. But when your retrieval is already sophisticated, you hit diminishing returns fast. That’s the Pareto frontier in action - focus on getting the fundamentals right rather than adding expensive bells and whistles.</p><h3>Performance Optimizations That Matter</h3><p>All of this sounds expensive, but with proper optimizations, it runs fast:</p><ul><li><strong>HNSW vector indexes</strong> for ~95% faster similarity searches compared to IVF indexes</li><li><strong>Materialized CTEs</strong> in PostgreSQL functions to avoid recomputing candidate sets</li><li><strong>Parallel processing</strong> of follow-up searches during self-reflective retrieval</li><li><strong>Smart caching</strong> of embeddings and query expansions</li></ul><p>The entire pipeline - from raw query to final ranked results - typically runs in under 2 seconds for enterprise datasets with millions of documents. Check out the full implementation of the hierarchical retriever <a href="https://codeberg.org/maxcodefaster/teammind/src/branch/main/packages/ai/src/storage/retrieve.ts" rel="noopener noreferrer" target="_blank">here</a> and the query orchestration logic <a href="https://codeberg.org/maxcodefaster/teammind/src/branch/main/apps/web/app/api/chat/route.ts" rel="noopener noreferrer" target="_blank">here</a>.</p><h3>Why This Approach Works</h3><p>The hierarchical approach with self-reflection feels significantly more accurate than basic vector search implementations and performs well compared to what I’ve seen from enterprise search solutions. While I haven’t done formal benchmarking against commercial solutions like Glean or Microsoft Viva, the approach addresses the core problems I’ve observed with simpler RAG implementations.</p><p>The secret sauce isn’t any single technique - it’s the <strong>thoughtful combination</strong> of proven methods, optimized for the realities of enterprise data and user behavior. Most academic papers test on clean datasets with perfect queries. Real users ask terrible questions about messy data, and this system is built for that reality.</p><h2>Conclusion</h2><p>The AI RAG space is rapidly evolving. There are thousands of papers, frameworks, and code repositories published every month. There are probably better ways to design a RAG system - from <a href="https://microsoft.github.io/graphrag/" rel="noopener noreferrer" target="_blank">Microsoft’s GraphRAG</a> to <a href="https://engineering.salesforce.com/the-next-generation-of-rag-how-enriched-index-redefines-information-retrieval-for-llms/" rel="noopener noreferrer" target="_blank">Salesforce’s “Next Gen RAG”</a> - it always depends on your needs.</p><p>I’m a big fan of the Pareto frontier, so the content above focused on performance while being feasible for a solo developer or small team and easy on the wallet. The techniques presented here represent battle-tested approaches that deliver real-world results without requiring massive infrastructure investments.</p><p>The key takeaways:</p><ul><li><strong>Document-based chunking with context preservation</strong> beats simple fixed-size chunking</li><li><strong>Late chunking</strong> significantly improves embedding quality for enterprise content</li><li><strong>Hierarchical search</strong> with proper metadata handling scales better than flat vector search</li><li><strong>HyDE and query expansion</strong> dramatically improve query understanding</li><li><strong>Self-reflective RAG</strong> fills information gaps that single-pass retrieval misses</li><li><strong>Hybrid search</strong> combining semantic and keyword approaches outperforms either alone</li></ul><p>Thanks for reading, and I look forward to hearing from you.</p>

content_text:
The following blog post is an opinionated result of hundreds of hours of intense research, implementation and evaluation while developing an enterprise AI chat system (source code can be found

here

). While I do not remember every source and arxiv paper, most will be linked directly in the text. Now please fasten your seatbelts and enjoy your deep dive into modern AI RAG architecture.

AI Apps Are Really Just RAG

AI is being crammed into every application. This guide will help developers understand what you actually need to build an AI app. Without debating whether the current state of cramming AI features into every existing app makes sense, we first need to understand what is meant by a text-based AI app.

In almost every commercial AI app, there’s no training of custom AI models. Instead, they use base models from big providers like OpenAI, Google, Anthropic, xAI, or open-source alternatives like Llama or Mistral. This is because training a model is highly resource-intensive, and state-of-the-art models have become commoditized - the gaps between models regarding intelligence, capabilities, and price are closing rapidly.

So if developing AI apps doesn’t typically involve building your own models, what do you need to do as a developer? The keyword is

RAG

(Retrieval Augmented Generation). Essentially, it means feeding base models the correct data and tuning requests to get desired answers. RAG isn’t new - it’s been around in data science for quite some time - but it has become a buzzword in the recent AI craze.

At its core, RAG involves two steps:

Ingestion

and

Retrieval

. This might seem trivial at first, but as with anything in data science, it’s always garbage in, garbage out. The AI space is evolving drastically - I can’t remember the last time I’ve seen any space evolve this fast. I’ve done the heavy lifting, scanning through dozens of papers and repositories while developing my own clone of enterprise applications like GleanAI, Zive, ChatGPT Enterprise, and Google Agentspace.

While there’s a lot of hocus pocus from snake oil vendors claiming their one magical solution will deliver enterprise RAG nirvana, the reality is far more nuanced. From my experience building production systems, it’s not any single technique that makes RAG work - it’s the thoughtful combination of proven methods that delivers results. The recommendations below sit on the Pareto frontier, balancing integration difficulty, performance, and cost. Ignore the shiny new papers promising 50% improvements on cherry-picked benchmarks; focus on techniques that actually work in messy enterprise environments with real users asking terrible questions.

Enterprise RAG Architecture: A comprehensive view of production-ready RAG systems, from heterogeneous data sources through sophisticated ingestion pipelines to hierarchical retrieval and intelligent response generation. Note the two-stage approach that reduces search scope while maintaining quality.

Here’s what you need to know to become an AI developer:

Ingestion

Streamlining Your Data

You want your AI apps to know about your data, but that data comes in different forms and shapes. Let’s say you have Microsoft SharePoint, Notion, or Confluence spaces. Text generation AI works best when fed cleaned-up text content, so what do you do with all the PDFs, Office docs, or custom spec files?

You need a pipeline that converts heterogeneous data into a homogeneous format so your retrieval algorithm always finds the most relevant (top K) results. Base models work exceptionally well with markdown content - a format that’s plain but has enough rich text elements and strong hierarchical features. There are different markdown flavors, but the most popular in terms of features is

GFM (GitHub Flavored Markdown)

.

In my enterprise app, I have several converters to convert most file types (PDFs, Office docs, images) and custom enterprise apps (Notion, Confluence) in a plug-and-play way to markdown. The code includes integrations for

SharePoint, Notion, Confluence, Jira & Google Drive via OAuth

,

PDF to GFM Markdown via Gemini 2.5 Flash

, and

Office Files (PowerPoint, Word, etc.) to GFM Markdown via Gotenberg

.

Chunking

Once you have your cleaned data in markdown format, you need to tackle the next challenge: LLMs have limited context window sizes, meaning you typically can’t cram all your data into the API call. While context windows have grown enormous, it still doesn’t make sense to push everything in there for two main reasons:

Cost factor

- Even with token caching, caching makes reflecting data changes non-viable

Retrieval performance

- Performance degrades with bigger contexts, leading to more hallucinations

Therefore, we need to “chunk” our data and optimize it for retrieval to feed the LLM with relevant information. Chunking is a meta-science in itself, with different approaches varying in complexity, cost, and performance implications:

Fixed Size Chunking

- Splits text into chunks of specified character count, regardless of structure

Recursive Chunking

- Divides text hierarchically using separators, recursively splitting until reaching desired sizes

Document-Based Chunking

- Splits content based on document structure (markdown headers, code functions, table rows)

Semantic Chunking

- Uses embeddings to group semantically related content together

Agentic Chunking

- Employs LLMs to intelligently determine chunk boundaries

Moving from top to bottom, we see increasing sophistication and performance, while Document-Based Chunking sits at the Pareto frontier. In my testing, this approach is perfect, especially since our data is already in GFM format.

The Context Problem

There’s one significant issue: keeping context. Imagine this scenario: We have text about Berlin and chunk it into different paragraphs, but lose context to the overall topic. When I say “it’s more than 3.85 million…” what is “it” referring to? We lose context in this chunk.

Context Loss in Traditional Chunking: Individual chunks lose connection to the document's main subject, making 'it's more than 3.85 million' ambiguous without knowing the document discusses Berlin's population [https://jina.ai/news/late-chunking-in-long-context-embedding-models/]

The Solution: Context Path Breadcrumbs

Here’s where we can apply a clever “hack” utilizing our markdown conversion. We create the

abstract syntax tree (AST)

from the markdown document hierarchy and prepend it as breadcrumbs to every chunk.

// Example: If our paragraph is in "Prehistory of Berlin" (H3)

// under "History" (H2) in document "Berlin" (H1)

const contextPath = "Berlin > History > Prehistory of Berlin";

const chunkWithContext = `${contextPath}\n\n${chunkContent}`;

This isn’t completely solved yet, as our document could live in a folder structure. To overcome this, we apply the same solution at the folder/file level: build a hierarchical path based on folder structure and prepend it to the markdown AST.

The complete GFM context path chunker with all its logic, including finding optimal chunk size and truncating context paths when they get too long, can be found

here

. This includes

incremental updates

and hierarchy management.

Embeddings

So far, we’ve cleaned and split our content while preserving context. We could use a simple text search algorithm to find relevant chunks. If I search for “inhabitants Berlin,” we’d probably get relevant results. But what about searching “inhabitants capital Germany”? With basic text search, we’d get no results.

That’s where

embeddings

come into play - one of the key concepts of AI RAG data handling.

What are embeddings?

Embeddings are numerical representations of text that capture semantic meaning in high-dimensional vector space. Instead of matching exact words, embeddings allow us to find conceptually similar content - so “inhabitants capital Germany” would match chunks about “Berlin population” because the AI understands these concepts are related.

Choosing the Right Embedding Model

The quality of your embeddings depends heavily on the model you choose. The

MTEB (Massive Text Embedding Benchmark) leaderboard

is your go-to resource for comparing embedding models across different tasks. It evaluates models across 8 different task types including retrieval, classification, clustering, and semantic textual similarity.

The Chunk Size Dilemma

Chunk size significantly impacts retrieval quality - include too much and the vector loses specificity, include too little and you lose context. There’s no “one size fits all” solution. Most developers start with chunk sizes between 100-1000 tokens, but the

recommended maximum

is around 512 tokens for optimal performance.

The Size Bias Problem

Longer texts generally show higher similarity scores when compared to other embeddings, regardless of actual semantic relevance. This means you can’t use cosine similarity thresholds to determine if matches are actually relevant.

Late Chunking: The Game Changer

This is where

Late Chunking

comes in as a breakthrough approach. Instead of chunking first then embedding, late chunking flips this process:

// Traditional approach

const chunks = chunkDocument(fullDocument);

const embeddings = chunks.map(chunk => embed(chunk)); // Individual embeddings

// Late chunking approach

const chunks = chunkDocument(fullDocument);

const contextualEmbeddings = embed(chunks); // Embed all chunks together as array

// This creates embeddings that consider inter-chunk relationships and context

This preserves broader document context within each chunk since the embeddings were created considering the full text, not just isolated segments. When you split text like “Its more than 3.85 million inhabitants make it the European Union’s most populous city” from a Berlin article, traditional chunking loses the connection to “Berlin” mentioned earlier, but late chunking preserves this context.

For an implementation of late chunking, take a look

here

.

Hybrid and Hierarchical Indexed Database Schema

We’ve prepared our data to be stored in the database and later retrieved. AI RAG databases usually need the embedding field type. There are many paid embedding databases, but usually the database you already have is the best database. If you’re working with PostgreSQL, you can use an extension called

pg_vector

.

Dedicated Vector Databases:

Pinecone

- Managed vector database with excellent performance and scaling

Weaviate

- Open-source vector database with GraphQL APIs

Qdrant

- Rust-based vector search engine with filtering capabilities

Chroma

- Developer-friendly open-source embedding database

Milvus

- Cloud-native vector database built for scalable similarity search

Traditional Databases with Vector Support:

Supabase

- Hosted PostgreSQL with built-in pgvector support

Redis

- In-memory database with vector search capabilities

Elasticsearch

- Search engine with dense vector field support

MongoDB Atlas

- Document database with vector search functionality

While dedicated vector databases offer optimized performance, I’ve found that

extending your existing database with vector capabilities often provides the best balance of simplicity, cost, and performance

for most enterprise applications. You avoid data synchronization issues, leverage existing backup and security infrastructure, and can perform complex queries that combine vector similarity with traditional filters.

Enterprises can have vast amounts of documents in different data silos. This can be challenging for a RAG system to always retrieve the most relevant chunks.

Hierarchical Indexing

is the approach of structuring your database schema to have multiple levels for your documents.

-- Parent level: document metadata and summary

CREATE TABLE documents (

id bigserial PRIMARY KEY,

title text NOT NULL,

summary text NOT NULL,

summary_embedding vector(1024) NOT NULL,

-- ... other fields

);

-- Child level: document chunks with foreign key

CREATE TABLE document_chunks (

id bigserial PRIMARY KEY,

content text NOT NULL,

embedding vector(1024) NOT NULL,

document_id bigint REFERENCES documents(id) ON DELETE CASCADE,

-- ... other fields

);

This helps us retrieve in a first step all possibly relevant documents, then only search over the chunks of those documents. The summary can be generated via a low-cost model such as

Gemini Flash Lite

. A prompt for generating such a summary can be found

here

.

While embeddings are great, multiple

papers

have proven that a mixture of classic text corpus search such as

BM25

or even simpler

n-gram

search yield significantly better results. That’s why on our hierarchical indexed database structure we also create a TokenNgram index.

Retrieval

We’ve finally reached the last puzzle piece: the retrieval step. We have cleaned and saved our data, and now it’s time to create a sophisticated search algorithm. This is where the magic happens - turning a user’s messy query into precise, relevant results.

HyDE - Making Queries Smarter

The first technique that’s absolutely game-changing is

HyDE (Hypothetical Document Embeddings)

. Instead of just embedding the user’s query directly, we generate a hypothetical answer to what the user is asking, then use that answer’s embedding for retrieval.

Why? Because user queries are often short and ambiguous (“quarterly results”), while the documents they’re looking for contain full, detailed content.

// Traditional approach

const queryEmbedding = await embed("quarterly results");

// HyDE approach

const hydeAnswer = await llm.generate("What would quarterly results contain?");

// -> "The quarterly results include revenue of $2.3M, profit margins of 15%..."

const hydeEmbedding = await embed(hydeAnswer);

HyDE was introduced by researchers at CMU

and consistently outperforms standard query embedding across most domains. In my implementation, I generate the HyDE response using a lightweight model like Gemini Flash, then embed both the original query and the hypothetical answer.

Hierarchical Document Retrieval

Here’s where our database design pays off. Instead of throwing embeddings at the wall and hoping for the best, we use a

two-stage hierarchical hybrid search

:

// Stage 1: Document-level candidate filtering (hybrid search)

const documentCandidates = await client.rpc('match_documents_hierarchical', {

query_embedding: queryEmbedding,        // For document summary similarity

hyde_embedding: hydeEmbedding,          // For chunk-level scoring

query_text: queryText,                  // For full-text search

// ... other params

});

// The SQL function performs sophisticated two-stage filtering:

// 1. Document filtering with OR condition:

//    - Vector similarity: (1 - (summary_embedding <=> query_embedding)) > threshold

//    - Full-text search: document.content &@~ query_text

// 2. Chunk scoring from candidate documents with hybrid scoring:

//    - Semantic: (1 - (chunk.embedding <=> hyde_embedding))

//    - Keyword: pgroonga_score when chunk.content &@~ query_text

The actual implementation is far more nuanced:

Stage 1 - Document Candidate Selection:

Uses

query embedding

against

document summary embeddings

for semantic similarity

Simultaneously runs

full-text search

against the

entire document content

using PGroonga

Documents qualify if they match EITHER condition (OR logic)

This dramatically reduces the search space while maintaining high recall

Stage 2 - Chunk-Level Hybrid Scoring:

Uses

HyDE embedding

against

individual chunk embeddings

for precise semantic matching

Runs

full-text search

against

chunk content

for exact keyword matches

Combines both scores with proper normalization and weighting

Only processes chunks from documents that passed Stage 1 filtering

-- Simplified view of the actual SQL implementation

WITH document_candidates AS (

SELECT d.id AS doc_id

FROM documents d

JOIN document_user_access ua ON d.id = ua.document_id

WHERE ua.user_id = target_user_id

AND (

(1 - (d.summary_embedding <=> query_embedding)) > similarity_threshold

OR d.content &@~ query_text  -- Full-text search on document content

)

LIMIT doc_search_limit

),

chunk_scores AS (

SELECT c.*, d.*,

-- Semantic score using HyDE embedding

(1 - (c.embedding <=> hyde_embedding)) AS semantic_score,

-- Keyword score using PGroonga

CASE WHEN c.content &@~ query_text

THEN pgroonga_score(c.tableoid, c.ctid)

ELSE 0 END AS keyword_score

FROM document_chunks c

JOIN documents d ON c.document_id = d.id

WHERE c.document_id IN (SELECT doc_id FROM document_candidates)

)

-- Score normalization and weighting happens here...

This approach is inspired by hierarchical passage retrieval methods but optimized for real-world enterprise scenarios. The key insight is that

most queries are looking for information from a small subset of documents

, so we can massively reduce the search space without sacrificing quality. The hybrid approach at both levels ensures we capture both semantic similarity and exact keyword matches, which is crucial for enterprise search where users might search for specific terms, project names, or concepts.

Query Expansion and Self-Reflective RAG

Raw user queries are often inadequate. “Meeting notes from last week” could mean anything. That’s where

query expansion

comes in. Before hitting the database, I use a lightweight LLM to:

Expand the query with related terms and synonyms

Extract time filters (“last week” → specific date range)

Identify the user’s intent and adjust search weights accordingly

Generate better search keywords

But here’s where it gets interesting - after the initial search, we don’t just stop. We implement a

self-reflective RAG approach

where the system evaluates its own search results:

// The complete orchestrated search pipeline

async function createOrchestratedStream(query: string, userId: string) {

// 1. Query Expansion with HyDE

const expansion = await generateObject({

model: tracedExpansionModel,

schema: queryExpansionSchema,

messages: [{

role: 'user',

content: `Expand query: ${query}`

}]

});

// 2. Initial hierarchical search

const initialResults = await hierarchicalRetriever.hierarchicalSearch(

expansion.expandedQuery,

expansion.hydeAnswer, // Key: using HyDE for better retrieval

{

target_user_id: userId,

embedding_weight: 0.7,

fulltext_weight: 0.3,

start_date: expansion.timeFilter?.startDate,

end_date: expansion.timeFilter?.endDate

}

);

// 3. Self-reflective gap analysis

const evaluation = await generateObject({

model: tracedGapAnalysisModel,

schema: gapAnalysisSchema,

messages: [{

role: 'user',

content: gapAnalysisUserPrompt.format({

query: expansion.expandedQuery,

initialSearchResults: JSON.stringify(initialResults)

})

}]

});

// 4. Follow-up searches if gaps identified

let allResults = [...initialResults];

if (evaluation.needsAdditionalSearches) {

const followupPromises = evaluation.informationGaps

.filter(gap => gap.importance >= 7) // Only high-importance gaps

.slice(0, 2) // Max 2 follow-ups

.map(gap =>

hierarchicalRetriever.hierarchicalSearch(

gap.followupQuery,

gap.followUpHydeAnswer,

searchParams

)

);

const followupResults = await Promise.all(followupPromises);

allResults = HierarchicalRetriever.combineResults([

initialResults,

...followupResults

]);

}

return allResults;

}

This is similar to the

ReAct pattern

but applied specifically to information retrieval. The magic number I’ve found is limiting this to 2 follow-up searches maximum - beyond that, you get diminishing returns and increased latency.

Hybrid Search That Actually Works

Everyone talks about “hybrid search,” but most implementations are inadequate. Here’s what actually works:

Adaptive weighting based on query intent

: By default, we bias toward semantic search (70% embeddings, 30% keyword) to capture documents with relevant meaning. However, during query expansion, we analyze the query to detect specific search patterns and dynamically adjust weights.

// Query expansion determines optimal search strategy

const expansion = await expandQuery(query);

const weights = expansion.weights || {

semanticWeight: expansion.queryType === 'specific_terms' ? 0.3 : 0.7,

keywordWeight: expansion.queryType === 'specific_terms' ? 0.7 : 0.3

};

const combinedScore = (

weights.semanticWeight * normalizedSemanticScore +

weights.keywordWeight * normalizedKeywordScore

);

Query type detection

:

Conceptual queries

(“project status”, “team performance”) → favor semantic search

Specific terms

(“Project Phoenix”, “JIRA-1234”) → favor keyword search

Mixed queries

→ balanced weighting

Proper score normalization

: You can’t just add cosine similarity and BM25 / PGroonga scores—they have completely different distributions. I normalize both to 0-1 ranges before combining.

Two-phase scoring

: Embedding similarity for document-level filtering, then detailed hybrid scoring only for chunks from candidate documents. This keeps it fast while maintaining quality.

This adaptive approach ensures we get the

conceptual relevance

that embeddings excel at, while not missing

exact matches

for specific terminology that enterprises rely on.

Enterprise data isn’t just text - it has structure, permissions, timestamps, and context. My retrieval system handles:

Temporal filtering

: Queries like “recent sales reports” automatically extract time ranges and filter documents by

source_updated_at

Permission-aware search

: The

document_user_access

table ensures users only see results they’re authorized to access, all handled at the database level for performance

Smart metadata filtering

: Instead of rigid JSON matching, I implemented flexible metadata filters that handle type mismatches gracefully

Source-aware retrieval

: Different document sources can have different retrieval strategies and weights applied automatically

Reranking - The Diminishing Returns

After all this sophisticated retrieval, we still have one more potential trick:

reranking

. In earlier versions of my system, I used Jina’s reranking models to take top candidates and reorder them based on deeper semantic understanding.

Here’s the thing though -

I’ve disabled reranking in the current version

. While academic papers show impressive improvements, in practice with a well-tuned hierarchical search, the quality gains are marginal while the latency cost is significant. The hierarchical approach with proper hybrid scoring and query expansion gets you 90% of the way there, and reranking that extra 5-10% isn’t worth doubling your response time.

The key insight is that rerankers work best when initial retrieval is poor. But when your retrieval is already sophisticated, you hit diminishing returns fast. That’s the Pareto frontier in action - focus on getting the fundamentals right rather than adding expensive bells and whistles.

Performance Optimizations That Matter

All of this sounds expensive, but with proper optimizations, it runs fast:

HNSW vector indexes

for ~95% faster similarity searches compared to IVF indexes

Materialized CTEs

in PostgreSQL functions to avoid recomputing candidate sets

Parallel processing

of follow-up searches during self-reflective retrieval

Smart caching

of embeddings and query expansions

The entire pipeline - from raw query to final ranked results - typically runs in under 2 seconds for enterprise datasets with millions of documents. Check out the full implementation of the hierarchical retriever

here

and the query orchestration logic

here

.

Why This Approach Works

The hierarchical approach with self-reflection feels significantly more accurate than basic vector search implementations and performs well compared to what I’ve seen from enterprise search solutions. While I haven’t done formal benchmarking against commercial solutions like Glean or Microsoft Viva, the approach addresses the core problems I’ve observed with simpler RAG implementations.

The secret sauce isn’t any single technique - it’s the

thoughtful combination

of proven methods, optimized for the realities of enterprise data and user behavior. Most academic papers test on clean datasets with perfect queries. Real users ask terrible questions about messy data, and this system is built for that reality.

Conclusion

The AI RAG space is rapidly evolving. There are thousands of papers, frameworks, and code repositories published every month. There are probably better ways to design a RAG system - from

Microsoft’s GraphRAG

to

Salesforce’s “Next Gen RAG”

- it always depends on your needs.

I’m a big fan of the Pareto frontier, so the content above focused on performance while being feasible for a solo developer or small team and easy on the wallet. The techniques presented here represent battle-tested approaches that deliver real-world results without requiring massive infrastructure investments.

The key takeaways:

Document-based chunking with context preservation

beats simple fixed-size chunking

Late chunking

significantly improves embedding quality for enterprise content

Hierarchical search

with proper metadata handling scales better than flat vector search

HyDE and query expansion

dramatically improve query understanding

Self-reflective RAG

fills information gaps that single-pass retrieval misses

Hybrid search

combining semantic and keyword approaches outperforms either alone

Thanks for reading, and I look forward to hearing from you.


url: https://www.ibm.com/think/insights/rag-problems-five-ways-to-fix
title: RAG Problems Persist. Here Are Five Ways to Fix Them | IBM
added_at: 2025-08-28 02:15:13
content_html:
<p>Data leakage is a known problem with AI systems in general, and LLMs that use RAG are no exception. Without the right measures in place, an LLM might provide low-level users information that they’re not authorized to access, from <a href="https://www.ibm.com/think/topics/pii" rel="noopener noreferrer" target="_blank">personal identifying information</a> (PII) to sensitive financial data.</p>
<p>“This is a reality with RAG,” Kairali said. “When you start with proof-of-concept, everybody is happy. But then when you want to push it to production and you want to make sure it’s production-grade, you start understanding that there’s a data protection issue.”</p>
<p>Addressing the issue means preserving access control lists (ACLs) and other <a href="https://www.ibm.com/think/topics/data-governance" rel="noopener noreferrer" target="_blank">governance</a> policies when unstructured data is ingested into multiple databases. “When the query is coming and data is retrieved, it’s important to make sure that the ACLs and governance policies are being honored,” Kairali said. “It’s basically an engineering problem.”</p>
<p>Solving this engineering problem can be made easier with the right data platforms, such as governed, open source-enabled <a href="https://www.ibm.com/think/topics/data-lakehouse" rel="noopener noreferrer" target="_blank">data lakehouses</a>. For instance, IBM’s <a href="https://www.ibm.com/products/watsonx-data" rel="noopener noreferrer" target="_blank">watsonx.data</a>, a hybrid, open data lakehouse, ensures that access controls are inherited from document source systems when data is retrieved. It also provides annotation for PII to prevent sensitive information from being shared.</p>
<p>As LLMs and other generative AI become more deeply ingrained in everyday workflows, improving RAG helps enterprises unlock greater value from their enterprise data. The right enterprise-level tools and strategies “enable higher performance and accuracy so that data becomes manageable and valuable,” Nirmal said. “That’s what every customer is looking for."</p>

content_text:
Data leakage is a known problem with AI systems in general, and LLMs that use RAG are no exception. Without the right measures in place, an LLM might provide low-level users information that they’re not authorized to access, from

personal identifying information

(PII) to sensitive financial data.

“This is a reality with RAG,” Kairali said. “When you start with proof-of-concept, everybody is happy. But then when you want to push it to production and you want to make sure it’s production-grade, you start understanding that there’s a data protection issue.”

Addressing the issue means preserving access control lists (ACLs) and other

governance

policies when unstructured data is ingested into multiple databases. “When the query is coming and data is retrieved, it’s important to make sure that the ACLs and governance policies are being honored,” Kairali said. “It’s basically an engineering problem.”

Solving this engineering problem can be made easier with the right data platforms, such as governed, open source-enabled

data lakehouses

. For instance, IBM’s

watsonx.data

, a hybrid, open data lakehouse, ensures that access controls are inherited from document source systems when data is retrieved. It also provides annotation for PII to prevent sensitive information from being shared.

As LLMs and other generative AI become more deeply ingrained in everyday workflows, improving RAG helps enterprises unlock greater value from their enterprise data. The right enterprise-level tools and strategies “enable higher performance and accuracy so that data becomes manageable and valuable,” Nirmal said. “That’s what every customer is looking for."

thumb_url:
https://www.ibm.com/content/dam/worldwide-content/creative-assets/ilrn/ul/g/e1/00/adobestock_868228347.jpeg/_jcr_content/renditions/cq5dam.web.1280.1280.jpeg


url: https://www.anthropic.com/news/contextual-retrieval
title: Introducing Contextual Retrieval \ Anthropic
added_at: 2025-08-28 02:32:47
content_html:
<p>For an AI model to be useful in specific contexts, it often needs access to background knowledge. For example, customer support chatbots need knowledge about the specific business they're being used for, and legal analyst bots need to know about a vast array of past cases.</p><p>Developers typically enhance an AI model's knowledge using Retrieval-Augmented Generation (RAG). RAG is a method that retrieves relevant information from a knowledge base and appends it to the user's prompt, significantly enhancing the model's response. The problem is that traditional RAG solutions remove context when encoding information, which often results in the system failing to retrieve the relevant information from the knowledge base.</p><p>In this post, we outline a method that dramatically improves the retrieval step in RAG. The method is called “Contextual Retrieval” and uses two sub-techniques: Contextual Embeddings and Contextual BM25. This method can reduce the number of failed retrievals by 49% and, when combined with reranking, by 67%. These represent significant improvements in retrieval accuracy, which directly translates to better performance in downstream tasks. </p><p>You can easily deploy your own Contextual Retrieval solution with Claude with <a href="https://github.com/anthropics/anthropic-cookbook/tree/main/skills/contextual-embeddings" rel="noopener noreferrer" target="_blank">our cookbook</a>.</p><h3>A note on simply using a longer prompt</h3><p>Sometimes the simplest solution is the best. If your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base in the prompt that you give the model, with no need for RAG or similar methods.</p><p>A few weeks ago, we released <a href="https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching" rel="noopener noreferrer" target="_blank">prompt caching</a> for Claude, which makes this approach significantly faster and more cost-effective. Developers can now cache frequently used prompts between API calls, reducing latency by &gt; 2x and costs by up to 90% (you can see how it works by reading our <a href="https://github.com/anthropics/anthropic-cookbook/blob/main/misc/prompt_caching.ipynb" rel="noopener noreferrer" target="_blank">prompt caching cookbook</a>).</p><p>However, as your knowledge base grows, you'll need a more scalable solution. That’s where Contextual Retrieval comes in.</p><h2>A primer on RAG: scaling to larger knowledge bases</h2><p>For larger knowledge bases that don't fit within the context window, RAG is the typical solution. RAG works by preprocessing a knowledge base using the following steps:</p><ol><li>Break down the knowledge base (the “corpus” of documents) into smaller chunks of text, usually no more than a few hundred tokens;</li><li>Use an embedding model to convert these chunks into vector embeddings that encode meaning;</li><li>Store these embeddings in a vector database that allows for searching by semantic similarity.</li></ol><p>At runtime, when a user inputs a query to the model, the vector database is used to find the most relevant chunks based on semantic similarity to the query. Then, the most relevant chunks are added to the prompt sent to the generative model.</p><p>While embedding models excel at capturing semantic relationships, they can miss crucial exact matches. Fortunately, there’s an older technique that can assist in these situations. BM25 (Best Matching 25) is a ranking function that uses lexical matching to find precise word or phrase matches. It's particularly effective for queries that include unique identifiers or technical terms.</p><p>BM25 works by building upon the TF-IDF (Term Frequency-Inverse Document Frequency) concept. TF-IDF measures how important a word is to a document in a collection. BM25 refines this by considering document length and applying a saturation function to term frequency, which helps prevent common words from dominating the results.</p><p>Here’s how BM25 can succeed where semantic embeddings fail: Suppose a user queries "Error code TS-999" in a technical support database. An embedding model might find content about error codes in general, but could miss the exact "TS-999" match. BM25 looks for this specific text string to identify the relevant documentation.</p><p>RAG solutions can more accurately retrieve the most applicable chunks by combining the embeddings and BM25 techniques using the following steps:</p><ol><li>Break down the knowledge base (the "corpus" of documents) into smaller chunks of text, usually no more than a few hundred tokens;</li><li>Create TF-IDF encodings and semantic embeddings for these chunks;</li><li>Use BM25 to find top chunks based on exact matches;</li><li>Use embeddings to find top chunks based on semantic similarity;</li><li>Combine and deduplicate results from (3) and (4) using rank fusion techniques;</li><li>Add the top-K chunks to the prompt to generate the response.</li></ol><p>By leveraging both BM25 and embedding models, traditional RAG systems can provide more comprehensive and accurate results, balancing precise term matching with broader semantic understanding.</p><figure><img loading="lazy" referrerpolicy="no-referrer" src="https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2F45603646e979c62349ce27744a940abf30200d57-3840x2160.png&amp;w=3840&amp;q=75" srcset="https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2F45603646e979c62349ce27744a940abf30200d57-3840x2160.png&amp;w=3840&amp;q=75 1x"/><figcaption>A Standard Retrieval-Augmented Generation (RAG) system that uses both embeddings and Best Match 25 (BM25) to retrieve information. TF-IDF (term frequency-inverse document frequency) measures word importance and forms the basis for BM25.</figcaption></figure><p>This approach allows you to cost-effectively scale to enormous knowledge bases, far beyond what could fit in a single prompt. But these traditional RAG systems have a significant limitation: they often destroy context.</p><h3>The context conundrum in traditional RAG</h3><p>In traditional RAG, documents are typically split into smaller chunks for efficient retrieval. While this approach works well for many applications, it can lead to problems when individual chunks lack sufficient context.</p><p>For example, imagine you had a collection of financial information (say, U.S. SEC filings) embedded in your knowledge base, and you received the following question: <em>"What was the revenue growth for ACME Corp in Q2 2023?"</em></p><p>A relevant chunk might contain the text: <em>"The company's revenue grew by 3% over the previous quarter."</em> However, this chunk on its own doesn't specify which company it's referring to or the relevant time period, making it difficult to retrieve the right information or use the information effectively.</p><h2>Introducing Contextual Retrieval</h2><p>Contextual Retrieval solves this problem by prepending chunk-specific explanatory context to each chunk before embedding (“Contextual Embeddings”) and creating the BM25 index (“Contextual BM25”).</p><p>Let’s return to our SEC filings collection example. Here's an example of how a chunk might be transformed:</p><pre><code>original_chunk = "The company's revenue grew by 3% over the previous quarter."

contextualized_chunk = "This chunk is from an SEC filing on ACME corp's performance in Q2 2023; the previous quarter's revenue was $314 million. The company's revenue grew by 3% over the previous quarter."</code></pre><p>It is worth noting that other approaches to using context to improve retrieval have been proposed in the past. Other proposals include: <a href="https://aclanthology.org/W02-0405.pdf" rel="noopener noreferrer" target="_blank">adding generic document summaries to chunks</a> (we experimented and saw very limited gains), <a href="https://arxiv.org/abs/2212.10496" rel="noopener noreferrer" target="_blank">hypothetical document embedding</a>, and <a href="https://www.llamaindex.ai/blog/a-new-document-summary-index-for-llm-powered-qa-systems-9a32ece2f9ec" rel="noopener noreferrer" target="_blank">summary-based indexing</a> (we evaluated and saw low performance). These methods differ from what is proposed in this post.</p><h3>Implementing Contextual Retrieval</h3><p>Of course, it would be far too much work to manually annotate the thousands or even millions of chunks in a knowledge base. To implement Contextual Retrieval, we turn to Claude. We’ve written a prompt that instructs the model to provide concise, chunk-specific context that explains the chunk using the context of the overall document. We used the following Claude 3 Haiku prompt to generate context for each chunk:</p><pre><code>&lt;document&gt; 
{{WHOLE_DOCUMENT}} 
&lt;/document&gt; 
Here is the chunk we want to situate within the whole document 
&lt;chunk&gt; 
{{CHUNK_CONTENT}} 
&lt;/chunk&gt; 
Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else. </code></pre><p>The resulting contextual text, usually 50-100 tokens, is prepended to the chunk before embedding it and before creating the BM25 index.</p><p>Here’s what the preprocessing flow looks like in practice:</p><figure><img loading="lazy" referrerpolicy="no-referrer" src="https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2F2496e7c6fedd7ffaa043895c23a4089638b0c21b-3840x2160.png&amp;w=3840&amp;q=75" srcset="https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2F2496e7c6fedd7ffaa043895c23a4089638b0c21b-3840x2160.png&amp;w=3840&amp;q=75 1x"/><figcaption><em>Contextual Retrieval is a preprocessing technique that improves retrieval accuracy.</em><br/></figcaption></figure><p>If you’re interested in using Contextual Retrieval, you can get started with <a href="https://github.com/anthropics/anthropic-cookbook/tree/main/skills/contextual-embeddings" rel="noopener noreferrer" target="_blank">our cookbook</a>.</p><h3>Using Prompt Caching to reduce the costs of Contextual Retrieval</h3><p>Contextual Retrieval is uniquely possible at low cost with Claude, thanks to the special prompt caching feature we mentioned above. With prompt caching, you don’t need to pass in the reference document for every chunk. You simply load the document into the cache once and then reference the previously cached content. Assuming 800 token chunks, 8k token documents, 50 token context instructions, and 100 tokens of context per chunk, <strong>the one-time cost to generate contextualized chunks is $1.02 per million document tokens</strong>.</p><h4>Methodology</h4><p>We experimented across various knowledge domains (codebases, fiction, ArXiv papers, Science Papers), embedding models, retrieval strategies, and evaluation metrics. We’ve included a few examples of the questions and answers we used for each domain in <a href="https://assets.anthropic.com/m/1632cded0a125333/original/Contextual-Retrieval-Appendix-2.pdf" rel="noopener noreferrer" target="_blank">Appendix II</a>.</p><p>The graphs below show the average performance across all knowledge domains with the top-performing embedding configuration (Gemini Text 004) and retrieving the top-20-chunks. We use 1 minus recall@20 as our evaluation metric, which measures the percentage of relevant documents that fail to be retrieved within the top 20 chunks. You can see the full results in the appendix - contextualizing improves performance in every embedding-source combination we evaluated.</p><h4>Performance improvements</h4><p>Our experiments showed that:</p><ul><li><strong>Contextual Embeddings reduced the top-20-chunk retrieval failure rate by 35%</strong> (5.7% → 3.7%).</li><li><strong>Combining Contextual Embeddings and Contextual BM25 reduced the top-20-chunk retrieval failure rate by 49%</strong> (5.7% → 2.9%).</li></ul><figure><img loading="lazy" referrerpolicy="no-referrer" src="https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2F7f8d739e491fe6b3ba0e6a9c74e4083d760b88c9-3840x2160.png&amp;w=3840&amp;q=75" srcset="https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2F7f8d739e491fe6b3ba0e6a9c74e4083d760b88c9-3840x2160.png&amp;w=3840&amp;q=75 1x"/><figcaption><em>Combining Contextual Embedding and Contextual BM25 reduce the top-20-chunk retrieval failure rate by 49%.</em></figcaption></figure><h4>Implementation considerations</h4><p>When implementing Contextual Retrieval, there are a few considerations to keep in mind:</p><ol><li><strong>Chunk boundaries:</strong> Consider how you split your documents into chunks. The choice of chunk size, chunk boundary, and chunk overlap can affect retrieval performance1.</li><li><strong>Embedding model:</strong> Whereas Contextual Retrieval improves performance across all embedding models we tested, some models may benefit more than others. We found <a href="https://ai.google.dev/gemini-api/docs/embeddings" rel="noopener noreferrer" target="_blank">Gemini</a> and <a href="https://www.voyageai.com/" rel="noopener noreferrer" target="_blank">Voyage</a> embeddings to be particularly effective.</li><li><strong>Custom contextualizer prompts:</strong> While the generic prompt we provided works well, you may be able to achieve even better results with prompts tailored to your specific domain or use case (for example, including a glossary of key terms that might only be defined in other documents in the knowledge base).</li><li><strong>Number of chunks:</strong> Adding more chunks into the context window increases the chances that you include the relevant information. However, more information can be distracting for models so there's a limit to this. We tried delivering 5, 10, and 20 chunks, and found using 20 to be the most performant of these options (see appendix for comparisons) but it’s worth experimenting on your use case.</li></ol><p><strong>Always run evals: </strong>Response generation may be improved by passing it the contextualized chunk and distinguishing between what is context and what is the chunk.</p><h2>Further boosting performance with Reranking</h2><p>In a final step, we can combine Contextual Retrieval with another technique to give even more performance improvements. In traditional RAG, the AI system searches its knowledge base to find the potentially relevant information chunks. With large knowledge bases, this initial retrieval often returns a lot of chunks—sometimes hundreds—of varying relevance and importance.</p><p>Reranking is a commonly used filtering technique to ensure that only the most relevant chunks are passed to the model. Reranking provides better responses and reduces cost and latency because the model is processing less information. The key steps are:</p><ol><li>Perform initial retrieval to get the top potentially relevant chunks (we used the top 150);</li><li>Pass the top-N chunks, along with the user's query, through the reranking model;</li><li>Using a reranking model, give each chunk a score based on its relevance and importance to the prompt, then select the top-K chunks (we used the top 20);</li><li>Pass the top-K chunks into the model as context to generate the final result.</li></ol><figure><img loading="lazy" referrerpolicy="no-referrer" src="https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2F8f82c6175a64442ceff4334b54fac2ab3436a1d1-3840x2160.png&amp;w=3840&amp;q=75" srcset="https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2F8f82c6175a64442ceff4334b54fac2ab3436a1d1-3840x2160.png&amp;w=3840&amp;q=75 1x"/><figcaption><em>Combine Contextual Retrieva and Reranking to maximize retrieval accuracy.</em></figcaption></figure><h3>Performance improvements</h3><p>There are several reranking models on the market. We ran our tests with the <a href="https://cohere.com/rerank" rel="noopener noreferrer" target="_blank">Cohere reranker</a>. Voyage<a href="https://docs.voyageai.com/docs/reranker" rel="noopener noreferrer" target="_blank"> also offers a reranker</a>, though we did not have time to test it. Our experiments showed that, across various domains, adding a reranking step further optimizes retrieval.</p><p>Specifically, we found that Reranked Contextual Embedding and Contextual BM25 reduced the top-20-chunk retrieval failure rate by 67% (5.7% → 1.9%).</p><figure><img loading="lazy" referrerpolicy="no-referrer" src="https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2F93a70cfbb7cca35bb8d86ea0a23bdeeb699e8e58-3840x2160.png&amp;w=3840&amp;q=75" srcset="https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2F93a70cfbb7cca35bb8d86ea0a23bdeeb699e8e58-3840x2160.png&amp;w=3840&amp;q=75 1x"/><figcaption><em>Reranked Contextual Embedding and Contextual BM25 reduces the top-20-chunk retrieval failure rate by 67%.</em></figcaption></figure><h4>Cost and latency considerations</h4><p>One important consideration with reranking is the impact on latency and cost, especially when reranking a large number of chunks. Because reranking adds an extra step at runtime, it inevitably adds a small amount of latency, even though the reranker scores all the chunks in parallel. There is an inherent trade-off between reranking more chunks for better performance vs. reranking fewer for lower latency and cost. We recommend experimenting with different settings on your specific use case to find the right balance.</p><h2>Conclusion</h2><p>We ran a large number of tests, comparing different combinations of all the techniques described above (embedding model, use of BM25, use of contextual retrieval, use of a reranker, and total # of top-K results retrieved), all across a variety of different dataset types. Here’s a summary of what we found:</p><ol><li>Embeddings+BM25 is better than embeddings on their own;</li><li>Voyage and Gemini have the best embeddings of the ones we tested;</li><li>Passing the top-20 chunks to the model is more effective than just the top-10 or top-5;</li><li>Adding context to chunks improves retrieval accuracy a lot;</li><li>Reranking is better than no reranking;</li><li><strong>All these benefits stack</strong>: to maximize performance improvements, we can combine contextual embeddings (from Voyage or Gemini) with contextual BM25, plus a reranking step, and adding the 20 chunks to the prompt.</li></ol><p>We encourage all developers working with knowledge bases to use <a href="https://github.com/anthropics/anthropic-cookbook/tree/main/skills/contextual-embeddings" rel="noopener noreferrer" target="_blank">our cookbook</a> to experiment with these approaches to unlock new levels of performance.</p><h2>Appendix I</h2><p>Below is a breakdown of results across datasets, embedding providers, use of BM25 in addition to embeddings, use of contextual retrieval, and use of reranking for Retrievals @ 20.</p><p>See <a href="https://assets.anthropic.com/m/1632cded0a125333/original/Contextual-Retrieval-Appendix-2.pdf" rel="noopener noreferrer" target="_blank">Appendix II</a> for the breakdowns for Retrievals @ 10 and @ 5 as well as example questions and answers for each dataset.</p><figure><img loading="lazy" referrerpolicy="no-referrer" src="https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2F646a894ec4e6120cade9951a362f685cd2ec89b2-2458x2983.png&amp;w=3840&amp;q=75" srcset="https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2F646a894ec4e6120cade9951a362f685cd2ec89b2-2458x2983.png&amp;w=3840&amp;q=75 1x"/><figcaption><em>1 minus recall @ 20 results across data sets and embedding providers.</em></figcaption></figure>

content_text:
For an AI model to be useful in specific contexts, it often needs access to background knowledge. For example, customer support chatbots need knowledge about the specific business they're being used for, and legal analyst bots need to know about a vast array of past cases.

Developers typically enhance an AI model's knowledge using Retrieval-Augmented Generation (RAG). RAG is a method that retrieves relevant information from a knowledge base and appends it to the user's prompt, significantly enhancing the model's response. The problem is that traditional RAG solutions remove context when encoding information, which often results in the system failing to retrieve the relevant information from the knowledge base.

In this post, we outline a method that dramatically improves the retrieval step in RAG. The method is called “Contextual Retrieval” and uses two sub-techniques: Contextual Embeddings and Contextual BM25. This method can reduce the number of failed retrievals by 49% and, when combined with reranking, by 67%. These represent significant improvements in retrieval accuracy, which directly translates to better performance in downstream tasks.

You can easily deploy your own Contextual Retrieval solution with Claude with

our cookbook

.

A note on simply using a longer prompt

Sometimes the simplest solution is the best. If your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base in the prompt that you give the model, with no need for RAG or similar methods.

A few weeks ago, we released

prompt caching

for Claude, which makes this approach significantly faster and more cost-effective. Developers can now cache frequently used prompts between API calls, reducing latency by > 2x and costs by up to 90% (you can see how it works by reading our

prompt caching cookbook

).

However, as your knowledge base grows, you'll need a more scalable solution. That’s where Contextual Retrieval comes in.

A primer on RAG: scaling to larger knowledge bases

For larger knowledge bases that don't fit within the context window, RAG is the typical solution. RAG works by preprocessing a knowledge base using the following steps:

Break down the knowledge base (the “corpus” of documents) into smaller chunks of text, usually no more than a few hundred tokens;

Use an embedding model to convert these chunks into vector embeddings that encode meaning;

Store these embeddings in a vector database that allows for searching by semantic similarity.

At runtime, when a user inputs a query to the model, the vector database is used to find the most relevant chunks based on semantic similarity to the query. Then, the most relevant chunks are added to the prompt sent to the generative model.

While embedding models excel at capturing semantic relationships, they can miss crucial exact matches. Fortunately, there’s an older technique that can assist in these situations. BM25 (Best Matching 25) is a ranking function that uses lexical matching to find precise word or phrase matches. It's particularly effective for queries that include unique identifiers or technical terms.

BM25 works by building upon the TF-IDF (Term Frequency-Inverse Document Frequency) concept. TF-IDF measures how important a word is to a document in a collection. BM25 refines this by considering document length and applying a saturation function to term frequency, which helps prevent common words from dominating the results.

Here’s how BM25 can succeed where semantic embeddings fail: Suppose a user queries "Error code TS-999" in a technical support database. An embedding model might find content about error codes in general, but could miss the exact "TS-999" match. BM25 looks for this specific text string to identify the relevant documentation.

RAG solutions can more accurately retrieve the most applicable chunks by combining the embeddings and BM25 techniques using the following steps:

Break down the knowledge base (the "corpus" of documents) into smaller chunks of text, usually no more than a few hundred tokens;

Create TF-IDF encodings and semantic embeddings for these chunks;

Use BM25 to find top chunks based on exact matches;

Use embeddings to find top chunks based on semantic similarity;

Combine and deduplicate results from (3) and (4) using rank fusion techniques;

Add the top-K chunks to the prompt to generate the response.

By leveraging both BM25 and embedding models, traditional RAG systems can provide more comprehensive and accurate results, balancing precise term matching with broader semantic understanding.

A Standard Retrieval-Augmented Generation (RAG) system that uses both embeddings and Best Match 25 (BM25) to retrieve information. TF-IDF (term frequency-inverse document frequency) measures word importance and forms the basis for BM25.

This approach allows you to cost-effectively scale to enormous knowledge bases, far beyond what could fit in a single prompt. But these traditional RAG systems have a significant limitation: they often destroy context.

The context conundrum in traditional RAG

In traditional RAG, documents are typically split into smaller chunks for efficient retrieval. While this approach works well for many applications, it can lead to problems when individual chunks lack sufficient context.

For example, imagine you had a collection of financial information (say, U.S. SEC filings) embedded in your knowledge base, and you received the following question:

"What was the revenue growth for ACME Corp in Q2 2023?"

A relevant chunk might contain the text:

"The company's revenue grew by 3% over the previous quarter."

However, this chunk on its own doesn't specify which company it's referring to or the relevant time period, making it difficult to retrieve the right information or use the information effectively.

Introducing Contextual Retrieval

Contextual Retrieval solves this problem by prepending chunk-specific explanatory context to each chunk before embedding (“Contextual Embeddings”) and creating the BM25 index (“Contextual BM25”).

Let’s return to our SEC filings collection example. Here's an example of how a chunk might be transformed:

original_chunk = "The company's revenue grew by 3% over the previous quarter."

contextualized_chunk = "This chunk is from an SEC filing on ACME corp's performance in Q2 2023; the previous quarter's revenue was $314 million. The company's revenue grew by 3% over the previous quarter."

It is worth noting that other approaches to using context to improve retrieval have been proposed in the past. Other proposals include:

adding generic document summaries to chunks

(we experimented and saw very limited gains),

hypothetical document embedding

, and

summary-based indexing

(we evaluated and saw low performance). These methods differ from what is proposed in this post.

Implementing Contextual Retrieval

Of course, it would be far too much work to manually annotate the thousands or even millions of chunks in a knowledge base. To implement Contextual Retrieval, we turn to Claude. We’ve written a prompt that instructs the model to provide concise, chunk-specific context that explains the chunk using the context of the overall document. We used the following Claude 3 Haiku prompt to generate context for each chunk:

<document>

{{WHOLE_DOCUMENT}}

</document>

Here is the chunk we want to situate within the whole document

<chunk>

{{CHUNK_CONTENT}}

</chunk>

Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else.

The resulting contextual text, usually 50-100 tokens, is prepended to the chunk before embedding it and before creating the BM25 index.

Here’s what the preprocessing flow looks like in practice:

Contextual Retrieval is a preprocessing technique that improves retrieval accuracy.

If you’re interested in using Contextual Retrieval, you can get started with

our cookbook

.

Using Prompt Caching to reduce the costs of Contextual Retrieval

Contextual Retrieval is uniquely possible at low cost with Claude, thanks to the special prompt caching feature we mentioned above. With prompt caching, you don’t need to pass in the reference document for every chunk. You simply load the document into the cache once and then reference the previously cached content. Assuming 800 token chunks, 8k token documents, 50 token context instructions, and 100 tokens of context per chunk,

the one-time cost to generate contextualized chunks is $1.02 per million document tokens

.

Methodology

We experimented across various knowledge domains (codebases, fiction, ArXiv papers, Science Papers), embedding models, retrieval strategies, and evaluation metrics. We’ve included a few examples of the questions and answers we used for each domain in

Appendix II

.

The graphs below show the average performance across all knowledge domains with the top-performing embedding configuration (Gemini Text 004) and retrieving the top-20-chunks. We use 1 minus recall@20 as our evaluation metric, which measures the percentage of relevant documents that fail to be retrieved within the top 20 chunks. You can see the full results in the appendix - contextualizing improves performance in every embedding-source combination we evaluated.

Performance improvements

Our experiments showed that:

Contextual Embeddings reduced the top-20-chunk retrieval failure rate by 35%

(5.7% → 3.7%).

Combining Contextual Embeddings and Contextual BM25 reduced the top-20-chunk retrieval failure rate by 49%

(5.7% → 2.9%).

Combining Contextual Embedding and Contextual BM25 reduce the top-20-chunk retrieval failure rate by 49%.

Implementation considerations

When implementing Contextual Retrieval, there are a few considerations to keep in mind:

Chunk boundaries:

Consider how you split your documents into chunks. The choice of chunk size, chunk boundary, and chunk overlap can affect retrieval performance1.

Embedding model:

Whereas Contextual Retrieval improves performance across all embedding models we tested, some models may benefit more than others. We found

Gemini

and

Voyage

embeddings to be particularly effective.

Custom contextualizer prompts:

While the generic prompt we provided works well, you may be able to achieve even better results with prompts tailored to your specific domain or use case (for example, including a glossary of key terms that might only be defined in other documents in the knowledge base).

Number of chunks:

Adding more chunks into the context window increases the chances that you include the relevant information. However, more information can be distracting for models so there's a limit to this. We tried delivering 5, 10, and 20 chunks, and found using 20 to be the most performant of these options (see appendix for comparisons) but it’s worth experimenting on your use case.

Always run evals:

Response generation may be improved by passing it the contextualized chunk and distinguishing between what is context and what is the chunk.

Further boosting performance with Reranking

In a final step, we can combine Contextual Retrieval with another technique to give even more performance improvements. In traditional RAG, the AI system searches its knowledge base to find the potentially relevant information chunks. With large knowledge bases, this initial retrieval often returns a lot of chunks—sometimes hundreds—of varying relevance and importance.

Reranking is a commonly used filtering technique to ensure that only the most relevant chunks are passed to the model. Reranking provides better responses and reduces cost and latency because the model is processing less information. The key steps are:

Perform initial retrieval to get the top potentially relevant chunks (we used the top 150);

Pass the top-N chunks, along with the user's query, through the reranking model;

Using a reranking model, give each chunk a score based on its relevance and importance to the prompt, then select the top-K chunks (we used the top 20);

Pass the top-K chunks into the model as context to generate the final result.

Combine Contextual Retrieva and Reranking to maximize retrieval accuracy.

Performance improvements

There are several reranking models on the market. We ran our tests with the

Cohere reranker

. Voyage

also offers a reranker

, though we did not have time to test it. Our experiments showed that, across various domains, adding a reranking step further optimizes retrieval.

Specifically, we found that Reranked Contextual Embedding and Contextual BM25 reduced the top-20-chunk retrieval failure rate by 67% (5.7% → 1.9%).

Reranked Contextual Embedding and Contextual BM25 reduces the top-20-chunk retrieval failure rate by 67%.

Cost and latency considerations

One important consideration with reranking is the impact on latency and cost, especially when reranking a large number of chunks. Because reranking adds an extra step at runtime, it inevitably adds a small amount of latency, even though the reranker scores all the chunks in parallel. There is an inherent trade-off between reranking more chunks for better performance vs. reranking fewer for lower latency and cost. We recommend experimenting with different settings on your specific use case to find the right balance.

Conclusion

We ran a large number of tests, comparing different combinations of all the techniques described above (embedding model, use of BM25, use of contextual retrieval, use of a reranker, and total # of top-K results retrieved), all across a variety of different dataset types. Here’s a summary of what we found:

Embeddings+BM25 is better than embeddings on their own;

Voyage and Gemini have the best embeddings of the ones we tested;

Passing the top-20 chunks to the model is more effective than just the top-10 or top-5;

Adding context to chunks improves retrieval accuracy a lot;

Reranking is better than no reranking;

All these benefits stack

: to maximize performance improvements, we can combine contextual embeddings (from Voyage or Gemini) with contextual BM25, plus a reranking step, and adding the 20 chunks to the prompt.

We encourage all developers working with knowledge bases to use

our cookbook

to experiment with these approaches to unlock new levels of performance.

Appendix I

Below is a breakdown of results across datasets, embedding providers, use of BM25 in addition to embeddings, use of contextual retrieval, and use of reranking for Retrievals @ 20.

See

Appendix II

for the breakdowns for Retrievals @ 10 and @ 5 as well as example questions and answers for each dataset.

1 minus recall @ 20 results across data sets and embedding providers.

thumb_url:
https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2F45603646e979c62349ce27744a940abf30200d57-3840x2160.png&w=3840&q=75


url: https://www.oreilly.com/radar/llm-system-design-and-model-selection/
title: LLM System Design and Model Selection – O’Reilly
added_at: 2025-08-28 03:12:42
thumb_url: https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/08/AI-Model-Pricing1.png
content_html:
<p>Choosing the right LLM has become a full-time job. New models appear almost daily, each offering different capabilities, prices, and quirks, from reasoning strengths to cost efficiency to code generation. This competition creates strong incentives for AI labs to carve out a niche and gives new startups room to emerge, resulting in a fragmented landscape where one model may excel at reasoning, another at code, and a third at cost efficiency.</p><p>AI, in one sense, is getting cheaper faster than any previous technology, at least per <em>unit of intelligence</em>. For example, input tokens for Gemini 2.5 Flash-Lite are approximately 600 times cheaper than what OpenAI’s GPT-3 (davinci-002) cost in August 2022, while outperforming it on every metric. At the same time, access to frontier capabilities is also becoming more expensive than ever. The reason is simple: we can now pay directly for more capability, which has led to the rise of $300+ per month Pro subscription tiers.</p><p>Today, any developer can run capable open-weight models locally for negligible marginal cost using tools like Ollama. At the same time, enterprise systems can experience sharp cost increases, depending on the model size (number of parameters, such as 3 billion, 70 billion, or even in the trillions), the number of internal processing steps, and the volume of input data. For developers, these are central system design choices that directly affect feasibility and cost structure. For end users, this complexity explains why a basic subscription differs so much from a premium plan with higher limits on advanced models.</p><p>The choices you make in these broader development decisions also determine which LLM and inference settings are optimal for your use case.</p><p>At Towards AI, we work across the LLM stack, building applications, designing enterprise systems, and offering online courses (<a href="https://www.oreilly.com/videos/building-and-operating/019283645221/" rel="noreferrer noopener" target="_blank">including one on O’Reilly</a>), custom corporate training, and LLM development consultancy. In our experience, model selection and system design have become central to getting meaningful results from these tools. Much of that, in turn, depends on where today’s models are gaining their capabilities. While scale still plays a role, recent progress has come from a broader mix of factors, including training-data quality, post-training methods, and especially how models are used at inference time.</p><h2><strong>The Shifting Foundations of Model Capability</strong></h2><p>While early gains in LLM performance tracked closely with increases in pretraining compute, larger datasets, bigger models, and more training steps, this approach now yields diminishing returns.</p><p>Recent improvements come from a broader mix of strategies. Pretraining-data quality has become just as important as quantity, with better filtering and AI-generated synthetic data contributing to stronger models. Architectural efficiency, like the innovations introduced by DeepSeek, has started to close the gap between size and capability. And post-training techniques, especially instruction tuning and reinforcement learning from human or AI feedback (RLHF/RLAIF), have made models more aligned, controllable, and responsive in practice.</p><p>The more fundamental shift, however, is happening at inference time. Since late 2024, with models like OpenAI’s o1, we’ve entered a new phase where models can trade compute for reasoning <em>on demand</em>. Rather than relying solely on what was baked in during training, they can now “think harder” at runtime, running more internal steps, exploring alternative answers, or chaining thoughts before responding. This opens up new capability ceilings, but also introduces new cost dynamics.</p><p>These varied improvement strategies have led to a clear divergence among AI labs and models, a rapid expansion in model choice, and in some cases, an explosion in model usage costs.</p><h2><strong>The Modern Cost Explosion: How Inference Scaling Changed the Game</strong></h2><p>Inference-time compute scaling has introduced a new dynamic in LLM system design: We’ve gone from a single lever model size, to at least four distinct ways to trade cost for capability at runtime. The result is a widening gap in inference cost across models and use cases, sometimes by factors of 10,000x or more.</p><p><strong>Larger models (size scaling): </strong>The most obvious lever is sheer model size. Frontier LLMs, like GPT-4.5, often built with mixture of experts (MoE) architectures, can have input token costs 750 times higher than streamlined models like Gemini Flash-Lite. Larger parameter counts mean more compute per token, especially when multiple experts are active per query.</p><p><strong>Series scaling (“thinking tokens”): </strong>Newer “reasoning” LLMs perform more internal computational steps, or a longer chain of thought, before producing their final answer. For example, OpenAI’s o1 used ~30x more compute than GPT-4o on average, and often 5x more output tokens per task. Agentic systems introduce an additional method of series scaling and an extra layer of cost multiplication. As these agents think, plan, act, reassess, plan, act, and so on, they often make many LLM steps in a loop, each incurring additional cost.</p><p><strong>Parallel scaling: </strong>Here, the system runs multiple model instances on the same task and then automatically selects the best output via automated methods, such as majority voting (which assumes the most common answer is likely correct) or self-confidence scores (where the model output claiming the highest confidence in its response is taken as the best). The o3-pro model likely runs 5–10x parallel instances over o3. This multiplies the cost by the number of parallel attempts (with some nuance).</p><p><strong>Input context scaling: </strong>In RAG pipelines, the number of retrieved chunks and their size directly influence input token costs and the LLM’s ability to synthesize a good answer. More context can often improve results, but this comes at a higher cost and potential latency. Context isn’t free; it’s another dimension of scaling that developers must budget for.</p><p>Taken together, these four factors represent a fundamental shift in how model cost scales. For developers designing systems for high-value problems, <strong>10,000x to 1,000,000x differences in API costs to solve a problem based on architectural choices are now realistic possibilities</strong>. Reasoning LLMs, although only prominent for about nine months, reversed the trend of declining access costs to the very best models. This transforms the decision from “Which LLM should I use?” to include “How much reasoning do I <em>want to pay for</em>?”</p><p>This shift changes how we think about selection. Choosing an LLM is no longer about chasing the highest benchmark score; it’s about finding the balance point where capability, latency, and cost align with your use case.</p><h2><strong>Core Model Selection Criteria</strong></h2><p>When choosing a model we find it is important to first clearly identify your use case and the minimum core AI capabilities and attributes needed to deliver it.</p><p>A common first step is to take a look at standard benchmark scores (for example LiveBench, MMLU-Pro, SWE-Bench). These benchmarks are a useful starting point, but some models are tuned on benchmark data, and real-world performance on tasks that are actually relevant to you will often vary. Filtering benchmark tests and scores by your industry and task category is a valuable step here. An LLM optimized for software development might perform poorly in creative writing or vice versa. The match between a model’s training focus and your application domain can outweigh general-purpose benchmarks.</p><p>Leaderboards like <a href="https://lmarena.ai/leaderboard" rel="noreferrer noopener" target="_blank">LMArena</a> and <a href="https://artificialanalysis.ai/" rel="noreferrer noopener" target="_blank">Artificial Analysis</a> offer broader human‑preference comparisons but still don’t replace custom real-world testing. It helps to have a set of your own example questions or tasks at hand to test out a new model for yourself and see how it performs. This should include a mix of easy tasks to establish a baseline and tough edge cases where it’s easy for a model to make mistakes.</p><p>As you move beyond ad hoc testing, for any serious development effort, <strong>custom evaluations are non-negotiable.</strong> They must be tailored to your use case and the types of problems you solve. This is the only way to truly know if a model, or a change to your system, is genuinely improving things for <em>your</em> users and <em>your</em> specific business goals.</p><p>Here are some core factors we consider:</p><p><strong>Multimodality</strong> is emerging as a major differentiator. Models like GPT-4o and Gemini can handle not just text but also images, audio, and in some cases video, unlocking applications that pure text models can’t support.</p><p><strong>Context window</strong> and effective <strong>context window utilization</strong> are also key: How many tokens or documents can the model process and how much of that advertised context window can the LLM <em>actually use</em> effectively without performance degradation relative to tasks that use less context?</p><p><strong>Latency</strong> is especially critical for interactive applications. In general, smaller or cheaper models tend to respond faster, while reasoning-heavy models introduce delays due to deeper internal computation.</p><p><strong>Reasoning </strong>is the ability to scale inference-time compute and perform multistep problem-solving, planning, or deep analysis.</p><p><strong>Privacy and security </strong>are often key considerations here. For example, if you want to keep your intellectual property private, you must use a model that won’t train on your inputs, which often points toward self-hosted or specific enterprise-grade API solutions.</p><p><strong>Trustworthiness</strong> is also becoming important and can come down to the reputation and track record of the AI lab. A model that produces erratic, biased, or reputationally damaging outputs is a liability, regardless of its benchmark scores. For instance, Grok has had well-publicized issues with its alignment. Even if such issues are supposedly fixed, it creates a lingering question of trust: How can one be sure it won’t behave similarly in the future?</p><p>Additionally, the <strong>knowledge cutoff date</strong> also matters if it is to be used in a fast-moving field.</p><p>After working out if a model meets your minimum capability, the next decision is often on optimizing trade-offs among cost, reliability, security, and latency. A key rule of thumb we find useful here: If the reliability gain from a more expensive model or more inference time saves more of your or your users’ time (valued in terms of pay) than the model costs, going with the larger model is a good decision!</p><h2><strong>The Pros and Cons of Open-Weight and Closed-API LLMs</strong></h2><p>The rise of increasingly competitive open-weight LLMs, such as Meta’s Llama series, Mistral, DeepSeek, Gemma, Qwen, and now OpenAI’s GPT-OSS has added a critical dimension to the model selection landscape. Momentum behind this open ecosystem surged with the release of DeepSeek’s R1 reasoning model, competitive with OpenAI’s o1 but priced at roughly 30x lower API <strong>costs</strong>. This sparked debate around efficiency versus scale and intensified the broader AI rivalry between China and the US. Reactions ranged from “OpenAI and Nvidia are obsolete” to “DeepSeek’s costs must be fabricated,” but regardless of hype, the release was a milestone. It showed that architectural innovation, not just scale, could deliver frontier-level performance with far greater cost efficiency.</p><p>This open-model offensive has continued with strong contributions from other Chinese labs like Alibaba (Qwen), Kimi, and Tencent (Hunyuan), and has put competitive pressure on Meta after its open-weight Llama models fell behind. China’s recent leadership in open-weight LLMs has raised new security/IP issues with some US- and European-based organizations, though we note accessing these model weights and running the model on your own infrastructure doesn’t require sending data to China.</p><p>This brings us back to the pros and cons of open weights. While closed-API LLMs still lead at the frontier of capability, the primary advantage of open-weight models is quick and affordable local testing, unparalleled flexibility, and increased data security when run internally. Organizations can also perform <strong>full fine-tuning</strong>, adapting the model’s core weights and behaviors to their specific domain, language, and tasks. Open models also provide <strong>stability and predictability</strong>—you control the version you deploy, insulating your production systems from unexpected changes or degradations that can sometimes occur with unannounced updates to proprietary API-based models.</p><p>Public closed-model APIs from major providers benefit from immense economies of scale and highly optimized GPU utilization by batching requests from thousands of users, an efficiency that is difficult for a single organization to replicate. This often means that using a closed-source API can be cheaper per inference than self-hosting an open model. Security and compliance are also more nuanced than they first appear. While some organizations must use self-hosted models to simplify compliance with regulations like GDPR by keeping data entirely within their own perimeter, this places the entire burden of securing the infrastructure on the internal team—a complex and expensive undertaking. Top API providers also often offer dedicated instances, private cloud endpoints, and contractual agreements that can guarantee data residency, zero-logging, and meet stringent regulatory standards. The choice, therefore, is not a simple open-versus-closed binary.</p><p>The boundary between open and closed models is also becoming increasingly blurred. Open-weight models are increasingly offered via API by third-party LLM inference platforms, combining the flexibility of open models with the simplicity of hosted access. This hybrid approach often strikes a practical balance between control and operational complexity.</p><h2><strong>Leading Closed LLMs</strong></h2><p>Below, we present some key costs and metrics for leading closed-source models available via API. Many of these models have additional complexity and varied pricing including options for fast modes, thinking modes, context caching, and longer context.</p><p>We present the latest LiveBench benchmark score for each model as one measure for comparison. LiveBench is a continuously updated benchmark designed to provide a “contamination-free” evaluation of large language models by regularly releasing new questions with objective, verifiable answers. It scores models out of 100 on a diverse set of challenging tasks, with a significant focus on capabilities like reasoning, coding, and data analysis. The similar LiveBench scores between GPT-4.5 and 2.5 Flash-Lite, despite 750x input token cost variation, highlights both that smaller models are now very capable but also that not all capabilities are captured in a single benchmark!</p><figure><img alt="AI model pricing and specifications comparison" loading="lazy" referrerpolicy="no-referrer" sizes="(min-width: 1024px) 800px, 100vw" src="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/08/AI-Model-Pricing1.png" srcset="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/08/AI-Model-Pricing1.png 1735w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/08/AI-Model-Pricing1-300x285.png 300w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/08/AI-Model-Pricing1-1600x1522.png 1600w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/08/AI-Model-Pricing1-768x730.png 768w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/08/AI-Model-Pricing1-1536x1461.png 1536w"/><figcaption><em>Source: Towards AI, Company Reports, <a href="https://livebench.ai/" rel="noreferrer noopener" target="_blank">LiveBench AI</a> </em><br/></figcaption></figure><h2><strong>Leading open-weight LLMs</strong></h2><p>Below, we also present key costs, the LiveBench benchmark score, and context length for leading open-weight models available via API. We compare hosted versions of these models for easy comparison. Different API providers may choose to host open-weight models with different levels of quantization, different context lengths, and different pricing, so performance can vary between providers.</p><figure><img alt="AI model pricing and specifications 2" loading="lazy" referrerpolicy="no-referrer" sizes="(min-width: 1024px) 800px, 100vw" src="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/08/AI-Model-Pricing-and-Specifications.png" srcset="https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/08/AI-Model-Pricing-and-Specifications.png 468w, https://www.oreilly.com/radar/wp-content/uploads/sites/3/2025/08/AI-Model-Pricing-and-Specifications-270x300.png 270w"/><figcaption><em>Source: Towards AI, Company Reports, <a href="https://livebench.ai/" rel="noreferrer noopener" target="_blank">LiveBench AI</a></em></figcaption></figure><p>Whether hosted or self-deployed, selecting a model only solves part of the problem. In practice, most of the complexity and opportunity lies in how that model is used: how it’s prompted, extended, fine-tuned, or embedded within a broader workflow. These system-level decisions often have a greater impact on performance and cost than the model choice itself.</p><h2><strong>A Practical Guide to Designing an LLM System</strong></h2><p>Simply picking the biggest or newest LLM is rarely the optimal strategy. A more effective approach starts with a deep understanding of the developer’s toolkit: knowing which technique to apply to which problem to achieve the desired capability and reliability without unnecessary cost. This is all part of the constant “<strong>march of nines” as you develop LLM systems modularly to solve for more reliability and capability.</strong> There is a need to prioritize the easiest wins that deliver tangible value before investing in further incremental and often costly accuracy improvements. The reality will always vary on a case-by-case basis, but here is a quick guide to navigating this process.</p><h3><strong>Step 1: Open Versus Closed?</strong></h3><p>This is often your first decision.</p><ul>
<li><strong>Go with a closed-API model (e.g., from OpenAI, Google, Anthropic) if:</strong> Your priority is accessing the absolute state-of-the-art models with maximum simplicity.</li><li><strong>Go with an open-weight model (e.g., Llama, Mistral, Qwen, DeepSeek) if:</strong><ul>
<li><strong>Data security and compliance are paramount:</strong> If you need to guarantee that sensitive data never leaves your own infrastructure.</li><li><strong>You need deep customization and control:</strong> If your goal is to fine-tune a model on proprietary data and to create a specialized expert that you control completely.</li></ul>
</li></ul><p>If you went open, what can you <em>realistically</em> run? Your own GPU infrastructure is a hard constraint. Assess your cluster size and memory to determine if you can efficiently run a large, leading 1 trillion+ parameter MoE model, such as Kimi K2, or if you are better served by a medium-size model such as Gemma 3 27B or a much smaller model like Gemma 3n that can even run on mobile.</p><h3><strong>Step 2: Gauging the Need for Reasoning</strong></h3><p>Does your task require the model to simply blast out a response, or does it need to <em>think</em> first?</p><ul>
<li><strong>Reasoning:</strong> For tasks that involve complex, multistep problem-solving, brainstorming, strategic planning, intricate code generation, or deep analysis, you need a dedicated reasoning model such as o3, Gemini 2.5 Pro, DeepSeek R1, or Claude 4. In some cases these models can be used in high-reasoning mode, which encourages the model to think for longer before responding.</li><li><strong>No reasoning:</strong> For straightforward tasks like simple Q&amp;A, summarization of a single document, data extraction, or classification, a powerful reasoning model is overkill.</li><li><strong>The middle ground:</strong> For tasks requiring moderate reasoning, such as generating a structured report from a few data points or performing basic data analysis at scale, a “mini” reasoning model, like OpenAI’s o4-mini or Gemini Flash 2.5, offers a balance of capability and cost.</li></ul><h3><strong>Step 3: Pinpointing Key Model Attributes</strong></h3><p>Beyond general intelligence and reasoning, modern LLMs are specialists. Your choice should be guided by the specific attributes and “superpowers” your application needs.</p><ul>
<li><strong>Prioritize accuracy over cost</strong> for high-value tasks where mistakes are costly or where a human expert’s time is being saved. o3-pro is a standout model here and it can even be used as a fact checker to meticulously check the details of an earlier LLM output.</li><li><strong>Prioritize speed and cost over accuracy:</strong> For user-facing, real-time applications like chatbots or high-volume, low-value tasks like simple data categorization, latency and cost are paramount. Choose a hyper-efficient “flash” or “mini” model such as Gemini 2.5 Flash-Lite. Qwen3-235B models can also be a great option here but are too complex to inference yourself.</li><li><strong>Do you need a deep, long-context researcher?</strong> For tasks that require synthesizing information from massive documents, entire codebases, or extensive legal contracts, a model with a vast and highly effective context window is crucial. <strong>Gemini 2.5 Pro</strong> excels here.</li><li><strong>Is multimodality essential?</strong> If your application needs to understand or generate images, process audio in real time, or analyze video, your choice narrows to models like <strong>GPT-4o</strong> or the <strong>Gemini</strong> family. For one-shot YouTube video processing, Gemini is the standout.</li><li><strong>Is it a code-specific task?</strong> While many models can code, some are explicitly tuned for it. In the open world, Codestral and Gemma do a decent job. But Claude has won hearts and minds, at least for now.</li><li><strong>Do you need live, agentic web search?</strong> For answering questions about current events or topics beyond the model’s knowledge cutoff, consider a model with a built-in, reliable web search, such as <strong>o3.</strong></li><li>Do you need complex <strong>dialogue and emotional nuance?</strong> GPT-4.5, Kimi K2, Claude Opus 4.0, or Grok 4 do a great job.</li></ul><h3><strong>Step 4: Prompting, Then RAG, Then Evaluation</strong></h3><p>Before you dive into more complex and costly development, always see how far you can get with the simplest techniques. This is a path of escalating complexity. Model choice for RAG pipelines is often centered around latency for end users, but recently more complex agentic RAG workflows or long-context RAG tasks require reasoning models or longer context capabilities.</p><ol>
<li><strong>Prompt engineering first:</strong> Your first step is always to maximize the model’s inherent capabilities through clear, well-structured prompting. Often, a better prompt with a more capable model is all you need.</li><li><strong>Move to retrieval-augmented generation (RAG):</strong> If your model’s limitation is a lack of specific, private, or up-to-date <em>knowledge</em>, RAG is the next logical step. This is the best approach for reducing hallucinations, providing answers based on proprietary documents, and ensuring responses are current. However, RAG is not a panacea. Its effectiveness is entirely dependent on the quality and freshness of your dataset, and building a retrieval system that consistently finds and uses the <em>most</em> relevant information is a significant engineering challenge. RAG also comes with many associated decisions, such as the quantity of data to retrieve and feed into the model’s context window, and just how much use you make of long-context capabilities and context caching.</li><li><strong>Iterate with advanced RAG:</strong> To push performance, you will need to implement more advanced techniques like hybrid search (combining keyword and vector search), re-ranking retrieved results for relevance, and query transformation.</li><li><strong>Build custom evaluation</strong>: Ensure iterations on your system design, additions of new advanced RAG techniques, or updates to the latest model are always moving progress forward on your key metrics!</li></ol><h3><strong>Step 5: Fine-Tune or Distill for Deep Specialization</strong></h3><p>If the model’s core <em>behavior</em>—not its knowledge—is still the problem, then it’s time to consider fine-tuning. Fine-tuning is a significant undertaking that requires a high-quality dataset, engineering effort, and computational resources. However, it can enable a smaller, cheaper open-weight model to outperform a massive generalist model on a specific, narrow task, making it a powerful tool for optimization and specialization.</p><ul>
<li><strong>Fine-tuning is for changing behavior, not adding knowledge.</strong> Use it to teach a model a specific skill, style, or format. For example:<ul>
<li>To reliably output data in a complex, structured format like specific JSON or XML schemas.</li><li>To master the unique vocabulary and nuances of a highly specialized domain (e.g., legal, medical).</li><li>Some closed-source models are available for fine-tuning via API such as Gemini 2.5 Flash and various OpenAI models. Larger models are normally not available.</li><li><strong>In open-weight models, </strong>Llama 3.3 70B and Qwen 70B are fine-tuning staples. The process is more complex to fine-tune an open-weight model yourself.</li></ul>
</li><li>Model <strong>distillation</strong> can also serve as a production-focused optimization step. In its simplest form, this consists of generating synthetic data from larger models to create fine-tuning datasets to improve the capabilities of smaller models.</li><li><strong>Reinforcement fine-tuning (RFT) for problem-solving accuracy</strong><br/>Instead of just imitating correct answers, the model learns by trial, error, and correction. It is rewarded for getting answers right and penalized for getting them wrong.<ul>
<li><strong>Use RFT to:</strong> Create a true “expert model” that excels at complex tasks with objectively correct outcomes.</li><li><strong>The advantage:</strong> RFT is incredibly data-efficient, often requiring only a few dozen high-quality examples to achieve significant performance gains.</li><li><strong>The catch:</strong> RFT requires a reliable, automated “grader” to provide the reward signal. Designing this grader is a critical engineering challenge.</li></ul>
</li></ul><h3><strong>Step 6: Orchestrated Workflows Versus Autonomous Agents</strong></h3><p>The critical decision here is how much freedom to grant. Autonomous agents are also more likely to need more expensive reasoning models with greater levels of inference scaling. Parallel inference scaling methods with multiple agents are also beginning to deliver great results. Small errors can accumulate and multiply during many successive agentic steps so the investment in a stronger more capable model can make all the difference in building a usable product.</p><ul>
<li><strong>Choose an orchestrated workflow for predictable tasks</strong> <br/>You design a specific, often linear, sequence of steps, and the LLM acts as a powerful component at one or more of those steps.<ul>
<li><strong>Use when:</strong> You are automating a known, repeatable business process (e.g., processing a customer support ticket, generating a monthly financial summary). The goal is reliability, predictability, and control.</li><li><strong>Benefit:</strong> You maintain complete control over the process, ensuring consistency and managing costs effectively because the number and type of LLM calls are predefined.</li></ul>
</li><li><strong>Build hybrid pipelines:</strong> Often, the best results will come from combining many LLMs, open and closed, within a pipeline.<ul>
<li>This means using different LLMs for different stages of a workflow: a fast, cheap LLM for initial query routing; a specialized LLM for a specific subtask; a powerful reasoning LLM for complex planning; and perhaps another LLM for verification or refinement.</li><li>At Towards AI, we often have 2-3 different LLMs from different companies in an LLM pipeline.</li></ul>
</li><li><strong>Choose an autonomous agent for open-ended problems.</strong> You give the LLM a high-level goal, a set of tools (e.g., APIs, databases, code interpreters), and the autonomy to figure out the steps to achieve that goal.<ul>
<li><strong>Use when:</strong> The path to the solution is unknown and requires dynamic problem-solving, exploration, or research (e.g., debugging a complex software issue, performing deep market analysis, planning a multistage project).</li><li><strong>The critical risk—runaway costs:</strong> An agent that gets stuck in a loop, makes poor decisions, or explores inefficient paths can rapidly accumulate enormous API costs. <strong>Implementing strict guardrails is critical:</strong><ul>
<li><strong>Budget limits:</strong> Set hard caps on the cost per task.</li><li><strong>Step counters:</strong> Limit the total number of “thoughts” or “actions” an agent can take.</li><li><strong>Human-in-the-loop:</strong> Require human approval for potentially expensive or irreversible actions.</li></ul>
</li><li>Gemini 2.5 Pro and o3 are our favourite closed-API models for agent pipelines, while in open-weight models we like Kimi K2.</li></ul>
</li></ul><p>Working through these steps helps translate a vague problem into a concrete implementation plan, one that’s grounded in clear trade-offs and tailored to your needs. This structured approach often yields systems that are not only more capable and reliable but also far more effective for specific tasks than a general-purpose chatbot ever could be.</p><h2><strong>Conclusion</strong></h2><p>The open-versus-closed race gives us rapid access to strong LLMs but also creates complexity. Selecting and deploying them demands both engineering discipline and economic clarity.</p><p>Developing in the LLM ecosystem demands a new level of engineering discipline and keen economic awareness. No single LLM is a cure-all. A practical, evolving toolkit is essential, but knowing which tool to pull out for which job is the real art. The challenge isn’t just picking a model from a list; it’s about architecting a solution. This requires a systematic approach, moving from high-level strategic decisions about data and security down to the granular, technical choices of development and implementation.</p><p>The success of specialized “LLM wrapper” applications like Anyscale/Cursor for coding or Perplexity for search, some of which are now valued at over $10 billion, underscores the immense value in this tailored approach. These applications aren’t just thin wrappers; they are sophisticated systems that leverage foundation LLMs but add significant value through custom workflows, fine-tuning, data integration, and user experience design.</p><p>Ultimately, success hinges on informed pragmatism. Developers and organizations need a sharp understanding of their problem space and a firm grasp of how cost scales across model choice, series and parallel reasoning, context usage, and agentic behavior. Above all, custom evaluation is non-negotiable because your use case, not a benchmark, is the only standard that truly matters.</p>

content_text:
Choosing the right LLM has become a full-time job. New models appear almost daily, each offering different capabilities, prices, and quirks, from reasoning strengths to cost efficiency to code generation. This competition creates strong incentives for AI labs to carve out a niche and gives new startups room to emerge, resulting in a fragmented landscape where one model may excel at reasoning, another at code, and a third at cost efficiency.

AI, in one sense, is getting cheaper faster than any previous technology, at least per

unit of intelligence

. For example, input tokens for Gemini 2.5 Flash-Lite are approximately 600 times cheaper than what OpenAI’s GPT-3 (davinci-002) cost in August 2022, while outperforming it on every metric. At the same time, access to frontier capabilities is also becoming more expensive than ever. The reason is simple: we can now pay directly for more capability, which has led to the rise of $300+ per month Pro subscription tiers.

Today, any developer can run capable open-weight models locally for negligible marginal cost using tools like Ollama. At the same time, enterprise systems can experience sharp cost increases, depending on the model size (number of parameters, such as 3 billion, 70 billion, or even in the trillions), the number of internal processing steps, and the volume of input data. For developers, these are central system design choices that directly affect feasibility and cost structure. For end users, this complexity explains why a basic subscription differs so much from a premium plan with higher limits on advanced models.

The choices you make in these broader development decisions also determine which LLM and inference settings are optimal for your use case.

At Towards AI, we work across the LLM stack, building applications, designing enterprise systems, and offering online courses (

including one on O’Reilly

), custom corporate training, and LLM development consultancy. In our experience, model selection and system design have become central to getting meaningful results from these tools. Much of that, in turn, depends on where today’s models are gaining their capabilities. While scale still plays a role, recent progress has come from a broader mix of factors, including training-data quality, post-training methods, and especially how models are used at inference time.

The Shifting Foundations of Model Capability

While early gains in LLM performance tracked closely with increases in pretraining compute, larger datasets, bigger models, and more training steps, this approach now yields diminishing returns.

Recent improvements come from a broader mix of strategies. Pretraining-data quality has become just as important as quantity, with better filtering and AI-generated synthetic data contributing to stronger models. Architectural efficiency, like the innovations introduced by DeepSeek, has started to close the gap between size and capability. And post-training techniques, especially instruction tuning and reinforcement learning from human or AI feedback (RLHF/RLAIF), have made models more aligned, controllable, and responsive in practice.

The more fundamental shift, however, is happening at inference time. Since late 2024, with models like OpenAI’s o1, we’ve entered a new phase where models can trade compute for reasoning

on demand

. Rather than relying solely on what was baked in during training, they can now “think harder” at runtime, running more internal steps, exploring alternative answers, or chaining thoughts before responding. This opens up new capability ceilings, but also introduces new cost dynamics.

These varied improvement strategies have led to a clear divergence among AI labs and models, a rapid expansion in model choice, and in some cases, an explosion in model usage costs.

The Modern Cost Explosion: How Inference Scaling Changed the Game

Inference-time compute scaling has introduced a new dynamic in LLM system design: We’ve gone from a single lever model size, to at least four distinct ways to trade cost for capability at runtime. The result is a widening gap in inference cost across models and use cases, sometimes by factors of 10,000x or more.

Larger models (size scaling):

The most obvious lever is sheer model size. Frontier LLMs, like GPT-4.5, often built with mixture of experts (MoE) architectures, can have input token costs 750 times higher than streamlined models like Gemini Flash-Lite. Larger parameter counts mean more compute per token, especially when multiple experts are active per query.

Series scaling (“thinking tokens”):

Newer “reasoning” LLMs perform more internal computational steps, or a longer chain of thought, before producing their final answer. For example, OpenAI’s o1 used ~30x more compute than GPT-4o on average, and often 5x more output tokens per task. Agentic systems introduce an additional method of series scaling and an extra layer of cost multiplication. As these agents think, plan, act, reassess, plan, act, and so on, they often make many LLM steps in a loop, each incurring additional cost.

Parallel scaling:

Here, the system runs multiple model instances on the same task and then automatically selects the best output via automated methods, such as majority voting (which assumes the most common answer is likely correct) or self-confidence scores (where the model output claiming the highest confidence in its response is taken as the best). The o3-pro model likely runs 5–10x parallel instances over o3. This multiplies the cost by the number of parallel attempts (with some nuance).

Input context scaling:

In RAG pipelines, the number of retrieved chunks and their size directly influence input token costs and the LLM’s ability to synthesize a good answer. More context can often improve results, but this comes at a higher cost and potential latency. Context isn’t free; it’s another dimension of scaling that developers must budget for.

Taken together, these four factors represent a fundamental shift in how model cost scales. For developers designing systems for high-value problems,

10,000x to 1,000,000x differences in API costs to solve a problem based on architectural choices are now realistic possibilities

. Reasoning LLMs, although only prominent for about nine months, reversed the trend of declining access costs to the very best models. This transforms the decision from “Which LLM should I use?” to include “How much reasoning do I

want to pay for

?”

This shift changes how we think about selection. Choosing an LLM is no longer about chasing the highest benchmark score; it’s about finding the balance point where capability, latency, and cost align with your use case.

Core Model Selection Criteria

When choosing a model we find it is important to first clearly identify your use case and the minimum core AI capabilities and attributes needed to deliver it.

A common first step is to take a look at standard benchmark scores (for example LiveBench, MMLU-Pro, SWE-Bench). These benchmarks are a useful starting point, but some models are tuned on benchmark data, and real-world performance on tasks that are actually relevant to you will often vary. Filtering benchmark tests and scores by your industry and task category is a valuable step here. An LLM optimized for software development might perform poorly in creative writing or vice versa. The match between a model’s training focus and your application domain can outweigh general-purpose benchmarks.

Leaderboards like

LMArena

and

Artificial Analysis

offer broader human‑preference comparisons but still don’t replace custom real-world testing. It helps to have a set of your own example questions or tasks at hand to test out a new model for yourself and see how it performs. This should include a mix of easy tasks to establish a baseline and tough edge cases where it’s easy for a model to make mistakes.

As you move beyond ad hoc testing, for any serious development effort,

custom evaluations are non-negotiable.

They must be tailored to your use case and the types of problems you solve. This is the only way to truly know if a model, or a change to your system, is genuinely improving things for

your

users and

your

specific business goals.

Here are some core factors we consider:

Multimodality

is emerging as a major differentiator. Models like GPT-4o and Gemini can handle not just text but also images, audio, and in some cases video, unlocking applications that pure text models can’t support.

Context window

and effective

context window utilization

are also key: How many tokens or documents can the model process and how much of that advertised context window can the LLM

actually use

effectively without performance degradation relative to tasks that use less context?

Latency

is especially critical for interactive applications. In general, smaller or cheaper models tend to respond faster, while reasoning-heavy models introduce delays due to deeper internal computation.

Reasoning

is the ability to scale inference-time compute and perform multistep problem-solving, planning, or deep analysis.

Privacy and security

are often key considerations here. For example, if you want to keep your intellectual property private, you must use a model that won’t train on your inputs, which often points toward self-hosted or specific enterprise-grade API solutions.

Trustworthiness

is also becoming important and can come down to the reputation and track record of the AI lab. A model that produces erratic, biased, or reputationally damaging outputs is a liability, regardless of its benchmark scores. For instance, Grok has had well-publicized issues with its alignment. Even if such issues are supposedly fixed, it creates a lingering question of trust: How can one be sure it won’t behave similarly in the future?

Additionally, the

knowledge cutoff date

also matters if it is to be used in a fast-moving field.

After working out if a model meets your minimum capability, the next decision is often on optimizing trade-offs among cost, reliability, security, and latency. A key rule of thumb we find useful here: If the reliability gain from a more expensive model or more inference time saves more of your or your users’ time (valued in terms of pay) than the model costs, going with the larger model is a good decision!

The Pros and Cons of Open-Weight and Closed-API LLMs

The rise of increasingly competitive open-weight LLMs, such as Meta’s Llama series, Mistral, DeepSeek, Gemma, Qwen, and now OpenAI’s GPT-OSS has added a critical dimension to the model selection landscape. Momentum behind this open ecosystem surged with the release of DeepSeek’s R1 reasoning model, competitive with OpenAI’s o1 but priced at roughly 30x lower API

costs

. This sparked debate around efficiency versus scale and intensified the broader AI rivalry between China and the US. Reactions ranged from “OpenAI and Nvidia are obsolete” to “DeepSeek’s costs must be fabricated,” but regardless of hype, the release was a milestone. It showed that architectural innovation, not just scale, could deliver frontier-level performance with far greater cost efficiency.

This open-model offensive has continued with strong contributions from other Chinese labs like Alibaba (Qwen), Kimi, and Tencent (Hunyuan), and has put competitive pressure on Meta after its open-weight Llama models fell behind. China’s recent leadership in open-weight LLMs has raised new security/IP issues with some US- and European-based organizations, though we note accessing these model weights and running the model on your own infrastructure doesn’t require sending data to China.

This brings us back to the pros and cons of open weights. While closed-API LLMs still lead at the frontier of capability, the primary advantage of open-weight models is quick and affordable local testing, unparalleled flexibility, and increased data security when run internally. Organizations can also perform

full fine-tuning

, adapting the model’s core weights and behaviors to their specific domain, language, and tasks. Open models also provide

stability and predictability

—you control the version you deploy, insulating your production systems from unexpected changes or degradations that can sometimes occur with unannounced updates to proprietary API-based models.

Public closed-model APIs from major providers benefit from immense economies of scale and highly optimized GPU utilization by batching requests from thousands of users, an efficiency that is difficult for a single organization to replicate. This often means that using a closed-source API can be cheaper per inference than self-hosting an open model. Security and compliance are also more nuanced than they first appear. While some organizations must use self-hosted models to simplify compliance with regulations like GDPR by keeping data entirely within their own perimeter, this places the entire burden of securing the infrastructure on the internal team—a complex and expensive undertaking. Top API providers also often offer dedicated instances, private cloud endpoints, and contractual agreements that can guarantee data residency, zero-logging, and meet stringent regulatory standards. The choice, therefore, is not a simple open-versus-closed binary.

The boundary between open and closed models is also becoming increasingly blurred. Open-weight models are increasingly offered via API by third-party LLM inference platforms, combining the flexibility of open models with the simplicity of hosted access. This hybrid approach often strikes a practical balance between control and operational complexity.

Leading Closed LLMs

Below, we present some key costs and metrics for leading closed-source models available via API. Many of these models have additional complexity and varied pricing including options for fast modes, thinking modes, context caching, and longer context.

We present the latest LiveBench benchmark score for each model as one measure for comparison. LiveBench is a continuously updated benchmark designed to provide a “contamination-free” evaluation of large language models by regularly releasing new questions with objective, verifiable answers. It scores models out of 100 on a diverse set of challenging tasks, with a significant focus on capabilities like reasoning, coding, and data analysis. The similar LiveBench scores between GPT-4.5 and 2.5 Flash-Lite, despite 750x input token cost variation, highlights both that smaller models are now very capable but also that not all capabilities are captured in a single benchmark!

Source: Towards AI, Company Reports,

LiveBench AI

Leading open-weight LLMs

Below, we also present key costs, the LiveBench benchmark score, and context length for leading open-weight models available via API. We compare hosted versions of these models for easy comparison. Different API providers may choose to host open-weight models with different levels of quantization, different context lengths, and different pricing, so performance can vary between providers.

Source: Towards AI, Company Reports,

LiveBench AI

Whether hosted or self-deployed, selecting a model only solves part of the problem. In practice, most of the complexity and opportunity lies in how that model is used: how it’s prompted, extended, fine-tuned, or embedded within a broader workflow. These system-level decisions often have a greater impact on performance and cost than the model choice itself.

A Practical Guide to Designing an LLM System

Simply picking the biggest or newest LLM is rarely the optimal strategy. A more effective approach starts with a deep understanding of the developer’s toolkit: knowing which technique to apply to which problem to achieve the desired capability and reliability without unnecessary cost. This is all part of the constant “

march of nines” as you develop LLM systems modularly to solve for more reliability and capability.

There is a need to prioritize the easiest wins that deliver tangible value before investing in further incremental and often costly accuracy improvements. The reality will always vary on a case-by-case basis, but here is a quick guide to navigating this process.

Step 1: Open Versus Closed?

This is often your first decision.

Go with a closed-API model (e.g., from OpenAI, Google, Anthropic) if:

Your priority is accessing the absolute state-of-the-art models with maximum simplicity.

Go with an open-weight model (e.g., Llama, Mistral, Qwen, DeepSeek) if:

Data security and compliance are paramount:

If you need to guarantee that sensitive data never leaves your own infrastructure.

You need deep customization and control:

If your goal is to fine-tune a model on proprietary data and to create a specialized expert that you control completely.

If you went open, what can you

realistically

run? Your own GPU infrastructure is a hard constraint. Assess your cluster size and memory to determine if you can efficiently run a large, leading 1 trillion+ parameter MoE model, such as Kimi K2, or if you are better served by a medium-size model such as Gemma 3 27B or a much smaller model like Gemma 3n that can even run on mobile.

Step 2: Gauging the Need for Reasoning

Does your task require the model to simply blast out a response, or does it need to

think

first?

Reasoning:

For tasks that involve complex, multistep problem-solving, brainstorming, strategic planning, intricate code generation, or deep analysis, you need a dedicated reasoning model such as o3, Gemini 2.5 Pro, DeepSeek R1, or Claude 4. In some cases these models can be used in high-reasoning mode, which encourages the model to think for longer before responding.

No reasoning:

For straightforward tasks like simple Q&A, summarization of a single document, data extraction, or classification, a powerful reasoning model is overkill.

The middle ground:

For tasks requiring moderate reasoning, such as generating a structured report from a few data points or performing basic data analysis at scale, a “mini” reasoning model, like OpenAI’s o4-mini or Gemini Flash 2.5, offers a balance of capability and cost.

Step 3: Pinpointing Key Model Attributes

Beyond general intelligence and reasoning, modern LLMs are specialists. Your choice should be guided by the specific attributes and “superpowers” your application needs.

Prioritize accuracy over cost

for high-value tasks where mistakes are costly or where a human expert’s time is being saved. o3-pro is a standout model here and it can even be used as a fact checker to meticulously check the details of an earlier LLM output.

Prioritize speed and cost over accuracy:

For user-facing, real-time applications like chatbots or high-volume, low-value tasks like simple data categorization, latency and cost are paramount. Choose a hyper-efficient “flash” or “mini” model such as Gemini 2.5 Flash-Lite. Qwen3-235B models can also be a great option here but are too complex to inference yourself.

Do you need a deep, long-context researcher?

For tasks that require synthesizing information from massive documents, entire codebases, or extensive legal contracts, a model with a vast and highly effective context window is crucial.

Gemini 2.5 Pro

excels here.

Is multimodality essential?

If your application needs to understand or generate images, process audio in real time, or analyze video, your choice narrows to models like

GPT-4o

or the

Gemini

family. For one-shot YouTube video processing, Gemini is the standout.

Is it a code-specific task?

While many models can code, some are explicitly tuned for it. In the open world, Codestral and Gemma do a decent job. But Claude has won hearts and minds, at least for now.

Do you need live, agentic web search?

For answering questions about current events or topics beyond the model’s knowledge cutoff, consider a model with a built-in, reliable web search, such as

o3.

Do you need complex

dialogue and emotional nuance?

GPT-4.5, Kimi K2, Claude Opus 4.0, or Grok 4 do a great job.

Step 4: Prompting, Then RAG, Then Evaluation

Before you dive into more complex and costly development, always see how far you can get with the simplest techniques. This is a path of escalating complexity. Model choice for RAG pipelines is often centered around latency for end users, but recently more complex agentic RAG workflows or long-context RAG tasks require reasoning models or longer context capabilities.

Prompt engineering first:

Your first step is always to maximize the model’s inherent capabilities through clear, well-structured prompting. Often, a better prompt with a more capable model is all you need.

Move to retrieval-augmented generation (RAG):

If your model’s limitation is a lack of specific, private, or up-to-date

knowledge

, RAG is the next logical step. This is the best approach for reducing hallucinations, providing answers based on proprietary documents, and ensuring responses are current. However, RAG is not a panacea. Its effectiveness is entirely dependent on the quality and freshness of your dataset, and building a retrieval system that consistently finds and uses the

most

relevant information is a significant engineering challenge. RAG also comes with many associated decisions, such as the quantity of data to retrieve and feed into the model’s context window, and just how much use you make of long-context capabilities and context caching.

Iterate with advanced RAG:

To push performance, you will need to implement more advanced techniques like hybrid search (combining keyword and vector search), re-ranking retrieved results for relevance, and query transformation.

Build custom evaluation

: Ensure iterations on your system design, additions of new advanced RAG techniques, or updates to the latest model are always moving progress forward on your key metrics!

Step 5: Fine-Tune or Distill for Deep Specialization

If the model’s core

behavior

—not its knowledge—is still the problem, then it’s time to consider fine-tuning. Fine-tuning is a significant undertaking that requires a high-quality dataset, engineering effort, and computational resources. However, it can enable a smaller, cheaper open-weight model to outperform a massive generalist model on a specific, narrow task, making it a powerful tool for optimization and specialization.

Fine-tuning is for changing behavior, not adding knowledge.

Use it to teach a model a specific skill, style, or format. For example:

To reliably output data in a complex, structured format like specific JSON or XML schemas.

To master the unique vocabulary and nuances of a highly specialized domain (e.g., legal, medical).

Some closed-source models are available for fine-tuning via API such as Gemini 2.5 Flash and various OpenAI models. Larger models are normally not available.

In open-weight models,

Llama 3.3 70B and Qwen 70B are fine-tuning staples. The process is more complex to fine-tune an open-weight model yourself.

Model

distillation

can also serve as a production-focused optimization step. In its simplest form, this consists of generating synthetic data from larger models to create fine-tuning datasets to improve the capabilities of smaller models.

Reinforcement fine-tuning (RFT) for problem-solving accuracy

Instead of just imitating correct answers, the model learns by trial, error, and correction. It is rewarded for getting answers right and penalized for getting them wrong.

Use RFT to:

Create a true “expert model” that excels at complex tasks with objectively correct outcomes.

The advantage:

RFT is incredibly data-efficient, often requiring only a few dozen high-quality examples to achieve significant performance gains.

The catch:

RFT requires a reliable, automated “grader” to provide the reward signal. Designing this grader is a critical engineering challenge.

Step 6: Orchestrated Workflows Versus Autonomous Agents

The critical decision here is how much freedom to grant. Autonomous agents are also more likely to need more expensive reasoning models with greater levels of inference scaling. Parallel inference scaling methods with multiple agents are also beginning to deliver great results. Small errors can accumulate and multiply during many successive agentic steps so the investment in a stronger more capable model can make all the difference in building a usable product.

Choose an orchestrated workflow for predictable tasks

You design a specific, often linear, sequence of steps, and the LLM acts as a powerful component at one or more of those steps.

Use when:

You are automating a known, repeatable business process (e.g., processing a customer support ticket, generating a monthly financial summary). The goal is reliability, predictability, and control.

Benefit:

You maintain complete control over the process, ensuring consistency and managing costs effectively because the number and type of LLM calls are predefined.

Build hybrid pipelines:

Often, the best results will come from combining many LLMs, open and closed, within a pipeline.

This means using different LLMs for different stages of a workflow: a fast, cheap LLM for initial query routing; a specialized LLM for a specific subtask; a powerful reasoning LLM for complex planning; and perhaps another LLM for verification or refinement.

At Towards AI, we often have 2-3 different LLMs from different companies in an LLM pipeline.

Choose an autonomous agent for open-ended problems.

You give the LLM a high-level goal, a set of tools (e.g., APIs, databases, code interpreters), and the autonomy to figure out the steps to achieve that goal.

Use when:

The path to the solution is unknown and requires dynamic problem-solving, exploration, or research (e.g., debugging a complex software issue, performing deep market analysis, planning a multistage project).

The critical risk—runaway costs:

An agent that gets stuck in a loop, makes poor decisions, or explores inefficient paths can rapidly accumulate enormous API costs.

Implementing strict guardrails is critical:

Budget limits:

Set hard caps on the cost per task.

Step counters:

Limit the total number of “thoughts” or “actions” an agent can take.

Human-in-the-loop:

Require human approval for potentially expensive or irreversible actions.

Gemini 2.5 Pro and o3 are our favourite closed-API models for agent pipelines, while in open-weight models we like Kimi K2.

Working through these steps helps translate a vague problem into a concrete implementation plan, one that’s grounded in clear trade-offs and tailored to your needs. This structured approach often yields systems that are not only more capable and reliable but also far more effective for specific tasks than a general-purpose chatbot ever could be.

Conclusion

The open-versus-closed race gives us rapid access to strong LLMs but also creates complexity. Selecting and deploying them demands both engineering discipline and economic clarity.

Developing in the LLM ecosystem demands a new level of engineering discipline and keen economic awareness. No single LLM is a cure-all. A practical, evolving toolkit is essential, but knowing which tool to pull out for which job is the real art. The challenge isn’t just picking a model from a list; it’s about architecting a solution. This requires a systematic approach, moving from high-level strategic decisions about data and security down to the granular, technical choices of development and implementation.

The success of specialized “LLM wrapper” applications like Anyscale/Cursor for coding or Perplexity for search, some of which are now valued at over $10 billion, underscores the immense value in this tailored approach. These applications aren’t just thin wrappers; they are sophisticated systems that leverage foundation LLMs but add significant value through custom workflows, fine-tuning, data integration, and user experience design.

Ultimately, success hinges on informed pragmatism. Developers and organizations need a sharp understanding of their problem space and a firm grasp of how cost scales across model choice, series and parallel reasoning, context usage, and agentic behavior. Above all, custom evaluation is non-negotiable because your use case, not a benchmark, is the only standard that truly matters.


