Retrieval and Indexing Methods

Why retrieval comes before generation

A model only knows what it saw in training. LLMs are trained on public data available at training time, so they can't answer from your private documents or from information that changes often.

Retrieval augmented generation (RAG) is the pattern that fixes this. It combines search with LLMs so that responses are grounded in your data. It follows a three-step flow:

1. Retrieve: when a user asks a question, your application queries an index or data store to find relevant content.
2. Augment: the app combines the user's question and the retrieved content, the grounding data, into a prompt.
3. Generate: the model receives the augmented prompt and generates a response grounded in the retrieved content, which reduces inaccuracies and enables citations.

Two terms recur through this lesson. Grounding data is retrieved content you give the model to reduce guessing. Embeddings are numeric representations of content, used for vector similarity search.

What an index is for

RAG works best when you can retrieve relevant content quickly and consistently. An index is a data structure optimized for retrieval, and it organizes your content for efficient retrieval.

Many RAG solutions use an index that supports one or more retrieval modes: keyword search, semantic search, vector search, and hybrid search, which is keyword plus vector, sometimes with semantic ranking. An index can also store fields that improve citation quality, such as document titles, URLs, or file names.

Azure AI Search is a recommended index store for RAG scenarios. It supports retrieval over vector and textual data in search indexes, and it can query other targets when you use agentic retrieval. Foundry connects your project to an Azure AI Search service and index; depending on the feature and API, that connection appears as a project connection or an index asset ID.

Query types in Azure AI Search

All queries in Azure AI Search run over a search index that stores searchable content. The query form decides what part of the index is searched.

In words: full-text queries iterate over inverted indexes of tokenized terms. Vector queries iterate over vector fields. Hybrid search combines text and vector search in a single query request. Agentic retrieval is an alternative path whose response is designed for agent consumption.

The classic query forms still matter, but some don't belong in RAG.

• Filters use OData syntax in a filter parameter on filterable fields. Filter text isn't analyzed: it's a verbatim, case-sensitive pattern that either matches or fails.
• Autocomplete and suggestions fire on partial input in a search-as-you-type box. They're intended for human interaction and you can skip them in a RAG pattern. Facets and order-by might be useful, but they're uncommon in RAG.
• Document look-up retrieves one document by its ID, with no index scan.

Classic RAG

Classic RAG uses the original query architecture. Your application sends a single query to Azure AI Search and orchestrates the handoff to an LLM itself. Your deployed LLM then formulates an answer from the flattened result set. It's simpler, with fewer components, and faster, because no LLM takes part in query planning.

To get the most relevant grounding data from classic RAG:

• Use hybrid queries, which combine keyword and vector search for maximum recall. A text string and its vector equivalent run as parallel queries, and the best matches from each return in one unified result set.
• Add semantic ranking, which re-scores results by meaning and identifies the top 50 most relevant results. It's optional in classic RAG.
• Apply scoring profiles to boost critical fields, and tune vector weighting, minimum thresholds, and result limits such as top-k for vectors and top-n for text.

During indexing, chunking subdivides large documents so that portions can be matched independently, and a vectorization step creates the embeddings used by vector queries.

Agentic retrieval

Classic RAG uses one query. Agentic retrieval, also called agentic RAG, uses a model to break a complex input into multiple focused subqueries, run them in parallel, and return structured grounding data that works well with chat completion models.

Its advantages over classic RAG:

• Context-aware query planning uses conversation history, so follow-up questions keep the context of earlier exchanges.
• Parallel execution of subqueries gives better coverage and reduces latency.
• Structured responses return grounding data, citations, and execution metadata.
• Built-in semantic ranking filters noise.
• Optional answer synthesis can include an LLM-formulated answer, or you can return raw passages.

The pipeline needs three new objects: one or more knowledge sources, each pointing to searchable content such as a search index or a remote SharePoint site; a knowledge base, the queryable object that unifies those sources; and the retrieve action that you call from application code, for example as a tool for your agent. The knowledge base also sets the retrieval reasoning effort: minimal, low, or medium.

Agentic or classic: making the call

Azure AI Search offers both approaches, and the choice follows a short list of signals.

Use agentic retrieval when:

• Your client is an agent or chatbot.
• You need the highest possible relevance and accuracy.
• Your queries are complex or conversational.
• You want structured responses with citations and query details.
• You're building a new RAG implementation.

Use classic RAG when:

• You need generally available (GA) features only, because LLM query planning in agentic retrieval is in preview.
• Simplicity and speed are priorities over advanced relevance.
• You have existing orchestration code you want to preserve.
• You need fine-grained control over the query pipeline.

For new RAG implementations, the recommendation is to start with agentic retrieval. And a RAG solution with agents can use Foundry IQ as the agent's single endpoint to a knowledge layer; Foundry IQ uses agentic retrieval.

Preparing content for the index

RAG quality depends on how you prepare content. A RAG implementation in Foundry typically covers five stages: prepare your data (organize and chunk it), set up an index, connect your Foundry project to the index, build the RAG application with the Foundry SDK or REST APIs, and test and evaluate retrieval quality and citations.

Azure AI Search helps with the hard parts of preparation:

In words: large documents are chunked automatically, images and PDFs go through OCR and image verbalization, and integrated vectorization creates embeddings without a separate pipeline.

The two approaches build that pipeline differently. For classic RAG, you use indexers and skillsets to build custom pipelines, or you push pre-processed content through the push API. Indexers pull from more than 10 Azure data sources, and incremental indexing keeps content fresh. For agentic retrieval, knowledge sources auto-generate the chunking and vectorization pipelines.

RAG, fine-tuning, or an agent tool

Foundry supports several patterns for private data, and they solve different problems.

• Use RAG when you need answers grounded in private or frequently changing data.
• Use fine-tuning when you need to change model behavior, style, or task performance, rather than add fresh knowledge.
• Use agent tools when you're building an agent that needs retrieval as a tool, such as the file search tool. To build a full RAG app with complete control instead, use the Foundry SDK.

The tell is the word "fresh". Knowledge that changes weekly goes into an index, not into model weights.

Security, cost, and latency

RAG can expose sensitive content if access and prompting aren't designed carefully.

• Apply access control at retrieval time. With Azure AI Search as a data source, use document-level access control with security filters, so users only retrieve authorized content.
• Prefer Microsoft Entra ID over API keys for production. API keys are convenient in development but aren't recommended for production.
• Treat retrieved content as untrusted input. System messages and app logic should reduce the risk of prompt injection from retrieved passages.

RAG also adds work compared with a model-only request: retrieval round trips, embeddings at indexing time and often at query time, and more input tokens from retrieved passages. When the token budget runs out, filter, rank, or summarize passages. When retrieval quality is poor, review chunking, the embedding model, and the search configuration.

Key takeaways

• RAG retrieves, augments, and generates, so answers are grounded in your data and can cite it.
• Full-text queries search inverted indexes, vector queries search vector fields, and hybrid queries combine both in one request for maximum recall.
• Classic RAG sends a single query and leaves orchestration to your app; it fits GA-only, simple, existing pipelines.
• Agentic retrieval plans parallel subqueries over a knowledge base and returns structured responses; start new RAG implementations there.
• Use RAG for fresh or private knowledge and fine-tuning for behavior or style.
• Enforce access control at retrieval time, and skip autocomplete and suggestions in RAG.