Matching Foundry Services to Tasks

A map from goal to starting point

Microsoft Foundry covers a wide range of features, and the hardest part is often knowing where to start. The fastest way through is to name the task first: generate text, ground answers, search vectors, run an agent workflow, or process speech and images. Each of those has a natural starting service.

Each capability in Foundry is a family, and some are marked (preview). Preview capabilities aren't recommended for production workloads, and preview status can vary by feature, region, and API version. Keep that in mind on every row of this lesson: the right service for a pilot can differ from the right service for production.

One more habit helps. Each option comes with a boundary, the point where another option is a better match. Learning the boundaries is what lets you choose, not only recognize names.

Generative tasks: start with a model

When the task is to send a prompt to a model, with no agent or tools, start with the quickstart for building with models and agents. It's the shortest path to a working call. Add an agent later only if you need tools or orchestration.

The services around that path:

• Foundry Models is the catalog you use to choose and deploy the model behind your app or agent.
• Playgrounds let you try models, prompts, and agents interactively before you write code.
• The model router routes requests automatically to the best model for cost and quality.
• Instant access models (preview) let you use selected models without provisioning a deployment first.

Two API surfaces carry generative calls from code:

• The Responses API is a stateful API for model and agent interactions.
• The Foundry v1 REST API is the OpenAI-compatible surface for chat, responses, embeddings, files, and fine-tuning.

So a team that only needs completions from a model doesn't start with an agent, a workflow, or a toolbox. It deploys a model and calls it, and it adds the agent layer when tools or orchestration become necessary.

Fine-tuning or retrieval

When quality isn't good enough, two levers look similar but fix different problems.

Fine-tuning customizes a model on your own data. Use it when prompting and retrieval can't reach the quality you need. Foundry Models points you to fine-tuning when a model needs your domain data.

But the boundary is sharp: if the gap is missing context, use retrieval instead. A model that answers badly because it has never seen your current product catalog doesn't need new training; it needs the right content supplied at query time.

Grounding answers in your content

Grounding means the agent answers from real content instead of guessing. Foundry offers several grounding services, and they differ by where the content lives.

• Foundry IQ provides agentic retrieval over a knowledge base, including private networking. Use it to ground answers in your content.
• File search grounds answers in uploaded files through managed vector stores. If you only need a few uploaded files, file search is lighter weight than Foundry IQ.
• Web search and Grounding with Bing ground answers in current web results.
• Azure AI Search integration uses Azure AI Search indexes as agent knowledge.
• Fabric IQ (preview) reasons over Microsoft Fabric data and semantic models, and Work IQ (preview) reasons over Microsoft 365 work data.

Retrieval-augmented generation (RAG) is the umbrella for these patterns: grounding model responses in your own content.

Vector search for agents

Vector search is how an agent finds passages by meaning, and in Foundry it hides behind familiar names.

Vector stores provide managed storage and indexing for file search. When you attach file search to an agent, the uploaded files are grounded through those managed vector stores, so you don't build or run the vector index yourself.

When your organization already runs Azure AI Search, the Azure AI Search integration lets an agent use those existing indexes as knowledge instead of copying content into new stores.

And when your code needs the vectors themselves, the Foundry v1 REST API includes embeddings in its OpenAI-compatible surface.

Agent workflows: choosing the build

Agents come in levels, and each level has a boundary.

• Prompt agents are declarative agents defined by instructions, a model, and tools. Use one when you want an agent with no runtime to manage.
• Hosted agents run your own agent code or framework in a Foundry-managed runtime. Use one when you need full control of agent logic and dependencies. If instructions and tools are enough, prompt agents are simpler.
• Workflows coordinate multiple agents and steps into a single orchestrated process. Use one when one task spans several agents or stages. If a single agent with tools can finish the job, stay simpler.

All of them run on Foundry Agent Service, which builds, runs, and operates agents as a managed service. For existing code, LangGraph, Microsoft Agent Framework, and Semantic Kernel map onto Foundry hosting.

A few neighbors sit around the agent itself. Agent-to-agent (A2A) lets agents call other agents across services and vendors. Routines package repeatable agent procedures for reuse. The Toolbox is a managed endpoint that packages the tools an agent can call, and Memory (preview) gives persistent recall across sessions.

Multimodal processing

When the task involves speech or images, Foundry has dedicated entry points.

• Voice-based prompt agents are prompt agents that speak and listen in real time, with managed voice orchestration. Use them when you want a spoken experience without hosting a speech pipeline. If you only need text responses, use prompt agents.
• Voice agents add speech input and output to agents.
• Foundry Tools for Speech, Language, and Translator add speech, language understanding, PII detection, and translation to an agent.
• The image generation tool (preview) lets an agent generate images.
• Healthcare AI models are domain-specific models for medical imaging and reporting scenarios.

Two more preview tools let an agent act on visual interfaces: browser automation navigates and acts on websites, and computer use operates a virtual computer interface.

Key takeaways

• Start a pure prompt-and-reply task with a model call; add an agent only for tools or orchestration.
• Use retrieval when the model lacks context, and fine-tuning when prompting and retrieval can't reach the quality you need.
• Foundry IQ grounds answers over a knowledge base; file search is lighter for a few uploaded files; Grounding with Bing covers current web results.
• Vector stores provide managed storage and indexing for file search.
• Prompt agents need no runtime, hosted agents run your own code, and workflows coordinate several agents or stages.
• Voice-based prompt agents give a spoken experience without hosting a speech pipeline; image generation is a preview agent tool.