Choosing LLMs, Small Models, and Multimodal Models

Start from the task, not from a model name

Picking a model is the first architectural decision in a Foundry app, and it is yours to make. For every model in the catalog, selecting an appropriate model for your use case is one of the responsibilities that stays with you, together with reviewing the model cards and complying with the law.

The choice is wide. The Foundry catalog includes over 10,000 models, with approximately 50 new models published each month. These models range from foundation models, reasoning models, small language models, multimodal models, domain-specific models, and industry models.

So the useful question is never "which model is best?". It is "which model fits this task, at this speed, at this cost, and in this place?". The rest of this lesson gives you the criteria to answer it.

The catalog gives you tools to narrow the list before you write any code:

In words: filter by inference task for the kind of work the model must do, by supported features when you need reasoning or tool calling, and by lifecycle to see whether a model is in preview, generally available, or deprecated. The Industry filter narrows by the industry-specific data used to train models, such as health, retail, or manufacturing. Then compare the candidates that remain with the leaderboard and the Benchmarks tab of each model card.

Reasoning model or fast responder

Two flagship OpenAI models in Foundry show the main trade-off when you choose a large language model. GPT-5 is a reasoning model. GPT-4.1 is a non-reasoning, fast response model.

GPT-5 is optimized for advanced enterprise use cases such as code generation and review, agentic tool calling, and business research. It excels at structured reasoning, multistep logic, and planning, but it might introduce higher latency because of its reasoning depth.

GPT-4.1 is optimized for high-speed, high-throughput applications such as real-time chat, customer support, and lightweight summarization. It doesn't offer the deep reasoning of GPT-5, but it delivers fast, concise responses with low latency and cost efficiency.

Read the table this way. GPT-5 takes up to 272K input tokens and returns up to 128K output tokens, 400K in total. GPT-4.1 handles 128K tokens in its short-context form and up to 1M tokens in its long-context form. Each family also comes in mini and nano variants.

The scenarios make the rule concrete. A research report that users are willing to wait many minutes for suits GPT-5 with medium or high thinking. A customer service assistant that answers live questions from an efficient search index needs to be fast, and for that scenario GPT-4.1 is a far better option.

How much a reasoning model thinks

GPT-5 is the first OpenAI model with four adjustable levels of thinking, set through reasoning effort. The level controls how much time and how many tokens the model spends before it answers, so it is also a dial for latency and cost.

In words: Medium is the default, the balanced and safe general-purpose choice. Minimal is the fastest and cheapest setting, and the least accurate on complex tasks. High gives the deepest reasoning at the highest latency and cost.

One limit catches people out: parallel tool calls aren't supported at Minimal reasoning effort. If an agent needs parallel tool use, choose Low, Medium, or High.

The pattern applies to GPT-5, GPT-5-mini, and GPT-5-nano. Absolute latency and cost scale down with the mini and nano variants, but the trade-offs between effort levels stay the same.

And if you're unsure which model to choose, the model router in Foundry Models is a ready-to-use option. It helps you get the most from the GPT-5 family and other models while saving up to 60 percent on inferencing cost with comparable quality. Model router for Foundry is in preview.

Small models that run on the device

Some tasks can't go to the cloud at all. For those, there is Foundry Local, an end-to-end local AI solution for shipping applications that run entirely on your device.

Use Foundry Local when you want to run AI locally, work offline, minimize latency, or keep data private. Your data never leaves the device, responses start immediately with zero network latency, and your app works offline. There are no per-token costs and no backend infrastructure to maintain.

Its catalog is deliberately limited. It's curated to models optimized for on-device use, tested across a range of consumer hardware, and small enough to distribute to end users. It covers chat completions (for example, GPT OSS, Qwen, DeepSeek, Mistral, and Phi) and audio transcription (for example, Whisper).

How Foundry Local runs inside your app:

• The runtime adds approximately 20 MB to your application package and runs inference through ONNX Runtime.
• It detects the hardware and accelerates inference on GPUs and NPUs when available, falling back to the CPU.
• Models download automatically on first use and are cached locally for later launches. Download them before travel if the app must work offline at the destination.
• It supports OpenAI request and response formats, so an app that already uses the OpenAI SDK can point to a Foundry Local endpoint with minimal code changes.
• The SDK comes in C#, JavaScript, Rust, and Python and runs inference in-process. An optional local web server exists for serving several processes or for tools like LangChain.

What Foundry Local is not

Foundry Local is optimized for hardware-constrained devices where a single user accesses the model at a time. You can technically install it on server hardware, but it isn't designed as a server inference stack.

Server runtimes such as vLLM or Triton Inference Server handle concurrent request queuing, continuous batching, and GPU sharing across many clients. Foundry Local doesn't provide those capabilities. If you need to serve models to multiple concurrent users, use a dedicated server inference framework.

Three more facts settle common doubts:

• No Azure subscription is required, because it runs entirely on local hardware.
• Prompts and outputs are processed locally. The network is still used to download model files and execution providers the first time a model runs, and for optional diagnostics if a user chooses to share logs.
• It supports Windows, macOS (Apple silicon), and Linux.

Models that read more than text

A multimodal model works with more than one kind of input. In the Foundry catalog, multimodal models are their own category, next to large and small language models.

The Claude family from Anthropic is a concrete example. These large language models support text and image input, text output, multilingual capabilities, and vision. They are models from partners and community, not models sold by Azure.

Before you commit, confirm what a candidate actually accepts. The model card's Details tab describes the model, its version information, and its supported data type. The Inference tasks filter narrows the catalog to task types such as chat completion, audio generation, or embeddings.

On the device side, the Foundry Local catalog includes audio transcription models such as Whisper, and Foundry Local keeps sensitive audio, text, and images on the device.

Models sold by Azure or by partners

The catalog has two main categories, and the category changes who supports you.

Models sold by Azure are hosted and sold by Microsoft under the Microsoft Product Terms. They come with Microsoft support, enterprise-grade SLAs, deep integration with Azure services, and Microsoft's internal Responsible AI review.

Models from partners and community make up the vast majority of the catalog. Their providers support them, with varying levels of SLA and support, and they offer rapid access to specialized innovations and niche capabilities.

A review isn't a guarantee. The Responsible AI reviews of models sold by Azure don't guarantee mitigation of potential risks, so you still review the model documentation and run your own evaluations before deployment.

Putting the choice together

The order of the questions matters. Where the model must run comes first, because a device-only or offline requirement removes every cloud option at once. The type of input comes next: if users send images, only a model that supports image input can do the job. Only then does reasoning depth decide between a reasoning model and a fast responder.

Take a field app for technicians at sites with no connectivity, where inspection notes are confidential. Download and cache its model files before travel. Foundry Local then fits, with no Azure subscription and no per-token cost. A cloud deployment, serverless or on managed compute and however small the model, still needs the network.

Key takeaways

• Selecting an appropriate model for your use case is your responsibility, across a catalog of over 10,000 models.
• GPT-5 is a reasoning model for complex, multistep work, with higher latency; GPT-4.1 is a non-reasoning, fast response model for real-time chat and high throughput.
• GPT-5 reasoning effort is Minimal, Low, Medium (the default), or High, and parallel tool calls aren't supported at Minimal.
• Foundry Local runs small, curated models on the user's own device: offline, private, no per-token cost, no Azure subscription, and single-user by design.
• A multimodal model such as the Claude family takes text and image input and returns text; confirm supported data types in the model card's Details tab.
• Models sold by Azure bring Microsoft support and enterprise SLAs; partner and community models are supported by their providers.