Prepstellar
Get the app

AIF-C01 · AI and ML Foundations

21 cards

Inference Modes

Swipe, scroll or use ← →

  1. Four ways to serve one model

    Training produces a model; serving it is a separate decision, and the same model can be deployed in more than one way. SageMaker AI offers four options to deploy models for inference:

    • Real-time inference — interactive workloads with low-latency requirements.
    • Batch transform — offline inference with large datasets.
    • Asynchronous inference — near-real-time inference with large inputs that require longer preprocessing times.
    • Serverless inference — workloads that have idle periods between traffic spurts.

    Choosing between them is not a matter of taste. Five properties of the workload decide it: the interaction pattern, the size of the dataset, the size of each input, how long processing takes, and whether traffic is continuous or intermittent.

    1 / 21

  2. Four ways to serve one model

    Read the four options as answers to two questions: does a caller wait for the response? and how big and how slow is one request?

    Option Designed for Waits for a response?
    Real-time Interactive, low-latency requests Yes, immediately
    Batch transform Offline inference over large datasets No
    Asynchronous Large inputs and longer preprocessing Yes, but not immediately
    Serverless Traffic with idle periods between spurts Yes, immediately

    Only batch transform is an offline job over an accumulated dataset. The other three are request-response modes, and they differ by how much they can carry and how long they can take.

    2 / 21

  3. Quick check

    Which option is designed for offline inference over large accumulated datasets?

    1. AServerless inference

      Serverless inference is a request-response mode for traffic that arrives in spurts with idle periods in between.

    2. BAsynchronous inference

      Asynchronous inference handles large inputs, but it stays request-oriented and near-real-time rather than offline.

    3. CBatch transform

      Right. Batch transform is the offline option for inference over large datasets, with no live caller waiting.

    3 / 21

  4. Real-time inference

    Real-time inference serves inference workloads with real-time, interactive, low-latency requirements: a recommendation on a product page, a fraud score during checkout, a chatbot reply.

    Its documented boundaries are tight because the caller is waiting:

    Property Real-time inference
    Payload size Less than 6 MB
    Request timeout Less than 60 seconds
    Autoscaling Supported
    Scale to zero Not supported

    That last row matters for cost. Real-time endpoints can scale out and back in with demand, but capacity never drops to nothing, so an idle endpoint still costs money. Real-time inference is also the option that supports multi-model and multi-container endpoints, so several models can sit behind one endpoint.

    4 / 21

  5. Quick check

    Which statement matches real-time inference?

    1. AIt serves interactive, low-latency requests, with autoscaling but no scale to zero

      Right. Real-time inference targets interactive low-latency traffic; it supports autoscaling, but scale to zero is not among its supported features.

    2. BIt is the offline mode, so its capacity drops to zero between accumulated dataset jobs

      Real-time inference is the interactive mode, not the offline one, and it is precisely the mode that cannot scale to zero.

    3. CIt accepts single requests up to 1 GB and allows almost an hour of processing

      Those are the asynchronous inference boundaries; real-time payloads stay below 6 MB and requests below 60 seconds.

    5 / 21

  6. Batch transform

    Batch transform performs offline inference with large datasets. Nobody is waiting for a single response: records accumulate, a job runs over them, and results are written out.

    Property Batch transform
    Payload size 100 MB or less
    Request timeout Days
    Autoscaling Not applicable
    Scale to zero Not applicable

    Autoscaling and scale to zero are marked not applicable rather than unsupported, and the difference is meaningful: there is no always-on endpoint to scale, because the job exists only while it runs. Nightly scoring of a day's transactions, monthly churn scores for a whole customer base, and one-off enrichment of an archive are its natural work.

    6 / 21

  7. Quick check

    A team has an accumulated offline dataset, individual payloads reach 80 MB, and processing may take several days. No interactive response is needed. Which option fits?

    1. AAsynchronous inference, the offline dataset mode whose jobs can last for several days

      Asynchronous inference is request-oriented, and its requests must finish in under one hour.

    2. BReal-time inference, because 80 MB stays inside its interactive payload boundary

      Real-time payloads must stay below 6 MB, so an 80 MB record is far outside its limit.

    3. CBatch transform: offline, payloads up to 100 MB, and jobs that can run for days

      Right. All three requirements — offline dataset, 80 MB payload, multi-day duration — sit inside the batch transform boundaries.

    7 / 21

  8. Keep your progress in the app

    That’s 3 of 10 quick checks. In the app they stay answered, and every lesson remembers where you left off.

    Get the app Keep reading here

  9. Asynchronous inference

    Asynchronous inference provides near-real-time inference for large inputs that require longer preprocessing times. The caller still submits a request and still gets a result back, but the result does not arrive within a single interactive round trip.

    Property Asynchronous inference
    Payload size 1 GB or less
    Request timeout Less than 1 hour
    Autoscaling Supported
    Scale to zero Supported

    This is the escape hatch for one big, slow request: a long video to analyze, a high-resolution scan, a large document to process end to end. It is the strongest fit when a single input is too large or too slow for the interactive modes, yet the result is needed sooner than an offline job would deliver it.

    8 / 21

  10. Quick check

    An application sends one 700 MB request, preprocessing may take 40 minutes, and the answer is needed sooner than an offline job would give it. Which option fits?

    1. AServerless inference, which accepts up to 1 GB per request

      Serverless payloads stop at 4 MB and its requests must finish in under 60 seconds, so 700 MB is out of range.

    2. BAsynchronous inference: up to 1 GB per request and under one hour of processing

      Right. Both constraints exceed the interactive limits yet stay inside the 1 GB payload and one-hour request boundaries.

    3. CBatch transform, the interactive mode for a single long-running request

      Batch transform is the offline mode over accumulated datasets, not an interactive path for one request.

    9 / 21

  11. Serverless inference

    Serverless inference targets inference workloads that have idle periods between traffic spurts: an internal tool used in bursts during office hours, a seasonal form, an API that goes quiet at night.

    Property Serverless inference
    Payload size 4 MB or less
    Request timeout Less than 60 seconds
    Autoscaling Supported
    Scale to zero Supported
    GPU support Not listed in the feature matrix

    Serverless inference is interactive like real-time inference, but it can scale to zero between spurts, so idle time costs nothing. The trade-offs are a smaller payload ceiling — 4 MB rather than 6 MB — and no GPU support in the documented feature matrix.

    10 / 21

  12. Quick check

    A prediction API receives 3 MB requests that finish in 20 seconds, but traffic arrives in short spurts separated by long idle gaps, and the team wants capacity to reach zero in between. Which option fits best?

    1. AAsynchronous inference, since a request under 4 MB requires its one-hour timeout window

      A 20-second request does not need an hour, and asynchronous inference is meant for inputs too large or slow for the interactive modes.

    2. BReal-time inference, which drops to zero capacity whenever traffic goes idle

      Real-time inference supports autoscaling but never scales to zero, so idle capacity keeps costing money.

    3. CServerless inference, because the requests fit its limits and the traffic goes quiet

      Right. Both 3 MB and 20 seconds sit inside the serverless boundaries, and idle periods between spurts are exactly its stated workload profile.

    11 / 21

  13. The size and time limits side by side

    Payload and timeout are the two numbers that decide most scenario questions, so learn them as one table rather than four separate facts.

    Option Payload size Request timeout
    Real-time Less than 6 MB Less than 60 seconds
    Batch transform 100 MB or less Days
    Asynchronous 1 GB or less Less than 1 hour
    Serverless 4 MB or less Less than 60 seconds

    Two patterns make the table easier to hold. Asynchronous inference carries the largest single request, and serverless inference the smallest. Duration climbs in three steps: under a minute for the interactive modes, under an hour for asynchronous, and days for batch.

    12 / 21

  14. Quick check

    Which set of payload boundaries is correct?

    1. AReal-time under 6 MB, batch 100 MB, asynchronous 1 GB, serverless 4 MB

      Right. Each option has its own ceiling, with asynchronous carrying the largest single request and serverless the smallest.

    2. BReal-time up to 100 MB, batch up to 4 MB, asynchronous below 6 MB, and serverless up to 1 GB

      These values are shuffled: an interactive real-time request cannot carry 100 MB, and serverless stops at 4 MB rather than 1 GB.

    3. CReal-time up to 1 GB, batch below 6 MB, asynchronous up to 4 MB, and serverless as high as 100 MB

      Real-time is the tightest interactive limit rather than the largest, and batch accepts far more than 6 MB.

    13 / 21

  15. The size and time limits side by side

    A duration mistake is just as costly as a size mistake, because a 40-minute job and a 40-second call belong to different options.

    Duration needed Option
    Under 60 seconds Real-time or serverless
    Under 1 hour Asynchronous
    Days Batch transform

    So the same 3 MB input can belong to three different options depending on how long it takes and who is waiting: a fast interactive call, a spurt-driven call that should cost nothing when idle, or one row inside an overnight job.

    14 / 21

  16. Quick check

    How do the documented request-duration boundaries compare?

    1. AReal-time can run for days, serverless under an hour, and asynchronous and batch under 60 seconds

      Real-time is the shortest of all, not the longest; days belong to batch transform.

    2. BReal-time is under an hour, batch under 60 seconds, and asynchronous and serverless can run for days

      Batch transform is the mode measured in days, and neither asynchronous nor serverless runs that long.

    3. CReal-time and serverless under 60 seconds, asynchronous under an hour, batch for days

      Right. The progression goes from short interactive calls, to longer asynchronous requests, to offline jobs that may last for days.

    15 / 21

  17. Scaling, cost, and hardware

    Beyond size and time, the feature matrix separates the options by how their capacity behaves — which is where the cost of idle time is decided.

    Option Autoscaling Scale to zero GPU support
    Real-time Supported Not supported Supported
    Batch transform Not applicable Not applicable Supported
    Asynchronous Supported Supported Supported
    Serverless Supported Supported Not listed

    Asynchronous and serverless inference are the two options that can scale to zero. Real-time inference cannot, and for batch transform the question does not apply, because a job holds resources only while it runs. Serverless inference is also the one option without GPU support in the matrix, so a GPU-bound model rules it out however bursty the traffic is.

    16 / 21

  18. Quick check

    Which pair of options can scale to zero?

    1. AReal-time inference and batch transform

      Real-time inference does not scale to zero, and for batch transform scaling is not applicable at all.

    2. BAsynchronous and serverless inference modes

      Right. Both asynchronous and serverless inference support scaling to zero, so idle periods cost nothing.

    3. CReal-time inference and serverless inference

      Serverless inference does scale to zero, but real-time inference keeps capacity running even when traffic stops.

    17 / 21

  19. A decision sequence you can reuse

    Work through the workload in the same order every time, and the four options separate themselves.

    1. Is a caller waiting at all? If not — an accumulated dataset, results written for later — choose batch transform.
    2. Is one request too large or too slow for an interactive call? More than a few megabytes, or minutes rather than seconds, points to asynchronous inference.
    3. Does traffic arrive in spurts with idle gaps, and should idle time cost nothing? Choose serverless inference, provided each request stays under 4 MB and 60 seconds and needs no GPU.
    4. Otherwise, is the traffic steady and latency-sensitive? Choose real-time inference.

    The order matters: checking the offline case first removes batch transform, and checking size and duration next removes the option that a payload limit would have blocked anyway.

    18 / 21

  20. Quick check

    Which sequence of questions separates the four options most reliably?

    1. AAsk about hardware first, then payload size, then whether the endpoint has a name

      Hardware and endpoint naming do not separate the options; interaction pattern, input size, and duration do.

    2. BAsk if the work is offline, then if one request is too large or slow, then if traffic is bursty

      Right. Removing the offline case, then the oversized or slow request, then the bursty traffic leaves real-time inference for steady low-latency work.

    3. CAsk whether traffic is bursty first, then choose batch transform for anything that exceeds 4 MB in one request

      Batch transform is chosen because the work is offline, not because one request happens to exceed a serverless payload limit.

    19 / 21

  21. Key takeaways

    • Batch transform is the offline option: large datasets, payloads of 100 MB or less, and jobs that can run for days, with no interactive endpoint to scale.
    • Real-time inference is the interactive default: payloads below 6 MB, requests below 60 seconds, autoscaling but never scale to zero.
    • Asynchronous inference absorbs the big, slow request: up to 1 GB, under one hour, with autoscaling and scale to zero.
    • Serverless inference fits intermittent traffic: up to 4 MB, under 60 seconds, scale to zero, and no GPU support in the feature matrix.
    • Decide by workload, not by habit: interaction pattern, dataset size, input size, processing time, and traffic continuity pick the option for you.
    20 / 21

  22. Quick check

    Which summary of the four options is correct?

    1. ABatch runs offline for days, real-time serves fast small requests without scaling to zero, asynchronous carries up to 1 GB, and serverless suits bursty traffic

      Right. Each clause matches the documented profile of its option, including the one mode that cannot scale to zero.

    2. BBatch is interactive, real-time scales to zero, asynchronous is limited to 4 MB, and serverless is the offline dataset option

      Every clause is inverted: batch is offline, real-time cannot scale to zero, asynchronous accepts up to 1 GB, and serverless is interactive.

    3. CBatch handles bursty traffic, real-time carries 1 GB inputs, asynchronous must finish in 60 seconds, and serverless runs for days

      These limits belong to other options: 1 GB and the one-hour window are asynchronous, and only batch transform runs for days.

    21 / 21

  23. 10 quick checks · then the test

    In the app, finishing the quick checks opens this lesson’s 10-question test, and the ones you miss come back exactly when you’re about to forget them.

    Take this lesson’s test → Next lesson: Data Types for AI ModelsGet the app
← Previous Keep your progress · Get the app Next card →

The whole course, on your phone

Lessons you can read, audio you can listen to on the way to work, and practice that remembers what you got wrong.

Get the app