Deep Learning Inference Optimization: What to Tackle First When Training Is Done but Serving Is Slow

A model that ran well in a research environment hits a different set of problems the moment it goes into production. It’s not accuracy—it’s response time and GPU costs. Both usually stem from the same root cause: the GPU is sitting idle, or it’s doing work it doesn’t need to do.

Optimization has an order. Start with the highest impact and the easiest to revert.

Step 1: Profiling — No Optimization Without Finding the Bottleneck

The most common mistake is skipping this step. Inference latency is the sum of multiple pieces.

• Request parsing and preprocessing (image decoding, resizing, normalization)
• CPU → GPU data transfer
• Model forward pass
• GPU → CPU result retrieval
• Post-processing and response serialization

It is not uncommon for preprocessing and transfers to account for over half the time. Quantizing the model in this state will barely reduce the total time.

Use the PyTorch profiler to view CPU and GPU activities together.

import torch
from torch.profiler import profile, ProfilerActivity

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    record_shapes=True,
) as prof:
    for _ in range(10):
        with torch.no_grad():
            model(sample_input)

print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=20))

Look for two things: the ratio of time the GPU kernels are actually running, and the periods where the GPU is idle. If the latter is large, the data supply cannot keep up, and that isn’t a model problem.

One thing to watch out for when measuring: GPU operations are asynchronous, so simple time measurements will return shorter times than reality. Synchronize with torch.cuda.synchronize() before measuring, or use the profiler. And discard the first few runs as warm-up.

Step 2: Lower Precision (FP16 / INT8)

If you confirm the bottleneck lies in model computation, start here. This is where you get the biggest impact with the least effort.

FP32 → FP16 is usually the first choice. Memory usage drops in half, tensor cores on modern GPUs accelerate the math, and the accuracy loss is negligible for most vision and language models.

model = model.half().eval().cuda()
with torch.no_grad():
    out = model(x.half().cuda())

INT8 quantization reduces it further, but accuracy loss can be noticeable. Therefore, you must always compare accuracy before and after quantization using validation data. Skipping this check often leads to discovering degraded service quality much later.

Going a step further, you can export to ONNX and run on an inference-specific runtime. Operator fusion and graph optimization apply, making it run faster on the same hardware.

# Example of building an ONNX model into an FP16 engine
trtexec --onnx=model.onnx --saveEngine=model.plan --fp16

During building, fixing the input size expands the room for optimization. If variable sizes are required, narrow the range by specifying minimum, optimal, and maximum sizes.

Step 3: Dynamic Batching — Keep the GPU Busy

Processing single requests one by one keeps GPU utilization low because GPUs are efficient when processing multiple inputs at once.

Dynamic batching gathers incoming requests over a short window and processes them in a single batch. Individual requests see a slight increase in wait time, but overall throughput goes up significantly.

Using a dedicated inference server (like Triton) allows you to enable this via configuration.

max_batch_size: 32

dynamic_batching {
  preferred_batch_size: [ 8, 16 ]
  max_queue_delay_microseconds: 2000
}

max_queue_delay is the key knob. Setting it higher increases the batch size and throughput, but also increases latency. A practical starting point is to allocate about 10 to 20% of the latency allowed by your SLA to this queue wait.

If your structure mounts the model directly onto a general web framework like FastAPI, switching to a dedicated inference server alone will drastically change throughput. However, because operational complexity increases, you should make this move only when actual traffic warrants it.

Step 4: Infrastructure — Utilization Over Specifications

The last part is hardware selection. A common misconception here is that “a more expensive GPU is the answer.”

• If you upgrade to a higher-end GPU while GPU utilization is low, costs only go up while latency remains the same. Raise utilization first through batching and pipelines.
• If the model barely fits in VRAM, switching to one with larger memory allows you to increase the batch size and throughput.
• If traffic is irregular, auto-scaling or serverless inference is more cost-effective than keeping a GPU instance running constantly.

Also, consider moving preprocessing to the GPU. Doing image decoding and resizing on the CPU often becomes the bottleneck. Shifting to GPU-based preprocessing reduces the “GPU idle periods” found in Step 1.

Summary Checklist

1. Profiling — Confirm whether the bottleneck is the model, preprocessing, or data transfer
2. Apply FP16 and check accuracy
3. ONNX export + inference runtime if needed
4. Raise GPU utilization with dynamic batching
5. Optimize preprocessing (GPU migration, asynchronous loading)
6. Then, adjust instance specs
7. INT8 only if it passes accuracy validation

Doing it in reverse order (buying an expensive GPU first, profiling later) results in spending money while leaving problems unsolved.

⚠️ Optimal settings vary by library version and hardware generation. This post covers the approach order, and actual figures must always be measured and verified on your own model and environment.