Deep Learning Inference Optimisation: Training Is Done but Serving Is Slow — Where Do You Start?

The moment a model that ran fine in a research environment is attached to a service, a different problem starts. Not accuracy but response time and GPU cost. Both usually come from the same root: the GPU is idle, or it is doing work it does not need to do.

Optimisation has an order — biggest effect first, and easiest to undo first.

Step 1: profiling — no bottleneck, no optimisation

The most common mistake is skipping this step. Inference latency is a sum of several pieces.

• Request parsing and preprocessing (image decode, resize, normalise)
• CPU → GPU data transfer
• The model’s forward pass
• GPU → CPU result retrieval
• Postprocessing and response serialisation

It is not unusual for preprocessing and transfer to account for more than half. In that state, quantising the model barely moves total time.

Use the PyTorch profiler to see CPU and GPU activity together.

import torch
from torch.profiler import profile, ProfilerActivity

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    record_shapes=True,
) as prof:
    for _ in range(10):
        with torch.no_grad():
            model(sample_input)

print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=20))

Two things to look at: the share of time GPU kernels are actually running, and the intervals where the GPU is idle. If the latter is large, data supply is not keeping up — and that is not a model problem.

One caution when measuring. GPU operations are asynchronous, so timing them naively reports shorter than reality. Synchronise with torch.cuda.synchronize() before measuring, or use the profiler. And throw away the first few runs as warm-up.

Step 2: lower the precision (FP16 / INT8)

Once you have confirmed the bottleneck is in model compute, start here. This is the band with the biggest effect for the least effort.

FP32 → FP16 is usually the first choice. Memory use halves, tensor cores on modern GPUs accelerate it, and the accuracy loss is negligible for most vision and language models.

model = model.half().eval().cuda()
with torch.no_grad():
    out = model(x.half().cuda())

INT8 quantisation cuts further but the accuracy loss can be noticeable. So always compare accuracy before and after quantisation on validation data before moving on. Skipping that check and discovering degraded service quality later is a common story.

A step further, there is exporting to ONNX and running on a dedicated inference runtime. Operator fusion and graph optimisation make it faster on the same hardware.

# ONNX 모델을 FP16 엔진으로 빌드하는 예
trtexec --onnx=model.onnx --saveEngine=model.plan --fp16

Fixing the input size at build time widens the room for optimisation. If you need variable sizes, narrow the range by specifying minimum, optimal and maximum shapes.

Step 3: dynamic batching — do not leave the GPU idle

Handling single requests one at a time keeps GPU utilisation low, because a GPU is efficient when it processes several inputs at once.

Dynamic batching gathers requests arriving within a short window and processes them together. Individual requests wait slightly longer, but total throughput rises sharply.

With a dedicated inference server (such as Triton) it is a configuration switch.

max_batch_size: 32

dynamic_batching {
  preferred_batch_size: [ 8, 16 ]
  max_queue_delay_microseconds: 2000
}

max_queue_delay is the key dial. The longer it is, the larger the batches, the higher the throughput and the higher the latency. Spending about 10–20% of the latency your SLA allows on this wait is a practical starting point.

If your setup puts the model directly on a general web framework such as FastAPI, moving to a dedicated inference server alone changes throughput substantially. It raises operational complexity, though, so make that move when you actually have traffic.

Step 4: infrastructure — utilisation over specification

Last comes hardware choice. The frequent misconception here is that “a more expensive GPU is the answer.”

• Switching to a higher GPU while utilisation is low raises cost and leaves latency where it was. Raise utilisation with batching and pipeline work first
• If the model barely fits in VRAM, moving to more memory raises throughput by allowing bigger batches
• If traffic is irregular, autoscaling or serverless inference beats an always-on GPU instance on cost

Also consider moving preprocessing onto the GPU. Image decoding and resizing on the CPU frequently become the bottleneck. Shifting to GPU-based preprocessing shrinks the “GPU idle” intervals you found in step 1.

Summary checklist

1. Profile — determine whether the bottleneck is the model, preprocessing or transfer
2. Apply FP16, then verify accuracy
3. If needed, export to ONNX + inference runtime
4. Raise GPU utilisation with dynamic batching
5. Optimise preprocessing (move to GPU, asynchronous loading)
6. Only then adjust the instance specification
7. INT8 only if it passes accuracy validation

Do it in reverse — buy the expensive GPU first, profile later — and you end up having spent the money with the problem intact.

⚠️ Optimal settings differ by library version and hardware generation. This article covers the order of approach; always measure the actual numbers on your own model and environment.