Comparing Cloud GPU Prices: What to Look at Before the Hourly Rate

Cloud costs for a computer vision project are routinely off by a factor of two when calculated as “GPU dollars per hour.” There are two causes: the time the GPU spends idle and the price of moving data.

Instead of an hourly rate table, this article focuses on how to look at those two. Rates change often; this structure does not.

How computer vision workloads differ

Language model training is mostly bottlenecked on compute itself. Computer vision is different.

• The input is heavy. High-resolution images and video have to be read every step
• Preprocessing eats CPU. If decoding, resizing and augmentation run on the CPU, the GPU waits
• Datasets are large. Tens of gigabytes to terabytes is normal, and moving that costs money

So choosing by the GPU spec sheet (compute performance) fails. The priority of what to check is different.

Get into the habit of checking GPU utilisation first and this ordering explains itself. If utilisation hovers below 50% during training when you check with nvidia-smi, renting a more expensive GPU just spends that money on waiting.

# 데이터 로딩 병목을 줄이는 기본 설정
loader = DataLoader(
    dataset,
    batch_size=64,
    num_workers=8,        # CPU 코어 수에 맞춰 조정
    pin_memory=True,      # CPU→GPU 전송 가속
    persistent_workers=True,
    prefetch_factor=4,
)

Leaving num_workers at 0 or 2 and then blaming the GPU is extremely common. This one line is cheaper than any instance upgrade.

Provider types and their character

Rather than listing vendors, here they are by character.

Large general-purpose clouds — the most regions and ancillary services, and they satisfy enterprise requirements (security, compliance, VPC). In exchange, GPU hourly rates are the highest and the pricing structure is complex. Choose them when you must integrate with company infrastructure or need compliance.

GPU-specialist clouds — focused on renting GPUs, so the same card is much cheaper. Fewer ancillary services, and availability can fluctuate by time of day. Most sensible for personal research and startup experiments.

Distributed and marketplace providers — you rent GPUs put up by individuals and small operators. Cheapest of all. Stability and security vary by provider, and they are inappropriate for sensitive data. Suited to experiments on public datasets.

Spot and preemptible instances — a discount option offered across all types. They can be reclaimed at any time in exchange for a big discount. Suited to work that can be interrupted and resumed, such as training. Code that checkpoints frequently is a precondition.

Hidden costs: egress and storage

This is where budgets go wrong.

Egress (transfer out) — putting data into a cloud is generally free, and taking it out is charged. Uploading a dataset, downloading results and checkpoints, and moving to another cloud can produce a figure rivalling the GPU bill.

Storage — turning an instance off does not stop storage charges. A volume with your dataset attached keeps billing. Clean up when the experiment ends.

Three countermeasures

1. Upload the dataset once. Do not re-upload per experiment; make a snapshot and reuse it
2. Push checkpoints to object storage where egress is free or cheap. Sync automatically during training and you are safe even if the instance is reclaimed
3. Download only what you need. The final weights, not every checkpoint

# 학습 중 체크포인트를 외부 스토리지로 주기적 동기화
while true; do
  rclone sync ./checkpoints remote:project/checkpoints --transfers 4
  sleep 600
done

The practical order for cutting cost

1. Finish the pipeline on a small instance first. Do not debug code on an expensive GPU
2. Check GPU utilisation. If it is low, fix data loading first
3. Turn on mixed precision (AMP). Most vision models gain speed and memory with no accuracy loss
4. Move to spot instances plus automatic checkpointing
5. Tear down instances and volumes the moment you finish. A GPU left running and forgotten is the most expensive mistake
6. Set budget alerts. Most clouds offer threshold notifications

⚠️ GPU hourly rates, egress fees and instance types differ by provider and region and change often. Verify actual amounts with each provider’s official pricing calculator.