gpu guide

Available GPU types on the cluster, from least to most powerful. Use the gpu: field in tccp.yaml or check live availability with gpus in the REPL.

available GPUs

GPUVRAMbest fornotes
t4 16 GB Inference, light training Oldest Turing card. Slow FP32 but decent FP16. Good for small models and debugging.
p100 16 GB Legacy workloads Pascal generation. No tensor cores. Mainly useful when nothing else is free.
rtx_6000 24 GB Medium training Turing with tensor cores. Decent for mid-size models.
v100 32 GB General training Volta. First-gen tensor cores. Solid all-rounder, good FP16 throughput.
a100-40gb 40 GB Large model training Ampere. 3rd-gen tensor cores, TF32, BF16. Fast interconnect. Great price/perf.
rtx_a5000 24 GB Medium training, multi-GPU Ampere workstation GPU. 8 per node — good for data-parallel training.
l40 48 GB Large models, fine-tuning Ada Lovelace. Good VRAM for big batch sizes or large models.
rtx_a6000 48 GB Large models Ampere workstation. 48 GB VRAM, 8 per node.
rtx_6000ada 48 GB Large models, fast training Ada Lovelace workstation. Newer arch than rtx_a6000.
a100-80gb 80 GB LLMs, very large models The workhorse. 80 GB handles most LLM fine-tuning. High bandwidth.
l40s 48 GB Fast training, inference Ada Lovelace server GPU. Better FP8 than L40. Newer generation.
h100 80 GB LLMs, fastest training Hopper. 4th-gen tensor cores, FP8, Transformer Engine. Top-tier throughput.

selection rules of thumb

VRAM estimation

Rough VRAM needs for common tasks:

taskapprox VRAMsuggested GPU
ResNet/EfficientNet training 4–8 GB t4
BERT fine-tuning 8–12 GB t4, v100
GPT-2 (124M) training 8–12 GB t4, v100
Stable Diffusion fine-tuning 16–24 GB v100, rtx_6000
LLaMA 7B LoRA fine-tuning 16–24 GB v100, a100-40gb
LLaMA 7B full fine-tuning 32–40 GB a100-40gb
LLaMA 13B LoRA fine-tuning 32–40 GB a100-40gb, a100-80gb
LLaMA 13B full fine-tuning 60–80 GB a100-80gb
LLaMA 70B LoRA (QLoRA 4-bit) 40–48 GB a100-80gb, l40s
Inference (quantized) model size / quant bits varies

using in tccp.yaml

Set a GPU at the project level or per-job:

# project-wide default
gpu: a100

# per-job override
jobs:
  train:
    script: train.py
    gpu: a100-80gb
    gpu_count: 2

Shorthand with colon syntax:

gpu: a100-80gb:2    # equivalent to gpu: a100-80gb + gpu_count: 2

When you set a GPU, tccp auto-bumps memory to 32G (64G for 80GB GPUs) and cpus to 4. Override if you need more.

v