GPU (Graphics Processing Unit)
Definition
Specialized hardware that runs AI models much faster than CPUs. NVIDIA A100, H100, etc.
Why It Matters
AI inference does the same matrix multiplication millions of times. CPUs are general-purpose and serial; GPUs have thousands of small cores running the same operation in parallel, they finish the same workload 10–100× faster. Without GPUs, hosting a 70B model would mean waiting 10 seconds per token.
Key Points
- H100 SXM5: 80 GB HBM3, ~3.35 petaFLOPS BF16, NVLink 900 GB/s inter-GPU bandwidth. Cloud spot pricing ~$2–4/hour per GPU.
- A100 80GB: ~2 petaFLOPS BF16, the most widely deployed GPU for production inference as of 2024.
- NVLink connects multiple GPUs into a shared VRAM pool, necessary for models whose weights exceed a single card's memory.
- Consumer RTX 4090: 24 GB GDDR6X, ~330 TFLOPS FP16. Fast for prototyping; cannot fit a 70B model in full precision.
- Memory bandwidth is the binding constraint for inference at typical batch sizes, H100's 3.35 TB/s vs. A100's 2 TB/s is why H100 produces ~2× higher token throughput despite a similar FLOPS ratio.
Example
An NVIDIA H100 has 80 GB of VRAM and ~3,000 trillion operations per second. Generating one image with FLUX.1 takes about 3 seconds on an H100. The same workload on a high-end CPU would take 5+ minutes.
Common Misconception
GPU FLOPS rating does not predict inference token throughput. Most LLM serving is memory-bandwidth-bound, not compute-bound, because batch sizes are small and the bottleneck is moving model weights from VRAM to compute units. A higher-bandwidth GPU produces more tokens per second even at similar FLOPS.
Related Terms
- InferenceThe process of running an AI model to generate a response. When you send a message to ChatGPT, the model performs inference.
- VRAMVideo RAM, the memory on a GPU used to store AI model weights during inference.
- ParameterA trainable weight in an AI model. Larger models have more parameters (7B, 70B, 400B).
GPU (Graphics Processing Unit) on Rewind.ai
Rewind.ai's self-hosted GPU pool runs every open-source model on the platform. Token pricing reflects GPU-second cost, that's why a long video render burns more tokens than a one-line chat reply.
Explore the Tools