Data Center Architecture

An AI data center is not a building with computers in it. It is a precision engineering system where power delivery, cooling, networking, and compute are co-designed and must all scale together. Understanding this physical infrastructure is essential for anyone making architectural decisions that have cost or sustainability implications.

The Power Chain

Grid power (110kV–500kV AC)
       │
       ▼
Substation (step down to ~12-35kV)
       │
       ▼
UPS (Uninterruptible Power Supply)
  → Battery backup for short outages
  → Converts to clean AC
       │
       ▼
PDU (Power Distribution Unit) per row
  → Distributes to individual servers
       │
       ▼
Server PSU (Power Supply Unit)
  → Converts AC → DC (12V or 48V for compute)
       │
       ▼
GPU / CPU / Memory

Typical data center PUE (Power Usage Effectiveness):
  PUE = Total facility power / IT equipment power
  Legacy data center: PUE 1.8–2.5 (overhead is 80-150% of compute)
  Modern hyperscale (Google, Meta, Microsoft): PUE 1.1–1.2
  Best-in-class AI facilities: targeting PUE 1.05

GPU Density - The Core Challenge

AI has pushed rack power density to unprecedented levels:

Traditional air cooling can handle up to ~25 kW/rack reliably. Modern AI GPU racks at 80–140 kW/rack cannot be air-cooled at all - they require liquid cooling. This is why data center retrofitting for AI is a $100B+ capital expenditure problem.

Networking - The Other Bottleneck

For distributed training across many GPUs, the network interconnect is as important as the compute. Training a large model requires constant all-reduce operations - synchronizing gradient updates across thousands of GPUs simultaneously:

• InfiniBand (NVIDIA) - up to 400Gb/s per GPU link with RDMA (Remote Direct Memory Access). The standard for high-performance AI training clusters. Used in all major AI data centers.
• NVIDIA NVLink - within a server, connects GPUs at 900GB/s total bandwidth (vs 32GB/s PCI-E). NVL72 rack connects 72 GPUs with NVLink, making them behave like a single giant GPU.
• Google TPU interconnect - Google's TPU pods use a custom 3D torus network topology for ultra-low-latency all-reduce across thousands of TPU chips.
• Meta's RDMA fabric - 400GbE Ethernet with RoCE (RDMA over Converged Ethernet) across the entire cluster; avoids InfiniBand licensing costs.

Storage for AI Workloads

AI training has unusual storage requirements: massive sequential reads (loading training data) rather than random reads/writes (typical for web workloads):

• Training datasets: petabytes; loaded in large sequential chunks
• Checkpointing: save model weights (hundreds of GB) every few hours; requires fast write bandwidth
• Inference: model weights loaded once into GPU memory; subsequent requests served from GPU VRAM
• Common: distributed storage systems (HDFS, Lustre, AWS FSx for Lustre) providing parallel I/O at 100+ GB/s aggregate bandwidth

The Facility Layer

Above the IT stack, a modern AI data center requires:

• Substation - most AI data centers require a dedicated substation, not a shared grid connection. Building a new 100MW+ substation takes 18–36 months in most jurisdictions.
• Backup generators - diesel generators for multi-hour outages; typically sized to power 100% of load for 24–48 hours
• Structural - AI racks at 80–140 kW have floor weight requirements that most existing data centers cannot support without structural reinforcement
• Water supply - for liquid-cooled or evaporative cooling systems, significant water supply infrastructure is required; increasingly a constraint for site selection