What These Hardware Terms Actually Mean

Why This Exists

Part 1 used terms like "IPC," "cache miss," and "memory bandwidth" as shorthand. This post breaks each one down so you can reason about hardware constraints without needing a computer architecture textbook.

Every term follows the same format: what it is, why it matters, and the one-sentence mental model to carry forward.

---

CPU Terms

Clock Speed (GHz)

How many cycles the processor ticks through per second. A 4 GHz CPU does 4 billion cycles per second. More cycles = more opportunities to do work, but how much work gets done per cycle varies wildly between architectures.

Mental model: Clock speed is how fast the engine revs. It doesn't tell you how much cargo it moves per revolution.

---

IPC (Instructions Per Cycle)

The number of instructions the CPU completes in a single clock cycle. Modern CPUs can retire 4–8 instructions per cycle when things go well. When things go poorly (branch mispredictions, cache misses), IPC drops toward 1 or below.

Mental model: IPC is how many lanes your highway has. Clock speed is the speed limit. Throughput = lanes × speed limit.

---

Core Count

The number of independent processors on a single chip. Each core can execute its own thread. More cores help when your workload can be split into parallel tasks. They do nothing for single-threaded work.

Mental model: Cores are workers. If the job can be split, more workers finish sooner. If it can't be split, extra workers stand idle.

---

Branch Prediction

The CPU guesses which direction an if/else will go before it actually knows. If the guess is right, execution continues without pausing. If wrong, the CPU throws away speculative work and restarts — a "branch misprediction" that costs 10–20 cycles.

Mental model: You're driving and guess which way the road turns. Guess right, no slowdown. Guess wrong, you brake and reverse.

---

Pipeline / Pipeline Stall

CPUs break instruction execution into stages (fetch, decode, execute, write-back) and overlap them — like an assembly line. A stall happens when one stage has to wait for another, and the whole pipeline idles for a cycle or more.

Mental model: A factory assembly line. If one station runs out of parts, every station downstream stops.

---

Hyper-Threading / SMT (Simultaneous Multi-Threading)

A single physical core presents itself as two logical cores. When one thread stalls (waiting for memory, for example), the other thread uses the idle execution units. Doesn't double performance — typically adds 15–30%.

Mental model: One desk, two people sharing it. When one is thinking, the other can write.

---

Memory Terms

RAM Capacity

Total bytes of physical memory available. When your working data exceeds RAM, the OS swaps to disk — which is thousands of times slower. More RAM means larger datasets fit without swapping.

Mental model: The size of your desk. Bigger desk = more documents open at once without going to the filing cabinet.

---

Memory Bandwidth

How many bytes per second can move between RAM and the CPU. Measured in GB/s. Limited by the number of memory channels and the speed of the DIMMs. Matters when the CPU processes data faster than memory can deliver it.

Mental model: The width of the pipe between storage and the factory floor. A wider pipe feeds the workers faster.

---

Memory Channels

Independent pathways between the CPU and RAM. A dual-channel setup can transfer twice the data per cycle compared to single-channel. Most server CPUs support 6–8 channels. Populating all channels maximizes bandwidth.

Mental model: Lanes on a highway. More lanes = more cars flowing simultaneously, even if each lane's speed limit is the same.

---

Memory Latency

The time it takes for data to travel from RAM to the CPU after being requested. Measured in nanoseconds (typically 50–100ns for DRAM). During this wait, the CPU may stall unless it has other work to do.

Mental model: The round-trip time to the warehouse. Even if the warehouse is huge, each individual fetch takes a fixed minimum time.

---

Cache (L1 / L2 / L3)

Small, fast memory built into the CPU die. L1 is smallest and fastest (1–4 cycles), L2 is larger and slightly slower (10–15 cycles), L3 is largest and slowest (30–50 cycles). All are dramatically faster than RAM (200+ cycles).

Mental model: L1 is your pocket. L2 is your desk drawer. L3 is the shelf behind you. RAM is down the hall. You want frequently-used data as close as possible.

---

Cache Line

The minimum unit of data the CPU fetches from memory — typically 64 bytes. Even if you need 1 byte, the CPU loads the entire 64-byte line. This is why sequential access patterns are fast (next data is already in the cache line) and random access is slow (each access loads a new line).

Mental model: You can't buy one egg at the store. You always get the full carton. If you need eggs in sequence, one trip covers you. If you need eggs from different stores, every egg is a new trip.

---

Cache Miss

When the CPU requests data that isn't in any cache level. It must wait for RAM to deliver it — a ~100ns penalty. A high cache miss rate means the CPU spends more time waiting for data than doing useful computation.

Mental model: You reach for a document on your desk and it's not there. You have to walk to the filing cabinet. If this happens constantly, you spend more time walking than working.

---

TLB (Translation Lookaside Buffer)

A small cache that stores recent virtual-to-physical address translations. Every memory access needs an address translation; the TLB makes this fast. A TLB miss means the CPU must walk the page table — expensive and serialized.

Mental model: An index card taped to your monitor with "document X is in drawer 3." Without it, you'd have to check the master filing system every time.

---

Storage Terms

IOPS (Input/Output Operations Per Second)

How many discrete read or write operations a storage device can handle per second. A spinning disk does ~100–200 IOPS. A good NVMe SSD does 500,000–1,000,000+ IOPS. This is the critical metric for databases and random-access workloads.

Mental model: How many individual requests the storage can answer per second — like how many customers a cashier can serve, regardless of basket size.

---

Throughput (Sequential Bandwidth)

How many MB/s or GB/s of data a storage device can read or write in a continuous stream. Different from IOPS — a device can have high throughput but low IOPS (spinning disks are decent at sequential reads, terrible at random ones).

Mental model: IOPS is how many small packages you can deliver per hour. Throughput is how much total weight you can haul if it's all going to the same place.

---

Latency (Storage)

The time from issuing an I/O request to receiving the first byte. Spinning disks: 5–15ms. SATA SSDs: 100–200μs. NVMe SSDs: 10–50μs. For latency-sensitive workloads, this is often more important than raw throughput.

Mental model: How long until the first byte shows up after you ask for it. A delivery truck might carry a lot, but if it takes 15 minutes to leave the warehouse, that's 15 minutes you're waiting.

---

NVMe (Non-Volatile Memory Express)

A protocol designed for SSDs, replacing the older AHCI/SATA protocol designed for spinning disks. NVMe talks directly to the CPU over PCIe lanes with minimal overhead — supports deep queue depths (64K queues × 64K commands each) vs. SATA's single queue of 32 commands.

Mental model: SATA is a single-lane drive-through. NVMe is a 64,000-lane highway with 64,000 cars per lane allowed.

---

Queue Depth

How many I/O requests are in-flight simultaneously. Higher queue depth allows storage devices to reorder and batch requests internally for efficiency. SSDs perform best with queue depths of 16–64. Low queue depth underutilizes the device.

Mental model: How many orders you've placed that haven't been delivered yet. More in-flight orders let the kitchen batch and optimize.

---

Network Terms

Bandwidth

Maximum data rate of a network link. Measured in Gbps. A 25G NIC can theoretically move ~3.1 GB/s. Actual throughput is lower due to protocol overhead, but bandwidth sets the ceiling.

Mental model: The diameter of the pipe. A wider pipe allows more water flow per second.

---

Latency (Network)

Round-trip time for a packet to travel from source to destination and back. Same-rack: <100μs. Same-datacenter: 0.5–1ms. Cross-region: 20–100ms. Matters enormously for workloads that make many sequential requests.

Mental model: If you need 1,000 round trips to complete a request and each trip is 1ms, that's 1 full second of pure waiting — regardless of how fast your CPU is.

---

RDMA (Remote Direct Memory Access)

Allows one machine to read/write another machine's memory directly, bypassing both operating systems' network stacks. Latency drops from ~50μs (kernel TCP) to ~1–2μs. Used in HPC, distributed databases, and ML training clusters.

Mental model: Instead of mailing a letter (pack, address, post office, delivery, open), you just reach into your neighbor's filing cabinet directly.

---

Kernel Bypass (DPDK / XDP)

Network processing that skips the OS kernel entirely. The application talks directly to the NIC hardware. Eliminates context switches, system calls, and kernel buffering. Adds complexity but can 10× packet processing throughput.

Mental model: Instead of all mail going through the central post office (kernel), you install a mailbox directly on your desk.

---

NIC (Network Interface Card)

The hardware connecting a machine to the network. Modern NICs handle 25G, 100G, or even 400G. Smart NICs offload work like checksums, segmentation, and encryption from the CPU.

Mental model: The on-ramp to the highway. A wider on-ramp lets more cars merge per second.

---

GPU Terms

CUDA Cores / Stream Processors

The individual compute units inside a GPU. A single core is weak compared to a CPU core, but GPUs have thousands (an RTX 4090 has 16,384). They excel at doing the same operation across thousands of data points simultaneously.

Mental model: A CPU is 8 expert engineers. A GPU is 16,000 assembly-line workers. Different jobs suit different teams.

---

VRAM (Video RAM)

Dedicated memory on the GPU, connected via a very wide bus (e.g., 384-bit). Much higher bandwidth than system RAM (1–2 TB/s vs. 50–100 GB/s), but limited capacity (8–80 GB). Your model and working data must fit in VRAM or performance collapses.

Mental model: The GPU's personal desk. If the model doesn't fit on the desk, it has to keep fetching from the filing cabinet (system RAM over PCIe) at 10–30× slower speeds.

---

Tensor Cores

Specialized GPU units that do matrix multiply-accumulate in one operation. A regular CUDA core does one multiply per cycle. A Tensor core does a 4×4 matrix multiply per cycle. This is why ML training accelerated dramatically in recent GPU generations.

Mental model: A regular calculator vs. a spreadsheet that multiplies entire tables at once.

---

NVLink

A high-bandwidth interconnect between GPUs (or GPU-to-CPU). Provides 600–900 GB/s between GPUs vs. ~32 GB/s over PCIe. Critical for multi-GPU training where gradients must be synchronized constantly.

Mental model: PCIe is a two-lane road between GPU buildings. NVLink is a 20-lane express highway between them.

---

GPU Memory Bandwidth

How fast data moves between VRAM and the GPU's compute units. An H100 has ~3.35 TB/s of memory bandwidth. For LLM inference, this is often the bottleneck — the GPU can compute faster than memory can feed it.

Mental model: The GPU's internal conveyor belt speed. Even with 16,000 workers, they starve if the belt can't deliver materials fast enough.

---

Interconnect & Bus Terms

PCIe (Peripheral Component Interconnect Express)

The main bus connecting CPU to GPUs, NVMe SSDs, NICs, and other devices. PCIe 4.0 provides ~2 GB/s per lane; PCIe 5.0 doubles that. Devices use multiple lanes (x4, x8, x16) for more bandwidth.

Mental model: The internal highway system on your motherboard. Lane count × lane speed = total throughput available to a device.

---

DMA (Direct Memory Access)

Hardware capability that lets devices (NICs, SSDs, GPUs) transfer data directly to/from RAM without involving the CPU. Frees the CPU from byte-shuffling work so it can focus on computation.

Mental model: Instead of you carrying every package from the mailbox to your desk, a robot does it while you keep working.

---

NUMA (Non-Uniform Memory Access)

In multi-socket servers, each CPU has "local" RAM that's fast to access and "remote" RAM (attached to the other CPU) that's slower. Software should allocate memory on the same NUMA node as the core that will use it.

Mental model: Two desks in different rooms, each with their own filing cabinet. Reaching your own cabinet is fast. Reaching the other room's cabinet takes a walk.

---

Putting It Together

These terms aren't isolated. They compose into bottleneck chains:

• A cache miss triggers a memory latency penalty. If misses are frequent, you saturate memory bandwidth.
• An NVMe drive with high IOPS still has a ceiling set by the PCIe lane count connecting it.
• A GPU with massive Tensor core throughput starves if VRAM bandwidth can't feed it, or if NVLink is too slow to synchronize multi-GPU gradients.

The key skill is tracing the chain: what is the first resource that saturates? Everything downstream of that saturation point is irrelevant until you fix the bottleneck.

---