TLDR: Study guide for the NVIDIA Certified Associate: AI Infrastructure and Operations (NCA-AIIO) exam, written from an operations background — if you already run Kubernetes or Slurm, this maps the AI infrastructure world onto what you already know.

Introduction: Why This Fuckin' Certification

I'm about to start working with NVIDIA GPUs at work, so I'd rather learn this properly instead of pick it up for fun purpose, because it won't going anywhere or further and the NCA-AIIO is a reasonable way to force that.

Everything here is written from an operations background. If you already run Kubernetes clusters, a lot of this world has a familiar shape — and where it doesn't. This is article have no hands-on, every commands output in this section is just I copied it somewhere xD

Where NCA-AIIO Sits

NCA-AIIO is the entry point, not the destination. Same idea as passing AWS Cloud Practitioner before Solutions Architect Associate, like everyone:

Worth knowing before you pick a path: NCP-AIO is the only one of these with hands-on labs. The lab environment is provisioned when the exam starts and you drive real Slurm, Kubernetes and Base Command Manager from a Linux CLI. Everything else on this list, including the one you are reading about, is multiple choice — you can pass those by studying. AIO you cannot.

Which makes AIO the friendlier next step if you already run clusters for a living, and the harder one if you don't. Verify the numbers on the official pages before you book anything, they move.

Exam Details

Full details on the official exam page — 50 questions, 1 hour, online proctored, registered through Certiverse.

Three weighted domains:

How to Read This

Everything here maps to a blueprint bullet unless the heading says otherwise. Two markers appear:

• (optional — NCP-AIO level) — real ops knowledge that the Associate exam does not test. Skip it on the first pass; come back when you run a cluster.
• (optional — reference) — numbers to look up, never to memorize.

If you are short on time, read everything unmarked, do the Exam Checklist, skim the Quick Recall cheat sheet, and ignore the rest.

Learning Resources

• AI Infrastructure and Operations Fundamentals on Coursera, the official course. Paid, worth it if your company sponsors training.
• YouTube full course, free alternative covering the same ground, no chapters though.

Mapping: What You Already Operate → AI Infrastructure

If you come from Kubernetes, most of this world has a familiar shape. Start here:

And the one thing that has no Kubernetes equivalent at all:

The compute fabric. In K8s you have one network and you mostly don't think about it (Like me for example xD). In AI infrastructure there are 4 physically separate networks, and the fastest of them exists purely so GPUs in different chassis can behave like one giant GPU. This is the biggest conceptual gap coming from the cloud-native world, and it's covered in Phase 3.

Phase 1: The AI Workload — What Are We Actually Running?

You can't size (capacity planning) a cluster for a workload if you can't describe it, right?

1.1 Training vs. Inference

These are two completely different workloads that happen to use the same hardware:

1.2 AI vs. ML vs. Deep Learning vs. GenAI

Nested circles, broadest to narrowest:

• AI — any system doing something that normally needs human intelligence, including plain rule-based systems that never learn.
• Machine Learning — learns patterns from data instead of being programmed. A human still picks which input features matter.
• Deep Learning — ML with multi-layer neural networks that work out the features themselves. Needs far more data and compute, which is what made GPUs mandatory.
• Generative AI — a deep learning application that produces new content (text, images, code) rather than a label or a number.

Rule of thumb: tabular data with thousands of rows → classical ML. Unstructured data (images, audio, text) at scale → deep learning.

1.3 Why Now — What Actually Changed

Neural networks are from the 1980s. The blueprint asks why they only took over recently. Four things landed at once:

• Data (Major) — self-supervised learning (Phase 1.5) removed the human-labeling bottleneck, so the raw internet became training data.
• Compute (Major) — Tensor Cores plus ever-lower precision (FP32 → BF16 → FP8 → FP4): same watt, far more math.
• Architecture — the transformer, a model can finally use ten thousand GPUs. Without this, the other three don't matter.
• Ecosystem — CUDA, PyTorch, pretrained models, GPU by the hour. Most teams fine-tune someone else's model rather than train one.

The consequence for infrastructure is scaling laws: more parameters + data + compute gives predictably better models, so the cluster keeps getting bigger.

1.4 Training Fundamentals

Example: Model 70B params -> gradient is 70B numbers, each number is direction for a weight (weight = weight - learning_rate × gradient). AllReduce sync gradient, not the fuckin' data!

Training is a loop: feed data in + let the model guess (forward pass), measure how wrong it was (loss), adjust slightly (optimizer step), repeat millions of times. The "adjust" step is matrix math, which is why it needs a GPU.

Terms worth recognizing: it is a wall of text but worth for a noob like me, that is why I put them below

• Batch size — how many samples the GPU processes before the model updates. Bigger batch = better GPU utilization but more GPU memory.
• Iteration (step) — one trip around the loop above: one batch in, one weight update out.
• Epoch — one complete pass over the whole dataset. It is a counter, not an extra step.
• Overfitting — the model memorized the training data instead of learning the general pattern. The opposite is underfitting — too simple, bad at both.
• Checkpoint — a periodic save of model state to shared storage, so a job that dies on day three resumes instead of restarting.

Batch, iteration and epoch are the same work counted three ways. Think of revising a question bank:

Question bank of 1,000 questions      <- the dataset
Answer 20, check them, learn          <- batch size 20 = one iteration
50 iterations = all 1,000 questions   <- one epoch

Epoch 2 is the same bank again, not a new one — each pass only nudges the weights a little, so you go around several times.

Overfitting is revising that bank until you have the answer key memorized: 10/10 on it, and lost on anything unseen. That is the trap, because the score you are watching looks great. Underfitting is not having studied enough — bad on both. It is the same failure as tuning a config against one recorded traffic pattern: the benchmark looks perfect right up until production shifts.

You catch it with a validation set the model never trains on, scored after each epoch. Training loss still dropping while validation loss starts climbing means the memorizing has begun, and early stopping is the usual answer.

Fixing it splits into three groups — past what the Associate exam asks, but I still like to write it down here xD:

1. Data, reach here first. More real data, or augmentation (crop, rotate, paraphrase — synthetic variety from what you already have). Nothing beats showing the model more of the world.
2. Architecture. Fewer layers or parameters, because a smaller model has less room to memorize. Dropout randomly disables neurons during training so they can't lean on each other.
3. Regularization & optimization. Weight decay / L1 / L2 penalize oversized weights. Early stopping ends the run when validation loss turns upward, instead of guessing an epoch count.

Exam-style question: a model scores 99% on training data but 60% on test data — what is this? → overfitting.

Two of these land directly on infrastructure: batch size drives GPU memory, and checkpointing drives storage bandwidth — every rank writes at the same instant, so the filesystem has to absorb the burst. Checkpoint sizing is a real capacity input, not an afterthought.

1.5 How Models Learn

Two output types under supervised learning: classification (discrete — spam or not spam) and regression (continuous — predicted price).

1.6 Evaluation Metrics

This section original is way too fuckin' hard for me, it is not fuckin' possible for me to understand how thing work! So I will try to memorize instead, it sucks but better than nothing:

• False positive — flagged it, nothing was wrong.
• False negative — stayed silent, something was wrong.
• Precision — of everything I called out, how much was right?
• Recall — of everything that actually happened, how much did I catch?
• Low precision = noise. Calling out a lot, wrong a lot.
• Low recall = misses. Staying silent while something was going on.
• Accuracy — correct overall. Useless when the data is imbalanced: a check that never fires is still 99.9% accurate, simply because almost nothing is ever wrong.
• F1 — precision and recall rolled into one number. Use it instead of accuracy when the classes are imbalanced.
• Picking one: which mistake is more expensive? Misses cost more => recall. False alarms cost more => precision.

Questions describe a business scenario and ask which metric matters. The shapes repeat:

1.7 Generative AI & LLMs

This is the part worth actually understanding, because everything NVIDIA sells right now is aimed at it.

• Transformer — the architecture behind every modern LLM (Attention Is All You Need, 2017). Its core trick, self-attention, lets every word weigh its relevance against every other word — and does it in parallel, which is exactly why GPUs suit it and older sequential architectures didn't.
• Token — the unit an LLM processes.
• Context window — how many tokens it can consider at once.
• Embedding — text turned into a vector, where similar meaning means nearby in space. Stored in a vector database (FAISS, Milvus, pgvector) so you can search by meaning instead of keyword.
• Hallucination: confident, fluent and fuckin' wrong. The fix is grounding the model in real documents, not more training.

Inference has two phases, and they stress the GPU differently:

That second row is why inference GPUs are chosen for High Bandwidth Memory and capacity. The KV cache, which stores the attention state of tokens already generated so each new token doesn't recompute the whole conversation, grows with batch size × context length and is frequently the real memory ceiling in production serving.

Customizing a model — guaranteed exam material:

RAG in one line: chunk your documents → embed them → store in a vector DB → at query time, retrieve the most relevant chunks and paste them into the prompt as context.

Ingest runs offline, query runs per request — and notice the LLM itself is never retrained. Source: NVIDIA, RAG 101

If a question says "must reflect data that changes daily" or "must cite internal documents", the answer is RAG, not fine-tuning. If it says "must always respond in our house style / output strict JSON", that's fine-tuning.

1.8 Who Uses What

Scenario questions like "a hospital wants X — which NVIDIA offering?" are common:

And for those content above, I feel like they are almost reflection of On-prem vs AWS (for example of Cloud) xDD

Phase 2: The Compute Layer

2.1 CPU vs. GPU vs. DPU

• CPU — latency optimized. Few powerful cores, big caches, branch prediction. Good at sequential logic and decisions.
• GPU — throughput optimized. Thousands of small cores doing the same operation on different data (SIMT). Good at matrix math, useless at branching logic.
• DPU (BlueField) — infrastructure offload. Takes networking, storage and security work off the host CPU so the CPU can stay busy feeding the GPUs. Programmed with the DOCA(Data Center on a Chip Architecture) SDK. If we need one thing for reflection, we can think about AWS Nitro I believe!

Amdahl's law is the practical takeaway. The part you can't accelerate sets the ceiling:

# Example!
A job takes 100 min: [==== GPU: 80 min ====][ CPU: 20 min ]
More and faster GPUs:                       [ CPU: 20 min ]  -> 20 min still here, not going anywhere!

Those 20 minutes are a floor no amount of GPUs can touch. In real clusters the leftover is almost always data loading and preprocessing, which is exactly why DALI and RAPIDS exist: move that work onto the GPU and the ceiling moves up with it.

2.2 GPU Generations — What Actually Changed

Skip the transistor counts. Remember what each generation introduced, because that's what gets asked (I have no fuckin' idea about TF32, BF16, FP8.... so it is fuckin' fine if you don't understand it xD):

Three relationships worth memorizing:

• H100 vs A100 — same 80GB capacity, but faster HBM3, 1.5x NVLink bandwidth, and FP8 via Transformer Engine.
• H200 = H100 compute + better memory — 141GB HBM3e, ~1.4x memory bandwidth. Drop-in compatible with HGX H100 systems. Best for large-model inference and long context, and now you know why (Phase 1.7: decode is memory-bandwidth bound).
• GB200 Superchip = 1 Grace CPU + 2 Blackwell GPUs over NVLink-C2C (900 GB/s, ~7x PCIe Gen5). GB200 NVL72 = 36 Grace + 72 Blackwell in one liquid-cooled rack acting as a single NVLink domain.

Also worth recognizing: L40S (Ada, 48GB, no NVLink — inference and graphics), L4 (Ada, 24GB, 72W — low-power inference/video), GH200 (1 Grace + 1 Hopper).

Detailed specs are in Appendix A — reference material, not memorization material.

2.3 DGX vs. HGX vs. MGX

• DGX — NVIDIA's own complete turnkey system, hardware plus software stack. Scales into a DGX SuperPOD reference architecture.
• HGX — just the GPU baseboard (8 GPUs + NVSwitch), sold to OEMs like Dell and Supermicro to build their own servers around.
• MGX — a modular reference design letting OEMs assemble many different CPU/GPU/DPU server configurations.

One-liner: DGX you buy finished, HGX is the engine block, MGX is the chassis blueprint.

2.4 Tensor Cores & Precision Formats

Tensor Cores are dedicated units that do fused matrix multiply-and-add (D = A × B + C), the single operation that dominates deep learning. Regular CUDA cores can do this too, just far slower.

Lower precision means fewer bytes per value: less memory, more throughput, some accuracy risk. That byte count is the number you actually use, it's what the sizing math in Phase 2.6 runs on.

A 70B model is 70 × 2 = 140 GB at BF16, half that at FP8. Which generation introduced which format is the other thing worth remembering — that's the shape the exam question takes.

Optional — none of this is on the exam. (This is the fuckin' AI Generated content xD) Each format splits its bits between exponent (buys range) and mantissa (buys precision). BF16 keeps FP32's full exponent, so it needs no loss scaling, which is why it displaced FP16 for training. TF32 is an Ampere-era internal compute format the math libraries apply transparently. FP8 ships in two flavors, E4M3 forward and E5M2 backward, because gradients span a wider range. NVFP4 stays accurate enough only because of per-block scaling.

2.5 Sharing a GPU: MIG vs. vGPU vs. Time-Slicing

A full H100 is overkill for most inference workloads, so there are three ways to split one. Ordered from strongest isolation to weakest:

MIG is the only one giving true QoS and fault isolation!

The same physical GPU, four legal ways to cut it. The crossed-out boxes are the capacity you lose to partitioning. Source: NVIDIA MIG User Guide

2.6 Sizing: How Much Hardware?

Two blueprint bullets ask you to pick hardware for a described use case. Both come down to memory. Why? this can be daily job if I'm not fuckin' wrong!.

Small example: a customer says they want to run a 70B model. The first question back is train it, or serve it? Serving it at BF16 takes ~170 GB once the KV cache is counted, so a single DGX H100 (8 × 80 GB = 640 GB) is already more than enough — three GPUs would do — and selling them a second box is wasted money. Training that same model needs over 1 TB before activations, so one DGX is not enough and you are looking at two nodes minimum. Same model, completely different bill.

Inference — memory holds the weights:

Add the KV cache and runtime overhead on top — budget roughly 20%. So a 70B model at BF16 does not fit one 80GB H100, but does fit one 141GB H200. That's the H200 story from 2.2 in concrete form.

Training — memory holds far more than the weights:

Weights + gradients + Adam optimizer state runs about 16 bytes per parameter, before activations. The same 70B model therefore needs ~1.1 TB of state — at least 14 GPUs' worth of HBM, so 16 in practice, and more once activations are counted.

Training needs roughly 10x the memory of inference. If you remember one number from this section, that's the one.

Scaling up vs. scaling out

Scale-up: every one of the 72 GPUs in a GB200 NVL72 wired into 18 NVLink switch chips. Any GPU reaches any other in a single hop at full NVLink speed — and it stops at 72. Source: NVIDIA DGX SuperPOD — Network Fabrics

Scale-out: leaf → spine → core InfiniBand joining 8 racks into a scalable unit, then 16 of those together. The C-UFM boxes are the fabric manager from Phase 3.4, and every one of those lines is an optic with a reach limit — which is Phase 8.3's problem. Source: NVIDIA DGX SuperPOD — Network Fabrics

Scale up first, because NVLink is far faster than anything crossing a chassis boundary; scale out when you run out of chassis. DGX SuperPOD is NVIDIA's reference architecture for doing the second properly.

Picking the part:

Phase 3: The Fabric Layer

This is the part with no Kubernetes equivalent, and the part most worth understanding.

3.1 Why Not Just Use PCIe?

Distributed training means every GPU must exchange gradients with every other GPU, every single step. That collective operation is AllReduce: each GPU finishes its batch holding its own gradients, AllReduce sums them across every GPU and hands back the identical result to all of them, so all the model copies stay in lockstep. It happens thousands of times per epoch, and nobody computes anything until it finishes.

PCIe Gen5 x16 gives ~128 GB/s bidirectional. NVLink 4 gives 900 GB/s bidirectional — roughly 7x. Run AllReduce over PCIe and your expensive GPUs spend most of their life waiting.

Quick note on units. These links are full duplex — separate lanes each way — so the same cable gets quoted two ways, and the two conventions differ by exactly 2x:

NVIDIA prints NVLink bidirectional, PCIe specs are per direction. Compare them as-printed and you find a 14x that isn't there. Networking flips it again: 400 Gb/s NDR InfiniBand means 400 Gb/s each direction.

• NVLink — direct GPU-to-GPU interconnect bypassing PCIe entirely (H100: 18 links × 50 GB/s bidirectional = 900 GB/s).
• NVSwitch — a non-blocking switch chip so all 8 GPUs in a chassis talk to each other simultaneously at full speed, not just in pairs.
• NVLink Switch (Blackwell) — extends NVLink outside the chassis, making 72 GPUs in a GB200 NVL72 behave as one NVLink domain.

3.2 The Four Fabrics

An AI datacenter runs four physically separate networks. Know which does what:

The rule of thumb for a DGX: one compute NIC per GPU (8 GPUs -> 8 NICs, a rail-optimized design), plus separate NICs or DPUs for storage and in-band, plus one RJ45 BMC port.

The storage and in-band fabrics share Ethernet switching but stay logically separate — and note the link legend: storage runs at 800 Gbps while in-band runs at 100. Source: NVIDIA DGX SuperPOD — Network Fabrics

The OOB fabric: 1 Gbps to everything with a management port — PDUs, BMCs, the IB switches themselves, the UFM appliances. Slow, dull, and the one network that must never be reachable from user space. Source: NVIDIA DGX SuperPOD — Network Fabrics

3.3 RDMA and Its Friends

• RDMA — a NIC writes directly into a remote machine's host memory, bypassing the OS kernel and the remote CPU. Zero-copy, very low latency.
• GPUDirect RDMA — the same, but straight into GPU memory, never touching host RAM. This is what NCCL uses for inter-node traffic, and the distinction between these two is exam material.

3.4 InfiniBand vs. RoCE vs. Spectrum-X

InfiniBand isn't Ethernet with different cables. It's a separate stack, its own switches, its own adapters (HCAs), no TCP/IP, built for RDMA from day one. It never drops packets under congestion, because a sender may only transmit once the receiver has granted it buffer credits, and routes are computed centrally by a Subnet Manager rather than by each switch on its own. Ethernet drops and retransmits, which is fine for web traffic and fatal for AllReduce. RoCE is the compromise: RDMA bolted onto Ethernet, which then has to be configured into behaving losslessly. Spectrum-X is NVIDIA selling that configuration pre-done.

If it helps: InfiniBand is to Ethernet what Fibre Channel is to iSCSI.

Two clarifications the exam likes:

• SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) does part of the AllReduce inside the switch, so less data crosses the wire.
• Subnet Manager: OpenSM is the open-source subnet manager that configures the fabric. UFM is NVIDIA's fabric management platform built around it (monitoring, telemetry, health). Don't mix them up.

Phase 4: The Storage Layer

4.1 Why Storage Is a GPU Problem

Traditional I/O wastes CPU and PCIe bandwidth on pure copying:

Traditional Path:
Storage/NIC ──► Host RAM (page cache) ──► Host RAM (bounce buffer) ──► GPU Memory

GPUDirect Storage (GDS) Path:
Storage/NIC ─────────────────(Direct PCIe P2P / RDMA)───────────────► GPU Memory

GPUDirect Storage opens a direct DMA path from storage into GPU memory, skipping the CPU bounce buffer. Local NVMe uses PCIe peer-to-peer; remote storage uses RDMA. Applications reach it through the cuFile API (kernel module nvidia-fs). Result: lower latency, higher throughput, and lower CPU utilization — that last one matters because the CPU is usually busy preprocessing data.

4.2 Storage Tiers

Local NVMe          →  Shared parallel FS        →  Object storage
(scratch, cache)       (datasets, checkpoints)      (data lake, archive)
fastest, smallest      fast, shared by all nodes    cheap, slow, huge

Parallel filesystems (Lustre / DDN EXAScaler, WEKA, VAST Data, IBM Storage Scale, BeeGFS) exist because a normal NFS server cannot feed hundreds of GPUs at once. They spread both data and metadata across many servers.

Remember the checkpoint problem from Phase 1.4: hundreds of GPUs writing simultaneously, all idle until the write finishes. Checkpoint burst bandwidth is usually what sizes the filesystem, not average throughput.

Phase 5: The Software Stack

5.1 The Layers

[ Applications & Models ]   NeMo, BioNeMo, Clara, Riva, Omniverse
            ▲
[ Serving & Optimization ]  NIM, Triton Inference Server, TensorRT, TensorRT-LLM
            ▲
[ Frameworks ]              PyTorch, JAX, TensorFlow  │  RAPIDS (data science)
            ▲
[ Accelerated Libraries ]   CUDA  │  CUDA-X: cuDNN, NCCL, cuBLAS, DALI  │  DOCA
            ▲
[ System Layer ]            GPU Driver, NVIDIA Container Toolkit
            ◄── DCGM monitors every layer above

• CUDA — the parallel computing platform and programming model. Everything sits on this.
• cuDNN — tuned primitives for neural networks (convolution, pooling, normalization).
• NCCL — the collective communication library for multi-GPU and multi-node (AllReduce, AllGather, Broadcast, ReduceScatter), optimized for NVLink and InfiniBand. Any question about optimizing inter-GPU communication is an NCCL question.
• CUDA-X — the umbrella name for the accelerated libraries sitting on top of CUDA (cuDNN, NCCL, cuBLAS, RAPIDS, DALI). NVIDIA uses the term constantly, so recognize it.
• DOCA — SDK for BlueField DPUs and SuperNICs.

5.2 Serving: TensorRT vs. Triton vs. NIM

These three get confused constantly, and the exam knows it: TensorRT optimizes the model; Triton serves it; NIM ships both as a supported container.

And three ways to shrink a model before you serve it — classic answer options, so don't mix them up:

• quantization (fewer bits per weight, what TensorRT does)
• pruning (drop weights that barely contribute)
• distillation (train a small student model to imitate a large teacher).

5.3 Distribution & Support

• NGC — NVIDIA's registry of GPU-optimized containers, pretrained models and Helm charts. Think like Docker Hub or container registry.
• NVIDIA AI Enterprise (NVAIE) — the paid layer: validated images, security patches, enterprise support, and NIM. This is what a company buys so someone else is accountable for CVEs.
• Apptainer (formerly Singularity) — containerization for HPC, not orchestration. Runs unprivileged and mounts the host filesystem naturally, which is why Slurm sites use it over Docker.

Phase 6: Operations — Scheduling & Orchestration

6.1 Slurm vs. Kubernetes vs. Run:ai

Slurm is still the standard for large training jobs, mainly because it allocates all or nothing — vanilla kube-scheduler will happily bind 60 of 64 pods and leave them idling.

Run:ai is worth understanding because it's essentially "Slurm's scheduling brain, but for Kubernetes" — teams get quotas, idle quota gets borrowed and reclaimed, and distributed jobs start all-or-nothing instead of deadlocking half-scheduled.

(Note: Slurm uses "gang scheduling" to mean time-slicing oversubscribed jobs — see PreemptMode=GANG. Everyone else uses it to mean all-or-nothing co-allocation)

6.2 NVIDIA GPU Operator

The GPU Operator is the reason you don't hand-install drivers on 200 nodes. It deploys and manages:

• NVIDIA driver + Container Toolkit
• Device plugin — exposes nvidia.com/gpu as a schedulable resource
• GPU Feature Discovery + Node Feature Discovery — labels nodes with GPU model, MIG profile, etc.
• MIG Manager — applies MIG partitioning declaratively
• DCGM + DCGM Exporter — monitoring
• Operator validator

The companion Network Operator handles the networking side: MOFED/DOCA drivers, RDMA shared device plugin, GPUDirect RDMA.

6.3 Base Command Manager

BCM (formerly Bright Cluster Manager) is bare-metal cluster provisioning for DGX and non-DGX clusters: OS deployment, InfiniBand fabric health checks, monitoring, and integration with both Slurm and Kubernetes. Closest thing to Cluster API in this world.

Do not confuse it with Base Command Platform, which is a SaaS for running training workloads on DGX Cloud.

6.4 MLOps, Briefly

• Lifecycle: data versioning → experiment tracking → training → model registry → validation → deploy → monitor → retrain.
• Model registry — versioned model artifacts with lineage and stage promotion. NGC plays this role for NVIDIA-published models.
• Deployment — canary and blue/green apply exactly as they do for services; Triton can serve multiple model versions concurrently on one GPU.
• Drift — data drift is the inputs changing; concept drift is the input→output relationship changing. Either one is the signal to retrain.
• Reproducibility — pin the container image, framework version, driver version, dataset version and random seed. Containers are the mechanism.

The blueprint asks you to "describe software components related to the AI development and deployment life cycle" — and it files that under Essential AI Knowledge, the 38% domain. So know which tool sits at which stage:

The pattern is the same CI/CD loop you already run, with a model artifact instead of a binary — and one extra failure mode (drift) that has no equivalent in ordinary software.

Phase 7: Operations — Monitoring & War Stories

7.1 The Tools

I don't have GPU hardware yet, so I'm not going to paste output I didn't generate. When I do, the hands-on version of this table gets its own article.

7.2 Xid Errors

Xid errors are codes the NVIDIA driver writes into the kernel log when something goes wrong — hardware fault, bad memory, driver bug, or an application doing something illegal. The number identifies the class of problem.

Operationally the reflex is the same one you already have for a flaky K8s node: detect → drain → diagnose → RMA or return to service. DCGM can watch for them across the fleet so you're not grepping dmesg on 200 machines.

For NCA-AIIO, knowing what an Xid is and where it surfaces is enough — matching specific codes to specific faults is NCP-AIO work.

7.3 Reading a Bottleneck

Where the Associate line is. The official NCA-AIIO study guide lists Run:ai's 6 Reasons for Low GPU Utilization and How to Improve It as suggested reading under AI Operations, and blueprint bullet 3.3 is "articulate the key measures and criteria related to monitoring GPUs". So "GPU util is low, what is starving it?" is exam material. Reading raw telemetry, matching Xid codes and running dcgmi yourself is NCP-AIO, which carries a whole 23% Troubleshooting and Optimization domain plus hands-on labs.

GPU utilization alone tells you nothing; read it alongside CPU, disk and network. These are the shapes the exam describes in words and asks you to name:

The pattern behind every row: the GPU is the most expensive part, so everything else exists to keep it fed. If it is idle, something upstream is the problem, never the GPU itself.

The rest is NCP-AIO (optional — NCP-AIO level). Same table, but now you are expected to know the memory math and the tooling, not just the name of the bottleneck:

7.4 Multi-Tenancy & Security

This one is on the NCA-AIIO blueprint: "identify key considerations for virtualizing accelerated infrastructure."

• Isolation ladder, weakest to strongest: time-slicing (none) → MPS (shared fault domain) → vGPU (hypervisor) → MIG (hardware) → dedicated GPU.
• Confidential computing (Hopper and Blackwell) encrypts data in use so even the hypervisor or cloud operator can't read GPU memory. For regulated workloads on shared infrastructure.
• The OOB fabric must never be routable from user networks. Same rule as any BMC network, higher stakes.

Phase 8: Data Center Realities

Three blueprint bullets live here — power and cooling, facility requirements, and on-prem vs. cloud — and all three sit inside the 40% infrastructure domain, not operations. More exam weight than the length of this phase suggests.

8.1 Power

• Density — a traditional enterprise rack is 5-10 kW. A single DGX H100 draws ~10.2 kW, and a GB200 NVL72 rack ~120 kW. You cannot drop these into an existing hall and hope.
• Delivery — that density means three-phase distribution and busways rather than ordinary PDUs, and it resizes the whips, breakers, UPS and generator behind them.
• Redundancy — N+1 is one spare unit; 2N is a complete second path at roughly double the cost. 2N is what you buy when a three-week training run must not die.
• Transients — GPUs are not a steady load. Thousands of them entering the same AllReduce swing power hard and fast, which stresses upstream gear in a way a rack of web servers never does.

One DGX SuperPOD scalable unit: 32 DGX H100 across 8 racks. Four systems per rack at ~10.2 kW each is roughly 40 kW in a rack sized for 10. Source: NVIDIA DGX SuperPOD Reference Architecture

8.2 Cooling

Direct-to-chip needs a CDU (coolant distribution unit) to separate the facility water loop from the clean loop touching the cold plates, plus pipework into the row. That is a building project, not a rack purchase.

PUE (Power Usage Effectiveness) = total facility power ÷ IT equipment power. Closer to 1.0 is better. A PUE of 1.5 means every 2 kW of compute costs another 1 kW in cooling and losses. Liquid cooling is one of the main levers for pushing it down.

8.3 Facility Requirements

The unglamorous list that stops projects:

• Floor loading — a fully populated NVL72 rack weighs over a tonne. Raised floors have limits, and so do freight elevators and the route from the loading dock.
• Water — direct-to-chip means supply, return, leak detection and a CDU. Most enterprise halls have none of it today.
• Cable runs — one compute NIC per GPU means eight high-speed links per node plus storage and management. Optics have reach limits, which constrains where a rack can physically sit relative to the spine.
• Space and service access — containment, ceiling height, and enough clearance to actually slide out a 10U chassis that takes two people to lift.

8.4 On-prem vs. Cloud vs. Hybrid

Utilization is the deciding number. A cluster busy 20% of the time should have been rented; one pinned at 90% for three years should have been bought. Hybrid keeps regulated data local and bursts the overflow out — DGX Cloud is NVIDIA's own version of that.

Exam Checklist

Study time should follow the blueprint weights, not your comfort zone.

Essential AI Knowledge (38%)

• Contrast training and inference across duration, bottleneck, failure behavior and scheduler.
• Place AI, ML, DL and GenAI as nested sets; say when classical ML beats deep learning.
• Name the four factors behind AI's recent takeoff, and why the transformer was the unlock.
• Explain batch size, epoch, overfitting, checkpoint — and which two hit infrastructure.
• Name the four learning paradigms with one use case each.
• Pick between accuracy, precision, recall and F1 given a described business cost.
• Explain self-attention in one sentence, and why prefill is compute-bound but decode is memory-bandwidth-bound.
• Choose between prompt engineering, RAG, LoRA and full fine-tuning for a requirement.
• Describe the RAG pipeline and where embeddings and the vector DB fit.
• Match an industry scenario to Clara / Merlin / Riva / Omniverse / Metropolis / Morpheus / NIM.
• Name the layers of the NVIDIA stack from driver to application, and what CUDA-X covers.
• Walk the AI lifecycle and name the tool at each stage, from RAPIDS to Triton to DCGM.

AI Infrastructure (40%)

• Explain why GPUs beat CPUs at matrix math, and what a DPU offloads.
• Say what Ampere, Hopper and Blackwell each introduced.
• Explain why H200 is the inference choice despite H100-identical compute.
• Estimate model memory at a given precision, and say why training needs ~10x inference.
• Distinguish scale-up (NVLink) from scale-out (InfiniBand), and say which you exhaust first.
• Differentiate DGX, HGX and MGX.
• Explain NVLink, NVSwitch and NVLink Switch, and why PCIe isn't enough.
• Differentiate FP32, BF16/FP16, FP8 and FP4 by bytes per value, and say which generation introduced FP8 and FP4.
• Describe MIG's hardware QoS and how it differs from vGPU and time-slicing.
• List the four fabrics and what each carries.
• Explain RDMA vs. GPUDirect RDMA.
• Compare InfiniBand, RoCEv2 and Spectrum-X; say what SHARP does.
• Explain how GDS removes the CPU bounce buffer, and name three parallel filesystems.
• Distinguish TensorRT, Triton and NIM.
• Distinguish quantization, pruning and distillation as ways to shrink a model.
• Explain what NCCL does and name its main collectives.
• Size power and cooling for a 120 kW rack; explain PUE and when liquid cooling stops being optional.
• List the facility requirements that block a deployment: floor loading, water, cable reach, service access.
• Compare on-prem, cloud and hybrid on cost, control, elasticity — and name utilization as the deciding number.

AI Operations (22%)

• Choose between Slurm, Kubernetes + GPU Operator, and Run:ai for a workload.
• List what the GPU Operator deploys and what the Network Operator adds.
• Describe what DCGM monitors and how it reaches Grafana.
• Distinguish Base Command Manager from Base Command Platform.
• Given utilization figures, name the bottleneck and the fix. (NCP-AIO level — lowest priority here.)
• Explain what an Xid error is and where it appears.
• Outline the MLOps lifecycle; distinguish data drift from concept drift.

Quick Recall — Exam-Day Cheat Sheet

Keyword → answer, nothing new.

Keyword → Product

One-Liners

Three core types — CUDA = general parallel, Tensor = matrix math / AI, RT = ray tracing.

GPUDirect — one idea (straight to GPU memory, no CPU/host RAM bounce), three routes: P2P = GPU↔GPU same box, RDMA = GPU↔GPU across the network, Storage (GDS) = GPU↔disk. The question names the route, which names the member.

Utilization ≠ memory usage — util is time busy, memory is capacity occupied. 80/80 GB at 15% util = starved data pipeline, not a busy GPU.

PUE = total facility power ÷ IT power. 1.2 = modern hyperscale, 1.5 = decent enterprise, 2.0 = overhead equals the compute, 3.5/5.0 = distractors. GPU racks are heat bombs, so liquid cooling is the lever — every PUE question with a density number is a cooling question.

Software: Name → Job

TensorRT makes the model fast, Triton puts it in production.

Monitoring: Symptom → Metric

Networking

• IB beats Ethernet on latency — not on "it has RDMA", Ethernet does RDMA too via RoCE.
• Three IB traits: low latency, high bandwidth, CPU offload. Traps: GPU offload, high latency.
• RDMA offloads the CPU through hardware on the NIC.
• IB needs a Subnet Manager to come up — OpenSM is the usual one (or embedded on a switch / UFM).
• Four fabrics: compute, storage, in-band mgmt, out-of-band mgmt. OOB = via BMC, alive when the OS is dead, off the production network.
• Training network = high throughput + low latency. Always eliminate Wi-Fi, 1 GbE, and Fibre Channel (that's storage).

The Rest

• Slurm = job scheduler (HPC). Apptainer = container runtime. MLOps = a methodology, not a tool.
• High density blocked by three things: power, cooling, physical space.
• On-prem wins on security + data sovereignty, cloud on low upfront cost, hybrid on mixed or bursty workloads.
• Predictive maintenance → manufacturing.

Appendix A: Hardware Spec Reference (optional — reference)

Reference only — do not memorize this. None of these numbers are worth exam study time at Associate level, and most aren't worth it at Professional level either. They're here because I keep needing to look them up. What matters is the why in Phase 2, not the digits.

GPU Specs

B100 also exists — a 700W Blackwell that drops into HGX H100 chassis — but NVIDIA ships mainly B200 and GB200.

DGX Systems

CPUs, for completeness: DGX A100 = 2× AMD EPYC 7742 (128 cores); DGX H100/H200 = 2× Xeon Platinum 8480C (112 cores); DGX B200 = 2× Xeon Platinum 8570 (112 cores). Storage on all of them is roughly "2× 1.92TB M.2 for the OS, 8× 3.84TB U.2 for cache".

Marketing Numbers

NVIDIA quotes these constantly; they occasionally show up as answer options:

• DGX B200 vs. DGX H100 — up to 3x training, 15x inference.
• GB200 NVL72 vs. H100 — up to 30x trillion-parameter inference, 4x training, 25x better energy efficiency.

Specs cross-checked September 2026 against the sources below. NVIDIA edits these pages, so re-verify before your exam.

Secret Weapons

I made a quiz set that mirrors this article. It is probably harder than the real exam, but it should test your knowledge as deep as it can xD

Udemy link

Summary & Next Steps

NCA-AIIO is a foundation credential, and the honest way to treat it is as the vocabulary exam before the real ones. Coming from a Kubernetes or Slurm background, most of the operations domain is already familiar — the genuinely new material is the fabric layer and the ML fundamentals.

Preparation order that made sense to me:

1. Watch the YouTube course (free) or take the Coursera course (paid).
2. Work through Phase 1 here until the training-vs-inference table is automatic — everything else hangs off it.
3. Check the official blueprint for anything that shifted.
4. Book it on Certiverse.

Good luck!

This article is a great start for me and I passed the exam as well xD

Image credit: every diagram and photo in this article belongs to NVIDIA, taken from the documentation and blog posts linked beneath each one. None of it is mine.

References

• NVIDIA Certification catalog
• NCA-AIIO official exam page
• NVIDIA MIG User Guide
• NVIDIA GPU Operator Documentation
• NVIDIA GPUDirect Storage Documentation
• NVIDIA Xid Errors
• NVIDIA DCGM
• NVIDIA Run:ai
• NVIDIA NCCL
• DGX A100 Datasheet
• DGX H100/H200 User Guide
• DGX B200 User Guide
• NVIDIA H200 product page
• GB200 NVL72 product page
• Inside NVIDIA Blackwell Ultra
• Attention Is All You Need