01 / Reconcile the delivered configuration

Match each system against the accepted bill of materials: chassis, GPU edition and quantity, CPU, DIMMs, drives, NICs and firmware. Attach serial numbers and resolve substitutions before testing.

For rented compute, document the assigned resources, tenancy and access boundary. This inventory is the baseline to which every subsequent result refers.

02 / Freeze the software baseline

Record OS image, driver, CUDA, framework, container digest and application configuration. Include GPU power policy, fabric settings and whether other jobs shared the test system.

Keep a copy of the test commands and input files. A result is reproducible only when the environment and inputs can be reconstructed.

03 / Test fabric and application separately

A collective-communication test records node count, GPU ranks, message sizes, NIC mapping and the exact test build. Separate intra-node and inter-node measurements. Then run the application with the agreed model, input and concurrency.

Retain the raw logs for both. NIC line rate, collective bandwidth and application throughput describe different parts of the system.

04 / Exercise restart and recovery

Agree the test duration and capture temperatures, throttling, hardware errors and job failures. Save a checkpoint to the intended storage, stop the test workload and restore it. Verify the recovered application state and data integrity.

Any disruptive failure test needs a defined scope, rollback procedure and an approved maintenance window.

05 / Close the acceptance record

For each test, retain the criterion, observed result, evidence location, exception and owner. Close failed items with a retest or an explicitly accepted exception.

The handover names the accepted configuration, operating owner and escalation route. Availability targets and support response times are recorded in the service agreement.

Acceptance record / minimum evidence

TestRecordDecision
InventorySerials, BOM, GPU/NIC mappingResolve substitutions
GPU / fabricDiagnostic logs, NCCL build, message sizes, rank layoutCompare agreed topology
ApplicationModel revision, precision, input, concurrency, p95 latencyApply workload criteria
RecoveryCheckpoint, interruption, restore, data integrityClose exceptions before sign-off

Agree thresholds and run duration before testing. Retain failed runs and retests alongside the final result.

Project checklist
  • Inventory and software baseline
  • Network + application + recovery evidence
  • Exceptions, acceptance owner and support route
Primary references