01 / Reconcile the delivered configuration
Match each system against the accepted bill of materials: chassis, GPU edition and quantity, CPU, DIMMs, drives, NICs and firmware. Attach serial numbers and resolve substitutions before testing.
For rented compute, document the assigned resources, tenancy and access boundary. This inventory is the baseline to which every subsequent result refers.
02 / Freeze the software baseline
Record OS image, driver, CUDA, framework, container digest and application configuration. Include GPU power policy, fabric settings and whether other jobs shared the test system.
Keep a copy of the test commands and input files. A result is reproducible only when the environment and inputs can be reconstructed.
03 / Test fabric and application separately
A collective-communication test records node count, GPU ranks, message sizes, NIC mapping and the exact test build. Separate intra-node and inter-node measurements. Then run the application with the agreed model, input and concurrency.
Retain the raw logs for both. NIC line rate, collective bandwidth and application throughput describe different parts of the system.
04 / Exercise restart and recovery
Agree the test duration and capture temperatures, throttling, hardware errors and job failures. Save a checkpoint to the intended storage, stop the test workload and restore it. Verify the recovered application state and data integrity.
Any disruptive failure test needs a defined scope, rollback procedure and an approved maintenance window.
05 / Close the acceptance record
For each test, retain the criterion, observed result, evidence location, exception and owner. Close failed items with a retest or an explicitly accepted exception.
The handover names the accepted configuration, operating owner and escalation route. Availability targets and support response times are recorded in the service agreement.
Acceptance record / minimum evidence
| Test | Record | Decision |
|---|---|---|
| Inventory | Serials, BOM, GPU/NIC mapping | Resolve substitutions |
| GPU / fabric | Diagnostic logs, NCCL build, message sizes, rank layout | Compare agreed topology |
| Application | Model revision, precision, input, concurrency, p95 latency | Apply workload criteria |
| Recovery | Checkpoint, interruption, restore, data integrity | Close exceptions before sign-off |
Agree thresholds and run duration before testing. Retain failed runs and retests alongside the final result.
- Inventory and software baseline
- Network + application + recovery evidence
- Exceptions, acceptance owner and support route