Evaluating AI storage for startup and checkpoint recovery

The model loads quickly and the GPUs begin working. Then part of the infrastructure fails, and the job needs its checkpoint. Storage now has another task: make that saved state reachable and usable from the recovery environment. The fast first start was useful, but it did not answer the restore question.

This guide uses NetApp and NVIDIA reference documents to follow those two paths. It does not walk through either design. It helps connect a supplier's tested configuration to the work you plan to run and recover.

What the reference configuration covers

NetApp's ONTAP AI design for NVIDIA DGX A100 specifies servers, storage, firmware, network and software together. A later component with a familiar family name can change compatibility or performance. Read the tested combination, not just the product label.

The newer NVIDIA-Certified Storage document adds certification context. Obtain the support guidance for the exact proposed components. A test describes what was evaluated; a contractual recovery commitment describes what someone agreed to provide. One does not automatically supply the other.

Keep the claims tied to storage. Infrastructure for AI workloads does not by itself demonstrate automated diagnosis, remediation or a particular downtime reduction. Ask what this proposed configuration can establish for your workload.

You need not reproduce every vendor test. Focus independent work on mismatched assumptions and consequential recovery paths; use supplied results where their scope actually fits.

Storage access patterns explain workload waits

Start with the workload's rhythm: a few large model files, many small opens, frequent checkpoint writes, or some combination. File sizes, concurrency, metadata work, checkpoint frequency and growth all shape storage demand. Include startup and restore, so a benchmark describes a path you recognize.

Large-file throughput may tell little about a job waiting on small metadata operations. A warm-cache run may differ from the first start after failover. Neither result has to be wrong; keep the conditions beside it so the first cannot masquerade as the second.

Compare useful workload throughput with input waits, storage latency and network saturation. Keep model, batch size and cache policy steady or document changes. Otherwise improved GPU utilization cannot be assigned confidently to the storage change.

Checkpoint recovery reaches beyond the file

Imagine a checkpoint behind a private storage endpoint. The primary compute pool can reach it; after that pool fails, the alternate starts but lacks a permitted network route. Ready machines now wait for the first byte. Adding compute has not addressed the blocked path.

Trace the restore from that byte. In a controlled exercise, can the alternate resolve the endpoint, connect and retrieve the named checkpoint? If access fails, investigate it first. Once bytes arrive, test integrity and application compatibility. This separates a path problem from an artifact or application problem; it is a hypothetical case, not a vendor test result.

Agree exercises with the owners of each dependency. A broken connection, interrupted checkpoint, limited capacity and unavailable management service pose different questions. Find where recovery is automatic and where a person must decide how work continues.

Keep watching after transfer. The application must read the checkpoint, accept its format and resume useful work. Check integrity using checksums or application-level validation, then measure continuation. A preserved snapshot alone does not prove that final step.

Include the handoff between compute, network and storage teams. An alternate that needs the same unavailable store can inherit the original failure. Identify the blocked path's owner and the evidence they need, rather than leaving new machines waiting elsewhere.

A spare compute pool can share the failed dependency

Read diagram description

Illustrative recovery dependency: both compute pools may need the same model or checkpoint store. Test access, artifact integrity and time to useful work from the alternate location. Diagram labels: Model + checkpoint store: Recovery artifacts and their versions; Primary compute pool: Reads artifacts through storage path; Alternate compute pool: May need that same path to recover; Recovery question: Can the alternate pool start useful work if storage is unavailable?.

The outcome now directs the next work. A blocked connection needs network or access changes; an unreadable checkpoint needs artifact or application work; slow successful continuation exposes a recovery-time limit. A successful installation can coexist with any of these unresolved restore questions.

The ongoing cost of the storage system

Put licensing, staffing, migration, backup and data movement beside performance. Compare candidate and current storage with the same workload and objective. The team will live with these recurring costs after the peak-throughput test ends.

The power and cooling guide follows another part of capacity. Compute, power, cooling and storage constrain the service together. A strong result in one still needs understandable limits and recovery behavior in the others.

Keep both the fast load and the checkpoint restore in the decision, including the alternate pool's route to the same store. You are choosing storage for work that should keep running and for work you may have to bring back.