Clockwork.io Guarantees AI Training Uptime

Clockwork.io, a startup specializing in AI cluster efficiency, is making a deal with its customers: The vendor promises that 90% of training failures on workloads deploying its TorchPass fault tolerance software for AI infrastructure will be resolved by automatically shifting to spare resources. And if they’re not, Clockwork.io will issue a 25% credit against the customers' next TorchPass renewal or additional licensing.

Clockwork.io calls it the “You Only Compute Once” or YOCO Guarantee, and it’s aiming it squarely at AI service providers, aka neoclouds or altclouds, which suffer regular GPU outages that only increase with the scale of the network.

In a study published by AI researchers at Meta in 2025, the authors says that Mean Time to Failure (MTTF) predictably decreases the larger the network gets. “[W]e project the MTTF for 16384 GPU jobs to be 1.8 hours and for 131072 GPU jobs to be 0.23 hours.” In other words, training jobs running on clusters of 131,072 GPUs can expect to fail nearly every 14 minutes.

Clockwork.io has some numbers of its own: In a setup of 2,048 GPUs, failures resulting in restarts can cost over $6 million annually in wasted compute. TorchPass can prevent expensive AMD and NVIDIA GPUs from losing their value to AI developers through failures and the time consumed to correct them.

GPU Failure and Fixes

GPUs fail for a variety of reasons. Some are catastrophic, or sudden, such as power or hardware failures. Others are gradual, such as diminishing GPU performance due to memory problems. Then there are “lemon nodes,” or GPUs that just don’t work in live AI environments even though they’ve tested well in the lab.

All these failures require engineering intervention, meaning downtime that can consume a significant portion of goodput, or the amount of work that can be completed in a given time period. According to neocloud Crusoe, if an organization spends 19% of a 24-hour training period fixing problems, that leaves goodput of 81%.

Whatever interferes with goodput reduces reliability, Clockwork says, a measure that hasn’t been accurately tagged in the industry. Clockwork CEO Suresh Vasudevan puts it this way:

"AI teams need their models to be done, not their nodes to be up. The industry has been measuring node uptime and calling it reliability. YOCO holds us accountable for the only thing that matters - your model, done.”

What TorchPass Does

As noted, Clockwork.io’s YOCO Guarantee is for users of its TorchPass software, which automatically “migrates” or shifts from a GPU or group of GPUs to other chips in the event of failures. The software tracks warning signs of potential failure and allows for planned maintenance or patching without stopping training.

TorchPass works with Clockwork.io’s basic FleetIQ technology, which deploys complex telemetry developed at Stanford, to create “a Waze for GPU clusters,” according to Dan Zheng, Clockwork’s VP of Products and Solutions. FleetIQ isn’t required to run TorchPass, but it relates signals for proactive or pre-emptive migration to TorchPass.

Clockwork.io isn’t alone in attempting to deal with GPU unreliability. Neocloud Crusoe, for instance, says its AutoClusters technology automatically detects and replaces failed GPUs in under 5 minutes. CoreWeave says its CoreWeave Cloud is built with tools and techniques that detect problems in training stalls and slowdowns that are otherwise tough to diagnose.

Clockwork.io claims to differentiate by the speed of GPU migration or replacement. Its YOCO Guarantee is an effort to prove its resolve to restore GPU access with, as the press release states, “no lost training progress, no checkpoint rollback, and no recompute.”

At least one customer, European cloud and AI infrastructure provider Scaleway, is ready to embrace the Clockwork.io’s guarantee: Fred Bardolle, Head of Products and AI at the provider, stated:

“The YOCO Guarantee is the first infrastructure commitment we've seen built around the right metric - whether progress is protected and the jobs keep running to completion, not whether the hardware stays up. That's the accountability model the AI infrastructure market has been missing.”

Futuriom Take: Clockwork.io has taken a big step toward validating GPU goodput with its YOCO Guarantee, which sends an important message about the reality of AI environments and the industry’s definition of reliability.