AI's hidden storage bill: What inference leaves behind
2026-10-23 · MIT Technology Review
AI's hidden storage bill: What inference leaves behind
When executives picture the cost of AI, they typically envision racks of GPUs and massive power consumption. However, this perspective overlooks a critical component: data storage costs. After a model responds to a prompt, the vast majority of back-end data generated must be stored. Because storage costs are difficult to plan for, they are frequently overlooked in initial AI infrastructure planning.
As inference scales, the data footprint grows with it
Traditionally, AI infrastructure conversations have centered on the compute-intensive process of model training. Training is finite: a run starts, consumes compute, and ends. Inference, however, runs continuously every time a model answers a prompt. For infrastructure planning, this distinction is crucial. A training run is a project you can budget for and retire. Inference never retires; every request leaves data behind.
Consider AI-generated video. According to internal measurements by WD, an 8-second 720p video clip returned to a user (3.97MB) leaves behind more than 500MB of associated back-end data, including logs, cached context, immediate renders, backups, and safety records. This represents a 130x increase over the output file size. Scaled to an enterprise generating longer, higher-resolution videos, the back-end footprint grows into hundreds of gigabytes per clip.
This footprint is expanding rapidly. McKinsey projects that AI inference demand will grow at a 35% CAGR through 2030, compared to 22% for training, accounting for over 40% of total data center demand by the decade's end. A recent WD-sponsored IDC white paper surveying 763 IT and business decision makers found that nearly 95% reported increased data volume due to generative AI over the past 12 months, with roughly 61% reporting growth of 25% or more.
> 'You can think of having more storage as an optimization for keeping the GPUs busy,' said a senior product manager at a hyperscale cloud provider. 'The biggest challenge is to ensure that the GPUs are not idle, waiting for data to show up—hours of GPU idle time becomes very expensive, very fast.'
Matching storage to the AI data life cycle
Compute demand fluctuates with training cycles and hardware refreshes, but retained data volume does not decrease when compute demand slows. Model logs, synthetic outputs, and inference results continuously accumulate, and organizations are holding onto them longer.
This changes the role of retained data. Information once archived and rarely accessed can now become an input for future AI workloads. As data is retained, retrieved, and reused, storage must support not only growing capacity but also dynamic access. The WD-IDC survey found that 96% of respondents anticipate needing faster archive retrieval to support inference or retrieval-augmented generation (RAG). For these massive volumes of retained data, HDDs provide a cost-effective, high-capacity foundation with a strong cost-per-terabyte advantage.
What this means for how AI gets budgeted
For CFOs and CIOs, this means 'AI infrastructure spend' is often scoped too narrowly on compute. A model idling while data loads costs as much as one idling for lack of GPUs. However, the storage bottleneck remains hidden until it becomes a problem, surfacing in operating costs and retrieval delays rather than as a CapEx line next to accelerators.
Whether it's a retailer generating AI product videos or an insurer processing claims through a RAG pipeline, compute costs are known and forecastable. Storage costs are different: they compound with every video generated and every claim processed, because the back-end data never disappears.
A team that budgets AI as 'compute plus a rounding error for storage' may discover in 12 to 18 months that the rounding error has become a significant fraction of the program's total cost—precisely when leadership decides on expansion.
Building for economically scalable capacity
Addressing this challenge begins with architecture. Storage strategies are often designed for expensive, hot-tier performance, yet most AI data by volume accumulates in warm, cool, and cold tiers. Enterprises need economically scalable capacity: infrastructure designed for the full AI data life cycle, balancing performance with capacity, power, reliability, and TCO.
This requires treating archives as active assets for RAG and inference, setting retention policies based on data's strategic value, and recognizing that storage decisions play out across fleets, not individual drives. Such infrastructure relies on tiered storage, where power consumption, reliability, density, and operational efficiency determine economic sustainability as capacity scales.
When evaluating your organization’s AI infrastructure budget, consider these questions:
- What is our data retention policy for AI-generated data?
- How are we balancing performance and capacity across different storage tiers?
- Can our cold storage archives support rapid retrieval for RAG pipelines?