Mount Object Storage to a GPU Instance for Training Data
A practical tutorial for mounting object storage on a cloud GPU instance so large training datasets stream in without overflowing local disk.
Training datasets routinely outgrow the local disk on a GPU instance. A single image or video corpus can run into terabytes, far more than the boot volume of a typical cloud GPU node. Mounting object storage lets you treat a bucket as if it were a local directory, so your training job reads data on demand instead of waiting for a giant copy to finish. This tutorial shows how to mount object storage to a GPU instance, tune it for throughput, and avoid the cost traps that catch people out.
Why mount instead of copy
The naive approach is to download the whole dataset to local disk before training. That works for small datasets, but it wastes time and money at scale. Your expensive GPU sits idle during the copy, the local volume may not be large enough, and you pay for storage twice. Mounting object storage solves all three. The data stays in the bucket, your code reads file paths normally, and a FUSE layer fetches bytes as they are requested.
The tradeoff is that reads now travel over the network. With good caching and prefetch this is usually fine for training, but it means you must think about throughput, latency, and provider egress policies.
Choose a mount tool
Most object stores speak an S3 compatible API, and several mature FUSE clients exist. The common choices are:
- s3fs style FUSE mounts that expose a bucket as a POSIX directory.
- Provider native CSI drivers and mount helpers, which often perform better than generic tools.
- Caching file systems that keep hot objects on local disk and fall back to the bucket for cold reads.
For training, a caching mount is almost always the right pick, because epochs reread the same files many times. The first epoch pays the network cost, and later epochs read from the local cache at disk speed.
Mount the bucket step by step
The exact commands vary by tool, but the shape is consistent. Here is the general flow on a Linux GPU instance.
- Install the FUSE client and the kernel FUSE module if it is not already present.
- Place your access credentials in an environment variable or a credentials file with tight permissions.
- Create an empty local mount point, for example a directory under your data path.
- Run the mount command pointing the client at your bucket, endpoint, and region.
- Verify the mount by listing the directory and reading one small file.
Set a generous local cache directory on the fastest local disk you have, ideally an NVMe scratch volume. Point the mount cache there so repeated reads stay off the network. Confirm the cache is working by watching network traffic drop on the second epoch.
Tune for GPU throughput
A GPU can consume data faster than a single threaded reader can supply it. If your GPU utilization sits low while training, the data path is probably starving the device. Several knobs help.
- Increase the number of data loader workers so multiple objects download in parallel.
- Enable readahead or prefetch so the next batch is fetching while the current one trains.
- Pack many small samples into larger shard files, because object stores handle a few large reads far better than thousands of tiny ones.
- Use sequential shard formats designed for streaming rather than random per file access.
Sharding is the single biggest lever for small file datasets. Reading one million tiny images one at a time over a network mount is painfully slow. Bundling them into a few hundred shard files turns that into a handful of large, efficient reads.
Watch egress and request costs
Mounting is cheap until the bill arrives. Two costs dominate.
| Cost driver | What triggers it | How to control it |
|---|---|---|
| Egress | Reading data out of a different region or cloud | Keep the bucket in the same region as the GPU |
| Request count | Many small GET operations | Shard data into larger objects |
| Repeated reads | Re-fetching the same files each epoch | Use a local cache mount |
The golden rule is to colocate. Put your bucket in the same region, and ideally the same provider, as your GPU instance. Cross region or cross cloud reads add latency and can add egress charges on every byte. If your GPU provider and storage provider differ, check whether a zero egress storage tier or a peering arrangement exists before you start a long training run.
Verify and harden the setup
Before you launch a multi day run, run a short smoke test. Train for a few hundred steps and confirm GPU utilization stays high, the cache fills, and network traffic falls on later epochs. Add the mount to a startup script so it survives reboots, and use read only credentials for training jobs so a bug cannot delete source data.
Plan for failures and resumability
Network mounts can hiccup, and a long training run should survive a transient read error rather than crash hours in. Build resilience into both the mount and the job. Configure the FUSE client to retry failed reads with backoff so a brief network blip does not surface as a hard error to your data loader. On the training side, checkpoint regularly to durable storage so an interruption costs minutes, not the whole run.
It also helps to validate the dataset before the GPU clock starts. Run a quick pass that lists every shard and reads a sample from each, so a missing or corrupt object surfaces immediately rather than three hours into training. Catching a bad path early on a cheap CPU step is far better than discovering it after a costly GPU has been spinning.
When a mount is the wrong choice
Mounting is excellent for large datasets read many times, but it is not always the answer. For a small dataset that fits comfortably on local disk, a one time copy is simpler and faster overall, because every read then comes straight from local storage with no network in the path. For workloads that read each file exactly once in a single pass, streaming directly from the object store without a cache can be cleaner than a full mount, since there is no reuse for a cache to exploit.
The decision comes down to dataset size relative to local disk and how many times you reread the data. Many epochs over a large corpus strongly favor a caching mount. A single pass over a modest dataset favors a plain copy or a simple stream. Pick the pattern that matches your access shape rather than mounting reflexively.
Conclusion
Mounting object storage gives a GPU instance access to datasets far larger than its local disk, without a slow upfront copy. The recipe is simple: pick a caching FUSE mount, shard your data into large objects, parallelize the loader, and keep storage in the same region as the GPU to dodge egress. Get those right and your GPU stays fed while your storage bill stays predictable.