Problem RestatementProblem
A new model version is ready: its weights are 500 GB (or more). It must be copied to 1,000+ GPU hosts as fast and reliably as possible so they can start serving it. The source (object storage or a seed machine) has limited bandwidth, and each host has a network link of limited speed (e.g., 25 Gbps shared for upload and download). Anthropic asked this in several forms: "stream a large file to 1,000 hosts fastest", "peer-to-peer under a shared link cap", "deploy a 500 GB model to GPU workers".
RequirementsRequirements
- Deliver the identical file (or set of files) to N hosts.
- Minimize the total time until all hosts have it.
- Verify integrity (no corrupted weights).
- Survive host and network failures, and resume without starting over.
- Switch hosts to the new version safely (no half-loaded models).
Bandwidth Math (do this out loud)
- File F = 500 GB = 4,000 Gb (gigabits). Link per host = 25 Gbps.
- One host downloading at full speed needs 4,000 / 25 = 160 seconds at the very least.
- Naive: every host downloads from one source with a 100 Gbps link. Total data = 1,000 × 4,000 Gb = 4,000,000 Gb → 40,000 seconds (11 hours). The source is the bottleneck.
- With peer sharing: once hosts have pieces, they upload to others. Total upload capacity grows with the number of hosts, so the ideal time approaches F / link speed, i.e., about 160 seconds plus overhead. That's why the answer is chunking + peer-to-peer.
Deep Dive — Getting 500 GB onto 1,000 hostsDeep dive
The arithmetic above sets the floor: 160 seconds, the time one host needs to pull 4,000 Gb down a 25 Gbps link. Every design below is judged against that number.
Every host downloads from the source
All 1,000 hosts pull the file from object storage or a seed machine.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
SRC[("Source - 100 Gbps")] --> H1["Host 1"]
SRC --> H2["Host 2"]
SRC --> H3["Host 1000"]
SRC --> MATH["1000 x 4,000 Gb = 4,000,000 Gb through one 100 Gbps pipe"]
MATH --> T["about 11 hours"]Every byte crosses the source's link once per host, so the source's bandwidth is divided a thousand ways. Eleven hours against a floor of 160 seconds — roughly 250x off — and meanwhile 1,000 hosts' worth of upload capacity sits completely unused.
Fan out through a tree, or pipeline down a chain
k forward chunk n to host k+1 while receiving chunk n+1. Every link runs at full speed at once, and the total approaches F / bandwidth plus a small pipeline fill — very close to the theoretical floor.
Both are enormous improvements and both are brittle in the same way: they impose a fixed topology on a fleet where hosts fail. A slow node in a tree delays its entire subtree; a dead node in a chain stops everything downstream of it. At 1,000 hosts, something being slow or dead is the normal state, not an exception, so the structure needs constant repair.
A swarm that has no fixed shape
Split the file into chunks — 64 MB is a reasonable size — and let every host fetch chunks from the source and from each other. A tracker records who holds what.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
SRC[("Source - seeds chunks")] --> A["Host A - has 1, 7"]
SRC --> B["Host B - has 3, 9"]
A <--> B
A <--> C["Host C - has 2, 5"]
B <--> C
TR["Tracker - who has which chunk"] --- A
TR --- B
TR --- C
C --> RARE["Rarest chunk first - every chunk spreads fast"]Total upload capacity now grows with the fleet: the more hosts that hold pieces, the faster the remaining hosts fill up, so the time approaches the 160-second floor plus overhead rather than degrading with scale.
Two details carry most of the benefit:
- Rarest chunk first. Hosts prefer the chunk the fewest peers hold. Without it, common chunks get replicated endlessly while one unlucky chunk becomes a bottleneck everybody waits on at the end.
- No repair logic needed. A slow or dead peer is simply a peer other hosts stop choosing. There is no subtree to re-parent and no chain to splice — which is precisely what the tree and the pipeline had to build by hand.
Verify each chunk against its hash on arrival, and the final file against the manifest. At this size a silently corrupted chunk that reaches a thousand GPU hosts is a much worse day than a slow transfer.
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
REG["Model Registry - version, manifest, checksums"] --> CO["Distribution Coordinator / tracker"]
OS[("Object storage - seed")] --> H1["GPU host - rack A"]
OS --> H2["GPU host - rack B"]
H1 <-->|"chunks"| H3["GPU host - rack A"]
H2 <-->|"chunks"| H4["GPU host - rack B"]
H1 <-->|"chunks"| H2
CO -->|"who has which chunks"| H1
CO --> H2
CO --> H3
CO --> H4- Manifest: the list of chunks with a SHA-256 for each. Hosts verify every chunk before sharing or using it.
- Coordinator: tracks chunk availability and suggests peers. Prefers peers in the same rack (fast, cheap links) and limits cross-rack or cross-zone traffic.
- Seeds: object storage plus a few "super-seed" hosts that get the file first and have full upload capacity.
Key Details
- Shared link cap: with one link used for both upload and download, balance them. Each host uploads roughly as much as it downloads, and the coordinator limits parallel connections per host.
- Topology awareness: seed at least one host per rack first, then spread within racks. Cross-rack links are often oversubscribed.
- Failures: a failed download of a chunk is retried from another peer. A host that dies just drops out. Progress is saved per chunk, so a restarted host resumes where it stopped.
- Integrity: verify per-chunk hashes and a final whole-file hash before loading.
- Loading into GPUs: many hosts can start loading completed shards (e.g., per tensor-parallel shard) before the full file arrives, if the model is split into shard files.
Rolling Out the New Version
- Pre-stage: distribute the new weights to local NVMe while the old version keeps serving.
- Verify the checksums on every host.
- Switch in waves: drain a subset of servers, load the new model, run a quick health and eval check, and put them back into rotation. Watch error and latency metrics before the next wave.
- Keep the previous version on disk for fast rollback.
- Garbage-collect old versions later, keeping N versions.
Trade-offs & AlternativesTrade-offs
| Approach | Good | Bad |
|---|---|---|
| Single source | Simple | Source bandwidth bottleneck (hours) |
| Tree fan-out | log(N) depth | Slow node delays its subtree, uneven link use |
| Pipeline chain | Near-optimal bandwidth use | Fragile to slow or failed hosts |
| P2P swarm (chosen) | Scales with hosts, robust | Needs a coordinator, more moving parts |
Wrap-UpWrap-up
Do the math first: a single source takes hours, while peer sharing approaches file size ÷ link speed. Split the weights into hashed chunks, seed one host per rack, and let hosts swap chunks rarest-first with a topology-aware coordinator, verifying every chunk and resuming after failures. Pre-stage weights on local disk, then switch serving to the new version in health-checked waves, keeping the old version for rollback.