◐ system-design/design-dropbox.md

37. Design Dropbox / Google Drive

Tests: file storage at scale, chunking, deduplication, sync engine, conflict resolution, sharing & permissions.

~4 min read·updated 5/29/2026

37. Design Dropbox / Google Drive

Tests: file storage at scale, chunking, deduplication, sync engine, conflict resolution, sharing & permissions.

37.1 Requirements

Functional

  • Upload / download files.
  • Sync across devices.
  • Share with other users (read / write permissions).
  • File versioning.
  • Browse folders.
  • Search.

Non-functional

  • Hundreds of millions of users.
  • Petabytes of storage.
  • Reliable: never lose user data.
  • Efficient: avoid re-uploading unchanged content.
  • Scalable sync (many devices per user).

37.2 Estimates

  • 500M users × 50 GB avg storage = 25 EB total. Stored as objects in commodity storage with replication.
  • Daily new uploads: ~1 TB/sec at peak globally (rough).
  • Average file size: photo ~3 MB; doc ~100 KB; long-tail of GB+ videos.

37.3 High-level architecture

[Desktop client]──→[Block svc]──→[Object store (chunks)]
[Mobile]──→[Block svc]
[Web]──→[Web svc]──→[File metadata svc]──→[Metadata DB]
[Sync svc]──notify changes via long-poll/WS──→[Clients]
[Share svc]──→[Permissions DB]
[Search svc]──→[ES]

37.4 The chunking insight

Don't store whole files. Split into chunks (e.g., 4 MB).

Why

  • Resumable uploads: only re-upload failed chunks.
  • Deduplication: same chunk across users stored once.
  • Sync efficiency: only changed chunks transferred.
  • Parallel transfer: many chunks in flight.

Content-addressed

Hash each chunk (SHA-256) → use hash as the storage key. Identical content → same chunk → automatic dedup.

A "file" is a manifest of chunk hashes.

file:
  id: f123
  name: presentation.pptx
  chunks:
    - hash: abc...  (offset 0-4MB)
    - hash: def...  (offset 4-8MB)
    - ...

Compression

Optional per chunk; skip if already compressed (jpg, mp4).

37.5 Storage tiers

  • Object store (S3, GCS, custom): chunks. Replicated 3-way or erasure-coded.
  • Metadata DB (sharded SQL, e.g., Postgres or Vitess): file metadata, folders, permissions.
  • Cache (Redis): recent listings, hot file metadata.
  • CDN for shared public files.

Chunks are immutable. Garbage collection deletes orphaned chunks.

Erasure coding

Beyond replication: encode N chunks into N+M shards; survive M failures. ~1.5× overhead vs 3× for replication. Used by Google Colossus, Facebook f4, Backblaze.

37.6 Upload flow

1. Client splits file into chunks; computes hashes.
2. Client → Block svc: "do you have these chunks?" (hash list).
3. Block svc returns list of missing chunks.
4. Client uploads only missing chunks (parallel).
5. Once all chunks present, client → File metadata svc with manifest.
6. Metadata svc creates file entry; emits FileChanged event.

This is the dedup-on-write trick. If a user uploads a file already present (someone else's identical), zero bytes go over the network.

37.7 Sync engine

Each client maintains a local index of files + hashes. On change:

  • Local watcher detects file change.
  • Chunk + hash; diff against cloud manifest.
  • Upload changed chunks; update manifest.

For incoming changes:

  • Sync svc subscribes to "FileChanged" events for files the client cares about.
  • Push to client via long-poll / WS.
  • Client downloads changed chunks; updates local copy.

Conflict resolution

Two clients edit same file concurrently:

  • Both upload manifests.
  • Server detects conflict (parent version mismatch).
  • Strategy: keep both ("Bob's conflicted copy") or merge (text files).
  • Notify users; let them resolve.

CRDTs would auto-merge; Dropbox doesn't because file types vary.

37.8 Metadata model

files (
  file_id, owner_id, parent_folder_id, name, size,
  current_version_id, latest_modified_at,
  trashed_at, ...
)
file_versions (
  version_id, file_id, manifest, created_at, created_by
)
chunks (
  chunk_hash, size, ref_count
)
folders (
  folder_id, owner_id, parent_folder_id, name
)
shares (
  file_id, shared_with_user, permission (read/write), created_at
)

Sharded by owner_id for personal files; queries by owner are fast.

For shared / collaborative files, look up via shares table.

37.9 Versioning

Each save = new version. Manifest of new chunks; reuse unchanged.

  • Old versions retained for N days (configurable).
  • Storage cost amortized via dedup of unchanged chunks.

37.10 Sharing

Per-file ACL:

  • Owner (full control).
  • Editors (read/write).
  • Commenters (read + comment).
  • Viewers (read).
  • Public link (anyone with link).

Authorization checked on every request.

For collaboration / multi-user editing: real-time sync via OT or CRDTs (Google Docs, not Dropbox classic).

37.11 Search

Full-text on file content + metadata.

  • On upload, async indexing pipeline extracts text (Tika, OCR for images).
  • Indexed into ES.
  • Per-user filter on query.
  • Permission-aware (don't return files user can't see).

37.12 Garbage collection

Chunks have ref count: +1 per file referencing them.

  • File deleted → manifest removed → chunk ref-counts decremented.
  • Chunks with ref-count 0 → garbage collected after grace period (in case of restoration).

GC is async; can be done as periodic job over chunk catalog.

37.13 Multi-device challenges

User has phone + desktop + iPad. Each maintains local state.

  • Selective sync: user picks which folders sync to each device.
  • Streaming download: file opened on demand; chunks fetched as needed (Smart Sync / Files On-Demand).
  • Offline access: cached chunks + delta sync on reconnect.

37.14 Mobile considerations

  • Background sync: respect OS limits.
  • Bandwidth aware: pause on cellular, resume on WiFi.
  • Battery: batch syncs.

37.15 Failure modes

  • Object store partition down: chunks unavailable; fall back to alternate region replicas.
  • Metadata shard down: that user's files unavailable; replica promoted.
  • Conflict storm: many clients editing one file; queue and serialize.
  • Sync lag: clients see out-of-date views briefly.

37.16 Sketch for interview

[Client]──hash chunks──→[Block svc]──"have you got these?"──→[Chunk store (object storage)]
[Client]──upload missing chunks──→[Object storage]
[Client]──manifest──→[File metadata svc]──→[Metadata DB sharded by owner_id]
                                                  ↓ FileChanged event
                                            [Sync svc fan-out]
                                                  ↓
                              [Other devices via WS / long-poll]
[Search] async-indexed via Kafka → ES
[GC job] sweeps unreferenced chunks

37.17 Google Drive specifics

  • Built on Spanner (metadata) + Colossus (objects).
  • Real-time collaboration in Docs/Sheets/Slides via OT (operational transformation).
  • Drive content vs Docs content: different storage backends.

37.18 What an interviewer wants

  • Chunking + content-addressed storage + dedup as the core idea.
  • Manifest model: a file is a list of chunk hashes.
  • Object storage for chunks; metadata DB for structure.
  • Async sync via change events.
  • Conflict resolution policy.
  • Erasure coding awareness for storage efficiency at scale.

Key takeaways

  • Chunk + hash = dedup, resumable, efficient sync.
  • Manifest model: file is a list of chunk hashes.
  • Object storage handles bytes; metadata DB handles structure.
  • Sync engine: change events + long-poll/WS to clients.
  • Erasure coding (Reed-Solomon) saves ~50% of storage at scale.

// 1 view

●main
UTF-8·typescript