◐ system-design/design-youtube.md

34. Design YouTube / Video Streaming

Tests: object storage, video encoding pipeline, CDN, adaptive bitrate streaming, recommendation, comments, view counting at scale.

~5 min read·updated 5/29/2026

34. Design YouTube / Video Streaming

Tests: object storage, video encoding pipeline, CDN, adaptive bitrate streaming, recommendation, comments, view counting at scale.

34.1 Requirements

Functional

  • Upload videos.
  • Watch videos (adaptive bitrate streaming).
  • Search videos.
  • Comments, likes, subscriptions.
  • Recommendations.
  • Live streaming (extension).

Non-functional

  • 2B users; ~500 hours of video uploaded per minute.
  • ~5B views per day.
  • High availability for playback.
  • Worldwide low-latency playback.
  • Years of content retention.

34.2 Estimates

  • Uploads: 500 hr/min × 1 GB/hr = 500 GB/min raw. Compressed multi-resolution: 5×.
  • 5B views/day ÷ 86,400 = ~58K plays/sec average; peak ~200K plays/sec.
  • Bandwidth: avg 1080p ~5 Mbps. 200K concurrent streams = 1 Tbps. Real numbers higher.

Storage:

  • Years of upload at multi-resolution (240p, 480p, 720p, 1080p, 4K, audio variants) = exabytes.

This is a problem of scale, but conceptually manageable: object storage + CDN + transcoding pipeline.

34.3 High-level architecture

[ Upload client ] ─→ [ Upload service ] → [ Object storage (raw) ] → Kafka(uploaded)
                                                                        │
                                              [ Transcode workers ] ─→ [ Object storage (variants) ]
                                                                        │
                                                            Metadata DB ↑

[ Watch client ] → [ CDN ] → [ Object storage (variants) ]
                       ↘ origin shielding
[ Player ] gets manifest from [ Watch service ] → [ Metadata DB ]
[ Search ] → Elasticsearch
[ Recommendations ] → ML serving
[ Counts service ] → Counter store

34.4 Upload pipeline

  1. Client requests a signed upload URL.
  2. Client uploads to object storage (resumable; chunked for large videos).
  3. Object storage emits UploadComplete event to Kafka.
  4. Transcode service picks up; spawns jobs.

Resumable / chunked upload

TUS protocol or S3 multipart. Large files (multi-GB) need resumability across network failures.

34.5 Transcoding pipeline

The expensive part. Each upload generates many outputs:

  • Resolutions: 144p, 240p, 360p, 480p, 720p, 1080p, 1440p, 2160p (4K).
  • Codecs: H.264 (compatibility), VP9, AV1 (bandwidth efficient).
  • HLS / DASH segments (~6 sec each).
  • Thumbnails at multiple timestamps.
  • Audio tracks.

Workflow:

  • Split video into chunks for parallel transcode.
  • Workers pull from Kafka job queue.
  • GPU/ASIC acceleration (Google uses VCUs).
  • Each chunk transcoded → uploaded to object storage.
  • Once all chunks done, manifest (HLS .m3u8 / DASH .mpd) generated.

Time to availability: minutes for short clips; hours for long uploads.

34.6 Adaptive Bitrate Streaming (ABR)

The player downloads a manifest listing variants (different bitrates) and downloads segments at the appropriate bitrate.

HLS (HTTP Live Streaming)

Apple's. The dominant protocol. .m3u8 master playlist → .m3u8 per-variant playlists → .ts (or .m4s) segments.

DASH (Dynamic Adaptive Streaming over HTTP)

Open standard. .mpd manifest → segments. Fewer client implementations.

Player behavior

  1. Fetch manifest.
  2. Start at low bitrate (fast first frame).
  3. Measure download speed of each segment.
  4. Step up/down bitrate based on bandwidth and buffer health.
  5. Maintain 10-30s buffer.

Why ABR

Network conditions vary per user, per minute. A fixed bitrate either buffers (too high) or wastes bandwidth (too low).

34.7 CDN: the heart of playback

Videos must be cacheable at the edge. 90%+ of bytes served from CDN.

Two-tier:

  • Edge PoPs cache popular content.
  • Regional shields cache the rest.
  • Origin (object storage) serves cache misses.

For YouTube scale, Google operates its own global CDN (Google Edge Network). For most companies, use CloudFront / Cloudflare / Fastly.

Hot videos (viral) are pre-positioned on many PoPs. Cold/unpopular only on a few; first request to a region warms.

34.8 Storage

Object storage holds variants. S3, GCS, Colossus (Google internal).

Lifecycle:

  • Recent videos: hot tier.
  • Older/unwatched: warm or cold (cheaper, slower).
  • Very old/popular: pinned in cache.

34.9 Metadata DB

videos (
  video_id, owner_id, title, description, duration, upload_date,
  status, primary_lang, manifest_url, thumbnail_url, ...
)
view_counts (video_id, count)
likes, comments, etc.

Sharded by video_id. Scale: billions of rows. Spanner / Bigtable / Vitess (sharded MySQL).

34.10 Search

(See chapter 25.) Elasticsearch / Vespa.

  • Index title, description, transcript, channel name.
  • Boost by views, recency, channel authority.
  • Faceted: category, duration, upload time.

YouTube uses Vespa (Yahoo origin) for ranking — handles billions of docs with rich ML scoring.

34.11 Recommendations

The killer feature. Two-stage:

Candidate generation

From a corpus of billions, get a few thousand candidates per user. Multiple sources:

  • Watch history (collaborative filtering).
  • Subscriptions.
  • Trending.
  • Topical interest.
  • Search context.

Approximate nearest neighbor over user × video embeddings.

Ranking

For top candidates, run a heavyweight neural ranker that scores: probability of click + watch time + likelihood of long-term satisfaction.

The classic 2016 paper "Deep Neural Networks for YouTube Recommendations" describes this two-stage system.

34.12 Comments

  • Per-video comments table; sharded by video_id.
  • Threading: parent_comment_id; max depth.
  • Live update via WebSocket / SSE.
  • Spam detection: ML + abuse rules.
  • Toxicity / moderation: classifier in pipeline.

34.13 Likes / subscriptions

  • Likes: counter per video; user has liked or not (set in Cassandra by user × video).
  • Subscriptions: edge in graph (user → channel).
  • Counters at scale: see counting system below.

34.14 View count

A counter that gets hammered. Approaches:

Naive

Increment on every play. At 200K plays/sec across millions of videos, this melts a single counter.

Sharded counter

Per video, N counter shards (e.g., 10). Increment a random shard. Read = sum.

Stream + materialized counter

Each play emits a play_event to Kafka. Stream processor (Flink) aggregates per video per minute, writes to durable counter store. View count = sum.

This is what most large systems do.

Approximate

At extreme scale, HyperLogLog for unique viewers; Count-Min for plays.

34.15 Live streaming (extension)

  • Ingest server (RTMP / WebRTC / SRT) accepts the stream.
  • Transcoded into HLS chunks in real time.
  • Chunks uploaded to object storage and propagated to CDN.
  • Latency: ~5-30 sec for HLS; sub-second for low-latency HLS or WebRTC.

For ultra-low latency (interactive): WebRTC. For sports / live events: low-latency HLS.

34.16 Failure modes

  • CDN PoP failure: BGP / DNS routes around.
  • Transcode worker pool overload: queue backs up; new uploads delayed (acceptable; not the watch path).
  • Object storage region failure: cross-region replication; failover.
  • Metadata DB primary down: replicas promoted; brief read-only blip.

34.17 Cost

Bandwidth dominates. Multi-billion-dollar bandwidth bill globally. Hence Google operates its own CDN; Netflix has Open Connect; Twitch has its own ingest network.

34.18 Sketch for interview

[ Uploader ]──→[ Upload svc ]──→[ Object store (raw) ]──→Kafka──→[ Transcode workers ]──→[ Object store (variants) ]
                                                                                            │
                                                                                  [ Metadata DB ]

[ Player ]──→[ Watch svc ]──→ manifest URL
[ Player ]──→[ CDN ]──→[ Object store ]   (HLS/DASH segments)
[ Search svc ]──→[ ES / Vespa ]
[ Recsys ]──→[ ML serving ]──→features from feature store
[ Comments svc ]──→[ Comments DB ]──→pubsub for live update
[ Counts svc ]──→Kafka(play events)──→[ Flink ]──→[ Counter store ]

34.19 What an interviewer wants

  • Object storage + CDN as the bulk of bytes.
  • Transcoding as async pipeline driven by events.
  • ABR streaming (HLS / DASH).
  • View counts via stream aggregation, not direct DB increments.
  • Awareness that recommendations is the platform's moat.

Key takeaways

  • Video lives in object storage; transcoded to many bitrates and formats; served via CDN.
  • HLS / DASH adaptive bitrate streaming is non-negotiable.
  • Counters at scale: sharded counters + stream aggregation, never direct DB counters.
  • Recommendations: two-stage (candidate gen + heavy ranker), the actual product moat.
  • Bandwidth is the cost driver; CDN and codec choices matter at scale.

// 2 views

●main
UTF-8·typescript