1. Fundamentals: Reliability, Scalability, Maintainability

Three properties define a "good" production system. Every architectural choice is a trade-off between them.

1.1 Reliability

Reliability = the system continues to work correctly, even when things go wrong.

"Things going wrong" = faults. A fault is a component deviating from spec. A failure is the system as a whole stopping doing what users need. The goal is fault tolerance: prevent faults from causing failures.

You cannot eliminate faults. You can only design so faults don't escalate to failures.

Categories of faults

1. Hardware faults. Disks fail (~2-5% AFR), RAM has bit flips, networks drop packets, datacenters lose power. Mean Time To Failure (MTTF) of a single disk is ~10 years; with 10,000 disks you lose ~3/day. Solution: redundancy (RAID, replication, multi-AZ deployments).
2. Software faults. Bugs, runaway processes, cascading failures, leap-second bugs (Linux kernel ~2012). Harder to detect because they correlate across nodes — one bad config or one poison message kills everything. Solution: process isolation, circuit breakers, gradual rollouts, careful monitoring.
3. Human errors. ~80% of major incidents are human-triggered (bad deploy, wrong config, accidental delete). Solution: minimize blast radius (staging, canary), recovery (point-in-time restore), automated checks (lint, tests, type systems), good UX in tooling.

Designing for reliability

• Redundancy: N+1 (one extra) or N+2 for critical components.
• Graceful degradation: when ML scoring is down, show recency-ordered feed instead of a 500.
• Bulkheads: isolate failures (per-tenant connection pools so one bad tenant can't drain the pool).
• Retries with backoff and jitter: don't herd; don't retry idempotency-unsafe ops blindly.
• Health checks and circuit breakers: stop sending traffic to broken instances.
• Chaos engineering: Netflix's Chaos Monkey kills random instances in prod to force the team to build for failure.

1.2 Scalability

Scalability = the system's ability to cope with increased load.

Not a yes/no property. Always ask: scalable in what dimension? (users, requests/sec, data volume, response time at percentile P).

Load parameters

Pick the right number to track. For Twitter, "tweets/sec" is fine for write load — but the real load problem was fan-out on read: one celebrity's tweet hitting 100M followers. The load parameter for the home timeline read isn't "reads/sec," it's "average followers per posting user."

Performance under load

Two questions:

1. When you increase a load parameter and keep system resources constant, how does performance degrade?
2. When you increase a load parameter, how much resource do you need to add to keep performance constant?

Latency vs throughput

• Latency = time for one request (a duration). Always quote percentiles, never averages — averages hide the long tail.
• Throughput = requests per unit time (a rate).

A system can have great average latency and terrible P99. The P99 customer is your loudest customer (they're a power user — many requests, so they hit tail latencies often).

Percentiles & tail latency

• P50 (median): half of requests are faster than this. Useful baseline.
• P95, P99, P99.9 ("three nines"): the slow ones. Amazon found 100ms extra latency dropped sales 1%; Google found 500ms extra dropped traffic 20%.
• Tail amplification: if one backend call has P99 = 1s and your request fans out to 100 backends, the probability all finish in 1s is ~37%. P99 of the parent = ~much worse than 1s.

Computing percentiles: don't average percentiles across machines (mathematically wrong). Use approximate algorithms like t-digest or HDR Histogram and merge.

Scaling axes (the AKF Scale Cube)

• X-axis: horizontal duplication. Run more identical instances behind a load balancer. Easy. Limited by stateful services and shared resources.
• Y-axis: functional decomposition. Split by service/domain (auth, payments, search). Microservices. Each service scales independently.
• Z-axis: data partitioning (sharding). Split data by a key (user_id mod N). Each shard handles a slice. Required for systems too big to fit on one node.

Most systems use all three.

Vertical vs horizontal

• Vertical (scale up): bigger machine. Simple, no code change. Caps at hardware limit. Single point of failure.
• Horizontal (scale out): more machines. Theoretically unlimited. Requires designing for distribution (statelessness, sharding, consensus).

1.3 Maintainability

The cost of software is dominated by ongoing maintenance, not initial development.

Three sub-properties:

1. Operability: easy for ops to keep it running. Good monitoring, runbooks, predictable behavior, automated routine tasks.
2. Simplicity: easy for new engineers to understand. Fight accidental complexity (Brooks: "no silver bullet" — there's essential complexity in the problem, but most complexity is accidental, our fault). Abstractions are the main weapon.
3. Evolvability (or extensibility): easy to change. Tied to TDD, refactoring, simple architecture, agile.

1.4 Availability

Reliability over time. Usually expressed in nines:

Availability	Annual downtime
99% (two nines)	3.65 days
99.9%	8.76 hours
99.95%	4.38 hours
99.99% (four nines)	52.6 minutes
99.999% (five nines)	5.26 minutes

Five nines is extremely expensive. Most B2B SaaS targets four nines. Internal tools, two nines.

SLI / SLO / SLA

• SLI (Service Level Indicator): a measurement. "Fraction of HTTP requests with status 200 in the last 5min."
• SLO (Service Level Objective): a target. "SLI ≥ 99.9% over 30 days."
• SLA (Service Level Agreement): a contract with consequences. "If SLO is missed, customer gets 10% credit."

Google's SRE rule: SLOs are looser than your real performance. The gap is your error budget — if you're under it, you can ship features faster (more risk). If you're over, you must slow down and stabilize.

Availability math

For independent components in series (request must traverse all): multiply.

• 3 services each at 99.9% → 99.7% combined.

For redundant components in parallel: 1 - product of failure probs.

• 2 replicas each at 99% → 1 - (0.01 × 0.01) = 99.99%.

In practice failures correlate (shared power, shared DNS, shared region) so independence is a fiction. Plan for correlated failures.

1.5 The hardest trade-off: reliability vs cost vs speed of delivery

There is no "100% reliable" system. There is "reliable enough that the marginal cost of more 9s exceeds the marginal value." For most products, 99.9% is the right target.

For Google L3/L4: when designing, state your availability target up front, then justify everything (replication factor, multi-region, ack policy) against that target. Don't over-engineer. Don't under-engineer.

Key takeaways

• Reliability = tolerate faults. Hardware fails, code has bugs, humans err. Build for all three.
• Scalability = define your load parameter, then measure performance at percentiles (not averages).
• Maintainability = operability + simplicity + evolvability. Most code-cost is post-launch.
• Availability = nines. State the target before you design.
• Trade-offs are explicit: reliability costs money; speed costs reliability; simplicity costs flexibility.