Scaling a Web App from 0 to 1 Million Users: A System Design Guide

Scaling from 0 to 1 million users is four separate problems — caching, asynchronous processing, database scaling, and CDN / static delivery — solved in that order, at predictable thresholds. A single well-built server comfortably serves the first ~10,000 users; what breaks next is almost always the database, and almost never the language runtime.

The Four Pillars of Scale

Every production scaling technique in this guide is one of four moves. Recognizing which pillar a bottleneck belongs to keeps you from reaching for the wrong tool:

1. Caching: stop computing or fetching the same thing twice. Static assets at the CDN, hot pages at the edge, expensive queries in Redis. Target an 80–95% hit rate per layer; below ~70%, the cache is cost without benefit.
2. Asynchronous processing: move everything off the request path that the user does not need to wait for — emails, image processing, report generation, webhook delivery. Queues turn a 3-second request into a 300-millisecond one.
3. Database scaling: index, then cache reads, then replicas, then partition. In that order — every step adds operational complexity and only the first is free.
4. CDN and static delivery: push bytes as close to the user as possible. For most content-heavy apps this offloads 60–90% of requests before they ever reach your code.

Stage 0–1k Users: One Server, Done Right

The correct architecture at 0–1,000 users is boring: one VM or container running the app, a managed Postgres (not the app box), object storage for uploads, and TLS at a proxy or platform edge. Managed database is non-negotiable — backups, failover, and patching are not a good use of a small team's time.

What breaks here is not load. It is N+1 queries, unindexed foreign keys, missing SELECT * on wide tables, and synchronous work on the request path. A page that issues 40 queries at 2ms each renders in 80ms of database time; at 100 rows per page it becomes 4,000ms — and no amount of horizontal scaling fixes a per-request flaw.

// The bottleneck is usually query shape, not server count.

// Before: 1 + N statements — latency grows with page size
const orders = await db.orders.findMany({ where: { tenantId }, take: 100 });
for (const order of orders) {
  order.customer = await db.customers.findUnique({
    where: { id: order.customerId },
  }); // +100 round trips
}

// After: one statement
const orders = await db.orders.findMany({
  where: { tenantId },
  include: { customer: true }, // planner emits a single join
  take: 100,
});

// Adding servers before this fix scales the flaw, not the app.

Also at this stage: response compression, an HTTP cache policy for static assets, a connection pool sized to your app instances (~5–10 connections each — Postgres does not want 500), and structured logs you can grep during the first real incident.

Stage 1k–10k Users: CDN, Replica, Queue

Three additions carry this entire tier:

• A CDN in front of static assets — hashed filenames with one-year cache lifetimes, HTML kept no-store. One configuration change, usually the single largest latency win available.
• A read replica for reporting, exports, and read-heavy pages tolerant of sub-second lag. Keep writes on the primary; direct only idempotent reads to the replica, and never route a read immediately followed by a write to different nodes.
• A background job queue (Redis-backed or a managed queue) with 2–10 workers. Anything the user does not need in the response — notifications, thumbnails, search indexing — leaves the request path permanently.

At this tier, a 10-person-team SaaS product typically sits comfortably on a 4-vCPU app instance and a 2-vCPU database, serving API p95 latencies under ~200ms with cache hits served in single-digit milliseconds.

Stage 10k–100k Users: Stateless Fleet and a Real Cache

Sessions and file uploads leave the app box. Every server becomes stateless — session state in Redis or a cookie, uploads in object storage — so a load balancer can distribute traffic across N identical app instances and the autoscaler can add them during peaks. This is the 12-factor sweet spot: scale by adding processes, deploy with zero-downtime rolling replacements.

The database usually becomes the bottleneck first: connections (each extra instance opens its own pool — cap it, and put PgBouncer or a transaction-pooling proxy in front), CPU on hot queries, and lock contention on hot rows. Fixes in order: fix the query, add the index, add a Redis cache layer in front of expensive reads (cache-aside with explicit invalidation on write), then add replicas for read distribution. An asset pipeline joins here too: images resized and converted at upload, variants served from the CDN — never resized on request.

This is also where rate limiting, per-tenant quotas, and proper metrics land. You cannot run a fleet blind: p95 latency, error rate, DB CPU, and cache hit ratio are the four numbers on the dashboard.

Stage 100k–1M Users: Partition, Decouple, Go Regional

Past this line, a single primary write path is the ceiling. The toolbox, again in order of when to reach for it:

• Partitioning before sharding: declarative range partitioning on time (event tables) or Citus-style distribution by tenant_id keeps one operational surface while distributing rows. Real DB-per-shard sharding is the last step, because rebalancing and cross-shard queries are permanent taxes.
• Read/write splitting: replicas absorb analytics, feeds, and search refreshes; writes stay pinned to the primary. Budget for replication lag in your UX — not every screen can be read-your-writes.
• A dedicated search cluster (OpenSearch/Elasticsearch, or Meilisearch/ Typesense for smaller fleets) once full-text queries exceed what ILIKE '%term%' scans can do under ~100ms. Index asynchronously from the event bus; the database stays the source of truth.
• Event-driven decoupling: a bus (Kafka, NATS, or a managed queue) fans out to indexers, analytics, notifications, and integrations so the write path only commits its own row.
• Multi-region reads: CDN at the edge everywhere; app + read replicas in a second region only when the latency or residency case is proven. True multi-region writes are a project, not a config flag — defer until regulation or revenue demands it.

Stage Table: Users, Bottlenecks, Solutions

The progression itself, condensed — each stage inherits the previous one rather than replacing it:

Stage A  0 - 1k users        one machine, nothing else

  Browser --> [ app + Postgres + files : single server ]

Stage B  1k - 10k users     CDN + background jobs + first replica

  Browser --> [ CDN: static ] -----------------> [ S3/object storage ]
           -> [ app ] --> [ primary DB ] <---- [ worker: queue jobs ]
                              ^
                              | replication (async, <1s lag)
                              +------ [ read replica ]

Stage C  10k - 100k users   stateless fleet, cache layer, LB

  Browser --> [ CDN ] --> [ load balancer ]
                              |
                 +------------+------------+
                 v            v            v
             [ app x N ]  [ app x N ]  [ app x N ]   (stateless, disposable)
                 |            |            |
                 +-----+------+------+-----+
                       v             v
                  [ Redis cache ]  [ primary ] --> [ replicas ]
                  [ object storage ]

Stage D  100k - 1M users    sharded writes, search, events, regions

  Browser --> [ CDN / edge ] --> [ LB ] --> [ app x N ]
                                            |       |
                        +-------------------+       +------------------+
                        v                          v                  v
              [ partitioned primary ]        [ search cluster ]   [ event bus ]
              (Citus / declarative       (OpenSearch/Elastic)  --> [ indexers ]
               shards by tenant)                                 --> [ analytics ]
              [ regional read replicas: us / eu / apac ]

When NOT to Scale Horizontally

Horizontal scaling is a multiplier, and it multiplies whatever is already true. Before adding a single instance, ask whether the bottleneck is per-request work you can eliminate: an N+1 query, a missing composite index, an unbounded SELECT * on a table with 200MB row blobs, JSON serialization of fields nobody renders, or an external API called synchronously inside the request loop.

The arithmetic is unforgiving. Halving per-request work is roughly equivalent to doubling your fleet — permanent, cheaper, and it helps every future instance too. Six signals say “fix the code, not the topology”: most queries under 5ms but request p95 is high; CPU idle while latency is not; database time dominates the trace; one endpoint absorbs most traffic; a cache you added returns <70% hits; and you cannot name which resource is actually saturated. If you cannot answer “what is the single constraint?” from a dashboard, you are not ready to scale anything.

What to Measure at Each Stage

Metrics are how you know a stage ended. The set grows, but the discipline does not:

• Stage A: request p95 and error rate (target: API p95 under ~200ms, 5xx under 0.1%), database slow-query log, and field core web vitals — LCP under 2.5s, CLS under 0.1.
• Stage B: cache hit ratio per layer, queue depth and oldest-job age, replica lag (keep it under a second for reads that matter).
• Stage C: connection-pool saturation, per-instance CPU and memory, autoscaling headroom, DB CPU and lock waits, p99 (the tail is where incidents live — see the Tail at Scale below).
• Stage D: per-shard hot spots, consumer lag on the event bus, search index freshness, regional latency per edge, and cost per thousand requests — at this scale, efficiency is a feature.

Common Anti-Patterns

• Microservices too early: splitting into services before you have a load problem gives you distributed problems without solving the original one. One deployable with clean internal modules scales to this entire guide.
• Premature sharding: partitioning by time or tenant gets most of the benefit; hard sharding adds rebalancing, cross-shard queries, and distributed transactions. Do it when a single primary is measured at its limit, not when a diagram looks impressive.
• Cache as a band-aid: a cache in front of a broken query hides the bug until the cold-start stampede exposes it. Caches are for genuinely expensive, genuinely reusable results — and they need TTLs, stampede protection (single-flight locking), and invalidation on write.
• Scaling state first: shared local disk, sticky sessions, and in-process caches all block the single cheapest scale move: adding and removing stateless instances freely.
• Multi-region from day one: active-active writes across regions buy consistency headaches long before most products have users in the second region. CDN plus regional reads covers ~95% of the latency case.

Verdict: scale in the order cache → async → database → CDN, one stage ahead of your metrics, never two. The teams that reach 1 million users smoothly are not the ones with the cleverest architecture — they are the ones who refused to build Stage D while they were still in Stage A.