Practical lessons from shipping APIs that handle millions of daily requests — covering contracts, middleware architecture, connection pooling, and observability.
Request throughput comparison across three connection strategies. A properly sized connection pool consistently delivers the lowest P99 latency under concurrent load.
The API Design Principles That Actually Scale
Building a REST API that serves ten users is straightforward. Building one that gracefully handles ten million requests per day while remaining maintainable by a growing team is an entirely different engineering challenge. The decisions you make in week one will echo through your architecture for years.
This guide distils lessons from building APIs that power production systems across fintech, healthcare, and e-commerce — industries where downtime is measured in real-world consequences.
Start With Your Data Contract
Before writing a single line of code, define your API contract. A contract specifies exactly what each endpoint accepts, what it returns, and what errors it can emit. Use OpenAPI 3.0 or JSON Schema to make your contract machine-readable and shareable with API consumers from day one.
- Input validation — reject malformed requests at the gateway, not deep inside business logic
- Response envelopes — standardise your success and error shapes across every endpoint
- Versioning strategy — decide on URI versioning (
/v1/) vs. header versioning before you ship - Pagination — cursor-based pagination scales to billions of rows; offset-based does not
Middleware Architecture for Scale
Node.js excels at I/O-bound workloads because its event loop processes thousands of concurrent connections on a single thread. But this strength becomes a liability if you block the event loop with CPU-intensive operations. Structure your middleware chain with this core principle in mind.
- Rate limiting and authentication at the edge — reverse proxy or dedicated API gateway layer
- Request parsing and schema validation in the first application middleware
- Business logic in dedicated service classes — never inline in route handlers
- Database access in repository classes with explicit connection pooling
- Response serialization and cache-header injection as the final middleware step
Make it work, make it right, make it fast — in that order. Premature optimisation is the root of all evil.
— Donald Knuth, paraphrased by every senior engineer who has rewritten a service
Connection Pooling and Query Optimisation
Every database query that executes inside a request-response cycle is a potential bottleneck. Use a connection pool (e.g. pg-pool for PostgreSQL) to avoid the overhead of establishing a new TCP connection per request. As a starting point, target a pool size of 2–4× your vCPU count and tune from there using load-test data.
Error Handling Done Right
Inconsistent error responses are the single biggest pain point for API consumers. Every error your API emits should follow the same structure: an error.code (machine-readable string), an error.message (human-readable sentence), and an optional error.details array for field-level validation failures.
Never expose internal stack traces or raw database error messages in API responses. Always map internal errors to sanitised, user-safe messages at the boundary layer before sending a response.
Observability From Day One
An API you cannot observe is an API you cannot operate. Instrument your Node.js service from the start with structured logging (e.g. pino), distributed tracing (e.g. OpenTelemetry), and RED metrics — Request rate, Error rate, and Duration percentiles. Your future on-call self will be grateful for every log line you write today.
