--- title: "Backend systems, technically — Thoughtful Robots" description: "How Thoughtful Robots engineers backend systems: framework and system shape, API contracts, data architecture, TDD, job queues, security, runtime topology, delivery, observability, and recovery." source: "https://thoughtfulrobots.ai/backend-systems-tech" --- # The request ends. The responsibility does not Backend engineering is the work of preserving meaning across boundaries: a transaction that commits once, a tenant that cannot see another tenant, a job that can be retried safely, an API that can evolve, and a failure that leaves enough evidence to recover. We choose frameworks, databases, queues and runtime topology around those guarantees — not the other way around. ## What the system has to keep true. These are backend responsibilities before they are technology choices. The tools show where our work has landed; each linked section explains the engineering contract behind them. | Layer | What we use | | --- | --- | | [Framework & system shape](#framework-system-shape) | Rails Payload SvelteKit server Modular monoliths Services | | [API & contracts](#api-contracts) | REST OpenAPI GraphQL RPC Webhooks | | [Data architecture](#data-architecture) | PostgreSQL D1 / SQLite pgvector Redis S3 / R2 | | [TDD & verification](#tdd-verification) | Minitest RSpec Vitest Playwright WebMock | | [Jobs, events & integrations](#jobs-integrations) | Solid Queue Sidekiq BullMQ Cloudflare Workflows | | [Security & tenancy](#security-tenancy) | OAuth 2 JWT Policy authorization Rate limits Encryption | | [Runtime topology](#runtime-topology) | Cloudflare Workers ECS / Fargate Kamal Serverless Containers | | [Delivery & infrastructure](#delivery-infrastructure) | Docker AWS CDK Migrations CI/CD Health checks | | [Observability & recovery](#observability-recovery) | Structured logs New Relic Metrics Job dashboards Backups | ## Choose the system shape *before the framework.* The first backend decision is not Rails versus Node. It is where consistency has to be immediate, which work can happen later, what the team must operate, and how many independent failure domains the product can afford. We usually begin with a modular monolith because one deployable unit keeps transactions, refactoring and operational ownership legible. We separate services only when a boundary earns independent scaling, security, release cadence or runtime requirements. - Domain depth, transaction boundaries and the rate the model will change Domain model Transactions - Request, job, streaming and scheduled execution profiles HTTP Workers Schedulers - Existing team fluency and the system already in production Ownership Change cost - Deployment target, cold-start budget and connection model Long-running Serverless Edge - The smallest architecture that can meet isolation and scale requirements Modular monolith Services Rails gives long-lived transactional products a coherent domain model, mature migrations, policy authorization and background work in one system. Payload gives TypeScript products a typed schema, generated admin surface and local server API beside REST and GraphQL. SvelteKit on Cloudflare Workers is useful when a small backend can live close to its users and depend on platform bindings rather than long-running processes. These are implementation choices, not agency boundaries. We can inherit another serious backend framework and apply the same decision model without first replacing it. A service boundary has to pay for its network hop, deployment surface and new failure mode. “Microservice” is not a synonym for well-structured code. ## Make the boundary *hard to misunderstand.* An API is a compatibility promise. We specify its inputs, outputs, authentication, errors and evolution rules where they can be tested, then generate documentation and clients from the same contract. REST is the default for durable public boundaries; GraphQL, RPC and framework-local APIs are selected when their coupling and query model are explicit rather than accidental. - Versioned HTTP resources with stable error envelopes and status semantics REST JSON - Executable request specifications that generate the published schema OpenAPI RSWAG - Typed clients generated from the contract instead of handwritten twice OpenAPI generators - Graph-shaped queries where clients genuinely control data selection GraphQL - In-process and RPC calls reserved for deliberately coupled services Local API RPC - Webhooks signed, replay-safe and explicit about delivery acknowledgement HMAC Event IDs - Additive changes stay backward compatible - Breaking changes receive a migration path and measurable adoption window - Authorization is tested at the endpoint, not inferred from the UI - Generated clients and examples are rebuilt in CI The schema is useful only when production behaviour and the published contract can fail the same test. ## Choose storage from *the consistency model.* PostgreSQL is our default because most product data has relationships, constraints and changes that need to commit together. We add document, search, vector, cache and object stores for access patterns they serve better—not to avoid modelling the source of truth. Each additional store needs an owner, a derivation path and a repair strategy. - Relational source of truth with foreign keys and explicit transaction boundaries PostgreSQL Active Record Drizzle - Edge-local relational storage for bounded serverless applications Cloudflare D1 SQLite - Search and semantic retrieval derived from canonical records Elasticsearch pgvector - Ephemeral coordination, counters and hot reads kept out of the primary path Redis Solid Cache - Large encrypted objects stored outside database rows and delivered by signed access S3 R2 Signed URLs - Expand and contract schemas across compatible releases - Backfills are resumable, observable and safe to run more than once - Indexes are introduced with realistic query plans and lock behaviour - Derived stores can be rebuilt from the source of truth - Sensitive data has an explicit encryption and retention lifecycle “Non-relational” is not one database category. A document, cache, vector index and object store solve different problems and fail in different ways. ## Drive the design *through executable behaviour.* Test-driven development is most valuable at the backend boundaries where a small ambiguity becomes durable data. We write the next behaviour as an example, implement the smallest coherent change, then refactor with the contract held in place. The suite is layered so domain feedback stays fast while requests, databases, workers and deployments receive the integration evidence they need. - Domain rules, state transitions and service objects exercised without HTTP Minitest RSpec Vitest - Request tests covering validation, response shape, authorization and tenancy Request specs OpenAPI - Integration tests against real migrations, database constraints and queue adapters PostgreSQL D1 Redis - External failures made deterministic at the network boundary WebMock Test doubles - Race conditions tested where uniqueness and last-writer behaviour matter Concurrency tests - Deployed critical paths checked as consumers see them Playwright API Smoke tests - A contract regression or undocumented response change - A migration that cannot roll forward safely - A broken tenant or authorization boundary - A worker that loses, duplicates or permanently hides failed work - A critical production smoke test failure Coverage is evidence of execution, not evidence of the right assertions. We optimise for meaningful boundaries and failure cases. ## Design what happens *after the response returns.* A job queue is not a reliability strategy by itself. Workers need idempotency, bounded retries, visible terminal failure and enough context to reconcile with the system of record. The same rules apply to schedules, notifications, webhooks, ingestion pipelines and third-party APIs: acknowledge only what is durable, assume delivery can repeat, and preserve a path to repair. - Database-backed work close to a transactional Rails application Solid Queue - Redis-backed workers with priority, scheduling and operational dashboards Sidekiq BullMQ - Durable multi-step serverless work with persisted progress Cloudflare Workflows - Scheduled sweeps that record their decision window before dispatch Cron Schedulers - Fan-out through domain events rather than controller side effects Events Notifiers - Every retried operation has an idempotency key or equivalent guard - Retries use bounded backoff and classify permanent failures - Poison work moves to an inspectable failed state - Webhook signatures and event identifiers are verified before mutation - Reconciliation can compare local truth with the external provider At-least-once delivery is common. Exactly-once business effect is something the handler earns through its data model. ## Make access control *part of every query.* Authentication proves an identity; authorization proves that identity may perform this action on this record in this tenant. We keep those decisions on the server boundary, scope data before it is returned, rate-limit expensive or abusable paths, and record access to sensitive material. Security is implemented as testable application behaviour and reinforced by the runtime—not postponed to an infrastructure checklist. - Session, OAuth, JWT and API-key authentication selected by client and trust model OAuth 2 JWT Better Auth - Policy checks over action, role, record and tenant Pundit Payload access - Tenant isolation applied centrally and tested against cross-tenant identifiers Scoped queries ActsAsTenant - Abuse constrained before expensive work or credential verification Rack::Attack Redis rate limits - Sensitive fields and files encrypted with rotatable key boundaries Envelope encryption KMS - Administrative reads and mutations retained as attributable events Audit logs Amendments - Authorization is denied by default - Secrets never enter public configuration or structured request logs - Webhook and upload capabilities expire and are scoped - Static analysis and dependency checks run in CI - Deletion and retention behaviour is explicit for regulated data A tenant ID in a request is user input. Isolation begins with the authenticated context, not with trusting that parameter. ## Place each workload *where its constraints fit.* Serverless and containers are runtime choices, not competing ideologies. Edge workers are excellent for bounded request work with platform storage and durable orchestration. Long-running containers are the better fit for connection pools, background workers, browser automation, media processing and workloads that need specialised binaries. We can combine both when the trust or execution boundary demands it. - Requests are stateless, bounded and benefit from global placement Cloudflare Workers - Storage and coordination are available as explicit platform bindings D1 R2 Queues - Durable work can be expressed as steps rather than a resident process Workflows - The application owns a long-lived server or database connection pool Rails Payload Puma Node - Workers need stable concurrency, memory or filesystem behaviour Sidekiq BullMQ - Untrusted files or browser processes require isolated execution Container pools Sandboxing - Migrations must complete before traffic reaches the new schema Migrator task Containerization packages a workload. It does not decide whether that workload should be long-running, independently scaled or publicly reachable. ## Ship the application *and its dependencies together.* A backend release changes code, schema, workers, configuration and sometimes network topology. We describe infrastructure in code, build immutable artifacts, sequence migrations explicitly, and make health checks prove more than process existence. Deployment is complete only when the new version can serve traffic, its workers understand the schema, and rollback has a defined data story. - Repeatable images with separate server, worker and migrator entry points Docker Multi-stage builds - Reviewable cloud topology and permissions expressed as source AWS CDK CloudFormation - Edge bindings and environments rendered from validated configuration Wrangler Cloudflare - Migrations applied as an explicit release phase before dependent traffic Rails migrations Payload migrator Drizzle - Health checks covering database and critical dependency readiness Readiness Liveness - Small releases with traceable rollback and environment parity CI/CD Kamal ECS - Configuration contract is complete without exposing secrets - Database changes remain compatible with the previous application version - Workers and schedules are deployed in the intended order - Health checks and smoke tests pass against the deployed environment - Rollback ownership and data consequences are known A green image build is not a green release. The system is the artifact plus the schema, bindings, workers and traffic policy around it. ## Leave enough evidence *to recover.* Observability starts with the questions an operator must answer: which tenant or request failed, what changed, whether the failure is spreading, and what can be retried safely. We connect structured events, service metrics, traces, job state and deployment identity so an alert points toward a decision. Recovery then turns that evidence into tested procedures for replay, rollback and restoration. - Structured logs carrying request, tenant, job and release correlation JSON logs Correlation IDs - Latency, throughput, saturation and error rates by critical path Metrics Service levels - Traces across API, database, queue and external calls OpenTelemetry New Relic - Queue depth, age, retries and terminal failures visible to operators Mission Control Bull Board - Audit and domain events retained separately from diagnostic logs Audit trail Domain ledger - Alerts map to a user or business impact and a named owner - Failed work can be inspected and replayed without duplicating effects - Backups and point-in-time recovery are exercised, not merely enabled - Derived indexes and caches have rebuild procedures - Runbooks record rollback, reconciliation and escalation decisions The useful question is not “do we have logs?” It is “can the person on call decide what is safe to do next?” ## We operate the systems after handover. These practices come from running multi-tenant commerce platforms, community products, AI workspaces, event discovery systems and encrypted document vaults — across Rails, Payload, PostgreSQL, Redis, AWS containers and Cloudflare's serverless runtime. Owning those systems after launch is why the page is opinionated about executable API contracts, migrations, tenant isolation, idempotent workers, dedicated migrators, encryption boundaries and recovery evidence. They are the details that decide whether a failure becomes a brief incident or corrupted state. ## An AI-native team, with a factory behind it. Team Foundry is our software factory — what the team uses day in, day out to deliver projects. It accelerates the work and validates it: every change arrives with the checks it passed, the session that produced it, and a person accountable for it.