Services / Cloud / Reliability

It stays up, and when it doesn't, you already know why

Reliability is not luck and it is not heroics. It is backups that have actually been restored, alerts nobody has muted, systems that restart themselves, and a written procedure for every failure you can reasonably expect.

Start the conversation →See the technology

Why it matters

The cost is the wait

The cost of being down is rarely just the sales you did not make. It is the support queue afterwards, the customers who quietly try a competitor and never mention it, your team's evening, and the trust you spend explaining what happened. And the bill arrives at the least convenient possible moment, because outages have an unhelpful habit of coinciding with your busiest day.

The uncomfortable part is that most outages are self-inflicted. A certificate expired. A disk filled with logs. A release went out that nobody could roll back. Someone changed a firewall rule by hand on a Friday afternoon. These are not exotic failures. They are known, boring and entirely preventable ones, and between them they cause far more downtime than any attacker or data-centre fire ever has.

So the work is unglamorous by design. Define what "up" means from your customer's point of view. Measure that, and alert only on it. Make the system restart itself before a human is involved. Restore the backups regularly, so you know they work and how long they take. Rehearse the bad day while it is not happening. And when something does break, review it without blaming anyone, so the same failure does not get a second turn.

What you actually get

Built to be trusted

Reliability is a set of habits more than a set of products. These are the habits worth paying for.

Define what "up" actually means

Availability targets written from the customer's point of view (can they check out, can they log in, does the order reach the warehouse) rather than from a graph of server load. A server can be perfectly healthy while nobody is able to buy anything.

Alerts nobody mutes

Deliberately few alerts, tied to symptoms a customer would notice, each with a documented action attached. A channel that fires forty times a day gets ignored, and then it misses the one that mattered. Reducing alert volume is real reliability work, not tidying up.

It fixes itself before you hear about it

Health checks that pull a failing instance out of rotation, workloads that restart on failure, capacity that follows demand, and releases that roll out gradually while watching their own error rate, with a one-step rollback the moment they do not like what they see.

Backups that get restored

Backups on a schedule, held where a compromise of the main system cannot reach them, written so they cannot be deleted for a defined period, and restored routinely so you know exactly how long recovery takes. An untested backup is a hypothesis, and you find out which at the worst possible moment.

A rehearsed plan for the worst day

We agree how fast you need to be back and how much data you can afford to lose, design to those numbers, then actually run a failover rather than describing one. You also get the honest cost curve. Near-instant recovery is achievable and costs meaningfully more than a four-hour target.

Runbooks, and reviews that change something

Every expected failure has a written procedure, so the fix does not depend on one person being reachable. Every significant incident gets a blameless review producing specific actions with owners and dates. The measure of success is simple: the same failure does not happen twice.

Where it earns its keep

Same pattern, different desks

How much reliability you should buy depends entirely on what an hour of downtime costs you, which is the first thing we work out together.

01 · Online retail & ticketing

The busiest hour of the year is the one that breaks

The problem: Traffic is flat most of the time, then multiplies within minutes for a sale, a campaign or an on-sale. The platform copes with ordinary days perfectly well and has fallen over during the precise hour that pays for the quarter.
What we build: Load testing against a realistic profile finds whatever gives way first, usually the database or a third-party dependency, rarely the web servers everyone suspects. Automatic scaling is tuned with proper warm-up, queues absorb the spike wherever instant processing is not required, caching takes read load off the origin, and the checkout path is built to degrade gracefully instead of failing outright.
What changes: Peak events handled without anyone watching a dashboard with their heart in their mouth, and a rehearsed answer for the parts that still cannot be scaled.

02 · Logistics & field operations

There is no maintenance window

The problem: Drivers, warehouses and customers use the system around the clock, so there is no quiet hour to deploy in. Updates are therefore rare, large and frightening, which makes each one more likely to cause the outage everybody was trying to avoid.
What we build: Environments are rebuilt from code so staging genuinely matches production, releases move to gradual rollouts with automatic health checks and instant rollback, and database changes are made in backwards-compatible steps so the old and new versions can run side by side during a deployment.
What changes: Small changes ship during the working day without ceremony, and deployment stops being the most dangerous thing that happens in a given month.

03 · Booking & hospitality platforms

A database failure on a Saturday night

The problem: Everything runs on a single database instance with a nightly backup and no tested recovery procedure. If it fails at the busiest point of the week, the honest answer to "how long until we're back" is that nobody knows, and the last day of bookings may be gone.
What we build: The database moves to a managed service with automatic failover to a standby, point-in-time recovery to the minute, and backups replicated into a separate account. Then we rehearse: a real failover, timed, with the results written down and the runbook corrected wherever reality disagreed with the plan.
What changes: A recovery time everyone has actually seen rather than hoped for, and a Saturday night failure that costs minutes instead of a weekend.

The technology

The tools behind it, named

Monitoring and recovery tooling should be boring, well-understood and operable by your own team. We choose accordingly.

6 layers · 34 technologies

Seeing what is happening

Metrics, logs and traces in one place, so an investigation starts with evidence rather than with somebody's guess about which component is misbehaving.

Prometheus
Grafana
OpenTelemetry
Datadog
Elastic
Jaeger

Knowing when it matters

Few alerts, routed to whoever is genuinely on duty, plus a status page so your customers hear it from you before they hear it from each other.

PagerDuty
Opsgenie
Sentry
Slack
Statuspage
Uptime Kuma

Systems that recover themselves

Restart, reschedule, scale and roll back, without waiting for a human to wake up, read a message and remember the right command.

Kubernetes
Docker
Helm
Argo CD
NGINX
Traefik

Rebuildable from code

If the worst happens, recovery means running a pipeline, not reconstructing a server from memory, old emails and a hopeful search through a wiki.

Terraform
OpenTofu
Ansible
Packer
GitHub Actions
GitLab

Data that survives

Managed databases with failover and point-in-time recovery, and backups held somewhere an attacker holding your production credentials still cannot reach.

PostgreSQL
MySQL
MongoDB
Redis
Backblaze B2
Immutable object storage

Proving it under load

A capacity claim is worth nothing until something has tried to break it on purpose, in a rehearsal you scheduled rather than one your customers scheduled for you.

k6
Apache JMeter
Grafana
Game days & failover drills

Product names and logos are the property of their respective owners and are shown to describe the technologies we work with. Their use does not imply any partnership, sponsorship or endorsement.

How we deliver it

Live behind a human first

Reliability work is continuous, but a first engagement has a shape: understand the cost of downtime, close the obvious gaps, then rehearse.

What does an hour of downtime cost you?

Everything else follows from this number. It decides whether you need recovery in minutes or in hours, whether a standby environment is worth its monthly cost, and which systems deserve the attention. It is a business conversation before it is a technical one.

We find the single points of failure

A review of architecture, dependencies and suppliers, looking for components with no redundancy, manual steps only one person knows, expiring things nobody is watching, and third parties whose outage automatically becomes your outage.

Monitoring you would actually trust

Metrics, logs and traces consolidated, dashboards built around customer-visible behaviour, and alerting rewritten from scratch, usually far fewer alerts than you had before, each one meaning something specific and each one with a documented response.

Automate the recovery

Health checks, automatic restarts, scaling rules, gradual rollouts and one-step rollback. The aim is that the majority of failures resolve themselves and appear in a report the next morning, rather than as a phone call at 3am.

Prove the backups and the failover

We restore from backup and time it. We fail over to the standby and time that too. Whatever the plan claimed, the rehearsal is the truth, and the runbook is corrected to match what actually happened rather than what was supposed to.

Then it becomes a routine

A monthly report in plain language covering uptime, incidents, spend and what changed; blameless reviews after anything significant; and rehearsals on a schedule, so the plan stays real instead of ageing quietly in a folder.

Before you commit

The questions worth asking

Can you guarantee we'll never go down?

No, and nobody honest will. Every link in the chain has a failure rate, including the cloud providers themselves, and past a certain point each additional nine of availability costs disproportionately more than the one before it. What we will do is agree a target that is worth what it costs you, engineer to it, and put a written availability commitment with defined remedies behind what we operate. Then we make the failures that do happen short, visible and well-rehearsed.

What availability target should we aim for?

The one your business case justifies, not the highest one available. Going from a few hours of downtime a year to a few minutes typically means duplicated infrastructure across regions, more complicated data handling and a permanently higher running cost. For plenty of businesses a well-rehearsed four-hour recovery is the right answer and the money is better spent elsewhere. We will show you the cost curve and let you choose with the numbers in front of you.

We already have backups. Isn't that enough?

It is, right up until you need them. The two things we find most often are backups that have never once been restored, so nobody knows whether they work or how long they take and backups reachable with the same credentials as production, which means ransomware takes those too. Backups only become a recovery plan once they are tested, isolated and timed.

Do we need Kubernetes for this?

Usually not. It is excellent at real scale and genuinely heavy below it. You inherit a platform that itself needs upgrading, securing and understanding. Plenty of businesses get better reliability from a couple of well-configured managed services than from a cluster nobody on the team can debug at 3am. We will tell you honestly when it is not worth it, and we say that fairly often.

Who gets called at 3am?

Whoever you decide, and it is a genuine choice rather than a formality. Some clients want on-call in-house with our support during working hours; others want us holding the pager under a written response commitment. Either way the goal is the same, automation handles the routine failures, so the number of times anyone is woken stays low enough that the arrangement remains sustainable.

Will this need changes to our application?

Some, usually. Infrastructure can absorb a great deal, but an application that cannot survive being restarted, cannot run as more than one copy, or keeps important state on a local disk will cap how reliable the platform underneath it can ever be. We identify those constraints early, tell you which ones actually matter for your target, and scope the changes as work you can plan, rather than presenting a rewrite as a precondition.

How long could you be down for?

If the honest answer is "nobody really knows", that's the most useful place to start. Tell us what you run and we'll show you where the single points of failure are hiding.

Start the conversation →

Explore more

The rest of Cloud & Infrastructure

Migration Cost optimisation Security & compliance ← All of Cloud