Asistencia 24/7/365 · Oficina 9–17 h MT, lun–vie +1 (208) 391-7176 team@alcousa.org

Migrating a production platform without downtime

A complete account of how we move a live system onto new infrastructure: what we check before we start, the order operations have to happen in, the failure modes that cost people their weekend, and the one step we will not automate.

Every migration that goes badly goes badly in the same way. The copy finished, the new environment looked right, someone changed DNS, and then a problem appeared that nobody had tested for — a missing PHP extension, a database user with the wrong grants, a cron job that never ran, a certificate that had not been issued. By the time it surfaced, the old host had stopped being authoritative and the fastest route back was measured in hours.

The migration itself was not the hard part. The hard part was that the plan had exactly one attempt in it.

This is how we do it instead, in the order it has to happen, with the reasoning attached. It is longer than a checklist because the reasoning is the part that transfers; a checklist you do not understand will be abandoned the first time it is inconvenient.

The principle everything else follows from

Sort every step into two piles: the ones you can retry for free, and the ones you cannot. Then automate the first pile relentlessly, and put a human gate on the boundary between them.

Copying files costs bandwidth. Restoring a database into a new server costs disk. Standing the application up under a temporary hostname costs nothing anybody will notice, because the real domain is still pointed at the old host and real users are still being served by it. You can run that loop five times in an afternoon and the client never knows there was a first attempt.

Changing the DNS record is where that stops being true. Resolvers begin caching the new answer within seconds, and you do not control their cache. If the destination is wrong you are not rolling back a deployment, you are waiting for other people's infrastructure to forget. Lowering the TTL beforehand converts a day-long problem into a five-minute one, which is worth doing and is not the same as making it reversible.

So: everything before the DNS change is automated and repeatable. The DNS change is confirmed by a person who is awake.

Stage one: assess before you touch anything

The purpose of the assessment is to find the things that will not survive the move, while you still have the option of not moving yet.

  • Runtime versions, exactly. Not "PHP 8" — the point release, and every loaded extension. A platform that quietly depends on an extension the new host does not have will fail on a code path nobody exercises during testing.
  • Every scheduled task. Cron entries, systemd timers, application-level schedulers, and anything a plugin registers internally. These fail silently on the new host and nobody notices until an invoice run does not happen.
  • Outbound mail. How does the application send, and from where? A platform that sends through the old host's local MTA will simply stop, and the failure looks like nothing at all.
  • Everything on disk that is not in version control. Uploads, generated files, caches that are actually stores, and the config file with credentials in it that is not in the repository because somebody was careful once.
  • Database size, engine, character set and collation. A collation mismatch is the classic way to corrupt names with accents in them across a migration and not find out for weeks.
  • What talks to it. Webhooks, API consumers, DNS records pointing at the same host for other purposes, and anything with the IP address hard-coded rather than the hostname.
  • Certificates and their issuance method. If the current certificate is issued by a process tied to the old host, that process needs an equivalent on the new one before cutover, not after.

Write this down. The assessment is the document you will argue with at 2am, and memory is not a substitute.

Stage two: copy, and copy again

The first copy is a rehearsal, not the migration. Its job is to surface the problems in stage one that you missed.

Copy files and database to the new host and bring the application up under a temporary hostname. Expect it to be broken. Fix what breaks — usually a hardcoded absolute URL, a file permission, a missing extension — and then throw the copy away and do it again from scratch with the fixes applied to the process rather than to the copy.

That second run is the one that matters, because it proves the procedure works rather than proving you can fix things by hand. A migration you can only perform once, by hand, at 2am, is not a procedure. It is a stunt.

Stage three: verify like you mean it

Verification is not loading the home page. A system can render its front page perfectly and be broken in every way that matters. Before we call a staged copy ready:

  • Every template renders. An index, a single item, a search result, a paginated archive, a 404 page. Different code paths, different failure modes.
  • The admin loads and a write succeeds. Read-only success hides a permissions problem completely.
  • A scheduled task runs to completion, observed, not assumed.
  • Mail leaves the server and arrives somewhere real.
  • The error log is empty under load, not merely at rest. Warnings that appear only under concurrency are the ones that ruin launch day.
  • The certificate is issued and the chain is complete. A certificate that is present but missing an intermediate works in your browser and fails in half the world's.
  • URLs have been rewritten correctly, and you have checked inside serialised data as well as in plain columns. This is where most WordPress migrations quietly corrupt themselves.

Then, and only then, do we produce DNS records.

Stage four: DNS, deliberately

We do not publish the target IP address before verification is complete. That sounds like an odd thing to withhold, and the reason is behavioral rather than technical: a client who has the address in an email will eventually paste it into their registrar at a moment of their own choosing, which is precisely the unattended cutover the whole procedure exists to avoid.

When the records go out they include the TTL, and we ask for the TTL to be dropped to 300 seconds a day before the change. That day of low TTL is the difference between a mistake costing five minutes and costing until tomorrow.

The cutover itself is one person, awake, typing the domain name to confirm. Our tooling requires that confirmation; you cannot click past it.

Stage five: the hour after

The migration is not finished when DNS propagates. For the first hour:

  • Watch the new host's error log directly, not a dashboard.
  • Watch the old host's access log. Traffic still arriving there tells you exactly who has a stale cache, and whether anything has the old IP hardcoded.
  • Confirm mail is still sending from the new host.
  • Confirm the first scheduled task fires.
  • Leave the old host running. Turning it off the same day removes your fallback for no benefit; it costs almost nothing to keep for a fortnight.

What this costs, and why we still do it

Staged properly, a migration takes longer in elapsed time and barely longer in engineering time, because the repeatable parts are automated. What it buys is that the risky step happens once, with somebody watching, having already proven the destination works.

Clients occasionally ask us to skip it. The argument is always reasonable — everyone is busy, the change looks small, the previous provider did it this way without incident. Our answer is that the previous provider got away with it, which is a different thing from it being safe, and the distinction only becomes visible on the day it does not work.

We have built the tooling so the careful path is barely slower than the reckless one. That is the actual goal: not to be cautious, but to make the safe procedure cheap enough that there is no reason to skip it.

Más de ALCO

Least privilege without breaking everything

Removing local administrator rights is one of the highest-value security changes available and one of the most frequentl

Leer más

What an hour of downtime actually costs

The published averages are enormous and nearly useless, because they average across organizations nothing like yours. He

Leer más

Infrastructure as code for organizations that are not software companies

The practice is usually explained by software companies, to software companies, with examples from software companies. T

Leer más

¿Encaja con lo que necesita?

Cualificamos cada proyecto antes de presupuestarlo. Eso significa una conversación técnica sobre su infraestructura, no una llamada comercial, y una respuesta clara si no somos la firma adecuada.

Veamos qué está ejecutando.

Una conversación de alcance con un ingeniero sénior. Le diremos qué cambiaríamos, cuánto costaría y si somos la firma adecuada para ello.