uBix Vault 1.2 — High Availability, Because Nodes Get Drained

The problem 1.1 didn’t solve

Up to 1.1, uBix Vault ran as one pod. With the MySQL backend the data survived anything — the node was replaceable — but the service didn’t. Every time the people who run the cluster drained a node, patched a kernel, or upgraded Kubernetes, the vault went away for a reschedule and an unseal. For the thing every other workload gets its credentials from, “please schedule your maintenance around the secrets manager” isn’t an answer. 1.2 is the answer.

What 1.2 does

It’s HashiCorp Vault’s own high-availability model, over the database uBix Vault already uses:

Every replica unseals; one is active. The active replica holds a lock row in MySQL. The others are standbys — unsealed, ready, and forwarding: send a request to any replica and it gets answered, so a plain Kubernetes Service in front of all of them just works.
Handovers are planned for. When the active replica is told to stop — a drain, a rolling upgrade, sys/step-down — it gives the lock up on the way out and a standby takes over in about half a second. Standbys hold incoming requests across that gap instead of failing them.
Crashes are bounded. If the active replica dies without letting go, its lease lapses and a standby takes over — 15 seconds by default.
Replicas talk on their own port, with mutual TLS from a small CA the vault keeps inside its own encrypted barrier. There’s nothing to configure and no certificate to hand it; only a process that unsealed the same vault can connect.
In the Helm chart it’s one switch: ha.enabled=true, replicaCount=3. That also brings a PodDisruptionBudget, so a drain can never take two replicas at once.

The part that matters: fencing

The hard bug in any active/standby system is the replica that thinks it’s still active. It stalls — a long GC pause, a frozen VM, a network partition — its lease runs out, another replica takes over, and then it wakes up and carries on writing. Timeouts can’t prevent that; they can only make it rarer.

So uBix Vault doesn’t rely on them. Once a replica has anything to do with the lock, every write it makes is fenced: inside the same database transaction, it checks that it still holds the lock at the generation it acquired it with, and a replica taking over has to wait for those in-flight writes before it can bump the generation. After that, the old replica’s writes are simply refused. I checked that the tests actually catch this by switching the check off — and watching the stalled replica overwrite the new one’s data.

Tested where it will run

Unit tests don’t drain nodes, so 1.2 ships with a failover test that does: three replicas on a real Kubernetes cluster (kind, in CI), a client reading through the Service every 50 milliseconds with no retries, and then every way the active replica can go away. Deleting the pod, draining its node, a full rolling restart, a step-down, a force-delete: zero failed requests out of 1,512. Then the nasty one — freezing the whole node the active replica runs on, like a hung machine. A standby took over in 17 seconds, and when the node thawed, the old active — which still believed it was in charge — noticed it had lost the lock and quietly became a standby.

Testing against real processes earned its keep before any of this shipped. Running two actual server processes against MariaDB — before the Kubernetes test existed — found that a step-down followed by a drain could leave no replica active, that standbys were failing requests during every planned handover, and that a waiting request could be stranded on a replica that had just become the active one. All three are fixed in 1.2, and all three would otherwise have been found by the people running it.

Why not Raft

Vault’s other HA flavour is integrated storage: Raft consensus, no external database. It’s the right design when you want to delete the database. It’s also months of work, a second storage engine, quorum-loss recovery as a brand-new way to have an outage, and a pile of new dependencies — all to remove a database that’s already run as a highly-available service here. So 1.2 does the smaller, sharper thing, and the core only ever sees an abstract lock, which is exactly the seam Raft would plug into if it’s ever worth it.

And what hasn’t changed

1.2 is an availability release, not an assurance one. uBix Vault still has not had an independent third-party security audit, and HA adds surface worth an auditor’s time — the cluster port, the fencing, the forwarding — which is now written into the review brief. A version bump doesn’t move that bar; only an auditor does.

uBix Vault 1.2.0 is BSD 3-Clause, on GitHub at github.com/cwolsen7905/uBixVault, tagged v1.2.0, with signed multi-arch images. The design, with every trade-off, is in docs/design/ha-active-standby.md.