uBix Vault 1.6 — Why I Rolled My Own Secrets Manager (and Why You Might Try It)

The problem

My ecosystem runs on secrets. The uBix projects, uBix Core among them, uBixShepherd and its LinkedIn token, Replikate, the sites, the Kubernetes cluster and the CI pipelines that deploy them: they all need API keys, database credentials and certificates, and none of that belongs in a repo or a ConfigMap. I needed a real secrets manager, one built to carry production traffic and light enough to run without it becoming a second job.

Why not HashiCorp Vault

Vault is the reference implementation, and I have nothing bad to say about what it does. It just wasn’t the right fit for what I needed:

The licence moved. HashiCorp relicensed Vault from MPL 2.0 to the source-available BUSL in 2023, and it’s IBM-owned now. For the thing every other workload gets its credentials from, I wanted terms that nobody else can change.
It’s a lot to operate. It’s a full platform with the breadth and operational weight to match. What I needed was a focused core: KV, Transit, dynamic database credentials, policies, leases, audit.
I wanted to own it. Barrier, seal and unseal, policy engine, lease lifecycle, all of it. I started from Vault’s shape and built the security-critical parts myself, then kept the scope tight on purpose: depth over breadth.

OpenBao is the open fork, it’s a good project, and if you want Vault’s whole system kept open it’s the right answer. I wanted something lighter than that.

The bet: speak Vault’s API

The thing that makes rolling my own a sane decision rather than a vanity project is that uBix Vault speaks Vault’s API. The vault CLI works against it, and so do External Secrets Operator and anything built on HashiCorp’s Go client. The tooling you already have keeps working.

And it’s built to be run in production, not just demoed:

High availability since 1.2. Replicas over MySQL with one active, fenced writes so a stalled replica can’t overwrite its successor, and handovers in about half a second. In the 1.2 failover test, through a rolling restart, a node drain, a step-down and a force-delete, a client with no retries saw zero failed requests out of 1,512.
Operations that exist from day one. Auto-unseal, snapshots, a scheduled backup CronJob and rolling upgrades.
Lighter than Vault. One Go binary, no plugin processes, a handful of dependencies, and a codebase you can read end to end.

1.6 removes the last big external dependency.

What 1.6 adds: it keeps its own data

Start it with -storage raft (or storage.type=raft in the Helm chart) and three replicas form a cluster that replicates every write to a majority before acknowledging it. No MySQL to provision, patch or lose.

Each replica has its own volume; any one can be lost. Lose two and there’s no quorum, so writes fail rather than diverge.
The same operations. Auto-unseal, snapshots, the backup CronJob and rolling upgrades work as before. operator raft handles peers, join tokens, snapshots and recovery, and vault operator raft list-peers works too.
Explicit joining. A new replica joins with a single-use, short-lived token against a pinned CA, and peer traffic is mutually authenticated TLS.
A way over from MySQL. One maintenance window: export the ciphertext, restore it into the first replica’s volume, start with the same unseal key, join the others. No token, lease or secret is invalidated, and rollback is a snapshot back into the database.

It ships EXPERIMENTAL. The label stays until it has soaked on my own dev and then prod.

The evidence: I attacked it until it broke

A consensus implementation that passes its unit tests has told you very little, so before release I went after it. A chaos suite on a real Kubernetes cluster (kind, in CI) with a client writing every 100 ms through the Service with no retries, every acknowledged write checked afterwards. Then a Jepsen-style checker: concurrent clients, crashes and partitions, and a per-key linearizability check on the history.

It found three real bugs:

An address bug. A bootstrapped node’s address was stored as a pod IP rather than its DNS name. A pod’s IP can change when it’s rescheduled; its DNS name doesn’t.
A stale read. A newly elected leader could answer a consistent read from a state machine that hadn’t applied the previous leader’s last entries. The leader now confirms it’s caught up before it answers.
Reads that could block forever. The same Jepsen run exposed reads that never returned after a crash or partition. Waits now honour the request deadline.

All three are fixed in 1.6. Here’s what I measured afterwards, three replicas on kind. Where a result is a range, it’s across the three chaos runs.

Event	Result
Leader’s node frozen	new leader in 6–7 s
Leader partitioned from the majority	new leader in 11–17 s; the isolated leader refused with 503
Acknowledged writes, all three runs	every one read back: 93/93, 85/85, 71/71
Whole cluster restarted (volumes kept, auto-unseal)	serving again in 18 s, nothing lost
One replica’s volume wiped (`-2`)	back as a voter in 19 s
Pod `-0` on a wiped volume	refused to bootstrap a new cluster while the existing one answered; rejoined once given a join token, data served, configuration intact
MySQL → Raft migration	48/48 entries intact, 33–35 s outage
Rollback to MySQL	serving in 41–46 s

I didn’t time the wiped -0 rejoin, so there’s no number for it. What I’m claiming is that it didn’t found a second cluster under its old identity, and that it came back.

The Jepsen-style run: three 2-minute runs under the race detector, no violation found, replicas converged. The checker also gave up on part of the history (it hit its step budget) on all three keys. So the reading is “no violation found”, not “proven linearizable”.

Against MariaDB, through the same Service: Raft reads are faster (p99 13.5 ms against 17.9 ms), and writes are slower (8 clients, p99 31–40 ms against 26.6 ms). These are CI-VM numbers, a regression baseline and not capacity planning.

Try it

It suits you if you run Kubernetes, you want a secrets manager that’s lighter to run than Vault and that Vault’s tooling already works with, and you want to own the terms it’s under. It’s BSD 3-Clause, and the repo is github.com/cwolsen7905/ubixvault. The Helm chart is in deploy/charts/ubixvault.

My advice on where to start:

Run it on the MySQL HA path. That’s the stable production path: ha.enabled=true with replicaCount=3.
Try Raft on a non-critical cluster first. Set storage.type=raft and replicaCount=3 for a new cluster; “Migrating from MySQL to Raft” in the deployment docs covers moving an existing one. Keep the database around for the soak.

Here’s where it stops:

Raft storage is EXPERIMENTAL in 1.6. It hasn’t had a production soak.
There’s no independent security audit. That’s still the open milestone, so weigh it against what your secrets are worth.
Raft wants 3 or 5 replicas on block storage with real fsync, not NFS or SMB. Auto-unseal is the practical mode, since a sealed replica doesn’t vote.
It isn’t Vault’s integrated storage byte for byte: autopilot covers membership and leader only, and a few response shapes differ.

Try Raft where a mistake is cheap, then tell me where it breaks. That’s how the last three bugs got found.