Replikate — Blue to Green, and the Switch I Forgot to Install

The Use Case

Blue/green is one of the most common ways to run production Kubernetes: two clusters, one live and one waiting, and a cutover when the new one is ready. The painful part is rarely the workloads. It’s everything around them: the ConfigMaps and Secrets (TLS certificates, registry credentials, shared config) that have to exist in green before anything scheduled there can start.

That’s exactly the job Replikate’s cross-cluster mode was built for. Pick a hub, register the clusters that should receive copies as spokes, and annotate the sources on the hub:

metadata:
  annotations:
    replikate.ubixsys.com/sync: "team=web"
    replikate.ubixsys.com/target-clusters: "green"

Every targeted source now has a managed copy in green, kept in step with blue. When a certificate renews on blue, the renewal shows up in green without anyone copying a Secret by hand at 2am.

What 1.3 Added

Two things in 1.3.0 make this fit real clusters better:

Per-target namespaces. A target-clusters entry can now be cluster:namespace, so blue’s default/wildcard-tls can land in green’s ingress namespace if that’s where green wants it. Change the override and the old copy is pruned.
A non-breaking domain move. The annotation prefix moved from replikate.brainchurts.com to replikate.ubixsys.com. Existing sources and copies under the old prefix keep working and self-migrate in place, with no coordinated re-annotation and no deleting and recreating live TLS secrets. A replikate_copies_by_domain gauge tells you when the migration is done.

The Switch I Forgot to Install

Then I sat down to write the Helm steps for turning this on, and there weren’t any.

The controller has had an --enable-cross-cluster flag since 1.1. Every test (unit, the two-control-plane envtest suite) wires the cross-cluster registry in directly in Go. But the Helm chart had no value that passed it, no way to say where the spoke credentials live, and no $POD_NAMESPACE for the default to fall back on. A Helm-installed Replikate could not become a hub at all. The feature worked in every test and was unreachable through the one install path most people use.

It’s a classic gap: the tests exercised the binary, and nothing exercised the packaging. Chart 1.3.1 adds the missing values:

helm upgrade --install replikate charts/replikate -n replikate-system \
  --set crossCluster.enabled=true

crossCluster.credentialNamespace defaults to the release namespace. The controller image is unchanged. 1.3.1 is chart-only and still runs ghcr.io/ubixsys/replikate:1.3.0, which always had the flag.

Which Cluster Is the Hub?

The obvious setup is to make blue the hub and green a spoke. It works, but it ties the source of truth to one colour: after a cutover, the idle cluster still owns everything the live one depends on.

The better setup, and one a reviewer pushed me toward, is a third cluster as the hub: an infrastructure cluster that already holds shared services and runs no application workloads. Blue and green are then both spokes. Neither colour depends on the other, and a blue/green swap never moves the source of truth. If you do go that route, keep in mind that the sync selector still applies on the hub itself, so give hub-only sources a selector that matches none of the hub’s namespaces.

Wiring It Up

On each spoke (green, and blue if you use a separate hub), apply deploy/spoke-rbac.yaml. It creates a least-privilege ServiceAccount (ConfigMaps, Secrets and Namespaces, nothing else) and a token. Build a kubeconfig from it.
On the hub, store that kubeconfig as a Secret named after the spoke (green), label it replikate.ubixsys.com/cluster-credential=true, and enable crossCluster.
Add target-clusters: "green" to the sources you want mirrored.

kubectl describe on a source shows RemoteReplicated or RemoteError events, and replikate_cluster_up{cluster="green"} tells you whether the hub can reach green.

Read This Before Cutover

Replikate treats the hub as the source of truth, and that has one consequence you need to plan for: deleting a source on the hub deletes its copies on every spoke. That’s the right behavior while everything is up. If blue is the hub, it’s the wrong behavior on the day you tear blue down. Deregister green first (delete its credential Secret on blue) and blue will leave green’s copies alone. With a separate hub this problem doesn’t arise: tearing down a colour is just removing a spoke.

The other honest edges: a hand-edited copy on green is repaired on the next source change or on a roughly two-minute recheck, not instantly the way local copies are. Update, same day: Replikate 1.4.0. The first reviewer to try this for real hit the limit I’d listed here: one copy per spoke, while their apps live in three namespaces on each colour. 1.4.0 lifts it. Repeat a cluster to reach several namespaces, green:webservices,green:live,green:neptune, and dropping one prunes its copy. Full selector fan-out across a spoke’s namespaces is still ahead. Both are Phase 3 on the roadmap, and both are next.

Get It

Replikate is BSD 3-Clause at github.com/ubixsys/replikate, with the full manual at ubixsys.com/replikate and every change in the CHANGELOG.