A new client was born on the old product

On 9 September we set up a new client workspace on our newest release. The setup job completed every step and reported success. The client's first screen was our old dashboard, and it stayed that way for two days.

Nothing was broken in the way a test looks for. The new home was there. It was switched off.

2

One switch, off by default

Our new home screen shipped on 6 August behind a switch that defaulted to off. That is a normal way to release something. You turn it on for each client when they are ready.

Every existing client had been switched on by hand, with database changes nobody recorded. Nothing in setting up a new workspace turned it on. Migrations laid down the defaults, the seed set only a name, a first-run marker, the demo setting and branding, and no step in setup wrote the switch. This was the only workspace our setup worker had created since the switch shipped, so it was the first to be born on the default.

Why nothing caught it

A finished setup job proved nothing about switches. No gate, check or audit read one back.

Our activation checks looked at other switches, and the written rule for them said a switch being off was a pass.

Our CI could not see the problem either. Every CI environment starts against a placeholder database where the settings read comes back empty, so the switch is off everywhere CI can reach. When the new home launched, its switched-on path had no browser test in CI.

What changed

Setup now writes the switches every live client runs, then reads them back. It first checks which switches the workspace's release actually has, so an older release is never asked for a column it does not ship. If a switch did not stick, that step turns red in our admin view and in our canary.

The database default is now on. Changing the default alone would not have reached the settings row, because that row is created roughly 1,350 migrations earlier, so the change also sets it for any workspace that has never been used. A dry run across our 14 databases was clean.

The activation audit now fails when the switch is off or missing, and each failure names the fix. A canary checks it. For the release on 13 September, a scripted browser check logged in as a client on every workspace running the product and confirmed the new home and its composer rendered. It passed on all six.

The rule we wrote down: a new workspace's starting configuration is asserted, not assumed.

What is still not right

When a switch does not stick, setup still finishes. The step turns red, but the job completes, because the welcome email has already gone out and a rerun would send it again.

What to ask any vendor

When you add a new customer, what checks that they got the current product and not the default? Which of your features are behind switches, and who turns them on? Can your setup report success while the customer sees the wrong thing?

Good answers come with a demo. The insights are free. If you want this level of engineering pointed at your operation, start with the free audit. The plan is yours to keep either way.