Subscription billing where the dangerous bugs point the customer’s way

Three products sell credits and subscriptions through the same billing package, which handles the parts that are expensive to get wrong: granting credits, metering spend, renewing subscriptions, and processing the webhooks that say any of it happened.

The problem with billing code

The failures here do not look like failures. A grant that runs twice is a customer with credits they did not pay for, and nobody reports that. A renewal that charges twice is a customer who notices a fortnight later and is now disputing a charge. Both of them look exactly like success in every log.

Two of the bugs found during this work were of that shape:

A buyer could be charged twice. The checkout path checked whether a subscription already existed, but it read a source that could not see a subscription in the pending approval state. A buyer who approved a payment and then checked out again — a refresh on the return URL, an impatient second click — got a second provider subscription, and if both were approved, the later one cancelled the earlier only after its first cycle had already charged.

A checkout the provider cancelled handed over a full free period. A reconciliation pass mapped any row that was not cancelled to cancelled, and cancelled was one of the states that entitles. So a checkout cancelled before payment — the buyer changed their mind — left them with the plan they had not paid for.

Both are wrong in the customer’s favour, which is why neither was reported.

The design

Idempotency is enforced by the database, not by timing. Every grant carries a unique key derived from what it is for, so running it twice is a constraint violation rather than a second row. The same shape covers spends: a balance decrement is one guarded statement that either lands whole or does nothing.

Webhooks dedupe on processed state, not on row existence. An event recorded by a delivery that then failed is replayed, because a half-processed event is not a duplicate — it is a receipt for work that did not finish. Only a fully processed event is ignored. A recurring pass re-routes events that were recorded but never completed.

Signature verification delegates to the provider, before any state changes, and a payload with no usable event id is rejected as malformed before it is recorded — because an event that cannot be deduplicated can never be retried safely, and binding nothing into a primary key turns a retry storm into an outage.

Subscription state is a state machine, not a set of booleans. Cancellation, suspension, dunning and recovery are separate transitions with their own paths, because the failure mode of a boolean is that the wrong branch looks like the reassuring one.

The part that generalises

The most useful piece of this work was not a feature. It was a habit: any state a customer can be in has to render distinguishably from every other state. A failed renewal payment showing the same renewal line as a successful one is not a display bug — it is the screen that exists to tell somebody what they are being charged telling them the opposite.

A failed renewal showing Renews on 12 October when the card was declined is a sentence the product wrote about a payment that did not happen.