When Systems Drift Apart

The characteristic integration failure in this category is not an outage. It is two systems that agreed yesterday and disagree today, with nothing reporting it.

How drift starts

A synchronisation ran and partially failed, writing half the records.

A record was rejected for a validation reason and nobody read the log.

Someone edited directly in one system, bypassing the integration.

A code was renamed at one end and the join stopped matching.

A retry created a duplicate.

A rule changed in one system and not the other, so the same hours classify differently.

All six are silent. None produces an alert by default.

The reconciliation

Scheduled, comparing counts and totals across the boundary.

People: active in the source against active in the timesheet system.

Projects: same.

Entries: count and total hours per period per pay group, both sides.

Exported against received, for payroll.

Differences reported to a person, with a threshold so that normal timing differences do not generate noise.

This is a day of work to build and it is what turns a silent failure into a Tuesday morning email.

What each difference means

Counts differ, totals match: duplicates or merges.

Totals differ, counts match: a rule or rounding difference.

Both differ, consistently in one direction: a filter is excluding something.

Both differ, growing over time: deletions not propagating.

A single large outlier: one bad record, usually findable by sorting.

Designing for detectability

Every integration run logs: start, end, records attempted, succeeded, failed, with reasons.

Failures are queued, not dropped, and visible.

A run that processes zero records raises an alert, because zero is almost never correct and is the signature of a broken filter.

Last-successful-run timestamps are exposed somewhere a person looks.

Without these, the first sign of trouble is a person's pay, which is the expensive way to learn.

Repair

Replay rather than manual entry, which means the integration needs a replay path from the start.

Idempotent writes, or replay creates the duplicates it was meant to fix.

A record of what was repaired, because the same failure will recur and the second investigation should be shorter.

And a look at why it was not detected sooner, which is usually the more valuable finding.

The zero-record alert

The cheapest integration alarm and the one most often missing.

A synchronisation run that processes zero records raises an alert.

Zero is almost never correct and is the signature of a broken filter, an expired credential or a changed field name.

Without it, a broken integration looks exactly like a quiet week.

Add the last-successful-run timestamp somewhere a person looks, which catches the case where the job stopped running at all.

Repair by replay

The design decision that determines how a failure is fixed.

Replay rather than manual re-entry, which means the integration needs a replay path from the start.

Idempotent writes, or replay creates the duplicates it was meant to fix.

A record of what was repaired, because the same failure recurs and the second investigation should be shorter.

And a look at why it was not detected sooner, which is usually the more valuable finding than the failure itself.

Reconcile campaigns and time records

Automated data movement must be observable and repeatable. Use this connection example to define retry, duplicate and reconciliation tests.