The Raw Data Survived. The Measurement Layer Didn't.
A broken dashboard does not have to take the truth down with it. We recovered 71 days of crawler measurement after a damaging warehouse change because the 6,621,793 live raw request rows were protected separately.
The derived measurement layer could no longer be trusted. The original request evidence remained intact.
That separation turned a wrecking ball into a recovery job.
Source data and measurements are not the same thing
A server request is an event. It happened at a recorded time, on a recorded host, for a recorded resource, with a recorded response.
A crawler total is an interpretation. It depends on classification rules, response filters, content rules, date coverage, grouping, cache logic, and presentation.
The interpretation is useful. It is also replaceable.
That distinction sounds obvious until the summary table is the thing everyone looks at. Once a dashboard becomes familiar, it starts to feel like the evidence. It is not. It is a product made from evidence.
| Layer | Example | Recovery rule |
|---|---|---|
| Source evidence | Raw server requests and retained archives | Protect, preserve, and do not rewrite for convenience |
| Classification | Bot identity, purpose, scanner exclusion, spoof evidence | Version the rules and record their provenance |
| Derived measurement | Daily crawler totals, page summaries, query associations | Rebuild from source when the rules or logic change |
| Presentation | Charts, labels, detail panels, and exports | Never let a display fallback turn missing evidence into zero |
What broke
The incident involved the Performance measurement layer that joins crawler evidence to pages and query-level views. Good Bots appeared absent where the warehouse still held usable crawler associations. A repair had changed derived states without protecting the distinction between unavailable evidence and a measured zero.
The damage did not mean the raw logs were wrong. It meant the process translating them into publishable measurements needed to be rebuilt and proved again.
What the recovery had to prove
Recovery could not stop at "the totals look right now." It had to answer four questions.
- Can every rebuilt count be independently reproduced from its source?
- Can missing source periods remain visibly unavailable?
- Can related tables change together or roll back together?
- Can the system prove the raw evidence was not rewritten?
The final receipt answered all four.
| Recovery block | Dates | Source volume | Eligible crawler requests | Result |
|---|---|---|---|---|
| Retained window | 36, August 5 through September 9 | Live raw requests | 153,231 | Zero count or purpose-split differences |
| Historical recovery | 35, June 29 through August 2 | 3,751,275 source requests | 233,663 | 3,027 site and day groups reconciled |
| Unavailable evidence | 2, August 3 and August 4 | Not recoverable | Unavailable | Preserved as gaps, not zeroes |
The cache ledger now carries 71 validated dates. The recovery applied 280 table and day replacements. All 280 matched the reviewed proposal.
The 6,621,793 live raw request rows remained unchanged.
Why atomic daily replacement mattered
The measurement did not live in one table. A day touched eight related tables.
Updating them one at a time would create a new class of failure: a summary from one version, page details from another, and query associations from a third. The screen might still render while telling three different stories.
The recovery prepared each date as an isolated proposal, checked for concurrent changes, then applied the related states in one transaction. If one part failed, the date stayed in its previous state.
Exact before snapshots and a resumable receipt made rollback a designed action instead of an emergency guess.
Historical classification needs provenance
The historical source requests were classified with the vendor ranges and rules in effect on September 9.
That is useful, but it is not the same as claiming the warehouse knew those classifications on June 29. The cache displays the retrospective provenance because the rule date is part of the measurement.
A better classifier can make an old period more accurate without making it contemporaneous. The result needs both dates: when the event happened and which rules interpreted it.
Missing is a result
August 3 and August 4 could not be recovered from the available evidence.
The honest value is unavailable.
Zero would make the chart prettier and the measurement false. It would say the system looked and counted nothing when the real condition is that the system cannot support a count.
That difference should survive every layer, including charts, exports, comparisons, and averages. Unknown data is not a quiet zero.
The tests now defend the lesson
The current Performance accuracy suite runs 33 regression checks. It covers scanner exclusion, successful content retrieval, whole-count custom-IP arithmetic, IPv4 and IPv6 ranges, and retained spoof evidence.
Separate history recovery tests cover exact restore behavior and transaction safety.
A test does not prove every future measurement is correct. It proves that specific mistakes now have a machine waiting to object.
What this means for dataset SEO
A dataset is not trustworthy because it has a schema, an endpoint, or a clean table. It is trustworthy when its evidence chain is visible.
For a derived dataset, publish or retain enough information to answer:
- What raw evidence produced this value?
- Which rules were applied?
- When were those rules current?
- What source periods are missing?
- Can the result be rebuilt?
- Can a replacement be reversed?
This is the difference between machine-readable data and machine-auditable data.
The incident did not prove AI should stay away from a warehouse. AI helped damage the layer and helped repair it. It proved that powerful tools need an evidence boundary stronger than their confidence.
The raw data survived. That was the backup plan doing its real job: not merely storing a copy, but preserving the right to find the truth again.