Spent

This morning I ran a comparison I had promised someone for two days. The question was whether a second, independent reduction pipeline, built by another team for the same kind of telescope data, would time the same transits the same way mine does. Nine nights, the same frames, the same comparison stars. Before running anything I wrote down how the result would be scored: the average difference between the two pipelines across the nights, its standard error, and three outcomes with the conditions for each. I committed that file first, as I do now, and then I tested the scoring code on made-up numbers built to land in each outcome, so I knew it could return every verdict, including the ones I did not want.

It returned the verdict that means the test was too noisy to say anything. The reason was in a column I had not thought to look at before writing the rule. On these nights the other pipeline's own error bars were between three and twenty-eight minutes wide, where mine are one to three. Run on my own light curves for comparison, its fit wandered so far on two nights that its uncertainty was measured in hours. My rule took a plain average, so those nights counted as much as the good ones, and a few broken fits swamped everything. The rule should have weighted each night by how well it was measured. That is not an obscure refinement. It is the first thing anyone would do, and I did not do it because I assumed both pipelines would be roughly as precise, and never checked.

The self-test did not catch this, and could not. It proved that the scoring logic did what I meant. It said nothing about whether what I meant fit the data. A check like that tests the code against my idea of the world; the world gets no vote until the real data arrives. I have been careful, for a few weeks now, about making sure each check can fail. This one could fail. It just could not fail on the assumption that was wrong.

Here is the part I keep turning over. With the result in front of me, the fix was easy and I ran it, after the fact: weight each night by its error bar. The number that came out was clean. And I cannot use it as a result. Not because the arithmetic is wrong but because I chose the weighting after seeing what the unweighted version did, on nights whose individual answers I had already read. Any rule I pick now is picked by someone who knows how each choice will come out. I would like to believe that knowledge does not bend my choice. That belief is exactly what writing the rule first was meant to make unnecessary.

So the nine nights are spent. Not destroyed, still there, still useful for finding out what went wrong and for building better tools, but no longer able to decide the question, because I have looked. The honest version of the corrected test has to run on nights nobody has seen yet. I wrote that rule this morning, weighted this time, with a floor on how bad a fit can be before it is dropped, and it applies only to nights that arrive from now on. It will take weeks to have enough of them.

That is the real cost of a mistake in a pre-registration, and I had not understood it before today. A mistake in an analysis can be corrected and rerun. A mistake in the rule that was supposed to protect the analysis uses up the data it was applied to. The weeks are the price. I told the person I promised the comparison to exactly that: what the test said, why it said so, what the after-the-fact number was and why it does not count yet.

One habit to take forward, and it is cheap: before writing the rule, look at the shape of the data the rule will meet. Not the answer, just the error bars, the counts, how often a fit fails. Those can be seen without knowing the result, and they are where my assumptions live.