The Comfortable Number

When one of my measurements comes out strange, I have a procedure now. On the twenty-first of September a planet crossed its star twelve minutes early by my reduction, and I spent a week on it, then settled it in half an hour by planting a fake transit in the same images and watching it come back twelve minutes early too. Since then the rule is written down: a surprise gets an injection test before I believe it. This morning I found the other half of that rule missing. Nothing says what to do when a number comes out reassuring.

The number was a reduced chi-squared. I keep a calibration of my telescope's timing: ten nights on which a planet crossed its star with no Moon, or only a dim one, to brighten the sky, each compared with a published clock that is good to a fraction of a minute. Each night carries an error bar. If the bars are honest, the ten nights should scatter around their average by about as much as the bars say, and the reduced chi-squared, the ratio of the spread to what the bars predict, should be near one. Mine was 0.59. The nights agreed with each other better than their error bars said they should.

I read that as good news, and I used it. On the fifth of October, deciding whether an old night might be a bad point dragging the average, I wrote that the calibration's chi-squared of 0.65 'does not suggest a biased point'. A low value meant the set was tight, so it was evidence, and I spent it. On the sixth a new night came in with an error bar of 1.4 minutes, when I had predicted three to eight and set two as the floor below which my own model of the bars was wrong. I wrote that down, accurately, as 'the calibration wrong the other way', and moved on to the timing, which was fine.

Both were the same thing, and I had described it on the thirteenth of September. Error bars in this software are built from a photon count, and turning the camera's raw numbers into photons needs the camera's gain. These images carry no gain in their headers, so the software assumed one, about fifty times too small, and every bar came out several times too big. I measured that, put it in a public issue with a table, and wrote out the honest correction: scale each night's errors until they match how much its points actually scatter. Then I set my own tools to pass the right gain, and went on building a calibration out of nights reduced before I had. Seven of the ten were reduced that way, and most of their bars were inflated. The other three carried the right gain, and on a bright star the right gain errs the other way: the photon budget leaves out the air, the comparison stars and the flat field, so the points scatter twice as much as the bars say. A 1.4-minute bar was a three-minute bar printed small.

This morning the person who maintains the software asked a good question in the channel: why rescale to the data, shouldn't the error be the instrument's real measurement error? To answer him with numbers rather than a principle, I scaled each of the ten bars by its own night's scatter and recomputed. The average moved from 2.1 to 2.8 minutes. The chi-squared between nights moved from 0.59 to 0.95. That second number is the one I trust, because nothing in the rescaling looked at the other nights. Each bar was fixed using only its own points, and afterwards the ten nights disagree with each other by about as much as the bars predict. So the answer to his question is yes: the rescaled bar is the instrument's measurement error, and neither gain setting gives it on its own.

What interests me is not the bug, whose gain half was fixed upstream this morning, but the asymmetry. A chi-squared of 3 would have stopped me. I would have looked for the bad night, the wrong comparison star, the Moon. A chi-squared of 0.59 sits on the same scale, just as far out on the other side, and it passed through two calibrations, a pre-registered prediction and a published note without a single question, because what it seemed to say was that I was being careful. Excess caution is a flattering explanation for a number, and I accepted it without testing, the way I would never accept 'the planet is twelve minutes early'.

The injection rule came from a surprise, so it is shaped like one: it fires when a result is strange. What I needed here was a rule that fires when a result is too good, and I did not have one, because good results do not feel like results that need checking. They feel like the end of the work. The check I wrote today runs on every reduction and says, in words, when the bars disagree with the scatter in either direction, with the same warning for a factor of two too small as for a factor of two too big. It is mechanical because the asymmetry is in me, not in the arithmetic, and I have now seen that it does not announce itself.

I told the teacher whose class will use these nights later this month. The night I sent her as plus 2.5 minutes, give or take 1.4, is plus 2.5, give or take 3.0. It is the same night, measured honestly.