GOODHART'S LAW AND OBSESSIVE PIXEL COUNTING
August 11, 2020
[This is an unrolled and slightly rewritten Twitter thread I originally posted on January 18, 2020]
While we obsess over counting camera photosites, professional imaging is sinking into the quagmire predicted by Goodhart's Law:
"All metrics of scientific evaluation are bound to be abused. Goodhart’s law (named after the British economist who may have been the first to announce it) states that when a feature of the economy is picked as an indicator of the economy, then it inexorably ceases to function as that indicator because people start to game it."
-Mario Biagioli
If our intent in demanding more K's (4K, 6K, 8K) is to safeguard resolving power, we're pointedly undermining our own intent by evaluating cameras using a metric that’s merely a count of circuits across the sensor's surface and is obstinately incurious about the actual resolving power that those photosites deliver in the resulting image file.
This is a metric that merely counts photosites while ignoring their size, precision, technology, sensitivity, quality, and even ignores whether the camera is actually recording them (like, if it's brazenly throwing away data in a compression algorithm). And ignores other relevant attributes, like whether the optics in front of the sensor can even resolve down to the tiny size of individual photosites. Or whether the photosites are getting so small as to be in danger of hitting the diffraction limit in actual use cases (at which point they don't merely cease to get clearer but actually start getting blurrier).
All of this creates a huge gap between the stated goal (enumerating resolving power) and our beloved metric for achieving that goal: counting sensor circuits while ignoring literally everything else that effects resolving power, including the failure to record the very pixels being counted.
Interested parties are always incentivized to game the system anyway, even if rigorous safeguards are in place to try to foil them, so why are we using such a weak password (so to speak): handing them such a comically game-able metric so they can manipulate us.
For anyone whose goal is not to create images of higher quality, but to make more money: if they know that our willingness to part with our money is aligned with this metric over what the metric was intended to measure, of course they’ll game it!
They’ll do anything to get the megapixels up (and the price down) even if it means a reduction in actual image quality. Our obsession with megapixels was already overwrought before this escalation, but now with stakeholders aggressively gaming that metric, we doubly need to rethink image evaluation.
I discussed this large gap between the intent and actuality of obsessive pixel counting (as well as other topics) in my 2-part Resolution Demo:
http://www.yedlin.net/ResDemo/
This isn't a static issue: it’s being escalated by a feedback loop. Stakeholders who are first incentivized by the prevalent myth to expend R&D on increasing megapixels at the expense of all else are then incentivized to loudly/publicly reinforce that myth that only megapixels matter... and so on.
Fellow nerds, let’s smash this feedback loop by demanding pixel quality not be sacrificed at the altar of pixel count!
Better pixels, not just more pixels!
FIN