most of what we know is inherited. somebody solved a problem in 2011, wrote a library, and now a lot of us hold their opinion without remembering that it belonged to a person.
frozen, not wrong
one morning every metric in the company stopped. not wrong — frozen, which is worse. wrong gets reported. frozen gets trusted.
the documentation was no help, because documentation describes the thing as designed: a queue, some disk, and one number that said fifty gigabytes. reality had turned up with two hundred. nothing was broken. it was doing exactly what it had been told to do, at four times the size.
inventory, not brilliance
that is what taking a thing apart actually buys you, and it is not brilliance — it is inventory. you need to know which three parts your system is made of, so that when it stops you are debugging physics instead of folklore.1
it took the whole day. the fix, when it came, was shorter than the incident report.
- folklore sounds like: we set that to fifty because of a thing that happened before you got here. ↩