mostlysignssomeportents:

Ashwin Parameswaran’s “People Make Poor Monitors for Computers” is a fascinating look at (and indictment of) the way we design automation systems with human fallbacks. Our highly automated, highly reliable systems – the avionics in planes, for example – are designed with to respond well to all the circumstances the designers can imagine, and use human beings as a last line of defense, there to take control when all else fails. But human beings are neurologically wired to stop noticing things that stay the same for a long time. We suck at vigilance. So when complex, stable systems catastrophically fail, so do we. Parameswaran quotes several sources with examples from air-wrecks, the financial meltdown, and other circumstances where human beings and computers accidentally conspired together to do something stupider than either would have done on their own.

https://boingboing.net/2012/04/08/when-human-beings-are-asked-to.html