Dangerous Analogies
May 11, 2005 by PingArgument by analogy is common in discussions of computer security. It’s often a useful way to reason about things, but sometimes an analogy can mislead you. I think a large class of misunderstandings about computer security are due to recurring problems in drawing analogies between what goes on in a computer and what happens in the real world.
The real world is bound by the laws of physics, whereas programming isn’t. Sure, there are physical limitations to the computing hardware we build, but systems engineers have worked hard to give us a mathematical abstraction to build on, and they’ve done a pretty good job. Within this abstraction, programmers basically get to make up whatever they want. Any hardware limitations that do make it through to the software level bear little resemblance to the physical world.
The huge difference is that objects in the real world come with existing relationships, but objects in the software world do not. Physical objects scatter light, exert forces on other objects, and so on. By default, every object in the universe is constantly affecting many other objects. Suppressing these relationships takes special work.
In software, nothing affects or communicates with anything unless you make it do so. Creating relationships takes special work.
Because people are forced to work from physical laws in the real world but have the flexibility to choose simpler laws for software, that ought to make it simpler to build reliable software than reliable real-world objects. It also ought to be possible to uphold a much higher standard of reliability with software, since formal proofs can actually be written about software, and that’s impossible with real objects. Paradoxically, we seem to have much more difficulty with software. All the time, people construct buildings and bridges that last dozens of years, but programmers seem hardly able to create a Web browser that runs for a week without crashing. Civil engineers are confident enough to certify their work and estimate low probabilities of failure, yet software companies routinely disclaim all responsibility for the proper functioning of their own products.
This problem happens because people are propagating and applying the wrong mental model for software security.
May 11th, 2005 at 11:23
I’ve thought about that too — more about the relation or contrast with civil engineers, but the same issue of explanations and metaphors. And always a bit confusing, *computers* actually fulfill all those engineering expectations pretty well, but the programs don’t. So how do we explain that? Lots of people try with process, describing how programmers work differently than engineers, but that’s not a very good justification or explanation; it doesn’t point out the qualitative differences in the work produced. I certainly don’t fully understand the combination of math, bureacracy, and engineering that makes up a typical program, which makes it pretty darn hard to explain.