Dangerous Analogies

May 11, 2005 by Ping

Argument by analogy is common in discussions of computer security.  It’s often a useful way to reason about things, but sometimes an analogy can mislead you.  I think a large class of misunderstandings about computer security are due to recurring problems in drawing analogies between what goes on in a computer and what happens in the real world.

The real world is bound by the laws of physics, whereas programming isn’t.  Sure, there are physical limitations to the computing hardware we build, but systems engineers have worked hard to give us a mathematical abstraction to build on, and they’ve done a pretty good job.  Within this abstraction, programmers basically get to make up whatever they want.  Any hardware limitations that do make it through to the software level bear little resemblance to the physical world.

The huge difference is that objects in the real world come with existing relationships, but objects in the software world do not.  Physical objects scatter light, exert forces on other objects, and so on.  By default, every object in the universe is constantly affecting many other objects.  Suppressing these relationships takes special work.

In software, nothing affects or communicates with anything unless you make it do so.  Creating relationships takes special work.

Because people are forced to work from physical laws in the real world but have the flexibility to choose simpler laws for software, that ought to make it simpler to build reliable software than reliable real-world objects.  It also ought to be possible to uphold a much higher standard of reliability with software, since formal proofs can actually be written about software, and that’s impossible with real objects.  Paradoxically, we seem to have much more difficulty with software.  All the time, people construct buildings and bridges that last dozens of years, but programmers seem hardly able to create a Web browser that runs for a week without crashing.  Civil engineers are confident enough to certify their work and estimate low probabilities of failure, yet software companies routinely disclaim all responsibility for the proper functioning of their own products.

This problem happens because people are propagating and applying the wrong mental model for software security.

I’ve thought about that too — more about the relation or contrast with civil engineers, but the same issue of explanations and metaphors. And always a bit confusing, *computers* actually fulfill all those engineering expectations pretty well, but the programs don’t. So how do we explain that? Lots of people try with process, describing how programmers work differently than engineers, but that’s not a very good justification or explanation; it doesn’t point out the qualitative differences in the work produced. I certainly don’t fully understand the combination of math, bureacracy, and engineering that makes up a typical program, which makes it pretty darn hard to explain.

 

It may be possible to hold programs to a higher standard of correctness, but the real question here is cost. If you offered people to build one bridge you could be sure worked, or 10 bridges that would probably work 90% of the time, people would go for the one bridge. Not the same with software, and with good reason: a browser that only crashes occasionally is much more useful to me than a bridge that only fails sometimes. And I think lots of people made the choice to go with something that may not function perfectly, but has more features, or is available today rather than a year from today. Enough of them in fact that not only did these low-assurance systems succeed, but the whole idea of making high-assurance systems has become very much of a niche field.

The adversarial model, which has always been a concern, but only recently a pressing one, has really turned this whole situation on its head. There are few useful incremental improvements in security. A program with 90% fewer security holes is basically as bad. I think that recently, people have started being more willing to pay the cost for making software more reliable and secure, but our whole process of creating software has a long way to go before that’s really possible.

 

“The huge difference is that objects in the real world come with existing relationships, but objects in the software world do not.”

Not true. Objects in the software world might not come with existing PHYSICAL relationships, but they are surely saddled with psychological relationship.

At a base level a button is there to be pushed and a slider there to be slid and an option group there to be…erm…optioned. At a slightly higher level there are expectations that, say, blue-colored text is probably a hyperlink (especially if underlined) and clicking it should open SOMETHING; a little disk icon should save whatever I am working on; a little printer should print; &c.

But those are all really simple / low-level expectations. The bigger issue is that a good GUI will often mirror elements of the problem-domain, and inherently brings along some expectations. For instance, if I make an application that models a slot machine then there should probably be a pull handle, drums that spin, an area little coins come out, and so forth. Technically I could model a slow machine where you push a button, random symbols are just displayed, and a counter just tallies wins and losses…but by pulling in the analogy of the real slot machine I add a lot of inherent usability…but at the cost of bringing along certain OTHER expectations that I might not want to include.

The point of all this being that analogies, whatever their evils, tend to be very helpful in creating abstractions, and we have to take the good with the bad. So, we are forced to work within certain restrictions as well, even if the medium doesn’t inherently include them.

You and I are talking about different things. I agree with you that there are certainly human expectations for how parts of a user interface behave. But by “relationships” I didn’t mean relationships in people’s minds; I meant cause-and-effect relationships among the software components. These don’t exist unless you write the code to make them happen.

 
 

Maybe this blog should include some sort of login to at least kinda-sorta authenticate posts?

 
 

Your comparison between software and a building/bridge misses the mark. Youre comparing a base product (software) with a completed product (bridge). A more acurate comparison is between software and bricks. You use both which are mass produced and identical to create a unique result. The bridge is analogous to your personal computer with the programme installed on it - the person who set this up is responsible for ensuring that it doesn’t break (ie: not the manufacturer).
The company that made the bricks made no guarentee that the bridge won’t fall down, they merely state that if used properly in a suitable environment then all should be fine. If an engineer built a bridge on sand or without enough support stands then the brick maker isn’t at fault when it falls down.
The software in the box IS secure, its when you install it on a system which allows other (bad) people to access it and exploit weaknesses that it fails. This is expressly outside the control of the programmer.
You can argue till your blue in the face that software should be designed with this use in mind (the lowest denominator) but the response is “WHY?”. As an informed user I personally suffer no spam, no viruses, no spyware or any of the other problems people complain about, so the system/software DOES work, only people don’t use it properly.
If this is too hard for the average user then this means that you have to make it easier, not more secure - ie: create a brick that knows how a bridge should be built.

 

[...] line, and sinker. This is exactly the type of mistaken comparison I was talking about in recent entries. Viruses are dangerous when they enter your body because they have comp [...]

 
amonynouse wrote:

Argument by analogy is just like trying to hit a large trout with a spade. While it’s underwater.

 

[...] entally, the explanation of this “Law” abuses the type of real-world analogy I recently described. There’s a nice analogy between running a program and eating [...]