A User Study of Off-the-Record Messaging
July 25, 2008 by Richard Conlanhttp://cups.cs.cmu.edu/soups/2008/proceedings/p95Stedman.pdf
Instant messaging has become a common form of information on the Internet, but most of the available services are not secure. There are available solutions, such as SecureIM, Pidgin-Encryption, and SILC, but they all have shortcomings compared to OTR (Off-The-Record).
The goal of OTR is to make conversations online as private and secure as face-to-face conversations. OTR was recently redesigned to be more easily used by non-technical users. The researchers for this study performed a user study on the new version of OTR.
Optimally using OTR requires initiating encryption per conversation and authenticating the user at the other end of the connection. In the original version of OTR the only way to authenticate was by manually verifying each users’ key fingerprint. The newer version allows users to authenticate by entered a shared secret, such as the place they first met.
The study was conducted using the “think aloud” method and included four pairs of friends. In some sessions friends were paired, and in others one friend from on pair of friends was talking to somebody from another pair of friends. This latter setup was intended to test the usability among users who didn’t know each other well. To test learnability of the system they ran a second session in which the users were paired differently.
By default OTR initiates encryption automatically, so nobody had problems getting the crypto going. Participants did, however, have trouble authenticating one another. The most common first attempt was to press the OTR button, but this does not actually authenticate a session (it actually rekeys the session). The next step was commonly to click the injected “authenticate” link provided in the IM window, which brings the user to a help page. Unfortunately, this did not actually help any participants because it did not say to “right-click”. Many users just looked at the images on the help page, which unfortunately lead to authentication errors because there is an image of “how not to authenticate” pictured before one describing how to do it properly.
Two participants tried to perform the “old style” authentication, which lead to much confusion as one buddy had thought they were verified while the other was not because the fingerprint verification method is one-way and must be performed on each side of the connection.
From these results the researchers proposed:
- have the OTR menu open when left-clicking the button
- the help page needs clearer information, such as saying to right-click on the button
- the help page should make it more clear that the “what not to do” image was what not to do by crossing it out or otherwise pictorially indicating the danger
- the authentication interface should itself help guide the user towards proper use of the system
- the interface should provide a box for a “question” in addition to the shared secret input