Personal Choice and Challenge Questions: A Security and Usability Assessment

July 16, 2009 by Richard Conlan

http://cups.cs.cmu.edu/soups/2009/proceedings/a8-just.pdf
Mike Just and David Aspinall

Challenge questions often serve as part of a password recover mechanism, though are sometimes included along with conventional authentication.  For a long time there was little research in this area, but some studies have emerged recently, generally concluding that challenge questions are neither very usable nor secure. 

This study sought to improve the current state by proposing a systematic and repeatable way to analyze the security and usability of challenge questions, with a focus on user-chosen questions.  They proposed a novel experiment for collecting 180 challenge questions.  The participants were directed to a website where they entered their security questions, but wrote their answers on a piece of paper that they then sealed in an envelope.  A few weeks later they were re-presented with their questions and asked to answer them again, and then compare against their original answers and report their success-rate at memorability to the researchers.  It was conducted in this manner to keep from actually gathering sensitive information from the participants.  Unfortunately, even after just twenty-three days the users were unexpectedly poor at properly remembering their answers.

The researchers then analyzed the submitted security questions, looking at three different attackers - one making a blind guess, one making a focused guess, and one allowed to make observations of the user.  For each attacker they scored each question as Low, Medium, or High security.  Low was chosen anything less than the threshold set by a 6-character alphabetic password, and Medium as anything less than an 8-character alphanumeric password.  It was found that of the 180 characters, 174 were Low security.  Still, since the attacker needed three questions to spoof authentication, these were combinable for the blind and focused guess attackers, meaning the cumulative security was often Medium or High.  However, the analysis left out various forms of attack such as another site asking the same questions, the answers to questions correlating, etc.