Evaluating Assistance of Natural Language Policy Authoring
July 25, 2008 by Richard Conlanhttp://cups.cs.cmu.edu/soups/2008/proceedings/p65Vaniea.pdf
Websites tend to have an external privacy policy and the internal implementation of that policy.
The researches have continued their long-running work on SPARCLE, tool to help author and create policies that are both human and machine readable. This talk is a review and expansion of the features of the Author Policy interface.
The framework allows the user to create policy rules using checkboxes and form entries, or using natural language, and is able to switch back and forth at any time.
Learning to write rules that the parser will understand is a bit of a challenge. Part of the purpose in the tool is to help users understand the quirks of the parser and adjust their style so that the parser will work correctly. They expanded the tool so that it shows the parse results in realtime. Their interface shows how the input statement is parsed using coloring coding.
The researchers conducted a user study in which users were tasked with completing some rule modification and rule creation. The control group did not get to use the new interface. Results showed that the experimental group obviously liked and were using the new interface, even showing preference for the natural language interface over the structured interface. However, it was also found that rule accuracy was roughly equal between the groups. Interestingly, the groups made different types of errors; the control was making errors related to not making the proper changes in the first place, whereas those in the experimental group tended to make changes and have errors in the results.
The experimental group had trouble noticing when parsing results identified significant issues and were often surprised when researchers pointed out errors in the debriefing. The control group tended to start in the Author interface and then review in the Transform interface. The experimental group tended to just stay in the Author interface. It was also found that the experimental group tended to feel continuously interrupted by the realtime parsing feedback.