Heuristic Evaluation of Enigmail

(706.100 Seminar/Projekt 6SP WS 2015/16)

Authors

Carina Yasmin Enzinger
Bianca Helbig

Supervisors

Keith Andrews
Ralph Wozelka

Expert Evaluators

Konrad Baumann
David Derler
Vesna Krnjic
Johanna Pirker

The Results of a Heuristic Evaluation of the Thunderbird Add-on Enigmail

Report of 8^th Jan 2017

1 Summary

Enigmail is a free Thunderbird Add-on for encrypting and signing email. This heuristic evaluation of Enigmail was performed in Jan and Feb 2016 in collaboration with four experts (see Table 1) in the fields usability and/or cryptography, who do not use Enigmail regularly. They were first given some background information about cryptography in general and some typical use cases for Enigmail. They were then asked to install Enigmail, to send signed and encrypted emails, and to review Enigmail by pointing out any positive and negative impressions (findings).

The experts were asked to think out loud while inspecting the interface, like in a thinking aloud test. After all four evaluations, all positive and negative findings were documented and assigned to heuristics (see Appendix A) by the project team. The findings were gathered in a table and rated for their positivity and severity, by all experts and the authors. The list of all positive findings was sorted in decreasing order of positivity (see Section 6). The list of all negative findings (problems) was sorted in decreasing order of severity (see Section 8).

The most positive finding was that none of the experts had any problem locating the Enigmail project page and downloading it. Although the evaluators struggled with the installation, all of them did eventually manage to install it. Vesna Krnjic remembered testing a much older version of Enigmail which she never really used, because it was too complicated. She remarked that the version of Enigmail (38.5.1) evaluated in this study had much better usability than the older version, and thus appears to be heading in the right direction.

For typical computer users (non-experts in cryptography), the largest difficulty appears to be the installation and setup of Enigmail. Howvere, even experts had problems installing Enigmail, since the setup wizard's manual mode does not function at all. The second most important issue occurred on the Mac: the user received feedback that a public key was successfully added, when in fact it had not been added to Enigmail's Key Management and encryption did not work. The parallel use of GPG Keychain and Enigmail on the Mac appeared to be the root cause of a number of problmes (problems 1, 2, 9, 19, and 24).

The problems ranked in 3^rd, 4^th and 5^th place are all related to a lack of feedback from Enigmail about what is currently happening. Problem 3 addresses the conflation of negative feedback with positive feedback, leading to confusion about whether or not the email will be encrypted. Problem 4 describes the issue that the message about saving the revocation certificate in a secure place (like a CD) comes after the user already saved it somewhere. Problem 5 reflects a lack of clarity about which public key Enigmail is using, in cases where a user has multiple key pairs.

2 Introduction

In 2014, Edward Snowden described the GPG (GnuPG) software [GPG] used to encrypt emails as "damn near unusable" [Sno2014]. Arne Padmos discussed some of the reasons for this in his talk “Why is GPG 'damn near unusable'?” [Pad2014]. The problems involved today are much the same as those specified in the 1999 paper “Why Johnny can't encrypt” [Whi1999]. Not much has changed in the past 17 years.

Enigmail [Bru2017] is a free add-on (extension) for the Thunderbird [TMF2017] open source email, news, and chat client. It can encrypt and sign emails using GPG. The Enigmail project is maintained by Patrick Brunschwig. The enigusab project [And2017, Woz2016] was initiated by Keith Andrews and Ralph Wozelka to help improve the usability of Enigmail.

3 Evaluation Methodology

Heuristic evaluation is a form of usability inspection, whereby a small team of usability specialists (typically 3 to 5) use their experience and judgement to analyse and assess a user interface. Heuristic evaluation is sometimes also called an expert review. The method was first formalised by Jakob Nielsen and Rolf Molich in 1990 [Nie1990].

Each evaluator assesses the interface alone, using a small set of usability principles (the heuristics) to guide their work. The output of each individual evaluator is a list of potential problems and positives, each with one or more representative screenshots.

The evaluation manager aggregates the 3 to 5 individual lists of problems into a larger combined list of problems, and then asks the evalutors to assign severity ratings to each problem in the combined list. Finally, the list is sorted in descending order of average severity, so that the most important problems are at the top of the list. Similarly, a combined list of positives is created and sorted by positivity. A good description of the heuristic evaluation method is given by Jakob Nielsen [Nie1995].

For this evaluation, Nielsen's revised set of ten usability heuristics [Nie1994] were adapted and extended with three more heuristics. These are described in Appendix A.

4 Evaluators and Evaluation Environments

The four evaluators and their evaluation environments are shown in Table 1. The evaluation sessions were recorded with Camtasia [TS2017], which captures the screen, webcam, microphone and any system sounds.

5 Main Positive Findings

The three most positive findings were as follows.

5.1 Easy to Find and Download

It is quite easy to find and download the Enigmail extension if its name is known. All four evaluators were immediately able to find and download Enigmail. Figure 1 illustrates three different ways to download Enigmail.

5.2 It Does What It Should

Enigmail does what it should straightforwardly. In particular, the simplicity of pushing a button to encrypt a message was complimented.

5.3 Better than Previous Versions

One evaluator (VK) had experience with a previous version of Enigmail. She remarked that the tested version 1.8.2 is definitely an improvement, but still needs some work to suit the needs of normal users.

6 List of All Positive Findings

Table 2 lists all of the positive findings from the evaluation, in decreasing order of average positivity (most positive first). The initials of the evaluators indicate who made which finding, and are shown in Table 3. The positivity ratings themselves are described in Table 4.

7 Analysis of the Main Problems

The five most severe problems found during the evaluation are described in detail here, most severe first.

7.1 Manual Mode in Setup Wizard

The most severe problem was discovered in the manual mode for experts, shown in Figure 2, where the setup wizard simply does not work correctly. After going through the setup wizard, when the user attempts to send a first encrypted or signed email, Enigmail gives the feedback "Enigmail is disabled for the selected identity", as shown in Figure 3. Redoing the setup wizard in beginner's mode fixed the problem without going through a whole new key pair generation, but there was no indication that anything was changed or fixed by beginner's mode, as shown in Figure 4.

At the end of GPG installation, a summary window like the one shown in Figure 5 confirms the successful installation. Unfortunately, the summary window is immediately obscured by the newly opened GPG Keychain window shown, as illustrated in Figure 6, so the user is unable to see (at this point in time) that GPG installation was successful. After key pair generation is completed and the GPG Keychain window closes, the summary window becomes visible, but at this point causes unnecessary confusion.

7.2 GPG Keychain

On one occasion, opening a public key received as an email attachment with GPG Keychain (the default) (see Figure 7), only added that public key in GPG Keychain (see Figure 8) and not in Enigmail Key Management (see Figure 9). This issue only occurred once on a MacBook. A screencast of the problem can be seen in Figure 10.

7.3 Unencrypted and/or Unsigned Message

The positive feedback that a message will be encrypted is not particulary prominent and could easily be overlooked, as can be seen in Figure 11. The negative feedback is shown in red but the wording could be mistaken as positive feedback if the prefix "un" is overlooked, as shown in Figure 12. Neither message appears directly adjacent to the button which produces it.

A better formulation would be: "This message will NOT be signed and NOT encrypted." It would be better to put the text next to the buttons anyway to make clear that it is a feedback for each button. The best solution, however, would be to overlay the button icons with either a green tick or a red cross to indicate the current status.

In fact, both the red warning message and the text "Enigmail:" to the far left are clickable, but there is no indication of this.

7.4 Location to Save Revocation Certificate Message Appears Too Late

The information window that a revocation certificate should best be saved on a safe external medium such as a diskette or CD only appears after the revocation certificate has been saved. The information window is shown in Figure 13.

7.5 Public Keys of the Same Email Address are not Distinguished

If a user has more than one public key associated with the same email address, it is not apparent which of the public keys will be attached by pressing the "Attach My Public Key" button in the write an email window, as shown in Figure 14.

8 List of Problems Found

Table 5 shows all the problems found during the evaluation, sorted in descreasing order of average severity. The severity ratings themselves are explained in Table 6. The evaluator(s) who found each problem are indicated by their initials. The evaluators and their corresponding initials are listed in Table 7.

9 Feedback Questionnaire Results

At the end of their session, each evaluator was given a feedback questionnaire, where they rated 12 criteria (questions) on a seven-point scale. The original questionnaire can be found in Appendix B. The seven-point scale was provided in the form of semantic differentials as 3 2 1 0 1 2 3. These were later converted to points from 0 (worst) to 6 (best) for analysis.

A tabular summary of the evaluator ratings is shown in Table 8. Most evaluators did not use the documentation provided by Enigmail, and therefore rated Question 4 neutrally with a "0", corresponding to 3 points in the analysis.

References

[And2017]

Keith Andrews and Ralph Wozelka; EingUsab: Improving the Usability of Enigmail; Visited 2017-01-08. http://projects.iicm.tugraz.at/enigusab/

[Bru2017]

Patrick Brunschwig; Enigmail: A simple interface for OpenPGP email security; Visited 2017-01-08. https://enigmail.net/

[GPG]

The GnuPG Project; GnuPG: The GNU Privacy Guard; Also known as GPG. Visited 2017-01-08. https://gnupg.org/

[Nie1990]

Jakob Nielsen and Rolf Molich; Heuristic Evaluation of User Interfaces; Proc. Conference on Human Factors in Computing Systems (CHI’90). ACM. Seattle, Washington, USA, Apr 1990, pages 249–256. doi:10.1145/97243.97281

[Nie1994]

Jakob Nielsen; Enhancing the Exploratory Power of Usability Heuristics; Proc. Conference on Human Factors in Computing Systems (CHI’94). ACM. Boston, Massachusetts, USA, Apr 1994, pages 152–158. doi:10.1145/191666.191729

[Nie1995]

Jakob Nielsen; 10 Usability Heuristics for User Interface Design; Visited 2014-04-27. https://nngroup.com/articles/ten-usability-heuristics/

[Pad2014]

Arne Padmos; Why is GPG "damn near unusable"?; Visited 2016-04-27. https://media.ccc.de/v/31c3_-_6021_-_en_-_saal_g_-_201412281130_-_why_is_gpg_damn_near_unusable_-_arne_padmos

[Sno2014]

Edward Snowden; Speaking at HOPE X in 2014 with Ellsberg; Visited 2016-04-27. https://youtu.be/onrTo1sYri8

[TS2017]

TechSmith; Camtasia; Visited 2017-01-08. http://camtasia.com/

[TMF2017]

The Mozilla Foundation; Thunderbird; Visited 2017-01-08. https://mozilla.org/thunderbird/

[Whi1999]

Alma Whitten and J.D. Tygar; Why Johnny can't encrypt: a usability evaluation of PGP 5.0; SSYM'99 Proceedings of the 8th conference on USENIX Security Symposium. Volume 8. USENIX Association Berkeley, CA, USA, 1999. Visited 2016-04-25. http://gaudior.net/alma/johnny.pdf

[Woz2016]

Ralph Wozelka, Keith Andrews, Carina Enzinger, Bianca Helbig, and Jacqueline Rinnhofer; (Un)Usability of Enigmail; Talk at Grazer Linuxtage 2016, on Sat 30 Apr 2016. http://keithandrews.com/talks/2016/glt-2016-enigusab/

Appendix A: Heuristics

Based on the feedback from the evaluators, the negative findings (problems) were classified according to the following 13 heuristcs:

• A01 Feedback
  The system should give appropriate feedback within an appropriate time. For example, for a delay of upto about 10 seconds display a busy cursor, for a delay of more than 10 seconds display a progress bar.

• A02 Speak the Users' Language
  The system should speak the users' language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions. Make information appear in a natural and logical order. Match the users' mental model. Beware of misleading metaphors.

• A03 Reversible Actions
  The user should be free to explore the system without penalty. Users sometimes choose functions by mistake and need a clear way to back up. Support undo and redo.

• A04 Consistency
  The same word, phrase, action, or situation should always mean the same thing. Follow platform conventions and standards. If there is a standard way of doing something, do it that way unless there is a very good reason to do it differently.

• A05 Error Prevention
  Prevention is better than cure. Careful design can prevent a problem from occurring in the first place. For example: selecting a file from a menu rather than typing in a file name, asking for confirmation before dangerous actions, avoiding modes (or making it extremely clear which mode the user is in), avoiding commands with similar names.

• A06 Recognition Rather Than Recall
  Place knowledge in the world. Make objects, actions, and options visible. For example, provide examples, default values, and easily retrievable instructions. The user should not have to remember information from one part of the dialogue to another.

• A07 Flexibility and Efficiency of Use
  Provide accelerators for frequent users, unseen by novice users. Allow users to tailor frequent actions. For example: abbreviations, command keys, type-ahead, the ability to edit and reissue previous commands, a menu of most recently used files, macros.

• A08 Aesthetic and Minimalist Design
  “Less is more”. Dialogues should not contain information which is irrelevant or rarely needed. Every piece of unnecessary decoration in a dialogue competes with the relevant units of information for the attention of the user.

• A09 Good Error Messages
  A good error message helps users recognize, diagnose, and recover from errors. Good error messages should be: in plain language (no codes), precise (precisely indicate the problem), defensive (never blame the user), constructive (suggest a solution), and multi-level (include a link to further information or the help system).

• A10 Help and Documentation
  Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Good help and documentation should be: easy to navigate (table of contents, index, and search facility), task-oriented (focused on the user's tasks), recipe-like (list sequences of concrete steps), and example rich (make liberal use of examples).

• A11 Bad Visibility
  The system should make important information as good as possible visible. Important information should be set apart from standard information. New windows should always be in the foreground of the screen.

• A12 Unclear Meaning The system should use distinctive and clearly formulated text and symbols. Postive and negative feedback should be clearly separable.

• A13 Bug
  A usability issue deriving from a software implementation bug.

The heuristics are based on Jakob Nielsen's original ten heuristics [Nie1995]. The final three heuristics were added by the authors.

Appendix B: Evaluator Materials

Each evaluator was given a paper copy of:

• Some background information about Enigmail and cryptography he-domain.pdf.
• Some typical use cases for Enigmail he-use-cases.pdf.
• Consent form consent.html.
• Feedback questionnaire feedback.html.