Thinking Aloud Test Report

3.1 Test Methodology

A thinking aloud test is a way of testing the usability of computer applications, websites, or any type of user interface. Test users are asked to perform specific tasks while at the same time expressing their thoughts verbally, as described in Keith Andrews' course notes [And2018]. The screen and test user are recorded with the help of software and a video camera. This way the footage can be reviewed and analysed later on. Through this, it is possible to closely analyze the behavior and the decisions of the test user. The method is very good at finding many usability problems as well as giving examples of what led to these problems. It requires very little materials/resources and is therefore a very cheap and simple to set up. The main disadvantages of the testing method are that users are slowed down by aound 17% when thinking aloud and that thinking aloud sometimes influences their problem solving behavior. For these reasons, measurements of task completion times do not make sense in a thinking aloud test. The thinking aloud testing method is described in further detail by Carol Barnum [Bar2010, Chapter 7].

It is generally recommended to run a thinking aloud test with at least five test users. This test was done with 10 test users (1 pilot test user, 9 regular test users) in order to get a good amount of feedback from the entire procedure. Further discussion about the number of test users and its impact on the test can be found in a paper by Jakob Nielsen [Nie1994]. As Nielsen concluded, 4-5 test users are usually sufficient to achieve good results. Further reading on thinking aloud tests can be found in Janni Nielsen's paper [Nie2002].

3.2 User Profiles

The spectrum of users who could use the d3-hypertree is very diverse, and heavily depends on the dataset used. For instance Michael Glatzhofer's Tree of Life will most likely be used by biology focused people, as most of the tree consists of the latin names of different species.

The test was done with users in the university student age range. This age range was chosen for two reasons. Firstly, this is a demographic which is very relevant for the first users of the d3-hypertree, since that is where it comes from. Secondly, this was the easiest group of test users to recruit. All of the test users either study or have studied at university, but in different fields from law to marketing, and architecture to computer science. This made for a good mix of test users in terms of prior knowledge.

3.3 Test Users

The test users were chosen to represent university students. All test users were aged 18-25 and are either in the process of completing studies or have already completed a university degree.

Table 1 gives an overview of the test users who participated in the study. The mapping of real names to aliases is included in Appendix B.1.

3.4 Test Environment

The test setup for the PC is shown in Figure 1 and for the tablet in Figure 2.

The hardware (devices, browser, internet connection, etc.) and environment used for the test is shown in Table 2.

The exact web browser version used for the tests is shown in Figure 3.

A fresh install of the browser was used for this test. No AdBlockers or other software/add-ons were in use.

3.5 Test Recording

The tests were captured with an external video camera on a tripod in 720p HD resolution. The first tests were done with the help of an external microphone, but this was removed after noticing sound quality problems. Mirrors were used to capture the face of the test user.

The screen recordings were done using Morae Recorder. The whole screen was captured and the user's voice was recorded using an external microphone placed on the desk.

3.6 Training

No special training on the interface was given to the users, in order to test the intuitiveness of the application for users with no prior knowledge of it. The thinking aloud technique was explained verbally to each test user, but was not practised. The test users were simply asked to follow the tasks and solve them one by one, whilst expressing their thoughts verbally in English.

3.7 Tasks

The task list used for the Product Tree is shown in Table 3. The task descriptions given to the users for the Mammals branch of the Tree of Life are shown in Table 4. The actual task slips for both tests are included in Appendix A.5. The test users were able to read the tasks immediately before starting the test and check with the facilitator in order to prevent vocabulary misunderstandings.

The presentation order of device and dataset was varied according to a pre-defined scheme. In this study, the pilot test with TP0 showed that the Tree of Life dataset was very hard to use on a tablet, so an executive decision was made to only test the Product Tree on the tablet for the remaining test users. The final presentation order for each test user is shown in Table 5.

3.8 Interview Questions

The following questions were asked of each user immediately after the final task, once for each test configuration. A summary of remarks made by test users can be found in Section 4.7.

• Opening Question
  • "How was it?"
• Standard Questions
  1. "Did anything strike you as particularly good?"
  2. "Did anything strike you as particularly bad?"
  3. "Do you have any other remarks, good or bad that come to mind?"
  4. "Now after filling out the questionnaire did anything else come to mind you would want to say about the application?"

3.9 Feedback Questionnaire

After the interview, which was done after each test, the user was asked to fill out the feedback questionnaire given in Appendix A.7. The feedback questionnaire was given on paper. An overview of the results can be seen in Section 4.8.

4.1 Discussion and Analysis

With each test, it became more clear what the main negative points to average users are. Almost every test user had the same issues, stemming from a mixture of performance problems and design decisions or missing features. In terms of performance, one of the main issues was the performance on a tablet, which was permanently laggy and would freeze sometimes. Another major problem were crashes of the tree when performing rash movements or zooming.

Design-wise, a pressing problem was the implementation of Wikipedia synchronized to the Tree of Life on tablets, which led to the cancellation of testing of the Tree of Life on tablet altogether. The other main design problems were the lack of overview and search capabilities. All users agreed that the overview provided by the d3-hypertree was mediocre at best. It is very easy to become lost and to not arrive at the desired location even if it has been explored before. Some users proposed an overview map or working with colors to make it easier to disinguish between locations.

Searching for a specific node or subtree without knowing its location in advance is very slow compared to text search. This is why most users proposed a search bar that would be linked to the nodes and would enable users to find nodes by entering a search term. Once a specific node has been found, the d3-hypertree is perfect for browsing neighboring categories and much more intuitive than other visualizations for similar datasets.

4.2 Task Completion

A summary of how many users completed each task within a reasonable timeframe and whether any assistance was given is shown in Table 6 and 7.

4.3 Top Three Positive Findings

The three most positive findings according to their average (mean) positivity ratings are described in more detail below. The positivity rating scheme used to rank positive findings is shown in Table 8.

4.4 List of All Positive Findings

Table 9 shows a list of all the positive findings which emerged from the test, sorted in descreasing order of average (mean) positivity, i.e. the most positive are at the top of the table. The positivity ratings were assigned by the author of this report.

4.5 Top Five Problems

The five most serious problems according to their average (mean) severity ratings are described in more detail below. Problem number 1 is the problem (negative finding) with the highest mean severity. The severity rating scheme used to rank problems is shown in Table 10.

4.6 List of All Problems Found

Table 11 shows a list of all the problems observed in the test, sorted in descreasing order of average (mean) severity, i.e. the most severe are at the top of the table. The severity ratings were assigned by the author of this report.

4.7 Interviews

Nine out of ten users liked the concept of hyperbolic browsing and the d3-hypertree and could imagine using this somewhere else more regularly. Only one user was strongly opposed to the idea to using this in everyday situations and thought it was impracticable. Seven users requested more distinguishability and every single one said more overview is needed. Since there is so much data on the screen, it can become overwhelming for users very quickly.

Four users did say the interface would need some more work in order to be implemented into existing platforms, since it is not that intuitive for a typical computer user. The utility for the Tree of Life was confirmed by all of the test users. Four test users said the Product Tree needed some adaptations in order to be useful for existing product taxonomies. That siad, three users also mentioned it being more neatly arranged, due to its alphabetical ordering and the 360 degree spread of the tree.

4.8 Feedback Questionnaires

Table 12 shows a summary of the ratings given by users in the feedback questionnaire at the end of the Tree of Life test. Table 13 shows a summary of the ratings given by users in the feedback questionnaire at the end of the product tree test. The neutral scale in the original feedback questionnaire has been mapped to a weighted scale between 6 (best) and 0 (worst). The numbers in bold indicate the (rounded) mean rating. The original blank questionnaire can be found in Section A.7. Scans of the questionnaires completed by each user can be found in Section B.2.

A.1 Checklist

The following checklist checklist.html was used for the tests.

A.2 Orientation Script

The following orientation script orient.html was used for the tests.

A.3 Background Questionnaire

The following blank background questionnaire background.html was used in the tests.

A.4 Consent Form

The following blank consent form consent.html was used in the tests.

A.5 Task Slips

The following task slips were presented to the test users:

• tasks-tol.pdf
• tasks-products.pdf

A.6 Data Collection Form

The following data collection form datacoll.pdf was used for the test.

A.7 Feedback Questionnaire

The following blank feedback questionnaires were used in the test:

• feedback-tol.html
• feedback-pt.html

B.1 User Aliases

The mapping of test users' real names to their aliases is documented in the file users.html.

B.2 Completed Background Questionnaires

For each test user, the completed background questionnaire was scanned as PDF:

• background_tp0.pdf
• background_tp1.pdf
• background_tp2.pdf
• background_tp3.pdf
• background_tp4.pdf
• background_tp5.pdf
• background_tp6.pdf
• background_tp7.pdf
• background_tp8.pdf
• background_tp9.pdf

B.3 Signed Consent Forms

For each test user, the signed consent form was scanned as PDF:

• consent_tp0.pdf
• consent_tp1.pdf
• consent_tp2.pdf
• consent_tp3.pdf
• consent_tp4.pdf
• consent_tp5.pdf
• consent_tp6.pdf
• consent_tp7.pdf
• consent_tp8.pdf
• consent_tp9.pdf

B.4 Completed Feedback Questionnaires

For each test user, the completed feedback questionnaire was scanned as PDF:

• feedback_tp0.pdf
• feedback_tp1.pdf
• feedback_tp2.pdf
• feedback_tp3.pdf
• feedback_tp4.pdf
• feedback_tp5.pdf
• feedback_tp6.pdf
• feedback_tp7.pdf
• feedback_tp8.pdf
• feedback_tp9.pdf

B.5 Data Log Forms

For each test user, the log file written by the scribe was scanned as PDF:

• data_tp0.pdf
• data_tp1.pdf
• data_tp2.pdf
• data_tp3.pdf
• data_tp4.pdf
• data_tp5.pdf
• data_tp6.pdf
• data_tp7.pdf
• data_tp8.pdf
• data_tp9.pdf

C.1 External Videos

For each user, the entire test was captured with a tripod-mounted external video camera:

Tree of Life Test:

• TP0 "Kim": tp0-tol-ext.mp4.

• TP1 "Marshall": tp1-tol-ext.mp4.

• TP2 "Teyana": tp2-tol-ext.mp4.

• TP3 "Jermaine": tp3-tol-ext.mp4.

• TP4 "Aubrey": tp4-tol-ext.mp4.

• TP5 "Kendrick": tp5-tol-ext.mp4.

• TP6 "Anthony": tp6-tol-ext.mp4.

• TP7 "Arlissa": tp7-tol-ext.mp4.

• TP8 "Gary": tp8-tol-ext.mp4.

• TP9 "Robyn": tp9-tol-ext.mp4.

Product Tree Test:

• TP0 "Kim": tp0-prod-ext.mp4.

• TP1 "Marshall": tp1-prod-ext.mp4.

• TP2 "Teyana": tp2-prod-ext.mp4.

• TP3 "Jermaine": tp3-prod-ext.mp4.

• TP4 "Aubrey": tp4-prod-ext.mp4.

• TP5 "Kendrick": tp5-prod-ext.mp4.

• TP6 "Anthony": tp6-prod-ext.mp4.

• TP7 "Arlissa": tp7-prod-ext.mp4.

• TP8 "Gary": tp8-prod-ext.mp4.

• TP9 "Robyn": tp9-prod-ext.mp4.

C.2 Session Capture Videos

For each user, the test session on the computer was captured with Morae and then exported in MP4 format:

Tree of Life Test:

• TP0 "Kim": tp0-tol.mp4.

• TP1 "Marshall": tp1-tol.mp4.

• TP2 "Teyana": tp2-tol.mp4.

• TP3 "Jermaine": tp3-tol.mp4.

• TP4 "Aubrey": tp4-tol.mp4.

• TP5 "Kendrick": tp5-tol.mp4.

• TP6 "Anthony": tp6-tol.mp4.

• TP7 "Arlissa": tp7-tol.mp4.

• TP8 "Gary": tp8-tol.mp4.

• TP9 "Robyn": tp9-tol.mp4.

Product Tree Test:

• TP0 "Kim": tp0-prod.mp4.

• TP1 "Marshall": tp1-prod.mp4.

• TP2 "Teyana": tp2-prod.mp4.

• TP3 "Jermaine": tp3-prod.mp4.

• TP4 "Aubrey": tp4-prod.mp4.

• TP5 "Kendrick": tp5-prod.mp4.

• TP6 "Anthony": tp6-prod.mp4.

• TP7 "Arlissa": tp7-prod.mp4.

• TP8 "Gary": tp8-prod.mp4.

• TP9 "Robyn": tp9-prod.mp4.

C.3 Session Capture Video Transcripts

The coding scheme for session capture video transcripts is shown in Table C.1. A chronological protocol of events during the test sessions is given in Tables C.2 through C.21. Times are given in minutes and seconds of elapsed time on the session capture video.