Thinking Aloud Test Report

3.1 Test Methodology

A Thinking Aloud (TA) test is a usability testing method that depends on the verbalisation of the thoughts of test users, or as Jacob Nielsen defines it: “In a thinking aloud test, you ask test participants to use the system while continuously thinking aloud — that is, simply verbalising their thoughts as they move through the user interface” [Nie2012]. Test users generally perform their tests independently of one another.

There are two ways of conducting the thinking out loud. In Concurrent Thinking Aloud (CTA), test users are encouraged to speak out loud while working on the povided tasks, while in Retrospective Thinking Aloud (RTA) test users are asked about their thoughts and experiences after the tasks. Research has shown the CTA method to be more effective [Als2015], and CTA was used in this study.

For this study, a short training in thinking aloud was provided before the start of each test, so test users could practice verbalising their thoughts. Changes in speech rate, volume, and pitch can indicate usability problems, in addition to negative remarks. The thinking aloud method is described in detail by Barnum [Bar2020].

Before starting the tasks, each test user was asked to fill out a consent form. Any questions about personal data collection were answered, and then the user's background data was collected by the facilitator in the form of a background questionnaire. Each test user was shown a short video on how to think aloud, and then thinking out loud was practiced with one or two training tasks.

For the first task, the users were asked to work through the interactive tutorial provided by the PCEE web application. Afterwards, five smaller more specific tasks were given, to cover workflows which are typically used less frequently. Once the tasks were completed, a short interview was conducted and the test users were asked to fill out a feedback questionnaire.

After all five test users had completed their tests, the faciliator's notes and session recordings were analysed and positive findings and negative findings (problems) were extracted with corresponding illustrative video clips. Then, the findings were aggregated into separate lists of positive and negative findings. Finally, ratings for positivity and severity were assigned by members of the project team, and the lists were sorted in decreasing order of positivity and severity, respectively.

3.2 User Profiles

Two user profiles were identified for PCEE:

• Students in an InfoVis class: PCEE can be used as lecture material and as an interactive tool during class to demonstrate the concept of parallel coordinates, which can be revisited by students at home.

• Independent Study: PCEE can be accessed online, so it is available to a wide audience for “independent study”.

The goals of both user groups are identical. Users of PCEE want to learn about parallel coordinates by working through an interactive tutorial. All tested users in this study fell into the category of “independent study”.

3.3 Test Users

Test users were selected with the minimum requirements of being fluent in English, having a basic understanding of maths, wanting to learn new tools or concepts, and having no prior knowledge about parallel coordinates. These criteria were confirmed in a short screening check a week before the tests.

Table 1 gives an overview of the test users who participated in the study. First name aliases are used throughout this report to protect their identity.

3.4 Test Environment

The test room is shown in Figure 1. The environment used for the thinking aloud tests is shown in Table 2. Test users worked on a laptop with its integrated touchpad, no mouse was provided.

The exact web browser version used for the tests is shown in Figure 2. During the tests, no adblocker was used and cookie selection was left to the user. This was only relevant for the training, because the web application does not store any cookies.

To record the external videos, a Sanyo Xacti HD1010 camcorder was mounted on a tripod and connected to a DeLOCK 66640 Stereo Table Microphone. In order to see the facial expressions of the test user in the external video recordings, a mirror was placed to the left of the laptop screen and the camera was positioned to record over the user's right shoulder.

For the session capture video, the integrated webcam and microphone of the laptop were used to overlay the user's face in the bottom left corner of the video.

3.5 Training

To practise thinking aloud, the user was first asked to navigate to YouTube (youtube.com) and search for a video of dog food. After finding a video, the user was asked to navigate to Youtube Music (music.youtube.com) and search for a German Rap song, all the time while talking aloud.

3.6 Tasks

The internal task list used by the project team is shown in Table 3.

3.7 Interview Questions

Each user was interviewed immediately after the final task using the interview script given in Section A.6. Summaries of the interviews can be found in Section 4.6.

3.8 Feedback Questionnaire

After the interview, the user was asked to fill out the feedback questionnaire given in Section A.7. The feedback questionnaire was given on paper. An overview of the results can be seen in Section 4.7.

3.9 Data Collection

Before starting the tests, the test users were given a short introduction. Then, they were given a consent form to read and sign, which can be found in Section A.3. For the background questionnaire, found in Section A.4, the facilitator asked the questions of the user and filled out the form, rather than handing the form to the user to fill out. Any questions regarding data collection were answered before the start of the recording. Once all tasks were finished, the test user was handed a feedback questionnaire to fill out, which can be found in Section A.7.

All video clips contained in this report have pixellated faces to protect the privacy of the test users. The pixellation was performed with Kdenlive. All test users and their data are identified with a first name alias or a unique test person id.

The mapping of user aliases, the signed consent forms, and the full session videos contain personally identifiable data of the test users. These files are not part of this report, are kept separate, and will be deleted within one year.

4.1 Task Completion

A summary of which users completed each task and whether any assistance was given is shown in Table 5.

4.2 Top Three Positive Findings

The three most positive findings according to their average (mean) positivity ratings are described in more detail below. The positivity rating scheme used to rank positive findings is shown in Table 6.

N01. Inserting 0 into Table Not Possible

Three test users discovered that the value 0 cannot be inserted into the chart by typing it into the table in Section 2, because the Redraw chart button becomes disabled, as can be seen in Figure 6.

N02. Zooming into Chart is Not Possible

Selecting a single polyline, especially on a smaller screen, becomes more difficult as the number of polylines increases, as can be seen in Figure 7. This can be solved by introducing a zoom function.

N03. Context Menu Poorly Described

Figuring out what and where the context menu is can be hard for a new user, as can be seen in Figure 8. This can be solved by adding an explanation.

N04. Polyline Selection Difficult

Selecting one or more polylines can be hard as the area that is clickable is very small, as can be seen in Figure 9. This could be solved by increasing the size a polyline's hit surface.

N05. Case Study Interactivity Poorly Described

The user is not informed what actions they should do during the case study, as can be seen in Figure 10. This could be solved by adding a short how-to part in the case study or nudging the user with well-known icons and smaller info texts.

A.1 Checklist

The following checklist was used for the tests: checklist.html.

A.2 Orientation Script

The following orientation script was used for the tests: orient.html.

A.3 Consent Form

The following blank consent form was used in the tests: consent-ta.pdf.

A.4 Background Questionnaire

The following blank background questionnaire was used in the tests: background.html.

A.5 Task Slips

The following task slips were presented to the test users: external-tasks-ta.pdf.

A.6 Interview Script

The following interview script was used in the tests: interview.html.

A.7 Feedback Questionnaire

The following blank feedback questionnaire was used in the tests: feedback.html.

B.1 Completed Background Questionnaires

For each test user, the completed background questionnaire was scanned as PDF:

• tp1-bq.pdf
• tp2-bq.pdf
• tp3-bq.pdf
• tp4-bq.pdf
• tp5-bq.pdf

B.2 Completed Feedback Questionnaires

For each test user, the completed feedback questionnaire was scanned as PDF:

• tp1-fq.pdf
• tp2-fq.pdf
• tp3-fq.pdf
• tp4-fq.pdf
• tp5-fq.pdf

C.1 User Aliases

The mapping of test users' aliases to their real names is documented in the file users.html.

C.2 Signed Consent Forms

For each test user, the signed consent form was scanned as PDF:

• tp1-consent.pdf
• tp2-consent.pdf
• tp3-consent.pdf
• tp4-consent.pdf
• tp5-consent.pdf

C.3 External Videos

For each test user, the entire test was captured with a tripod-mounted external video camera in the following files:

• TP1 “Peter”: tp1-ext.mp4
• TP2 “Bob”: tp2-ext.mp4
• TP3 “Julian”: tp3-ext.mp4
• TP4 “James”: tp4-ext.mp4
• TP5 “Robert”: tp5-ext.mp4

C.4 Session Capture Videos

For each test user, the entire test session on the computer was captured in the following files:

• TP1 “Peter”: tp1.mp4
• TP2 “Bob”: tp2.mp4
• TP3 “Julian”: tp3.mp4
• TP4 “James”: tp4.mp4
• TP5 “Robert”: tp5.mp4