Thinking Aloud Test Plan

1 Introduction

The Parallel Coordinates Explorable Explainer (PCEE) is an interactive tutorial about the parallel coordinates visualisation technique for multidimensional datasets. It is designed for newcomers to the topic to learn the basics about parallel coordinates and become familiar with this kind of graphical chart. PCEE is an open-source project [And2025a]. A live version of the tool is hosted at [And2025b].

The web application is available only in English and will be tested in English with English-speaking test users. The evaluation will cover the entire PCEE tool. A separate branch will be created for the thinking aloud tests, so as to preserve the exact version used.

2 Test Methodology

A Thinking Aloud (TA) test is a usability testing method that asks test users to talk out loud as they work, or as Jacob Nielsen defines it: “In a thinking aloud test, you ask test participants to use the system while continuously thinking aloud — that is, simply verbalising their thoughts as they move through the user interface” [Nie2012]. Test users should perform their tests independently of one another. The thinking aloud method is described in detail by Barnum [Bar2020].

Before starting the tasks, each user will be asked to fill out a consent form. Any questions about data collection will be answered, and then the user's background data will be collected by the facilitator in the form of a background questionnaire. Each test user will be shown a short video on how to think aloud, and will then practice thinking out loud with one or two training tasks.

For the first task, the user will be asked to work through the interactive tutorial provided by the PCEE web application. Afterwards, five smaller, more specific tasks will be given, to cover workflows which may be used less frequently. Once the tasks have been completed, a short interview will be conducted and the test user will be asked to fill out a feedback questionnaire.

After all five test users have completed their tests, the faciliator's notes and session recordings will be analysed and positive findings and negative findings (problems) will be extracted with corresponding illustrative video clips. Then, the findings will be aggregated into separate lists of positive and negative findings. Finally, ratings for positivity and severity will be assigned by members of the project team, and the lists will be sorted in decreasing order of positivity and severity respectively.

3 User Profiles

Two user profiles were identified for PCEE:

• Students in an InfoVis class: PCEE can be used as lecture material and as an interactive tool during class to demonstrate the concept of parallel coordinates, which can be revisited by students at home.

• Independent Study: PCEE can be accessed online, so it is available to a wide audience for independent study.

4 Test Users

Table 1 gives an overview of the test users planned to participate in the study.

5 Test Environment

All users will use the same test environment for the thinking aloud test. The planned environment for the tests is shown in Table 2.

For this test, no adblocker will be used. The PCEE web application does not require any cookies. The browser data will be cleared before each test to provide a clean environment for each user.

6 Recording

For external video recording, a Sanyo Xacti HD1010 video camera will be mounted on a tripod and used to film over the user's right shoulder. A mirror will be placed to the left of the laptop screen. The camera's field of view will be set up to capture the screen, keyboard, and user's facial expressions in the mirror. A DeLOCK 66640 table microphone will be used to capture audio for the external recording.

For session capture, OBS Studio 32.0.4 will be used to record the screen, audio, and the user's face. The laptop's internal webcam will provide audio and video of the user's face, with the facecam video window placed in the bottom left corner of the screen.

Video clips will be extracted from the session recordings to illustrate findings for the report using Kdenlive 24.12.3. The same software will be used to pixellate the faces of the test users and to transcode the video clips where necessary.

7 Training

First, a small video clip of how to conduct a thinking out loud test will be shown. To practice thinking out loud, users will then be asked to perform one or two tasks on YouTube (youtube.com). The first training task will be to find a video of dog food. The second training task will be to find a German rap song on Youtube Music.

8 Test Tasks

Table 3 shows the tasks which will be used to test PCEE.

9 Interview Questions

The following questions will be asked of each user immediately after the final task.

• Opening Question
  • “How was it?”
• Standard Questions
  1. “Did anything strike you as particularly good?”
  2. “Did anything strike you as particularly bad?”
  3. “Do you now know what parallel coordinates are?”
• Individual Questions

  If any questions come up during the test with each test user, the user will be asked those questions at this point.

10 Data Collection and Collation

The consent form which test users will be asked to read and sign can be found in Section A.4 Consent Form. For the background questionnaire, found in Section A.3 Background Questionnaire, the facilitator will ask the questions of the user and fill out the form, rather than handing the form to the user to fill out. Any questions regarding data collection will be answered before the start of the recording. Once all tasks are finished, the test user will be handed a feedback questionnaire to fill out, which can be found in Section A.6 Feedback Questionnaire.

In any retained data, all test users will be referred to by a first name alias and a test user id. All personally indentifiable data will be deleted within one year.

References

[And2025a]

Keith Andrews; Parallel Coordinates Explorable Explainer (PCEE); visited 07 Dec 2025 https://github.com/tugraz-isds/pcee

[And2025b]

Keith Andrews; Parallel Coordinates: An Explorable Explainer; visited 07 Dec 2025 https://tugraz-isds.github.io/pcee/

[Bar2020]

Carol M. Barnum; Usability Testing Essentials; 2^nd Edition, Morgan Kaufmann, 2020. ISBN 0128169427.

[Nie2012]

Jakob Nielsen; Thinking Aloud: The #1 Usability Tool; 15 Jan 2012. https://nngroup.com/articles/thinking-aloud-the-1-usability-tool/

A Test Team Materials