Conduction of main study

Final User Study Design

Before we conducted our main study, we changed some stuff from the preliminary:

1. We changed some questions for the interview
2. We added SEQ (Single ease question) after every single task to get how hard or easy the task was (will be used to compare the learning effect)
3. We adjusted the tasks a bit

The Setup: All of our studies (almost) were conducted in the Usability Lab in the university. We had 3 different notebooks for our subjects to use, depending on which Operating System they are using at home/work. One Laptop with Windows 10, a Linux Laptop with ubuntu and a Macbook with a rather old Mac OSX (10.4). We tested 28 in total with 2 more people missing (will be done this weekend), with a total of 10 Windows user, 10 Mac User and 8(10) Linux User. We counterbalanced the tasksgroups, so we can check if there was a major difference in task completion if people did the “easier” task beforehand and so had the so called “learning effect”. The tasks groups: We prepared the folder structure beforehand for the subjects, each task has its own folder, also we counter balanced the order to avoid the learning effect.

The tasks:

• Navigiere in den Ordner Desktophintergrund, suche die 3 neuesten Bilder heraus, lege einen neuen Ordner „Bilder“ im Unterordner Task 1 an und verschiebe die drei Bilder dort hinein.
• Suche zwei Dateien mit dem selben Namen in den Ordnern „Notizen“ und „Notes“ im Unterordner Task 2, und lösche die ältere Datei der beiden.
• Navigiere in den Ordner Tiere, suche die Katzenbilder heraus und lösche diese. Benenne den Ordner Tiere in „Hunde“ um.
• Erstelle ein Tagebuch: Erstelle für 10 Tage jeweils eine Text Datei und nenne Sie „Tag x “.
• Suche im Unterordner Task 5 nach dem File „findMich.txt“ und kopiere es in den Unterordner Task 5.
• Öffne das Textfile „Löse mich“ berechne das Ergebnis, schließe das File und nenne das File um in das Ergebnis.
• Sortiere die Files nach Filetypen und lösche die Filetypen mit den meistens Einträgen.
• Sortiere die Files nach ihrer Größe, erstelle im Ordner Task8 einen Ordner „Size“ und kopiere die größten drei Files dahin.

Group A: 8 Tasks on Standard Explorer Group B: 8 Tasks on Dual pane Group C: 8 Tasks on command line

Therefore a total of 24 Tasks in total per person

A = Standard Explorer (Finder, Nautilus, Windows Explorer) B = Dual pane (Double commander on all 3) C = Command Line

ABC – ACB – BAC – BCA – CAB – CBA

The “Interview”:

Introduction:

• How old are you?
• What is your working environment?
• How long are you using your PC/Tablet/Smartphone in average a day?
• How would you rate yourself as a multi media user?
• Which OS are you using and why?
• Do you know what a file manager is?
• Which file manager are you using?
• What are your goals, when you use are using a file manager?
• What are the key components of a file manager?
• Are you using any shortcuts?

ATI (Affinity for Technology Interaction)

After User Tasks:

• What was your first impressions of the respective file manager?
• Which one would you like to use?
• What would you improve on the respective FM?
• Could you imagine using a terminal based FM in the future (if yes, why)?
• Which FM is appealing the most esthetically?

For every group of OS (Windows, Linux, Mac) we had different orders, so we can avoid having the learning effect. This will be an important aspect of our analysis.

The Process:

We started the main study with a round about of our topic, so the subject knows what exactly he is doing and why we are conducting our study. Afterwards he got a data privacy statement, since we recorded the screen they are using and for their insurance of privacy. Afterwards we started the interview, asking about some demographical stuff and their multi media knowledge in general. After the introduction the subjects filled in the ATI, a scale which gets a score to the affinity of a person towards technical systems. This score will be used to check if there is a correlation between their ATI Score (basically how or if their technical affinity affects the task completion) and the performance in the user tasks. The user tasks were conducted right after and depending on the knowledge of CLI (we asked beforehand if the person knows the CLI and how experienced he/she is with it), we gave them a short introduction how to use a CLI and explained shortly how the commands are used and which shortcuts are commonly (tab for autocomplete, arrow key to repeat a command…). Depending on the OS and subject, we changed the order of tasks groups (counter balancing) to avoid the learning effect. The duration of one user study last from 50 minutes (fastest) to 2,5 hours (slowest).

As conclusion we asked the subjects some questions about the used file manager and some questions about esthetics (check “After User Tasks”).

What we found out (without doing the evaluation and analysis yet):

• The normal Windows user are usually more used to the User Interfaces and functionality

  1. Linux User are commonly more used to the Terminal than the other 2
  2. The Mac Version was rather old, some people didn’t know exactly what they were doing since it was not the same as their own Mac (their technical affinity in general was not too good)
  3. The tasks were repetitive, we changed some stuff within the tasks (e.g. looking for a certain file, we changed the file of course to avoid the learning effect), but the “core” tasks were identical  We needed the same conditions
  4. Performance related, Linux and Windows people were generally faster than Mac user (rationally seen, without the evaluation and analysis)
  5. The standard Explorer was generally the best one in terms of performance, then the double commander and lastly the commandline
  6. If people were experienced in the CLI, they were really fast with navigating through the tasks, on the other hand if people were not experienced, you could see that they were struggling at the beginning but graduately got used to it and got faster in navigating
  7. Some people were kinda nervous and didn’t want to try stuff themselves and kept asking the test supervisor for help  we did not help them too much, just telling them to try it, only if they really couldn’t solve it we would give them a hint what to do (mostly only on the commandline part)
  8. People had no trouble to find the commands on the cheat sheet, but the usage of them were problematic for some (Usually it goes: Command, affected File and Path where the file should be moved/copy)
  9. Creating folders and moving files into certain folders were easily done by the most, but the most problems people had was to get information from files trough the terminal, most people were used to the graphical UI, so they had trouble to extract information from the terminal
  10. Most of the people didn’t bother to change anything from the default view (e.g. not using the Thumbnail view for selecting certain pictures)
  11. Nummerierter ListenpunktThe double commander was missing some basic commands (mosty on linux and mac, people didn’t know how to create a text file, since on Windows you can right click and easily create a textfile and in general the dual pane wasn’t too well received)