First run with 3 subjects

We tested 3 subjects, who are studying media informatics at the university and therefore should be a bit experienced about file manager - for our first preliminary - to estimate and check the validity of our study.

It was conducted to avoid errors and mistakes throughout our main study. The study took place in the university and took about 1 hour and a half.

The subjects had no prior experiences in CLI, but have used PCs with windows on it frequently.

First off we asked the subjects about their demographical background and specifically asked about their experience about file manager in general. Afterwards we conducted the main part (user tasks) and as conclusion we asked them about their opinion about the different file manager they just tested and asked them, which one they liked the most (both functionally and esthetically).

The User tasks

We let all three participants complete all tasks (which included a set of 10 Tasks * 3 File manager Types * 3 Operating systems, which makes it 90 Tasks) after another and didn’t count in things such as counter balancing.

What we found out

• The task themselves were rather simple, a joint of different primitive tasks to a more complex task.
• Tasks on the standard explorer and two panel were completed rather fast on every OS compared to the CLI (we gave them the structured folder, so they only needed to move it around)
• The participants told us, that it is getting rather repetitive, since the different OS are not too much of a difference
• The 3 subjects had some problems with the command line, while doing the task, they were kind of lost and didn’t know what to do  It was their first time working with the CLI and they didn’t want to make a mistake, that’s why they were trying to think of a solution themselves instead of just asking
• The main problem for the subjects was not having their usual GUI to work with, upon asking, they revealed, that they kind of lost their navigation point and didn’t like that there is no interface to work with
• The navigation with “cd” and the constant checking of the cheatsheet made dem lose focus of the tasks
• As conclusion, we need to refine our tasks, also we might consider giving the subjects a time scope to get familiar with the commandline in case they do not know how to use it yet

Second test attempt for the main study

Before we actually conducted the second part of our preliminary, we had another Contextual Inquiry with Dennis, a semi expert of using the CLI.

This CI was for us to refine our tasks, especially to get information about the usual task he does when using the CLI. As summary, he is indeed using the CLI a lot, but still prefer using the GUI based one for media consumption (mixed form).

For media consumption (such as photos, videos and music) he is still using the GUI based file manager, meanwhile for programming stuff he is using the CLI to work more efficiently.

He specifically pointed out, that he prefers to work on his keyboard only and doesn’t want to switch to the mouse, since that is interrupting his working flow.

What we found out:

1. Generally there are no tasks, which are not „doable“ on GUI as well as the CLI, but he prefers the CLI when he is working on his programming stuff
2. Main reason for using CLI is not disturbing his workflow by switching to the mouse
3. The amount of time getting used to using the VIM commands can be detterent
4. Generally is highly dependent on the personal preferences

Before our second preliminary, we refined the tasks and now have 8 Tasks we want to conduct. Another subject was being tested, he had a bit of experience with the CLI.

This study went through similar to the first one, with the difference, that the subject could solve the tasks faster and also we decided to reduce the amount of tasks with the different operating systems.

We asked at the beginning of the study which OS he is familiar with (what he is probably using) and let him use that OS for our study.

Tasks

Based of the information we gained from the preliminary study and the contextual inquiries we conducted, we created the tasks that we are going to use in the main Study.

For the tasks we identified various task primitives, which are common tasks, users perform when interacting with file managers. Examples of those are: copy, navigate, delete, create, etc.

We then combined those task primitives to complexer tasks to give a valid use case. We plan to collect logs of the keys the subjects press, te time they took for each task and a recording of their screen during the study.

Participants

For our study we focus on subjects who are familiar with PCs and preferably computer science students. This is important, because we want to gain information about which file manager is best for which task, which would assume the users are capable of performing said tasks to begin with.

Further we learned from the preliminary that our subjects showed various levels of proficiency with the three operating systems (OS) we provided. For the main study we decided to let each participant use the OS he feals most comfortable with.

This means we will split our subcets into three groups. Windows, Linux and Mac. We can then compare the data within the groups for each OS. We would also like to compare the data between the various groups. Therefore we aim to get equally big groups, in order for the data to be comparable.

From ouzr experience, Linux users are much harder to come by than any of the other two groups, so the number of Linux users would dictate the overall number of participants we will test. We aim for at least 8 per group and would preferably test between 10 and 12 per group, if we can find enough participants for that. If we manage to get 10-12 users per group we would deal with 30-36 participants.

Other

Given the fact, that we are going to use the same tasks for every subects, this means each subject has to perform each individual task 3 times, as they would have to do so once, for each file manager we want to test.

Because of that the participants would naturally demonstrate a learning effect and would acustome themselves to the tasks. To counter act that we want to counterbalance the task sequence depending on the final number of participants.

We also observed that our participants showed various levels of experience with the command line. To counter act the knowledge gap in that area, we plan to give a short introduction/revision of how the command line works and what the commands for the OS at hand are.

Every participant, no matter the level of expertise, will get the same introduction for their OS.

Afterwards we will also provide a cheat sheet for all commands needed in the various tasks, to ensure the user can complete all the tasks by themselves, no matter their experience.

Procedure

For the study we rented a room of our university's chair, for a week an plan to conduct most of the study there.

We also want to rent a Mac Book because we dont own one and cant use the one we borrowed for the preliminary, for the whole study.

From what we gathered from the preliminary study, our studies should take anywhere between 40-70 minutes, depending on the participants overall knowledge and experience of file maangers and especially the command line.