There are a few things we learned from the PreStudy and want to make better in the main study. Following, there is a list of our lessons learned:

• no more videos of the participants, but screencasts with audio. Furthermore we want to get the UX (emotions) with an ingame questionnaire (it´s really difficult and extremely costly to extract emotions through video logs and the results seemed not reliable)
• only one of us as a test adviser instead of two → the study can be conducted at two locations at the same and more participants can be tested
• this time, we´ll tell the participant that the test is about latency
• time per participant was way to high (about 15 minutes live study and about 30 minutes evaluation → improvement: don´t make the study ‘open end’, but use a fixed timeframe
• The permutation of the predeceeding test iteration strongly influenced the performance of the following test iteration→ when permutation values strongly differed, the player was „calibrated“ to the previous latency and got stuck at the first cactus

          → Improvement: prevent "dying" in the game, 
          → Improvement: give the player a short "decalibration time" 
          (e.g. type in a player name or fill in a short questionnaire (zero-latency))

* The game library showed stuttering (library problems or small objects moving fast over the screen → assets moved many pixels per frame), influenced experience of users? → Improvement: Use bigger assets and/or a better screen

• Latency parametrization was not set in respect to the system latency - parametrization A-C seemed to „disappear“ in regard to the overall system latency (When the end-to-end latency is already 70ms, adding 10ms is maybe to little)

        → Improvement: Use higher latency values, especially one, which is ridiculously high 
        --> maybe compare to real-world network latency values? (e.g. in online games)
        → Improvement: Make system latency test before starting the main study, in order to allow adaptions, when needed.

As in the PreStudy, first some demographic data will be collected. This time, the influence of alcohol is not important for our study and our results, but still the age. Furthermore, we want to know the experience in playing computer games of the participant. This will happen without a questionnaire: the test adviser asks them and takes notes.

The data we want to collect:

• age
• nickname for the high score
• computer gaming experience (means „Spielerfahrung“ and not „Erfahrung während des Spielens“)

     How we want to measure the experience of the player?
     This is the question we developed: 

     How often do you play computer games normally?
      * daily
      * several times a week
      * once a week
      * less than once a week
      * never

     (This is only translated, the testing will be in German)

Instead of capturing the emotions and the user experience of the participants with videos, we want to collect this data with an ingame questionnaire which has to be filled in after every permutation. To find the most adequate questionnaire for our study, we talked to Florian Bockes. These are the results:

• GEQ?
  • The Games Experience Questionnaire is pretty wide.
  • But there´s is a short module (ingame, which we want) with 14 items, still there are questions with are completely irrelevant for our study (for example „I was interested in the games` story).
  • Furthermore, the GEQ is in English and needs to be translated.
  • –> GEQ is not the best choice , especially because we want to learn more about the emotions and the user experience than about the game experience.

• UEQ?

Could be User Experience Questionnaire an option?This one is too wide as well and contains too many questions which are irrelevant for our study and the questionnaire should be filled in quickly

• Developing an own questionnaire?
  • Then we were thinking about creating our own questionnaire by picking only 2 or 3 questions of those questionnaires. But with this “quick and dirty method”, mistakes can happen very easily.

• SAM

Finally, we chose our questionnaire: the self-assessment manikin (SAM), which looks like this:

After the study we want to learn more about the emotions and the user experience, so we will conduct an open interview with the participants.

In the PreStudy we learned that measuring the high score is very motivating for the participants. For this reason, we want to give them an incentive: every test person should get candy for the participation and the highest 3 high scores should get monetary prices. Since cash gifts are difficult, we want to give them coupons (With values of 15,10 and 5 Euros). Maybe from the Arcaden.

As in the PreStudy, the MainStudy will be conducted with a little game. This time, it will be an Asteroids game. A detailed description of the game can be found in an own blog entry.

As for the PreStudy, we want to conduct a study before the real study (PreMainStudy) for testing parameters, chose the most adequate latencies and review the duration.

Bradley and Lang (1994): Measuring emotion The self-assessment manikin and the semantic differential.

Drachen et al. (2018): Games User Research.