Action disabled: revisions

09 - Start of the interview period for the questionnaire and questionnaire details (2021-02-03)

Tagged as: blog, questionnaire, interaction
Group: J_20/21 This post outlines general information about the questionnaire (administration period, content).

After we finished the qualitative analysis of the chat-logs, we gathered an insight into what students write in the chat with the resulting categories. We used these categories as a framework to build a questionnaire to gain further insight into how students and instructors interact in Twitch courses (a course with only one feedback channel).

Creating the first draft of the questionnaire

We first used the categories from our qualitative analysis to create questions that we want to ask students or instructors that have attended at least one university or college course that was held on Twitch. The following categories were used in the questionnaire:

  • Features that are unique to a live chat or Twitch
  • Discussions that are off-topic or that do not address the content or information from the live stream
  • Unwanted behavior or unwanted events in live chats
  • External audience
  • Anonymity for chat messages
  • Discussions about the content of the live stream between students or with the instructor with the goal of providing information
  • Discussions about streaming itself
  • Evaluating the quality of the course and the instructor

We also added the Classroom Community Scale to the questionnaire for students to see if there is a sense of unity or community in online courses, since creating a sense of community was one aspect that appeared relevant in the qualitative analysis.

Creating the second draft of the questionnaire

After the first draft was created, we got feedback and changed the questions accordingly. We then used https://www.soscisurvey.de to host our online survey. This enabled us to split the questionnaire depending on who the study participant is (a student or an instructor). The questions for the instructors are based on the questions for the students, but there are some questions that were added for instructors, especially some free text fields. We also added an introduction page that presented information about the study and we informed the participants how we are processing their data (anonymously and confidentially) and that they give informed consent that we could use their data for the purpose of their study. We also noted that students of media informatics or information science are able to gain credits for participation and that we gather their e-mail addresses separately.

We also added the option to change the language at the beginning and during the survey. Participants are able to view the survey in either German or English.

Pretest of the questionnaire

We conducted a pretest for our questionnaire before the questionnaire went live. 4 participants were recruited for the pretest. The first participant noted several inconsistencies or improvements to the structure of the questionnaire. The following participants commented mostly on the wording of sentences and on small mistakes.

The biggest change because of the pretest comments is the addition of demographic questions, information about behavior on Twitch, information about the visited courses and the addition of several free text questions. We also added an ending page to the questionnaire so that participants know that they can go back and look at their answers one more time before they finish. A free text field for questions or comments about the study was also added on that page.

One aspect that was also being discussed, was if we want to evaluate a single course or all courses. We decided to use the experience from all courses, since we expect a larger amount of students and instructors to take part in our study with this method. But we still used the evaluation of one course for the Classroom Community Scale, since it is not suited for the evaluation of multiple courses at the same time. We therefore decided to ask the students that they answer the Classroom Community Scale for the favorite course they have visited so far.

We also asked two media informatics instructors that were currently in charge of Twitch courses if they had any additional questions, since their input was especially interesting. From their feedback, we were able to include additional questions in the survey.

Administration period and recruitment of participants

The study went live on 01.02.2021 and will end on 14.02.2021. We posted the study in the media informatics and information science forum on the e-learning platform GRIPS of the University of Regensburg. We also contacted the instructors that were currently in charge of Twitch courses and asked them to promote the study in their Twitch courses, since the students in these courses are all potential participants for our study. We are looking into additional methods of recruitment (Reddit, asking instructors that are streaming on Twitch, contacting other instructors that are using Twitch at the University of Regensburg). At this point in time, 40 participants have already participated in our study, although they are all either media informatics or information science students.