07 - Documentation for the qualitative analysis of the collected chat-logs (2020-12-28)

Tagged as: blog, interaction, chat, chat-log, qualitative analysis, analysis, documentation
Group: J_20/21 The rough documentation for our qualitative analysis of the previously collected chat-logs with grounded theory.

Strategy for categorizing some labels

  • For links, we tried to look at which website was linked in the chat and then assigned a label depending on the context of the site
  • We also grouped features that were unique to Twitch together (such as chat commands, chat bots, automatic messages by the streamer/instructor)
  • Emojis and Emoticons were also grouped together and counted as their own label since they both symbolize things and Emojis are an essential part of Twitch (unlocking your own Emojis is a reward on Twitch)
  • We only mentioned the appearance of Emojis & Emoticons in the coding and we plan to look at them separately at the end.

After first round (Vorlesung Software Engineering, 81 messages)

  • The first round of open coding resulted in 43 codes
  • 14 categories were then assigned for our first axial coding
  • Interesting things about this chat-log specifically were found
  • There were some watchers that were not necessarily students that watched the stream and they found it interesting
  • Some of them mentioned that they wished that their own professors or instructors streamed or had a better access to online lectures
  • The stream was interrupted by the son of the instructor who wanted attention, the students did not mind and even commented on it being „wholesome“ and they made some jokes about it

Second round (lectures with very few chat messages)

  • We then looked at the streams that only had a few messages (less than ten) and tried to see which of our labels we could apply to them
  • It should probably be noted that these messages are very interesting since they are some of the only messages across the entire duration of the stream
  • These lectures were: Kombinatorische Strukturen (2 messages) and TUB Fachgebiet IIT (8 messages) in that order:
    • Kombinatorische Strukturen: A student made a joke about a drawing presented as the only message, which aligns with a previous code
    • TUB Fachgebiet IIT: Students were mainly giving examples for a question asked by the instructor. There was also spam, possibly posted by a bot, that was ignored.

After second round (Kombinatorische Strukturen, TUB Fachgebiet IIT)

  • 5 additional codes were found with open coding
  • 3 new categories were found for axial coding
  • In two of those lectures, students criticised the instructor or lecturer, unlike the first lecture we looked at which was also longer than those lectures and had a higher student engagement.

    Note: One of the streams included in this round was removed since it was streamed on YouTube and not Twitch and our research only focuses on chat-logs from Twitch. The stream contained only one message and therefore did not influence the results much. Because of this, one open and axial code less were found during this round, although the axial code appeared later for a different lecture. he mistake was corrected in the relevant files and the paper.

    Third round (lecture with the most chat messages)

  • Next we looked at the lecture Allgemeine Krankheitslehre (354 messages), which had the most chat messages out of our dataset
  • It should be noted, that in this lecture some students started arguing a bit until the student who used crude language was timed out or banned
  • The lecture, unlike the one form the first round, was not recieved as positively and it was asked if they could switch to Zoom, probably because the chat felt distracting to some (off-topic talk, jokes, the argument in the beginning)
  • In some cases students split their messages instead of posting just one, compared to the other lectures up until now (maybe influenced by the higher amount of messages?)
  • Users paulee and Mister_Lyon most likely did not pay attention to the lesson and had a lot of non-relevant discussions (probably to annoy others or to entertain themselves during the lecture)

After third round (Allgemeine Krankheitslehre, 354 messages)

  • We added 30 new codes that were found by open coding
  • Only 2 new axial codes were found among the new codes from open coding, the rest was assigned to previous codes that fit (this suggests that our current list of axial codes is fairly conclusive)
  • The by far highest amount of messages were examples given by students to answer questions from the instructor (106 messages)
  • In this lecture we also found messages that were unclear in their meaning and we put these messages into a separate category that we did not assign an axial code for.

Fourth round (the two remaining smaller lectures, both had the instructor writing messages in the chat themselves)

  • We then looked at the two remaining smaller lectures, Grundlagen der Informatik (42 messages) and Vorlesung Media Engineering (4)(54 messages):
    • Grundlagen der Informatik: The most interesting aspect in this lecture is that the instructor himself was active in the chat and posted several messages
    • Vorlesung Media Engineering (4): In this lecture the instructor was also interacting with students in the chat, mainly to answer a student that wanted to recruit the instructor for an event as a moderator

After fourth round (Grundlagen der Informatik, Vorlesung Media Engineering(4))

  • 16 new codes were added, 6 for Grundlagen der Informatik, 10 for Vorlesung Media Engineering
  • No new axial codes needed to be added, the other codes were assigned to the previously created axial codes
  • Emoticons were used fairly often during these lectures (in 21 out of 96 messages)

Fifth and final round (Vorlesung Rechnerarchitektur, 189 messages)

  • We looked at the lecture Vorlesung Rechnerarchitektur last for any additional open codes
  • No (or at the most one) additional axial code was expected to emerge from this final open coding

After fifth round (Vorlesung Rechnerarchitektur, 189 messages)

  • 22 new codes were added in total, which lead to a final number of 117 codes for open coding
  • There were a lot of questions and answers in the chat that were answered or that other students at least attempted to answer (tutors or very knowledgable students were present in chat)
  • At the beginning of the lecture, an event hosted by the university called inDay students was promoted, which led to a larger amount of off-topic messages

General Findings for axial codes and across lectures

  • General small talk such as greetings, partings, thanking or jokes and reactions to them were the most present across all lectures with 154 messages (greetings and partings as well as jokes or words of thanks were all present in more than half of the lectures)
  • This is directly followed by answers from students for the instructor with 156 messages, although this number is heavily influenced by the lecture with the most messages (Allgemeine Krankheitslehre). A lot of students in that lecture gave examples following a question by the instructor (106 messages) or they simply typed an affirmate response (such as yes, okay) to a question asked by the instructor.
  • The use of emojis, emoticons and emotes was also high with 92 messages. They were also used in more than half the lectures (except the three with less than ten messages) and students frequently combined these with a message or they posted them just as is.
  • Although messages about the audio of the instuctor were posted 32 times, these messages still appeard in half of the lectures because of several different problems with the audio (hearing an echo, hearing background noises, microphone was accidentally left on, instructor can't be heard clearly)

The final overall results and the results for selective coding will be published in a separate blog post that focuses on these results, while this post focuses on the documentation. This post outlined every step in our qualitative analysis, our decisions, the order in which we looked at the lectures and what our thoughts were during the coding process. Some additional information can be found in the files we used for our qualitative analysis.