Action disabled: index

Writing of the paper (2019-02-19)

Tagged as: blog,paper,ranking,likert,evaluation
Group: D After conducting the user tests (pre-study, main-study), central thoughts and findings were discussed and the writing of the paper was initiated.

While last tests were carried out within the main-study, first thoughts and notes on structuring and scientific preparations were discussed and summarized. After the compilation of the first test results of the main-study, the writing of the paper (considering the predefined template) was officially initiated. We paid special attention to the structuring and distribution of the contents and the highlighting of the delimitation and transition of both studies (in pre-study and main-study). The findings from both studies were repeatedly questioned and discussed before and during the writing process. In the paper, qualitative findings and observations were extensively summarized and quantitative results were evaluated and interpreted. After the evaluation of the studies, we realized that the pre-study dealt with a large number of different, wide-ranging findings, while the main study concentrated on a concrete spectrum of results. The relative magnitude of the results was to be considered, but the pre-study primarily served to weigh up a broad spectrum of possibilities and interests, whereupon we decided not to withhold any results. As important core points of the paper we used a detailed description of both studies, a detailed listing, interpretation and discussion of the results, an outlook in Future Work and core thoughts and topics in Related Work. In Related Work, we wanted to address important areas of interest in rating and ranking. Since the pre-study provided numerous insights (above all for the ranking of items and for final design decisions in the main-study), we wanted to express this on a larger scale. The aim was to address various aspects, such as a precise description of the participants (number, general data, distribution of nationalities, experiences with evaluation systems/ranking, etc.), the study design (within-subject design, presentation of categories, number of runs, counterbalancing/randomisation, subjectivity/objectivity of the objects of investigation, test profiles, personality tests, etc.), and the design of the study.), the setup (technical data on reproducibility, printed and digital slides, etc.), the documentation (pre-questionnaire, Big5), the study process and the results (for pre-questionnaire, ranking winner, acceptance or refutation of hypotheses, etc.). In addition to analyzing qualitative feedback, we also developed the following null hypotheses with respect to temporal measurements to gain knowledge:

  • H0: There is no difference between the number of items and the amount of time.
  • H0: There is no difference between the category of items and the amount of time.
  • H0: There is no difference between the category-sequence and the amount of time.
  • H0: There is no difference between the ranking type and the amount of time.
  • H0: There is no difference between the interaction method and the amount of time.
  • H0: There is no difference between the nationality and the amount of time.

Within the main-study, on the other hand, we addressed similar aspects, such as the objects of investigation (within Fitts' Law tests), the technical and general setup (including a mix of printed and digital media), the use of a pre-questionnaire, the participants and the course of the study. For the writing of the paper, we therefore relied on a detailed investigation of the following hypotheses:

  • H0: The order does not influence the evaluation of the ranking system.
  • H0: The order does not influence the evaluation of the two subjective systems.
  • H0: The evaluation of the systems has no impact on the ranking result among the winners.

Regarding the tips and suggestions that we could gain from the explicitly requested feedback session for the literature review, we planned to refine our scientific writing style to match those needs. Therefore, we e.g. tried to consider consistency and to eliminate superfluous phrases in terms of the scientific character of our paper.