Analysis and Discussion of the Literature Research Results (2019-03-07)

Tagged as: blog, results, ar, taxonomies
Group: G This blog article will show you the most important results, limitations and conclusions of our research.

Last week we finished the analysis of our literature research results and handed in our paper. This means that we finished all five phases of a literature research according to Brocke et al. (2009). The last phase contains the interpretation of our results,identification of lesson learned as well as possible, future activities (Brocke et al., 2009).

This blog article will show you the most important results, limitations and conclusions of our research. We will only give a short overview about the main aspect in these categories. Further information can be retrieved from our paper.

Topic of Publications

Next to the general results, that we already presented in the last blog, we additionally analyzed the main topics of publications. Therefore, approaches from Dey et al. (2016), Billinghurst et al. (2015), Carmigniani et al. (2011), Papagiannakis et al. (2008) and Van Krevelen and Poelman (2010) were used and slightly adapted (cf. table 1). We added the topics Nature for nature-related publications, Application Development for presentations of supporting tools for the development of AR applications, and System and Technology for publications that presented technological algorithms and models. Most publications were assigned to the topic System and Technology (49, 36.3%), followed by Education and Application Development (14, 10.37%) and Health & Medicine (13, 9.63%).

Table 1: Publication Topics adapted from Dey et al. (2016), Billinghurst et al. (2015), Carmigniani et al. (2011), Papagiannakis et al. (2008) and Van Krevelen and Poelman (2010)

Results of Technological Analysis

Classes of AR Displays

Based on our results we added the subcategory Multiple to our technological taxonomy for combinations of multiple display types in one application. Table 2 shows the distribution of classes of AR displays. With 127 (70.17%) of all displays the visual display was the preferred type. Visual displays still represent the most popular display type among AR publications. Olfactory and gustatory displays were not used in any of the analyzed publications. 13.26% of all display usages were combinations of multiple displays or multimodal ones, if different senses were involved (Schmalstieg and Höllerer, 2016).

Table 2: Distribution of AR Displays based on our Taxonomy

Following combinations of displays occurred:

  • monitor-based & optical see-through: 4 (2.21%)
  • monitor-based & monitor-based: 3 (1.66%)
  • monitor-based & video see-through: 2 (1.10%)
  • video see-through & optical see through: 1 (0.55%)
  • spatial & optical see-through: 1 (0.55%)
  • spatial & monitor-based: 1 (0.55%)

Following multimodal displays occurred:

  • video see-through & aural: 3 (1.66%)
  • monitor-based & aural & haptic: 2 (1.10%)
  • optical see-through & aural: 2 (1.10%)
  • spatial & haptic: 1 (0.55%)
  • optical see-through & haptic: 1 (0.55%)
  • closed-view & aural: 1 (0.55%)
  • monitor-based & aural: 1 (0.55%)
  • optical see-through & haptic & monitor: 1 (0.55%)

AR Display Postioning

Head-mounted (44, 33.08%) and handheld (41, 31.3%) displays were the most preferred display types. Figure 2 shows an increase of head-mounted displays since 2015 and a decrease of handheld displays from 2016 to 2017. The least preferred displays were body-attached ones with 3 (2.26%) and near-eye displays with no occurrences.

Figure 2: AR Display Positioning based on our Taxonomy

AR Tracking Methods and Environment

The technological taxonomy was expanded by the sensor types EMG (Electromyography) and EEG (Electroencephalogram) based on our results. Table 3 shows the distribution of tracking methods. Optical tracking is the most preferred method of all tracking systems with 44 (23.40%) occurrences for 3D structured light tracking and 30 (15.96%) occurrences for outside-looking-in tracking in our analysis. GPS is the preferred mobile tracking system (5.32%). Stationary displays including mechanical, electromagnetic and ultrasonic displays were never used in the analyzed publications. Regarding the environment of usage, most applications are utilized indoors (103, 76.29%), in contrast to outdoor environments (10, 7.41%) or both environments (17, 12.59%).

Table 3: Distribution of Tracking Methods based on our Taxonomy


Based on our results the technological taxonomy was adapted by adding Desktop Computer UI, subdivided in Mouse and Keyboard to the class 2D. The class 3D UI was expanded by the subcategories Presenter, Controller, Handle, Stylus and Handheld- Clicker. Hand Motions and Face Recognition were added to Natural UI. Furthermore the category Combinations was divided into the subcategories Multi-View UI and Multiple Interfaces consisting of the subcategories Multiple Classes and Same Class. Finally, the class Adaptive was expanded by the subcategory EMG (Electromyography) Interface.

Figure 3: Types of AR User Interfaces (aggregated from 2015 - 2017) based on our Taxonomy

Touch-based (25, 18.52%) and body-motionbased input modalities (24, 17.78%) were the most preferred ones (see Figure 3). Tangible User Interfaces represented only 10.37% of all displays. Augmented paper and Multi-view UI occurred in no publication.

24 (17.78%) of all interfaces represented combinations of multiple types at once. Due to issues in defining the terms Hybrid UI and Multimodal UI, we decided to replace both with the class Multiple Interfaces, subdivided into Multiple classes and Same Class based on our results. The subcategory Multiple Classes contains interfaces from at least two of the eight main interface classes described in section 3 (10), whereas the subcategory Same class contains interfaces from a single class (7). The combination of touch and body motion (7, 5.18%) occurred most, whereas other combinations occurred once or twice in the analysis.

Results of Evaluation Structure

General Results

In general, 152 evaluations were conducted in 135 publications, whereof most publications (101, 78.81%) included one evaluation and 22 (16.3%) publications two to three evaluations. Regarding the focus, 83 publications conducted solely user-based evaluations, 36 publications solely system-based ones and 3 publications included both evaluation types. 61,48% of the AR related paper conducted a user-based evaluation. In most publications evaluations were conducted in laboratory environments (96, 63.16%), in contrast to final usage contexts (field study, 18, 11.84%). 2 (1.32%) publications conducted evaluations in both environments. Preliminary and pilot studies were included in 26 (19.3%) publications and 51 publications included within-subject experiment design, and 17 between-subject design.


85 (69.11%) publications conducted an evaluation involving participants, 37 (29.84%) publications conducted an evaluation without participants and in two (1.61%) publications evaluations were conducted with and without participants. In total, 4567 participants were involved in all evaluations. The amount of participants was mentioned in 83 (95.40%) publications. The values range from two to 1000 participants per evaluation. Five publications conducted evaluations with more than 100 participants (107, 169 [46], 683, 865 and 1000). The percentage of the three highest amounts of participants involved approximately 55.79% of all participants.

The average age of participants was mentioned by 54 evaluations (47.06% of participant involving publications).74% of the participants in our analysis had a mean age of less than 30 years. We split the average age into the following ranges:

  • 0 - 9 years: 3.7%
  • 10 - 19 years: 9.26%
  • 20- 29 years: 61, 11%
  • 30 - 39 years: 16,67%%
  • 40 - 59 years: 5.56%
  • >= 60 years: 3.7%

    The gender distribution was mentioned by 54 (62.07%) publications. In total, 1516 (48.11%) participants were male and 1630 (51.73%) were female (c.f. table 4). The gender Other was mentioned in three publications with a total of five (0.16%) participants. For this reason the gender Other was added to the taxonomy.

Table 4: Percentage of Gender

Regarding the participants’ profession, 52.08% came from an university context, 16.67% from the general public and 31.25% from other sources. In 30 evaluations the profession was not explicitly mentioned (38.46%). 40% of evaluations did not mention the profession of participants.

Results of Evaluation Areas

We classified evaluations after the areas from Swan II and Gabbard (2005), Dünser and Billinghurst (2005) and Dünser et al. (2008). An evaluation can be represented by one or more areas. As we additionally analyzed system-based publications, we added the category System Technology to the exclusively user-based approach from Swan II and Gabbard (2005) and Dünser et al. (2008). Table 5 shows the adapted evaluation areas with references, examples from the publication analysis and the percentage of all analyzed areas.

Tabel 5: Evaluation Areas adapted from Dünser and Billinghurst (2005), Dünser et al.(2008), Swan II and Gabbard (2005)

Furthermore, combinations of areas were analyzed. The combinations of User Performance and Usability (12, 31.58%) and User Performance and Perception (10, 26.31%) occurred most. Other combination occurred one to three times.

Results of Evaluation Methods

The distribution of evaluation methods is presented in the following:

  • Qualitative Analysis: 33.33%
  • Technological Measurements: 29%
  • Subjective Measurements: 16.67%
  • Objective Measurements: 16.33%
  • Usability Evaluation: 2.33%
  • Informal Evaluations: 2.33%

We added the category Technological Measurements to our taxonomy, as we included system-based evaluations next to user-based ones. Qualitative analysis was the preferred evaluation method, followed by Technological Measurement. Subjective and objective measurements occurred in similar quantity and usability and informal evaluation methods were used in very few publications.

16 different combinations of evaluation methods occurred in the analyzed publications. The combination of Subjective measurements and Qualitative Analysis 15 occurred most (15, 24.59%), followed by Objective Measurements and Qualitative Analysis (13, 21.31%) as well as Objective Measurements and Subjective Measurements (11, 18.03%). Other frequencies of combinations ranged from 1 to 4.

The following figures 4-9 show the results of each evaluation method in detail:

Figure 4: Qualitative Evaluation Techniques (aggregated from 2015 - 2017) based on our Taxonomy

Figure 5: Technological Measurements (aggregated from 2015 - 2017) based on our Taxonomy

Figure 6: Subjective Measurements (aggregated from 2015 - 2017) based on taxonomy

Figure 7: Objective Measurements (aggregated from 2015 - 2017) based on our Taxonomy

Figure 8: Usability Evaluation Techniques (aggregated from 2015 - 2017) based on taxonomy

Figure 9: Informal Evaluation Techniques (aggregated from 2015 - 2017) based on our Taxonomy

Combination of Evaluation Areas and Methods

Due to lack of inconsistency in the classification of publications, two publications had to be excluded from this comparison. However 143 evaluations were analyzed and 57 different combinations occurred. The frequency of occurrence ranged from 1 to 48. The combination of the evaluation area System Technology with the evaluation method Technological Measurements occurred 48 times (33.57%), followed by the combination of User Performance and Objective measurements with 7 occurrences (4.9%). Unique combinations appeared 36 times in the analysis.

Limitations & Conclusions

The proposed taxonomies represent a collection of results from various literature-based sources that were set in relation to each other by combining them in hierarchical classifications. Our results are not representative for the total amount of AR related publications available in the literature, but were only collected from the analysis of our limited publication corpus. Like the exclusion criteria for publications that did not correspond to our definition described in the section Methodology, the venue proceedings and time interval were selected subjectively, but based on approaches or succeeding works from the literature. In general, our result is influenced by multiple subjective judgments and external factors, but however founded on various methodologies from the literature, so that we were able to draw comparisons between the results. Our analysis of 135 state-of-the-art AR publications has shown the generalizability of our taxonomies, as we were able to classify our entire publication corpus. Therefore, our taxonomies can be applied to each type of AR publications that contains at least one technological aspect, either including an evaluation or not.

Outlook and next Steps

We just peer-reviewed papers from other groups and are intended to the retrieve the feedback from our paper. We hope to retrieve constructive criticism, that we can discuss in our team and improve our paper.
Next steps are writing the rebuttal, improve our paper and prepare the final presentation.

See you in our last blog article in two weeks :)


[12] Mark Billinghurst, Adrian Clark, and Gun Lee. 2015. A Survey of Augmented Reality. Foundations and Trends® in Human–Computer Interaction 8, 2-3 (2015), 73–272.

[14] Jan vom Brocke, Alexander Simons, Björn Niehaves, Kai Riemer, Ralf Plattfaut, and Anne Cleven. 2009. Reconstructing the giant: On the importance of rigour in documenting the literature search process. In ECIS.

[15] Julie Carmigniani, Borko Furht, Marco Anisetti, Paolo Ceravolo, Ernesto Damiani, and Misa Ivkovic. 2011. Augmented reality technologies, systems and applications. Multimedia Tools and Applications 51, 1 (Jan. 2011), 341–377.

[22] Arindam Dey, Mark Billinghurst, Robert W. Lindeman, and J. Edward Swan II. 2016. A Systematic Review of Usability Studies in Augmented Reality between 2005 and 2014. In 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct). 49–50.

[25] Andreas Dünser and Mark Billinghurst. 2011. Evaluating Augmented Reality Systems. In Handbook of Augmented Reality, Borko Furht (Ed.). Springer New York, New York, NY, 289–307.

[26] Andreas Dünser, Raphaël Grasset, and Mark Billinghurst. 2008. A survey of evaluation techniques used in augmented reality studies. In ACM SIGGRAPH ASIA 2008 courses on - SIGGRAPH Asia ’08. ACM Press, Singapore, 1–27.

[62] George Papagiannakis, Gurminder Singh, and Nadia Magnenat-Thalmann. 2008. A survey of mobile and wireless technologies for augmented reality systems. Computer Animation and Virtual Worlds 19, 1 (Feb. 2008), 3–22. 10.1002/cav.221

[67] Dieter Schmalstieg and Tobias Höllerer. 2016. Augmented Reality: Principles and Practice. Addison-Wesley Professional. de/9780133153217

[71] J. Edwards Swan II and Joseph L. Gabbard. 2005. Survey of User-Based Experimentation in augmented Reality. In In 1st International Conference on Virtual Reality, Las Vegas.

[76] Rick Van Krevelen and Ronald Poelman. 2010. A Survey of Augmented Reality Technologies, Applications and Limitations. International Journal of Virtual Reality (ISSN 1081-1451) 9 (June 2010), 1.