first step: what kind of UIs do scientists evaluate?

There are heterogenous UIs researchers evaluate. E.g. tools that visualise code and therefore support programmers, ephemeral UIs, iTV platforms, gestures UIs, voice authoring UIs, 3D virtual books, tangibles, Photo sorting, documents, Mail, story telling.

second step: Limitation of possible usage contexts

We decided to proceed with UIs that can be used by everybody currently to have the possibility of gathering many test persons. Additionally we need UIs access to so we decided to proceed with usage contexts of photo sorting, document sorting, organisation of e-Mails.

third step: tasks and metrics researchers use for evaluation photo- and document sorting

We again did some literature research and looked for tasks and metrics scientists use to evaluate their UIs in our chosen contexts. Especially for the photo context, we found lot of possible tasks and metrics (see following section). This deeper research confirmed our motivation, that there is a leck of standardised tasks. When reading papers, many authors do not name the tasks they use. Regarding the replication crises, this is an additional indicator that some standards need to be introduced in the research of UIs.
fourth step: Why photo sorting is a good task context to evaluate

We decided to deepen our research in the context of photo sorting. This can be established by the fact that giant photo collections need to be organised by users in the digital age. Instant Messaging Services such as WhatsApp, snapchat etc. support sharing images so the amount of pictures rises. It is also a good context because we can base our tasks on existing literature, such as Rodden et. al (2001), who introduced searching tasks that every user of photo-tools does. (searching for events, searching for a single specific image, searching for images that share common properties). Photo sorting tools are available for everybody so they can be evaluated easily. Since we could not find any standardised tasks, our research will provide some new insights and is therefore useful to any future study that evaluates tools that sort photos.

fifth step: Creation of survey: Including the user's perspective

questionnaire is currently being created. - future steps: evaluation of survey, creation of task and metric corpora, pre-study, main-study

Metrics - Photo sorting

• Think Aloud Protocol
• Video Protocol
• Interview
• TPT / TCT
• Survey / Quetsionnaire
• Observation: field notes
• Data logging
• average time spent in tool
• tool is being rated by users
• audio recording

Tasks - Photo sorting

• searching for events (birthday, holiday, .. )
• searching for single specific image
• searching for images that share common properties (e.g. all fotos contain best friend)
• from all photos taken, identify your favorite photographs
• from set of favourits, identify thouse that YOU took
• from those you took, identify those that you would choose to share with the other group members
• retrieve photos by tag
• retrieve photos by contact
• drag photos in software
• rearrange placement of photos
• creating and writing in text boxes and resizing them to annotate single or multiple photos
• select photos from a short-time event (1-2 days) and send it to someone over email
• select photos from a long time event (more than two days) to be uploaded to a web page or shown to someone
• select photos for a book representing events and happenings in the past 6-12 months

Literature

Ames, M. G., & Manguy, L. (2006, October). PhotoArcs: ludic tools for sharing photographs. In Proceedings of the 14th ACM international conference on Multimedia (pp. 615-618). ACM.

Clawson, J., Voida, A., Patel, N., & Lyons, K. (2008, September). Mobiphos: a collocated-synchronous mobile photo sharing application. In Proceedings of the 10th international conference on Human computer interaction with mobile devices and services (pp. 187-195). ACM.

Fujita, H., & Arikawa, M. (2009, November). Spatial sorting function for a map-based slideshow editor. In Proceedings of the 17th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (pp. 520-523). ACM.

Gaggi, O. (2013, March). Discovering local attractions from geo-tagged photos. In Proceedings of the 28th Annual ACM Symposium on Applied Computing (pp. 730-735). ACM.

Landry, B. M., & Guzdial, M. (2006, June). iTell: supporting retrospective storytelling with digital photos. In Proceedings of the 6th conference on Designing Interactive systems (pp. 160-168). ACM.

Li, C., Loui, A. C., & Chen, T. (2010, October). Towards aesthetics: A photo quality assessment and photo selection system. In Proceedings of the 18th ACM international conference on Multimedia (pp. 827-830). ACM.

Lu, Y., Wong, T. T., & Heng, P. A. (2004, October). Digital photo similarity analysis in frequency domain and photo album compression. In Proceedings of the 3rd international conference on Mobile and ubiquitous multimedia (pp. 237-244). ACM.

Ojala, J., & Malinen, S. (2012, October). Photo sharing in small groups: identifying design drivers for desired user experiences. In Proceeding of the 16th International Academic MindTrek Conference (pp. 69-76). ACM.

Patel, D., Marsden, G., Jones, M., & Jones, S. (2009, October). An evaluation of techniques for image searching and browsing on mobile devices. In Proceedings of the 2009 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists (pp. 60-69). ACM.

Ren, K., Sarvas, R., & Calic, J. (2009, October). FreeEye: intuitive summarisation of photo collections. In Proceedings of the 17th ACM international conference on Multimedia (pp. 1127-1128). ACM.

Ryu, D. S., Chung, W. K., & Cho, H. G. (2010, March). Photoland: a new image layout system using spatio-temporal information in digital photos. In Proceedings of the 2010 ACM Symposium on Applied Computing (pp. 1884-1891). ACM.

Strong, G., & Gong, M. (2009, July). Organizing and browsing photos using different feature vectors and their evaluations. In Proceedings of the ACM International Conference on Image and Video Retrieval (p. 3). ACM.

Sun, Y., Zhang, H., Zhang, L., & Li, M. (2002, December). MyPhotos: a system for home photo management and processing. In Proceedings of the tenth ACM international conference on Multimedia (pp. 81-82). ACM.

Tankoyeu, I., Stöttinger, J., Paniagua, J., & Giunchiglia, F. (2012, October). Personal photo indexing. In Proceedings of the 20th ACM international conference on Multimedia (pp. 1341-1342). ACM.