Development and evaluation of an approach for predicting mouse positions beyond the system’s latency

Latency is defined as the time between a user’s action and the response of the system to it (Friston et al., 2016). It is known to decrease the performance (Friston et al., 2016; MacKenzie & Ware, 1993; Tochioka et al., 2019) and user experience (Caserman et al., 2019; Ng & Dietz, 2014) when interacting with computers. The decrease in performance has been noticed in simple Fitts’ Law tasks (MacKenzie & Ware, 1993), steering law tasks (Friston et al., 2016; Tochioka et al., 2019) and gaming scenarios (Claypool, 2005; Liu et al., 2021). For this reason, many researchers in Human-Computer-Interaction (HCI) tried to minimize and compensate latency. One way to achieve this is by reducing the latency in the system’s hardware (Le et al., 2017). However, faster components, which cause less latency, are more expensive (Ushirobira et al., 2016) and latency will always be present to some degree with every hardware (Tochioka et al., 2019).

Instead of reducing the latency of a systems’ hard and software components, previous work also tried to reduce latency using software that predicts the user’s actions. Researchers, as well as practitioners, experimented with several methods including time derivates, heuristic approaches, linear and polynomial extrapolation, and curve fitting to predict the user’s actions (Nancel et al., 2018). Promising results by Henze et al. (2016) suggest that using neural networks for this prediction can be an effective solution (Henze et al., 2016). Neural network predictions of user behavior have been effectively used in the context of touchscreens (Henze et al., 2016; Le et al., 2017), shooter games (Halbhuber et al., 2021), and virtual reality (Schwind et al., 2020). They were able to improve users’ performance and/or experience.

Predicting users’ mouse movement to reduce perceived latency could be expanded by predicting the position of the mouse cursor beyond the system’s latency, therefore decreasing the latency to a negative value. As some research found effects even of very low latencies on performance and user experience (Deber et al., 2015; Ng et al., 2012), and intuitively, a system that performs user actions preemptively appears more performant, it can be suspected that such a system would lead to a higher performance. However, some experts think there is a lower limit for the effects of latency (Friston et al., 2016). Moreover, users might be irritated by a prediction beyond their actions. Consequently, it remains unclear whether such a prediction can improve performance and user experience.

The goal of this thesis is to investigate how prediction beyond the system’s latency influences user performance and experience in simple tasks. For this purpose, a state-of-the-art neural network will be trained to predict the mouse cursor position at a time point that is beyond the latency of the system.

To train the neural network, mouse movement data from participants completing a simple browser-based task must be collected. Thus, a standard task known from HCI research will be implemented in JavaScript and made available through the Internet. With this data, the network architecture will be optimized, and a prediction algorithm will be trained. After training the network its predictions will be integrated into the web-based implementation of the HCI task. Thereupon, an empirical user study will be conducted to assess the influence of different prediction timeframes on user performance and experience.

• Research and preparation of literature on latency, prediction, latency mitigation, and HCI methods
• Development of a web-based HCI task for mouse data collection
• Conduction of an online data collection study
• Development and training of a neural network for mouse position prediction
• Integration of predicted positions into the task
• Design of a user study to assess the influence of different prediction timeframes
• Conduction of the study
• Analyzation of results

• Machine learning methods
• Web development in JavaScript
• Standard methods in Human-Computer-Interaction
• Designing, conducting, and evaluating empirical studies

• Caserman, P., Martinussen, M., & Göbel, S. (2019). Effects of End-to-end Latency on User Experience and Performance in Immersive Virtual Reality Applications. In E. van der Spek, S. Göbel, E. Y.-L. Do, E. Clua, & J. Baalsrud Hauge (Eds.), Lecture Notes in Computer Science. Entertainment Computing and Serious Games (Vol. 11863, pp. 57–69). Springer International Publishing. https://doi.org/10.1007/978-3-030-34644-7_5
• Claypool, M. (2005). The effect of latency on user performance in Real-Time Strategy games. Computer Networks, 49(1), 52–70. https://doi.org/10.1016/j.comnet.2005.04.008 Deber, J., Jota, R., Forlines, C., & Wigdor, D. (2015). How Much Faster is Fast Enough? In B. Begole (Ed.), ACM Digital Library, Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp. 1827–1836). ACM. https://doi.org/10.1145/2702123.2702300
• Friston, S., Karlström, P., & Steed, A. (2016). The Effects of Low Latency on Pointing and Steering Tasks. IEEE Transactions on Visualization and Computer Graphics, 22(5), 1605–1615. https://doi.org/10.1109/TVCG.2015.2446467
• Halbhuber, D., Henze, N., & Schwind, V. (2021). Increasing Player Performance and Game Experience in High Latency Systems. Proceedings of the ACM on Human-Computer Interaction, 5(CHI PLAY), 1–20. https://doi.org/10.1145/3474710
• Henze, N., Funk, M., & Shirazi, A. S. (2016). Software-reduced touchscreen latency. In F. Paternò, K. Väänänen, K. Church, J. Häkkilä, A. Krüger, & M. Serrano (Eds.), Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services (pp. 434–441). ACM. https://doi.org/10.1145/2935334.2935381
• Le, H. V., Schwind, V., Göttlich, P., & Henze, N. (2017). PredicTouch. In S. Subramanian, J. Steimle, R. Dachselt, D. Martinez Plasencia, & T. Grossman (Eds.), Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces (pp. 230–239). ACM. https://doi.org/10.1145/3132272.3134138
• Liu, S., Claypool, M., Kuwahara, A., Scovell, J., & Sherman, J. (2021). The Effects of Network Latency on Competitive First-Person Shooter Game Players. In 2021 13th International Conference on Quality of Multimedia Experience (QoMEX) (pp. 151–156). IEEE. https://doi.org/10.1109/QoMEX51781.2021.9465419
• MacKenzie, I. S., & Ware, C. (1993). Lag as a determinant of human performance in interactive systems. In B. Arnold, G. van der Veer, & T. White (Eds.), Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '93 (pp. 488–493). ACM Press. https://doi.org/10.1145/169059.169431
• Nancel, M., Aranovskiy, S., Ushirobira, R., Efimov, D., Poulmane, S., Roussel, N., & Casiez, G [Géry] (2018). Next-Point Prediction for Direct Touch Using Finite-Time Derivative Estimation. In * P. Baudisch, A. Schmidt, & A. Wilson (Eds.), Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (pp. 793–807). ACM. https://doi.org/10.1145/3242587.3242646 Ng, A., & Dietz, P. H. (2014). The effects of latency and motion blur on touch screen user experience. Journal of the Society for Information Display, 22(9), 449–456. https://doi.org/10.1002/jsid.243
• Ng, A., Lepinski, J., Wigdor, D., Sanders, S., & Dietz, P. (2012). Designing for low-latency direct-touch input. In R. Miller (Ed.), ACM Conferences, Proceedings of the 25th annual ACM symposium on User interface software and technology (p. 453). ACM. https://doi.org/10.1145/2380116.2380174
• Schwind, V., Halbhuber, D., Fehle, J., Sasse, J., Pfaffelhuber, A., Tögel, C., Dietz, J., & Henze, N. (2020). The Effects of Full-Body Avatar Movement Predictions in Virtual Reality using Neural Networks. In R. J. Teather, C. Joslin, W. Stuerzlinger, P. Figueroa, Y. Hu, A. U. Batmaz, W. Lee, & F. Ortega (Eds.), 26th ACM Symposium on Virtual Reality Software and Technology (pp. 1–11). ACM. https://doi.org/10.1145/3385956.3418941
• Tochioka, H., Ikeda, H., Hayakawa, T., & Ishikawa, M. (2019). Effects of Latency in Visual Feedback on Human Performance of Path-Steering Tasks. In T. Trescak, S. Simoff, D. Richards, A. Bogdanovych, T. Duval, T. Kuhlen, H. Nguyen, S. Morishima, Y. Itoh, R. Skarbez, & M. Masek (Eds.), 25th ACM Symposium on Virtual Reality Software and Technology (pp. 1–2). ACM. https://doi.org/10.1145/3359996.3364726
• Ushirobira, R., Efimov, D., Casiez, G [Gery], Roussel, N., & Perruquetti, W. (2016). A forecasting algorithm for latency compensation in indirect human-computer interactions. In 2016 European Control Conference (ECC) (pp. 1081–1086). IEEE. https://doi.org/10.1109/ECC.2016.7810433