Classification of Multimodal Social Media Crisis Data – Evaluation and Comparison of two Multimodal Machine Learning Models

With the rise of the internet came a flood of free accessible information. Especially social media distributes huge amounts of unstructured information. With the emergence of a humanitarian disaster, there is also always a surge of crisis-related posts and other mentions of said crisis. Helpers in the region can use this information to distribute help to places where it is needed. However, filtering this information requires either human resources, classification, or filtering by automation. The field of this form of information processing is called Crisis Computing. Crisis Computing and Crisis Informatics have shown promising development in the last years. This turn is towards multi-modal fusion models that allow good classification results. But most of the research was conducted with the same data, namely the CrisisMMD (Alam et al., 2018) data set, and evaluated with said data set. For example, in the paper of Zou et al. (2021) ,where they fuse Twitter data and evaluate it in comparison with unimodal approaches. Also, the same researchers that created the CrissisMMD use it in their paper on multimodal learning for disaster response. (Ofli et al., 2020) So even though there is a good base of approaches and research, application on current crisis data is relatively rare.

This thesis discusses and evaluates two architectures for multimodal crisis image and text classification, which can then be evaluated on the generated Data Set as well as the CrisisMMD dataset, to ensure right implementation. The goal is to evaluate if the architectures perform well on evaluating data sets that are not contained in the CrisisMMD and which of the evaluated architectures show the best performance. The specific evaluation will be done on the classification tasks specified in the paper of Adwaith et al. (2022) which evaluates the problems of former papers with state-of-the-art machine learning tools. The aim is to classify Twitter information on its relevance and if relevant, which kind of response they would require.

The tasks in Order:

• Collection of a social media data set of disaster tweets and images
• Training and evaluating the architectures on CrisisMMD
• If results are worse than related work: reevaluation and adjustment
• Testing the models on the collected data set
• Evaluation and discussion of the results

• Machine learning basics and NN modeling
• Multimodal data analysis and classification
• Social media scraping
• Study Design

Adwaith, D., Abishake, A. K., Raghul, S. V., & Sivasankar, E. (2022). Enhancing multimodal disaster tweet classification using state-of-the-art deep learning networks. Multimedia Tools and Applications, 81(13), 18483–18501. https://doi.org/10.1007/s11042-022-12217-3

Alam, F., Ofli, F., & Imran, M. (2018). CrisisMMD: Multimodal Twitter Datasets from Natural Disasters (arXiv:1805.00713). arXiv. http://arxiv.org/abs/1805.00713

Ofli, F., Alam, F., & Imran, M. (2020). Analysis of Social Media Data using Multimodal Deep Learning for Disaster Response (arXiv:2004.11838). arXiv. http://arxiv.org/abs/2004.11838

Zou, Z., Gan, H., Huang, Q., Cai, T., & Cao, K. (2021). Disaster Image Classification by Fusing Multimodal Social Media Data. ISPRS International Journal of Geo-Information, 10(10), Article 10. https://doi.org/10.3390/ijgi10100636