Emotion Detection Research: A Systematic Review Focuses on Data Type, Classifier Algorithm, and Experimental Methods
DOI:
https://doi.org/10.24002/ijieem.v5i1.7077Keywords:
emotion detection, emotion recognition, wearable device, smartphone, arousal-valence, classifierAbstract
There is a lot of research being done on detecting human emotions. Emotion detection models are developed based on physiological data. With the development of low-cost wearable devices that measure human physiological data such as brain activity, heart rate, and skin conductivity, this research can be conducted in developing countries like Southeast Asia. However, as far as the author's research is concerned, a literature review has yet to be found on how this research on emotion detection was carried out in Southeast Asia. Therefore, this study aimed to conduct a systematic review of emotion detection research in Southeast Asia, focusing on the selection of physiological data, classification methods, and how the experiment was conducted according to the number of participants and duration. Using PRISMA guidelines, 22 SCOPUS-indexed journal articles and proceedings were reviewed. The review found that physiological data were dominated by brain activity data with the Muse Headband, followed by heart rate and skin conductivity collected with various wristbands, from around 5-31 participants, for 8 minutes to 7 weeks. Classification analysis applies machine learning, deep learning, and traditional statistics. The experiments were conducted primarily in sitting and standing positions, conditioned environments (for developing research), and unconditioned environments (applied research). This review concluded that future research opportunities exist regarding other data types, data labeling methods, and broader applications. These reviews will contribute to the enrichment of ideas and the development of emotion recognition research in Southeast Asian countries in the future.
References
Abdullah, M. K., Mohd Suradi, N. R., Jamaluddin, N., Mokhtar, A. S., Abu Talib, A. R., & Zainuddin, M. F. (2006). K-chart: a tool for research planning and monitoring. Journal of Quality Management And Analysis, 2(1), 123–130.
Abujelala, M., Abellanoza, C., Sharma, A., & Makedon, F. (2016). Brain-ee: Brain enjoyment evaluation using commercial eeg headband. Proceedings of the 9th Acm International Conference on Pervasive Technologies Related to Assistive Environments, 1–5.
Akçay, M. B., & Oğuz, K. (2020). Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers. Speech Communication, 116, 56–76.
Al-Nafjan, A., Hosny, M., Al-Ohali, Y., & Al-Wabil, A. (2017). Review and Classification of Emotion Recognition Based on EEG Brain-Computer Interface System Research: A Systematic Review. Applied Sciences, 7(12), Article 1239.
Alghowinem, S., Goecke, R., Wagner, M., Parker, G., & Breakspear, M. (2013). Eye movement analysis for depression detection. 2013 IEEE International Conference on Image Processing, 4220–4224.
Alsaggaf, M. A., & Althonayan, A. (2018). An empirical investigation of customer intentions influenced by service quality using the mediation of emotional and cognitive responses. Journal of Enterprise Information Management, 31(1), 194–223.
Banan, A., Nasiri, A., & Taheri-Garavand, A. (2020). Deep learning-based appearance features extraction for automated carp species identification. Aquacultural Engineering, 89, Article 102053.
Bin Suhaimi, N. S., Mountstephens, J., & Teo, J. (2020). Emotional State Classification with Distributed Random Forest, Gradient Boosting Machine, and Naïve Bayes in Virtual Reality Using Wearable Electroencephalography and Inertial Sensing. ISCAIE 2020 - IEEE 10th Symposium on Computer Applications and Industrial Electronics, 12–17.
Bulagang, A. F., Mountstephens, J., & Teo, J. (2021a). A Novel Approach for Emotion Classification in Virtual Reality using Heart Rate (HR) and Inter-beat Interval (IBI). 2021 IEEE International Conference on Computing, ICOCO 2021, 247–252.
Bulagang, A. F., Mountstephens, J., & Teo, J. (2021b). Electrodermography and Heart Rate Sensing for Multiclass Emotion Prediction in Virtual Reality: A Preliminary Investigation. 2021 IEEE Symposium on Industrial Electronics and Applications, ISIEA 2021.
Bulagang, A. F., Mountstephens, J., & Teo, J. (2021c). Support Vector Machine Tuning for Improving Four-Quadrant Emotion Prediction in Virtual Reality (VR) using Wearable Electrodermography (EDG). Journal of Physics: Conference Series, 1878(1), Article 012020.
Bulagang, A. F., Mountstephens, J., & Teo, J. (2022). Investigating Electrodermography (EDG) and Heart Rate (HR) Signals for Emotion Classification in Virtual Reality. IVIT 2022 - Proceedings of 1st International Visualization, Informatics and Technology Conference, 29–34.
Bulagang, A. F., Mountstephens, J., & Wi, J. T. T. (2020). Tuning Support Vector Machines for Improving Four-Class Emotion Classification in Virtual Reality (VR) using Heart Rate Features. Journal of Physics: Conference Series, 1529(5), Article 052069.
Casciola, A. A., Carlucci, S. K., Kent, B. A., Punch, A. M., Muszynski, M. A., Zhou, D., Kazemi, A., Mirian, M. S., Valerio, J., & McKeown, M. J. (2021). A deep learning strategy for automatic sleep staging based on two-channel EEG headband data. Sensors, 21(10), Article 3316.
Dao, M.-S., Dang-Nguyen, D.-T., & Kasem, A. (2018). HealthyClassroom-a proof-of-concept study for discovering students’ daily moods and classroom emotions to enhance a learning-teaching process using heterogeneous sensors. Proceedings of the 7th International Conference on Pattern Recognition Applications and Methods, 1, 685-691.
Dzedzickis, A., Kaklauskas, A., & Bucinskas, V. (2020). Human Emotion Recognition: Review of Sensors and Methods. Sensors, 20(3), 592.
Feng, H., Golshan, H. M., & Mahoor, M. H. (2018). A wavelet-based approach to emotion classification using EDA signals. Expert Systems with Applications, 112, 77–86.
Fernández-Aguilar, L., Martínez-Rodrigo, A., Moncho-Bogani, J., Fernández-Caballero, A., & Latorre, J. M. (2019). Emotion detection in aging adults through continuous monitoring of electro-dermal activity and heart-rate variability. Understanding the Brain Function and Emotions: 8th International Work-Conference on the Interplay Between Natural and Artificial Computation, IWINAC 2019, Almería, Spain, June 3–7, 2019, Proceedings, Part I 8, 252–261.
Garcia-Moreno, F. M., Bermudez-Edo, M., Rodríguez-Fórtiz, M. J., & Garrido, J. L. (2020). A CNN-LSTM deep Learning classifier for motor imagery EEG detection using a low-invasive and low-Cost BCI headband. 2020 16th International Conference on Intelligent Environments (IE), 84–91.
Godfrey, A., Hetherington, V., Shum, H., Bonato, P., Lovell, N. H., & Stuart, S. (2018). From A to Z: Wearable technology explained. Maturitas, 113, 40–47.
Gonzalez, H. A., Muzaffar, S., Yoo, J., & Elfadel, I. M. (2020). BioCNN: A Hardware Inference Engine for EEG-Based Emotion Detection. IEEE Access, 8, 140896–140914.
Guo, H. W., Huang, Y. S., Chien, J. C., & Shieh, J. S. (2015). Short-term analysis of heart rate variability for emotion recognition via a wearable ECG device. 2015 International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS), 262–265.
Gupta, R., Khomami Abadi, M., Cárdenes Cabré, J. A., Morreale, F., Falk, T. H., & Sebe, N. (2016). A quality adaptive multimodal affect recognition system for user-centric multimedia indexing. Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval, 317–320.
Haddaway, N. R., Page, M. J., Pritchard, C. C., & McGuinness, L. A. (2022). PRISMA2020: An R package and Shiny app for producing PRISMA 2020-compliant flow diagrams, with interactivity for optimised digital transparency and Open Synthesis. Campbell Systematic Reviews, 18(2), Article 1230.
Hartono, M. (2020). The modified Kansei Engineering-based application for sustainable service design. International Journal of Industrial Ergonomics, 79, Article 102985.
Huynh, S., Lee, Y., Park, T., & Balan, R. K. (2016). Jasper: Sensing gamers’ emotions using physiological sensors. MobiGames 2016 - Proceedings of the 3rd Workshop on Mobile Gaming, Co-Located with MobiSys 2016, 1–6.
Jaharuddin, N. S., & Zainol, L. N. (2019). The Impact of Work-Life Balance on Job Engagement and Turnover Intention. The South East Asian Journal of Management, 13(1), Article 7.
Jain, S., Jain, V., & Das, S. (2018). Relationship analysis between emotional intelligence and service quality with special evidences from Indian banking sector. Revista Espacios, 39(33), Article 3.
Jalilifard, A., da Silva, A. G., & Islam, M. K. (2017). Brain-tv connection: Toward establishing emotional connection with smart tvs. 2017 IEEE Region 10 Humanitarian Technology Conference (R10-HTC), 726–729.
Kadariya, S., Gautam, R., & Aro, A. R. (2019). Physical Activity, Mental Health, and Wellbeing among Older Adults in South and Southeast Asia: A Scoping Review. BioMed Research International, 2019, Article 6752182.
Kamdar, M. R., & Wu, M. J. (2016). PRISM: a data-driven platform for monitoring mental health. Biocomputing 2016: Proceedings of the Pacific Symposium, 333–344.
Kanjo, E., Younis, E. M. G., & Ang, C. S. (2019). Deep learning analysis of mobile physiological, environmental, and location sensor data for emotion detection. Information Fusion, 49, 46–56.
Koppe, G., Meyer-Lindenberg, A., & Durstewitz, D. (2020). Deep learning for small and big data in psychiatry. Neuropsychopharmacology, 46(1), 176–190.
Krithika, L. B., & Priya, G. G. L. (2016). Student emotion recognition system (SERS) for e-learning improvement based on learner concentration metric. Procedia Computer Science, 85, 767–776.
Le-Quang, B. L., Dao, M.-S., & Nazmudeen, M. S. H. (2019). Wemotion: A system to detect emotion using wristbands and smartphones. In Communications in Computer and Information Science, 968, 92–103.
Liliana, D. Y., Ismail, I. E., Zain, A. R., & Balgis, S. S. (2021). Emotion Intelligent Application for Independent Wellbeing Management using Certainty Factor. Proceedings - 2021 4th International Conference on Computer and Informatics Engineering: IT-Based Digital Industrial Innovation for the Welfare of Society, IC2IE 2021, 338–343.
Lu, X., Liu, X., & Stolterman Bergqvist, E. (2019). “ It sounds like she is sad” Introducing a Biosensing Prototype that Transforms Emotions into Real-time Music and Facilitates Social Interaction. Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, 1–6.
Maier, M., Elsner, D., Marouane, C., Zehnle, M., & Fuchs, C. (2019). DeepFlow: Detecting Optimal User Experience From Physiological Data Using Deep Neural Networks. AAMAS, 2108–2110.
Minhad, K. N., Ooi, J. S. K., Md Ali, S. H., Reaz, M. I., & Ahmad, S. A. (2017). Low-cost wearable sensor for human emotion recognition using skin conductance response. IEICE Transactions on Information and Systems, E100D(12), 3010–3017.
Mullins, J. T., & White, C. (2019). Temperature and mental health: Evidence from the spectrum of mental health outcomes. Journal of Health Economics, 68, 102240.
Najafabadi, M. M., Villanustre, F., Khoshgoftaar, T. M., Seliya, N., Wald, R., & Muharemagic, E. (2015). Deep learning applications and challenges in big data analytics. Journal of Big Data, 2(1), 1–21.
Nakisa, B., Rastgoo, M. N., Rakotonirainy, A., Maire, F., & Chandran, V. (2018). Long short term memory hyperparameter optimization for a neural network-based emotion recognition framework. IEEE Access, 6, 49325–49338.
Naruetharadhol, P., & Gebsombut, N. (2020). A bibliometric analysis of food tourism studies in Southeast Asia. Cogent Business & Management, 7(1), Article 1733829.
Nguyen, N. T., Nguyen, N. V, Tran, M. H. T., & Nguyen, B. T. (2017). A potential approach for emotion prediction using heart rate signals. Proceedings - 2017 9th International Conference on Knowledge and Systems Engineering, 221–226.
Noelke, C., McGovern, M., Corsi, D. J., Jimenez, M. P., Stern, A., Wing, I. S., & Berkman, L. (2016). Increasing ambient temperature reduces emotional well-being. Environmental Research, 151, 124–129.
O’Brien, N., Latessa, S., Evangelopoulos, G., & Boix, X. (2018). The language of fake news: Opening the black-box of deep learning-based detectors. https://dspace.mit.edu/handle/1721.1/120056
Pollreisz, D., & TaheriNejad, N. (2017). A simple algorithm for emotion recognition using physiological signals of a smart watch. 2017 39th Annual International Conference of the Ieee Engineering in Medicine and Biology Society (EMBC), 2353–2356.
Prabowo, D. S. P., & Aji, H. M. (2021). Visual packaging and perceived emotional value: A study on Islamic branded cosmetics. The South East Asian Journal of Management, 15(1), Article 4.
Ramli, I., Jamil, N., Ardi, N., & Hamzah, R. (2018). Emolah: A Malay language spontaneous speech emotion recognition on iOS platform. International Journal of Engineering and Technology(UAE), 7(3), 151–156.
Ramos, A. L. A., Flor, J. M., Casabuena, M. M., & Estrada, J. E. (2018). Face expression extraction using eigenfaces, fisherfaces, and local binary pattern histogram towards predictive analysis of students emotion in programming. In Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, 248, 38–46.
Rattanyu, K., Ohkura, M., & Mizukawa, M. (2010). Emotion monitoring from physiological signals for service robots in the living space. ICCAS 2010, 580–583.
Razali, N. A. M., Malizan, N. A., Hasbullah, N. A., Wook, M., Zainuddin, N. M., Ishak, K. K., Ramli, S., & Sukardi, S. (2023). Political Security, Threat Prediction Framework, Using Hybrid Lexicon-Based Approach and Machine Learning Technique. IEEE Access, 11, 17151–17164.
Saganowski, S., Dutkowiak, A., Dziadek, A., Dzieżyc, M., Komoszyńska, J., Michalska, W., Polak, A., Ujma, M., & Kazienko, P. (2020). Emotion recognition using wearables: A systematic literature review-work-in-progress. 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), 1–6.
Schmidt, P., Reiss, A., Duerichen, R., Marberger, C., & Van Laerhoven, K. (2018). Introducing Wesad, a multimodal dataset for wearable stress and affect detection. Proceedings of the 20th ACM International Conference on Multimodal Interaction, 400–408.
Sofian Suhaimi, N., Mountstephens, J., & Teo, J. (2021). Class-Based Analysis of Russell’s Four-Quadrant Emotion Prediction in Virtual Reality using Multi-Layer Feedforward ANNs. ACM International Conference Proceeding Series, 155–161.
Suhaimi, N. S., Mountstephens, J., & Teo, J. (2020). Parameter tuning for enhancing inter-subject emotion classification in four classes for vr-eeg predictive analytics. International Journal of Advanced Science and Technology, 29(6 Special Issue), 1483–1491.
Suhaimi, N. S., Mountstephens, J., & Teo, J. (2021). Explorations of A Real-Time VR Emotion Prediction System Using Wearable Brain-Computer Interfacing. Journal of Physics: Conference Series, 2129(1).
Suhaimi, N. S., Teo, J., & Mountstephens, J. (2018a). Emotional State Classification in Virtual Reality Using Wearable Electroencephalography. IOP Conference Series: Materials Science and Engineering, 341(1).
Suhaimi, N. S., Teo, J., & Mountstephens, J. (2018b). Empirical analysis of intra vs. Inter-subject variability in VR EEG-based emotion modeling. Journal of Engineering and Applied Sciences, 13(8), 2137–2144.
Teo, J., & Chia, J. T. (2018). Deep Neural Classifiers for Eeg-Based Emotion Recognition in Immersive Environments. 2018 International Conference on Smart Computing and Electronic Enterprise, ICSCEE 2018.
Teo, J., Suhaimi, N. S., & Mountstephens, J. (2020). Pushing the boundaries of EEG-based emotion classification using consumer-grade wearable brain-computer interfacing devices and ensemble classifiers. International Journal of Advanced Science and Technology, 29(6 Special Issue), 1475–1482.
Ujir, H., Jee, E. M., Farhaan Iqbal, M., Mun, Q. K., & Hipiny, I. (2021). Real-Time Driver’s Monitoring Mobile Application through Head Pose, Drowsiness, and Angry Detection. Proceedings of the 8th International Conference on Computer and Communication Engineering, ICCCE 2021, 1–6.
Utami, K., Rauf, A., & Salmiah, S. (2019). Consumer Behavior Analysis in Organic Vegetables Decision Making in Medan City. Agrotekma: Jurnal Agroteknologi Dan Ilmu Pertanian, 3(2), 75.
Valliappan, S., Mohan, B. P. R., & Kumar, S. R. (2017). Design of low-cost, wearable remote health monitoring and alert system for elderly heart patients. 2017 International Conference on IoT and Application (ICIOT), 1–7.
Wahyono, I. D., Saryono, D., Putranto, H., Asfani, K., Rosyid, H. A., Sunarti, Mohamad, M. M., Said, M. N. H. B. M., Horng, G. J., & Shih, J.-S. (2022). Emotion Detection based on Column Comments in Material of Online Learning using Artificial Intelligence. International Journal of Interactive Mobile Technologies, 16(3), 82–91.
Xu, T., Yin, R., Shu, L., & Xu, X. (2019). Emotion recognition using frontal EEG in VR affective scenes. 2019 IEEE MTT-S International Microwave Biomedical Conference (IMBioC), 1, 1–4.
Yadollahi, A., Shahraki, A. G., & Zaiane, O. R. (2017). Current state of text sentiment analysis from opinion to emotion mining. ACM Computing Surveys (CSUR), 50(2), 1–33.
Yang, B., Li, X., Hou, Y., Meier, A., Cheng, X., Choi, J.-H., Wang, F., Wang, H., Wagner, A., & Yan, D. (2020). Non-invasive (non-contact) measurements of human thermal physiology signals and thermal comfort/discomfort poses-a review. Energy and Buildings, 224, 110261.
Zaheer, S., Amjad, M. S., Rafique, M. Z., & Khan, M. A. (2020). A K-Chart-based implementation framework to attain lean & agile manufacturing. International Journal of Production Management and Engineering, 8(2), 123–135.
Zhao, B., Wang, Z., Yu, Z., & Guo, B. (2018). EmotionSense: Emotion recognition based on wearable wristband. 2018 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI), 346–355.
Zheng, L. J., Mountstephens, J., & Teo, J. (2020). Comparing eye-tracking versus EEG features for four-class emotion classification in VR predictive analytics. International Journal of Advanced Science and Technology, 29(6 Special Issue), 1492–1497.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 International Journal of Industrial Engineering and Engineering Management
This work is licensed under a Creative Commons Attribution 4.0 International License.
The copyright of the received article shall be assigned to the journal as the publisher of the journal. The intended copyright includes the right to publish the article in various forms (including reprints). The journal maintains the publishing rights to the published articles.
This work is licensed under a Creative Commons Attribution 4.0 International License
