Adaptation of the Learning Process using the Internet of Things
MATERIAL AND METHODS
Emotion assessment methods can be divided into two main groups according to the basic techniques used to recognize emotions: healing techniques based on self-assessment of emotions by completing various questionnaires (Isomursu, Tähti, Väinämö,
& Kuutti, 2007) (Wallbott & Scherer, 1989) and machine evaluation techniques based on the measurement of various parameters of the human body. Also, there are frequent cases of the simultaneous use of several methods to increase the reliability of the obtained results.
According to research by Scherer and Gonçalves, each emotion can be assessed by analysing the five main components of emotion (behavioural tendencies, physiological responses, motor expressions, cognitive assessment and subjective feelings), but only the first four can be assessed automatically and can indicate information about the user's emotional state during interaction without its interruption. Subjective feelings are usually assessed only using self-assessment methods (Scherer, 2005) (Gonçalves et al., 2017).
Automatic recognition of emotions is usually performed by measuring various parameters of the human body or electrical impulses in the nervous system and analysing their changes. The most popular techniques are electroencephalography, measurement of skin resistance, blood pressure, heart rate, eye activity and motion analysis.
Heart rate variability (HRV)
HRV is a technique for assessing emotional state based on measuring heart rate variability, which means fluctuations in rhythm over some time. In contrast to the mean deviation of the heart rate, which is expressed in the period of 60s, the HRV analysis examines the fluctuation of the nuance in each cycle of the heart rhythm and its regularity (Hsieh & Chin, 2011). Heart rate variability is regulated by the synergistic action of two branches of the autonomic nervous system, namely the sympathetic and parasympathetic nervous systems. Heart rate is the net effect of parasympathetic nerves, which slow down the heart rhythm, and sympathetic nerves, which speed it up. These changes are influenced by emotions, stress and physical exercise (Benezeth et al., 2018). Besides, HRV depends on age and gender, and other factors include physical and mental stress, smoking, alcohol, coffee, overweight and blood pressure, as well as glucose levels, infectious agents and depression. Hereditary genes also significantly affect heart rate variability. Low HRV indicates a state of relaxation, while increased HRV indicates a potential state of mental stress or frustration (Haag, Goronzy, Schaich, & Williams, 2004).
The classic technique for measuring HRV is the ECG, which measures the primary electro-biological signal related to cardiac activity and provides the ability to define the time between heart rate pulses as a function of time (Hsieh & Chin, 2011). The interval from the ECG signal can be extracted using conventional peak detection techniques, which allow the duration between each peak to be defined and form an HRV signal that expresses the change in the interval between peaks over time.
The common method of HRV analysis usually includes analytical methods in the time and frequency domain (Hsieh & Chin, 2011). The various studies based on analyses in one or both domains are briefly summarized in a study by Mikuckas et al. (Mikuckas et al., 2014).
The application of HRV to emotion recognition is complicated by the fact that HRV influences other factors, and various signal filtering and function extraction techniques are implemented to address this problem. There are approximately 14 different parameters that can be extracted by HRV analysis. A detailed description of these parameters and their relationship to the main emotions is given by the authors' Zhu, Ji and Liu in the research (Zhu, Ji, & Liu, 2019). The most common technique used for HRV analyses is to calculate the power spectral density (PSD) of the signal (Mikuckas et al., 2014). PSD represents the spectral power density of the time series as a function of frequency. Typical HRV measurements obtained from frequency domain analysis are forces within frequency bands and force ratios. The amount of energy contained in a frequency band can be obtained by integrating the PSD into the limits of the frequency bands (Mikuckas et al., 2014).
The main disadvantages of ECG-based HRV are the properties of the ECG, in particular, the complexity of the sensors and the high requirements for the measurement procedure to minimize the impact on the environment. An alternative to ECG-based HRV is photoplethysmography (PPG). Photoplethysmography is a technique for detecting changes in the microvascular volume of blood in tissues. The principle of this technology is very simple and requires only a light source and a photodetector. The light source illuminates the
tissue and the photodetector measures small changes in transmitted or reflected light (Figure 2) associated with changes in tissue perfusion (Benezeth et al., 2018).
Figure 2: Principles of PPG left reflection mode and right transmission mode (Benezeth et al., 2018).
The PPG signal (Figure 3) consists of two main components:
▪ The static part of the signal depends on the structure of the tissue and the average blood volume of the arterial and venous parts of the blood changes very slowly depending on the breathing,
▪ The dynamic part represents the changes in blood volume that occur between the systolic and diastolic phases of the heart cycle (Tamura, Maeda, Sekine, & Yoshida, 2014).
PPG signals, which are analogous to time-domain voltage values, are analysed using methods similar to those used for ECG-based HRV analysis. The main difference between PPG and ECG-based analysis is signal filtering using high-pass filters before defining peaks and generating the HRV signal. PPG can only be performed with one sensor attached to the finger or with multiple sensors attached to the right and left earlobes (Allen, 2007).
Figure 3: Example of the PPG signal (Tamura et al., 2014).
There are several studies that demonstrate the successful implementation of this technique and demonstrate its advantages over an ECG (Jeyhani, Mahdiani, Peltokangas, &
Vehkaoja, 2015). In research by Allen (Allen, 2007) a comparison between the ECG signal and the PPG signal is given (Figure 4), which demonstrates the strict relationships between the two signals. The delay of PPTp and PPTf in the PPG signal represents the time of transition until the heart rate reaches the measurement point.
Figure 4: Comparison of signals from PPG and ECG. (Elgendi et al., 2019)
Recently, there has been a growing interest in remote photoplethysmography (rPPG), which can restore the cardiovascular pulse wave by measuring variations in backscattered light at a distance, using only ambient light and inexpensive vision systems (Benezeth et al., 2018). Remote sensing makes it possible to significantly increase the level of human comfort during the measurement process, but this reduces the signal-to-noise ratio and increases the need for more advanced signal processing and analysis algorithms. In research by Maritsch (Maritsch et al., 2019) Machine learning algorithms have been implemented to increase the accuracy of HRV measurements performed by smartwatches. The results of this research prove that ML is a useful tool for analysing PPG measurement data and extracting the required functions.
A brief overview of research aimed at recognizing emotions using HRV is given in Table 1.
Table 1: An overview of scientific research focused on the recognition and evaluation of emotions using HRV (own design).
Bearing Emotions Methods Hardware and software
This study aimed to
Happiness and sadness HRV, skin temperature (SKT)
SKT sensor, PPG sensor (Park, Kim, Hwang, &
Lee, 2013)
This project aimed to
Mental stress HRV, GSR, breathing Heart rate monitor (HRM) (Polar WearLink recognizing emotions. The results of the review that the situation in this area is at odds with the situation with EEG or ECG, where researchers focus on the full development of PPG and rPPG techniques, including the development of new configurations, wearable PPG sensors, improved signal analysis and measurement methods and research new areas of application.
The main advantage of PPG-based HRV lies in the absence of a requirement for special training on humans for measurement. Usually, it is enough to touch the active surface of the sensor for a few seconds. The rPPG method provides the possibility of non-contact measurements. The cheap PPG device and its accessibility for all potential users are so simple that even the touch screen of a regular smartphone can be used as a PPG sensor.
These features of methodology reveal the potential for its implementation in a wide range of applications, especially in the field of human-machine-IoT interaction, as sensors of this type can be easily installed in joysticks and other machine controllers and can be hidden from the end-user.
In special cases where the number of emotions or their accuracy of detection requires conditions, the HRV technique needs to be supplemented by other techniques such as ECG, GSR and data fusion. This situation develops a high potential for the application of big data analysis techniques.
EXPERIMENT AND RESULT
As there has been a recent increase in interest in remote photoplethysmography (rPPG) as already mentioned in (Benezeth et al., 2018) we compared smart wristbands (Francisti &
Balogh, 2019). We also compared the individual wristbands with a reference device, which was the BOSO TM-2430 holster, which is commonly used in the medical environment.
Simultaneously with the pressure holster, we also measured the heart rate using smart wristbands at precisely set time intervals. Subsequently, we evaluated the measured data from the holster and the individual wristbands and compared their accuracy based on comparative statistics. When measuring the heart rate, we used a holster (A&D BOSO TM -2430), which recorded the pressure and heart rate, and we also used the following wristbands:
▪ Mi Band 2
▪ Mi Band 3
▪ Mi Smart Band 4
▪ Fitbit Charge 3
▪ Huawei band 3 pro
▪ Samsung Galaxy fit e
▪ Watchking Smart T8s
▪ Watchking Smart Q8s
The holster was set to record a pulse every 30 minutes. The exception was the night mode from 10 pm to 7 am, when the holster recorded a pulse for each hour. Since the holster also recorded data other than the pulse, we first had to modify the file from unnecessary data so that only information about the date, time and pulse remained in the table, to which we also added a column with information about the person's activities.
During measuring the heart rate, we recorded the changes of the wristbands in the table (Table 2) to remember the intervals putting a wristband on the wrist. According to that schedule, the names obtained from the holster were comparable to the data obtained from a particular wristband.
Table 2 Time schedule for pulse measurement
Examining (comparing), we found that among the smart wristbands with accurate heart rate measurement we can include Fitbit and Mi Band 4.
Table 3 Comparison of measured data from the holster, Fitbit and Mi Band 4 bracelet
When we summarize all the results from the comparison of the Mi Band 4 wristband with the holster, we can say that the measurement was more accurate than with the Fitbit Charge 3 wristband (Table 3). It follows that the Mi Band 4 is the most accurate measuring device of all the smart wristbands used in the experiment, which means that the measured values between the holster and the Mi Band 4 wristband are related and the differences between the values are not statistically significant.
We determined the statistical significance based on the percentage deviation between the measured values of the pulses, which we calculated as follows:
𝑂 = |(100 − ((𝑁 ∗ 100)/𝐻)|
Where:
▪ O the deviation is given in %,
▪ N is the value of the pulse measured using a wristband,
▪ H is the pulse value measured with a holster.
Deviation values are recorded in absolute value to remove negative values if the pulse values measured with the holster are smaller than those measured with the wristband.
CONCLUSION
The reliability, accuracy and speed of evaluating emotions strongly depend not only on the measurement method and sensor used but also on the signal processing and technique used in the analysis. The choice of measurement methods and sensors is a complex process in which a large number of questions are asked. Physiological parameters can be measured in the same way as the physical principles of signal acquisition. The measurement of technology concerning individual sensors creates a huge number of choices. More attempts have been made to classify emotions, sensors and universal selection algorithms. The first
step is to select the measurement parameters and methods, while the second step is to select the sensors.
We assume that at the beginning of the method selection it is necessary to define whether we are interested in a conscious or unconscious reaction, or maybe both methods at the same time. The research, based on conscious answers is relatively simple and does not require any special hardware but requires a lot of attention when preparing questionnaires. On the contrary, the results of self-evaluation are not so reliable and there is a possibility that a person will not correctly recognize his emotions or provide inaccurate answers to unpleasant questions. Methods based on unconscious responses usually provide more reliable results, but require more measurement attempts and increase high hardware requirements.
Methods based on unconscious reactions provide many possibilities. Because all reactions in the human body are controlled by electrical signals generated in the central nervous system, we can conclude that electrical parameters are the primary entities that provide most accurate results, and measuring non-electrical signals returns the human body's response to an electrical signal. Measurement of electrical parameters has two properties: it is possible to use methods based on direct (self-generating) sensors when measuring the signal covered by the central nervous system (EEG, ECG, HRV, EMG, EOG), or measurements based on modulation, i.e. when changes in the human body modulate the properties of the sensor (GSR).
By further research and by creating a comprehensive system for measuring physiological states and subsequent classification of the emotional state, we want to confirm the impact and influence of the emotional state on the teaching process. We assume that we will be able to adapt the teaching materials, learning style and approach to specific students according to their current emotional state. We assume that the adaptive system will be more flexible and effective for students in acquiring new knowledge.
ACKNOWLEDGEMENT
This research has been supported the projects KEGA 036UKF-4/2019, Adaptation of the learning process using sensor networks and the Internet of Things.
REFERENCES
Allen, J. J. P. m. (2007). Photoplethysmography and its application in clinical physiological measurement. 28(3), R1.
Benezeth, Y., Li, P., Macwan, R., Nakamura, K., Gomez, R., & Yang, F. (2018). Remote heart rate variability for emotional state monitoring. Paper presented at the 2018 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI).
Buyya, R., & Dastjerdi, A. V. (2016). Internet of Things: Principles and paradigms: Elsevier.
Coates, A., Hammoudeh, M., & Holmes, K. G. (2017). Internet of things for buildings monitoring:
Experiences and challenges. Paper presented at the Proceedings of the International Conference on Future Networks and Distributed Systems.
Feidakis, M., Daradoumis, T., & Caballé, S. (2011). Endowing e-learning systems with emotion awareness. Paper presented at the 2011 Third International Conference on Intelligent Networking and Collaborative Systems.
Francisti, J., & Balogh, Z. (2018). Wireless Sensor Network as a part of Internet of Things. In M.
Turcani, Z. Balogh, M. Munk, J. Kapusta, & L. Benko (Eds.), Divai 2018: 12th International Scientific Conference on Distance Learning in Applied Informatics (pp. 419-429).
Francisti, J., & Balogh, Z. (2019). Identification of Emotional States and Their Potential. In S. K.
Bhatia, S. Tiwari, K. K. Mishra, & M. C. Trivedi (Eds.), Advances in Computer Communication and Computational Sciences, Ic4s 2018 (Vol. 924, pp. 687-696).
Gómez, J., Huete, J. F., Hoyos, O., Perez, L., & Grigori, D. J. P. C. S. (2013). Interaction system based on internet of things as support for education. 21, 132-139.
Gonçalves, V. P., Giancristofaro, G. T., Geraldo Filho, P., Johnson, T., Carvalho, V., Pessin, G., . . . Ueyama, J. J. S. C. (2017). Assessing users’ emotion at interaction time: a multimodal approach with multiple sensors. 21(18), 5309-5323.
Haag, A., Goronzy, S., Schaich, P., & Williams, J. (2004). Emotion recognition using bio-sensors:
First steps towards an automatic system. Paper presented at the Tutorial and research workshop on affective dialogue systems.
Hsieh, P.-Y., & Chin, C.-L. (2011). The emotion recognition system with heart rate variability and facial image features. Paper presented at the 2011 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE 2011).
Chanel, G., Ansari-Asl, K., & Pun, T. (2007). Valence-arousal evaluation using physiological signals in an emotion recall paradigm. Paper presented at the 2007 IEEE International Conference on Systems, Man and Cybernetics.
Choi, J., Ahmed, B., & Gutierrez-Osuna, R. J. I. t. o. i. t. i. b. (2011). Development and evaluation of an ambulatory stress monitor based on wearable sensors. 16(2), 279-286.
Isomursu, M., Tähti, M., Väinämö, S., & Kuutti, K. J. I. J. o. H.-C. S. (2007). Experimental evaluation of five methods for collecting emotions in field settings with mobile applications. 65(4), 404-418.
Jeyhani, V., Mahdiani, S., Peltokangas, M., & Vehkaoja, A. (2015). Comparison of HRV parameters derived from photoplethysmography and electrocardiography signals. Paper presented at the 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).
Kummerfeld, B., & Kay, J. (2017). User modeling for the internet of things. Paper presented at the Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization.
Lee, M. S., Lee, Y. K., Pae, D. S., Lim, M. T., Kim, D. W., & Kang, T. K. J. A. S. (2019). Fast Emotion Recognition Based on Single Pulse PPG Signal with Convolutional Neural Network. 9(16), 3355.
Lee, S., Barker, T., & Kumar, V. (2011). Learning Preferences and Self-Regulation–Design of a Learner-Directed E-Learning Model. Paper presented at the International Conference on Advanced Software Engineering and Its Applications.
López, T. S., Ranasinghe, D. C., Harrison, M., McFarlane, D. J. P., & Computing, U. (2012). Adding sense to the Internet of Things. 16(3), 291-308.
Madakam, S., Lake, V., Lake, V., Lake, V. J. J. o. C., & Communications. (2015). Internet of Things (IoT): A literature review. 3(05), 164.
Maritsch, M., Bérubé, C., Kraus, M., Lehmann, V., Züger, T., Feuerriegel, S., . . . Wortmann, F.
(2019). Improving heart rate variability measurements from consumer smartwatches with machine learning. Paper presented at the Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers.
Mehta, R., Sahni, J., & Khanna, K. J. P. c. s. (2018). Internet of things: Vision, applications and challenges. 132, 1263-1269.
Mikuckas, A., Mikuckiene, I., Venckauskas, A., Kazanavicius, E., Lukas, R., & Plauska, I. J. E. i. E.
(2014). Emotion recognition in human computer interaction systems. 20(10), 51-56.
Park, M. W., Kim, C. J., Hwang, M., & Lee, E. C. (2013). Individual emotion classification between happiness and sadness by analyzing photoplethysmography and skin temperature. Paper presented at the 2013 Fourth World Congress on Software Engineering.
Quazi, M., Mukhopadhyay, S., Suryadevara, N., & Huang, Y.-M. (2012). Towards the smart sensors based human emotion recognition. Paper presented at the 2012 IEEE International Instrumentation and Measurement Technology Conference Proceedings.
Russell, J. A. J. J. o. p., & psychology, s. (1980). A circumplex model of affect. 39(6), 1161.
Scherer, K. R. J. S. s. i. (2005). What are emotions? And how can they be measured? , 44(4), 695-729.
Tamura, T., Maeda, Y., Sekine, M., & Yoshida, M. J. E. (2014). Wearable photoplethysmographic sensors—past and present. 3(2), 282-302.
Thompson, D. (2015). Formalisation vs. understanding. Paper presented at the International Conference on Unconventional Computation and Natural Computation.
Wallbott, H. G., & Scherer, K. R. (1989). Assessing emotion by questionnaire. In The measurement of emotions (pp. 55-82): Elsevier.
Zhu, J., Ji, L., & Liu, C. J. P. m. (2019). Heart rate variability monitoring for emotion and disorders of emotion. 40(6), 064004.