EAPRIL Conference Proceedings 2015

(1)

EEG MIND READER 1.0 AS A PART OF COMPLEX MEASURING METHOD (CMM) FOR EDUCATION

L Devosa*, Á. Maródi**, T. Grósz ***, Zs. Búzás****, J. Steklács*****

*Lead researcher. Health Science and Health Promotion Research Group, College of Kecskemét, H-6000 Kecskemét Kaszap str. 6-14., devosa.ivan@tfk.kefo.hu, **PhD student. Institute of Education, University of Szeged, H-6720 Szeged, Dugonics sq. 13., agimarodi@gmail.com *** PhD student. Institute of Infonnatics, University of Szeged, H-6720 Szeged, Dugonics sq. 13., grosz.tamas@inf.u-szeged.hu ****Researcher, Health

Science and Health Promotion Research Group, College of Kecskemét, H-6000 Kecskemét Kaszap str. 6-14., buzas.zsuzsa@tfk.kefo.hu, *****Researcher, Health Science and Health Promotion Research Group, College of

Kecskemét, H-6000 Kecskemét Kaszap str. 6-14., steklacs.janos@tfk.kefo.hu.

ABSTRACT

In the Complex Measuring Method (CMM) two devices are used together simultaneously: Neurosky MindWave and TOBII T120 connected together with the Health Science and Health Promotion Research Group (HSHP-RG) - MindReader 1.0 application running on a Microsoft Windows 7 based personal computer, thus the recorded data o f both devices get the timer stamps from the same source, therefore the received data (TOBII & EEG) can be compared easily and exactly. Studies based on TOBBI eye-tracking device, researchers still cannot answer one o f the most important question: when the eye is on a fixation point fo r a long time, the subject is thinking (concentrating), or just relaxing fo r a while? Using CMM, we can answer this question, following up the appearance o f fatigue and

differentiate the required concertation levels fo r the same exercise for different pupils.

(2)

INTRODUCING CMM

The Complex Measuring Method (CMM) is a methodological tool for education, developed by our research group: Health Science and Health Promotion Research Group (HSHP-RG) working at College of Kecskemet. The basic idea was to connect the existing eye-tracker tools with a portable, easy-to-use EEG device. The CMM makes possible to record the eye movement and the EEG-signals in real time, therefore researchers can describe the connection between the recorded eye movements and its neurological backgrounds (Devosa, Maródi, Grósz, Búzás, & Steklács, 2015). Why is this so important for a pedagogical researcher? CMM can answer important questions about the reading process, including one of the most important one: when the reader fixated for a long time on an area (word, picture etc.), it happened because of trying to understand the meaning, or it was just for stop for relaxation (Maródi et al., 2015). Therefore the CMM could help summarising the real problematic and difficult parts of textbooks (Maródi & Devosa, 2015), which should be important part of assessments and evaluation of future textbooks (Devosa, 2014).

PARTS OF CMM

The CMM contains four essential parts:

• NeuroSky MindWave

• TÖBBI T120

• HSHP-RG MindReader 1.0

• Personal Computer running Microsoft Windows 7 Details o f NeuroSky MindWave

NeuroSky MINDWAVE: is a simplified EEG machine based on a personal computer running any of the following operating systems: Windows: XP, Vista, Windows 7, Windows 8, Windows 8.1, or MacOS : 10.7.5, 10.8.x, 10.9.x, 10.10, 10.11. With the bundled softwares this tool of brainwave technology is mostly used for playing, and the educational researchers use to record attention levels during the math, memory and pattern recognition exercises. Lots of softwares (games, educational softwares etc.) can be downloaded from the official site of the device: http://store.neurosky.com/ On figure 1 a pattern registration sample exercise can be seen. During the solution process, the subject is wearing the EEG device therefore the data is being recorded continuously.

* A . B 4 c _

T raining

• ^■

T esting m J j

• ■

•

^►

88 12 133 4 7 162 38

o □

69 31

•

152

■

₄₈

•

143

►

55

o O

₃₉ ₄₁ ^•₂₇ ^■₂₃ ^•₇₉ ^►₂₁

110“

• 56

► 44

Figure 1: Pattern recognition sample

(3)

The device itself is ideal for researches working with young pupils, because: easy to carry and handle, light, children like how it looks, easy to buy and cheap to replace, if something happens with the device in the classrooms.

Sensors can be found on the ann o f the device (figure 2):

Adjustable Head Band

Figure 2: Sematic look of MindWave

The device has two sensors which are connected to forehead and ear clip. These sensors can gain strong enough power to detect raw EEG signals (Alpha, Beta, Gamma, Delta waves). For researchers the attention, meditation levels, blink time - which are calculated data - are the most important (Salabun, 2014).

The list of record-available signals are below:

• Alpha, Beta, Gamma, Delta waves

• eSense meter for Attention

• eSense meter for Meditation

• on-head detection

• eSense Blink Detection

• blink detection (which can be a reference point to TOBII data - described later).

Specifications

• Weighs 90g

• Sensor arm up: Height: 225mm x Width: 155mm x Depth: 92mm

• Sensor Arm down: height: 225mm x width: 155mm x depth: 165mm

• 30mW rate power; 50mW max power o - 2.471 GHz RF frequency

• 6dBm RF max power

• 250kbit/s RF data rate

• 10m RF range

• 5% packet loss of bytes via wireless

• UART Baudrate: 57,600 Baud

(4)

• lmV pk-pk EEG maximum signal input range

• 3Hz - 100Hz hardware filter range

• 12 bits ADC resolution

• 512Hz sampling rate

• 1Hz eSense calculation rate (NeuroSky, 2011) Details o f TOB11 T120

TOBII T120 is a market leader brand in eye-tracking. The device is very popular among the educational and reading researchers (Steklacs, 2015) because it makes quite precise assessment, and easy to setup. The TOBII T120 uses the technique called pupil centre corneal reflection (PCCR). The PCCR uses highly visible reflections caused by a light source to illuminate the eye (Senzio-Savino, Alsharif, Gutierrez, & Yamashita, 2011). An image of the eye showing these reflections are captured by a camera, which is then used to identify the reflection of the light source on the cornea (glint) and in the pupil. The gaze direction is calculated by the direction of vector fonned by the angle between the cornea and pupil reflections combined with other geometrical features of the reflections. (See figure 3.)

Figure 3: TOBII device T120

TOBII T120 uses near-infrared illumination to create the reflection patterns on the cornea and pupil of the eye. Images of the eyes and the reflection patterns are capture by image sensors. Physiological 3D model of the eye with advanced image-processing algorithms are used to estimate the position of the

(5)

eye in space and the point of gaze with high accuracy (TOBII Library, 2015). (See figure 4.)

1An ey e tracker consists of cam eras, projectors and algorithms.

The Eye Tracker

The projectors create a pattern of near-infrared light on the eyes.

i The cam e ra s I take high-frame-rate

images of the user s eyes and the patterns.

The im age processing algorithm s find specific details in the user's eyes and reflections pattens.

B ased on these details, mathematical algorithms calculate the e y e s' position and g aze p o in t for instance on a computer monitor.

Figure 4: Working method ofTOBII T120 (TOBIIpro Library, 2015) Gaze plots and heat map

Gaze plot and heat map outputs generated by the TOBII T120 device (see figure 5) (Maródi et al., 2015), where dots represents the fixations graphically, where chubby dots specify a longer fixation period, and lines among fixations are marks the saccades. “Gaze plot" means all the fixations the subject completed on a particular illustration or webpage shown by screen shot markings (see figure 5).

Figure 5: Gaze Plot or Scanpath image.(Maródi et al., 2015 )

Gaze Plots illustrates the total eye tracking gathering or numerous subjects in a tiny period. “Heat map” (Maródi et al., 2015) also visualize eye tracking infonnation. (see figure 6)

(6)

Figure 6: Heat map image(Maródi et al., 2015)

A heat map demonstrates the quantity, and period of fixations by altered colours which subjects completed within certain areas. Green typically indicates the smallest amount of quantity of fixations or the shortest time, with altering levels in between green and red. If the subjects did not fixate in the region at all, a colourless area signifies it on a heat map. The colourless area could mean it may have been in their peripheral vision of the subjects (Maródi & Devosa, 2015).

Details o f MindReader 1.0

MindReader 1.0 is programmed in C# language (uses .NET 4.5) by Health Science and Health Promotion Research Group, College of Kecskemét. This software is essential in our studies, because this application connects the EEG data and TOBII data together. Meanwhile many EEG + eye-reading studies are in progress, according to the read literature we could not find any, where scientist could solve the problems of easy caring devices, and the precise sample timing (Bensalem-Owen, Chau, Sardam, & Fahy, 2011). The basis of our idea was to have the both devices driven from the same timing source. Therefore we decided to use devices in our studies that can be driven from one PC together and quite easy to use. (Devosa et al., 2015), because in many cases the researchers visit the school, not the subjects the lab: it is much better for the pupils to stay in well-known environment during the experimental process. Therefore have had found the MindWave device, which has stream data format as output. Our MindReader runs on the same PC receiving the data, and marking it with time stamps, fonn the same source: the timer of the PC, hence researcher can compare the EEG and TOBII data easily and precisely (see figure 7).

COMMON TIMER

Figure 7: Our idea: Complex Measuring Method (CMM)(Devosa et al., 2015)

(7)

The Mind Reader application has two important tasks to do:

1. receive the signals form MindWave tool, and displays it as a function therefore the researcher can immediately and obviously see the results;

2. write the values into a Microsoft Excel workbook for further analysis.

Inside MindReader 10

As all modern software development, the design - develop - test processes were done within an up-to- date software developing environment. Our group decided to use C#, as a programming language, hence the Visual Studio 2012 has been used to write the code (see figure 8).

EEGRecorder - Microsoft Visual Studio Express 2012 foe Window* Desktop Quick Launch (Ctrl-* Q) f i ~ O X

FILE EDIT VIEW PROJECT BUILD DEBUG TEAM TOOLS TEST WINDOW HELP

i: O - ^O d Q u J* • ► Start - Debug - Any CPU - &^_ ^J |l = jf fT^_

Fonml.es [Design]’

Forml System.WindowsForms.Form

>■

El Ac c nubility AccessibleDescnption Accessible Name

AccessibleRole Default

E Appearance

BackColor | | Control

Backgroundimage 1 1 (none) BeckgroundlmagcLsyout Tile

Cursor Default

E l Font Microsoft Sans Serif: 8,2Spt

F or eC olor Control Text

FormBorderStyie Suable

RightToLeft No

RightToLeft Lay out False

| EEG Mind Reader 1.0

UseWaitC ursor False

□ Behavior

AllowOrop False

AutoValidote EnablePreventFocusChange

ContextMenuStn p (none)

DoubleBufFered False

Enabled True

ImeMode NoControl

■s' EEG Mind Reader 1.0 1 o II S II S3 1

Record

Raw data - Empty

Lependl - Empty

51 saveFileDialogl

Error List ... ... ... * i X T • I O 0 Errors I I 0 Warnings Q 0 Messages Search Error List f i ’

“ Uata Detenption File Line ^ Colu-.. Project -*■

E) (ApplicationSettings) E) (DataBindmgs)

Too T e s t

The text associated with the control.

Properties Solution Explorer Error List Output

Ready

Figure 8: The developing phase of Mind Reader 1.0’s GUI in Visual Studio

The software has a clear and simple GUI, to make the data recording process as simple as possible.

One button implemented the GUI, which function changes according to the actual circumstances:

record => save as => start => stop.

The software’s displaying method uses the C# source code shown on figure 9. The software updates the displayed chart if any new signal received from attention or meditation data.

(8)

private void chartUpdate()

{

while (! shouldStop)

{

if (eegDataHandler != null)

{

object[] attention = eegDataHandler.attentionData.ToArray();

object[] meditation = eegDataHandler.meditationData.ToArrayO;

object[] raw = eegDataHandler.rawData.ToArrayO;

if (attention.Length > 0 && meditation.Length > 0 && raw.Length > 0)

(

attentionMeditationChart.lnvoke((Action)delegate()

{

attentionMeditationChart.Series[”Attention"].Points.Clear();

attentionMeditationChart.Series["Meditation”].Points.Clear();

for (int i = 0; i < attention.Length -1 ; i++)

{

attentionMeditationChart.Series[''Attention,,].Points.AddY(attention[i]);

}

for (int i = 0; i < meditation.Length - 1; i++)

{

attentionMeditationChart.Series["Meditation"].Points.AddY(meditation[i]);

});}

rawChart.Invoke((Action)delegate()

{

ra wChart. Series ["Raw"]. Points.Cleat));

for (int i = 0; i < raw.Length - 1; i++)

{

ra wChart. Series ["Raw"] .Points.AddY(raw[i]);

});}

}

System.Threading.Thread.Sleep(50);

}

>

Î

Figure 9: Source code of displaying method of Mind Reader 1.0 in Visual Studio

The structure of the source code makes easy to develop our software in the future for the new requisites.

EXAMPLE RESULTS FROM PRE-TESTS OF CMM

During 2015 pre-test experiences were carried out using CMM. On figure 10, it is possible to see - at the lower left comer - MindWave 1.0 working in progress during a CMM experience held in a Kecskemét primary school.

(9)

Figure 10: MindReader 1.0 in working in a CMM experience

The exercises were getting more and more difficult, in this paper two exercises are described and shown with MindReader 1.0. In the exercises the 10 years-old pupils had to find the correct route on the display of TOBII, according to the description and legend.

On figure 11 (exercise 1) a smoothed attention level record can be seen. We received quite noisy signals because of high sampling rate, that is why the smoothing. At „break downs” we believe the student changed among the exercise’s parts: the legend and the picture.

»1---1---1---1---1---1---1---

•0 4 Á

7 0 -

-

1». . / W l \ A A/S A A A I l/\ /v VW UW \ J \ , J \ \

140 \í \ v M / S r ^ T n A j v i J

5 _{30 •}v \ / \ J \ v \ \ V W \ \ A r vV ^ - |

20 10

0* I--- - í - ---- L ----¿ --- I - ---- L ----

0 100 200 300 400 500 600 700

Tim* (mc)

L aa m « g h « tib a rtt|M . Zolii m > |u lh o i M ival Zoli nam tudta, hogy hol lakik Laci, árért

•dót« ne tt egy falu in éren r«|(M é» a k ö v e ik e t* utaiééM:

indulj el aziskolÉtöl a kővetkező irányoknak (É éizak, D dal, K-kalet. ÉK aszakkalot) m egfelelően K - > 0 - > K - > K -> É K -> É

M elyik széznú h arb an lakik L a c i?

Figure 11: Exercise 1.

On figure 12 (exercise 2.) The pupil has to solve a same type exercise like the previous one. The researchers were interested in fatigue, which can be exactly recognized from the signals.

(10)

1 . M n 4 M r U k « t ( M « k e if o if d a m lr ! 2. F w * 4 M * . W U I • k n r l t r M l H i |!

S. M n j M r h t r f | t « M I

4. lo t duty e w«b> *t A k o v rlh r/ u *•< Kalo lJ k e t r li k *nyt>«l V A t « i c k liu lr i n w n i t u < H k a n t á i e n 1« l o l l 6- F*r<fc4bH»til> *»*> •'

Figure 12: Exercise 2

The results of the preliminary experiences fairly shows that CMM methodological tool works as the designers planted it to work. According to the plans of Health Science and Health Promotion Research Group (HSHP-RG) the regular experiences will start in February of 2016, and by 2017 the CMM tool will be ready for assessment and evaluate the new teaching materials before they will be used in everyday education.

REFERENCES

Bensalem-Owen, M., Chau, D. F., Sardam, S. C„ & Fahy, B. G. (2011). Education research:

evaluating the use of podcasting for residents during EEG instruction: a pilot study. Neurology, 77(8), e42-44. doi: 10.1212/WNL.0b013e31822b0017

Devosa, I. (2014). Az e-kommunikáció lehetőségei és csapdái az oktatásban. In A. Dombi & M.

Dombi (Eds.), Pedagogikum és kommunikáció: Comprehensive pedagogy> and communication.

(pp. 197-205). Szeged: Universitas Szeged Kiadó.

Devosa, I., Maródi, A., Grósz, T., Búzás, Z., & Steklács, J. (2015). The Complex Measuring Method (CMM) in education. In Z. R. Prof Aleksandar Sedmak, Simon Sedmak, Snezana Kirin (Ed.), TEAM 2015: 7th International Scientific and Expert Conference o f the International TEAM Society, (pp. 283-286). Beograd: Beograd: University of Belgrade, Faculty of Mechanical Engineering.

Maródi, A., & Devosa, I. (2015). “Kisgyermekek körében végzett eye tracking vizsgálatok

eredményeinek statisztikai elemzése’’. Paper presented at the International Conference on Eye Movements 2015.

Maródi, A., Devosa, I., Steklács, J., Fáyné Dombi, A., Búzás, Z., & Vanya, M. (2015). Eye-Tracking Analysis of the Figures of Anti-Smoking Health Promoting Periodical’s Illustrations. Practice and Theory in Systems o f Education, 10(3), 285-293. doi: 10.1515/ptse-2015-0027

(11)

NeuroSky. (2011). MindWave User Guide, from

http://developer.neurosky.com/docs/lib/exe/fetch.php?media=mindwave_user_guide_en.pdf Salabun, W. (2014). Processing and spectral analysis of the raw EEG signal from the MindWave.

Przeglad Eleketrotechniczny, 90(2), 4. doi: 10.12915/pe.2014.02.44

Senzio-Savino, B., Alsharif, M. R., Gutierrez, C. E., & Yamashita, K. (2011). The Eyes and Games: A Survey of Visual Attention and Eye Tracking Input in Video Games. Proceedings ofSBGames 2011.

Steklács, J. (2015). Szemmozgásvizsgáló Oktatásmódszertani Kutatóműhely és Labor. Anyanyelvi Pedagógia, 7(2).

TOBII Library. (2015). An introduction to eye tracking and Tobii Eye Trackers. Retrieved 2016.01.23., 2015, from http://www.tobii.com/eye-tracking-research/global/library/white- papers/tobii-eye-tracking-white-paper/

TOBIIpro Library. (2015). How do Tobii Eye Trackers work? Retrieved 2016.01.23., 2016, from http://www.tobiipro.com/learn-and-support/leam/how-do-tobii-eye-trackers-work/