This documentation is for the old eyetracker, which is no longer in use.
OpenSesame Documentation: http://osdoc.cogsci.nl/devices/pygaze/#pygaze-opensesame-plug-ins
EyeTribe Documentation: \\fustore\wamsleylab\Technical Stuff\EyeTribe Files\EyeTribe Documentation
PyGaze Github repository: https://github.com/esdalmaijer/PyGaze
EyeTribe Tracker Setup
First, the EyeTribe SDK (software development kit) must be downloaded in order for the tracker to work. Check Program Files (x86) for the EyeTribe directory, if there is none you need to download the app. IMPORTANT: the currently newest version of the SDK (v0.9.77) does not work properly with OpenSesame. You can find the installation package for the older version that works in \\fustore\wamsleylab\Technical Stuff\EyeTribe Files\TheEyeTribeSDK-0.9.56-x86.exe.
To use the EyeTribe tracker with Opensesame, you must do the following:
- Plug in the tracker to a USB 3.0 port on the computer. The tracker does not work with USB 2.0.
USB 3.0 (left) and USB 2.0 (right)
- Open the EyeTribe Server application (Found in C:\Program Files (x86)\EyeTribe\Server\EyeTribe.exe OR you can find it in Windows Start > All Programs). Make sure that the application says it is connected to the tracker before continuing. The Server application must be running whenever you want to use the tracker in OpenSesame.
As of 6/20/2016, we have found that the tracker DOES work for OpenSesame 3.0.7, but DOES NOT work for OpenSesame 2.x.
For an example of how to create a basic experiment using the EyeTribe with OpenSesame, see the OpenSesame experiment file BasicEyeTracking.osexp located at \\fustore\wamsleylab\Technical Stuff\EyeTribe Files\OpenSesameExamples (you will not actually be able to run the experiment unless the EyeTribe Server is running and the tracker is plugged in).
Within OpenSesame, there are multiple modules you need to use to operate the tracker.
- pygaze_init module: This plugin initializes the tracker, connecting OpenSesame to it, and is required in order to use the tracker. Under the "select tracker type" setting, the "EyeTribe" option must be selected. You will always want to calibrate the tracker as well when it is initialized, so make sure "Calibrate Tracker" has a check next to it. The calibration process allows the tracker to know where on the screen the participant is actually looking.
- pygaze_start_recording module: This indicates when to start collecting data from the EyeTribe, and should be at, or before, the point in the experiment where you are interested in collecting tracker data. This module must be placed some point after the pygaze_init module.
- pygaze_stop_recording module: Indicates when to stop collecting data from the EyeTribe.
Syncing Eyetracker Data with EEG Data
In order to analyze the EyeTribe data with EEG recordings, there are additional steps to the ones above to setup and analyze the data. The EyeTribe data (ET data) will need to be aligned with the simultaneous EEG data. To do this, both the ET and EEG data must have shared "events" or "triggers". In order to create these events in each data log, follow the instructions below.
First, make sure the program Input Director (website here) is installed on both the experiment room's computer as well as it's respective control room computer. This program allows us to send virtual “keypresses”, marking events, from the participant testing computer in the experiment room to the EEG recording computer in the control room. If Input Director is not installed those computers, you can find the setup program in \\fustore\wamsleylab\Technical Stuff\EyeTribe Files\InputDirector.v1.4.build110.Setup.exe or it can be downloaded here (Get the most recent version). Once installed, follow the instructions here to set the program up properly (The control room computer will be the "slave", and the experiment room computer will be the "master"). Don't worry about steps 3.2.2 and on, they don't matter for what we are doing with the program.
Next, make sure that some version of Python 2.7 is installed (you can also run the python-2.7.12 file found on the shared drive directory). You can just install it with the default settings - but when installing the program, take note of the installation directory, you will need this later. Then you need to install a python package called pywin32. This allows python programs to simulate keypresses on Windows. Download it here (or find it on the shared drive, named pywin32-220.win32-py2.7).
Now we are going to create a file that will direct OpenSesame to this new package (see the official instructions here, under the section "Customizing your environment with environment.yaml"). Open a text editor (for example Notepad), and create a new file. Copy and paste the following into the text file:
You will need to replace the first part of each of these directories to point to your python installation directory. For example, if Python were installed at "C:/Python27", I would just replace "DriveNameHere" with "C:/". Save this file as environment.yaml (no .txt at the end).
Now we need to know where to move this file so that OpenSesame can read it. Run OpenSesame, and click on the large black box area at the bottom, where it looks like code is (highlighted in red in the picture below). This is the Debug window, and serves as an interface for Python.
Type the following in, pressing return after each line:
The first line imports a package called "os". The second line uses a method from the os package called getcwd(). This method returns the current working directory, which tells us where the OpenSesame Python interpreter is operating from. Take note of the directory that is printed out.
Now, go back to where you saved the environment.yaml file, and copy/paste it into the directory that OpenSesame just printed out in the console. Once you have moved the environment.yaml file there, close OpenSesame and reopen it. Now OpenSesame can read that file. When OpenSesame reads the file, it can now find the pywin32 package (referred to as win32com in the script).
Now we will test to see the steps were performed correctly. Create a new experiment in OpenSesame, and insert an inline script module (the inline script is the white module with the half blue half yellow looking cross). Inline scripts are a way to run custom code within the OpenSesame experiment. In the inline script module, paste the following:
Run the OpenSesame experiment. If there is no error, and the experiment finishes properly, you are done. Otherwise you probably made a mistake in the environment.yaml path names or put it in the wrong place. Go through all of the steps again until you are able to run the program without an error (IMPORTANT!! Every time you edit the environment.yaml, you will need to close and reopen Opensesame for the paths to be updated within the program).
### START TO BE EDITED ### - See erins comments on the word document
Consider now, how many different events you will want to use in your experiment; 2 is the required minimum, one at the start and one at the end, however you can use additional events if you want to know when a certain stimulus or task was presented. Once you know what they will be, return to the control room computer and open "Event Descriptor" located in C:\TWin\Bin\EventDescrEditor.exe. Here you will add your events and what key press creates the event on the EEG timeline. See Creating Events in Twin for instructions on how to do this. It is important that the names of your events are numbers, for example when the key F11 is pressed, the event "10" is created to signify the start of the recording period, and when the key F12 is pressed, the event "20" is created to signify the end of the recording period. To make things easier for yourself later, make sure you write down the events you create here including: the key pressed, its label (event name), and what it signifies (start, stimulus, or end).
On the experiment room computer, open Input Director and click on the "Macros/Key Bindings" tab. Key Bindings allow you to press a key on the master computer which sends a signal to the slave computer as if the assigned key was pressed on the slave computer. We will use this to create events on the EEG data. For each event you created for Twin, you will now create a key binding to. Under Key bindings, click "Add". There are three required fields for each key binding: the key (more specifically the key on the experiment room computer which generates the signal), the target (which will always be the control room computer), and the target key (the key you want to simulate pressing on the control room computer; the event keys). For example, if I could setup the key as "Page Up", the target as the control room computer, and the target key as F11- this would mean that any time "Page Up" was pressed on the experimental computer, the control room computer would have its F11 key pressed, and if F11 had an event assigned to it an event would be created on the EEG data (See figure below). You must create a key binding for every event that you are using for your experiment.
### END NEED TO EDIT###
Next, you need to configure your OpenSesame program to actually generate the keypresses and events. As mentioned in the overview section above, you will need to use the pygaze_init, pygaze_start_recording, and pygaze_stop_recording OpenSesame modules to initialize the eyetracker and to indicate at what time during the experiment you want to start and stop logging ET data. Between the start and stop recording modules is where the code for sending EEG and eyetracker events should be placed. You can see an example of this on \\fustore\wamsleylab\Technical Stuff\EyeTribe Files\EyeTrackingWithEvents.osexp. In your OpenSesame experiment, 3 inline script modules are necessary. These necessary inline scripts are described below.
But first, a quick and important note about InputDirector. If InputDirector doesn't receive a keypress after a while (~60 seconds), InputDirector will "go to sleep". Thus, if the last simulated keypress was >60 seconds ago, we will need to "wake up" InputDirector before it will register the next keypress. As a result, you need one extra keybinding in InputDirector on the master computer, which will send a key to the slave but does not create an actual event in Twin (meaning there is no event in the Twin event database for this keypress). One absolutely necessary place to put this "wake up" keypress in the OpenSesame program is at the start of the experiment, right before the Start script described below - otherwise the start event will not be registered and recorded in Twin. It is also necessary that you add this "wake up" keypress before sending another keypress more than 60 seconds after the last keypress.
Again, refer to the example OpenSesame program to get a better idea of how this should be laid out.
The Setup Script
The setup script imports the package needed to simulate keypresses, and defines event labels as well as which hotkeys should be pressed for which events. The setup script should be placed IMMEDIATELY AFTER the pygaze_init module, and should include something similar to the script below
The Start script
The start script ultimately causes an event to be created in twin as well as the eyetracker log to indicate the starting of recording. This start script should be placed IMMEDIATELY AFTER the pygaze_start_recording module, and should include something similar to the script below:
Notice that the two lines of code in the Start Script both call the variables we defined in the Setup Script. This is why it is important to make sure the events for the Setup Script are defined correctly.
The Stop Script
Like the start script, the Stop Script ultimately creates an event in Twin as well as the eyetracker log at the same time to indicate the end of a recording. The stop script should be placed IMMEDIATELY BEFORE the pygaze_stop_recording module, and should include something similar to the script below:
By now you can see the obvious pattern used to create events on the EEG and ET data. If your study uses stimulus/task event markers, you will want to add an inline script right before the stimulus or task starts, and should follow the same pattern as above. The code below is an example of how this would be setup:
Recording EEG and ET data
To run the experiment and record EEG and ET data, you must make sure the following programs are running:
- Input Director
- OpenSesame experiment
- EyeTribe Server (see Connecting the EyeTribe Tracker above)
Make sure that the experiment and control room computers are actively connected in Input Director. Make sure the EyeTribe Server program indicates that the EyeTribe tracker is connected. When you are ready to begin recording, remember these two extremely important things:
- Make sure the the Twin recording window is the active window on the control room computer for the duration of recording- this is absolutely necessary, otherwise Twin will not generate events in response to key presses.
- Make sure the computer mouse on the experimental computer is not hovering over the EEG signals while recording, otherwise events will be placed wherever your mouse is; instead, move your mouse to the bottom of the window and keep it there.
Once everything is ready, you can begin recording in Twin, and run the OpenSesame program.
Exporting/Prepping Recorded EEG and ET Data
NOTE: This is an older method not currently used by the lab. This documentation was kept in case it was needed again.
Once done with an experiment, the EEG and ET data must be prepped a bit before analysis. For the EEG data, export the EDF from Twin. Then, export the event markers from Twin. Instructions for exporting can be found here. The ET data is saved as a .tsv file (standing for tab separated values). You can find the file located in the same directory as the OpenSesame log file, with the same name as the OpenSesame log file (but with .tsv instead).
Open the word document that contains the EEG event markers. Copy all of the text (CTRL+A), and open the excel file ASCIIEventCreator.xls which is probably located in the EyeTribe Files directory. Paste the copied text into the cell indicated by the file. Before clicking on anything else (AKA the pasted cells are still selected), click on the DATA tab in excel, and within the data tools section click on "Text to Columns". Click on "Delimited", click next, and make sure "Comma" and "Space" have checks next to them and click next again, then click Finish. If you pasted the text in the correct cells, you should see numbers and text under latency and type.Copy the first three columns, including the header down to the last event. Open Notepad++ (download here if you dont already have it) and paste the copied text. Save the file as a normal .txt file in the directory of the rest of your data. This is your event file that will be used to import markers onto the EEG timeline.
Next, we should add the event markers to the EEG timeline. To do this, open matlab, and run EEGlab. Once the EEGlab window has appeared, click "File">"Import Data">"Using the BIOSIG interface". Select the EDF file you exported from Twin. Click "File">"Import Event Info">"From Matlab array or ASCII file". The settings should be as follows
- Append: checked
- Event file: select .txt event file you created
- input field column names: latency, type, position
- number of file header lines: 1
Click "OK". Verify that the events were imported by clicking "Plot">"Channel Data (scroll)", and making sure that the events are on the EEG.
EEGLab: Syncing and Merging EEG and ET Data
NOTE: This is an older method not currently used by the lab. This documentation was kept in case it was needed again.
Once the EEG and ET data have been prepared, you are ready to combine the two data files. To do this, we will use a plugin for EEGlab, EYE-EEG. As a technical note, this plugin basically upsamples the ET data to the sampling frequency of the EEG data by interpolating the data for the ET file and then adds the ET data as additional channels to the EEG file. Make sure that this plugin is installed by going to the EEGlab directory, then look in the plugins folder. If you you do not see eye-eeg, you can copy the plugin found on the network drive at \\fustore\wamsleylab\Technical Stuff\EyeTribe Files\eye_eeg0.41. You could also download it here. Move the unzipped eye-eeg folder into the EEGlab's plugin folder.
As of 6/30/2016, the EYE-EEG plugin does not support our eyetracker. To fix this, we have edited a matlab script which should be moved into the EYE-EEG plugin's folder. You can find the edited matlab script on the network drive in the EyeTribe Files folder. It is named parsesmi.m. Copy the file, and paste it into the EYE-EEG plugin folder, replacing the original file.
You will also want to edit another matlib script. Within the eye_eeg folder, open the synchronize.m script. Once it opens, press CTRL+F and search for "radius". On about line 37, you should see the variable RADIUS defined as some number. Make sure that RADIUS is set to 200 or so. This variable is important because it determines how many samples away from the EEG events it will search for an ET event. If too low, it will not find the corresponding ET event.
The computer should now be ready to sync and merge the EEG and ET data. First, you will need to parse the ET .txt data file. To do this, open Matlab and run EEGlab. In EEGlab, you should see "Eyetracker" on the toolbar. If you do not, make sure that you have the EYE-EEG plugin in the correct place. Click on "Eyetracker", then "Parse ET data", then select "Parse eyetracker raw data", then "text file from SMI". For the first input, select your ET .txt data file, and for the second input save the Matlab object wherever you want. Check the box asking if special messages were sent. In the text box below the checkbox, delete all of the text, leaving it empty. Click OK. The Matlab window should have no errors other than a warning that you used a keyword that was empty. You should now have a .mat file in the directory you chose which contains the data extracted from the .txt file.
If not already loaded, load the EEG data file with the event markers into EEGlab. In the toolbar, click "Plot", and select "Channel Data (scroll)". Scroll through the epochs to make sure the markers are on the EEG timeline. Close the Channel Data window. Now, click on "Eyetracker" on the toolbar, and select "Import & synchronize ET". Select the parsed matlab file you just created and click OK. For "Start-event", select the event you used on both data files to signify the start of data collection. Obviously do the same for "End-event". Uncheck all of the channels you don't want to import- you should definitely remove the time and the state column because the information is unnecessary. You can find a key to these data channels below. Click OK when done selecting channels.
'timestamp': string time representation,
'time': integer timestamp in milliseconds,
'fix': Boolean indicating whether there is a fixation,
'state': integer 32bit masked tracker state,
'rawx': integer raw x gaze coordinate in pixels,
'rawy': integer raw y gaze coordinate in pixels,
'avgx': integer smoothed x gaze coordinate in pixels,
'avgx': integer smoothed y gaze coordinate in pixels,
'psize': float average pupil size,
'Lrawx': integer raw x left eye gaze coordinate in pixels,
'Lrawy': integer raw y left eye gaze coordinate in pixels,
'Lavgx': integer smoothed x left eye gaze coordinate in pixels,
'Lavgx': integer smoothed y left eye gaze coordinate in pixels,
'Lpsize': float left eye pupil size,
'Lpupilx': integer raw left eye pupil centre x coordinate,
'Lpupily': integer raw left eye pupil centre y coordinate,
'Rrawx': integer raw x right eye gaze coordinate in pixels,
'Rrawy': integer raw y right eye gaze coordinate in pixels,
'Ravgx': integer smoothed x right eye gaze coordinate in pixels,
'Ravgx': integer smoothed y right eye gaze coordinate in pixels,
'Rpsize': float right eye pupil size,
'Rpupilx': integer raw right eye pupil centre x coordinate,
'Rpupily': integer raw right eye pupil centre y coordinate
You can save the new dataset if you want. If you chose to show synchronization results in the "Import & synchronize ET data", a window should appear presenting the results. The top figure demonstrates how well the ET (top) and EEG (bottom) event markers line up on the timeline. The bottom left figure provides an R2 statistic for these two data file events. The bottom right figure shows a histogram of how many events were x-number of frames offset from each other.
You are done. Just export the file if you want to analyze it somewhere else.