17 Machine Learning for Gaze Prediction in Eye-Tracking Systems

Faculty Mentor: Haohan Zhang (Mechanical Engineering, University of Utah)

Introduction

Commercial eye trackers like the Neon (Pupil Labs, Berlin, Germany) use high-resolution images of the eyes along with proprietary, black-box algorithms to estimate gaze direction (Baumann & Dierkes, 2023). However, these commercial trackers are inaccessible to many users due to their high cost and their black box nature, which make it difficult to integrate with other devices such as robotic neck braces (Demaree, 2024; Chang, 2020). The objective of this research project is to build alternative machine learning (ML) models that estimate eye gazes with near-commercial accuracy, thereby potentially reducing the cost of commercial trackers and enabling cross-platform integrations.

Data Collection, Preprocessing, and Model Development

To attain this objective, we first used the Neon tracker to collect raw ground truth data, consisting of eye images as input and the corresponding 2D screen space (x, y) gaze coordinates as target outputs for each frame. Specifically, the data were collected from 4 different young healthy adults (3 male and 1 female, age 23-27) wearing the Neon tracker.

During data collection, participants were instructed to perform everyday activities that involved natural visual exploration (e.g., navigating a hallway, climbing stairs, working at a desk with a computer, engaging in a conversation), both in indoor and outdoor settings, with the head free to move. Data recorded from each participant ranged between 5 to 10 minutes.

This raw subject data was then transformed in an OpenCV, Pandas, and PyTorch pre-processing pipeline. In this pipeline, gaze coordinates were normalized to scale values between [0,1]. Additionally, eye images were down sampled by a factor of 3 in each dimension, resulting in a compressed image size of 128 by 64 pixels. A downsampling factor of 3 was chosen based on preliminary testing to balance GPU memory efficiency while preserving key features in the eye images. Overall, our data collection and preprocessing stage yielded a training pipeline consisting of 380,000 total frames (or 32 minutes of video).

We randomly selected data from 3 participants to develop a training set, and used data from the last remaining participant to develop a testing set. As a result, the training dataset consisted of 300,000 frames and the testing dataset consisted of 85,000 frames.

Based on this processed data, we developed end-to-end deep learning models for gaze prediction. Convolutional neural network (CNN) architectures were chosen in our model development, as they excel at extracting spatial features from images and have been used in previous literature for similar eye-tracking tasks (Krafka, 2016; Yang, 2023; Pathirana, 2022). We evaluated multiple specific models based on the CNN architecture, including a pre-trained ResNet-18 CNN model, variations of a custom 2D CNN model, a spatiotemporal (2+1)D CNN model, and a new CNN model augmented with gated recurrent units (GRU).

Results & Discussion

After initial experimentation and prototyping across the aforementioned architectures, two of the most promising models were identified. First, a spatiotemporal model that stacks 10 historical frames as input channels and captures temporal dynamics of the eye movements. Second, a custom 6-layer 2D CNN model consisting of 4 convolutional layers and 2 fully connected layers. To fine-tune these two promising models, we evaluated the effects of hyperparameters such as learning rate, batch size, weight decay, and optimizer type on the mean squared error of training and validation sets.

Ultimately, these fine-tuned models were evaluated on a testing set containing data from an unseen individual, collected across both indoor and outdoor environments and various activities such as standing, walking, and computer tasks. Specifically, the custom 6 Layer CNN model achieved a mean squared error of 0.0038, corresponding to an average euclidean error of 6.16%. Similarly, the stacked CNN model achieved a mean squared error of 0.0043, corresponding to an average euclidean error of 6.56%. These results demonstrate that our CNN-based deep learning models can estimate eye gaze with near-commercial accuracy, and maintain robust performance across unseen and diverse testing conditions (e.g., lighting, different subjects, differences in eye-tracker fitting). Furthermore, because we achieved this level of accuracy using downsampled, lower-resolution images, it is plausible that lower-cost, lower-resolution cameras can be used to obtain accurate tracking performance.

In addition to these findings, an interesting takeaway from our experiment is that the stacked CNN model performed comparably but slightly worse than the single CNN model. We initially hypothesized that incorporating information from the previous 10 spaced-out frames (which captures 1 second of historical context) would enhance performance. However, the superior results of the single-frame CNN model suggest that the most relevant predictive cues for eye tracking may be largely contained within a single frame. Thus, we propose that including temporal context may introduce additional noise rather than meaningful information, which negatively impacts model accuracy.

Extension

After identifying our 6-layer CNN as the best-performing model, we extended our project to investigate whether similar accuracy can be achieved using limited input data where only one eye image is used. Our rationale was that images from both eyes are similar and therefore it may be possible that a single eye image contains sufficient information for accurate gaze prediction. This single-eye approach was motivated by practical considerations for future research. For example, integrating gaze sensing into wearable robotic systems is simpler and more feasible when relying on a single eye camera, because it avoids the complexities of setting up, cropping, and synchronizing two camera inputs, enables simpler mechanical design for camera attachment, and minimizes the obstruction of the field of vision due to the cameras.

To that end, we trained a separate instance of our 6-layer 2D CNN model using only right eye images as input. To accommodate the difference in input dimensions, we made a minor adjustment to one of the model’s fully connected layers, ensuring a fair comparison with the original dual-eye model. When evaluated on a testing set, the single-eye model achieved a mean squared error of 0.0127, corresponding to an average euclidean error of 11.3%.

While the single-eye model did not match the accuracy of the dual-eye model, it still achieved reasonable accuracy given the limited input. This result suggests that some predictive power is retained even when using only one eye. However, the performance gap does show the value of incorporating both eye views for more accurate gaze estimation.

Conclusion

To conclude, results from this project are promising, representing an important first step towards accessibility of low-cost eye tracking technology and new capability to integrate gaze-sensing with robotic systems. As a future direction, we will work on integrating our models with wearable robotics systems (such as a neck exoskeleton developed in the Utah Wearable Robotics Laboratory), to evaluate its real world usage in helping people with mobility limitations.

Bibliography

Baumann, C., & Dierkes, K. (2023). Neon accuracy test report. Pupil Labs, 10.

Chang, B. C., Zhang, H., Trigili, E., & Agrawal, S. K. (2020, November). Bio-inspired gaze-driven robotic neck brace. In 2020 8th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob) (pp. 545-550). IEEE.

Demaree, D., Brignone, J., Bromberg, M., & Zhang, H. (2024). Preliminary study on effects of neck exoskeleton structural design in patients with amyotrophic lateral sclerosis. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 32, 1841-1850.

Krafka, K., Khosla, A., Kellnhofer, P., Kannan, H., Bhandarkar, S., Matusik, W., & Torralba, A. (2016). Eye tracking for everyone. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2176-2184).

Pathirana, P., Senarath, S., Meedeniya, D., & Jayarathna, S. (2022). Eye gaze estimation: A survey on deep learning-based approaches. Expert Systems with Applications, 199, 116894.

Yang, H., Yang, Z., Liu, J., & Chi, J. (2023, February). A new appearance-based gaze estimation via multi-modal fusion. 3rd International Conference on Neural Networks, Information and Communication Engineering (NNICE) (pp. 498-502). IEEE.