Understanding Cognitive Processes Behind "I Don't Know" Responses in Sustainability Literacy Assessment Using Eye-Tracking and Usability Evaluation
DOI:
https://doi.org/10.51747/energy.v16i2.p431-444Keywords:
Eye Tracking, Hybrid System Usability Scale, Sustainability Literacy Test, User Interface, Visual AttentionAbstract
Assessing sustainability literacy is essential for ensuring that sustainability literacy tests accurately measure individuals' understanding of sustainability concepts. However, conventional evaluation methods primarily rely on test scores and response accuracy, providing limited insight into the cognitive processes underlying participants' decision-making. This study investigates the use of eye-tracking technology to analyze participants' visual attention while completing a Sustainability Literacy Test (SLT). Eye movement data, including scanpaths, were analyzed to identify user interactions with test items and determine which visual elements attracted the greatest attention, particularly when participants selected the "I Don't Know" response option. In addition, system usability was evaluated using the Hybrid System Usability Scale (H-SUS). The results indicate that participants required an average response time of more than 60 seconds for items associated with the "I Don't Know" option, suggesting increased visual attention and cognitive load in specific areas of the interface. The website achieved an average H-SUS score of 63.34, indicating an acceptable level of usability, corresponding to Grade C and an "OK" adjective rating. These findings demonstrate that eye-tracking data can reveal hidden patterns of cognitive effort and user uncertainty that cannot be captured through response accuracy alone. The proposed approach provides valuable insights for improving test design, enhancing question readability, and optimizing the user interface of web-based Sustainability Literacy Tests, thereby supporting more effective assessment of sustainability literacy.
References
[1] V. Ardaniah, Z. D. Billah, L. E. Rahmantyo, and P. Ardhianto, “Embedding Sustainability Literacy with Language Teaching in Higher Education,” JL3T (Journal of Linguistics, Literature and Language Teaching), vol. 11, no. 1, pp. 130–148, Jul. 2025, doi: 10.32505/jl3t.v11i1.11027.
[2] F. Anastasya, “Pengembangan Sistem Informasi Tes Literasi SDGs ITS Berbasis Web sebagai Media Pengukuran Literasi Mahasiswa Mengenai Pembangunan Berkelanjutan,” Institut Teknologi Sepuluh Nopember, 2026.
[3] V. Yaneva, B. E. Clauser, A. Morales, and M. Paniagua, “Using Eye-Tracking Data as Part of the Validity Argument for Multiple-Choice Questions: A Demonstration,” J. Educ. Meas., vol. 58, no. 4, pp. 515–537, Dec. 2021, doi: 10.1111/jedm.12304.
[4] V. Yaneva, B. E. Clauser, A. Morales, and M. Paniagua, “Assessing the validity of test scores using response process data from an eye-tracking study: a new approach,” Advances in Health Sciences Education, vol. 27, no. 5, pp. 1401–1422, Dec. 2022, doi: 10.1007/s10459-022-10107-9.
[5] J. Bartolomé, P. Garaizar, and L. Bastida, “Validating item response processes in digital competence assessment through eye-tracking techniques,” in ACM International Conference Proceeding Series, Association for Computing Machinery, Oct. 2020, pp. 738–746. doi: 10.1145/3434780.3436641.
[6] B. Maddox, A. P. Bayliss, P. Fleming, P. E. Engelhardt, S. G. Edwards, and F. Borgonovi, “Observing response processes with eye tracking in international large-scale assessments: evidence from the OECD PIAAC assessment,” European Journal of Psychology of Education, vol. 33, no. 3, pp. 543–558, Jul. 2018, doi: 10.1007/s10212-018-0380-2.
[7] M. G. Carneiro Queiroz et al., “Comparison of cognitive workload between very short answer questions and multiple-choice questions: an eye-tracking experiment,” Med. Educ. Online, vol. 31, no. 1, 2026, doi: 10.1080/10872981.2026.2621434.
[8] L. Hahn and P. Klein, “Clustering eye-movement data uncovers students’ strategies for coordinating equations and diagrams of vector fields,” Educational Studies in Mathematics, vol. 118, no. 3, pp. 359–385, Mar. 2025, doi: 10.1007/s10649-023-10243-y.
[9] Y. Hu, B. Wu, and X. Gu, “An Eye Tracking Study of High-and Low-Performing Students in Solving Interactive and Analytical Problems,” 2017.
[10] J. Nielsen, “How Many Test Users in a Usability Study?,” Nielsen Norman Group (NN/Group). Accessed: Apr. 27, 2026. [Online]. Available: https://www.nngroup.com/articles/how-many-test-users/
[11] A. Lopez-Cardona, P. Emami, S. Idesis, S. Duraisamy, L. A. Leiva, and I. Arapakis, “A Comparative Study of Scanpath Models in Graph-Based Visualization,” in Eye Tracking Research and Applications Symposium (ETRA), Association for Computing Machinery, May 2025. doi: 10.1145/3715669.3725882.
[12] A. Mohamed Selim, M. Barz, O. S. Bhatti, H. M. T. Alam, and D. Sonntag, “A review of machine learning in scanpath analysis for passive gaze-based interaction,” 2024, Frontiers Media SA. doi: 10.3389/frai.2024.1391745.
[13] K. Brosnan, B. Grün, and S. Dolnicar, “Cognitive load reduction strategies in questionnaire design,” International Journal of Market Research, vol. 63, no. 2, 2021.
[14] X. Wang and V. Aryadoust, “An eye-tracking and neuroimaging study of negative wording effects on cognitive load in a metacognitive awareness tool,” System, vol. 128, Feb. 2025, doi: 10.1016/j.system.2024.103552.
[15] R. Niu, W. Wu, Q. Li, and Y. Fang, “Evaluating the Effect of Font Size and Line Spacing in Chinese Online Education User Interfaces on Children’s Learning Experience,” Int. J. Hum. Comput. Interact., 2025, doi: 10.1080/10447318.2025.2524495.
[16] J. Baumgartner, N. Ruettgers, A. Hasler, A. Sonderegger, and J. Sauer, “Questionnaire experience and the hybrid System Usability Scale: Using a novel concept to evaluate a new instrument,” International Journal of Human Computer Studies, vol. 147, Mar. 2021, doi: 10.1016/j.ijhcs.2020.102575.
[17] J. Baumgartner, N. Frei, M. Kleinke, J. Sauer, and A. Sonderegger, “Pictorial System Usability Scale (P-SUS): Developing an instrument for measuring perceived usability,” in Conference on Human Factors in Computing Systems - Proceedings, Association for Computing Machinery, May 2019. doi: 10.1145/3290605.3300299.
[18] J. Brooke, “SUS: A ‘Quick and Dirty’ Usability Scale,” Usability Evaluation in Industry, 1996.
[19] S. Sugawara, N. Nangia, A. Warstadt, and S. R. Bowman, “What Makes Reading Comprehension Questions Difficult?,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Long Papers, 2022, pp. 6951–6971. [Online]. Available: https://github.
[20] S. Gruhn, E. Segers, J. Keuning, and L. Verhoeven, “Understanding variation in children’s reading comprehension: A dynamic approach,” J. Comput. Assist. Learn., vol. 40, no. 2, pp. 876–901, Apr. 2024, doi: 10.1111/jcal.12923.
[21] E. Z. Nisa’, “Analisis Usability dan Perancangan Perbaikan Interface pada Aplikasi E-Health Menggunakan Eye Tracking Technology (Studi Kasus: E-Health Surabaya),” Institut Teknologi Sepuluh Nopember Surabaya, Surabaya, 2023.
[22] R. A. Purnamasari, “Evaluasi Usabilitas dan Perbaikan Desain Website UPT Bahasa dan Budaya ITS Menggunakan Eye Tracking, Retrospective Think Aloud (RTA) dan System Usability Scale (SUS),” Institut Teknologi Sepuluh Nopember Surabaya, Surabaya, 2023.
[23] I. M. Mahendra, “Analisis Dampak Layout, Warna, dan Time Pressure pada Usabilitas Business Intelligence Dashboard menggunakan Eye Tracking, Retrospective Think Aloud (RTA), dan System Usability Scale (SUS),” 2024.
[24] J. Dobres, B. Wolfe, N. Chahine, and B. Reimer, “The effects of visual crowding, text size, and positional uncertainty on text legibility at a glance,” Appl. Ergon., vol. 70, pp. 240–246, Jul. 2018, doi: 10.1016/j.apergo.2018.03.007.
[25] M. G. Zolbin and S. Kujala, “Evaluating the Usability of a Patient Portal: Insights from Older Migrant Adults,” Stud. Health Technol. Inform., vol. 336, pp. 2344–2348, May 2026, doi: 10.3233/SHTI260689.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 ENERGY: JURNAL ILMIAH ILMU-ILMU TEKNIK
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
