A Scoping Review Of Usability Metrics For Assessing ehealth Systems
DOI:
https://doi.org/10.59645/tji.v4i1.420Keywords:
usability, eHealth, usability metrics, usability evaluationAbstract
eHealth systems have exploded in popularity worldwide in recent years, fundamentally altering how health services are delivered. However, there has been a long discussion about what usability metrics should be used to evaluate eHealth systems. This paper assesses the usability metrics mostly applied in evaluating eHealth systems. A scoping review method was used, whereas 15 papers were reviewed after being extracted from 2112 studies from PubMed, Emerald Insight, and SAGE. The search terms were "usability" in combination with "metrics", "evaluation metrics", "factors", "attributes", "framework", "models", "taxonomy", "eHealth", "health", "telehealth", and "mHealth". The study established that usability metrics, including ease of use, task-technology match, navigation, information quality, technical quality, guide and support, consistency, visibility, flexibility, accessibility, and collaboration, are mostly applied in evaluating eHealth systems’ usability. Although the metric named collaboration had a low frequency, this study recommends that it be used in assessing eHealth systems due to its necessity. Thus, the healthcare process involves multiple healthcare professionals collaborating to accomplish the patients’ healthcare process. Additionally, the study revealed limited studies on the usability of eHealth systems in developing countries, specifically Africa. Subsequently, a few African studies applied generic usability metrics only to evaluating eHealth systems compared to developed countries. Future studies should consider validating these metrics' applicability in contexts in developing countries with limited resources.
Downloads
References
Alanazi, F. (2015). Evaluating the Usability of the Laboratory Information System (LIS) in Coombe Hospital and Hail Hospital. Dublin Institute of Technology (Masters Dissertation), 1-105.
Alathas, H. (2018, November 19). How to Measure Product Usability with the System Usability Scale (SUS) Score. Retrieved November 19, 2019, from UX PLanet: https://uxplanet.org/how-to-measure-product-usability-with-the-system-usability-scale-sus-score-69f3875b858f?gi=2f7c9a7a5a5f
Aljaber, T., Gordon, N., Kambhampati, C., & Brayshaw, M. (2015). An Evaluation Framework for Mobile Health Education Software. 2015 Science and Information Conference (SAI) (pp. 786-790). London, UK: IEEE. https://doi.org/10.1109/SAI.2015.7237233
Allas, T., Bravo-Biosca, A., Phipps, J., Hart, M., Laatsit, M., & Roper, S. (2018). Innovate UK. Swindon, UK: February. Retrieved February 28, 2020, from https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/681741/17.3253_Innovate_UK_Evaluation_Framework_RatherNiceDesign_V2_FINAL_WEB.pdf
Aziz, H. A., & Abochar, H. (2015). Telemedicine. Clinical laboratory science: Journal of the American Society for Medical Technology, 28(4), 256-259. https://doi.org/10.29074/ascls.28.4.256
Bergström, A., Skeen, S., Duc, D. M., Blandon, E. Z., Estabrooks, C., Gustavsson, P., & Hoa, D. T. (2015). Health system context and implementation of evidence-based practices—development and validation of the Context Assessment for Community Health (COACH) tool for low- and middle-income settings. Implementation Science, 10(120), 1-15. https://doi.org/https://doi.org/10.1186/s13012-015-0305-2
Bosch, B., & Mansell, H. (2015). Interprofessional collaboration in health care: Lessons to be learned from competitive sports. Canadian Pharmacists Journal (CPJ), 148(4), 176-179. https://doi.org/10.1177_1715163515588106
Broekhuis, M., Velsen, L. v., Peute, L., Halim, M., & Hermens, H. (2021). Conceptualizing usability for the eHealth context: A content analysis of usability problems of eHealth applications. Journal of Medical Internet Research (JIMR), 5(7), e18198. https://doi.org/10.2196/18198
Broekhuis, M., Velsen, L., & Hermens, H. (2019). Assessing usability of eHealth technology: A comparison of usability benchmarking instruments. International Journal of Medical Informatics, 128, 24-32. https://doi.org/https://doi.org/10.1016/j.ijmedinf.2019.05.001
Brown, W., Schannal, R., Rojas, M., & Yen, P.-Y. (2013). Assessment of the health IT usability evaluation model (Health-ITUEM) for evaluating mobile health (mHealth) technology. Journal of Biomedical Informatics, 46(6), 1080-1078. https://doi.org/10.1016/j.jbi.2013.08.001
Burney, S. A., Ali, S. A., Ejaz, A., & Siddiqui, F. A. (2017). Discovering the Correlation between Technology Acceptance Model and Usability Model and Usability. International Journal of Computer Science and Network Security (IJCSNS), 17(11), 53-61. Retrieved from https://www.researchgate.net/publication/321883220_Discovering_the_Correlation_between_Technology_Acceptance_Model_and_Usability
Busagala, L. S., & Kawono, G. C. (2013). Underlying Challenges of E-Health Adoption in Tanzania. International Journal of Information and Communication Technology Research, 3(1), 34-41. Retrieved from http://www.esjournals.org
Chen, Y., Chen, H., Huang, W., & Ching, R. (2006). E-government strategies in devel-oped and developing countries: An implementation framework and case study. Journal of Global Information Management (JGIM), 14(1), 23-46.
Dowding, D., & Merrill, J. A. (2018). The Development of Heuristics for Evaluation of Dashboard Visualizations. Applied Clinical Informatics, 9(3), 511-518. https://doi.org/10.1055/s-0038-1666842
Farzandipour, M., Riazi, H., & Jabali, M. S. (2018). Proposing Electronic Health Record Usability Requirements Based on Enriched ISO 9241 Metric Usability Model. Acta Informica Medica, 26(1), 29-34. https://doi.org/10.5455/aim.2018.26.29-34
Fynn, J. F., Hardeman, W., Milton, K., & Jones, A. P. (2020). A scoping review of evaluation frameworks and their applicability to real-world physical activity and dietary change programme evaluation. BMC Public Health, 20(1000). https://doi.org/https://doi.org/10.1186/s12889-020-09062-0
Gregory, M., & Tembo, S. (2017). Implementation of E-health in Developing Countries Challenges and Opportunities: A Case of Zambia. Science and Technology, 7(2), 41-53.
Halim, M. (2019). The eHealth Usability Matrix: Developing a usability evaluation framework for patient-facing eHealth technologies (Bachelor Thesis). Enschede, Netherlands: University of Tewente. Retrieved from http://essay.utwente.nl/78323/1/Halim_BA_BMS.pdf
Hamad, W. B. (2019). Current position and challenges of ehealth in Tanzania: A review of literature. Glopbal Scientific Journa (GSJ) , 7(9), 364-376. Retrieved August 14, 2020, from http://www.academia.edu/attachments/60627529/download_file?ct=MTU5NzQxODg0MywxNTk3NDE4ODQzLDY1MjMyMTk=&s=swp-toolbar&iid=0280d766-c855-4659-b330-164888bde483
Hardenbol, A. X., Knols, B., Louws, M., Meulendijk, M., & Askari, M. (2020). Usability aspects of medication- related decision support systems in the outpatient setting: A systematic literature review. Health Informatics Journal, 26(1), 72 –87. https://doi.org/https://doi.org/10.1177/1460458218813732
Heeks, R. (2002). Information Systems and Developing Countries: Failure, Success, and Local Improvisations. The Information Society An International Journal, 101-112.
Heponiemi, T., Kujala, S., Vainiomäki, S., Vehko, T., Lääveri, T., Vänskä, J., . . . Hyppönen, H. (2019). Usability Factors Associated With Physicians' Distress and Information System-Related Stress: Cross-Sectional Survey. JMIR Med Inform, 7(4), e13466. https://doi.org/ 10.2196/13466
Househ, M. S., Shubair, M. M., Yunus, F., Jamal, A., & Aldossari, B. (2015). The Use of an Adapted Health IT Usability Evaluation Model (Health-ITUEM) for Evaluating Consumer Reported Ratings of Diabetes mHealth Applications: Implications for Diabetes Care and Management. Acta Inform Med, 23(5), 290-5. https://doi.org/10.5455/aim.2015.23.290-295
Hyppönen, H., Kaipio, J., Heponiemi, T., Lääveri, T., Aalto, A.-M., Vänskäi, J., & Elovainio, M. (2019). Developing the National Usability-Focused Health Information System Scale for Physicians: Validation Study. Journal of Internet Medical Research (JIMR), 21(5), e12875.
Inal, Y. (2019). Heuristic-based user interface evaluation of the mobile centralized doctor appointment system: A case study. The Electronic Library, 37(1), 81-94. https://doi.org/https://doi.org/10.1108/EL-06-2018-0114
International Organization for Standardization. (1998). ISO 9241-11:1998 - Ergonomic requirements for office work with visual display terminals (VDTs) Part 11 Guidance on usability. Geneva: International Organization for Standardization.
International Organization for Standardization. (2018). ISO 9241-11:2018 - Usability: Definitions and concepts. Geneva: ISO. Retrieved January 13, 2020, from International Standards Organization: https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en
International Organization for Standardization. (2018). ISO-9241-11:2018 Ergonomics of human-system interaction — Part 11 Usability: Definitions and concepts. Switzerland: ISO. Retrieved January 13, 2020, from International Standards Organization: https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en
Islam, M. N., Karim, M. M., Inan, T. T., & Islam, A. K. (2020). Investigating usability of mobile health applications in Bangladesh. BMC Medical Informatics and Decision Making, 20(19), 1-13. https://doi.org/https://doi.org/10.1186/s12911-020-1033-3
Jacobs, M., Hopkins, J., Mumber, M., & Mynatt, E. (2019). Usability Evaluation of an Adaptive Information Recommendation System for Breast Cancer Patients. AMIA Annual Simposium Proceedings (pp. 494–503). AMIA.
Johnson, S. G., Potrebny, T., Larun, L., Ciliska, D., & Olsen, N. R. (2020). Usability Methods and Attributes Reported in Usability Studies of Mobile Apps for Health Care Education: Protocol for a Scoping Review. JMIR Res Protoc, 9(8), e19072. https://doi.org/10.2196/19072
Kavuta, K. D., & Magoiga, P. C. (2023). The assessment of the usability of health information systems in referral hospitals in Tanzania: A case of Geita Regional Referral Hospital. The Journal of Informatics, 3(1), 68-82. https://doi.org/https://doi.org/10.59645/tji.v3i1.108
Kavuta, K., Msanjila, S., & Shidende, N. (2023). Assessment of the Influence of Contextual Issues in Evaluating eHealth Systems Usability in Tanzania. East African Journal of Information Technology,, 6(1), 155-170. https://doi.org/https://orcid.org/0000-0003-4218-7144
Kawakyu, N., Nduati, R., Munguambe, K., Coutinho, J., Mburu, N., DeCastro, G., . . . Gimbel, S. (2019). Development and Implementation of a Mobile Phone–Based Prevention of Mother-To-Child Transmission of HIV Cascade Analysis Tool:Usability and Feasibility Testing in Kenya and Mozambique. JMIR Mhealth Uhealth, 7(5), 1-11. https://doi.org/10.2196/13963
Khajouei, R., & Farahani, F. (2020). A combination of two methods for evaluating the usability of a hospital information information system. BMC Medical Informatics and Decision Making, 20(1), 84-93,. https://doi.org/doi: 10.1186/s12911-020-1083-6
Khan, M. T., Qamara, S., & Sinha, R. R. (2017). Adaptation of Open Source Software for Healthcare in India: Cas of CARE2X. Journal of Engineering Applied Science, 12(10), 2603-2612. https://doi.org/10.3923/jeasci.2017.2603.2612
Kikoba, B., Sukums, F., Abel, Z., Shidende, N., Kijuu, Y., Ilomo, S., & Kisanga, A. (2019). Assessing How Health Data Systems Support Data Use in the Tanzanian Health System. Conference: 12th Health Informatics in Africa Conference HELINA’ 19 (pp. 1-7). Gaborone, Botswana: Journal of Health Infoarmatics in Africa.
Krishna, S., & Walsham, G. (2005). Implementing public information systems in developing countries: Learning from a success story,. Information Technology for Development, 11(2), 123-140.
Liew, M. S., Zhang, J., See, J., & Ong, Y. L. (2019). Usability Challenges for Health and Wellness Mobile Apps: Mixed-Methods Study Among mHealth Experts and Consumers. JMIR Mhealth Uhealth, 7(1).
Lungo, J. (2008). The reliability and usability of district health information software: case studies from Tanzania. Tanzania Journal of health Research, 10(1), 39-45.
Maguire, M. (2001). Context of Use within usability activities. International Journal of Human-Computer Studies, 55, 453-483.
Ministry of Health, Community Development, Gender, Elderly and Children [MOHCDGEC]. (2019). Digital Health Strategy July 2019 - June 2024. Dodoma: United Republic of Tanzania.
Mtebe, J. S., & Nakaka, R. (2018). Assessing Electronic Medical Record System Implementation at Kilimanjaro Christian Medical Center, Tanzania. Journal of Health Informatics in Developing Countries, 1-16.
Mugo, D. M., & Nzuki, D. (2014). Determinants of Electronic Health in Developing Countries. International Journal of Arts and Commerce, 3(3), 49-60.
Nielsen, J. (1993). Usability Engineering. San Francisco: Mogan Kaufmann.
Nielsen, J. (2022, February 8th). Inroduction to Ussasbility: Current Issues in Web Ussability. Retrieved from NN Group: https://www.nngroup.com/articles/usability-101-introduction-to-usability/. [Accessed 2022 February 8th].
Niranjanamurthy, M., Archikam, N., Himaja, G., & Shetty, P. K. (2014). Research Study on Importance of Usability Testing/ User Experience (UX) Testing. International Journal od Computer Science, 3(10), 78 - 85.
Noraziani, K., Ain, A. N., Azhim, M., & Eslami, S. R. (2013). An Overview of Electronic Medical Record Implementation in Healthcare System: Lesson to Learn. World Applied Sciences Journal, 25(2), 323-332.
Peltola, J. (2019). Adoption and Use of Hospital Information systems in Developing Countries: Experiences of Health Care Personnel and Hospital Management in Tanzania. Tampere: Tampere University.
Price, M., Weber, J., Bellwood, P., Diemert, S., & Habibi, R. (2016). Evaluation of eHealth System Usability and Safety. In F. Lau, & C. Kuziemsky, Handbook of eHealth Evaluation: An Evidence-based Approach (pp. 337-350). Victoria, Canada: University of Victoria.
Ramadan, A. A., Jackson-Thompson, J., & Schmaltz, C. L. (2017). Usability Assessment of the Missouri Cancer Registry’s Published Interactive Mapping Reports: Round One. JMIR Human Factors, 4(3), e19. https://doi.org/10.2196/humanfactors.7899
Rodriguez-Paras, C., & Sasangohar, F. (2017). Usability Assessment of a Post-Traumatic Stress Disorder (PTSD) mHealth App. Proceedings of the Human Factors and Ergonomics Society 2017 Annual Meeting Texas. 61, pp. 1824-1828. USA: SAGE Journals. https://doi.org/https://doi.org/10.1177/1541931213601
Sadiku, M. N., Shadare, A. E., & Musa, S. M. (2017). Mobile Health. International Journal of Engineering Research, 6(11), 450-452.
Sauro, J., & Lewis, J. R. (2016). Standardized Usability Questionnaires. In J. Sauro, & J. R. Lewis, Quantifying the User Experience: Practical Statistics for User Research (pp. 185–240). Elsevier. https://doi.org/10.1016/b978-0-12-384968-7.0000
Sousa, V. E., & Lopez, K. D. (2017). Towards Usable e-Health: A Systematic Review of Usability Questionnaires. Applied Clinical Informatics, 8, 470-489.
Squires, J. E., Aloisio, L. D., Grimshaw, J. M., Bashir, K., Dorrance, K., & Coughlin, M. (2019). Attributes of context relevant to healthcareprofessionals’use of research evidence inclinical practice: a multi-study analysis. Implementation Science, 14(52), 1-14. https://doi.org/https://doi.org/10.1186/s13012-019-0900-8
Taiwo, O., Awodele, O., & Kuyoro, S. (2016). A Usability Framework for Electronic Health Records in Nigerian Healthcare Sector. International Journal of Computer Science Engineering (IJCSE), 5(1), 16-20.
Tao, D., Shao, F., Wang, H., Yan, M., & Qu, X. (2020). Integrating usability and social cognitive theories with the technology acceptance model to understand young users’ acceptance of a health information portal. Health Informatics Journal, 26(2), 1347-1362. https://doi.org/10.1177/1460458219879337
The Republic of South Africa. (2019). National Digital Health Strategy for South Africa. Pretoria: National Department of Health.
The Republic of Kenya. (2016). Kenya National eHealth Policy 2016 – 2030. Nairobi: Republic of Kenya: Ministry of Health.
Tiihonen, T., Vesisenaho, M., & Sutinen, E. (2008). Concept on Context: IS Meeting Context in Developing Countries. Technology for Innovation and Education in Developing Countries.
Tyllinen, M., Kaipio, J., & Laaveri, T. (2018). A framework for usability evaluation in EHR procurement. Studies in Health Technology and Informatics, 247(1), 446-450. https://doi.org/10.3233/978-1-61499-852-5-446
Wallace, S., Reid, A., Clinciu, D., & Kang, J. S. (2013). Culture and the importance of usability attributes. Information Technology & People, 26(1). https://doi.org/10.1108/09593841311307150
Wasylewicz, A. T., & Scheepers-Hoeks, A. M. (2019). Clinical Decision Support Systems. In K. P, D. M, & D. A, Fundamentals of Clinical Data Science [Internet]. Springer.
Zahabi, M., Kaber, D. B., & Swangnetr, M. (2015). Usability and Safety in Electronic Medical Records Interface Design: A Review of Recent Literature and Guideline Formulation. Human factors, 57(5), 805-834.
Zakaria, N., Wahabi, H., & Qahtani, M. A. (2020). Development and usability testing of Riyadh Mother and Baby Multi-center cohort study registry Nasriah. Journal of Infection and Public Health, 13(10), 1473-1480. https://doi.org/https://doi.org/10.1016/j.jiph.2020.02.035
Downloads
Published
How to Cite
Issue
Section
