Background: The lack of an international standard for assessing and communicating health app quality and the lack of consensus about what makes a high-quality health app negatively affect the... Show moreBackground: The lack of an international standard for assessing and communicating health app quality and the lack of consensus about what makes a high-quality health app negatively affect the uptake of such apps. At the request of the European Commission, the international Standard Development Organizations (SDOs), European Committee for Standardization, International Organization for Standardization, and International Electrotechnical Commission have joined forces to develop a technical specification (TS) for assessing the quality and reliability of health and wellness apps.Objective: This study aimed to create a useful, globally applicable, trustworthy, and usable framework to assess health app quality.Methods: A 2-round Delphi technique with 83 experts from 6 continents (predominantly Europe) participating in one (n=42, 51%) or both (n=41, 49%) rounds was used to achieve consensus on a framework for assessing health app quality. Aims included identifying the maximum 100 requirement questions for the uptake of apps that do or do not qualify as medical devices. The draft assessment framework was built on 26 existing frameworks, the principles of stringent legislation, and input from 20 core experts. A follow-up survey with 28 respondents informed a scoring mechanism for the questions. After subsequent alignment with related standards, the quality assessment framework was tested and fine-tuned with manufacturers of 11 COVID-19 symptom apps. National mirror committees from the 52 countries that participated in the SDO technical committees were invited to comment on 4 working drafts and subsequently vote on the TS.Results: The final quality assessment framework includes 81 questions, 67 (83%) of which impact the scores of 4 overarching quality aspects. After testing with people with low health literacy, these aspects were phrased as "Healthy and safe," "Easy to use," "Secure data," and "Robust build." The scoring mechanism enables communication of the quality assessment results in a health app quality score and label, alongside a detailed report. Unstructured interviews with stakeholders revealed that evidence and third-party assessment are needed for health app uptake. The manufacturers considered the time needed to complete the assessment and gather evidence (2-4 days) acceptable. Publication of CEN-ISO/TS 82304-2:2021 Health software - Part 2: Health and wellness apps - Quality and reliability was approved in May 2021 in a nearly unanimous vote by 34 national SDOs, including 6 of the 10 most populous countries worldwide.Conclusions: A useful and usable international standard for health app quality assessment was developed. Its quality, approval rate, and early use provide proof of its potential to become the trusted, commonly used global framework. The framework will help manufacturers enhance and efficiently demonstrate the quality of health apps, consumers, and health care professionals to make informed decisions on health apps. It will also help insurers to make reimbursement decisions on health apps. Show less
Background: The COVID Radar app was developed as a population-based surveillance instrument to identify at-risk populations and regions in response to the COVID-19 pandemic. The app boasts of >8... Show moreBackground: The COVID Radar app was developed as a population-based surveillance instrument to identify at-risk populations and regions in response to the COVID-19 pandemic. The app boasts of >8.5 million completed questionnaires, with >280,000 unique users. Although the COVID Radar app is a valid tool for population-level surveillance, high user engagement is critical to the success of the COVID Radar app in maintaining validity. Objective: This study aimed to identify optimization targets of the COVID Radar app to improve its acceptability, adherence, and inclusiveness. Methods: The main component of the COVID Radar app is a self-report questionnaire that assesses COVID-19 symptoms and social distancing behaviors. A total of 3 qualitative substudies were conducted. First, 3 semistructured focus group interviews with end users (N=14) of the app were conducted to gather information on user experiences. The output was transcribed and thematically coded using the framework method. Second, a similar qualitative thematic analysis was conducted on 1080 end-user emails. Third, usability testing was conducted in one-on-one sessions with 4 individuals with low literacy levels. Results: All 3 substudies identified optimization targets in terms of design and content. The results of substudy 1 showed that the participants generally evaluated the app positively. They reported the app to be user-friendly and were satisfied with its design and functionalities. Participants' main motivation to use the app was to contribute to science. Participants suggested adding motivational tools to stimulate user engagement. A larger national publicity campaign for the app was considered potentially helpful for increasing the user population. In-app updates informing users about the project and its outputs motivated users to continue using the app. Feedback on the self-report questionnaire, stemming from substudies 1 and 2, mostly concerned the content and phrasing of the questions. Furthermore, the section of the app allowing users to compare their symptoms and behaviors to those of their peers was found to be suboptimal because of difficulties in interpreting the figures presented in the app. Finally, the output of substudy 3 resulted in recommendations primarily related to simplification of the text to render it more accessible and comprehensible for individuals with low literacy levels Conclusions: The convenience of app use, enabling personal adjustments of the app experience, and considering motivational factors for continued app use (ie, altruism and collectivism) were found to be crucial to procuring and maintaining a population of active users of the COVID Radar app. Further, there seems to be a need to increase the accessibility of public health tools for individuals with low literacy levels. These results can be used to improve the this and future public health apps and improve the representativeness of their user populations and user engagement, ultimately increasing the validity of the tools. Show less
Background: The COVID Radar app was developed as a population-based surveillance instrument to identify at-risk populations and regions in response to the COVID-19 pandemic. The app boasts of >8... Show moreBackground: The COVID Radar app was developed as a population-based surveillance instrument to identify at-risk populations and regions in response to the COVID-19 pandemic. The app boasts of >8.5 million completed questionnaires, with >280,000 unique users. Although the COVID Radar app is a valid tool for population-level surveillance, high user engagement is critical to the success of the COVID Radar app in maintaining validity.Objective: This study aimed to identify optimization targets of the COVID Radar app to improve its acceptability, adherence, and inclusiveness.Methods: The main component of the COVID Radar app is a self-report questionnaire that assesses COVID-19 symptoms and social distancing behaviors. A total of 3 qualitative substudies were conducted. First, 3 semistructured focus group interviews with end users (N=14) of the app were conducted to gather information on user experiences. The output was transcribed and thematically coded using the framework method. Second, a similar qualitative thematic analysis was conducted on 1080 end-user emails. Third, usability testing was conducted in one-on-one sessions with 4 individuals with low literacy levels.Results: All 3 substudies identified optimization targets in terms of design and content. The results of substudy 1 showed that the participants generally evaluated the app positively. They reported the app to be user-friendly and were satisfied with its design and functionalities. Participants’ main motivation to use the app was to contribute to science. Participants suggested adding motivational tools to stimulate user engagement. A larger national publicity campaign for the app was considered potentially helpful for increasing the user population. In-app updates informing users about the project and its outputs motivated users to continue using the app. Feedback on the self-report questionnaire, stemming from substudies 1 and 2, mostly concerned the content and phrasing of the questions. Furthermore, the section of the app allowing users to compare their symptoms and behaviors to those of their peers was found to be suboptimal because of difficulties in interpreting the figures presented in the app. Finally, the output of substudy 3 resulted in recommendations primarily related to simplification of the text to render it more accessible and comprehensible for individuals with low literacy levels.Conclusions: The convenience of app use, enabling personal adjustments of the app experience, and considering motivational factors for continued app use (ie, altruism and collectivism) were found to be crucial to procuring and maintaining a population of active users of the COVID Radar app. Further, there seems to be a need to increase the accessibility of public health tools for individuals with low literacy levels. These results can be used to improve the this and future public health apps and improve the representativeness of their user populations and user engagement, ultimately increasing the validity of the tools. Show less
Background: During the COVID-19 pandemic, several home monitoring programs have described the success of reducing hospital admissions, but only a few studies have investigated the experiences of... Show moreBackground: During the COVID-19 pandemic, several home monitoring programs have described the success of reducing hospital admissions, but only a few studies have investigated the experiences of patients and health care professionals.Objective: The objective of our study was to determine patients' and health care professionals' experiences and satisfaction with employing the COVID-box.Methods: In this single-center, retrospective, observational study, patients and health care professionals were asked to anonymously fill out multiple-choice questionnaires with questions on a 5-point or 10-point Likert scale. The themes addressed by patients were the sense of reassurance and safety, experiences with teleconsultations, their appreciation for staying at home, and the instructions for using the COVID-box. The themes addressed by health care professionals who treated patients with the COVID-box were the characteristics of the COVID-box, the technical support service and general satisfaction, and their expectations and support for this telemonitoring concept. Scores were interpreted as insufficient (<= 2 or <= 5, respectively), sufficient (3 or 6-7, respectively), or good (>= 4 or >= 8, respectively) on a 5-point or 10-point Likert scale.Results: A total of 117 patients and 25 health care professionals filled out the questionnaires. The median score was 4 (IQR 4-5) for the sense of safety, the appreciation for staying at home, and experiences with teleconsultations, with good scores from 76.5% (88/115), 86% (56/65), and 83.6% (92/110) of the patients, respectively. Further, 74.4% (87/117) of the patients scored the home monitoring program with a score of >= 8. Health care professionals scored the COVID-box with a minimum median score of 7 (IQR 7-10) on a 10-point scale for all domains (ie, the characteristics of the COVID-box and the technical support service and general satisfaction). For the sense of safety, user-friendliness, and additional value of the COVID-box, the median scores were 8 (IQR 8-10), 8 (IQR 7-9), and 10 (IQR 8-10), respectively, with good scores from 86% (19/22), 75% (15/20), and 96% (24/25) of the health care professionals, respectively. All health care professionals (25/25, 100%) gave a score of >= 8 for supporting this home monitoring concept, with a median score of 10 (IQR 10-10).Conclusions: The positive experiences and satisfaction of involved users are key factors for the successful implementation of a novel eHealth solution. In our study, patients, as well as health care professionals, were highly satisfied with the use of the home monitoring program-the COVID-box project. Remote home monitoring may be an effective approach in cases of increased demand for hospital care and high pressure on health care systems. Show less
Background: During the COVID-19 pandemic, several home monitoring programs have described the success of reducing hospital admissions, but only a few studies have investigated the experiences of... Show moreBackground: During the COVID-19 pandemic, several home monitoring programs have described the success of reducing hospital admissions, but only a few studies have investigated the experiences of patients and health care professionals.Objective: The objective of our study was to determine patients’ and health care professionals’ experiences and satisfaction with employing the COVID-box.Methods: In this single-center, retrospective, observational study, patients and health care professionals were asked to anonymously fill out multiple-choice questionnaires with questions on a 5-point or 10-point Likert scale. The themes addressed by patients were the sense of reassurance and safety, experiences with teleconsultations, their appreciation for staying at home, and the instructions for using the COVID-box. The themes addressed by health care professionals who treated patients with the COVID-box were the characteristics of the COVID-box, the technical support service and general satisfaction, and their expectations and support for this telemonitoring concept. Scores were interpreted as insufficient (≤2 or ≤5, respectively), sufficient (3 or 6-7, respectively), or good (≥4 or ≥8, respectively) on a 5-point or 10-point Likert scale.Results: A total of 117 patients and 25 health care professionals filled out the questionnaires. The median score was 4 (IQR 4-5) for the sense of safety, the appreciation for staying at home, and experiences with teleconsultations, with good scores from 76.5% (88/115), 86% (56/65), and 83.6% (92/110) of the patients, respectively. Further, 74.4% (87/117) of the patients scored the home monitoring program with a score of ≥8. Health care professionals scored the COVID-box with a minimum median score of 7 (IQR 7-10) on a 10-point scale for all domains (ie, the characteristics of the COVID-box and the technical support service and general satisfaction). For the sense of safety, user-friendliness, and additional value of the COVID-box, the median scores were 8 (IQR 8-10), 8 (IQR 7-9), and 10 (IQR 8-10), respectively, with good scores from 86% (19/22), 75% (15/20), and 96% (24/25) of the health care professionals, respectively. All health care professionals (25/25, 100%) gave a score of ≥8 for supporting this home monitoring concept, with a median score of 10 (IQR 10-10).Conclusions: The positive experiences and satisfaction of involved users are key factors for the successful implementation of a novel eHealth solution. In our study, patients, as well as health care professionals, were highly satisfied with the use of the home monitoring program—the COVID-box project. Remote home monitoring may be an effective approach in cases of increased demand for hospital care and high pressure on health care systems. Show less
Background: During the COVID-19 pandemic, several home monitoring programs have described the success of reducing hospital admissions, but only a few studies have investigated the experiences of... Show moreBackground: During the COVID-19 pandemic, several home monitoring programs have described the success of reducing hospital admissions, but only a few studies have investigated the experiences of patients and health care professionals.Objective: The objective of our study was to determine patients’ and health care professionals’ experiences and satisfaction with employing the COVID-box.Methods: In this single-center, retrospective, observational study, patients and health care professionals were asked to anonymously fill out multiple-choice questionnaires with questions on a 5-point or 10-point Likert scale. The themes addressed by patients were the sense of reassurance and safety, experiences with teleconsultations, their appreciation for staying at home, and the instructions for using the COVID-box. The themes addressed by health care professionals who treated patients with the COVID-box were the characteristics of the COVID-box, the technical support service and general satisfaction, and their expectations and support for this telemonitoring concept. Scores were interpreted as insufficient (≤2 or ≤5, respectively), sufficient (3 or 6-7, respectively), or good (≥4 or ≥8, respectively) on a 5-point or 10-point Likert scale.Results: A total of 117 patients and 25 health care professionals filled out the questionnaires. The median score was 4 (IQR 4-5) for the sense of safety, the appreciation for staying at home, and experiences with teleconsultations, with good scores from 76.5% (88/115), 86% (56/65), and 83.6% (92/110) of the patients, respectively. Further, 74.4% (87/117) of the patients scored the home monitoring program with a score of ≥8. Health care professionals scored the COVID-box with a minimum median score of 7 (IQR 7-10) on a 10-point scale for all domains (ie, the characteristics of the COVID-box and the technical support service and general satisfaction). For the sense of safety, user-friendliness, and additional value of the COVID-box, the median scores were 8 (IQR 8-10), 8 (IQR 7-9), and 10 (IQR 8-10), respectively, with good scores from 86% (19/22), 75% (15/20), and 96% (24/25) of the health care professionals, respectively. All health care professionals (25/25, 100%) gave a score of ≥8 for supporting this home monitoring concept, with a median score of 10 (IQR 10-10).Conclusions: The positive experiences and satisfaction of involved users are key factors for the successful implementation of a novel eHealth solution. In our study, patients, as well as health care professionals, were highly satisfied with the use of the home monitoring program—the COVID-box project. Remote home monitoring may be an effective approach in cases of increased demand for hospital care and high pressure on health care systems. Show less
Despite significant efforts, the COVID-19 pandemic has put enormous pressure on health care systems around the world, threatening the quality of patient care. Telemonitoring offers the opportunity... Show moreDespite significant efforts, the COVID-19 pandemic has put enormous pressure on health care systems around the world, threatening the quality of patient care. Telemonitoring offers the opportunity to carefully monitor patients with a confirmed or suspected case of COVID-19 from home and allows for the timely identification of worsening symptoms. Additionally, it may decrease the number of hospital visits and admissions, thereby reducing the use of scarce resources, optimizing health care capacity, and minimizing the risk of viral transmission. In this paper, we present a COVID-19 telemonitoring care pathway developed at a tertiary care hospital in the Netherlands, which combined the monitoring of vital parameters with video consultations for adequate clinical assessment. Additionally, we report a series of medical, scientific, organizational, and ethical recommendations that may be used as a guide for the design and implementation of telemonitoring pathways for COVID-19 and other diseases worldwide. Show less
