Satisfaction, Usability, and Compliance With the Use of Smartwatches for Ecological Momentary Assessment of Knee Osteoarthritis Symptoms in Older Adults: Usability Study

Introduction

Mobile devices are becoming commonplace in patient-based research [1]。Their ability to capture sensor data and enable interaction with participants in both observational and interventional studies makes mobile devices a powerful tool to augment traditional data collection approaches [2]。For example, these devices passively record activity with an accelerometer and location via GPS sensors to track physical activity and mobility. This information could be useful in understanding patients’ symptoms in the free-living environment. Such knowledge would be ideal for patients with osteoarthritis who exhibit variable pain experiences that may also interact with their mood and fatigue levels [3,4]。Coupled with sensor-based mobility data, smart devices offer a rich portrait of the interplay between symptoms and mobility levels.

Osteoarthritis is a degenerative and progressive disease affecting approximately 250 million patients worldwide [5]。Pain experiences greatly differ between patients and are often irregular within the same patient [6]。The complexity of symptoms is partly due to the site (knee, hip, or hand), genetic predisposition, initial cause of damage (ie, injury), obesity status, level of inflammation, and environmental factors [5,7]。Traditionally, patients receive treatment after reporting pain complaints and a physical examination along with optional imaging (eg, radiographs) [8,9]。Physical activity patterns, mobility function, and symptoms are used by clinical practitioners to inform treatment decisions [8,10,11]。However, difficulty in retrospective assessment of complex experiences like pain and the recall bias of self-assessing activity patterns present obstacles for care management of patients with osteoarthritis [12]。As a result, there has been considerable interest in using smart mobile devices—phones and wearables—for ascertaining symptoms and objective activity measures for informing practitioners [13]。In 2019, approximately 30 to 40 apps were designed for logging pain symptoms, but only one-fifth of those apps engaged the patients for which they were designed [14]。Moreover, none were solely designed for a smartwatch interface. Mobile devices and smart wearables have the potential to better characterize symptoms in the free-living world, but involvement of end-users (eg, patients) are necessary for appropriate design and long-term adoption.

New tools are needed to collect symptoms, experiences, and patterns of mobility and activity in real time in the free-living environment. Ecological momentary assessment (EMA) is a method based on data collection originally developed by Larson and Csikszentmihalyi in 1983 [15] for the psychological assessment of what activities people engage in, how they feel, and what they are thinking during their daily lives. It was developed because people are poor at reconstructing psychological experiences after they have occurred [16,17]。Rather, EMA considers experiences in the moment in a real-world environment and is potentially more representative of reality [18]。EMAs were first collected using paper diaries, followed by dedicated electronic diaries [19]。Recently, however, smartphone and smartwatch apps are becoming a pervasive means of assessing medical symptoms [20,21]。Work by Murphy and Smith demonstrated that tracking activity patterns with daily EMA fatigue reports yielded insights into the manifestation of activity-induced fatigue in participants with knee or hip osteoarthritis [22]。另一个最近的报告使用专门设计的smartwatch app to prompt older adults with knee osteoarthritis to report their pain 4 to 5 times per day for approximately 3 months. Results demonstrated that older adults wore the watch for 75% of the study duration and answered 50% to 60% of the twice-daily prompts to rate their pain. Despite some drawbacks, including battery drain and technical issues, participants generally thought the watch was convenient and acceptable [23]。Although this previous work is encouraging, additional research is clearly needed to document smartwatch satisfaction, usability, and compliance for knee osteoarthritis symptoms.

The large increase in mobile medical apps has prompted the US Food and Drug Administration (FDA) to release a guidance statement [24]。The FDA is clearly supportive of evaluating patient-reported outcomes [25]; however, the framework for regulating medical mobile apps is still in its infancy [24]。Moreover, FDA guidance documents state that any patient-based software should undergo evaluation for overall design, usability, and acceptability for use in clinical care and research settings [26]。In that regard, the objective of our study was to evaluate satisfaction, usability, and compliance using the real-time and online assessment and mobility monitoring (ROAMM) mobile app designed for smartwatches. This study builds on initial input from interviews about the ROAMM app interface and usability in both patients and practitioners [27,28]。We hypothesized that older adults with knee osteoarthritis would provide positive satisfaction and usability ratings while being compliant with wearing the smartwatch and answering EMA prompts over an approximately 2-week evaluation period.

Methods

参与招聘和访问的设计

Community-dwelling older adults aged 65 years and above with symptomatic unilateral or bilateral knee osteoarthritis were enrolled in the study. Recruitment sources included community advertisements and participant-based registries. Exclusion criteria included significant cognitive impairment, neurological conditions that severely inhibited mobility, inability to communicate because of severe hearing loss or speech disorder, terminal illness with life expectancy less than 12 months, severe pulmonary disease, renal failure with hemodialysis, severe psychiatric disorder (eg, bipolar, schizophrenia), excessive alcohol use (>14 drinks per week), drug addiction, or treatment for cancer (radiation or chemotherapy) within the past 1 year. All participants provided written informed consent, and the protocol was approved by the University of Florida Institutional Review Board.

Participants were asked to attend 2 clinic visits: one at baseline and another approximately 2 weeks later. After providing written informed consent, participants were administered the Mini-Mental Status Examination and then instructed on how to use the ROAMM app as previously described [27,29]。Participants were provided a simple user guide on how to use the wireless charging station and USB cable. They were also provided with a demographic questionnaire and an “exit” questionnaire that asked about their satisfaction with watch functionality and usability (seeMultimedia Appendix 1) to be completed at the end of the second week. At the second visit, participants were asked to return the smartwatch and completed questionnaires.

ROAMM App and EMA

The ROAMM app was developed at the University of Florida to enable real-time capture of patient-generated information. The smartwatch app collects wearable sensor (accelerometer and GPS) data simultaneously with symptom EMAs, as described previously [27]。Briefly, the ROAMM app is composed of a server and smartwatch app that are remotely connected through a secure https protocol. This integrated framework is designed and developed to perform several tasks, including remote data collection, storage, retrieval, and analysis. The primary goal of this project was to evaluate usability, satisfaction, and compliance of wearing the smartwatch and responding to EMA prompts in free-living conditions. Participants were asked to wear and charge the smartwatch every day for approximately 2 weeks during waking hours. The ROAMM app was programed to prompt the participant three times a day in a stratified random manner at prespecified windows: 8:00-11:59, 12:00-15:59, and 16:00-19:59.

While wearing the watch, participants were prompted in the morning to report their prior night’s sleep quality. Thereafter, EMA pain, fatigue, mood, and activity were assessed throughout the day. Participants used the rotating bezel on the Samsung Gear S3 to dial in responses and then saved their responses by pressing a button located on top of the bezel. Rating scales were chosen based on the previous literature and the ability to scale down the content for the watch interface [30-34]。In the morning, participants rated their previous night’s sleep quality on a scale of 0 to 10 [35,36], with the following anchors: 0 to 1, “very poor”; 2, “poor”; 3 to 4, “OK”; 5 to 8, “well”; and 9 to 10, “very well”. EMA pain was evaluated using a valid and reliable numerical rating scale—the 11-point Box Scale (BS-11) of pain intensity that ranges from 0 to 10 [37,38]。There is a wide variety of versions of this scale and its inclusion of text anchors [39]。Because of the small watch face, we preferred to include more anchors than the traditional numeric scales. The following text anchors were shown as the participant rotated the dial: 0, “none”; 1 to 3, “mild”; 4 to 5, “moderate”; 6 to 7, “severe”; 8 to 9, “very severe”; and 10, “worst possible.” A depiction of the interface is shown inFigure 1and in our previous publications [27,28]。

‎ View this figure

Fatigue severity was also assessed using a scale of 0 to 10, using the abovementioned anchors, according to other similar validated scales previously reported [40-42]。Mood ratings were scaled slightly differently to more closely follow previously validated visual analogue scales [43,44]。By default, the zero value for “neutral” was placed at the bottom of the screen; rotation to the right reported negative mood ratings, with text anchors “negative” for –1 to –3 and “very negative” for –4 to –5. Rotation to the left reported positive mood ratings. Finally, participants rotated the bezel to choose an icon representing one of the following activity categories that they were presently engaged in: lying down, standing, walking, sitting, and other activities (representing other possible activities such as gardening and exercise). Thus, participants were prompted to report pain, fatigue, mood, and activity three times per day. To reduce burden, prompts were delivered in a contiguous manner—one after another. The total time to answer a set of prompts was very short, typically <30 seconds.

ROAMM Exit Questionnaire to Evaluate Satisfaction and Usability

A 13-item exit questionnaire was administered at the end of the second week of the study (see questionnaire inMultimedia Appendix 1). The questions dealt with wearing comfort (eg, size, weight, wristband material), usability of the ROAMM app (eg, responding to prompts, font size, battery life), ease of using the inductive charger, and willingness to participate in future research studies. Participants were also asked to provide feedback to improve the app and its usability. Questions that used a 4-point Likert scale were reduced to two categories for statistical analysis (eg “very satisfied and satisfied” vs “somewhat satisfied and not satisfied”). Some questions asked participants to select as many options as possible that apply. Participants were also asked to provide any additional opinions of the ROAMM app and the smartwatch. Responses to this question were categorized into 4 major areas: technical issue; usability or functionality issue; size, weight, or display issue; and no issue (ie, positive opinion).

ROAMM EMA Compliance

Compliance with each ROAMM app prompt was calculated in two ways. First, a raw compliance rate was calculated as the number of actual responses divided by the total number of possible responses assuming the watch was delivering the EMAs during programmed times:

(Total responses / Total number of possible responses) × 100.

Second, it was important to adjust the compliance rate to notpenalizeparticipants for potential technical issues or for when the watch was not being worn (ie, when charging). For this calculation, time windows with <3 hours of sensor data (ie, the watch was turned off during a time when an EMA could be delivered) or if the watch was charging for >30 minutes were flagged. Flagged time windows were not countedagainstthe participant for nonresponsiveness (ie, they were not included in the denominator of the compliance rate). We considered this form of ”adjusted“ compliance in the stratified analysis described below. Only days where there were >3 hours of data, signifying a sufficient time to judge compliance, were considered in the analysis.

Data Analysis

Comparisons of dichotomous responses on the patient satisfaction surveys were described as proportions and analyzed using Fisher exact test. Questions that contained multiple answers or free text were tallied, but formal statistical comparisons were not performed owing to the low number of responses. Adjusted compliances were compared using the Studentttest between two groups and one-way analysis of variance with posthoc tests for more than two comparisons. Differences and associations were considered statistically significant at an α level <.05.

Results

Characteristics of the Study Population

Table 1provides demographic characteristics of 27 of the 28 participants who completed the demographic questionnaire. Their mean age was 73.2 (SD 5.5) years, with a total of 19 (70%) female participants, 21 (78%) White participants, and 24 (89%) participants with a college-level education. Participants were moderately active, and most were overweight (n=10, 37%) or obese (n=9, 33%).

ROAMM Exit Questionnaire to Evaluate Satisfaction and Usability

Of the 26 participants, 81% (21) reported that they would be willing to wear the smartwatch while sleeping, and 85% (22) reported the text was large enough to read (Table 2). Moreover, all 26 participants reported it was easy to enter ratings using the smartwatch. About 77% (20/26) of the participants reported that the smartwatch’s battery life ended while they were wearing it. A similar proportion of participants regularly wore a wristwatch (16/26, 62% vs 10/26, 38%;P=.16) and answered that they would wear the smartwatch as their personal watch (11/24, 46% vs 13/24, 54%;P=.77).

Approximately half of the participants (14/26, 54%) reported the smartwatch was “very comfortable” or “comfortable” (Table 3). A follow-up question asking participants how the smartwatch comfort could be improved received the following responses: no changes (n=7), reduce weight of the watch (n=11), improve wristband clasp function (n=7), reduce display size (n=6), change the material of wrist band (n=6), reduce wrist band size (n=5), and other (size, weight, display and motion detection) (n=8). Despite these criticisms, a majority of the participants reported that they were satisfied with the function of the watch (19/26, 73%;P=.002) and charging the battery (20/26, 77;P<.001;Table 3).

Furthermore, a majority of the participants (19/26, 73%;P=.002) expressed their willingness to use the ROAMM app for a 1-year research study. In a follow-up question that asked the participants the reasons for responding ”not likely“ or ”somewhat likely“ (n=11), participants cited lack of comfort (n=5), (the watch was) not stylish (n=3), gets in the way (n=4), screen was hard to read (n=3), screen was unresponsive (n=4), privacy issue (n=1), technical issue (n=5), and size or weight issues (n=1). However, some of these participants were willing to wear the smartwatch for 1 month (n=5) or 3 months (n=1). Only 3 participants reported not willing to wear the watch at all.

All participants were asked to provide additional comments on the ROAMM app and the smartwatch. Those who opted to respond commented on technical issues (battery charging: n=10; temperature of the watch being too hot: n=2) and usability issues (resetting the watch: n=5; unresponsive screen: n=1; and size, weight, or display issues: n=7). There were positive opinions about the health monitoring aspects (n=4) and the ability to use the device as a phone or for email and calendar use (n=2).

ROAMM EMA Compliance Rates

Twenty-eight participants wore the smartwatch for a mean of 13.9 (SD 0.4) days. When considering only those days with >3 hours of wear-time, participants wore the watch for a mean of 11.3 (SD 0.6) days. The accumulated total was 316 days recorded along with a total of 2505 smartwatch responses. The raw compliance rate was 61% (2505/4108) and the adjusted compliance rate was 83% (2505/3036). Specific to different windows throughout the day, the adjusted compliance rate was 86% (1004/1161) in the morning, 79% (800/1016) in the afternoon, and 77% (701/908) in the evening; details of adjusted compliance rate according to EMA responses in each window are shown inTable 4.

Average adjusted compliance for EMA prompts were similar for pain, mood, fatigue, activity, and sleep (P=.14), although compliance was consistently lowest for reporting activity, which was the final question of the bundle. Moreover, average adjusted compliance rates were similar across the three time windows (P=.92). We explored potential reasons for compliance differences in a stratified analysis. Adjusted compliance was lower among those who do not regularly wear a wristwatch (88% vs 71%;P=.03) and was better among those who thought the text was large enough to read (86% vs 60%;P=.01) (Figure 2). No differences in adjusted compliance rates were observed for participants who reported higher satisfaction levels, those who were more likely to wear the watch for a 1-year study, those who would wear the smartwatch as a personal watch, and those who reported the smartwatch did run out of battery (Figures 2and3).

‎ View this figure

‎ View this figure

Discussion

Gerontechnology is a relatively new concept that aims to promote health and well-being through technology that considers older adults’ needs and preferences [45]。The ROAMM app was developed based on these guiding principles and was designed to capture information about gerontological symptoms in the free-living environment. To ensure the technology is appropriate for this population, our research team and others have conducted focus groups to gather feedback aboutgero-friendlyvisualization (eg, display size) and functionality [27,46-49]。In the next phase of this study, we evaluated the technology in a small target sample. In this context, the purpose of this study was to evaluate the ROAMM smartwatch app for usability, satisfaction, and compliance in a patient population of older adults with knee osteoarthritis. Subsequent paragraphs interpret the results within the framework of gerontechnology and compare the current results to the existing literature. Based on our exit questionnaire, a majority of participants positively rated the ROAMM app display and functionality (eg, rotating dial). About half of the participants felt the smartwatch was uncomfortable, but almost three-fourths were likely to participate in a long-term study asking them to wear the smartwatch. Additionally, EMA compliance rates reported here were similar to a recent meta-analysis that pooled data from 701 participants across 12 EMA studies [50]。The high EMA compliance rates also indicate that older adults were able to use the app in free-living conditions. Participants also responded that it was easy to enter information using the rotating bezel, the text was sufficiently large, and they were satisfied with charging the smartwatch and effectively charging it every night. These responses culminated in a high likelihood of participating in research asking them to wear the smartwatch in a 1-year study—a goal for research related to health monitoring. However, it should be noted that willingness to participate in a long-duration study might not transfer to long-term compliance. Overall, our results suggest that older adults with knee osteoarthritis were generally satisfied with the ROAMM app and smartwatch, but the next intervention requires improved comfort and wearability for planning long-term studies.

观察duri电池消耗是一致的问题ng the study. The ROAMM app collects sensor data simultaneously with EMA data. We previously reported that the battery is most susceptible to the GPS sensor, with approximately 1% battery drain per collected sample [29]。这下水道成倍增加sensors are collected simultaneously and further affected when the screen is activated during EMA responses. In a similar study, investigators from the KOALAP (Knee Osteoarthritis, Linking Activity and Pain) study also struggled to ensure the smartwatch battery lasted during the day—about 15 hours. They also found that the lack of battery life significantly impacted engagement with the smartwatch [51]。Additional innovation is needed on battery technology, smart sensor triggering (eg, activate accelerometer during movement only, activate GPS outside a geofence), and energy efficiency to ensure that apps like ROAMM are capable of health monitoring for an entire waking day. Advances in sensor technology and EMA tools for health monitoring are only effective if sufficient compliance is demonstrated [52]。The compliance rates reached in this study were consistent with systematic reviews of EMA for assessing chronic pain in adults (eg, 83% [53] and 86% [54]). However, achieving good compliance is a multifactorial challenge, as it involves the type of behavioral coaching, perceived burden, demographics of the population, and the usability of the technology [55]。Regarding the demographics, older adults tend to have higher compliance (88%-90% at 75 years old) than younger adults (72%-74% at 25 years old) even in technology-based evaluations, as reported in a chronic pain study [50]。In fact, an EMA-based study in older African American adults reported over a 90% compliance rate when rating their activity and stress, four times per day, on a smartphone [56]。There was also some evidence that fewer questions yielded higher compliance. We observed that a single sleep quality question in the morning yielded the highest compliance. In prior work, microinteraction EMAs—where people are prompted with fast, glanceable questions that could be answered in a few seconds similar to ROAMM—were developed on smartwatches and compared to less-frequent EMA prompts on smartphones. Researchers found that although prompts on the smartwatch were eight times more frequent than those on the smartphone, participants were 35% more compliant to short microinteraction EMAs on the smartwatch [57]。Participants also responded to EMAs in less time and reported the EMAs to be less distracting on the smartwatch than on the smartphone [58]。Therefore, EMAs on a smartwatch might serve as an excellent approach for longitudinal studies, which was also conveyed by a majority of older adults in our study who were willing to participate in a 1-year research study.

分层分析合规率产生了即时通讯portant information for practice and for planning future research. In general, compliance was similar between participants with different opinions of the comfort and satisfaction with the function of the smartwatch and ROAMM app. Unexpectedly, compliance was similar among participants not likely to wear the smartwatch as their own personal watch and those who would not volunteer for a 1-year research study. Participants regularly wearing a wristwatch had significantly higher compliance than nonwearers. Furthermore, individuals who had difficulty reading the text on the watch had lower compliance than those who did not experience difficulties. In the focus group study, approximately 80% of the respondents reported the display text size was adequate [27]。In the current study the same results were found (24/28, 79%) and participants reported the text was large enough. To be more inclusive and generalize to the population as a whole, future studies will need to consider whether people regularly wear watches and ensure text size or fonts are optimized for compliance.

There are strengths and weaknesses of this study that will aid in conducting future research using smartwatch devices for monitoring health. One of the weaknesses is that this study was performed on a relatively small, homogenous sample of older adults with knee osteoarthritis. In particular, this was a well-educated sample, and the results may not be generalizable to individuals with lower levels of education. Furthermore, we did not employ a commonly used ”usability“ scale for assessing the ROAMM app, which makes comparisons to the literature difficult. At the time of data collection, existing scales were not appropriate for assessing both the software and hardware of wearable devices. Moreover, despite internal pilot testing, rapid battery drainage found during wear in the free-living environment remained to be an issue. These weaknesses are balanced with some strengths such as the thorough investigation of usability and user compliance following an extended use of the ROAMM app in real-world settings.

总之,用膝盖osteoarthrit老年人is positively rated and were generally satisfied with the ROAMM app on the Samsung smart watch. Battery life remains a concern and will need to be carefully considered in future studies. Compliance rates were generally high but were impacted by personal experiences wearing a watch and text readability. After using the ROAMM app for about 2 weeks, a majority of older adults were willing to participate in a 1-year study requiring them to wear the smartwatch. Overall, the results support new opportunities to monitor health symptoms while capturing objective sensor information from a smartwatch in older adults with knee osteoarthritis.