If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, the NetherlandsClinical Laboratory, Catharina Hospital, Eindhoven, the NetherlandsExpert Center Clinical Chemistry Eindhoven, Eindhoven, the Netherlands
Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, the NetherlandsExpert Center Clinical Chemistry Eindhoven, Eindhoven, the NetherlandsDepartment of Vascular Medicine, Amsterdam University Medical Centers, Amsterdam, the Netherlands
Department of Biomedical Engineering, Eindhoven University of Technology, Eindhoven, the NetherlandsClinical Laboratory, Catharina Hospital, Eindhoven, the NetherlandsExpert Center Clinical Chemistry Eindhoven, Eindhoven, the Netherlands
Department of Anesthesiology, Intensive Care & Pain Medicine, Catharina Hospital, Eindhoven, the NetherlandsDepartment of Electrical Engineering, Eindhoven University of Technology, Eindhoven, the Netherlands
The shift toward remote patient monitoring methods to detect clinical deterioration requires testing of wearable devices in real-life clinical settings. This study aimed to develop a remote early warning scoring (REWS) system based on continuous measurements using a wearable device, and compare its diagnostic performance for the detection of deterioration to the diagnostic performance of the conventional modified early warning score (MEWS).
Materials and methods
The study population of this prospective, single center trial consisted of patients who underwent major abdominal cancer surgery and were monitored using routine in-hospital spotcheck measurements of the vital parameters. Heart and respiratory rates were measured continuously using a wireless accelerometer patch (HealthDot). The prediction by MEWS of deterioration toward a complication graded Clavien-Dindo of 2 or higher was compared to the REWS derived from continuous measurements by the wearable patch.
A total of 103 patients and 1909 spot-check measurements were included in the analysis. Postoperative deterioration was observed in 29 patients. For both EWS systems, the sensitivity (MEWS: 0.20 95% CI: [0.13–0.29], REWS: 0.20 95% CI: [0.13–0.29]) and specificity (MEWS: 0.96 95% CI: [0.95–0.97], REWS: 0.96 95% CI: [0.95–0.97]) were assessed.
The diagnostic value of the REWS method, based on continuous measurements of the heart and respiratory rates, is comparable to that of the MEWS in patients following major abdominal cancer surgery. The wearable patch could detect the same amount of deteriorations, without requiring manual spot check measurements.
]. To anticipate on these abnormalities, regular measurements of vital parameters combined with early warning scoring systems have been adopted in hospitals. However, the best strategy is a much-debated subject, and limitations such as lack of validation, generalizability, or improvement of clinical outcomes are often discussed [
]. A drawback of early warning scoring systems is that they are manually measured and calculated, and therefore, labor intensive. Recent advances in wearable sensors that measure vital parameters provide an opportunity to replace the current manual procedures with remote monitoring systems and potentially improve the recognition of deterioration [
The current early warning scores are based on intermittent spotcheck measurements and a response protocol based on manually calculated scores. A commonly used tool is the modified early warning score (MEWS), which assigns points to impaired vital parameters and triggers follow-up care when the score exceeds a threshold [
]. Patient data can be derived from electronic medical records (EMRs), laboratory data, and wearable sensors.
One recently developed wearable sensor is Healthdot (Philips Electronic Nederland B.V.), a small patch that is attached to the patient's left lower rib using an adhesive patch. The device has the ability to continuously, remotely and unobtrusively measure vital parameters for a period of up to two weeks, which makes it suitable for remote monitoring. In addition, this wearable device has previously been shown to be clinically acceptable in terms of accuracy [
]. However, the clinical performance of wearable data compared to current practice in the surgical ward is unknown.
The aim of this study was to develop a remote early warning score (REWS) using vital parameters collected by a wearable sensor, and compare its diagnostic performance for the detection of deterioration to the diagnostic performance of the conventional modified early warning score (MEWS).
2. Materials and Methods
2.1 Study design
The TRICA study, NCT03923127, is a prospective single-center study of wearable sensors in postoperative patients in a tertiary hospital (Catharina Hospital, Eindhoven, the Netherlands). The trial was approved by the Medical Ethics Committee (Maxima Medical Center, Veldhoven, the Netherlands [W19.001]).
2.2 Patient population
All adult patients scheduled to undergo major abdominal cancer surgery between April 2019 and August 2020 were eligible to participate in the study. The exclusion criteria were: pregnancy or breastfeeding, allergy to tissue adhesives, antibiotic-resistant skin infection, active implantable devices, and any skin condition in the area of device application. With an expected complication rate of up to 25% for the different types of major abdominal cancer surgery a sample size of 143 inclusions was selected in order to collect a reasonable amount relevant complications. Patients that met the inclusion criteria were identified by their surgeon during a preoperative routine visit, patients interested in participation met with a member of the clinical trial team to be further informed about the study. Informed consent was obtained prior to the start of the study procedures.
2.3 Data collection
After the surgical procedure the investigational device Healthdot (Philips Electronic Nederland B.V.), a wearable patch of 5 × 3 × 1 cm that weighs 13.6 g, was applied to the patient's lower left rib on the midclavicular line. Heart rate (HR) and respiratory rate (RR) measurements were stored in the internal memory of the patch at time intervals of 8 s and 1 s, respectively. The accuracy of the device in postoperative patients has previously been established [
]. The HR data were resampled to a 1 s interval using linear interpolation. Every 5 min, the averages (mean) of the data collected over the past 5 min were transferred to a cloud server using built-in wireless communication technology (LoRa). Immediately after surgery, the wearable patch was applied in the post-anesthesia or intensive care unit, whichever was applicable to the patient. The wearable patch sensor then collected vital parameters (HR and RR) over a two-week period. Although the device can also collect data at home, only the vital parameters collected during the patients' hospital stay were considered in the present analysis. During the data collection both the regular care team and research team did not have access to the data collected by the wearable patch.
Once a patient was admitted to the general ward, MEWS scores were collected and registered in the electronic medical records (EMR) by the nursing staff as part of routine care. In general, MEWSs were measured every shift of 8 h. The local early warning score table is included in Appendix A, and the score can have a value between 0 and 19. The MEWS scores were retrospectively extracted.
All postoperative complications were retrospectively documented and graded by physicians according to the Clavien-Dindo classification [
]. Complications of grade 2 or higher were considered clinically relevant deteriorations, as those patients needed at least pharmacological, surgical, endoscopic, or radiological intervention.
2.4 Pre-processing of data
HR and RR measurements extracted from the internal memory of the wearable patch were used in the present analysis. A quality index (range 0–100) was reported together with the HR and RR measurements, low-quality measurements (quality index = 0) were excluded. Thereafter, the HR and RR data were down sampled to 5-min averaged data by calculating the mean of the data within the 5-min windows to match the sampling frequency of the cloud data storage transmissions.
2.5 Remote early warning score
As an alternative to the current manually collected early warning scores, the introduction of wearable sensors that automatically collect vital parameters allows the introduction of a remote early warning score (REWS). The REWS was built based solely on automatically collected HR and RR measurements from the wearable. Using the point scoring table of the local MEWS score for the HR and RR measured by the wearable patch, EWS points were assigned at every 5-min interval. (HR < 40 bpm: 2 points, HR 40–50 bpm: 1 point, HR 51–100: 0 points, HR 101–110: 1 point, HR 111–130 bpm: 2 points, HR > 130 bpm 3 points. RR < 9 rpm: 2 points, RR 9–14 rpm: 0 points, RR 15–20 rpm: 1 point, RR 21–30 rpm: 2 points, RR > 30 rpm: 3 points [
].) These scores were subsequently averaged over a 1-h period to avoid major influence of short deviations of vital parameters, resulting in the REWS, a numeric score ranging from 0 to 6.
2.6 Data analysis
The REWS was calculated retrospectively for the same time points when a MEWS score was registered in the EMR whenever wearable sensor data were available. If no MEWS was extracted from the EMR, the patient was excluded, as no comparison to the REWS could be made.
Baseline characteristics are expressed as totals and percentages in the case of binominal data or as medians and interquartile ranges (IQRs) in the case of numerical data. The significance of the differences between the deterioration and no-deterioration group was assessed using Fisher's exact test or Wilcoxon test for binominal or numerical data, respectively.
The performance of both early warning scores was then assessed by evaluating whether the patient experienced deterioration (complication CD ≥ 2) within 24 h after the spot-check measurement. Spotchecks within 24 h after a deterioration are excluded as deviating vital parameters shortly after deterioration are expected to be associated with the recent deterioration rather than an upcoming one. A visual representation on the in- and excluded spotchecks can be found in appendix B. The early warning scores were plotted against the occurrence of deterioration using a receiver operating characteristic (ROC) curve, which describes the discriminative power of the EWS scores for the range of all possible cutoff values [
]. The 95% confidence intervals (CIs) around the area under the ROC (AUROC) were computed using 2000 stratified bootstrap replicates. Statistical comparison of the two ROC curves was performed using the Delongs test for correlated ROC curves [
The distributions of the early warning scores were plotted as normalized histograms, and the data were split into the deterioration (CD ≥ 2 complications within the next 24 h) and no-deterioration groups. For the MEWS, the cutoff value was set at ≥ 3 to match routine practice, as action by the medical team is undertaken when the MEWS rises above this cutoff value. The cutoff value for the REWS was selected to match the number of true-positive cases from the MEWS score. The sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and 95% CI were then calculated based on these cutoffs [
A total of 143 patients were enrolled in this study. After excluding 23 patients to whom no wearable sensor was applied, 2 patients without spot-check data, and 15 patients without wearable sensor data, 103 patients were included in the analysis (Fig. 1). The baseline characteristics and outcomes of the study population are summarized in Table 1.
Table 1Baseline characteristics, data availability and outcomes table. Continuous data is represented as mean (sd) or median [IQR].
29 out of the 103 patients(28%) experienced clinically relevant deterioration during the study period. Patients who experienced deterioration were older (median 68 vs. 63 years, p = 0.01) and had a significantly longer length of hospital stay (median 13 vs. 8 days, p < 0.01) than those who did not experience deterioration. Interventions in case of deterioration included unanticipated ICU admission (n = 8), pain medication (n = 1), radiological (n = 5), surgical (n = 4) and antimicrobial (n = 18) intervention. No significant differences in comorbidities were found between the groups (Appendix C).
1909 spotchecks were used in the analysis, of which 665 (35%) originated from patients who experienced deterioration during their hospital stay. A median of 16 spotchecks were obtained for each patient. For patients who experienced deterioration, a significantly higher number of spotchecks was available (median 22 vs. 15, p < 0.01), which can be explained by the significantly longer length of stay. In total, 105 spotchecks (5.5%) were collected within the 24 h prior to deterioration (complication with CD ≥ 2) and were thus labelled as positive cases.
Fig. 2 shows the ROC curves for both MEWS and REWS to predict clinical deterioration within 24 h after the assessment. The AUROCs were comparable, the AUC for REWS (0.71, 95% CI: 0.66–0.77) was numerically higher than that of MEWS (0.62, 95% CI: 0.57–0.68). Despite overlapping confidence intervals the p-value of a delongs test was p < 0.01.
To match the number of true positives of the MEWS spotchecks, the cutoff for the REWS score was placed at ≥ 2.4. As shown in Fig. 3 and Table 2, 1816 (95.1%) of the REWS spot-checks did not exceed this threshold. A total of 93 (4.9%) REWS spotchecks did exceed the threshold, of which 21 (22.6%) were followed by deterioration within the next 24 h. The ≥2.4 cutoff for the REWS results in a sensitivity of 0.20 and a specificity of 0.96. An overview of the outcomes for different REWS and MEWS cutoff values is provided in Appendix D and E.
Table 2Distribution of the patients over the different areas of the Early Warning Scores. The cutoff values match the cutoff values shown in Fig. 3. A Clavien Dindo (CD) score of 0 is assigned when no deterioration is present.
The MEWS and REWS both exceeded their respective thresholds in 9 patients that experienced a deterioration, 1 patient only exceeded the MEWS threshold and 1 only exceeded the REWS threshold. The e 18 patients that did experience a clinical deterioration did not exceed the threshold in the 24 h prior to a deterioration for both systems. Both MEWS and REWS spotchecks had the same number of true positives by design (21 spotchecks); however, the number of false positives was higher for REWS than for MEWS (72 vs. 65 spotchecks).
In the present study, which enrolled patients after major abdominal cancer surgery, the REWS, based on vital parameters measured by a wireless patch sensor (Healthdot) through accelerometry, was compared with routine MEWS spotchecks performed by nursing staff. The cutoff point for the REWS was chosen to match the number of true positives obtained in current practice, which resulted in comparable sensitivity and specificity for the REWS. These findings are important because early warning scores for predicting clinical deterioration are of great interest in current clinical practice. However, the collection of information required for these systems is labor intensive. Therefore, intermittent assessments with relatively large time intervals are used, although this weakens the concept of early warning of deterioration because deterioration may occur between assessments.
In the present cohort, 1816 spot-check measurements did not exceed the REWS cutoff value. These spotchecks would automatically be classified as low-risk and thus would not require any action from the nursing staff. Simultaneously, the number of spotchecks exceeding the alarm threshold increased (MEWS: n = 86, REWS: n = 93), leading to a larger number of patients requiring follow-up care. However, translating these results into an impact on the net workload, the 1816 automatically and remotely performed spotchecks that did not require follow-up checks by the nurse would amply compensate for the 7 extra checks that exceeded the EWS threshold. While the exact amount of time saved is highly dependent on the future monitoring protocols that are implemented in the general wards, replacing part of or all monitoring in the general ward with an automated early warning scoring system could result in reduction of the manual EWS measurement originating workload of nursing staff. With an estimate of at least 10 min of time needed to manually measure and report early warning score variables (partly) replacing this system with a remote monitoring system could free up time and allow nursing staff to focus on other important tasks rather than on the routine measurement of EWS scores.
Another notable finding is that in the present cohort, the REWS based on two continuously measured vital parameters had a comparable AUC as the standard MEWS assessments consisting of seven vital parameters. A previous study evaluating early warning scores supported the finding that HR and RR were the most important predictors for deterioration among the vital parameters in the MEWS and that simplification of the number of parameters could be justified [
]. Previous clinical studies investigated comparable wearables in similar populations. Although different outcomes were analyzed, all studies reported promising results of the wearable device in terms of the detection of adverse events, decreased hospital length of stay, and 30-day readmission rate [
]. Although promising, the authors encountered significant limitations in the use of such a system when the patients gained mobility.
The present study aimed to assess the performance of REWS compared to MEWS, and other trials comparing continuous measurements with intermittent spotchecks also found promising results. Weenk and colleagues. compared continuous measurements of vital parameters with MEWS and found good agreement between both types of data collection, suggesting that the MEWS system might be replaced [
]. However, they did not report the outcomes or predictive values, complicating comparison to our findings. A different and more personalized approach was analyzed by Keim and colleagues, who investigated the deviations of vital parameters from a patient's baseline using continuous measurements compared to intermittent measurements [
]. They concluded that continuous monitoring could provide alert thresholds as the degree of change from a patient's own baseline.
The strengths of the present study include its prospective design and the real-world setting in a clinically relevant patient population. The wearable sensor was compared with current clinical practice, while allowing unobtrusive automated EWS assessments. Another potential advantage of wearable monitoring is that patients can move freely, which is important for early post-operative recovery. Furthermore, sleep disturbance for the measurement of vital parameters by the nursing staff is reduced.
A limitation of the present study is that it was performed at a single center and investigated the local performance of the EWS, which might impair external validity. Second, the primary outcome consisted of heterogeneous deteriorations. This heterogeneity challenged the definition of the forecast window and generalizability of patterns in vital parameters before deterioration occurred. A 24 h forecast window length was chosen because changes in vital parameters can be found up to 24 h prior to severe deterioration, and a forecast window of 24 h is commonly used in the literature [
]. However, mild deterioration is assumed to impair vital parameters less than severe complications, and their detection could benefit from a shorter forecast window. Additionally, a shorter forecast window could align with the length of working shifts, which are typically 8 h in most hospitals. Third, the population size did not allow splitting into training and validation datasets. Therefore, the performance of the proposed REWS cutoff value should be evaluated using an independent dataset. Fourth, the analysis was challenged by unavailable wearable data in 15 patients. The unavailability of wearable data is caused by pre-processing issues, such as corruption of the internal flash drive of the device and problems with data synchronization. This cause of missing data will be mitigated in future introduction of the wearable device into routine clinical practice and/or its use in follow up trials, as the wirelessly sent data will be used instead of data extraction from internal memory.
The present trial is an important first step toward remote monitoring and wearable-based decision support for patients in general wards. This trial provides a ground for larger scale follow up trials that are needed to optimize and validate protocols for the use of wearable based EWS in clinical practice. In the future, wearable based remote monitoring could be further explored for home monitoring as well. The present analysis focused on the comparison of wearable data collected at the same time as the routine MEWS assessment. However, as wearable patches continuously measure vital parameters, follow-up trials can aim to develop novel, continuous wearable-based early warning algorithms. Considering the current sensitivity of both EWS systems for the detection of deteriorations it will be interesting to assess whether the use of continuous vital parameters can lead to better and/or earlier identification of deteriorating patients.
A remote early warning scoring system based on vital parameters measured by a wearable patch offers a predictive performance comparable to that of the MEWS in patients following major abdominal cancer surgery. The wearable patch detected the same number of true deteriorations without requiring manual spotchecks. Overall, the present study strengthens the idea that remote continuous monitoring with a wearable accelerometer sensor patch has potential as a clinical decision support tool while improving patient mobility and comfort as well as workload of nursing staff.
The trial was partly financed by a grant from the Rijksdienst voor Ondernemend Nederland (Grant No: RVO ITEA 161006).
CRediT authorship contribution statement
Jonna A. van der Stam: Conceptualization, Data curation, Funding acquisition, Formal analysis, Writing – review & editing, Quality control of data and algorithms, Data analysis and interpretation, statistical analysis, Manuscript preparation, Manuscript editing. Eveline H.J. Mestrom: Conceptualization, Data curation, Funding acquisition, Formal analysis, Writing – review & editing, study design, data acquisition, Quality control of data and algorithms, Data analysis and interpretation, statistical analysis, Manuscript preparation, Manuscript editing. Simon W. Nienhuijs: Conceptualization, Data curation, Funding acquisition, Formal analysis, Writing – review & editing, study design, data acquisition, Data analysis and interpretation, Manuscript Review & Editing. Ignace H.J.T. de Hingh: Conceptualization, Data curation, Funding acquisition, Formal analysis, Writing – review & editing, study design, data acquisition, Data analysis and interpretation, Manuscript Review & Editing. Arjen-Kars Boer: Conceptualization, Data curation, Formal analysis, Writing – review & editing, Data analysis and interpretation, Manuscript Review & Editing. Natal A.W. van Riel: Data curation, Formal analysis, Writing – review & editing, Data analysis and interpretation, Manuscript Review & Editing. Wim Verhaegh: Data curation, Formal analysis, Writing – review & editing, Quality control of data and algorithms, Data analysis and interpretation, statistical analysis, Manuscript Review & Editing, Data curation, Formal analysis, Writing – review & editing, Quality control of data and algorithms, Data analysis and interpretation, statistical analysis, Manuscript Review & Editing. Volkher Scharnhorst: Conceptualization, Data curation, Formal analysis, Writing – review & editing, study design, Data analysis and interpretation, Manuscript Review & Editing. R. Arthur Bouwman: Conceptualization, Data curation, Formal analysis, Writing – review & editing, study design, Data analysis and interpretation, Manuscript Review & Editing.
Declaration of competing interest
Arthur Bouwman acts as a clinical consultant for Philips Research and Koen de Groot and Wim Verhaegh are employees of Philips Research. The other authors declare no potential conflicts of interest.
The authors would like to thank Alberto Bonomi for contributing to the data pre-processing and providing feedback on the manuscript and Linda Eerikainen for contributing to the data pre-processing.
Appendix A. Supplementary data
The following is the Supplementary data to this article: