This is the first RCT undertaken which evaluated a JITAI smartphone app for smoking cessation that uses mobile sensing to trigger support. In addition to assessing the feasibility of key trial parameters and estimating intervention effects, a qualitative process evaluation was undertaken as well as a SWAT to investigate incentives for improving follow-up rates.
The innovative online trial design employed was feasible and enabled the sample size target to be reached within the planned period of recruitment and for a lower cost than originally planned (£19 vs. £50 per participant). Importantly, given that the sample was exclusively identified online via adverts, the follow-up rate estimate was close to anticipated levels and is towards the higher end of the rates achieved in other web-based cessation trials with comparable samples in terms of mean age, gender and education.61 The SWAT findings suggested that higher incentives (i.e. £20 compared to £10) would reduce the proportion of people requiring manual follow-up, which is time-consuming and takes up study resources, and speed up the completion of follow-up. However, rates of response to the request to return a saliva sample were lower than anticipated. It is possible that this was affected by the COVID-19 pandemic, potentially due to more limited access to postal services due to changes in movement and time spent outside of the home or hesitation to provide a sample. Given the positive influence of a higher incentive on follow-up identified by the SWAT, it is likely also though the relatively low incentive of £5 for a returned sample may have also been a factor. This may have also contributed to the imbalance in return rates between arms.
The trial also demonstrated that three-quarters of smokers assigned to the Quit Sense app will install it on their phones and engage with it at least to the point of setting a quit date. There have been very few studies reporting the rates of uptake for cessation apps and those that do have either offered incentives for installation63 or required participants to install an app for study inclusion.75 Rates of engagement in the app were slightly lower than anticipated, with around half of those who installed it continuing to use the app for 1 week or more. Compared to the largest cessation app evaluation undertaken to date, for the acceptance and commitment therapy-based app ‘iCanQuit’,27 Quit Sense participants had a higher median number of days use (10 days; server recorded, vs. 6 days; self-report).27 Few other app evaluations provide engagement duration data although these can vary considerably across apps and countries, as demonstrated by a cessation app evaluation in Japan showing that after 6 months 88% of the intervention arm were continuing to engage with the ‘CureApp Smoking Cessation’ system.75
The trial findings provided moderate evidence that Quit Sense increased verified cessation at 6-month follow-up compared to usual care, having found a larger effect size than originally anticipated, though estimated with considerable imprecision in line with a feasibility trial. Self-reported outcomes showed smaller non-significant effects favouring Quit Sense. No effects of Quit Sense were observed at 6-week follow-up, suggesting that any benefit from Quit Sense, relative to usual care, was more likely due to maintaining abstinence in the longer term among those attempting to quit rather than from increasing the proportion of participants initiating a quit attempt. However, there was no quantitative evidence that Quit Sense affected hypothesised mechanisms of action at 6-week follow-up relative to usual care, including use of lapse prevention strategies, urge strength and frequency, self-efficacy or the extent to which smoking is automatic and associated with environmental cues, although these analyses may have been underpowered. Smoking rates early on in the study, potentially reflecting lapse rates among those attempting to quit, were lower among Quit Sense participants, which could indicate one potential pathway towards cessation, although this difference was not statistically significant. The qualitative process evaluation, however, provided greater depth into the potential pathways to abstinence from using the app. One unexpected pathway, although supported from a Quit Sense pilot study,10 was the participants’ description of benefiting in particular from the training phase of using the app (stage 1). They reported how this self-monitoring had helped them better understand and reflect on the drivers of their smoking behaviour and reinforce their commitment to quit. Self-monitoring is not a commonly employed BCT in smoking cessation interventions and few, if any, prompt the systematic logging of cues and triggers for smoking with feedback. Self-monitoring of experiences during a quit attempt has been found to reduce the intensity of cravings and other emotional triggers for smoking76 and may be an under-used approach to support cessation. Participants also indicated valuing the support delivered once they embarked on their quit attempt and that this ‘bolstered’ their defences against urges to smoke and helped keep them motivated and committed to quitting. Though other participants found they disengaged from the app because they did not feel they needed or wanted the support or because they relapsed back to smoking.
The health economics analyses indicated that the mean total costs for the Quit Sense arm were £35 higher than for usual care, though there were no differences in QoL at 6 months between arms. However, most benefits from stopping smoking will be realised beyond the 6-month follow-up time point77 and the cost-effectiveness analysis did not model long-term health outcomes, for example due to cessation differences between arms. As such, an economic model would be required in any future more definitive study. A further potential limitation of the health economic analysis is that an NHS/PSS cost perspective was not included as a further sensitivity analysis. That said, we did include a sensitivity analysis based on more favourable cost assumptions and in that the intervention was still not estimated to be cost-effective. Were an NHS/PSS cost perspective to be adopted then, based on results reported here (see Table 15), this would likely give the same result, as the estimated cost of the intervention is not estimated to be offset by any differences in NHS costs (e.g. NHS stop smoking services/GP visits). Finally, one cost that is challenging to estimate is the maintenance costs of the app. While during the trial these were estimated based on programmer time, if rolled out and Quit Sense was made available and used by far higher numbers of people, the annual maintenance costs would likely be much greater, though when spread across a larger number of people the per participant costs would likely be lower.
In the original grant proposal for this project four key feasibility criteria were outlined to help guide whether to proceed to a definitive trial of Quit Sense. As specified in Table 18, for each criterion, thresholds are considered to have been met if they are included in or are lower than the 95% CI around the estimate value (i.e. the upper 95% CI is higher than the threshold).
A priori progression criteria thresholds and observed values
All but the saliva sample return rate progression criterion reached a priori thresholds. As already indicated, several factors were believed to have contributed to the low return rate of saliva samples, though the relatively low financial incentive was considered to be the primary factor. Other online cessation trials using procedures that this trial was informed by but with higher incentives for returning saliva samples experienced higher response rates. For example, the StopAdvisor trial59 provided a £20 incentive for returning a sample and obtained a 75% response rate (personal communication). Other trials of digital cessation interventions with hard-to-reach populations (pregnant smokers) have provided £30 incentives and attained a 70% return rate.78 Therefore, it is expected that increasing the incentive in a future trial should substantially increase the response rate.
Duration of engagement with Quit Sense was slightly lower than anticipated, with half of those installing Quit Sense using the app for more than 1 week. Engagement with digital interventions is challenging to assess and device-derived data are considered limited for doing so, particularly as it fails to differentiate between effective and routine engagement and misses active engagement with the process of behaviour change outside of the intervention.79 This was demonstrated in part in our qualitative findings. Some participants reported feeling that a key benefit from the app was gained from the initial training stage and some individuals actively avoided using the app once their quit attempt started in order to avoid smoking-related objects. Participants also reported stopping using the app when they relapsed, and relapse is most likely to occur in the first week or so of a quit attempt. Therefore, it is difficult to determine what an ideal engagement rate is given most participants of any cessation intervention will fail to quit and most of these will do so early on in their quit attempt. Having participants actively engage in the preparation of quitting and gain benefit from the app through effective engagement79 may be a more important and realistic goal. This was supported by evidence indicating that all those installing the app set a quit date, with the 10-day (median) period spent using the app reflecting engagement in the process of training the app and self-monitoring smoking behaviour.
Strengths and limitations
A key strength and novelty of this trial was that most procedures were fully automated, from recruitment to follow-up, and so the resources required to run the trial were relatively low. In terms of manual follow-up, for those receiving a £20 incentive for completing the 6-month questionnaire, only a minority required contacting by telephone, further lowering the required resources for this and any future definitive trial. The trial also tested a novel ‘auto-process evaluation’ feature where Quit Sense participants could orally describe their views of the app in a high ecologically valid context, further investigating novel approaches for efficient trial design. In terms of quality, the trial used gold standard assessments of smoking abstinence,52 applied robust randomisation and intervention delivery fidelity and published protocols (trial and SWAT) and made the statistical and health economic analysis plan publicly accessible prior to analysis on the open science framework, embracing the principles of open science.80
Reinforcing the value of undertaking a feasibility trial, there were limitations identified which, if rectified, would improve the quality of a future definitive trial. A key limitation, as identified above, was the poor return rate of saliva samples and the potential bias created from these being higher in the Quit Sense arm compared to usual care. A further limitation was that the oldest participant in the trial was 61 years old and so our sample did not include those in older age groups. This likely reflects the online advertising approach adopted. Trials undertaken in the USA of digital cessation apps have found Facebook and Google adverts resulted in 11% and 4% of participants aged over 65, respectively,81 which, although still relatively low, was higher than found in this trial. Interestingly, the mean age of participants in the present trial aligns closely with trials of digital cessation interventions where smokers were recruited offline,61 and so may reflect a narrow age distribution of participants rather than one shifted towards younger age groups. However, users of cessation apps from app stores, more closely representing the ‘real world’, have a lower mean age than that found in online cessation trials, around 30 years [standard deviation (SD) 10],82 and those participating in app store-based cessation app trials also have a similar lower mean age.83 Nevertheless, working on approaches to increase the age diversity of research participants would be valuable in future work.
A further limitation was that this trial was undertaken during an unprecedented time of changed habits and routines due to the COVID-19 pandemic. Participants reported reduced movement due to lockdown and similar measures to reduce movement outside of the home. It is likely this affected the exposure and time spent in different smoking locations and consequently the app’s ability to deliver context-specific support. It is not possible to say whether COVID-related changes made quitting easier or harder for trial participants, but evidence from a nationally representative survey indicates that on average smoking cessation activity increased during the 2020 lockdown, relative to the same time in 2018–9, though smoking prevalence in younger age groups (18–34) also increased.84 The extent to which increased cessation activity continued into 2021 and beyond, however, is unclear.
Implications and future research
This feasibility trial supports progress to a definitive trial, with some minor amendments relating to trial procedures and minor intervention optimisation.
The project generated evidence that incentives were likely important for reducing trial workload related to follow-up, speeding up follow-up responses and potentially increasing the proportion of data completed in the final follow-up questionnaire. Although speculative, there is also an expectation that increased incentives for returning a saliva sample would increase response rates. These changes to incentives are warranted based on the findings and are also highly relevant for informing online automated trials in general.
The trial also found that Facebook advertising was much less expensive than Google advertising and that when Facebook adverts dominated in the second phase of recruitment, they led to higher rates of non-white ethnicity participants. Such findings are important beyond this project given the growing interest in online recruitment.
In terms of the intervention, some participants mentioned experiencing some technical issues during the qualitative interviews, although it was not possible to determine for how many across the whole Quit Sense arm. In addition, a small proportion of participants (9%) seemed to be able to use the app normally but without their data being uploaded to the server. Minor technical issues are common with smartphone apps, particularly those with relatively low development budgets compared to commercially funded apps. Further ‘bug fixing’ combined with extensive testing as well as updating the app to the latest Android operating system is required prior to a definitive trial to reduce the risk of technical issues affecting the user experience, potentially leading to disengagement. Ultimately an iOS (Apple iPhone) version will need to be developed should the app be ‘rolled out’, although given that there are few differences found between Android and iOS users of cessation apps,85 there is no strong rationale for doing this prior to a definitive trial.
The qualitative findings indicate a few areas where the app could be enhanced to improve usability, satisfaction and potential efficacy. These include increasing the options for reporting smoking in stage 1 of the app, such as increasing the smoking trigger and situation options, adding gamification elements such as rewards or ‘badges’, and improving the tailoring of the timing for support delivery when users dwell in the same location for long periods of time, such as at home. There were no suggestions or indications that the overall structure of the app or its user interface should be changed and, given the vast majority reported that the app was easy to use, only very minor modifications appear to be required.
Finding ways of enhancing engagement with apps remains a challenge in the field of digital health. While the engagement rates with Quit Sense were positive, they were slightly lower than those anticipated, and those rates found in the feasibility study.10 Opportunities for enhancing engagement with Quit Sense, based on review and qualitative evidence,86,87 could include greater use of statistical progress information, coping games, community networking and social interaction, personification of the app and use of rewards (e.g. badges/trophies).
The efficacy estimate findings, combined with the qualitative findings, suggest that there is potential for using an app to help both the individual and a system to learn about smoking patterns and triggers to inform extrinsic (app) and intrinsic (individual) support during a quit attempt. Greater emphasis on the potential benefit of engagement with this pre-quit learning phase could help users gain greater insight into their own smoking behaviour to help them more successfully maintain abstinence long term. Other smoking cessation apps may also benefit from providing this feature. The moderate efficacy estimate evidence generated in this trial suggests that this uncommon approach for promoting cessation should be more widely investigated. Additionally, it would be useful to better understand the mechanisms with which Quit Sense might lead to increased abstinence relative to usual care.
Further research investigating the mechanism of action of Quit Sense would help improve our understanding of how apps might support people to quit smoking and could be used to optimise Quit Sense support. Furthermore, exploration into additional methods of passively identifying high-risk situations for smoking, such as through the use of Bluetooth to identify other individuals the app user smokes with (co-location) or geographic information system data, may help improve the accuracy of JITAIs like Quit Sense.
Patient and public involvement
A PPI panel contributed throughout all stages of the project. This included input at the pre-submission stage of the original grant application to advise on the project plan (four panel members who were smokers or ex-smokers), during the study to advise on design, procedures and intervention content and delivery (two panel members who were smokers/ex-smokers) and anticipated input for dissemination activities (two panel members who were smokers/ex-smokers). PPI input was achieved through dedicated PPI-based in-person and online meetings, from written feedback via e-mail, involvement in quarterly team meetings and through a PPI-based independent member of the TSC. During the project we attempted to increase the number of PPI members in the panel by advertising formally and approaching individuals informally though without success.
In addition to active participation in study meetings, to help with study preparation PPI members provided feedback on study design, the participant journey and promotion of study engagement, participant-facing materials and study information including the full and basic participant information sheet, the screening process, consent form and baseline measures, and they helped to choose the Quit Sense logo. PPI members also reviewed the intervention messages delivered by the Quit Sense app, reviewed wording of study procedural SMS messages and provided more general feedback on the Quit Sense app overall including aesthetics and functionality. Furthermore, PPI members participated in meetings specifically to discuss and help validate qualitative findings.
The input from the PPI panel has been impactful across numerous aspects of the study. One such aspect, made prior to the project starting, was the PPI recommendation that the design of the study should enable a reasonable estimate of the impact of the Quit Sense app. This led to a proposed sample size that was at the higher end of what would typically be recommended for a feasibility trial and added weight to the mid-study proposal to increase the sample size further due to lower than anticipated cost and speed of recruitment. This was felt to be important to increase the precision of the effect estimate of the app and for feasibility outcomes. A further example of impactful input related to the design and refinement of app content. This included PPI members reviewing and providing suggestions for intervention message content that needed to be updated to accommodate changes related to COVID such as new working from home messages and reviewing other messages to ensure that they were COVID appropriate and relevant. PPI members also provided highly valuable input by reading through and validating the summary of qualitative findings to help with interpretation relating to app refinement. Both PPI members felt the qualitative findings relating to the app resonated with their own experiences and highlighted the importance and novelty of the training stage (stage 1) of the app. Based on their own experiences, PPI members felt that participants’ suggestions for additional ways of creating positive feedback, such as the option for the user to indicate times when they had resisted smoking and adding gamification components such as badges and rewards, would be valuable additions to the app.
Patient and public involvement members have indicated that they are happy to continue involvement in project dissemination and in the proposed full trial application and subsequent project if funded. Their continued involvement in dissemination activities will likely include recommendations on dissemination channels, advising on dissemination content and headline findings and on approaches to communicate findings to the public via infographics and media-based press releases.
Equality, diversity and inclusion
As identified in Strengths and limitations, the diversity of age was limited, having excluded those in higher age groups. This is an area for improvement in a future trial.
While it was considered a strength that a target was set for the proportion of the sample who were from low SES backgrounds, the final sample fell short of the proposed target of 45%, with only 29% from low SES recruited. One factor influencing this was the decision to use a conservative definition of low SES – either in a semiroutine or routine and manual occupation, class 5 in the NSSEC, or having never worked or were long term unemployed. Including class 4 of the NSSEC may have been justifiable and would have resulted in a sample proportion of low SES very close to reaching the 45% target, although this would have diluted the low SES classification. While in the second recruitment period we attempted to increase the proportion of participants from low SES groups, and from non-white backgrounds, this did not appear to be effective for SES and coincided with increased advertising costs, though this was not formally evaluated. Improved targeting of low SES groups may help to increase the proportion from the lowest SES group. This might be achieved via a more systematic application of targeting criteria for the online adverts from the outset, consideration of approaching online forums or organisations well represented by key diverse groups or through more tailored adverts, supported by PPI input and co-design.
The diversity of the trial sample in terms of non-white ethnicity was also relatively low (9%) and was lower than the proportion in England and Wales (15%).88 The proportion of non-white ethnicity participants recruited in the second recruitment period, including those missing, when this characteristic was purposively targeted for in the advertising filters, was higher (14%) than in the first period (5%). Improving ethnic diversity through advertising filters may, therefore, be feasible and effective, though this may still fall short of ensuring good representation from minority groups. For the qualitative process evaluation, we had a higher diversity of participants, including 25% of participants from non-white ethnicity and 50% classed as low SES.
Throughout the study and for all procedures, we sought input from our PPI panel, representing our target group, on wording and images used in participant-facing materials. This was considered to help identify potential issues with wording or images and help ensure that materials were accessible to a range of reading levels and backgrounds. While we attempted to identify further PPI members, particularly from diverse groups, we had no success. This will remain an objective for a future definitive trial.
Reflecting on the research team’s composition, there was a lack of diversity in team members’ characteristics, although there was even gender balance. There was, however, slightly higher ethnic diversity in the Trial Steering Group, who had oversight responsibility for the trial. Increasing the diversity of researchers and those involved in oversight as well as study samples remains an important goal for research projects. There was a mixture of career levels across the research team, although no one was below a PhD or equivalent level of experience, and earlier career researchers had significant leadership roles in the project.
