Study advertisements were promoted on Facebook (including Instagram) and Google AdWords in two phases – the first ran from 27 November to 12 December 2020 and the second from 5 January to 25 January 2021. Participant identification and recruitment is summarised in the study Consolidated Standards of Reporting Trials flow () and the advertisement engagement flow diagrams (). The total number of ‘impressions’ (when the advert is shown on a web page or app) were 30,835 and 98,767 for Facebook for phases 1 and 2, respectively (total of 129,602), and 24,143 and 630 for Google across phases 1 and 2 (total of 24,773). This led to 560 and 858 advert ‘link clicks’ (total of 1418) for Facebook and 407 and 18 for Google (total of 425) for phases 1 and 2, respectively. There was a total of 1275 landings on the study website, of which 323 (25%) people completed the eligibility assessment, 299 (23%) were eligible, 267 (21%) provided informed consent to participate and 209 (16%) continued to randomisation (117 in phase 1 and 92 in phase 2). Of those assessed for eligibility, 93% were eligible, and of those eligible, 70% consented and were randomised. One individual was randomised but had been enrolled by their partner and so was removed from the randomised sample as they were not eligible as they had not provided informed consent.
Flow diagram of advert reach, engagement and enrolment rates and costs.
Removing the one participant randomised but enrolled by their partner, 209 remaining individuals were randomised, 104 allocated to the Quit Sense arm and 105 to the usual care arm. At 6 weeks, 149 (71%; 95% CI 65% to 77%) were followed up and at 6 months this was 160 (77%; 95% CI 71% to 82%). There were six withdrawals, four from the Quit Sense arm and two from the usual care arm.
Table 2 provides baseline sample characteristics. The sample had a mean age of 41 years (range 18–61), 56% were female, 29% classified as low SES and 9% were of non-white ethnicity. Mean baseline smoking rate was 15 cigarettes per day and 31%, 59% and 10% were classed as low, moderate and high heaviness of smoking, respectively.
Participant characteristics at baseline
Measurement, recruitment, intervention engagement and smoking outcomes
Completion rates for the anticipated primary outcome for a full trial
Completion of self-reported abstinence for the primary outcome at 6 months was 77% (160/209; 95% CI 71% to 82%) (Table 3). By arm, response rates were 78% and 75% for the Quit Sense and usual care arms, respectively. The return of a viable saliva sample for biochemical validation of those self-reporting abstinence for the primary outcome was 39% (16/41; 95% CI 24% to 54%), and by arm 52% (13/25) and 19% (3/16) for the Quit Sense and usual care arms, respectively. In addition, one participant (in the Quit Sense arm) returned a sample which had insufficient saliva in order to assess cotinine/anabasine levels. More detailed outcome data at 6 weeks and 6 months can be found in Appendix 6, Tables 19 and 20.
Feasibility outcome completion rates
Usual care arm cessation rate
Three participants in the usual care arm were classified as abstinent according to the primary outcome, a rate of 2.9% (95% CI 0.0% to 6.0%).
Cost of recruitment using online advertising
Advertising costs were split into running costs for the campaigns, managed by the commercial partner organisation Nativve, and the costs of the adverts on Facebook (which includes Instagram) and Google (all costs inclusive of value-added tax) (see ). Total advertising running costs were £2796, broken down into a set-up fee (£1068) and 6-week campaigns for both Facebook and Google (£864 each). Total advert costs were £804.44 for Facebook and £412.49 for Google. Total cost per recruit was £19.20, which included running costs (£13.38) and advert costs (£5.82). One hundred and ninety-five participants were recruited via Facebook and 14 from Google. The advert cost per recruit was lower for Facebook (£4.13) than for Google (£29.46). In phase 2, Facebook adverts were purposively targeted towards non-white and low SES groups by Nativve using filters including for language and interests to attempt to increase the proportion of participants from these groups. While the proportion of non-white (including missing) participants was higher in phase 2 (13.8%) relative to phase 1 (5.2%), there were no meaningful differences in the proportion classed as low SES (phase 1 30.4% vs. phase 2 29.9%). The estimated cost per recruit was higher in phase 2 (calculated range £2.27–8.11) compared with phase 1 (calculated range £1.16–4.61), although the data comes from web analytics and so is not fully reliable.
Rates of app installation and use and acceptability
The installation rate of the Quit Sense app, defined as the submission of the unique code provided to intervention participants as recorded by the server hosting the app, was 75% (95% CI 67% to 83%; 78/104) (Table 4). All but one participant (99%; 77/78) who installed the app did so before the installation text message reminder was sent. Among the 28 participants sent a reminder, one installed the app on the day of the reminder and three replied to the subsequent text message inviting a reason for not installing, where two indicated they had had problems installing (‘option 5’) and one that had had no internet (‘option 1’). The three responding to the text message were contacted by the lead researcher but none of them installed the app. A small number of participants who installed the app (n = 9) did not have engagement data uploaded to the server due to a technical issue. In a few of these cases, missing engagement data for some engagement variables were determined from data recorded in follow-up questionnaires or during process evaluation interviews. Among those who installed the app, 100% (95% CI 95% to 100%; 70/70) set a quit date in the app and 51% (95% CI 39% to 63%; 38/74) engaged with the app for more than 7 days, and 23% for more than 30 days (95% CI 13% to 33%), with a median duration of use of 10 days [interquartile range (IQR) 30)]. The median number of days between the date when the quit date was set and the actual quit date (‘quit date delay’) was 9 (IQR 8). Among those who engaged with the app up until their quit date (36/74; 49%), the total median duration of app engagement was 27 days (IQR 91) and duration of app engagement after their quit date had passed was 22 days (IQR 99), excluding two participants whose app engagement period could not be accurately calculated due to a technical issue.
Use of Quit Sense app (Quit Sense arm only)
Among intervention participants who installed the app and were followed up at 6 weeks, 67% (29/46) said they would recommend Quit Sense to a friend trying to quit, with 30% (13/46) saying ‘maybe’ and 2% (1/46) saying they would not recommend it. The majority of participants either strongly agreed (55%; 24/44) or agreed (32%; 14/44) that Quit Sense was easy to use, with 11% (5/44) neither agreeing nor disagreeing and 2% (1/44) disagreeing.
Completion of smoking cessation-related resource use and quality-of-life data
At 6-month follow-up, the response rate for both resource use and QoL data was 147/209 (70%; 95% CI 64% to 77%).
Intervention effect on anticipated primary outcome and secondary abstinence outcomes
Analysis of the primary outcome, self-reported abstinence as smoking no more than five cigarettes within the 6-month study period, biochemically verified by saliva test, found a higher abstinence rate in the Quit Sense arm (11.5%; 12/104) compared to the usual care arm (2.9%; 3/105) (unadjusted OR 4.44, 95% CI 1.21 to 16.21; p = 0.024) (Table 5). This shows a positive association, with moderate evidence to suggest that the odds of abstinence for the Quit Sense group is 4.4 times that of the usual care group. When adjusting for stratification variables and prognostic factors, results and conclusions in terms of direction, size, precision and statistical significance of the effect remain consistent (adjusted OR 4.46, 95% CI 1.19 to 16.69; p = 0.023) (see also Appendix 6, Table 21). This analysis adjusted for heaviness of smoking index as a key prognostic variable, which appeared potentially imbalanced between arms at baseline, but due to relatively small numbers of abstinent participants, the model fit was potentially problematic. A more appropriate approach of exact inference (using SAS 9.4 statistical software) was therefore used, which produced more conservative (wider CIs), but overall consistent results and conclusions (adjusted OR 4.36, 95% CI 1.10 to 25.22; p = 0.033) (see Appendix 6, Table 22). Sensitivity analyses were undertaken where (1) withdrawals were excluded and (2) complete cases only included. Other than changing the abstinence proportions, results and conclusions of effect size, direction, uncertainty and statistical significance also remain consistent (see Appendix 6, Tables 23–27).
Between-arm differences in abstinence for primary and secondary outcomes
By using the Bayesian approach relevant in preliminary trials, it is estimated that there is 90% probability that the underlying OR favouring the intervention is 1.7 or higher, 93% that it is 1.5 or higher and 85% that it is 2.0 or higher, indicating good support for a subsequent trial in which this range of effect sizes is considered.
Secondary smoking outcomes (see Table 5 and Appendix 6, Tables 27–34) collected at 6 months also favoured Quit Sense over usual care, although this was only statistically significant for the biochemically validated outcome: biochemically validated 7-day point prevalence abstinence (Quit Sense 15.4%; usual care 4.8%, OR; 3.64, 95% CI 1.28 to 10.33; p = 0.015) and self-reported 7-day point prevalence abstinence (Quit Sense 26.9%; usual care 19.1%, OR 1.57, 95% CI 0.82 to 3.01; p = 0.18). Self-reported 7-day point prevalence abstinence at 6 weeks did not favour Quit Sense over usual care (Quit Sense 19.2%; usual care 20.0%, OR 0.95, 95% CI 0.48 to 1.89; p = 0.89). Due to there being potential between-arm differences in saliva sample return rates, we undertook an additional post hoc sensitivity analysis for the primary outcome analysis but using self-reported prolonged abstinence only. The findings favoured the Quit Sense arm, though the between-arm difference was not statistically significant (Quit Sense 24.0%; usual care 15.2%, OR 1.76, 95% CI 0.88 to 3.53; p = 0.11).
Intervention effect on hypothesised mechanisms of action of app at 6 weeks post enrolment
At 6-week follow-up, the proportion of participants who reported smoking in the first 2 weeks of a quit attempt (or since enrolment if no quit date set) was 70.4% in the Quit Sense arm and 80.8% in the usual care arm, which was not a statistically significant difference [X2, 1 (degrees of freedom), N = 149] = (X2 statistic = 2.17), p = 0.14] (Table 6). Use of lapse prevention strategies varied, with ‘contacting a friend/family member for support or distraction’ as the least commonly used strategy (47% participants used) to ‘focusing on other tasks’ as the most commonly used (91%) (Table 7). There was no evidence of a difference between arms on average lapse prevention strategy use [MD −0.07, 95% CI −0.26 to 0.12, T-statistic: −0.75; p = 0.46] or when broken down into avoidance (MD −0.10, 95% CI −0.33 to 0.13, T-statistic: −0.58; p = 0.39) or coping strategies (MD −0.06, 95% CI −0.26 to 0.14, T-statistic: −0.58; p = 0.56) (Table 8).
Between-group differences in scores for mechanism of action measures at 6 weeks (non-parametric statistical tests)
Proportion of participants using lapse prevention strategies at 6 weeks
Between-group differences in scores for mechanism of action measures at 6 weeks (parametric statistical tests)
There were no evidence of differences between arms on self-efficacy (Mann–Whitney U = 5550.50, p = 0.39) and strength of urges to quit (Mann–Whitney U = 5016.00, p = 0.23), frequency of urges to quit (Mann–Whitney U = 5267.50, p = 0.83) and WISDM automaticity (Mann–Whitney U = 5153.50, p = 0.51) and associative processes (MD 0.14, 95% CI −0.36 to 0.64, T-statistic: 0.56; p = 0.58) subscale scores at 6 weeks (see Tables 6 and 8).
Process evaluation findings
Participants who made use of the ‘auto-process evaluation’ feature did so primarily to provide positive suggestions for app improvement, which mainly focused on logging unique contexts when smoking, but also to raise some issues and problems with the app. These are summarised in Table 9. However, this feature was not well-used. Six participants out of those who installed the app (6/78; 7.7%) chose to submit an audio message. Two of these participants also took part in a qualitative interview. One participant submitted an audio message on two occasions (once on commencement of the app and another having used it for some time).
Content summary for auto-process evaluation audio recordings submitted in the Quit Sense app
Participants of the qualitative interview study (N = 20) were purposefully sampled, with 15 recruited from the intervention group, to inform aspects of app engagement and trial procedures, and 5 from the usual care group, to capture experiences of participating in that group. We specifically chose participants from those consenting to a qualitative interview who had both achieved smoking abstinence and continued to smoke, or had relapsed to smoking (Table 10)
Qualitative sample by randomised group and smoking status
The qualitative sample were broadly representative of the total trial sample with two exceptions – higher rates of highest qualification as degree and higher rates of those from lower SES groups in the interview sample, with the latter due to deliberate oversampling to ensure a broader range of views were captured (Table 11).
Qualitative sample in comparison to total trial sample
Five main themes were identified from the qualitative analysis: ‘engaging’, ‘disconnecting’, ‘suggestions for app improvement’, ‘pathways to change’ and ‘time and context’.
‘Engaging’
Finding out about the study and reasons for taking part
Participants did not have concerns about being recruited remotely online and providing personal details such as their names, addresses and telephone numbers as part of registering for the study. They felt that the study advertisements appeared genuine. Participants commented that they were reassured by the associations the study had with both a university and the NHS. Participants tended to report that they signed up immediately after landing on the study website and based only on reading the basic participant information sheet rather than downloading and reading the longer participant information sheet.
I’ve seen it’s also in relation with the NHS. So […] I enrolled straight away […].
Female, usual care arm, ID110
Everything was kind of straight forward and […] informative […] So, I didn’t download the detailed information sheet to be honest [laughs]. […] no concerns whatsoever.
Male, usual care arm, ID116
The main reason given for participation was a desire to quit smoking. Related motivations included the desire to be held accountable in making a quit attempt, seeking extra support and a sense of belonging (i.e. being part of a group with others who had similar motivations):
It was the accountability almost; just to be able to, you know, put a bit of pressure on myself rather that than just say, ‘I’m going to finish’ and inevitably, I never do.
Male, usual care arm, ID116
I thought, I’ll have a stab at it and give something a go. Just sort of as many aids as we can get really.
Male, Quit Sense arm, ID115
I found it on Facebook [ … ] fed to me on the feed. It felt good to be a part because it’s certainly something that I’m interested in or at any rate, something that I need [ … ] and it also felt good to be a part of a research group [ … ] like you were part of something.
Male, Quit Sense arm, ID116
‘Disconnecting’
Those participants who were retained in the study reported disconnecting with the study and intervention app due to a variety of reasons connected to either the use of the technology or changes in their own motivations and behaviours.
Technical issues
Several participants reported having technical issues with the app resulting in feelings of frustration and annoyance. Issues included the following: unwanted and repeated battery optimisation messages, crashing other applications, the app opening when it was not wanted, an inability to complete surveys and inaccurate recording of cigarettes reported.
Not being able to use it at the beginning of the quitting smoking process, when it was like the most important [ … ] I found it quite frustrating [ … ].
Female, Quit Sense arm, ID106
Change in smoking behaviour
Some participants had uninstalled Quit Sense from their phones. Reasons included not quitting and lapsing:
No, I used it for about a month. Of course, when I started smoking again after that, I felt bad and embarrassed, so I didn’t use it much after that [ … ] Just disengaged by that point.
Female, Quit Sense arm, ID210
Other participants disengaged because they felt that they no longer needed app support, such as because they had quit cigarettes and/or transitioned to vaping. Interestingly, one participant who thought the app was very useful in the preparation stage ceased actively engaging with it once quit day arrived because they associated it with smoking:
So, quit day came and although I checked back in on the app, I did find that I really didn’t use the app at all. Partially because when I quit, I wanted to be, ‘I’ve quit’. When you quit you throw your cigarettes and lighters away. One of the top tips [from the app] is to throw anything away that reminds you of your life when you were smoking. In my mind, that was also to get rid of the stop smoking app as well because you’ve stopped smoking. The messages did still pop up.
Male, Quit Sense arm, ID216
Some participants disengaged with the app and/or study more broadly because they either did not make a quit attempt or lapsed:
I haven’t felt like I’ve been particularly involved in a study [ … ] me enrolling on the stop smoking and the timing of me stopping smoking [ … ] I hadn’t stopped smoking.
Female, usual care arm, ID108
Not meeting motivational needs
Some participants had been interested to try an app to assist their quit attempt but had found that this form of support (at least alone) did not meet their needs:
I removed it after a few days. I thought yeah, it’s not going to work for me [ … ] I just realised it’s all, shall we say, psychological support. [ … ] To stop smoking it has to be at least a week or two of something physical [ … ] Supervised or patches, pills, I don’t know.
Female, usual care arm, ID110
Suggestions for app improvement
Participants felt that alcohol, daily routine and boredom were important triggers for smoking that were missing from the app:
I think it could have been a bit more specific. As someone that smokes and likes a bit of a drink every now and again, I felt that just clicking ‘socialising’ didn’t quite cover it. I don’t know, I felt like it might have been beneficial for your data to say, ‘okay, so alcohol was involved’.
Male, Quit Sense arm, ID115
Sometimes it can be as simple as after a meal.
Female, Quit Sense arm, ID109
Boredom is probably a lot of people’s triggers for smoking. But for me, quite often, I didn’t fit into a category. I didn’t have one because I felt a certain way, I just had one because I felt like I should have one now. There’s no real reason or purpose. [ … ] It was having one for the sake of it, for the routine of it.
Male, Quit Sense arm, ID216
Some participants wanted to be able to record that they had smoked while en route to other places, for example, while walking or driving:
I like to smoke whilst I’m walking. So in between actual locations and venues; it doesn’t give you the option to say that.
Male, Quit Sense arm, ID116
How I personally see how it could be slightly improved for future use, would be the ability to tell the app that you are driving, or for the app to recognise that. [ … ] the roads where I would smoke the most [ … ] trigger roads.
Male, Quit Sense arm, ID216
Related to the desire to record smoking while being out and about was the suggestion that the app should have a ‘quick report’ button or allow users to add details later after the smoking event:
When you’re travelling around, the process of logging that you’ve had a cigarette can be quite a long process. Which isn’t a problem at all, if you have a job working behind a desk or if you physically stop what you’re doing to go and have a cigarette break, you’ve got time to log it and go through your phone etc, etc.
Male, Quit Sense arm, ID216
Yeah, so when there’s like a questionnaire at the end of the day that say’s how many cigarettes you’d had that day, you could put it in there but you couldn’t log the time of day and what the triggers were. So, you have to sort of like put a ‘I smoked 10 in that day’ but if you didn’t log it at the time, you couldn’t go through the options of what triggered that craving etc.
Female, Quit Sense arm, ID111
One participant liked the idea of being able to make notes in the app, for example, how she was feeling on quit date that she could look back on as an additional aid to monitoring progress:
Yeah, I mean, maybe just have a section that would explain more of how we felt that day. I think sometimes just getting it off your chest. I know I ticked a box, ‘did you find it easy’, maybe you could write or type something, as it’s on your phone [ … ] you could look back on it and see how you’ve improved.
Female, Quit Sense arm, ID210
Several participants thought that the app could further boost motivation by having a reward system such as points for not smoking or badges:
Every time you read a message maybe it could get something like a star or a tick or whatever. [ … ] Just to get people a bit more engaged with the app. With a points system, maybe they could have like free books to read or whatever.
Female, Quit Sense arm, ID215
Yeah, it just gives you a little more of a boost that you’re doing something good. I might not notice it, but it is doing something. You also get little, you know, badges – ‘Yay, one week smoke free!’ that you can then share and things like that, which then gives you a sense of achievement; especially if you’ve told family and friends. It’s things you can share and then go – look how well I’m doing. Which I think we all … it’s really tough to quit smoking so I think that little pat on the back is really nice [laughs].
Female, Quit Sense arm, ID109
One participant who found that messages about smoking occasionally prompted him to think about smoking suggested that some could be reworded to congratulate users on not smoking in locations where they previously had reported:
I think, and again, I don’t know how possible this even is, but I always found it interesting to think back about ‘how many cigarettes have I not smoked?’ So, if for example, the app knew that you were driving and then said afterwards that I would usually have five cigarettes on that journey, it could be that when you were driving it would say, ‘well done, that’s five cigarettes you didn’t have.’ So, it’s still recognising that it is somewhere where you usually would have smoked but instead of reminding you and then having a cigarette, it would be like, ‘oh great, a little milestone or achievement that I haven’t had any cigarettes on that journey.
Male, Quit Sense arm, ID216
Some participants felt that when they were in the same location for prolonged periods of time, such as due to COVID-19 lockdown restrictions, messages could be better tailored by arriving at the times that they reported smoking, so based less on location but more on reported times of smoking (e.g. after meals):
I was working from home, so I was pretty much sat in the same place all of the time [ … ] either I was at the dining room table, or I was on the sofa. They were the only two places.
Female, Quit Sense arm, ID114
They just sort of came at times that I felt was a bit random. I didn’t feel that they came at specific times, like tea breaks and stuff because that’s the key periods really when you’re more likely to go and have a smoke.
Male, Quit Sense arm, ID115
I liked the idea of the application training itself to my pattern and what I’m doing. But then I expected at the time that the application would identify the time that I was more likely to be longing for a cigarette and send me a message. So, I’m not sure if that is in the app or not but I would find that helpful.
Female, Quit Sense arm, ID208
‘Pathways to change’
Three main mechanisms by which the app supported behaviour change were identified. A further mechanism (‘disrupting’) which was associated more with contextual factors is presented under ‘Time and Context’. Participant vignettes which expand on this theme can be found in Appendix 6, Table 35.
Committing, reporting and preparing for quit day
The preparation phase (stage 1) of the app was valued as an opportunity to make a formal commitment to quitting, including setting a fixed goal by establishing a quit date. This stage was also seen as something novel and not offered by other quit smoking apps:
All the other smoking apps don’t do that. On those ones, you’ve quit now. So, that was a nice one to be like, right, this is the day I’m going to quit, and it was that countdown [ … ] it gave you that focus.
Female, Quit Sense arm, ID109
It made me make a commitment [ … ]. It was something a bit more concrete and an actual set date and it really did keep me on track.
Female, Quit Sense arm, ID315
Some participants attributed a reduction in their cigarette intake during stage 1 to the act of reporting smoking via the app. One reason provided was that logging smoking required effort and, in some cases, it appeared easier to forgo the cigarette than to answer the questions about the smoking event. An alternative explanation of this point could be that participants felt more accountable for their smoking behaviour when asked to report it, so became more aware of it, and that in itself prompted behaviour change. Participants also said that repeatedly reporting smoking made smoking a conscious decision again, which in turn prompted them to question whether they really wanted a cigarette and raised their self-awareness about triggers:
When I did need to go for a cigarette, I would do it without thinking about it. Using that app made me think about it a lot. Do I really need one? Should I really go for one right now? Using that app, it just helped us to cut down after the first four days to be fair. Now, the first four days, they were a bit stressful in my life, but the app did help. Every time going for one, I had to write it up. [ … ] The training was very good. Every time I was writing in one of those little boxes, when I was going for a smoke. The app would come up and be like, ‘Do you really need to do this right now?’ Little subliminal messages in my head just stuck there. Every time I was pulling one out of the box, it just kept going over in my head – ‘Do you really need to have one right now?’ [ … ] Every time I would have to go for one, I’d have to report it. So, in the end I was just thinking, I’m not going to go for one because I’d have to report it. That would then take my mind of wanting one to be fair.
Male, Quit Sense arm, ID206
So, I found on the lead up to my quit date, it firstly made me realise how many I smoked. [ … ] When you’re on the app and you see how many cigarettes you’ve had. Moreover, I would have a cigarette and then 10 minutes later, I would have another one. Why? Why did I have that cigarette that I didn’t need anyway and then half an hour later, I’ll have another one? I found it useful, looking at that to see my own reasons for validating my own smoking. There was no hiding behind the ‘Oh, well, I’ve only had a few cigarettes’ [ … ] it’s in black and white and actually, no I haven’t.
Male, Quit Sense arm, ID216
I very much attribute my quitting to the app because it helped me subtract the patterns of me smoking. It allowed me to see when I was smoking more, and it then allowed me to unpick why I was smoking more. That was the key to the lock for me. It was helping me see that was why I was smoking more, when the only solution I could draw was because I felt I should.
Male, Quit Sense arm, ID311
One participant discussed how the combination of reporting and looking at the progress calendar in ‘My Profile’ helped to motivate her to reduce her cigarette intake:
To be honest, I really thought I was going to fail at quitting, but it actually works. [ … ] I think it was that because if you smoked more than five in a day, you got a little sad face as well. So, I noticed as I was getting towards the quit date, I was actually thinking, do I need to really have this smoke or not? I was cutting down before which I think helps for when I did quit. I hadn’t been smoking as much just before then.
Female, Quit Sense arm, ID114
Interestingly, another participant used the report smoking feature to understand and then to avoid locations, which for him triggered cravings to smoke, like a personal diary:
It didn’t last very long because I set a very short quitting date, but I like the idea of reporting where and when. I did that for a couple of days and then I purposely went to different routes; when I was on [a] walk and that in the mornings, I would go on different routes to avoid that place. It became a routine that I would light a cigarette at that place, so I went a different way.
Male, Quit Sense arm, ID211
‘Validating’: the role of app messages
App messages were largely deemed to be encouraging, supportive and motivational, offering validation of efforts to quit and helping to re-enforce the goal of quitting:
What they do is they sort of they just bolster your defences a little bit. The odd fact here, the odd figure there. [ … ] a lot of the messages are very good. They give that validation that you’re looking for. They give that little reason to smile about why you’re doing it.
Male, Quit Sense arm, ID216
Yeah, they were very motivational, without being lecturing, if you know what I mean? They were just on the right level, for me I felt.
Female, Quit Sense arm, ID210
Having given up smoking, it kind of suggested that you’d be very proud. You’d be able to say, ‘I’ve given up’. [ … ] I found that very useful. [ … ] I think they’re very, very good. Very supportive.
Male, Quit Sense arm, ID211
I guess the main effect for me is in terms of it inspiring me; the tips that you would … you know, every time I recorded that I’d had a cigarette, it would say something. Some of those ideas were new [ … ] that encouragement was needed and definitely had an effect.
Male, Quit Sense arm, ID116
Some participants described the app as a ‘positive voice’ and one which some participants felt able to internalise:
Yes, it’s definitely like a positive voice in your head; that’s how I view the app. Having the encouragement coming from inside you. Sometimes you are experiencing a negative mood, so I would just be like, ignore it and let me have my cigarette. Having the [app] is like another positive sound in your head. It’s something that’s constantly encouraging you.
Female, Quit Sense arm, ID208
All in all, the app is really great. Little messages really, really help. Especially when you know in the back of your head that you don’t need to go for one just because you’re bored. With the little messages there, for extra support, just telling you what your mind knows, that you don’t need to go for one. An app knows that you don’t need to go for one, so why are you going for one?
Male, Quit Sense arm, ID206
While the information presented in the messages was not necessarily new to participants, they reported finding the content provided a helpful reminder:
I think it was mostly stuff that I already knew but it’s always helpful to be reminded and was more like of a prompt to think about stuff that I already knew. If that makes sense, because sometimes there’s stuff that you know about but you kind of need to be reminded of it.
Female, Quit Sense arm, ID106
‘Equipping’
When asked, several participants recalled strategies suggested by the app that they had attempted including breathing exercises, delaying smoking for a few minutes, removing visual cues to smoke and using distraction techniques:
I think the ones that were just sort of saying, you know, 'make sure you’ve cleared everything away beforehand.' So, I would make sure that the night before, I would have my last smoke and then make sure that the tobacco I had then went away, out of sight. Got rid of the ashtrays [ … ] Like I said, in the first week, I found it the most useful. I think I’d gone and brought a pack of Mint Imperials, so if I’m sat bored in a meeting, at least I can have one of those, to keep my hands and mouth busy for a few minutes.
Female, Quit Sense arm, ID114
It was about changing up something about when you had a trigger.
Female, Quit Sense arm, ID210
I did the exercise ones, and I spent a lot more time indoors with my daughter. That was the strategy that said, 'be indoors with people who don’t smoke.'
Female, Quit Sense arm, ID315
I used the ones that said, try and preoccupy yourself with something else. I found that if I happened to catch it at that point when I was having a craving then I did tend to find it helpful.
Male, Quit Sense arm, ID115
‘Time and context’
COVID-19
As interviews were collected during a national COVID-19 lockdown in England, participants were asked how their smoking habits had been impacted by the pandemic, if at all.
Several participants felt that due to lockdown they had more free time, especially at home and they attributed these circumstances to an increase in boredom and consequently of increased smoking:
If anything, I’m smoking more because I’ve got more time on my hands.
Male, Quit Sense arm, ID308
I think it’s probably true that I actually smoke more now because I’m indoors, at home, in a situation that’s quite comfortable, to be quite truthful. In that, I’m at home and doing my own thing, so I can do as I please. So, I probably smoke more than if I were going to work or going out and about, I would probably reduce intake. If I was in public or in a building where smoking wasn’t allowed etc., then I’d be smoking less.
Male, Quit Sense arm, ID116
I think it kind of confirmed. I knew as soon as I was working from home that I was getting bored. So, I’d be sitting there working, particularly in a meeting. When you’re sat in a meeting for 2 hours; there’s nothing much else you can do, so I’d think, oh, I’ll just have a smoke [laughs].
Female, Quit Sense arm, ID114
Well, unfortunately, we started smoking more at the beginning of it because of nothing to do; sheer boredom.
Female, Quit Sense arm, ID315
Other participants had found lockdown had helped to reduce their smoking and had been useful in making a quit attempt. One reason for this was the lack of social contact including with other smokers and not going to locations where they would ordinarily have smoked:
The only time I will have been out is when we’ve been shopping, as of recently. Or we’ll have been food shopping, or I’ll have gone to work when I’ve needed to. But apart from that, I haven’t really seen anyone who would influence me on my smoking. If that makes sense?
Male, Quit Sense arm, ID206
To be fair, it probably helped in a strange way, just because of the lack of socialising, which was kind of my key issue. My days in the office are a lot lower, so again, the smoke breaks, things like that; the temptations go. It has made it easier with, I’m not going say, for dinner or for a drink and things like that. Where normally my previous attempts have ended, let’s say.
Male, usual care arm, ID116
Some participants had come under increasing financial pressures due to the pandemic which motivated a reduction in smoking:
I don’t think [Covid] had a massive impact, apart from being a slight motivator. Mostly, to be honest, my motivation to stop smoking was financial. I think I, like a lot of people, got made redundant towards the start of the pandemic. Again, financially, it was that I can’t continue to spend money that I don’t have on smoking. And you know, the knowledge too that if I continue to smoke, it improves my chances of bad health if I were to get Covid. I also think the fact that I’ve not been around other people that smoke.
Female, Quit Sense arm, ID106
There were additional references to health as a motivator, including the possibility of quitting providing improved resilience to COVID. These were from a participant who was still smoking at the time of interview. However, for this participant concerns about smoking and COVID had disappeared over time and were no longer a primary motivator.
It’s been scary at times hasn’t it. I mean, really, when it first came out a year ago, it made me think that I need to stop smoking because if I catch this, I’m going to have a better chance of getting over it if I don’t smoke. To now, when we’re all just so depressed aren’t we? How long is this going to go on, you know [laughs]? I think alcohol and cigarettes … everyone’s behaviour. You’re not going to the gym, you’re not …
Female Quit Sense arm, ID210
For some other participants who continued to work as usual throughout lockdown, there were no significant changes reported:
Exactly the same. Yeah, there’s no difference in lockdown for me. I’m still working. It’s the same.
Female, Quit Sense arm, ID208
Feasting and fasting
Recruitment for the study took a break between mid-December 2020 and early January 2021 to avoid the main festivities of Christmas and New Year. Nonetheless, a small number of participants commented that their attempts at quitting had been negatively impacted by Christmas and New Year festivities:
Christmas and New Year’s intruded within the time period that I’d set myself. I think I’d started using the app around the beginning of November, possibly late December. It came to Christmas and New Year and I found myself smoking and reaching for cigarettes even more rapidly than I had been, certainly more so than the early stages of me using the app. The app had a direct effect on reducing the number of cigarettes at first, probably for the first two or three weeks, so not really so long. Then towards Christmas, it would be too easy to pick up … and when I realised that I’d failed; I’d crossed the line, and I hadn’t given up. It just picked up again.
Male, Quit Sense arm, ID120
Some interviews coincided with preparations for Ramadan and Ramadan itself, with Muslim participants discussing how this was a positive factor – helping to motivate and enforce their quit attempt:
Something good that is going to happen – I’m Muslim, so Ramadan is going to begin in less than two weeks. When you’re fasting, you cannot eat, drink or smoke as well. So, I always look forward to Ramadan because it does help with stopping smoking. Because you’re fasting, here in the UK, you’ll be fasting for more like 10 hours a day.
Female, Quit Sense arm, ID208
To be fair, yeah, it’s been okay. It’s kind of gave me a bit of a push to cut down a lot more. I don’t know if you’re aware but it’s Ramadan at the minute as well and I’m fasting. So, I’m hoping that by the end of the 30 days I will have eliminated it altogether, but we’ll just have to wait and see.
Male, Usual care arm, ID209
Economic evaluation
From the reported levels of smoking cessation aid resource use over the 6-month follow-up period (Table 12) we can see that these are generally at relatively low levels, and that e-cigarettes/vaporisers (which are not routinely funded by the NHS) seem to be the most frequently used. This coupled with the finding that 80% of NRT users reported paying for NRT themselves (see Table 12) demonstrates that the recommended NHS and Personal Social Services (PSS) cost perspective (National Institute for Health and Care Excellence, 2013) would miss much of the costs associated with smoking cessation aids. The importance of taking a more societal cost perspective can also be seen when the levels of resource use are combined with estimated unit costs (Table 13) to give total costs (in Table 14 it can be seen that e-cigarettes/vaporisers and NRT account for more than 70% of the total cost when excluding intervention costs).
Unit costs (at 2020–1 financial year levels)
Summary costs (over 6-month follow-up period)
In terms of the cost of the intervention, above it was reported that the mean advertising cost per participant recruited was estimated to be £14.09 (across all participants). We assumed that this was our best estimate of what the recruitment costs would be if the intervention were to be provided again. Additionally, it was assumed that the cost of maintaining the Quit Sense app would be £1500 per annum. When apportioned across the 104 participants in the intervention arm, this equates to £14.42 per participant, and this was assumed to be our best estimate of what the Quit Sense app maintenance costs would be if the intervention were to be provided again. Together this gave a total intervention cost (for future provision) of £28.51 per participant. When added to the total non-intervention cost (for those with data on this variable), the mean per participant total cost can be seen to be approximately £35 higher in the intervention group (without adjustment for any differences there might be between groups) (see Table 14).
The completion rates for the QoL measure (EQ-5D-5L) at the baseline, 6-week and 6-month follow-up points were 100% (208/209), 71% (148/209) and 70% (147/209), respectively. Also, in both groups, compared to the baseline EQ-5D-5L score (see Table 2), the mean follow-up scores for both those who completed the EQ-5D-5L at baseline and 6 weeks, and at baseline and 6 months, were lower than that at baseline (see Appendix 6, Tables 19 and 20). The total QALY scores are also reported in Appendix 6, Table 20; however, as the intervention group had a higher mean EQ-5D-5L score at baseline these need to be treated with caution (adjustment for this is made in the below regression analysis).
Acknowledging that the results need to be treated with caution, for example due to the small sample size, regression analysis was conducted in order to estimate the mean difference in cost between treatment groups (incremental cost of the intervention) and mean difference in effect between treatment groups (incremental QALY gain associated with the intervention). Based on the total costs, which included the estimated total intervention cost (for future provision), the results show that the incremental cost of the intervention was estimated to be £6.68, with an incremental QALY gain of −0.006 (Table 15). However, it should be noted that there was no significant difference in cost or effect on QoL between the two groups and that such differences could therefore have arisen due to chance. Nonetheless, with these mean estimates, as the intervention is estimated to be both more costly and less effective it is referred to as dominated by standard care, and the intervention would not be estimated to be cost-effective. In keeping with this, the cost-effectiveness acceptability curve (CEAC) estimated that there was a 32.4% probability that the intervention was cost-effective at a threshold of £20,000 per QALY (a cost/QALY below £20,000/QALY has been deemed to be cost-effective).71
Estimates of incremental cost, incremental effect and cost-effectiveness
Finally, a sensitivity analysis was conducted using different assumptions about the total intervention cost for future provision. This was not specified in the health economics analysis plan but was justified on the basis that we sought to assess whether the above results might change when more favourable assumptions about the intervention cost were used. We assumed that, in terms of recruitment costs, the difference in cost between the intervention and standard care would be zero as in this trial standard care also had the same estimated recruitment costs. Additionally, the cost of maintaining the app was assumed to be negligible as this might be an appropriate assumption if the app were adopted at scale by a large number of people. The results of this sensitivity analysis (see Table 15) show that when the intervention is assumed to have a zero cost, the incremental cost is estimated to be −£21.84 (the estimated incremental QALY gain was unchanged). It has been argued that the incremental cost-effectiveness ratio values can be misleading with negative cost/effect estimates.74 Accordingly, we estimated the net benefit74 at the threshold value of £20,000 per QALY. In this sensitivity analysis, the net benefit was estimated to be negative (see Table 15), which denotes that the intervention is again not estimated to be cost-effective at the £20,000/QALY threshold. Additionally, the intervention was estimated to have a 38.1% probability of being cost-effective according to the CEAC (see Table 15).
When estimating the treatment effect, estimates have been adjusted for baseline EQ-5D-5L score (QALYs only), age, gender, baseline smoking rate, baseline SES and baseline heaviness index; 95% CI = 95% CI; Nc (Ni) = number of intervention (app) and standard care (control) participants who were included in the analysis; a, estimated probability of the intervention being cost-effective on the CEAC at the threshold (λ) of £20,000 per QALY.
SWAT assessing effect of incentives for increasing follow-up
Baseline characteristics for SWAT groups can be found in Appendix 6, Table 36. At 6-month follow-up, there was no evidence of a difference in overall response rate among participants offered a £20 voucher incentive compared with a £10 incentive (79% vs. 74%, unadjusted OR = 1.35, 95% CI 0.71 to 2.60; p = 0.36; fully adjusted OR = 1.29, 95% CI 0.66 to 2.54; p = 0.46) (Table 16). However, the likelihood of requiring manual follow-up differed between incentive groups; 46% of those in the £20 incentive group required manual follow-up (by text, phone or e-mail) compared with 62% of the £10 incentive group, representing a 49% reduction for the £20 incentive (unadjusted OR = 0.51, 95% CI 0.29 to 0.89; p = 0.018; adjusted OR = 0.53, 95% CI 0.29 to 0.95; p = 0.032).
Effects of incentive value on response rate of smoking status questions at 6 months (self-reported prolonged abstinence) and rate of manual follow-up at 6 months
The log-rank test, which compares expected and observed questionnaire completions over time by incentive group, shows moderate evidence that there was a difference in survival probability between incentive groups, with the £20 incentive group showing on average a lower completion time (median: 7.0 days, IQR 0.08, 18.13) than the £10 incentive group (median: 14.9 days, IQR 0.15, 24.55), which was statistically significant (unadjusted p = 0.008) (Table 17). This is confirmed by a Cox proportional hazards analysis showing that at any particular point in time, the £20 incentive group were 55% more likely (1.55 times) to respond to the 6-month questionnaire, compared to those participants in the £10 incentive group (unadjusted HR = 1.55, 95% CI 1.12, 2.15, p = 0.008). The failure plot () suggested that failure probability increased over time, with a higher probability of completing the 6-month questionnaire when close to 0 day. Adjusting for stratification and prognostic variables for the above analyses did not alter the findings meaningfully.
Effect of incentive value on response time to completing smoking status questions at 6 months (median days [interquartile range])
Failure plot (estimate of 1 minus the survival function) for time taken to complete the 6-month follow-up questionnaire by incentive arm.
Response rates to questionnaire items other than the primary outcome were also higher in the £20 incentive group compared to the £10 group (see Appendix 6, Table 37). For example, 78% versus 68% of participants reported urge items data and 78% versus 66% completed EQ-5D-5L questions in the £20 and £10 incentive groups, respectively.
