Some text in this chapter has been reproduced from Naughton et al.40 This is an open access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) licence, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The text below includes minor additions and formatting changes to the original text.
Study design
This was a two-arm parallel-group randomised controlled feasibility trial, allocating smokers recruited online to a ‘usual care’ arm (link to NHS SmokeFree website) or an intervention arm who received ‘usual care’ plus access to the Quit Sense app (link with a code to app via Google Play app store).
The study also had a nested qualitative process evaluation with a subsample of participants to assess participant experiences of the intervention and how these experiences might help explain the causal pathway towards smoking behaviour change. Usual care participants were also asked about their experience of participating in the study, informing an assessment of the feasibility of randomised study design for a definitive trial.
Before initiation of the trial, the protocol (Naughton et al.,40 all informed consent forms and any material given to the prospective participants were approved by the Wales REC7 NHS Research Ethics Committee (reference 19/WA/0361). A Trial Steering Committee (TSC) with a majority of independent members oversaw the conduct of the trial.
Participants
Inclusion criteria for participation were: a current smoker; age 16 years and above; smoked at least 7 cigarettes per week; willing to make a quit attempt within 14 days of enrolling in the study; own an Android smartphone (version 5.0 or above); a resident in England; and able and willing to provide informed consent using the web-based form. Exclusion criteria were that participants must not have previously participated in the trial.
Setting
The setting was online and the intervention was delivered online via the participants’ smartphones. This matches the ‘real-world’ setting after implementation to maximise validity. Participants saw online recruitment adverts and completed screening, consent, baseline and follow-up measures online.
Online recruitment
Recruitment took place through paid-for online adverts with Facebook (which includes Instagram) and Google search, limited to England-based IP addresses and targeted at Android devices. The period of recruitment was specifically planned to be over 6 weeks. Online adverts were managed by a partner company called Nativve who specialise in digital marketing for recruiting to research studies.
In line with our patient and public involvement (PPI) panel’s recommendations, Nativve, on our behalf, initially tested different advert wording and images to assess which led to the highest recruitment rate using imbedded Google analytics in our study website (using REDCap, Research Electronic Data Capture, a web-based software platform designed to support research studies, with built-in audit trails and data export functions41,42) (see Appendix 1 for advert examples). They then prioritised the best-performing adverts primarily based on the lowest cost per recruit.
The distribution of socioeconomic status (SES) was monitored throughout the recruitment via the study website, using the National Statistics Socio-Economic Classification (NSSEC).43 This was undertaken to monitor whether the sampling strategy was promoting major socioeconomic inequalities of access. It was pre-planned that if after 3 months of recruitment or when 35% of the target sample was reached, < 45% of our sample were categorised as low SES, then oversampling in favour of low SES participants would be undertaken to increase low SES representation. Low SES was defined as individuals who have a semiroutine or routine and manual occupation, class 5 in the NSSEC, or who have never worked or are long term unemployed. This action primarily included using Facebook advert targeting, which allows the targeting of users according to their education, job titles, workplace, etc. We also used Facebook advert targeting to attempt to increase the ethnic diversity of the sample.
Procedure
Enrolment and engagement
Once an individual had clicked on a study advert, they were directed to the study website where they were given information on the study including the downloadable participant information sheet. If, after reading information about participating, they were interested in taking part, they completed a screening survey to ensure that they met the study inclusion criteria. If eligible, they were asked to provide consent to participate by providing an e-signature. Participants were then asked to complete an online baseline questionnaire and afterwards they were randomly allocated to either the usual care or the Quit Sense app arm (see Table 1 for study assessments).
Schedule of participant assessments
To promote continued engagement in the study, 5 days after enrolment all participants were sent a study text message thanking them for their continued involvement in the study. After 12 weeks, participants were sent a postcard (in an envelope for privacy) with study information. The PPI panel recommended that we should provide summary information on the characteristics of participants in the study, such as age groups, and provide recruitment rates and follow-up rates to promote continued engagement, particularly among those in the usual care arm (see Appendix 2).
Interventions
Usual care arm
After randomisation, both arms received an automated text message to their phone providing them with a link to the NHS SmokeFree website (www.smokefree.nhs.uk), and, if requested, also by e-mail. At the time the trial was run, this website provided access and signposting to digital, telephone and in-person cessation support in England. The support offered on the SmokeFree website included information and links to six main types of cessation support: in-person (via a Stop Smoking Service), the SmokeFree National telephone helpline, an online chat facility with an adviser, an e-mail cessation programme, a SMS text message programme and the NHS SmokeFree smartphone app. The main difference between the NHS app and Quit Sense is that the NHS app does not require users to log their smoking behaviour, it does not proactively deliver real-time support to help smokers manage urges, and in addition it is not tailored to the user and most of the tips and support it provides are user- rather than phone-triggered.
Quit Sense app arm
In addition to usual care, Quit Sense arm participants received an automated text message providing them with a link to the Quit Sense app on the Google Play store, along with a unique app activation code. If participants in the Quit Sense arm had not installed the app after 3 days, a reminder text message was sent to encourage participants to install the app. If the participant still had not installed the app 5 days later, a further text message was sent enquiring why they had not installed the app and inviting a reply by text message. They were asked to respond with one of the following reasons for not having installed the app: (1) for no internet; (2) for not enough storage; (3) for forgot; (4) for don’t want app; (5) for problem installing; and (6) for other. If participants replied ‘5’ or indicated a problem installing the app, the lead researcher contacted them manually (by text or phone) to offer help with installation.
Quit Sense is a ‘context aware’ smartphone app, also known as a JITAI, designed for smokers willing to make a quit attempt. It is informed by learning theory (LT) and two theory-guided SMS text message systems,35,44 which are in turn informed by social cognitive theory (SCT).45 Quit Sense targets three key determinants that map on to LT and six that map on to SCT: learned associations between smoking and an individual’s physical environment (LT); learned associations between smoking and mood triggered by the environment (LT); the presence of other smokers (LT, SCT); awareness of smoking triggers (antecedents) (SCT); outcome expectancies of a lapse (SCT); goal/intention for complete abstinence (SCT); self-efficacy in resisting urges to smoke (SCT); and knowledge of and self-efficacy for lapse prevention strategy use (SCT). Twenty-one corresponding behaviour change techniques (BCTs)46,47 are used to target these determinants. See Appendix 3 for a ‘logic model’ for Quit Sense and connecting evidence.
The central feature is ‘geofence-triggered support’ (GTS), which is orientated around three stages within the app (see Appendix 4 for screenshots):
Stage 1 (‘train the app’): The user sets a quit date (default suggestion is 7 days’ time) and trains the app to learn about their smoking behaviour via the app’s smoking reporting tool. This requires the user to report in the app each smoking episode and the situational context when they ‘light up’ in real time [stress, mood,48 urge strength,49 setting description (home, work, working from home, socialising, other), presence of other smokers], while the app records geolocation using location sensors. If a user reports smoking more than once in the same location (as defined by the app), the app creates a geofence (a circular virtual perimeter) around that location. Each geofence zone represents a high-risk area for smoking for that user, with users typically having multiple geofence zones in different locations.
Stage 2 (‘commit to quit’ – a 28-day abstinence challenge): After their quit date has passed, the app monitors the user’s location. If they enter (and remain in for at least 5 minutes) a geofence, the app will then determine whether to trigger a GTS message. This decision is informed by the user’s smoking reporting history, including time of day, for that location. Messages are individually tailored using the context information collected in stage 1 and provide lapse prevention support (strategies, motivation enhancement, encouragement, etc.) to help users avoid or cope with triggers they are likely to experience in that location that cue urges to smoke. Further decisions about whether to trigger messages are made after each 3-hour interval of remaining in that location (default between 8 a.m. and 9.30 p.m. or defined by the user). New geofences are created in this stage if smoking is reported more than once in the same location, as with stage 1.
Stage 3 (maintaining abstinence – ‘maintain the change’): The app continues to deliver GTS for 2 further months but reduces the frequency by one-half every month (i.e. one in every two scheduled messages sent for first month, then one in every four scheduled messages sent). After 3 months post quit date, the GTS support stops, unless the user opts to restart their quit attempt, which they can do at any time (see feature description below).
Quit Sense has six additional features (relevant to all stages, unless specified otherwise):
After each smoking report is submitted, tailored feedback is provided on screen. This feedback in stage 1 includes messages focused on preparing to quit, boosting motivation and self-efficacy. These messages are tailored to nine characteristics collected during app initiation (hardest situation to avoid smoking, number of cigarettes smoked, living with a smoker, longest quit attempt to date, primary reason for quitting, perceived primary downside to quitting, self-efficacy, gender and pregnancy status). In stage 2, the feedback after a smoking report is submitted is orientated around lapse and relapse prevention.
An ‘End of Day’ (
EoD) survey that users are invited to complete each day, recording the number of cigarettes smoked that day, craving
48 and self-efficacy
44,45 (stages 1 and 2 only). After the
EoD survey is completed, the app provides a feedback message. If the number of cigarettes reported in real time during the day is different from the number reported in the
EoD survey, the feedback message promotes adherence to logging smoking behaviour.
A ‘My Profile’ section including number of days quit, money saved (based on smoking rate), a calendar showing emoji feedback with use of a smiley, neutral and sad emoji to reflect value ranges for smoking, cravings and self-efficacy for each day. In addition, smoking pattern feedback is provided using graphical and written summaries for smoking triggers based on the user’s historical reporting of smoking behaviour.
A ‘Library’ of quitting advice split into six categories: ‘getting ready’, ‘boost your motivation’, ‘effects of smoking and quitting’, ‘early days of quitting’, ‘staying quit’, ‘smoking after your quit date’. It includes an option for users to write and submit their own support messages.
Scheduled non-tailored daily support messages delivered each morning orientated around the quit date (stages 1 and 2 only) – targeting outcome expectancies, motivation, preparation, self-efficacy (three sets of different messages to prevent message duplication in successive attempts).
The option for the user to reset their quit date in stage 1, if they are not feeling ready, or stage 2 if they relapse. Relapse is defined as more than one smoking episode reported each day over 2 consecutive days during a quit attempt (either through smoking episode logs or the
EoD survey). If this occurs, users are invited to either recommit to their quit attempt and continue or reset their quit date, and they then go back to stage 1.
As described earlier, Quit Sense users are also able to use an audio-record feature embedded in the app where they can provide feedback on their views and experiences of using Quit Sense, as part of the process evaluation. They receive two prompts and two reminders to provide such feedback at key time points: 2 days before quit date, with a reminder 1 day later if no feedback provided, and 10 days after quit date, with a reminder 1 day later if no feedback provided.
Follow-up
One day before the 6-week questionnaire was due to be completed, participants received a text message letting them know that they would receive the 6-week questionnaire link the next day, which they were sent the following day and informed that they would receive a £5 Amazon voucher for completing it. If participants had not completed this after 4 days, they were sent a reminder text message. If it was still not completed a further 3 days later, they were informed by text message that a researcher would get in contact to collect the information over the phone, and they were also given the questionnaire link again in the text message. Shortly after this last message was sent, if the questionnaire had still not been completed, the researcher contacted them by telephone. Up to five contact attempt episodes were made.
Included in the 6-week questionnaire was a question asking about interest in participating in a qualitative interview by telephone to investigate experiences of participation in the trial and, from Quit Sense arm participants, to gain feedback on the app. We planned to interview 20 participants (15 in the Quit Sense arm, 5 in the usual care arm). Purposive sampling was used to identify participants with varied characteristics. For further information, see Qualitative process evaluation.
Other approaches to increase response rates were also used. For example, participants were issued with a £5 voucher following completion of the 6-week follow-up questionnaire and interview participants were given a £20 gift voucher.
Six and a half months after enrolment (referred to as the 6-month follow-up), all participants were sent a text message inviting them to complete the final questionnaire. As with the 6-week follow-up procedure, a reminder text message was sent and a message about contacting them by telephone instead to complete the questionnaire at 4 and 7 days after the initial text message, respectively. As part of the SWAT nested within the trial, participants were randomly allocated to be offered either £10 or £20 to complete the 6-month follow-up questionnaire (see SWAT for more details). If manual follow-up by telephone was unsuccessful, participants were sent a text message inviting them to respond to the primary smoking outcome question. Participants who reported a 7-day abstinence in the 6-month questionnaire were informed at the end of the questionnaire that they would be sent a saliva test kit in the post and that they would receive a further £5 for returning it. This information was also sent by text message.
Measures
Feasibility outcomes
Feasibility outcomes used to estimate key parameters to inform a fuure trial50 include:
completeness of the anticipated primary outcome for a future definitive trial (6-month self-reported abstinence with biochemical validation; see
Smoking and related outcomes);
abstinence rate of usual care arm, using the anticipated primary outcome for a future definitive trial;
cost per recruit, based on advertising costs;
rates of app installation and use and acceptability [recommend Quit Sense to a friend and ease of use (five-point scale)];
completion of smoking cessation-related resource use, including usual care use, and
QoL (
EQ-5D-5L)
51 data;
hypothesised mechanisms of action of Quit Sense;
participant experiences and feedback on app usage.
Smoking and related outcomes
The primary abstinence outcome for which we monitored completeness of ascertainment was based on the Russell standard.52 Abstinence was defined as: self-reported abstinence in the previous 6 months allowing for no more than five cigarettes and not smoking in the previous week, biochemically validated by a saliva cotinine concentration of < 10 ng/ml,52,53 and for those using nicotine substitution (e.g. NRT, or e-cigarettes), an anabasine concentration of < 0.2 ng/ml.53,54 Given evidence that some e-liquid, believed to be that made outside of the UK, can contain anabasine53 we monitored the proportion of e-cigarette users who reported abstinence from smoking but had an anabasine level above the chosen threshold.
Given recommendations to include additional smoking outcomes based on different time periods of abstinence and time points,55 we also measured 7-day point prevalence abstinence at 6-month follow-up (self-report and biochemically verified) and 7-day point prevalence abstinence at 6-week follow-up (self-report).
We collected the following hypothesised mechanisms of action at 6-week follow-up:
Lapse incidence (any smoking, even a puff) in the first 2 weeks since the initial quit attempt or since enrolment,
9 which is highly predictive of relapse.
6,77Lapse prevention strategy use,
22 which is associated with lapse prevention.
16 This was a mean score of the frequency of use [4 categories: not at all (1), 1–5 times (2), 6–10 times (3), more than 10 times (4)] for 7 strategies for avoiding smoking and 13 for coping with the desire to smoke.
Smoking cessation self-efficacy,
44,45 which prospectively predicts lapse and relapse to smoking.
56 This self-efficacy measure includes four items which are scored on a five-point scale from not at all (1) to extremely (5) and then averaged; self-efficacy to quit smoking for good (adapted from original) and self-efficacy relating to avoiding smoking during habitual (after eating), social (when with other smokers) and emotional (when anxious or stressed) situations. This measure demonstrates good internal consistency (α = 0.81).
44The strength of urges to smoke (
SUTS) measure,
48 which prospectively predicts abstinence and is superior to other urge-measures in doing so,
49 and frequency of urges to smoke (
FUTS).
48Automaticity and associative processes subscales from the Wisconsin Inventory of Smoking Dependence Motives (WISDM-37).
57
WISDM items are scored on a seven-point scale from ‘not true of me at all’ (1) to ‘extremely true of me’ (7). The automaticity subscale is derived from four items and the associative processes subscale is from three items.
Sample size
As this was a feasibility study, sample size was determined as that likely to provide reasonably accurate estimates for key ‘full trial’ parameters, rather than to detect a difference in smoking outcome between the two arms.
In line with recommendations that feasibility studies with binary outcome measures recruit at least 60 participants per group (minimum of 120) and a maximum of 100 per group (maximum of 200),58 the proposed sample size for this feasibility trial was 200 smokers (100 per arm). Originally, we planned to recruit 160 participants but during recruitment we found that the cost per participant was much lower than anticipated and the speed of recruitment could be controlled by increasing the daily spend on adverts. Therefore, with approval from the Trial Steering Group, sponsor and National Institute for Health and Care Research, and having notified the REC, we increased the planned sample size to 200. This sample size would enable key parameters to inform a future definitive trial to be estimable within the following precision [defined as the 95% condifence interval (CI) half-width]:
Primary outcome completion – we estimated a follow-up rate for self-reported smoking status at 6 months of 80%, with precision of ± 6%. This was based on smoking cessation evaluation trials with online recruitment that were most similar to the proposed trial, namely trials of a cessation smartphone app
59 (2-month follow-up = 84%) and a cessation website
60 (7-month follow-up = 72%). A Cochrane Review of web-based cessation trials showed that few report retention rates higher than 80%.
61 For biochemical validation, we estimated that 75% of participants reporting abstinence would return a saliva sample by post, with precision of ± 22%. This was based on estimates from a cessation website trial with a 6-month follow-up
60 (75%) and a SMS text message cessation intervention trial with a 3-month follow-up
33 (80%).
Cessation rate in usual care arm – based on the control arm abstinence rate in a Cochrane Review of mobile phone-based cessation interventions,
62 we estimated an abstinence rate of 5% at 6-month follow-up, providing precision of ± 5%
App installation and initiation – we estimated that 85% of intervention participants would install the app, with precision of ± 7%. This was based on installation rates (92%) provided by a study examining engagement with a cessation app among an online population of smokers
63 – we assumed slightly lower installation rates as, unlike this other study, we did not use financial incentives to promote initial engagement.
App engagement – in our acceptability study, 71% used the app for more than 1 week (a time frame deemed meaningful by our
PPI panel). With a similar rate in this trial, precision would be ± 9%. This is in line with engagement rates reported in other studies identified when this study was designed.
63
Randomisation
Sequence generation, allocation and blinding
Randomisation was stratified by smoking rate (< 16 vs. ≥ 16 cigarettes/day; based on mean smoking rates from trials recruiting smokers online)59,64 and SES (low vs. high), based on the NSSEC self-coded method.43 Participants were randomised to the usual care or Quit Sense arms in a 1 : 1 ratio. Allocation sequences were generated by a computer program (using REDCap) with random permuted blocks (varying block sizes). Randomisation was integrated into the enrolment process on the study website. As allocation was integrated into the study website, the allocation sequence was concealed from participants until assignment and concealed from members of the trial team, other than the statistician, developers of the study database and the lead researcher who needed to be unblinded for potentially providing app installation support and selecting participants to interview as part of the qualitative evaluation.
Statistical methods
The feasibility and acceptability of all measures were assessed by their level of completeness and via interviews with a subsample of participants.
Outcomes
Completeness of the anticipated primary outcome for a future trial (feasibility outcome i.) and the abstinence rate using this anticipated primary outcome for the usual care arm (outcome ii.) are described as proportions with 95% CIs and translated into interpretable probabilities using the Bayesian approach relevant in preliminary trials65 with the objective of powering the definitive trial. This method produces the estimates of the probability that the underlying odds ratio (OR) is 1.7 or higher, or at other values relevant to providing plausibility to the effect size used to power a subsequent trial such as 1.5 or higher, or 2.0 or higher. It is based on treating the CI for the log OR as a normal distribution. Due to the small number of events in the feasibility trial,65 a method involving a continuity correction was used for the log OR and its standard error.66
Cost per recruit (outcome iii.), rates of app installation (outcome iv.) and completion of smoking cessation-related resource use and QoL (EQ-5D-5L) (outcome v.) are also described using summary statistics with 95% CIs.
For the primary outcome measure, the intervention effect on abstinence at 6 months (self-reported, with abstinence validated by a saliva test) was estimated using multiple logistic regression, providing ORs with 95% CIs, while adjusting for stratification variables (smoking rate and SES at baseline), prognostic covariates (heaviness index at baseline) and treatment group, defined by the Statistical Analysis and Health Economics Plan (SHEAP; https://osf.io/mt6s5/). For the primary outcome measure, we assumed that those participants withdrawn, non-responders or missing were not abstinent,52 as a conservative model. Analysis results using exact inference are also presented, as a more appropriate approach, due to the relatively small numbers of abstinent participants.
Between-group analyses of mechanisms of action variables at 6 weeks were undertaken for lapse incidence (2 weeks, 1 month), lapse prevention average scores and subscale scores, WISDM subscale scores, strength of urge to smoke, frequency of urge to smoke and self-efficacy average scores. An appropriate parametric/non-parametric statistical test was used to test for a difference between treatment groups, with the two-sample t- test for continuous variables reasonably satisfying the normality assumption, and the chi-squared or Mann–Whitney test otherwise, or for categorical variables.
Sensitivity analyses of abstinence at 6 months were conducted using an analogous logistic model as described for the primary outcome, excluding withdrawals, as well as a complete case analysis. Secondary analyses, again using an analogous logistic model, were carried out for 7-day abstinence at 6 months with biochemical validation by saliva test and self-reported 7-day abstinence at 6 weeks and 6 months. A missing data analysis using the full information maximum likelihood method was originally planned; however, for a binary outcome, it was unfortunately not currently technically possible in commonly available statistical software.
Economic evaluation
The methods used to estimate the costs and benefits have been described in detail in the SHEAP. Briefly, they are as follows: we aimed to collect baseline and follow-up data that would enable us to inform the decision as to how cost (based on resource use data) and benefit (in terms QoL) data are estimated in any future more definitive study. Completion rates were thereby estimated.
In terms of costs, a societal cost perspective was adopted, where we estimated the costs associated with the intervention (e.g. app maintenance) and smoking-related costs to both the individual (e.g. NRT) and the NHS [e.g. NHS stop smoking services/general practitioner (GP) visits]. Appropriate unit costs67 were attached to all reported items of resource use (at 2020–21 price levels) in order to identify major cost drivers.
In line with guidance at the analysis stage,68
QoL was measured via the EQ-5D-5L51 [as this enables the calculation of quality-adjusted life-year (QALY) scores] and scored via the crosswalk mapping function.69
Additionally, as outlined in the SHEAP, a preliminary cost-effectiveness analysis was conducted, the results of which need to be treated with caution due to the small sample size. A complete case analysis was conducted, where only those with complete cost (based on the 6-month resource use questionnaire) and outcome (EQ-5D-5L data at baseline, 6 weeks and 6 months) were included. Seemingly unrelated regression analysis70 was undertaken to estimate the mean incremental cost and mean QALY gain associated with Quit Sense compared to usual care. In the absence of dominance (where one intervention is both more costly and less effective), the incremental cost-effectiveness ratio (incremental cost/QALY gain) was estimated and assessed in relation to a range of cost-effectiveness thresholds (e.g. £20,000–30,000 per QALY)71 to provide a preliminary assessment of whether Quit Sense constitutes value for money. The associated level of uncertainty was also estimated.
SWAT assessing the effect of incentives on follow-up rates
SWAT objective
The objective of the SWAT was to determine whether the offer of a higher financial incentive for questionnaire completion improved response rates at 6-month follow-up. Findings will be used to inform a future definitive trial.
Participants were randomly allocated to either receive an incentive worth £10 (the rate advertised to them originally) or £20 (the increased incentive) for completing the ‘primary outcome’ question at 6-month follow-up.
All participants were sent automated notifications by SMS 1 week before the 6-month questionnaire was due (pre-questionnaire notification as planned), at the 6-month time point with a link to the questionnaire and 7 days after the questionnaire due date with a reminder message. Two weeks after the questionnaire due date, the Research Associate attempted manual follow-up to collect the data by telephone (as previously planned and this still entitled the participant to the incentive) and a final text message primary outcome question was sent if there was no success from the manual follow-up. The automated notifications reminded the participant of the follow-up incentive, at the previous or increased rate, depending on SWAT randomisation group. The SWAT protocol has been accepted on to the Medical Research Council (MRC)-funded SWAT store (SWAT #164; www.qub.ac.uk/sites/TheNorthernIrelandNetworkforTrialsMethodologyResearch/SWATSWARInformation/Repositories/SWATStore/).
SWAT outcome measures
Primary: Questionnaire response rate. This included those responding by all methods (text, online or by telephone follow-up) within the allowable window for the primary outcome question (smoking status at 6 months).
Secondary:
proportion of participants requiring manual follow-up by the lead researcher (i.e. cost implications of staff time);
time taken for participants to respond to the questionnaire;
completeness of responses (counted as the number of missing items from all items asked, including secondary outcomes) and comparison to the response rate for the primary outcome (smoking status at 6 months).
SWAT analysis plan
Sample characteristics at baseline (by incentive arm), response rates at 6-month follow-up and other related response data, including completeness of responses, were summarised. For the primary analysis, a logistic regression model was run, using Wald CIs for the OR, including stratification variables as covariates in the model and any participant characteristics that appeared to be imbalanced between the different incentive groups, with response (binary) as the dependent variable. For secondary analyses, further logistic regression models were run using the same approach, and covariates for assessing the effect of incentives on manual follow-up rates and Cox proportional hazards analysis were undertaken with the log-rank test to assess the effects of incentives on response time.
Qualitative process evaluation
An embedded qualitative study was undertaken to assess participant perspectives. The aims of the qualitative process evaluation were:
to understand participant experiences of participating in the study;
to gather user views on app usage to inform further optimisation.
A purposively selected subsample of participants was invited to take part in a qualitative interview. The process evaluation was guided by the MRC guidance on process evaluation of complex interventions.72 Part of this was to use interview data to better understand participant experiences of the intervention and how these experiences might help explain the causal pathway towards smoking behaviour change where it was observed. Qualitative interview topic guides (see Appendix 5) were used to probe for in-depth descriptions of experiences (e.g. of cravings experienced and how the app delivered support that may have helped ‘in the moment’, or not). Detailed information gathered qualitatively allowed for a richer understanding of situations and experiences than would be possible to gather through the app or quantitatively. In addition to using interview data, to meet the second aim of the process evaluation study, the Quit Sense app included an audio recording feature (‘auto-process evaluation’) so that users could leave feedback on their views and experiences of using the app. The study therefore also assessed the feasibility of this user-initiated process evaluation approach where data can be collected in a highly ecologically valid context. Any relevant data submitted were explored alongside the interview data, to potentially provide a broader range of views from a more user-controlled perspective.
At the 6-week follow-up, participants, including those in the Quit Sense and usual care arms, were invited to participate in a telephone qualitative interview as part of the process evaluation (aiming for a recruitment ratio of 3 : 1 from the Quit Sense and usual care arms, respectively). We used purposive sampling for intervention participants, where possible, by aiming to recruit individuals with high and low SES (as per study stratification), varied rates of app engagement and to include both abstainers and continuing smokers. For control participants, a mixture of high and low SES individuals, including smokers and abstainers, were purposively sampled. During the interview, intervention participants were asked about their experience and views of self-monitoring smoking using the app, including any environmental context factors/triggers missing from the smoking report survey, probing for detail and specific examples in order that causal pathways to behaviour change could potentially be identified. The interview topic guide was employed flexibly, allowing participants to discuss the study and app engagement in a manner that was comfortable for them. According to the topic guide, interviewees were also asked about the ease of using the app; types of messages they liked most and least; their views on the timing of messages when entering or dwelling in a smoking zone (geofence); their views on the ‘My Profile’ and ‘Library’ sections of the app; their views on features they would like that are not currently provided; and how personalisation of the app could be improved. Control participants were asked about their experience of participating in the study, informing an assessment of the feasibility of randomised study design for a definitive trial. Both groups of participants were asked about their use or interest in the use of other smoking cessation aids and different types of cessation support available including e-cigarettes; use of other smoking cessation apps; other health and well-being apps; and NHS-orientated smoking cessation support options.
Interviews were audio-recorded by the interviewer (AH) and transcribed to intelligent verbatim standard by an administrator. Interview transcripts were checked for accuracy, and all were anonymised (removing names, places and other distinguishing features). NVivo was used to support the qualitative analysis.
Analysis of process evaluation data
An inductive thematic analysis of the first three to four interview transcripts was initially undertaken (by AH).73 AH and CN then met to discuss coding and to develop a coding framework. This was agreed by consensus. AH then continued to code the remaining data according to the framework. Coding consistency was checked through the independent coding of 10% of the data (conducted by CN). At verification meetings, AH and CN agreed codes by consensus and were satisfied that coding was consistent. Refinement of themes continued until the themes were agreed. CN and AH met regularly to discuss the coding process and ultimately agreed on the key themes arising from data analysis for reporting purposes.
Analysis of process evaluation data focused on identifying key themes associated with using the app and views and experiences of using other relevant apps, supplemented with a descriptive analysis of relevant experiences to begin to hypothesise intervention causal pathways towards behaviour change. A second stage of analysis worked with descriptions of experiences of lapse or lapse avoidance in the context of the app and considering unique participant contexts. These are reported as vignettes and used to explore participant experiences and to identify mechanisms of action and casual pathways towards behaviour change. There were not many audio-recordings submitted via Quit Sense’s auto-process evaluation feature and so, as pre-planned, we did not code these data for thematic analysis but instead provided a basic summary of the content.
