1. Introduction
On 23 March 2020, the UK government enacted measures that were included in the Coronavirus Act 2020 and recommended that everyone (except in certain essential circumstances) must stay in their homes [1]. Consequently, many of the elements enabling and supporting participants in the Integrated Healthy Lifestyle Service (IHLS) programme became impossible to deliver due to the pandemic of COVID-19. In order to continue delivery, the IHLS weight management (WM), smoking cessation and physical activity (PA) intervention sessions transitioned from face-to-face to online delivery.
Current evidence for the successful implementation of exclusively online health behaviour change offers is limited [2]. A single form of service delivery is never likely to meet all individual’s needs [3]. However, given the frequency of online health interventions is increasing rapidly, more advanced methodologies are needed to explore the components that can make such interventions successful for as many individuals as possible [4]. Consequently, process evaluations of implementation fidelity should become an integral part of the delivery and evaluation of all digital health behaviour change research.
Process evaluations can contribute to pandemic preparedness by improving both robustness and agility of systems [5]. It is argued that robust systems employ expertise and tools that are fit for purpose, while agile systems are built to be adaptable and flexible in quickly changing conditions [6]. Classical implementation fidelity tools and measures can be operationalised within the context of the COVID-19 pandemic, and at each stage of the pandemic. Doing so empowers researchers to apply their skill sets toward optimising uptake of evidencebased interventions [6]. Implementation focuses on characterising and enhancing the acceptability, adoption, appropriateness, feasibility, fidelity, cost, penetration, and sustainability of evidence-based interventions in distinct settings [7]. Within the context of COVID-19, it is proposed that such are assessed across individual, provider (clinical providers and public health workers broadly), and organisational levels to identify determinants (barriers and facilitators) to implementation across all levels of an organisation [6]. Assessing these elements of implementation provides a set of guidelines for translating research into practice and enables more accurate inferences to be made about intervention effectiveness [8].
Assessing intervention fidelity is identified as being a key challenge for health behaviour change interventions [9] as public health impact is dependent on the extent to which efficacious interventions are disseminated with fidelity into real world settings, then maintained, and institutionalised [10]. Consequently, implementation fidelity assessment can be used to expand effective uptake of online health behaviour change interventions, as well as communicating data back to decision makers in a format that is actionable. To ensure that evaluations are representative and operational, data collection should account for the perspectives of multi-level stakeholders with responsibilities ranging from planning to ground level service delivery.
The aim of the current study was to investigate the impact of the changes made to a UK-based IHLS during the COVID-19 lockdown across WM, smoking cessation and PA services offered, with the following objectives posed as evaluation questions:
- How does the transition from face-to-face to online sessions affect staff adherence towards key performance indicators (KPIs)? How are IHLS staffs supported to deliver sessions remotely? How does this support affect their confidence and ability to deliver the intervention as intended? Are there ways in which staff can be better supported to deliver interventions remotely?
- How does the transition from face-to-face to online sessions affect client uptake and retention? Are there ways in which clients can be better supported to engage with online sessions?
Given the aim and objectives, the Evidence Integration Triangle [11] was adopted as the overarching theoretical framework. Through the prompt identification of barriers and facilitators of reasons for attrition, the framework allows for the exploration of the three main evidence-based components of program/policy, implementation processes and measures of progress. In addition to publication, results and analysis from this study are to be fed back to key stakeholders, IHLS staff and clients in order to assess, evaluate and promptly inform adapted future iterations of each offered service. Adoption of such an integrated framework allows for more consistent mapping, evaluation and incorporation of successful methods and strategies for modifying behavioural determinants [12].
2. Materials & Methods
The current study provides qualitative data to assess the implementation fidelity of an IHLS’s transition from face-to-face to online delivery during the COVID-19 lockdown. The observed IHLS focuses on reducing health inequalities among vulnerable and at-risk groups, including within areas of deprivation. Specifically, the WM, smoking cessation and PA services are compliant with respective National Institute for Clinical Excellence (NICE) guidelines [13-15]. Compliance with such guidelines includes the recruitment, training and support of staff to ensure fidelity. The WM service is for all adults (aged >16 years) with a body mass index (BMI) of 30 kg/m2 or above (or 27.5 kg/m2 with comorbidities), with a focus on enabling clients from the 40% most deprived lower super output areas (LSOAs) to access the service. The smoking cessation service is suitable for clients of any age who have smoked a tobacco product in the last 48 hours. The service can be accessed via self-referral or referral from a health or social care practitioner. The PA service is designed for adults (aged >16 years) who are currently participating in less than 30 minutes of moderate activity per week. In line with NICE guidelines, the service supports and encourages physically inactive individuals to increase their levels of PA.
The service is a partnership between a UK-based university, established IHLS provider and was commissioned by a County Council in the East of England. The UK-based University commits a direct investment into research and evaluation to support the IHLS. Although reflecting key elements of the Public Health England (PHE) research and analysis guidelines [16], each service is predominantly developed and delivered in line with the required annual KPIs as stipulated by the commissioning body. This remained the case during the COVID-19 lockdown.
2.1 Design
A qualitative research design was adopted to enable a deep understanding of IHLS implementation fidelity.
Between March and September 2020, a pragmatic sample of 167 online Qualtrics surveys were conducted across IHLS staff (n= 44) and clients (n= 123). IHLS staff consisted of senior management, team lead and practitioner roles. Clients who were currently attending or had attended one or multiple IHLS services since the transition to online sessions in January 2020 were invited to take part. Pragmatic sampling has been adopted in survey research previously [17] and thus, the current study extends the applicability of these methods.
Two online surveys were designed for IHLS staff and clients, respectively. Surveys were administered via the Qualtrics Online Survey software, with participants providing informed consent prior to completion. Each survey consisted of ten questions addressing barriers, facilitators and opportunities towards delivering (IHLS staff) and attending (IHLS clients) online service sessions during the COVID-19 lockdown. An example question from the client survey was: “Has the COVID-19 pandemic had an impact on your ability to attend IHLS sessions? If yes, which sessions and what impact has this had?” Consequently, questions demonstrated aspects of face validity as they were transparent and relevant to the priority population [18]. Objectivity was maintained by the lead investigator as the resultant qualitative data aligned to the a priori Evidence Integration Triangle [11] framework and was fit to serve as evidence for satisfying the research question [19] of evaluating implementation fidelity of a UK-based IHLS.
Institutional ethical approval was received from Leeds Beckett University’s Research Ethics Sub Committee (approval number 68268). All data were anonymised, and survey responses coded throughout to ensure confidentiality.
2.2 Data coding and analysis
The pen profile approach presents findings from content analysis via a diagram of composite key emerging themes. In summary, deductive content analysis was initially adopted to categorise IHLS staff and client survey data into two a priori staff and client pen profiles. Inductive analysis then allowed for emergent themes to be created to expand the pen profiles. Data were then organised schematically to assist with interpretation of the themes and verbatim quotations used to provide contextand verify participant responses. Quotations were labelled by participant number (Pn).
Methodological rigour was demonstrated through a process of triangular consensus between members of the research team. This offered transparency, credibility and trustworthiness of the results, as the data were critically reviewed using a reverse tracking process from the pen profiles back to the verbatim transcripts, providing alternative interpretations of the data. All investigators were in agreement with the initial interpretation of results made by the lead investigator.
3. Results
The a priori staff and client COVID-19 pen profiles and emergent themes are presented in Figures 1 and 2, respectively. Figure 1 outlines largely positive staff comments in relation to the emergent theme's integration, training, support, and client engagement. Negative staff comments are noted in relation to the emergent theme KPIs. Figure 2 outlines largely positive client comments across all emergent themes including session delivery and content, support, perceived competence, and health benefits. Table 1 displays a specific frequency count of positive and negative emergent theme mentions across the IHLS staff and client sub-groups.
4. Discussion
This is one of the first studies to provide a comprehensive exploration of implementation fidelity within a UK-based IHLS during the COVID-19 lockdown. Such findings will be critical in ensuring online health behaviour change interventions are tailored as best as they can be to individual needs to positively impact as significant a proportion of the population as possible.
4.1 COVID-19 Staff
4.1.1 Integration
IHLS staff noted that the COVID-19 lockdown had provided a positive (n=20) opportunity for increased communication and subsequent integration across the varying levels of staff hierarchy (e.g. senior management, management, team lead, and practitioner team members). Specifically, good communication between IHLS senior management, management and ground level practitioner team members removed the top-down approach to information dissemination noted among practitioners prior to COVID-19.
“There is more integration than ever before across staff and services. We are all communicating and starting to share ideas for the first time.” (P17)
Working in an integrated manner has been shown to result in greater impact, insight and subsequent benefit for all parties involved within an organisation [11]. Through this increased level of sharing successes and failures, an understanding of the optimal content and delivery of online health behaviour change interventions can be achieved. In contrast, negative comments (n=7) were noted regarding communication channels between key stakeholders outside of the IHLS staff hierarchy. Specifically, commissioner and IHLS staff communication fell short of enabling the creation of the sustainable, integrated and collaborative structure warranted to alleviate mental health problems [20]. Specifically, a top-down approach to information dissemination was noted by IHLS practitioners (n=5).
“Outcome expectations are not communicated clearly by commissioners we are just told what to do and we feel like we have no choice or say in what works or doesn’t work. This causes emotional responses such as anxiety, stress and general worry.” (P19)
Efforts should be made to develop spaces that facilitate ongoing dialogue and mutual support among key stakeholders and staff across all hierarchical levels. Specifically, initiatives adopting a bottom-up approach to information dissemination have greater potential to change working routines as they enable staff to move towards more collaborative and coordinated work [20]. Learning from the transition from face-to-face to digital delivery during COVID-19, a bottom-up approach facilitating discussion and support could be conducted digitally to further increase staff efficacy and competence of online communication and delivery.
4.1.2 Training
Staff training and development is key to ensuring intervention effectiveness and overall success, especially when moving away from traditional interpersonal interventions to those that are exclusively digital [21]. Whilst recognising that there were challenges in the speed of transition from face-to-face to online delivery, which was necessary for IHLS continuation, the largely positive comments (n= 28) regarding training to deliver online sessions highlights the capabilities of the IHLS staff in adapting to adverse situations in pursuit of a common goal.
"I am relishing the adaptations to my role and overall job expectations as I am expanding my skill set and meeting a broader range of clients." (P39)
Along with incorporating prior knowledge and expertise, practitioners received tutorials and fact sheets detailing how to set up and run online sessions.
“Tutorials and fact sheets were sent out to staff which has been invaluable.” (P17)
A previous evidence-based group health behaviour change intervention (Healthy IDEAS) noted that providing practitioners with detailed scripts, descriptions and guidelines for each intervention component could increase fidelity to provider training [22].
4.1.3 Support
Staff were keen to develop and strengthen support links both within and outside of their own intervention teams. Specifically, staff at ground level noted positive comments (n=29) regarding support and felt they had the necessary guidance at the organisational level to engender a culture where they felt confident to make decisions and take actions.
“Regular virtual team meetings have built rapport and an overall team morale and unity across all services. Within the IHLS, we feel we are valued, and we all believe in the products we are delivering.” (P35)
Within a collaborative organisation, authority should be only partly tied to a given position or role and partly worked out as partners and team members agree on the best way to address a specific problem and reach a common goal. Evidence supports the use of a fluid hierarchical system that allows for organisational adaptive responses in times of instability and changes, such as was experienced during the COVID-19 lockdown [23]. Although such a system was noted to be the case for the majority of practitioners, team leads and managers, a minority of practitioners (n=6) described staff as being resistant and obstructive towards the transition from face-to-face to online delivery methods.
"Some team members are resistant to change, despite coming from commissioners, they are very obstructive to the changes in delivery and working methods which the contract has dictated." (P4)
Previous research shows that resistance to change is not uncommon where organisations decide or are required to take a sudden innovative change of approach, such as had to be undertaken during the COVID-19 lockdown [24]. To overcome such resistance to change, it is important that organisations implement a supportive and open communication culture to understand the motivation behind various levels of resistance to change [24]. Adopting a bottom-up approach is a first step in achieving this environment [20].
4.1.4 Key performance indicators (KPIs)
COVID-19 has challenged the way staffs have been taught to deliver care, as roles and responsibilities have been redefined, new tools and processes implemented, and cross-professional and cross-sectoral collaboration formalized [20]. This can often lead to resistance, resignation or disregard. Existing literature suggests that both trustbased (relationship dynamics) and control-based (organisational dynamics) governance mechanisms play a crucial role in partnership development [25]. There is widespread agreement that a bottom-up approach is required, whereby the purpose and benefits of the change should not only be understood and embraced by staff at ground level, but also coproduced with them through incremental changes and by fostering distributed leadership [26]. Conversely, staff expressed negative views (n=36) regarding the KPIs set during the COVID-19 lockdown.
“Current KPI targets don’t reflect the drastically changed circumstances we are dealing with (during the COVID-19 lockdown). This creates more anxiety, stress and depression in the workplace and affects team morale from the outset.” (P3)
“Commissioners seem slow to react to changing circumstances and don’t seem to understand what is actually happening at ground level.” (P5)
Such results reflect a top-down approach to information dissemination with regards to commissioners and the KPI targets set for IHLS staff during the COVID-19 lockdown. In the absence of shared professional and organisational visions, goals and targets, there is an absence of collaboration and this inevitably raises much frustration among staff, who feels vulnerable, as expectations are placed on them, which too often feel unrealistic. Although the IHLS is grounded in systems thinking, the underlying approach to overall targets remains a top-down process. Consequently, efforts should be made by key stakeholders outside of the IHLS staff hierarchy to further share their thoughts, reasoning and evidence behind goals and targets to IHLS staff, as well as taking onboard suggestions from those staff operating at ground level.
4.1.6 Client engagement
Although client engagement was noted as positive (n=15) by practitioners for those who did attend online sessions, negative comments (n=7) were noted in relation to vulnerable and under-served groups within areas of deprivation, both key target groups stipulated by current IHLS KPI targets.
“Referrals are lower across the board as can’t get into schools, events and leisure centres to promote.” (P8)
“It is hard to reach thelow-income families as they lack the technology to engage.” (P34)
Ensuring that plans for encouraging and maintaining meaningful engagement are in place before rolling out health behaviour change programmes is vital [27]. Engagement approaches that help clients understand the benefits of health behaviour change (e.g. weight loss and mental health) through online campaigns, social media exposure, website information, and radio advertising are warranted to ensure as many clients as possible are aware that such programmes are available, appropriate and beneficial.
4.2 COVID-19 clients
4.2.1 Session delivery and content
The main change to the IHLS during the COVID-19 lockdown was a transition from face-to-face to online delivery. These changes might have important implications for the clients’ experience of the intervention. Although many clients were aware that a transition to online delivery was compulsory, concurrent with previous research [28], negative comments (n=11) regarding session delivery and content noted by clients revolved around a preference to return to ‘normal’ face-to-face delivery as soon as possible. Previous studies have found that face-to-face delivery methods facilitate the creation of a strong practitioner-client rapport, and in the case of group sessions, client-client rapport, both of which prove an effective tool for intervention adherence [29]. Contrastingly, many comments regarding session delivery and content in the current study were positive (n=67), with clients praising the IHLS practitioner’s delivery.
"X (a practitioner) is so very motivating. X makes us believe we are worth it with a true meaning and belief. X is able to deliver sessions in a fun and clear way; however X is also empathetic with all group members acknowledging everyone’s circumstances and life patterns." (P70)
It has been reported in previous process evaluations that adding travelling times on top of existing client workloads might thwart attendance [30]. The adoption of remote health behaviour change interventions might alleviate issues of location and transport. This was noted by clients (n=48) in the current study.
“I am able to attend online daytime sessions as I can attend from the comfort of my home or wherever I am at that time. It was the travelling to and from specific venues that has always stopped me attending the weight management sessions previously.” (P134)
The COVID-19 pandemic also provided an opportunity for the rollout of novel online behaviour change offers to maintain the delivery of lifestyle interventions remotely. Harnessing the surge in interest, enthusiasm and acceptance of digital behaviour change offers during lockdown has immediately been recognised as an opportunity for service providers [31]. Incorporating both quantitative (e.g. frequency counts of number of session items delivered) and qualitative (e.g. interviews, focus groups and surveys) measures of online session implementation fidelity can allow future researchers to accurately measure delivery and session impact and consequently, whether the intervention is perceived to be efficacious to behaviour change from both practitioner and client viewpoints.
4.2.2 Support
Client support, as well as key information outlining how to access online WM, smoking cessation and PA sessions were maintained during the COVID-19 lockdown through the adoption of telephone calls, social media messages and advertisements, website posts, and resource downloads sent via email. This effort was positively (n=83) received by clients.
"They have done the best they could in a challenging situation, I think. Information and support through telephone calls, emails, the website, as well as social media platforms is available for those who want it. Can't complain really." (P37)
However, learning from decades of prior research and experience, a single form of service delivery is never likely to meet all individual’s needs and negative client comments (n=18) were also noted.
“Telephone calls of very limited benefit. I am not into social media as a means of effective communication and have no interest in virtual (e.g. Zoom) meetings for this kind of activity.” (P116)
“The occasional Facebook message didn't cut much ice with me.” (P93)
Clients attending the current IHLS have an average age of 57 years, important to note given that previous research shows methods to engage onto essential online health, finance, education, and other social services are often designed with the younger population in mind [32]. Research shows that older participants (≥55 years of age) demonstrate considerable interest in learning how to use the internet for accessing particular services, however, service providers’ ambitions to engage with older adults online appear more limited as a result of entrenched stereotypes of older non-users, a lack of internal digital skills, as well as organisational and funding constraints [32]. The current study's findings emphasise the importance of balancing the views of older adults and service providers in the design of online engagement strategies. These insights are critical for improving online service delivery affected by an increasing withdrawal of traditional interpersonal services. Further research is warranted in exploring the best methods to deliver client training and support for digital health behaviour change offers to enable as many individuals as possible access.
4.2.3 Perceived competence
The COVID-19 pandemic exacerbates the importance of a hidden form of social inequality, digital inequalities [33]. Indeed, differences exist between individuals and social groups in terms of access to technologies but also in terms of their competence to obtain benefit from their use of technology. Clients noted a lack of competence to access online IHLS sessions to be a key barrier (n=29) to attendance.
“Found it difficult to join virtual sessions as I have a very old computer and it is too slow to load the online sessions. I tried once and didn’t try again.” (P61)
"I did not want to join the virtual sessions, not my thing." (P92)
However, there is clearly merit in supporting increased access to online health behaviour change offers [34], and this was reflected in the many positive client comments (n=57) regarding access to the online sessions.
“I prefer the online sessions to be honest as I can attend from the comfort of my own home or wherever I happen to be at that moment in time. The links to the sessions (adult weight management) meant they were easy to join even for me, and I am useless on the computer!” (P107)
As the use of technology increases substantially during the COVID-19 crisis, so do the impacts of digital inequalities. Given the increasing dependency on technology in all spheres of life, digital inequalities put the most digitally disadvantaged more at risk of adopting adverse behaviour change habits during prolonged isolation [33]. The current study is a first step in considering and documenting individuals’ experiences of accessing an online IHLS during the COVID-19 lockdown. Further mixed methods studies combining large-scale quantitative analyses with rich qualitative observations across differing cultures, populations and contexts are now warranted.
4.2.4 Health benefits
Digital interventions have great potential to improve population health and the efficiency and reach of health care delivery [35]. Mobile apps, text messages, wearable and ambient sensors, social media, and interactive websites can improve health by supporting behaviours involved in disease prevention, self-management of long-term conditions, and delivery of evidence-based health care practice [35]. Positive comments (n=27) were noted by clients regarding both physical and psychological benefits of attendance at online WM and smoking cessation sessions.
"Each session (smoking cessation) still reinforces positive behaviours and really, the online stuff is no different to the face-to-face sessions I attended. I am enjoying it and each session provides me with the confidence boost again to not smoke.” (P97)
"Marvelous service". Not dieting but healthy eating for life. I can move better because of my weight loss but perhaps most importantly, I feel better in myself!’ (P64)
Technology can provide a detailed, unobtrusive assessment of behaviour and its context. A challenge for future research assessing the benefits of online health behaviour change interventions is to find the most valid and efficient combinations of methods of measuring physical and psychosocial health benefits. Consequently, complementary qualitative evaluations are crucial to fully understand and interpret user experiences, as well as developing and evaluating user engagement and overall effectiveness.
A strength of the evaluation was the comprehensive assessment of intervention fidelity of an ongoing IHLS during the COVID-19 lockdown using multiple sources of data theoretically underpinned by the Evidence Integration Triangle [11]. The triangulation of data is a further strength which enhanced understanding of intervention implementation and subsequently, overall intervention fidelity. Finally, to ensure completeness, the manuscript was prepared in line with the 21-point checklist outlined in the Standards for Reporting Qualitative Research (SRQR) [36]. Study limitations are also noted. A small pragmatic sub-sample of clients from each of the offered services were recruited via convenience sampling methods and hence results cannot be considered generalisable. The subjective nature of the data is also a limitation, as is the presence of self-selection bias which resulted from the pragmatic sampling methods adopted.
5. Conclusions
This process evaluation represents one of the first efforts to document how an ongoing IHLS was adapted as a result of the COVID-19 pandemic. Lessons learned show that incorporating a bottom-up approach to information dissemination and incorporating rapid feedback from commissioners through to ground level staff is necessary for ensuring that programme targets are fully understood and agreed upon. It should be noted that the IHLS is commissioned by the public health team of the council, with the scale and seriousness of the COVID-19 pandemic the demands on public health teams were especially high, which may have been a factor in their responsiveness to this service. This study will support the critical reflection by key stakeholders, the IHLS team and IHLS clients on the positive and negative aspects of these adaptations. It will also provide transferable information to develop strategies to effectively deliver online health behaviour change interventions in the presence of extraordinary circumstances (e.g. future lockdown situations). In the context of health behaviour change interventions, the ubiquity of digital technologies and their adoption into day-to-day life translates into greater potential reach than traditional interventions, and consequently greater potential for positive public health impact. However, the potential public health impact of these digital health behaviour change interventions can only be realised to the extent of their availability, accessibility and efficacy.
Competing Interests
The authors declare that they have no competing interests.
Author Contributions
GS conceived the study. GS contributed to the study design and methodology. GS was responsible for the oversight of the study. GS contributed to the recruitment of participants. GS was responsible for data analysis. All authors contributed to data interpretation, and the writing of the manuscript. All authors contributed to critical revision of the manuscript for important intellectual content and gave final approval.
Acknowledgments
The authors would like to thank all participants for their involvement, as well as OneLife Suffolk and Suffolk County Council for their support during data collection and manuscript write-up.
Availability of Data and Materials
Participants did not provide informed consent and assent for non-anonymised transcript and survey data to be shared beyond the research team, therefore data is not available for open access.
Consent for Publication
In accordance with the ethical approvals mentioned all participants consented for their data to be included in the published manuscript.
Ethical Approval
Institutional ethical approval was received by Leeds Beckett University’s Research Ethics Sub Committee (approval number 68268).