2.1 Study area

Kambia District is located in the Northern Province of Sierra Leone. Its capital and largest city is the town of Kambia (Figure 1). As of the 2015 census, the District had a population of 343,686 [8]. Kambia District borders the Republic of Guinea to the north, Port Loko District to the south and Bombali District to the east.

Figure 1: Kambia District.

Kambia district is overwhelmingly Muslim (over 70%) and ethnically diverse. The Susu are the predominant ethnic group followed by the Temne, Limba, Fula and Mandingo.

All the Kambia chiefdoms have medical centre or posts with the only referral hospital located in Kambia town, the district headquarters. There are few transport services to facilitate access to the referral hospital. Consequently, Kambia has the highest death rate compared to other districts in the country, especially infant and maternal mortality [9].

2.2 Intervention

For MOTS modules, each module is made up of about 5-6 units with a pre-and post-test, and each unit is a maximum of 5 minutes of listening time. The two module descriptions are described below:

1. Module 1: Vaccination - The overarching goal of Module 1 is to build CHWs’ capacity to support national vaccination programs by promoting vaccination. Learning objectives are to ensure CHWs understand the basics of vaccination: why vaccinating, who, when and where, and are able to share this information with the community.
2. Module 2: Outbreak Response - The overarching goal of Module 2 is to build CHWs’ capacity to contribute to the country disease surveillance system and to respond to outbreaks of emerging infectious diseases. Learning objectives include: understanding the importance of community-based surveillance, identifying potential outbreaks, knowing how to report, knowing how to identify notifiable conditions (such as newborn death, maternal death, neonatal tetanus, clustered deaths, polio, cholera, guinea worm, Ebola, yellow fever).

Various levels of the MoHS, such as Peripheral Health Units (PHU) In-charges can monitor module and quiz completion, quiz results, and listening patterns to assess the performance of the modules and the CHWs. Figure 2 shows the architectural illustration of MOTS concept and workflow figure.

Figure 2: MOTS Concept and Workflow.

2.3 Study design

To assess the effectiveness of the MOTS education, a quasiexperimental design with two treatments and one comparison group was used. Both treatment groups received MOTS training as well as the MOTS quizzes associated with each module. Treatment Group 1 received the Vaccination module first followed by the Outbreak Response module. Treatment Group 2 received the Outbreak Response module followed by the Vaccination module. The Comparison Group received no MOTS training nor the MOTS quizzes.

2.4 The study hypothesis

The quasi-experimental study aimed to answer two key questions:

1. To what degree does MOTS improve knowledge and behavior of CHWs related to vaccinations and outbreak response (compared to a comparison group)? The hypothesis: MOTS improves knowledge/behaviors of CHWS compared to CHWs who receive no mobile training.
2. To what degree does the order of the refresher modules (Vaccination followed by the Outbreak Response module, or vice-versa) matter on knowledge and behavior change? The hypothesis: Participation in MOTS sessions back-to-back can lead to user fatigue, resulting in reduced effectiveness of the second refresher module in which a CHW participates.

2.5 Study population

The primary study participants included randomly-selected CHWs from targeted chiefdoms in Kambia District. CHWs from the comparison group occupied the chiefdoms of Mambolo and Samu while those from treatment groups were from Bramia, Magbema, Tonko Limba, Gbinle Dixing and Masungbala chiefdoms.

Prior to the project, CHWs were screened for their participation in MOTS in a functionality assessment where CHW ownership of functional mobile phones and connectivity was assessed. From the list of qualifying CHWs, CHWs were chosen to participate. The location of the CHWs determined which treatment group a CHW was placed in. Those in hard to reach areas were purposively placed in the comparison areas. This minimised spill-over effect and a feeling of being left-out by other CHWs. Table 1 outlines the number of CHWs per Chiefdom and their allocation for the treatment and comparison groups. A total of 378 treatment and comparison CHWs were visited and consented to voluntarily enroll into the study at baseline, while a total of 375 treatment and comparison CHWs were found for the endline interviews follow-up.

Table 1: Sample communities and sample sizes.

Figure 3 outlines the total sample of CHWs that participated in the study. Please note that the graphic captures CHWs who participated in the quasi-experimental design as well as those who completed the pre- and post-tests associated with the MOTS platform.

Figure 3: MOTS Sample Framework.

2.6 Study variables

The survey included variables related to the socio-demographic data of the CHW, such as their gender and education level; CHW duties such as the number of households served; participation in the MOTS education, including knowledge questions associated with both the Vaccination and Outbreak Response modules, CHW attitudes such as their confidence to serve their communities in a time of disease outbreaks and the degree to which they shared messages with their community related to the two modules. The MOTS quiz for the pre-and post-tests for both modules comprised 20 questions. Only 12 to 14 of these 20 questions for the Vaccination and Outbreak Response modules, respectively, were included in the outcomes survey data collection. Questions in the outcomes survey entail the same content as those in MOTS though a few tweaks were made to fit the structure of the survey tools. Table 2 outlines the data sources used below as they relate to the quasi-experimental assessment of the MOTS system data.

Table 2: Study variables by method.

2.7 Data management and analysis

Six enumerators were trained to collect the quantitative data from the selected CHWs. A digital data gathering platform, Survey CTO, was used for data collection and this minimized errors that are often experienced with paper-based data collection. This was for the preand post-assessment study for all the CHWs enrolled.

Data was processed in Excel and exported to Statistical Package for Social Scientists (SPSS) for analysis where frequency tables and charts were produced and incorporated into the report.

Regression analysis on the survey data using SPSS was used primarily to help identify factors that were influencing knowledge change among the CHWs. Analysis included a binary logistical regression under the assumptions of the Ordinary Least Square (OLS) method. The significance of the relationships was tested at 95 percent level of significance against the p-value of 0.05, which were then noted with the subsequent beta value and the Nagelkerke R2 was used to identify the strength of the fit of the regression. A knowledge index for the regression analyses was created to assess factors that influenced knowledge improvement. The Knowledge Index was created by coding all knowledge questions to 1 or 0 (1=correct answer, 0=incorrect answer) and then summing the scores to all questions (maximum score was 11). The highest knowledge quartile was determined as those above eight correct answers. Those 8 and below were considered below the threshold. CHWs were then coded as 1 (Highly knowledgeable) or 0 (Not highly knowledgeable).

2.8 Ethics approval

The government of Sierra Leone, Office of the Sierra Leone Ethics and Scientific Review Committee, Ministry of Health and Sanitation Directorate of Policy, Planning & Information (DPPI), Youyi Building, Fifth Floor, East Wing, Freetown approved the study and provided a Protocol number 12/03/2017.

3.1 Socio-demographic characteristics

Of the 375 CHWs included in the study, the majority (85.2%) of them from both the comparison and treatment groups were male with a high attendance of secondary education, earned most of their income from crop farming, and were living below the $2.50 international poverty line (as measured by the Poverty Probability Index). Majority (69.8%) of the CHWs in the coverage area were covering less than 90 households. Details of the break down between Treatment 1 (TI), Treatment 2 (T2) and the Comparison (C) group can be seen in Table 3.

Table 3: Social demographics.

3.2 MOTS progress and completion of training

Overall, at the end of training, the MOTS system indicated that 62 percent of Treatment 1 had completed training for both modules while 68 percent of Treatment 2 had completed training for both modules by the close date. Progress-wise, 93 percent of Treatment 1 and 84 percent of Treatment 2 completed or were in progress in the Vaccination module while 81 percent of Treatment 1 and 93 percent of Treatment 2 had completed or were in progress for the Outbreak Response module. The initial module pushed to the CHWs had the highest completion rates (Figure 4).

Figure 4: MOTS refresher training module progress per treatment group.

3.3 MOTS quiz results

Using the MOTS quiz data for the CHWs that also participated in the outcomes survey, for Treatment 1, out of the total number of quiz questions for both modules, 55 percent of the answers were correct at the pre-test, while at post-test, 68 percent of the answers were correct. This represented a 13 percentage-point improvement in knowledge for Treatment 1. For Treatment 2, they experienced a 7 percentagepoint improvement between the pre- and post-test knowledge improving from 58 to 65 percent between the pre- and post-test. Overall, the Vaccination module performed better than the Outbreak Response module. It was expected that the CHWs would achieve an average score of 80 percent achievement in knowledge by the posttest; however, neither treatment group exceeded 70 percent (Figure 5).

Figure 5: MOTS Quiz Knowledge Questions: Percent Scored as Correct.

3.4 MOTS outcome survey data

Utilizing the outcomes survey data, which was collected in-person through enumerators, (Figure 6), knowledge change was compared between the treatment and comparison groups. While the MOTS quiz questions were disaggregated by Treatment 1 and Treatment 2, the combined Treatment 1 and 2 averages are compared to the Comparison given very little difference between Treatment 1 and 2 averages according to the survey data. With results from both the Vaccination and Outbreak Response modules combined, both the treatment group and comparison groups scored on average 85 percent correct on the knowledge questions at the baseline. At endline, the treatment group scored on average of 87 percent while the comparison group scored 84. Vaccination knowledge started out higher than knowledge associated with the Outbreak Response module and slightly improved for the Treatment Group while slightly decreasing for the Comparison group. Vaccination knowledge also exceeded the 80 percent target for endline knowledge. Knowledge associated with the Outbreak Response modules slightly increased for both groups, with the Treatment Group reaching the 80 percent target at endline.

Figure 6: MOTS Outcome Survey Knowledge Questions: Percent Scored as Correct.

3.5 CHW community engagement on the modules

Results at both baseline and endline suggest consistent CHW-community engagement regarding vaccinations and slightly less engagement regarding outbreak responses. Almost 100 percent of the Comparison and Treatment groups had spoken about Vaccination in the month prior to the baseline and endline surveys (data not shown). Discussions about Outbreak Responses were only slightly lower at an average of 99 percent for both groups at both baseline and endline. Both groups felt they’d spoken more about both topics in the month prior to the survey than what they would normally (Figures 7 and 8). The treatment group slightly gained in confidence for both modules, while the comparison group felt they were slightly more confident at endline regarding vaccinations but slightly less confident regarding outbreak response.

Figure 7: Community Engagements on Vaccination.

Figure 8: Community Engagement on Outbreak Response.

3.6 Attitudes towards CHW work

Overall, CHW attitudes towards their work was high at both baseline and endline for both the Comparison and Treatment groups; however, the Comparison group saw a slight decrease in their belief that they would trust disease outbreak information if received through their mobile device (Figure 9).

Figure 9: Attitudes towards CHW work.

3.7 Predictors of knowledge change

Given the limited variation between the Treatment and Comparison groups, regression analyses were conducted to further assist in understanding the variables that influenced knowledge change (Table 4). The first analysis aimed to determine whether completion of the training influenced knowledge, given this is an assumed critical precursor for “refreshing” a CHW’s existing knowledge. A knowledge index was created, consisting of 11 questions pulled from both the Vaccination and Outbreak Response modules. The description of the creation of this index was covered in the Methods section. The analysis found that participation in both modules (or lack thereof) was not associated with higher knowledge scores. However, when assessing each module separately, there was an association between completing the vaccination module as well as if they completed both modules. Similarly, for the Outbreak Response knowledge, completion of the module was associated with improved knowledge, but completion of both modules was not associated with higher knowledge scores.

Table 4: Predictors of Knowledge Change.

An additional analysis also found that being male and below the poverty line were associated with improved knowledge scores but food security was negatively associated with improved knowledge; however, whether they were in a treatment or comparison group, membership to a particular chiefdom and confidence levels were not associated with improved knowledge.

3.8 MOTS costs

As part of the MOTS implementation, a costing study was commissioned to analyse

1. the regular, ongoing cost to the MoHS of providing mobile-based refresher training to CHWs throughout Sierra Leone,
2. the expected cost to MoHS of offering MOTS refresher training compared to that of the alternatives in-person refresher training or no refresher training at all,
3. the benefits of the mobile-based refresher training for the MOH (or another implementing organization), and
4. the tradeoffs compared to in-person refresher training.

The full study and its associated methodology are described elsewhere [10]. The high level findings are shared here as a way to illustrate a possible cost and benefit assessment of MOTS for the MoHS of Sierra Leone.

In investigating the costs and benefits of MoHS implementing mobile-based refresher training, based on the MOTS pilot project, the costing study found that:

1. The cost for MoHS to provide mobile-based refresher training to all CHWs in a single district within one year would be about $25,000, compared to $52,000 for in-person refresher training-a savings of $27,000;
2. The annual cost for the MoHS to extend a mobile-based refresher training to all 16,000 CHWs nationwide is estimated to be around $190,607, which amounts to approximately $12 per year per CHW, and $1.19 per household reached;
3. The MoHS could put in place the foundation for mobile-based refresher training and CHW communication for a low, fixed annual fee (under $14,000 for the technology infrastructure and translation and recording of the IVR audio content), and then disseminate education modules selectively according to available funds to cover the variable expenses;
4. CHWs incur low opportunity cost when attending in-person training and incur low/no opportunity and airtime costs in accessing the mobile-based IVR refresher trainings, which are available according to their schedule and preferences.