Click here to download:

Project_Mwana_Zambia_Evaluation.pdf (3.73 MB)

(download)

Click here to download:

Project_Mwana_Zambia_Evaluation.pdf (3.73 MB)

Introduction

Zambia began to scale a national Early Infant Diagnosis (EID) program in 2006.  There have been significant logistical challenges in getting samples from rural facilities to the three reference labs in the country and then the EID results from the labs back to the facilities.  In 2010 Zambia started a pilot to use a RapidSMS based mobile health system to deliver the results from reference labs back to the facility using SMS. This program evaluation looks at the efficacy of the SMS system.

Material & Methods

The program evaluation was run in 21 health facilities in Zambia.  Eleven facilities were located across three rural districts in the Luapula Province and ten facilities were located across two districts in the Southern Province.  The evaluation looks at 710 EID results that were delivered from the lab back to the facilities by SMS between July 1st 2010 and February 15th, 2011.  The evaluation looked at the change in the turn-around time (TAT) between a sample being collected and a result being returned to the facility it was collected from, after the SMS system was utilized. This was looked at in two different ways. For the 10 facilities in the Southern Province a comparison was made between the TAT of 1141 samples collected during an 18 month baseline assessment and 467 samples collected during 5.5 months of the pilot. For the 11 facilities in Luapula province a comparison was made between the TAT of hardcopy versions versus SMS versions of 243 EID results delivered to the same facilities by both methods. For Luapula Province, censored data was used to take into account a large volume of EID results that never arrived by hardcopy.  Results were further broken down to compare urban versus rural facilities and facilities with good and bad network coverage.

Results

The evaluation found results in two areas; 1) improved TATs for EID results being sent from lab to facility 2) increased volume of EID results arriving by SMS as opposed to hardcopy.  In the baseline vs pilot comparison done for the Southern Province there was found to be a 57% decrease in the total TAT.  In the hardcopy vs SMS analysis that was done in Luapula province there was a 28% decrease in the overall TAT but when just the 9 rural facilities were looked at there was a 46% decrease in TATs. When comparing the volume of results arriving by hardcopy versus SMS it was found that 30% more arrived by SMS.

Desmond Connolley from the frog team that traveled to Zambia in November writes about the challenges of transporting Dried Blood Spot (DBS) samples to a hub where all DBS from the area are collected before they are sent on to a national laboratory for testing.  This is a cross post from frog's on the road blog.

An infant born to an HIV positive mother is at risk for transfer of the virus, which can be devastating to his or her tiny system. An impressive effort is underway in Zambia to reduce this type of transfer as well as to get HIV positive babies onto life-saving treatment as soon as possible.  At six weeks of age, an infant can be tested using a Dry Blood Sample (DBS) test, and the faster a determination can be made, the faster a baby can be given treatment. The time between taking a sample and getting a result back is a race against the clock.

There are very few labs equipped to run this test and are usually located far from the rural clinics we have been working in.  The RapidSMS system developed by UNICEF is currently being used to send DBS results directly from the distant lab back to clinics via SMS message.  This is reducing the overall time between the time a sample is taken from the baby and the time a clinic receives a result.  But a key challenge remains: how can we shorten the time it takes to get a sample to the lab?

One of the challenges Zambians face is the lack of an efficient and speedy postal system in rural areas. To get around this, rural clinic workers often have to get creative about how to get babies’ DBS samples where they need to go. 

Enter Grace, a nurse in the village of Kabuta, located in the northern part of Nchelenge district. As her clinic is based in one of the more remote areas, she will sometimes travel the 50+ kilometers to the hub at St. Paul’s to deliver her DBS samples by hand. However, more often than not, she will enlist the help of anyone passing by, be it an NGO vehicle or, in the case of this particular day, our UNICEF team that came to interview her.

After giving us a thorough demonstration of the packaging and logging of four DBS samples from the past week, she asked us to take them back with us to St. Paul’s Hospital. We were more than happy to accommodate her. I remarked that by taking these samples into our custody, we were actively participating in the system we were there to interview her about. It was a profound moment to realize that the future of four infants’ lives was in our very hands.

Kate Canales from the Frog Team that traveled to Zambia in November writes about how clinic workers are participating in Project Mwana even in areas with barely any coverage.  This is a cross post from frog's on the road blog

A large part of our research in Zambia is focused around an SMS communication system for healthcare. This system relies on clinic workers and community volunteers to interface with various services on their personal phones via SMS (users are reverse billed, so they do not pay from their own pockets). When we asked Steve, a Health Officer in the rural district of Kawambwa, to show us how he uses his phone to enable his work, he told us that he gets no network coverage at the clinic itself so his phone is kept at home.

We followed Steve back to his house in the neighboring village and he showed us the two places he CAN connect: a high shelf in his bedroom where he leaves the phone in order to hear it ring, and the enormous ant hill just adjacent to his house where he runs to take a call or send an SMS. From the top of the hill, you get the best connection in town.  And from this anthill, he does the clinic’s mobile-enabled work.

(download)

Click here to download:

phase-1-user-trainings-hpGuEzkvnmcbCHEFHkGb.zip (602 KB)

As we were preparing for the trainings we made some easy-to-use user manuals that we printed on cards for the users to easily refer to. The manuals were in both English and Bemba (the most commonly used local language in Luapula) to make it easier for the users to understand. We had to make separate training materials for RemindMi and Results160 since we didn't want the CBA (community based agents) to try and register as Results160 users. One thing to note is that some facilities were not being trained to use RemindMi because most of their villages had no network coverage.
Since the pilot sites are located in 3 districts of Luapula Province, namely Mansa, Kawambwa and Nchelenge, the team had to split into 3 groups. Since we were based in Mansa, Team Mansa had the extra job of monitoring and administering the SMS server to make sure nothing went wrong during the trainings. The journey to Kawambwa and Nchelenge was awesome. Luapula has a very wonderful landscape with so many rivers and beautiful waterfalls This just made the trip more exciting. The road is along the great Luapula River that goes all the way to Lake Mweru in Nchelenge District where one the teams was heading.

Kawambwa District is on highlands and this is where Zambia's tea plantations and The Ntumbachushi Falls are located. Three of our pilots sites are located here. Even though GSM signal is bad in all three villages, two of them have very bad network signal. Since the two are located near Kawambwa town (often referred to as the Boma by the locals), we decided to pick the users from their villages and conduct the trainings from a central location that had better signal. We trained two users and an officer from the DHO (District Health Office). One of the things I learned from the strategy of making users use their own phones was that we didn't really have to teach them to use the phones because they were already familiar with them. This really helped a lot because our focus was on the use of the system and not how to use the phones. After introducing Mwana, we gave the users the manuals and asked them to follow the instructions. Two of them went through the steps with ease but we had a user who could barely use SMS technology. Helping her learn was quiet challenging and interesting. We were so happy to see her learn as her face brightened up after each step. Seeing the other users follow the user manual without any problems made us really excited because we were now sure that the training materials we made were effective. The next training was equally exciting because we were also going to do RemindMi training. We started with Results160 training which went as planned. Before starting the RemindMi training, we had no idea that we were in for a surprise. We were all surprised to receive responses from Mwana in Bemba. We didn't know that Team Mansa had enable the Bemba locale. The users (and us) were so amazed. This change was going to make it easier for CBAs to use Mwana. Apart from the erratic signal, the training went smoothly. The clinic stuff were happy to be part of the first users to use the system and said they would use it effectively. The feedback we got from the two trainings was very positive and encouraging.

Nchelenge is on the banks of Lake Mweru. The view is magnificent and the fish very tasty. Team Nchelenge had so much fun and they went through similar challenges to team Kawambwa though the network signal was generally not as bad as the Kawambwa Rural Health Centres. One of the extra challenges they had was the larger number of users they had to train because the clinics were busier due to the larger population. One the the clinics was so busy that the DHO had to take over some of the clinic duties so that the clinic staff could be trained. This was a very encouraging gesture that left us all touched because his commitment was beyond expectation. Team Nchelenge didn't train everyone how to use RemindMi because some areas had very bad signal which would make the use RemindMi difficult. The signal on some phones was so weak at the clinic that we had to do the rest of the training by the roadside. The trainings were successfully and the feedback was positive.

Mansa is the provincial capital of Luapula province and is the largest town in the province. We have more pilot sites here than the other districts. The population here is also larger and the clinics and quite busy as well. Of all the teams, Team Mansa had the largest job of training most users and also doing system administration. Even though Mansa is the provincial capital, the network signal in the areas surrounding some of the clinics is very bad. Inspite this, they successfully carried out the training and got positive feedback from the users.

Team Mwana successfully conducted the trainings and learned a lot from them. This is evident by the success of the launch and the usage log of the activities going on in the system. One other proof of this is that we have only received less than 10 help requests from the users ever since we launched it on June 14 2010.

I am really looking forward to more of this when Project Mwana Phase2 begins.

-- Nchimunya Hamakando

This narrated slideshow describes the process that the project mwana team went through in phase 1 of the deployment.  It also describes what the team built and how it works.

I got the correspondence below from one of the developers on Project Mwana who is currently up in Mansa.  The text in red is what Zain sent to our reverse billing toll free sim card which is our GSM modem attached to the server that is running RapidSMS. It is basically telling us to pay what we owe for all the text messages that have been sent.  RapidSMS of course did not recognize this syntax so sent a very 'helpful' message back to Zain - the text in blue.

This interchange reminds me a lot of a situation we had in both Ethiopia and Malawi where RapidSMS sent back a confirmation and a 'thank you' to the reporters who were sending in data to the system.  Many of the reporters - being polite human beings - would reply with a 'you're welcome' or 'no problem, happy to help', upon which RapidSMS would send back a similar message to the one above in blue, telling the reporter that it couldn't understand them.

For some reason this makes me giggle.  We are bending the rules of what SMS was originally designed for - quick 1 to 1 communication - into a 'system' that improves and even saves people's (in our case mothers and children) lives, gets them information faster, and makes their work easier.  Yet occasionally, when at least 1 or those 1s is a machine, then nuance and human context is lost.  This reminds me of the phrase 'does not compute'.  About the phrase, Wikipedia says: 

Does not compute, and variations on it, is a phrase often spoken by computers, robots and other artificial intelligences in science fiction works of the 1960s to 1980s. The phrase indicated cognitive dissonance on the part of the device, conventionally leading to its self-destruction.

According to The Random House Historical Dictionary of American Slang, the phrase was first used as a catch phrase by the television show My Living Doll in 1964. It was then popularised in Lost In Space (1965), along with "Affirmative!", "Warning! Warning!" and "Danger, Will Robinson!"

The phrase "does not compute" and robots who self-destruct when considering emotions or paradoxes is frequently satirized in popular culture.

Let's hope not.  I wonder what the machine on Zain's end will write back to RapidSMS...

- Erica Kochi

After all the efforts that have been put into the coding and a series of pilot site visits and demos in Mansa, one of the provincial towns of Zambia, consultations plus demos with key players which include the office of the Provincial Medical Office, District Health Office, lead stakeholder Zambia Prevention Care and Treatment Partnership II (ZPCTII) a U.S. funded Agency for International Development, the team is now ready to implement the system in the selected pilot sites of Mansa, Kawambwa and Nchelenge.

The implementation process will include among other things the following;- introductions between team members and clinic staff, discussions on DBS and how it is handled in that clinic, an overview of the Results160 and RemindMi program and how it is going to strengthen health services for mothers and infants. Emphasis will be placed on the importance of active participation of all players in the process.

After all that is said and done, the demos for Results160 and RemindMi will be done to ensure that the staff will be able to implement and train the CHWs for their respective clinics. This is one very important activity that will shape and determine the direction the program is going to take.

This program has been designed specifically to benefit the Zambian people in a number of ways, some of which are as follows:

1. It will improve the health of the communities 
2. It will improve immunizations coverage as mothers will be timely reminded to bring children to the facilities 
3. It will reduce the turn-around time for the DBS sent for analysis 
4. It will help the Government to collect data on all births occurring in the communities 
5. Build capacity and teach new skills in life 
6. Reduce on the number of travels by the CHWs to the facilities 
7. Reduce paper-work 

I strongly believe that with proper implementation and the involvement of appropriate staff, the program will translate to improved health and therefore contribute to the attainment of the 4th MDG.

DBS Results160 + RemindMI = Better health for our community

--James Mkandawire , Zambian MOH

We've just been in Ndola for the past week building a bridge from the lab computer system to our RapidSMS server back in Lusaka. Our goal is to get fresh DBS results as they're entered onto the (non-internet connected) lab computer, and funnel them to RapidSMS, where we can then distribute them to clinics. It's a critical piece of the puzzle; without results, we have nothing to send. And yet, for all its importance, we have no on-site staff to leave in Ndola, the machine cannot be remotely accessed, and even if we had staff to spare, Ndola is a half-day's ride from Lusaka and a full-day from Mansa. Whatever software we leave there will largely have to administer and maintain itself.

To that end, I cannot overstate the importance of such software giving you detailed feedback about itself. Having the software maintain thorough logs about its health and operation and having it make a heroic effort to get those logs to you is the key to remotely diagnosing and fixing any problem that may arise.

We deployed our solution on the lab computer Friday afternoon -- about forty minutes left in our departure window to get back to Lusaka that night -- and actually managed to squeeze in a live test or two. Upon seeing real results trickle into the Lusaka server, we combination cheered/sighed in relief, packed up, and left. The true test, however, would come on Monday. We would see if the system still worked without the magical effect of us standing right there in front of it.

The first sync is scheduled daily at 9:30am. I sat nervous at my laptop, trying to limit myself in how often I clicked 'Refresh'. 9:30 came and went, then 9:35. The lab connection was slow; these things could take time... 9:40... 9:45... Perhaps the computer hadn't been turned on... By 10:00, hope had faded, and I drafted an email to the lab manager asking him to assist us in troubleshooting. As I was about to click 'Send', a new submission from Ndola popped into the queue.

We had successfully sent results end-to-end! The team was thrilled. But something was odd. On closer inspection, there were no results. Instead, just this:

...
Apr-26 09:30:01: records deleted from ZPCT database! (12-00000)
Apr-26 09:30:01: querying records of interest from ZPCT: 280 total; 4 new; 257 resolved; 1 in limbo; 18 untested
Apr-26 09:30:01: error accessing ZPCT database (read-only); staging database not touched
Traceback (most recent call last):
  File "C:\unicef-mwana\script\extract.py", line 323, in pull_records
    records = query_prod_records()
  File "C:\unicef-mwana\script\extract.py", line 315, in query_prod_records
    records.append((source, read_sample_record(id, conn)))
  File "C:\unicef-mwana\script\extract.py", line 181, in read_sample_record
    sample_row = query_sample(sample_id, conn)
  File "C:\unicef-mwana\script\extract.py", line 175, in query_sample
    raise ValueError('sample [%s] not found' % sample_id)
ValueError: sample [12-00000] not found
Apr-26 09:30:01: db sync attempt 1 of 6 failed; trying again in 2 minutes
...
Apr-26 10:06:04: all db sync attempts failed

It turns out we had a bug. During our testing on Friday, we had created a new sample result for testing, then deleted it. Our software duly noted the deleted record, but then still tried to check it for updates. As it no longer existed in the main database, the software crashed. Despite this, it still tried again another five times, with increasingly lengthy waiting periods between each attempt. This was by design -- database queries can fail for any number of ephemeral reasons, so this was yet another layer of built-in resiliency. Finally, almost forty minutes later, it phoned home about its failure.

Without the reporting infrastructure we had put in place, debugging our lack of results from the lab would have been a nightmare. With many, many hours of debugging, severely depleted goodwill of the lab manager, and perhaps an emergency visit to Ndola, we would have figured it out. It also may have even been days before we realized there was actually a problem! But instead, we knew the precise cause of the failure immediately, as if we had been sitting right at the computer at the exact moment the failure happened. A fix was ready in two hours, and would be deployed hopefully within the day.

The takeaway is that if you're deploying a critical piece of software in an environment you don't have control over and don't have ongoing access to, build in layer upon layer of redundancy, resiliency, and fail-safes. In particular, we employed the following tactics:

• software keeps detailed logs about its normal operation
• software traps any unexpected errors -- no matter where they occur -- and adds debugging information to the logs
• logs are sent along with our actual data of interest
• software phones home at regular intervals even if there is no new data to send
• software requires positive acknowledgment from the other end that the data it sends have been received safely; without this acknowledgment, it will re-send the data on each scheduled run
• the software attempts unreliable operations (querying a database, sending data over the internet) several times before giving up

--Drew Roos