The Power of a New, Web-Based Collaborative Tool to Deliver Essential Information to Health Professionals in Developing Countries

Describe The Approaches Utilized To Measure / Assess This KM Initiative: 

Ongoing monitoring of user feedback and technical support questions have helped inform improvements to the eToolkit Application. For example, several users indicated that they need the ability to manually decide the ordering of resources on a page, in lieu of the automated design currently in place. K4Health implemented an interim solution and is currently updating the software to address this issue fully in the next release. During 2011, K4Health will also conduct formal usability testing of the eToolkit Application with new and existing users in both developed and developing countries. Usability testing is concerned with how well users can use the functionality in a given interactive interface. It involves asking users to perform one or more tasks and then recording their performance and answers to questions. A well-established evaluation method in the development lifecycle of computer systems and software, usability testing can be a simple and inexpensive method that yields high returns in improving the interface of a product. Employing the “think-aloud” technique during usability testing further enhances the evaluation methodology. While test participants are using the system, the evaluators ask them to “think out loud” to gain insight into the users’ thought processes. This combined technique allows evaluators to not only observe what the participants are doing with the interface, but also why they are doing it. K4Health has already conducted pilot tests with one user in South Africa and two users in the U.S. Evaluators recorded audio from the test sessions and the participants’ mouse movements using screen video capturing software (CamStudio and Jing). The pilot tests identified useful and actionable findings about usability of the eToolkit Application. In addition, the evaluators revised some of the test tasks that were confusing to the test participants and scaled down the number of tasks to ensure they could be accomplished in the time allotted for each session. K4Health plans to conduct usability testing on an annual basis. In addition, K4Health will conduct interviews or focus-group discussions with existing users to explore their perceptions of the software and understand better how they integrate it into their workflow.

What Do You Think Are The Main Unanswered Questions Or Challenges Related To This Field Of Work?: 

One of the main challenges of usability testing is promoting a culture in your organization where management and staff recognize the need for it and understand the value of it. Another challenge is getting usability testing incorporated into the entire product development lifecycle, especially in the early phases. Managers and software developers are anxious to produce deliverables as quickly as possible and may overlook the importance of the principle to test early and to test often in order to save valuable time and resources later on.

What Was The Purpose Or Motivation For Assessing This KM Initiative? : 

Usability testing will ensure that the eToolkit Application meets the needs of the intended audience and it will inform the iterative design of the software, helping K4Health to decide the future direction of the software.

What Would You Do Differently Next Time?: 

K4Health had worked on toolkits with intended users of the eToolkit Application before developing the software, which informed the software design. However, more formal formative research to understand specifically what users want and how they think and behave would have better informed the software design, saving time and resources. In addition, conducting usability testing with simple paper prototypes in the early phases of the development process would have helped avoid unnecessary software coding and recoding.

What Were The Most Important Lessons Learned About The Assessment Process?: 

Pilot usability testing has already identified useful information about usability issues with the software that cannot be gained from indirect monitoring methods, such as analysis of technical support questions. For example, the evaluators discovered several usability problems with the software’s resource record forms including difficulty understanding the meaning of certain fields and not being clear on which fields were required. Pilot testing was an important part of the evaluation process. The evaluators discovered that they had to revise certain test tasks to avoid confusing the test participants. Also, the evaluators allotted 10 to 20 minutes for the test, but it actually took about one hour to complete each test session. Many of the test tasks were actually more difficult for the test participants than the evaluators had expected, which therefore required more time to complete. Using the “think-aloud” technique takes some practice to implement. While it is not intended to be used as a method to guide the user, test participants who are unfamiliar with the technique tend to use it as a way to gain reassurance about their performance and guidance about “what to do next.” Evaluators need practice to explain clearly the intent of the usability test and think-aloud technique, and most of all, to build rapport and put the test participant at ease before starting the test.

What Advice Would You Give To Others Based On Your Experience?: 

Monitoring and evaluation specialists recommend conducting formative research about the intended audience before designing or implementing a new program. User experience professionals also recommend conducting user research before embarking on new software development. This phase tends to be neglected due to limited time or resources. However, conducting this critical phase at the beginning will only save time and resources later in the process. In addition, usability testing is such a valuable evaluation method that it should take place throughout the lifecycle of a product, particularly in the early stages before the interface is even designed.

Describe The KM Initiative: 

Limited access to information remains a barrier to evidence-based health care in developing countries. The INFO Project at Johns Hopkins and partner organizations developed online collections of essential information vetted by public health experts for use by health professionals in developing countries. These toolkits required considerable resources due to the collaborative work flow and programming for the interface. The follow-on project, Knowledge for Health (K4Health), endeavored to use information technology to improve efficiency in and simplify the work flow. An inter-disciplinary team at K4Health worked together to develop an easy-to-use, web-based, collaborative tool, called the K4Health eToolkit Application, that public health practitioners without specialized IT skills could use to create these electronic libraries or websites. Public health practitioners and information scientists identified goals, tasks, and work-flow patterns that drove the interface design. Software developers identified open-source software called Drupal that could automate and support the work flow and defined the scope of possibilities for the interface design. Through an iterative process, software developers adapted the open-source software to meet the defined goals while the public health practitioners and information scientists evaluated web-based prototypes.