Q&A – Reflections on educational technology programs and their evaluation

Q&A – Reflections on educational technology programs and their evaluation

A recently published report by HEART suggests that a lack of evidence persists regarding how teaching practice changes as a result of edtech programmes.

In a short Q & A session, one of the report’s contributors and PI of our project, Niall Winters, who recently moved from the London Knowledge Lab to take up a position as an Associate Professor at the University of Oxford, Department of Education, reflects on challenges that need to be considered when designing and implementing edtech programmes.

In your discussion, you highlight a number of challenges that persist with regard to the design and implementation of edtech programmes. What would you consider to be the main challenges?

The first challenge is taking a participatory approach with local teachers or trainers to co-design the interventions in a bottom-up manner that addresses their needs. Teachers should have a greater role in the design, development, implementation and evaluation of edtech interventions. I explored this question, particularly the focus on mobile, in more detail in this Guardian blog post.

The second challenge is one of equality, and designing with those at the margins. I think there is a real danger that educational technology focuses on those with existing privilege, for example, children who already have lots of access to resources. This risks – even somewhat unintentionally – increasing inequality rather than reducing it. I’m sorry to say that we are not doing enough for those who are deemed “hard-to-reach” either geographically, socio-economically or physically (for example because of a disability, including learning disabilities). So there needs to be a sustained and targeted focus on these groups.

Another challenge is asking the hard question: what justifies the use of technology? If the intervention that you are designing can be done equally well with pen and paper, then what is the justification for technology? To address this question, one needs to have a strong understanding, not only of the functionalities of technology, but also of how one can design an effective and appropriate pedagogy that leverages these technologies in new and productive ways. Teacher training is an obvious one, and particularly developing teacher training linked to design in technology enhanced learning.

How would you describe the current status quo of evaluation in edtech programmes?

The focus of evaluation – although things are changing – remains on learning outcomes. The question that keeps being asked is how technology can achieve better learning outcomes. However, I think we need to take a step back and look at the context in which educational technology is being implemented. So how I like to think about edtech interventions is as ‘complex interventions’: as an intervention that has several interactive components and is very sensitive to features of the local context. Given that this is the case, we are often finding that some projects or some evaluations jump to randomized control trials too soon, without doing a program-in-context evaluation first. These program-in-context evaluations should be the focus of evaluations for the next few years.

What are the implications of the findings of this report?

I think this report helps DFID in-country advisors understand the complexities of educational technology research and the different factors that need to be taken into account when deciding on the level of funding to provide to this area (in collaboration with ministries of education and health). As the report makes clear, the key point is not really to focus on information dissemination or technology delivery, but instead focusing on teacher training to develop more innovative and complex uses of technology in support of improving teaching practice and student learning.

And finally, and I am aware that this may come across as a bit a controversial implication to some: Appropriate technologies are not often those that are currently available, but instead should be thought of as those that help to maintain the highest level of support. In the case of mobile learning, for example, this may be smartphones, not low-end phones that rely on weak pedagogies. Viewing edtech in this way has far reaching implications, which often require large long-term investments.

Comments are closed.