In an era where schools’ investment in technology seems to be accelerating, in both scale and cost, an accompanying and understandable questioning of the evidence base for these decisions is often heard. Lots has been written about this subject (some of it by me) and debates have raged impotently across the Twitterverse. One thing we can all agree on is that a better understanding of technology’s impact is desirable, and that research is the way to gain this.
With many of the schools within the Group for which I work in the process of embarking on their own mobile learning tablet projects, I have a definite opportunity to design a study which might help other educators to see the impact of this technology on learning and on learners.
Sounds great, huh? Of course, were it that simple, I wouldn’t be writing this.
The litmus test for any research finding is replicability – can you take the same approach elsewhere and get the same results? This shows that the treatment being researched is generalisable to other contexts and in short, that it has a genuine effect that can’t be explained by other factors. It is this fundamental hurdle which educational research has always had trouble scaling and many think that the only way to remove all of the confounding variables is to use Randomised Control Trials (RCTs). I’ve got a lot of time for this viewpoint, especially as espoused by Ben Goldacre for the DfE a while back, and the scientific purity of it all is frankly quite delightful.
The problem with RCTs is that they’re very hard to do well, for lots of reasons. I’ll leave the debate over the ethics of control groups aside and focus purely on the reality of most school-based research – and that’s the fact that it tends to be opportunistic. Most often the intervention (in this case, the use of tablets in classrooms) is designed primarily with logistical and educational concerns in mind and any research study has to fit around this.
Randomisation of treatment/ control groups, so crucial to removing the obscuring veil of other factors, is almost never the starting point. For practical reasons, groups are usually formed using existing classes of pupils, which are very un-random in their nature. It’s hard to imagine a school putting randomisation of groups in a study ahead of all the other things they have to do in order to function properly. As for the notion of proper control groups (let alone blind or double blind ones), I haven’t ever seen this in an educational study.
So, if an RCT is impractical in the context of the schools I’m working with and their timeframes, can we come up with an alternative study design that is still going to produce useful and valid data? Hopefully so, perhaps with enough input from those reading this. I’ve sought advice from colleagues and the NFER, and this is my proposed design at this stage:
1. The theory of change is that this technology, deployed in this manner, will accelerate progress and improve attitudes to learning.
2. The research questions are:
- Does the provision of tablet computers in a 1-to-1 model have an impact on academic progress?
- Does the provision of tablet computers in a 1-to-1 model have an impact on pupils’ views on their learning?
3. The study will use six different schools with overlapping contexts, to go some way towards removing school quality as a variable. Schools 1 and 2 have many shared characteristics; they are academically-successful, single-sex schools without a history of 1-to-1. Both will be deploying tablets to the entirety of the certain year groups by September 2014. Schools 2 and 3 are existing co-educational 1-to-1 schools in the maintained sector which are characterised by strong leadership and year-on-year improvements to academic outcomes.
4. One measure of impact will be progress made between data capture points. For Schools 1 & 2, these points will be Y7 baseline assessments and the Y7 assessments carried out next summer. Schools 3 & 4 will take the KS2 SATs as the baseline, which benefits from being externally assessed. In both cases, the cohorts’ rates of progress from previous years will be taken into account.
This is problematic, as this data will be subject to all sorts of external variables and will come from multiple feeder schools. It could be argued, the more impactful intervention these students have been influenced by is their change of school. To counter this, we will also look at the progress data of previous equivalent cohorts from all 4 schools (e.g. the last 3 years’ Y7s) to try and isolate an effect of the treatment.
5. A pupil survey which is carried out before and then repeated after the treatment. The content of the survey will not be related to the treatment and will focus rather on respondents’ views of school, of their learning and of the teaching they receive. The reason for separating the survey from the treatment is to control for acquiescence bias which may be related to individuals’ positive perceptions of technology. The survey will instead seek to reveal if there has been a quantifiable change in the populations’ views of these aspects of their school life. In order to provide a measure of control for the effects of history (experiencing the world for another year), Schools 5 & 6 will provide pre- and post- survey respondents from equivalent cohorts in similar contexts, the difference being that there will be no intervention in these schools.
6. Another obvious variable to control for is the quality of the implementation of the 1-to-1 projects. This is mitigated somewhat by the nature of our work as a Group, because of the centralised support and processes that are in place. All 4 treatment schools are also using the same product and ecosystem, albeit with differences in device size.
7. Results will be analysed by gender, year group and context (maintained/ independent).
Can you suggest ways in which this design could be changed to improve the quality of the data uncovered?