Literature about peer review in blended and online environments has until recently been sparse (Swinglehurst, Russell & Greenhalgh 2006). However, for some time there has been useful work associated with the evaluation and review of: high quality learning objects (Taylor & Richardson 2001); learning designs (Wills et al. 2009); e-learning materials and resources (Ruiz, Candler & Teasdale 2007); and online courses and materials (Wood & George 2003). Some of these are outlined below. Recent literature exploring peer review in blended learning contexts is then considered.

Learning objects

There is an established tradition of peer review of learning designs or learning objects. Taylor & Richardson’s (2001) authoritative study focuses on ICT-based teaching and learning resources and suggests three forms of evidence for peer review: documentation explaining design considerations, a description of the resource and reflection on this.

Learning designs

Documenting the pedagogy, reuse and sharing of ‘learning designs’ also provides a useful space for peer review. Hung & Chen (2001), in their examination of e-learning design, emphasise the importance of connectivity and dialogue as well as being able to manage content and participant involvement. The four dimensions they consider essential for this — situatedness, commonality, interdependency and infrastructure — are also drawn on by Boud & Prosser (2002). Their framework for reviewing learning activities focuses on their impact on learning, and informed the development of formative and summative review guidelines for learning designs. They highlight four key influences on high quality learning in this context: learner engagement, acknowledgement of contexts, challenging learners and providing practice. Littlejohn and Pelger (2007) usefully differentiate between the ‘media blend’ and the ‘activity blend’, and Sharpe (2006) notes that blended learning designs can change the roles of the different participants. More recently, the ALTC-funded Project EnRoLE (Wills et al. 2009) has built a community of university teachers using online role play as well as a repository of sharable and reusable role play learning designs. This project incorporated an associated peer review process using the ALTC Exchange.

e-Learning materials

Evaluation or review of e-learning materials commonly involves using a checklist (Knox 1999; Oliver 2000). Checklists have a useful role in helping to identify questions and parameters, but they are limited in terms of resolving complex aspects of teaching in context in blended learning environments because they tend to narrow the focus to reviewing e-learning materials rather than the teaching and learning practices which use these materials. Conole et al. (2004) propose a model for “pedagogically driven approaches to e-learning” (p17) for guiding both design and auditing of e-learning but not the actual learning that takes place (see also Conole & Fill 2005). Evaluation issues in an online environment highlight that ‘subtle changes’ are inherent in the nature of these environments (Oliver 2000).

Widely cited in the medical peer review literature, Ruiz et al. (2007) also focus on materials. They clarify what they see as the significant differences between e-learning and traditional (print) materials in terms of: pedagogy (going beyond what could be achieved in traditional formats and exploiting temporal and spatial choices); format (ie incorporating elements like hyperlinks – multimedia to effectively go beyond traditional methods); usability (HCI factors and the quality of student experiences when interacting in the learning environment); navigation (exploitation of flexibility along with a recognition of the need for clear organisation); interactivity (evaluating effective or ineffective use of this option); delivery (differentiation of various media for delivery); and currency (of materials and more broadly). Written and framed in terms of clinical medical teaching, the additional dimensions of peer review in an e-learning versus a traditional environment are discussed in the context of providing evidence of scholarship. The authors of the current report acknowledge Glassick, Huber and Maeroff’s (1997) six criteria for assessing scholarly work. Those authors conclude by suggesting the need to develop peer review training, multidisciplinary peer review, guidelines and incentives. The interest for peer review in the current project is not online course materials per se but rather the online interactions among colleagues and students.

Online courses

Denise Wood and her colleagues provide a review of instruments (checklists and generic descriptors) for online course development and review, identify their concerns with these and explain their development of a new peer review instrument (Wood & George 2003; George, Wood & Wache 2004). Their broader considerations and categories and the three case studies (Health Sciences, Pharmacy and Planning) used to trial their instrument are particularly useful. As found in face-to-face settings, trial participants:
felt that it was important to identify which aspects of the online course component needed to be peer reviewed … [and] the academics indicated that they would feel ‘safe’ asking a colleague who had already developed an online learning environment … to provide feedback on their online course. (George, Wood & Wache 2004, p297)

Wood and her colleagues consider the ways in which their instrument supports the scholarship of teaching and learning, and provide a framework based on elements of good practice. The ALTC project Peer Review of Online Learning and Teaching (Wood & Project Team 2009) refines the development of this online tool for peer review of online materials and teaching. Wood & Friedel (2009) discuss encouraging academics to take a ‘Web 2.0 approach’ to peer review by sharing in the creation of criteria and contributing their own examples. They also built a feature to create reviews automatically around themes, for example, the first year experience (Scutter & Wood 2009).