TEU response to the TEC analysis of submissions on the paper outlining options for the PBRF 2012 Quality Evaluation

12 September 2011

For further information please contact:

Sandra Grey

National president

Jo Scott

Policy analyst

Introduction

As we noted in our submission to the TEC (22^nd July 2011), the June aftershocks in the Canterbury region have put a different slant on recovery from the earthquakes and the impact this may have on the PBRF 2012 Quality Evaluation.  A greater degree of uncertainty about the extent of further seismic activity has increased the strain on the whole community and has disrupted timelines for re-building and re-establishing services.

It was in this context that the TEU proposed that the TEC revisits the issue of the PBRF 2012 Quality Evaluation and considers other options, as members at the affected TEIs feel that the original proposal (to use the existing special circumstances category) is insufficient to address the ongoing impact of the earthquakes on institutions and staff.

Finding an acceptable solution

Since our meeting with the TEC on 18^th August 2011, we have held further meetings with TEU members at Canterbury TEIs.  At these meetings, members reiterated the importance of the TEC settling on a solution that both minimises the impact on individual staff and affected institutions and that is regarded by the rest of the sector as fair and equitable.

The key point that members felt needed to be understood by the TEC and TEIs is that if at all possible individuals should have the chance to choose from a range of acceptable solutions that best addresses their circumstances.  Members also emphasised the need for a decision to be made within the next month, so that if the quality evaluation exercise proceeds, staff (both academic and general) and institutions can get on with the necessary preparations.

During these meetings TEU members also indicated strong support for considering a partial round where, as in 2006, individual institutions make a decision as to whether some or all staff submit their portfolios.  At the affected institutions, individuals should be free to decide whether they want to submit or not.  Allowing academics who completed their portfolios in 2003 or 2006 to carry over their ‘original’ score in this round will not significantly affect the aggregate scores of the affected TEIs, therefore preserving the mechanism of the quality evaluation in deciding funding provision.

The TEU is supportive of considering this option.  Although we have previously only mentioned it as a possibility in our original options paper to the TEC,  we now feel that the situation has significantly changed over time and that this option merits reconsideration.

If a partial round is rejected as the best option, we would again emphasise that the solution needs to be as non-intrusive as possible, and would therefore favour using an additional category, such as the model we proposed in our options paper.  It is crucial that academics who have been affected by the ongoing earthquakes are assured that the magnitude of the events for their research communities is seen as distinct, hence the need for an additional earthquake category that is understood to be separate from the somewhat maligned ‘special circumstances’ category.

Moderation processes

In our meeting with the TEC, we agreed to seek additional feedback from members about the issue of moderation.  Members recognised the importance of the moderation process in identifying and then ameliorating any discrepancies in individual and institutional scores.

The process should follow the type of models used in examination moderation, where the panel makes a first assessment, and then an external audit panel moderates their assessment.

Paramount for members was an assurance that the moderation process be open and transparent, and that moderators with specific disaster/trauma expertise be brought in to undertake any external moderation function.

Members were firmly of the view that individual panels or panel members should not be left to make a judgement call on a portfolio that has been flagged as affected by the earthquakes.

The moderation – both for individual evidence portfolios and institutions – will be of a comparative nature, for example: comparing individuals’ scores in 2012 with their 2006 scores; comparing individual TEIs and the overall aggregate results of other institutions both in 2006 and 2012.

This is perhaps more easily done when it comes to the institutions, as there will be clear evidence of the impact of the earthquakes if Canterbury based TEIs ‘slip’ in the ‘rankings’ which result from the aggregation of individual scores.

Other issues

Early career researchers:  Panels need to think very carefully about the quality evaluation categories given to early career researchers because of the negative effects of a low score at this popint in their career.  Even though individual scores are awarded for the purposes of deciding the aggregate research income allocated to an institution, indiviudal scores are being used by institutions as career markers and have real consequences for individuals for the period at least until the next PBRF round, hence the merit of making 2012 a partial round.

Impact of PBRF scores on employment:  TEU urges the TEC to require affected institutions to provide some kind of guarantee that lower QCs will be accepted in the context of the events of the past 12 months and will not be used to assess performance as part of departmental or institutional reviews.  We recognise that if there are concerns with an individual staff member’s research performance prior to these events, then these should be dealt with as part of normal processes of performance review and development.