Options for the 2012 PBRF Quality Evaluation to address the impact of the Canterbury earthquakes

Posted By TEU on Apr 12, 2011 |


A paper prepared for the Tertiary Education Commission

12 April 2011

The Canterbury earthquakes of September 2010 and February 2011 have had a devastating effect on participants in the 2012 PBRF Quality Evaluation process. Tertiary education institutions have focused in the first instance on restoring teaching as soon as possible, so that staff and students can return to their day-to-day routines. However, the return to normal research activity may take significantly longer.

Whilst all staff will not have suffered the same degree of trauma or disruption, we know that research work of many individuals has been very badly affected, and that the three TEIs in the region have had significant infrastructural damage that has hindered their ability to operate in their usual manner. Further, the impact goes beyond the Canterbury region, as many other individuals, collaborating with Canterbury region academics, or having personal connections to the city are also affected.

Although some work will have already been completed in readiness for the 2012 Quality Evaluation, the events of September and February have diverted energy and resources to more pressing matters. For example, administrative and library staff will be fully occupied for much of this year recovering their worksites from the devastation of the earthquakes. Rescheduling of teaching, developing on-line teaching materials, reorganising offices, increased pastoral care needs for students and so on, will all divert attention – quite rightly – to the most pressing concern, earthquake recovery.

Whilst PBRF is meant to be a system of allocating funding at an institutional level, it has become a highly-personalised process, where individual staff are under considerable pressure to produce research outcomes, and academic units are under pressure to perform or lose funding. This kind of pressure is particularly untenable for those affected by the earthquake: struggling with major changes in their domestic lives, working in difficult conditions, and suffering from the emotional impacts of the devastation of their homes or workplaces.

Because of our concerns about the impacts of the earthquakes on our members, the TEU has engaged with the Tertiary Education Commission (TEC), by letter and also face-to-face, signalling the need to consult widely about how the 2012 Quality Evaluation should take into consideration the impact of these two earthquakes.

The TEU has now conducted wide consultation amongst its members which has generated a number of options we wish to put to the TEC. Foremost in the minds of members is:

  • That there be recognition of the hard work they have done to improve or maintain their standing with regard to the limited and specific metrics of the PBRF;
  • That any changes to the Quality Evaluation process does not unduly favour affected academics; and
  • That the Quality Evaluation is seen to be fair to academics from all parts of the country, in all disciplines, and in all stages of their career development.

Key assumptions

We base this options paper on the following assumptions and/or principles which we believe need to underpin decision-making about modification to the 2012 Quality Evaluation:

  • The PBRF 2012 Quality Evaluation must adjust for the Canterbury earthquakes whose effects are significant and long-term.
  • Any adjustment to the Quality Evaluation process must take into consideration that the impact of the earthquakes may be felt outside of the Canterbury region.
  • To be acceptable, any adjustment would need to achieve a “level playing field” without increasing the reporting burdens on Canterbury region academics, especially at a time when normal reporting requirements are already difficult to meet.
  • The tertiary education community should be consulted fully but in a timely manner on all adjustments to the 2012 Quality Evaluation under consideration by the TEC.
  • The decision-making process needs to be transparent for those being assessed.
  • Changes to the 2012 Quality Evaluation must not financially disadvantage Canterbury tertiary education institutions or impact on perceptions of their current standing as high-quality research institutions.

Canterbury region TEU member accounts of life after the earthquakes

These few representative accounts listed below have helped shape the proposal that we table for consideration by TEC and the tertiary education institutions. Each illustrates a key problem we believe is important for any adjustment to the 2012 Quality Evaluation to address.

“I am unable to access any of my printed research files. My personal laptop does not have the software to continue coding etc. my research material (this is loaded on my work PC, which is in my office, which is only accessible for 10 minutes at a time under escort from USAR people). I cannot access my research books that are in my office as the library is inaccessible and there are very few specialist e-books available – hardly any on New Zealand – so anybody doing research that is NZ orientated cannot access material they may want.”

“I am working all the hours in the day doing extra work to reorganise courses and communicate with students, administrative staff, technical staff and academic colleagues, and management.
It is barely possible for me to fulfil my teaching duties. It is literally impossible for me to advance my research at all. I have no time, no access to the data, articles and other research materials in my office, no access to the library, no proper access to printing. And this is at the most critical time in the PBRF calendar – the last period when you could reasonably expect to get publications ready for submission in time to be published this year.”

“The gap in my work towards the initial phase of my PhD has been a significant set-back and emotionally I am still struggling to see the point. I hope that soon I will recover some motivation but feel pretty empty. I have conference funding approved for this July but don’t know if it is worth going with so little progress.”

“The impact of 4th September alone was not just the month we waited to get back to the laboratory, but the many months of culturing that could not be replaced. We were approximately four months back into the culturing when the quake of 22nd February again hit our city. As I write this I know that the cultures would have been destroyed. But whether we only lost a day, week or all four months I won’t know until I can access the freezer stocks to see if the power came on in time to rescue our historical library.

Thus, even though we have just under two years remaining on this contract, the first year has effectively been made null and void and these steps can’t be accelerated or conducted in parallel over the next two years because the products of these steps are necessary for later steps. The two quakes added together were probably only about a minute of time in a 6 month window, but they have erased 12 months of research.”

These stories primarily describe the impact of lost time. The TEC should be aware that lost time is not just a quantitative factor for academics. The loss of time may also result in a missed opportunity to establish priority in competition with other research groups; e.g., being the second to publish may mean publishing in a lower impact journal, getting a lower score on a grant application, or missing a grant application deadline because the work has not been published in time. The effect of the earthquakes may have meant that some events that contribute to quality evidence never happened: e.g., meetings in Christchurch were cancelled or academics from Christchurch could not attend.

While it is clear that most academics in New Zealand suffer from personal tragedies from time to time, and collectively these tragedies can have similar quantitative and qualitative effects, the scale of the earthquakes is unprecedented and has destroyed the ability of the institutions and communities of researchers to compensate because the very infrastructure of the research environment has compromised everyone’s capacity. And this is on top of the normal personal tragedies that academics of this region must also endure.

Possible options for the 2012 Quality Evaluation

Whilst there is unlikely to be a perfect solution for dealing with the impact of the earthquakes, we believe nonetheless that it is possible to palliate the effects, without disadvantaging other institutions participating in the PBRF.

The TEU has discussed seven possible options with members in the Canterbury region and with our advisory group (Appendix One). No one solution was without shortcomings and all have their supporters and detractors. PBRF as a tool overall is not without its critics in the academic community and while the Canterbury earthquake has highlighted some of the problems for members, we realise that in the immediate term we want TEC to respond to ensure members affected by the earthquake receive the care and consideration they deserve.

As no solution is without shortcomings, we have thus settled on a composite solution that offers several options to each staff member.

This solution recognises the key assumptions listed earlier, as well as additional concerns that many members expressed, including:

  • Impacts of options on early career researchers;
  • The difficulty of assessing the impact of the earthquakes on an individual researcher through a criteria-based system.

Our proposed solution, described in detail below, relies upon: 1) adding a field to signal a portfolio which has been impacted by the earthquakes; 2) shifting the time frame of assessment for some individuals; and 3) a pro-rating of some quality scores at an institutional level.

  • Adding a field “Impact of Canterbury earthquakes”.
  • Individuals submitting EPs can opt to tick the statement “I have been affected by the Canterbury earthquakes”.

This also permits researchers outside of the Canterbury region that have been affected by the earthquakes, because of collaboration with Canterbury-based researchers or because they have personal ties to the affected region, to seek consideration.

  • Ticking this option then allows the individual to opt to select either:
    • I wish to post-date the start of my assessment period from September 2004; or
    • I wish to be given a grade based on my work since 2006, but have my portfolio listed among those that have been severely affected by the earthquake.

Providing these choices allows the TEC to hear from those who know their circumstances best rather than to make “top down” assumptions about the impacts on individuals. Importantly, it allows individual researchers to take their personal circumstances into consideration e.g. a new researcher may choose to have his or her portfolio considered amongst those which are pro-rated, whereas a more established researcher may choose to extend the start date.

  • Pro-rating applied to the allocation of institutional funding.

Once the allocation of Quality Categories is established, a pro-rating formula would be applied to the distribution of funding within these categories, and final rankings would be adjusted accordingly. The pro-rating would recognise the impossibility of assessing the many portfolios whose evidence was not available for the exercise because of events outside of the control of the academics.

Our proposal requires that a clear set of instructions be developed, with input from stakeholders, which would provide clarity to panel members about implementation of these options. This may need to include instructions such as how to deal with lesser numbers of NROs. It also requires a commitment for additional training for panels on how to follow and apply these instructions, providing confidence to the academic community and the TEIs that the criteria were being applied uniformly. We propose that during these training sessions, panels are briefed by a staff member from one of the affected institutions about the challenges institutions and staff face as they deal with the preparation of evidence portfolios in the context of day-to-day and long-term impacts of the earthquakes.

Conclusion

The Tertiary Education Union has confidence that TEIs and the TEC recognise the additional issues which will colour the 2012 Quality Evaluation exercise. Whatever the proposed course of action, the PBRF can only continue if the proposed approach has the confidence of academics, TEI management and the TEC. For these reasons, the solution should benefit from a transparent process that has been subjected to fair debate.

We present this proposal in the spirit of providing TEC with ideas that possibly have not been generated within the organisation. We are of course open to other suggestions from TEC and the TEIs that also meet what we believe are fundamental requirements of fairness to those affected by the earthquakes.

The advantages of this proposed scheme are that:

  • It recognises that there is unlikely to be a single way of adjusting the round that uniformly delivers fairness to the TEIs and the academics:
  • It does so by trusting the academic to know her or his circumstances best and opting to either extend the age of evidence that qualifies for affected academics or allows the accumulated impact of academic portfolios that cannot be objectively graded to be addressed by a pro-rating of institution scores;
  • It requires the least amount of change to the work of the panels and will generate fewer unintended effects on the outcome (especially from inconsistent application of complex adjustments at the portfolio level);
  • It minimises the invention of new or untested criteria or procedures to achieve the purpose of the PBRF.

We acknowledge as well that this proposal comes with an additional challenge. It will be assumed that individual scores of those who do not include older evidence in their portfolios should be lower because of the earthquakes. As a result, an additional penalty to the individual will be the diminished value of their ‘grade’ in comparisons both within institutions and between institutions. Therefore it would be absolutely critical that, for the 2012 round, scores remained confidential to the academic alone.

Appendix 1: Options considered by TEU members at Canterbury TEIs

Below are the seven options considered by TEU members from affected Canterbury TEIs during consultation on this paper. As we noted in the body of the paper, member opinion ranged across all the options, and issues were identified with all the scenarios.

Cancelling or postponing the entire Quality Evaluation exercise

This option does not address the problem of how funding could still be allocated to the Canterbury region relative to other institutions. Freezing funding ratios at the levels determined by the last round would not provide the Government with an indication that each institution was indeed properly investing to ensure quality outcomes.

It would perpetuate the inequities for Canterbury TEIs. The Canterbury region will never get the months of disruption back, and postponement or cancellation may only further distort the outcomes of unaffected institutions.

This option will probably put additional burdens on Canterbury region academics, who will have to prepare portfolios over a longer period of time.

The Quality Evaluation is delayed only for Canterbury TEIs

This option does not address how funding allocations to Canterbury TEIs relative to other TEIs would be managed. Any decision to decrease, maintain or increase the Canterbury proportion of PBRF-allocated funding would not be based on an evaluation of management effectiveness at leading a tertiary institution.

This option does not address the impact of a shorter period for generating future outputs if evaluation of the Canterbury region were ever again reconciled with all other TEIs.

We also suspect that this option will be seen as potentially disadvantaging other TEIs. Any extension period will be arbitrary and whether it was too short or too generous will never be known, increasing discontent with the PBRF.

Special circumstances category used by Canterbury region institutions/staff

While we are confident that panel members do try to be as even handed as possible when evaluating portfolios, there is no doubt that there is an element of subjectivity involved in evaluations and this would be true of the weight given to researchers affected by the Canterbury earthquakes.
Firstly, there is subjectivity in completing the special circumstances category. Some researchers may underplay the impact of the earthquake, while others may overplay the impact on their work. Secondly there is subjectivity in evaluating the research portfolios. Can panellists who are outside the Canterbury region really get a sense from a short paragraph in a research portfolio whether the circumstances being described matter to with regard research outputs, contributions to the research environment, or peer esteem measures?

Such an option is also likely to introduce new work for academics attempting to qualify for the adjustment, and new work for panels trying to implement it.

It may be possible to try and develop some kind of actuarial formula which guides panellists in their evaluation of the impact on the earthquakes on research outputs; however, this type of actuarial modelling is highly complicated. Do we really have any way of establishing what number of ‘points’ losing a house or office has been; how many ‘points’ a person would be allocated for a person who has suffered very little loss as a result of the earthquakes but has been traumatised by the experience; or how many ‘points’ losing a loved one equals?

Representative list of nominees for evaluation

All institutions nominate on a per-head basis a representative list of nominees for evaluation, but they have to be representative across all panels.

To account for the disruption in Canterbury, 1/3 (or thereabouts) of academic staff from unaffected institutions submit profiles while only ¼ (or thereabouts) of academics from Canterbury-based institutions submit profiles. This provision accounts for possible effects of the quake on academics in institutions located outside of Canterbury but collaborating with those who reside in Canterbury or have personal connections in Canterbury.

All institutions will be equally advantaged as they will all put in top candidates, but the scale of the call is large enough to prevent the institutions from having only top candidates.

Extend the ‘start’ date of the PBRF quality evaluation exercise back in time

Extend the start date for this exercise by one, two, or three years. Therefore rather than 2006-2011, the exercise is taken from 2005-2011, 2004-2011, or 2003-2011.

The Quality Evaluation part of the PBRF model is a snapshot in time therefore this option simply changes when the snapshot is taken. It has no impact on individual research development as what is being undertaken currently exists in its own right and is subject to the normal performance management that should occur within a department.

This option equally ‘advantages’ all institutions. It will mean that some people who participated in the 2006 partial round may end up submitting nominated outputs etc. that they have already submitted – but the overall effect for institutions will be a smoothing out of the impact of the earthquake.

This option makes up the estimated loss of 16 months of research time by bringing in an earlier research period of 16 months.

Partial round for all participating TEIs

All institutions put forward a range of representative academics for the round (as per the 2006 round).

Acknowledges that not all Canterbury region academics have had the same degree of disruption to PBRF-monitored activities. Their work during the inter-round period may even have raised the overall evaluation of their institution, and this should be taken into account.

It also recognises that since all academics may not have been disadvantaged by the quake, it would be unfair to lift their evaluations in any arbitrary way.

It acknowledges that academics who, through no fault of their own, may have portfolios that do not accurately reflect their efforts.

This option is likely to lead to front-loading of As and excluding Rs and Cs. However as all institutions will be doing the same thing, the effect on funding will be minimal.

Reduced amount of material required for EPs

The amount of evidence to be included in the EP to be reduced for affected individuals on a pro-rated basis, enabling affected individuals to be judged on a 56-month assessment period.

The “special circumstances” option would be used to describe any quality (rather than quantity) issues that arise as a result of the earthquakes or the reduced term of the assessment.

This option would take into account the extensive period of disruption for those in Canterbury, and recognises that the PBRF Quality Evaluation is supposed to evaluate an individual researcher’s quality of work.

Again however, we are concerned that this approach will be highly subjective.

Print Friendly