TEU submission on the PBRF Review

Posted By TEU on Dec 16, 2013 |


Submission of the Tertiary Education Union Te Hautū Kahurangi o Aotearoa

To the Ministry of Education (Tertiary Strategy) on the “Review of the Performance Based Research Funding Model”

4 October 2013

For further information please contact:

Jo Scott
Policy Analyst
Email: jo.scott@teu.ac.nz

Sandra Grey
Vice President (Industrial and Professional Committee)
Email: sandra.grey@vuw.ac.nz

Introduction

The Tertiary Education Union Te Hautū Kahurangi o Aotearoa welcomes this opportunity to respond to the Ministry of Education’s consultation document ‘Review of the Performance Based Research Fund.

The TEU is the largest union and professional association representing staff in the tertiary education sector (in universities, institutes of technology/polytechnics, wānanga, private training establishments, and REAPs).  Since the introduction of the PBRF model, we have raised concerns about its implementation.  At this point, we continue to have concerns about the range and extent of the unintended consequences that have emerged during the model’s implementation, particularly where these have impacted on employment conditions, teaching, learning and research.  We acknowledge that the model has benefitted some; however we would argue that there are sufficient concerns that a much more extensive, independent analysis is required before committing to continuing with this model.

Nonetheless, while we have presented arguments about the distortions, unintended consequences, and issues with the implementation of the PBRF model in our accompanying paper “Why the PBRF needs to go”, we are mindful that if the model is to be kept it must be modified to ameliorate the very worst effects.

Timeframe for consultation

Before moving on, we would like to note that the time frame for consultation will contribute further to the public policy failures of the PBRF model, because a number of the proposed changes do not provide the necessary detail to fully assess their impact.  Further time and effort needs to be taken to talk with staff in tertiary education institutions about exactly what the changes would mean, and what the implications may be for specific groups of staff, including Māori, Pacific peoples, and women in order to ensure no further unintended consequences result from trying to correct current problems in the system.

We have structured our response to address each of the 18 consultation questions, although some of the questions and our responses have been grouped in blocks.

Response to the consultation questions

  1. Do the proposed objectives provide a clear statement of the Government’s expectations for the PBRF? What are the key advantages or disadvantages of the proposed objectives?

The objectives are clearly stated in the consultation document and in other documents, but it is their interpretation and application at the institutional level that in the past has contributed to a number of distortions and unintended consequences that over time stifles innovation, knowledge generation, and compromise universities’ statutory obligations to act as the critic and conscience of society.

We outline specific concerns relating to the proposed objective “support commercialisation…” in questions five and six.

  1.  What are the benefits or risks for tertiary education organisations and teaching and research staff of retaining the core design features of the PBRF, including the individual as the unit of assessment?

All methods of evaluating discovery and the dissemination of knowledge have flaws.  However, the worst abuses of PBRF in terms of manipulation of the system by institutions, and demotivation of staff who fail to fit ‘the model’, have been a direct consequence of using the individual as the unit of assessment.  Our view is that these could largely be overcome by moving to a unit of assessment beyond the individual (for example, by department).  We acknowledge this is not a simple process or an absolute solution to current problems (as is noted in the consultation document) but policy evaluation and change must address any major unintended consequences and distortions caused by the policy approach.  We also note that regardless of what unit of assessment is used, careful evaluation of possible detrimental impacts on individual staff, groups of staff, departments, and types of research should be a priority.

Moving to a unit of analysis beyond the individual would also help to restore the balance between teaching and research.  Since teaching generally involves a large number of academics, peer and institutional recognition has always been more difficult to achieve compared with the high profile of research accomplishments.  Under PBRF, our members have reported that this imbalance is exacerbated, even though teaching is perhaps the way knowledge is disseminated most widely.  A shift to a measure beyond the individual may prove to be fairer on skilled teaching staff (particularly those in practice-based areas such as teacher education, nursing and medicine, law and commerce) and in rebalancing the priorities of the sector.

Therefore our recommendation would be that further work be undertaken to evaluate other levels of assessment.  In doing so, particular attention needs to be given to evaluating whether a different model would ensure all staff engaged in teaching and in research can fulfill these roles in an environment that preserves and enhances the exercise of academic freedom, that encourages creativity and innovation, and that recognises the importance of diversity in teaching and research.

This work needs to be undertaken in a very transparent manner, so that the sector is able to reflect and comment on options, rather than (as has been the case with this review) starting from the assumption that the current measures are the correct ones, and that problems with them can be addressed simply with a series of adjustments.

  •  In your view, would increasing the financial weighting for new and emerging researchers encourage tertiary education organisations to recruit, develop and retain new and emerging researchers?

AND

 

  • What are the benefits or risks for tertiary education organisations and teaching and research staff of introducing a financial weighting for new and emerging researchers?

 

A simpler way of overcoming some of the biases against new researchers may be to reduce the weighting given to the various categories.  We would recommend the following:

A = 3

B = 2

C = 1

This ensures recognition of research excellence but means that there are fewer dis-incentives to hire and support new and emerging researchers.  The change in weighting will also mean that there are fewer disincentives with regard to hiring and retention of academics from ‘practice-based disciplines’ such as nursing or teacher education.  We believe that reducing the weighting will ensure that institutions continue to hire a diverse range of teaching and research staff, an essential approach for innovation, knowledge generation, and quality teaching.

We would also support evaluating the impact of an overall institutional weighting of the staff profile, and doing this in several ways, to overcome some of the biased institutional practices (which we do not believe the Government intended).

Firstly, each institution should be weighted with regard to the number of early career researchers on staff.  Secondly, departments and institutions should be weighted with regard to the proportion of teaching done by permanent/continuing staff who are research active.  At present, teaching is increasingly being handed over to ‘contract staff’ who usually have no allocated time to research and are often on inferior employment conditions to permanent staff.  Moreover, doing so undermines the critical purpose of universities to expose students to research-active instructors.  Given the prioritising of research-led teaching in government and institutional documentation, as well as evidence showing this is the best approach in the tertiary sector, this cannot continue.

  • In your view, would redefining PBRF – eligible External Research Income to include commercialisation income earned from research outputs encourage individual researchers and tertiary education organisations to seek more commercialisation income?

AND

  • What are the benefits and risks for tertiary education organisations, teaching and research staff, and research end users of redefining PBRF – eligible External Research Income to include commercialisation income earned from research outputs?

We think that the change proposed in (5) would encourage individuals and institutions to seek more commercialisable income.  However this would not be without risks.  Our concern would be that research with an obvious commercial outcome may end up being favoured above non-commercialisable investigations, to the detriment of understanding and knowledge-generation.  We believe that research is for the public good, and while commercialisation may result from some of the work done in tertiary education institutions, it is not and should not be the primary goal of these institutions.

The monetary rewards from commercialisation in fact create sufficient incentive without adding additional public investment in commercialisation via the PBRF.  Unlike the peer-review process applied to the outcomes of other research activities, the quality of commercialisation activities cannot be assured.  Likely, any net benefits would be realised only well after the work has resulted in licensable IP.  We would be concerned about public money in the form of a PBRF premium going uncritically to the commercial developments at our tertiary education institutions.

Furthermore, if also ‘incentivised’ in the PBRF process, a risk is again created in terms of maintaining a balanced teaching and research profile.  Making commercialisation a measurable outcome also implies it is not already adequately measured through other metrics.  Indeed, before commercialisation returns are factored in, a means needs to be developed to genuinely capture the ‘public good’ value of non-commercialisable research that currently is not adequately addressed.

  • In your view, would placing a weighting on External Research Income received from non-government sources encourage researchers and tertiary education organisations to seek out research funding from industry, iwi and other community sources?

AND

  •  What are the benefits and risks for tertiary education organisations and teaching and research staff of requiring reporting of PBRF – eligible External Research Income by source and placing a weighting on External Research Income received from non-government sources?

These proposals raise a number of problems and could lead to unintended consequences.  Internationally when the focus has shifted to rewarding universities for seeking (for example) industry funding, this has had disastrous consequences.  In Canada when attention turned to seeing industry funding in pharmaceutical research at universities, work on blue-skies treatments ceased, with the focus turned to minor modifications of easily marketable drugs.  The Ministry of Education must investigate all of the unintended consequences of such a weighting change before considering this.

Additionally, the source of the funding should not factor into an evaluation of an academic’s performance.  For example, competition for funding through the Marsden Fund is intense, but those who receive these grants are no less accomplished than an academic who receives a grant from the fishing industry.  Indeed, industry may use very different criteria for awarding grants than excellence alone.  Relationships, perceived business friendliness, willingness to delay publication and interest in incremental problem solving and other factors can be more important to the private sector than an academic’s overall excellence as a researcher.

In terms of research from other non-government sources (particularly Māori and community organisations), it seems that the Government is instructing universities and other TEIs to ‘double-dip’ from the public purse.  Much non-government revenue in New Zealand is provided by the Government, and TEIs are funded by the Government.  Why therefore would we need to measure the amount of funding obtained from government-funded non-government organisations, unless the intention is to use this as a proxy measurement for effectiveness of networks with these organisations?

  • What are the benefits and risks for tertiary education organisations and teaching and research staff of excluding overseas-based staff from the Quality Evaluation?

The risk that has been associated with the inclusion of overseas-based academics is that institutions may choose to invest in partial appointments of these researchers to inflate their research outcome claims.  This creates an incentive for our universities in particular to export their research staff out of the country, leaving less access on balance by the public, students and government.  We believe therefore that the PBRF should be restricted to academics based in New Zealand.

  • In your view, would reducing the number of research outputs that can be included in an Evidence Portfolio significantly reduce the amount of time spent by teaching and research staff and tertiary education organisations preparing Evidence Portfolios, while still providing sufficient information to allow for robust assessment?

AND

  • In your view, would reducing the number of examples of Peer Esteem and Contribution to the Research Environment significantly reduce the amount of time spent by teaching and research staff and tertiary education organisations preparing Evidence Portfolios, while still providing sufficient information to allow for robust assessment?

Panel members we have spoken to have certainly noted that a reduction in the ‘quantity’ of information provided would reduce the time spent reviewing portfolios.  However the number proposed may be too few, especially if assessment remains at the level of the individual.

  • In your view, would the introduction of a single Research Contribution component together with more explicit guidelines improve recognition of research contribution outside of academia within the Quality Evaluation?

Again for panel members this is a positive change.  For academics often significant time is spent during internal ‘practice’ rounds for PBRF and in the formal evaluation processes, moving items from Peer Esteem to Contribution to Research Environment and back again, as institutions debate where the item best fits and how to ‘get the best value’ out of a contribution made by the staff member.  Having a single component would be a time saving.

  •  What are the benefits and risks for tertiary education organisations and teaching and research staff of removing the special circumstances provisions from the Quality Evaluation?

We have repeatedly been told that the special circumstances category is not taken in to account in the evaluation process, so it does not seem worth people spending significant periods of time filling in these provisions, nor panel members having to read this detailed information.

However, the solution is not to remove the category altogether – as there are some special circumstances which need to be advised in this type of process – but rather to modify how it is recorded.  If we think to the preparation of CVs, candidates would always note any major gaps in time (perhaps due to having taken time out to care for children or elderly parents; having spent time away from a profession to learn new skills and so on).

In the case of the PBRF, it seems important that panels know if a staff member had taken a significant block of leave from the institution and the impact this may have had on outputs.  What must not be done is to require any greater detail than saying “This staff member was on leave from the institution for a period of 6 months/12 months/2 years”.  This keeps details to a minimum, but signals clearly that someone employed for six years will have completely different-looking Evidence Portfolio than someone employed for two of six years.

There are many legitimate circumstances which prevent people completing research for large block of time including, but not limited to: parental leave; medical leave; secondment into a government department or NGO; return to industry for up-skilling; a period of volunteering to an international organisation such as VSA.  We believe that removal of the category altogether may have a particular negative, gendered impact.

  • What are the benefits and risks of disestablishing expert advisory groups, establishing a new Pacific research peer review panel, and limiting cross – referrals to interdisciplinary subject areas identified in the PBRF guidelines?

We are unsure what is intended here or why, so do not feel comfortable making a final evaluation.  However, there are concerns that there will be a rejection of interdisciplinary research if expert advisory groups are removed.

  • What are the benefits and risks for tertiary education organisations and teaching and research staff of removing component scoring for Evidence Portfolios and assigning a single quality category score with accompanying commentary?

There is some merit to looking to simplify the process by removing component scoring.  However the risk would be the possibility of less contestable information and thus a possible reduction in transparency.

  • What are the risks and benefits for tertiary education organisations, teaching and research staff and the public of publishing more detailed breakdowns of PBRF funding allocations to tertiary education organisations?

If there are minimal breakdowns of PBRF funding allocations, then the process becomes more of a funding exercise than a bench-marking one.  It may also reduce the ferocious competitiveness and game playing at the TEO level, which has cost individual staff members so dearly.  If no one “wins” then there is no race, and TEOs can return to evaluating the professional development of their employees by a range of means, rather than using PBRF for ends for which it was not intended.

  • What are the key benefits and risks for tertiary education organisations, teaching and research staff, and students of removing AQS(N) and AQS(P) and using AQS(S) and AQS(E) to report research performance at the institutional level?

AND

  • What are the benefits and risks for tertiary education organisations, teaching and research staff, and students of removing AQS(N) and using AQS(S) to report research performance at the level of subjects and nominated academic units?

AQS(N) and AQS(P) have been used as tools to bypass broader institutional responsibilities for evaluation of performance, and as the consultation document notes, for other ‘negative incentives’ associated with recruitment, development and retention of (in particular) new and emerging researchers.  We therefore support the proposed changes.

  •  What are the benefits and risks for tertiary education organisations, teaching and research staff of this proposal to change data collection to assess staff eligibility and inform reporting?

We think this proposal will assist in reducing the ‘gaming’ associated with collecting information about eligible staff numbers.  We are therefore supportive of the proposed changes.

Additional comments

Some of the questions presented in the consultation document cannot be adequately answered because of a lack of detail or a lack of data and evidence.  If this model is to be retained, the Ministry of Education should seek the following information before making any further judgments about the PBRF and its utility:

  1. Information from editorial boards of New Zealand-based journals, to establish whether they are struggling to get submissions, or if the type of submissions they are receiving and publishing has changed.
  2. Independent research commissioned, seeking interviews with key informants about their perceptions of the utility of the system (for example panel members, designers, research directors, team leaders, and of course, academics, at varying levels of experience) to evaluate their perceptions of the positive and negative aspects of PBRF, and the surrounding processes that have developed.
  3. The full impact on the workforce profile must be assessed; this includes collating and publishing the age demographics of the tertiary education workforce since inception of the PBRF.
  4. The full human resource management impact of PBRF must be assessed in any proper evaluation of the system.  To do this it is necessary to have independent research which looks at hiring and firing and redundancy patterns around census dates, as well as the creation of new categories of employees since the inception of PBRF.

References

 

Smith, R 2011 “Performance-Based Research Funding: why it should end now” NZCPR Research 25th March 2011 http://www.nzcpr.com/category/nzcpr-research-papers/

Lee, Frederic S. 2007  “The Research Assessment Exercise, the state and the dominance of mainstream economics in British Universities”, Cambridge Journal of Economics, 2007.

Malcolm, W. and Tarling N. 2007. Crisis of identity?  The mission and management of universities in New Zealand.  Dunmore Press, Wellington.

Print Friendly