PBRF submission on the process of evaluating Evidence Portfolios

Posted By TEU on Nov 6, 2009 |

Submission of the Tertiary Education Union (TEU) Te HautūKahurangi o Aotearoa

to the Tertiary Education Commission on the consultation document “Process of evaluating Evidence Portfolios”

6 November 2009

For further information please contact:

Jo Scott

Policy Analyst

Tertiary Education Union

Te HautūKahurangi o Aotearoa

Process of evaluating Evidence Portfolios


The New Zealand Tertiary Education Union (TEU) Te HautūKahurangi o Aotearoa is the largest tertiary sector union in this country.”  Our membership currently sits at approximately 11,500 members, covering all types of TEOs in the sector.

Prior to the formation of the TEU on 1st January 2009, both AUS and ASTE made submissions on a range of topics related to the PBRF.”  When the PBRF was introduced, we raised a number of concerns about the model, many of which have unfortunately emerged as issues in the sector.”  Our stance on the PBRF model some years on from its introduction is that minimal changes should be implemented as a result of the current review.”  We have taken this position with the current funding model as we believe that leading up to and post-2012 the sector should vigorously debate alternative models that could replace the PBRF, which we regard as fundamentally flawed.

For this reason the TEU holds the view that further changes to the model, other than minor adjustments (such as clarifying guidelines) would put unnecessary pressure on academic and general staff involved in research and on TEOs, would risk further undermining the data obtained so far from assessment rounds, and should therefore be rejected.

Response to Sector Reference Group requests

Section 5.1: Options A-D – dual assessment (self-assessment by TEOs)

At this point in the PBRF’s history, the TEU is primarily concerned with ensuring that staff are protected from misuse and abuse of the process and that minimal changes are made.”  In line with this view, we would be most supportive of option B as it retains the current system and ensures continuity.”  Option D also has some merit, however this is probably outweighed by the disadvantages noted in the paper; certainly history has shown that many TEOs resort to ‘gaming-type’ behaviour in the quest to gain additional funding.”  We would also be concerned at the potential for individual staff facing additional pressure through such an approach.

Section 5.2: Options E-L – design of EPs

The TEU supports option L, which leaves the design of EPs unchanged.”  Whilst the other options are seeking to address some of the difficulties encountered in the past, the degree of change required is in our view too extensive for this stage of the PBRF.

Section 5.3: Options M-O – “impact factors”

The TEU supports option O which enables panels to make a decision regarding the use of “impact factors”.”  Presumably panels intending on including “impact factors” will provide explanatory information to assist those preparing EPs.

Section 6.1: Options P & Q – selection of panel chairs and members

Whilst it could be argued that the current process for selecting panel chairs and members may lack impartiality in some parts of the process, the TEU favours retaining this system.”  Post-2012 in our view is the appropriate time to undertake a full review of the process and the PBRF model.

Section 6.2: Options R & S – number of panel members

The TEU supports option S, which allows for an increase in panel membership for some panels.”  According to the available data, this seems to affect four of the panels; the addition of one more panelist for each of these panels would probably address the issue and would mean minimal additional financial outlay for the TEC.

Section 6.2: The possible addition of peer review panels

The TEU does not support including new panels at this stage of the PBRF model.

Section 6.3: Auditing of information in EPs

Presumably the purpose of the two recommendations is to ensure that all information has been received and that the auditing team has sufficient time to complete their work.”  The TEU would be supportive of the two recommendations on that basis.

Section 7.3: The indicative assessment timetable

The TEU is supportive of the indicative timetable provided by the TEC.

Section 8.1: The descriptors and tie-points

Overall each of the three components is comprehensively described.”  We would however ask that consideration be given to making more explicit the contribution of research to teaching and teaching quality.

Section 8.1: Quality Categories

We do not see the need for wording changes in the current descriptions for Quality Categories.

Section 8.2: The evaluation procedure

As we have previously noted, the TEU does not favour large-scale change to the PBRF model, including any significant change to processes such as applying the evaluation criteria.

Section 8.3: Options T-X – cross-referral of EPs

The TEU favours option T, however option W (provide rationale for grades/scores) also has some merit in terms of transparency for both panels and individuals.

Section 8.4: Options Y & Z – management of conflicts of interest

Presumably the processes outlined in this section will be continued for the next round.”  The TEU would support option Y as a means for further assuring submitters that their work will be assessed impartially.

Section 9.1: The moderation process

The TEU does not support changes to the moderation process.

Section 9.2: The complaints process

Our members’ experiences of the current complaints process have generally been negative.”  Those who have made complaints have found it difficult extracting information about the evaluation of EPs from the TEC, with some having to resort to the Privacy Commissioner in order to obtain this.”  Many have indicated that they found the process to be a waste of time, which indicates to the TEU that change is needed.

It seems unjust to us that a decision made that affects an individual’s career then leaves little effective rights to appeal the substance of the decision.”  In our view this falls short of notions of natural justice.”  The complaints process should therefore be reviewed to ensure that it enables a fair, simple and transparent means for individuals to raise concerns about their Quality Category.

Print Friendly