Submission on reporting PBRF results

Posted By TEU on Jul 1, 2009 |

Submission of the Tertiary Education Union (TEU) Te HautūKahurangi o Aotearoa

on the PBRF Sector Reference Group consultation paper

“Reporting of results”

19th June 2009

The New Zealand Tertiary Education Union (TEU) Te HautūKahurangi o Aotearoa is the largest tertiary sector union in this country.”  Our membership currently sits at approximately 11,500 members, covering all types of TEOs in the sector.

Prior to the formation of the TEU on 1st January 2009, both AUS and ASTE made submissions on a range of topics related to the PBRF.”  When the PBRF was introduced, we raised a number of concerns about the model, many of which have unfortunately emerged as issues in the sector.”  Our stance on the PBRF model some years on from its introduction is that minimal changes should be implemented as a result of the current review.”  We have taken this position with the current funding model as we believe that leading up to and post-2012 the sector should vigorously debate alternative models that could replace the PBRF, which we regard as fundamentally flawed.

For this reason the TEU holds the view that further changes to the model, other than minor adjustments (such as clarifying guidelines) would put unnecessary pressure on academic and general staff involved in research and on TEOs, would risk further undermining the data obtained so far from assessment rounds, and should therefore be rejected.

Our concerns about current arrangements for reporting of results

The 2006 PBRF guidelines refer to a number of principles that underpin the public reporting of PBRF results.”  These include the need to protect the confidentiality of individual staff members’ Quality Categories; and maintaining the confidence and co-operation of the academic community; which are the two areas we will primarily focus our comments on in this section.

The TEU supports processes that ensure tertiary sector staff are provided with the means to undertake quality research that advances the pursuit of scholarly knowledge and that informs teaching practice.”  However what we do not support are funding mechanisms that bulk-fund institutions but require individuals to be accountable for the level of this funding (as is the case with the PBRF research assessment process).

We have never been able to understand why this is so.”  If we must engage in assessment of research quality for the purposes of funding at an institution level, surely this process should be the responsibility of senior management within TEOs, and should minimise the time that researchers need to spend away from doing their work.

When the individual scores that staff receive are then able to be released back to the institution, this embeds the belief that individuals are personally responsible for the funding that their TEO receives.”  As well, by allowing this, the TEC potentially leaves itself open to individual cases under the Privacy Act particularly where a staff member has been disadvantaged as a result of the misuse of their PBRF scores.

Evidence of misuse of the scores abounds, particularly in the area of appointments, progression and promotion.”  Institutions have become closely involved in monitoring, screening, assessing and criticising individual portfolios.”  The personnel involved in these processes are often also involved in decisions about internal recruitment, allocation of variable salary payments (additional salary increases have been made available to some staff) and promotion processes such that the privacy of an individual’s score is easily violated.”  Thus the principles of confidentiality, confidence and co-operation are undermined and the initial purpose and intent of the PBRF is compromised.

Our members have also reported that some institutions are distributing their discretionary funding primarily to “winners” of the evaluation process (i.e. those receiving A’s for their individual score), thus further accentuating the impact of the PBRF on individual careers, by further narrowing the disbursement of funding across the profile of researchers.

As we noted in our introduction to this submission, we believe that any changes to the current assessment model should be minimal, allowing for a substantive review post-2012.”  However in the case of reporting of individual results to institutions, we are particularly concerned that measures be taken to secure each individual’s rights to privacy, and to halt the many abuses and misuses of these scores by institutions.

Responses to the Sector Reference Group proposals

Section 5.1: Reporting of individual quality categories to individuals

The TEU recommends that option B: “Individual staff members are only advised of the individual quality category on request” be implemented.”  Current PBRF processes mean that quality categories are returned to staff automatically. ” The Privacy Act gives individuals the right to know their personal score on request, but does not compel an individual to have this knowledge.”  Providing these scores only on request deals with situations where an individual may not wish to receive this information.

Section 5.2: Reporting of individual quality categories to TEOs

The TEU recommends that option D: “The TEC releases Quality Categories to individuals, but only provides summary information to TEOs (headcount and FTE etc)” be implemented.

Maintaining the status quo, (option C), violates the Privacy Act, for the following reasons:

  • The individual has not given permission to TEC to pass on this personal information to their institution or employer.”  Individual staff are required to submit an EP and therefore do not have a genuine opportunity to give free and informed consent in relation to subsequent use of their information.
  • The individual has no certainty regarding ongoing security of information held by TEOs and access to this information.
  • If an individual disagrees with their quality category or component score, again there is no genuine means available to address this.”  The current system allows appeals only on the grounds of process, effectively denying appeals on the basis of erroneous or misleading information.
  • TEOs are using PBRF scores for purposes for which they were never intended, especially for various employment-related purposes, as outlined in section 5.2 of the consultation paper.”  A very recent example at Victoria University of Wellington has seen academic staff at the college of education receive letters stating that they have a contractual obligation to perform research at a standard that would be accorded a “B” PBRF rating. ” The letters vary according to the report received by the staff member on their internal PBRF assessment.”  Letters received by staff whose research was rated “R” in the assessment state that the staff member is required to meet with their head of school and an human resources representative to agree revised research performance objectives.”  These must be met within three months to avoid implementation of a performance improvement plan. ” Similar letters are to be sent to staff in other faculties of the university. ” A further example at another institution is the practice of permitting staff to report their PBRF scores on promotion applications, leaving the applicant in an untenable position – if they do not report their score, assumptions can then be made that may affect their promotion prospects.”  That such practices have been permitted to occur has in these and many other verifiable instances negatively impact on individuals and bring the PBRF into disrepute amongst researchers and other staff.
  • Some staff are being pressured to reveal their scores, so to have the option ‘not to request scores’ (or not to reveal that they have requested them from TEC) gives individuals greater control over their private information in this respect.
  • Disclosure of individuals’ scores does not directly serve or meet the purposes for which PBRF information was collected.

The TEU supports option D because it brings this part of the process in line with the principles of the Privacy Act (particularly principle 11 – dealing with limits on the disclosure of personal information).”  As Adams noted in his report, TEOs already have sufficient information which can be used to inform strategies to build research capacity and quality.”  Therefore to argue that PBRF individual scores are also required for this purpose is misleading, and allows TEOs to abdicate their responsibilities regarding their own internal management and strategy processes.

It is true from the point of view of bulk funding that TEC relates directly and solely with individual TEOs. ” From the point of view of the Privacy Act, however, the TEC also has an obligation to every individual eligible researcher in regard to the accuracy, storage, accessibility, usage and non-disclosure of personal information concerning that individual. ” While TEC may not wish to deal with each individual, how they currently obtain information about research quality means this legal obligation cannot be avoided.

Section 5.3: Reporting of component scores to TEOs and individuals

The TEU recommends option F: “Status quo: The TEC releases all component scores from each point in the assessment to the individual on request.” be implemented.

This option conforms with the above points regarding maintaining personal privacy of information.

Section 5.4: Use of data by TEOs

The TEU recommend option J: “The TEC produces recommended guidelines for TEO use of data and requires them to be adopted.” be implemented.

The significant number of misuses and abuses of PBRF data indicates clearly that there is a need to have enforceable guidelines put in place about the use of of data.”  However the only way to have certainty about ensuring individual data is not misused is not to release this data back to institutions.

Section 5.5: Public availability of research outputs

The TEU recommend the status quo, option L: “No information from evidence portfolios is made publically available.” be implemented.

Any move from this position would violate individual privacy.

Section 6.1.1: Threshold for reporting

The TEU recommends option S: “No quality category profiles are reported at sub-TEO level.” be implemented.

Adopting option S should prevent the problem of inference of individual staff member scores that has occurred with lower-level reporting.

Section 6.1.2: Publicly available aggregate information – reporting of indicators

Our view is that individual TEO results and overall score distributions should be reported in a number of different formats in order to prevent violation of individual privacy and excessive interest in ‘ranking’ TEOs based on average quality scores only.

In line with this position, the TEU supports dimension VII which proposes changing reporting of average quality scores to one decimal place, as there is a ‘false accuracy’ effect with the use of two decimal places.

Section 6.2: Reporting of PBRF results after TEO mergers

The TEU recommends option U be implemented.”  If TEOs choose to merge, they have presumably done so with full awareness of the current research capacity of both parties; there is no good argument therefore for separating the results of the two institutions post-merger.

Section 6.3: Reporting at sub-discipline level

As we noted in section 6.1.1, the TEU does not support reporting quality category profiles at sub-TEO level.”  Given our small research community, we also do not support reporting at sub-discipline level, as in some instances field of research codes may include such a small number of individuals that individual privacy could be violated through reporting at this level.”  The TEU therefore recommends that option Y be implemented.

In our view, reporting broad subject areas nationally and by each panel ensures provides a sufficient level of information for the sector, whilst also ensuring the integrity of personal information collected during the evaluation process.

Section 7: Reporting by subject and by nominated academic unit

As with the previous section, our concern is that reporting must maintain the privacy of individual information.”  Our view is that reporting by subject and nominated academic unit (given the size of our research community) does not satisfy this criteria.”  Therefore the TEU does not support either of the proposed options.


If the TEC continues to allow individual scores to be provided to TEOs, we believe that it will then leave itself open to future challenges from individual researchers under the terms of the Privacy Act.”  Researchers, as employees, have no choice but to complete an evidence portfolio – refusal to do so would mean disciplinary action.”  Any notion therefore that an individual has ‘consented’ to the disclosure of their personal scores to their employer is fallacious.”  For this reason alone, the practice should cease.

Additionally however, the disclosure of private information in the manner that the TEC has been permitting flies in the face of the very strict guidelines that researchers themselves must adhere to when undertaking their own research and disseminating data from human subjects.”  Any ethics committee presented with a research proposal that included distribution of personal information without properly informed consent would no doubt reject it, and rightly so.

The practice of releasing individual scores to TEOs breaches individual privacy and has resulted in use of this information far beyond the original intent.”  Given this, our expectation is that the main outcome of this part of the review of the PBRF is that the practice of releasing individual scores to TEOs ceases. ” This practice encourages (amongst other things) ‘league-tabling’ of individuals and institutions, a situation that is detrimental to the ongoing development of quality research in all parts of the sector.”  As we have noted in our submission, TEOs should not be permitted to abdicate their management and strategic responsibilities and use PBRF data as a proxy for adequate internal systems for assessing research capacity and performance.

Print Friendly