Announcements: QUFA President’s Message #5: Evaluation of Teaching, April 7, 2020

Dear Colleagues,

I promised that I would write something specific about teaching evaluation and what the Collective Agreement (CA) allows QUFA instructors to do to represent their teaching in the absence of USATs.  Last week I attended a helpful webinar by CTL on developing your own survey in the absence of USAT and I spoke there about some considerations to bear in mind from a Collective Agreement perspective.  I am concerned though, that I might have sowed some confusion, so I want to clarify what, from QUFA’s perspective, are good strategies for our Members at this juncture.

First, however, it will also help if I acknowledge where I am coming from, because teaching evaluation is an oddly charged topic, and while I am confident that I understand what, objectively, the CA does and doesn’t allow, my sense of where the opportunities in the current situation lie are linked to my attitude to teaching evaluation, which may in turn be tied to my own position as a tenured Professor for whom all personnel decisions save merit are in the rear view mirror. I am aware that the QUFA Members for whom the question of what happens in the absence of USAT is crucial, are those with precarious, or at least not yet permanent appointments, and from that perspective the situation may look different.  From my perspective, USATs are garbage: biased in multiple directions, unreliable as an indicator of teaching effectiveness and, because of their statistical presentation, an easy end-run for assessors around the work of actual evaluation of teaching.  This view does not make me special but is just an echo of scholarly consensus about “student evaluations of teaching” (SETs), and the views set out in a very widely cited arbitration decision two years concerning the use of SETs at Ryerson.[1] From that perspective, the cancellation of USATs is no loss but actually an opportunity to foreground the richer evidence of teaching that can be contained in a teaching dossier one term ahead of the instantiation of the new regime of teaching evaluation associated with QSSET, that would have spelled the death of USAT anyway.[2] However, I also understand that one quality that the pseudo-objectivity that makes USAT bad in my view, also could be valuable to those facing personnel decisions.  A 4.5 that puts you well above the mean for your Unit, is an incontestable demonstration that you had fulfilled your part of the bargain.  USATs may be garbage but at least they are garbage you can stand on.

So, the opportunity and challenge of this moment for instructors facing reappointment, renewal, or tenure is to use the remaining basis for teaching evaluation the CA prescribes, the teaching dossier, in a manner that exploits its nuance while ensuring that its contents force favorable conclusions.  From QUFA’s perspective it is essential that this task be accomplished in a manner that respects the CA.  Even though we are in a situation where one CA provision cannot be fulfilled, Article 29.3.1 that provides that “(USAT) (or another University-wide student evaluation scheme approved by the Parties) shall be used in the by the University in the assessment of a Members’ teaching performance,” the rest of Article 29 including  29.4 that permits Members to survey their students, remains in force.  29.4 affords that permission in the context of a number of restrictions including, the one at 29.4.2: “Member’s course surveys are not for the same purposes as the USAT and shall not be used in its stead in whole or in part.” So, what are the purposes of the Member’s course survey, how is it different from USAT and how might you use it to advantageous effect in the current situation?

First, I’ll say that I think you are going to get better nitty gritty advice on that point from the staff at CTL than you will from me, and I strongly encourage you to rely on their expertise.  I will however try to explain the conceptual framework underpinning Article 29 to help you negotiate this tricky course.  The first point to observe is that all evaluation of teaching has two purposes, a formative one, concerned with making you a better teacher, and a summative one, concerned with making determinations about who should be reappointed, renewed, tenured and given merit awards for teaching.  The formative function fits especially well within the CTL’s mandate, which is the cultivation of pedagogy.  The summative function is the primary concern of the Collective Agreement, because the CA regulates the terms and conditions of employment.  Nevertheless, the CA also ensures the scope for formative activity because both QUFA and the University want to ensure the professional flourishing of academic staff as a condition of employment.  For this reason, the USAT has two purposes: a formative one, to give instructors feedback to improve teaching and a summative one to form the basis for the evaluation of teaching in personnel procedures. Because of its summative function, the administration and use of USAT is specified in detail in the CA, a reflection of careful negotiation between the Parties. In other words, these provisions reflect your union’s hard-won ability to say to your employer, “you can only evaluate our Members’ teaching in a manner that we, their representatives, agree to.” If that control may seem a pyrrhic victory given that it secures as poor an instrument as USAT, it was nevertheless also the condition that let us negotiate the QSSET which will set student responses about instructor effectiveness in the context of specific information about working conditions.  The fact that the surveying of students for the summative purposes of personnel process is negotiated by the Parties to the CA, also means that QUFA will view it as a violation of the CA if the University administration, at any level, devises alternative surveys or elements thereof, and encourages their use in the absence of USATs. 

The other basis for summative evaluation contemplated by the CA is the Member’s Teaching Dossier.  In contrast to the USAT, which is negotiated between the Parties, and has uniform questions (although there are additional questions instructors can add), the teaching dossier is entirely under the individual instructor’s control. It is an opportunity to offer all the context, nuance, and specificity that USAT lacks, and importantly, to put focus on the inputs of teaching—the labour of the teacher—rather than on what the students thought about that labour.  In addition, Article 29.4 provides for the Member’s Course Survey but makes plain that its purpose is formative: “to improve course design and/or teaching effectiveness.” In other words, it is not intended to be a means for the students to communicate to the University that you were or were not an effective teacher. Nevertheless, a Member can submit the survey in a teaching dossier as long as it has been administered in the manner provided for in 29.4.2.  This provision is not as contradictory as it sounds: a survey, undertaken for the purpose of finding out how different elements of your course worked, or how your presentation of material might be improved, is an input in the same way that the design of your syllabus or assessments are.  Such a survey shows you pedagogically hard at work, investigating and correcting your own practice to better assist your students.  USAT does not do this, precisely because it is administered by the University so that it may make personnel decisions.

All of these distinctions between different assessment instruments in Article 29 are of course predicated on the existence of USATs; together they constitute an eco-system and if one organism, the big fish that is USAT, is removed, there are opportunities for the other parts to operate in new ways.  If it’s a great moment for the teaching dossier to get the consideration deserves, it’s also moment for Members facing personnel decisions to be tempted to use the Member’s survey to recreate the apparently firm ground of USATs, potentially in violation of the CA.

As I have said, the CTL is there to help you navigate this problem—but here are some suggestions for how to make the Member’s survey work in your favour.  First, remember that you control the survey and thus you do not need to present the raw responses, but can instead present your reflections on them, even if you use a Likert scale. That means you can take some risks with what you ask. Note also that if you use Qualtrics it always lets you have boxes for students to write qualitative comments.  Questions that are genuinely formative in their intent are specific and risk negative answers, but they can also be phrased in a manner that forces students to accept premises about what the course needed to do, whether they like it or not.  Thus, I might ask my students in ENGL 200 The History of Literature in English: “Did I clearly teach core concepts and terminology for the study of English literature?” Because I have designed my course to foreground the building and use of a critical vocabulary, I think I’m pretty likely to get mostly good scores, and the question doesn’t give students much room to complain that I am boring because I make them learn terminology.  I also know that here in my teaching dossier, I can accompany the survey with a few sample lecture slides that show how I teach terms and definitions by guiding students to use them to describe what happens in a particular literary text. I can present the result by stating: “most students gave strongly positive responses to this question; the few negative responses were accompanied by comments indicating that the student disliked having to memorize terms.” Since my slides clearly show that I am teaching the students why they need this vocabulary, and I am confident that my Head, will acknowledge the value of this practice, I am showing the assessor of my teaching—who is my Head, not my students—the care and precision with which I approach materials.  Four questions crafted this way, and you have a good representation of student response to put in your dossier. And if the numerical responses are in your favour, of course you can put those in too and even stand on them.

Regards,

Elizabeth Hanson
QUFA President
Hansone@queensu.ca

[1] https://www.canlii.org/en/on/onla/doc/2018/2018canlii58446/2018canlii58446.html

[2] QSSET is also a survey of students, though one that cannot be used the way USATs are, and the CA has just been amended so that when use of QSSETs begin, all evaluators of teaching will be required to base evaluation on teaching dossiers when the Member submits them, as well as on the survey itself, which in turn presents results in such a way to force contextualization because there will no longer be easily extracted scores as there are with USAT.