Tag: library assessment

The Sisyphean Task of Evaluating Information Literacy: A Beginning

Today is the first official day of my 6-month sabbatical. While I have some ambitious ideas for the time and I hope to be able to expand into many areas of professional interest, my first priority and ostensible purpose is to evaluate the ELIP program, based on both the feedback and the assignment data that were collected this past year.

In the last, lazy days of June, I was able to make a start with my analysis, reading the librarian journals, the student survey answers, and the faculty survey answers. What remains is the bulk of the student assignment data, especially the search strategy worksheets, which formed the major assignment in both semesters.

Even taking into account the increased communication and collaboration with course instructors, truly understanding the long-term impact of our instruction (if any) remains outside of our capabilities. The bulk of literature on one-shot instruction sessions notes the difficulties incumbent upon evaluating any lasting impact of the instruction; we at least have the perspective of the whole semester, as reported by course instructors.

The initial goal of the research was (somewhat vaguely) to evaluate the impact of the program on students. After teaching, collecting data, and keeping this project on my mind almost constantly throughout the past months, the questions I think we can answer are the following:

  1. Did the ELIP program help students succeed in their English 1100 courses?
  2. Does further integration of instruction aid in relationship-building between the library and the TRU community?
  3. How can we improve our instruction practices to better meet student and program needs?

I feel like potential answers have been swirling in my brain since the beginning of all this, but at this point, my most important goal will be to forget all that and start at the beginning.

“I can see that being unobtrusive will be difficult:” Brief notes on Makerspace observations

I took advantage of some increasingly rare downtime this afternoon to continue writing up the results of the Makerspace study. While I hope that we can get shape this into a fully submittable form at some point before my sabbatical in July, my project for today was to massage the results from the naturalistic observations into a narratively digestible format.

I remember doing this observations, almost a year ago now, and honestly thinking that these weren’t much, these tiny thoughts. Part of the goal of having me carry out this exercise, of course, was to fumble towards the futility of objective research. I’m not connected to the space; I don’t have a preconceived picture of it on any scale. I don’t have built relationships with the users there. I’m almost surprised when I find patterns in my own notes, the same words used over and again in disperate sessions.

A note from my very first session reads, “I can see that being unobtrusive will be difficult. People want to talk to me.” And then, in talking through the observations with the staff, hearing their thoughts about their influences on the space culture. It’s the same for the library as a whole of course, not to mention the university, but in these larger spaces we lose sight of it: there is no firm line between users and staff, and for a productive pedagogical culture, we don’t really want there to be. Learning is playing, discussing, and collaborating, and the way that works best is when no one is truly separate.

The library as unreality: The case of process mapping

This week I am in London (Ontario – which I did not expect to lean quite so hard into the whole London but Canada thing) at the Canadian Library Assessment Workshop, a conference I have previously enjoyed and found very useful. This year, I am finding it…unnecessary, a description I could expand on, except that I’d rather shift this post to more interesting musings.

It’s becoming increasingly clear (to myself) not only that I am intellectually bored with my job but that this boredom is increasingly bubbling out in varying degrees of weirdness that I’m sure are visible to others. The reason I made the trek to CLAW this week is to give a workshop on process mapping, which is useful in practice but dreadfully boring to explain. My slides were due a couple of weeks ago, so I put together a milquetoast workshop and sent it off. The problem started when I set up a lunch date with a former STU professor, and in my head I kept picturing her asking me what I was there for, and (again, in my head) I kept having a fair amount of trouble justifying this topic, and what I finally landed on was: I won’t get promoted if I do anything too interesting. And I cannot find the lie.

Is process mapping interesting? I admit that there is a kernel of interest to me, but I had not heretofore examined why that should be. Unearthing interest in what I already do would of course be infinitely better, or at least more convenient, than attempting to forge into something new. Fortunately, the book I happened to bring for the plane was William Egginton’s The Rigor of Angels: Borges, Heisenberg, Kant, and the ultimate nature of reality. The book engages in an interdisciplinary discussion of reality, the self, perception, and uncertainty; and in the introduction I encountered the following quote from Borges:

“We (the undivided divinity operating within us) have dreamt the world. We have dreamt it as firm, mysterious, visible, ubiquitous in space and durable in time; but in its architecture we have allowed tenuous and eternal crevices of unreason which tell us it is false.”

Borges, cited in Egginton

Therein lies much of the interest in both process mapping particularly and libraries as an intellectual project. We have spent our days maintaining (and at one time – I fear passed – developing) structures to describe reality and are the first to be shocked when these are developed or executed imperfectly. Our processes are the world that we dream, but they refuse to emerge consistently in practice. Egginton argues that “minimal forgetting” is conditional to the formation of the self and therefore no one person can access the truth. Process mapping as a collectively-formed representation, exposes these cracks, whether or not we admit they exist.

The importance of a figure like Heisenberg in this conversation is that the act of observing reality changes it. We make choices about what aspects we wish to observe and in doing so change the future (Heisenberg’s entire premise is of course predicated on the quantam world playing by different rules, but neither Kant or Borges would of course be proscriptive to that domain). Libraries are inclined to “make idols out of our tools,” and need benefit from an overly healthy dose of unreality.

So, in the words of Borges, “Let us admit what all idealists admit: the hallucinatory nature of the world. Let us do what no idealist has done: seek unrealities that confirm that nature.”

500 dollars, a room of ones own, and a secret third thing

It’s time, mostly. The secret third thing.

I think a lot about the library as a space for play, because whenever I wonder what exactly what we’re for, I come back to thoughts about third places, which I think, for adults even more than children have to be for play. In the past, I’ve been frustrated by liminality during times when I’ve been focused on progress, but that’s not what liminality is for.

I don’t know if the academic library can always be a true third place, because for most of these students, we essentially are their workplace. But I think there are still degrees to which we can cultivate safety: for experimentation, for difference, for freedom from regular expectations.

Though the trend I see in academic libraries is that we don’t want to.

We don’t take student cards at the doors here like some academic libraries, but our study rooms are very clearly for study only and the list of rules about them has always been longer than I’m comfortable with. The resistance against coffee shops and relaxing furniture always simmers in the background. The academic library is a place to study, of course, but particularly with the diminishment of print collections and the distribution of research help away from building reference desks, there needs to be something to mark the library as place other than a giant study house. We should the place on campus where you don’t get graded, where you can think about things and try things out without judgement and with structures to encourage you.

I spent the morning coding interview data from Makerspace users, which I think is one place in the library that does this very well. Students go in, specifically to experiment, often with no particularly academic purpose, and they feel welcome, they feel encouraged, and then they feel inspired. The staff are seen as helpful and present, and the community and space encourage creation and accept failure. It would be hard to replicate that atmosphere for the entire library, simply because the Makerspace is relatively compact and the library building is very large, but when our mission is “Inspiring knowledge creation” I sometimes wonder, how exactly we are doing that, beyond the regular provision of study carrels, and books and journals. What could make our library spaces more welcoming? More inspiring?

But of course, it’s not just the space, it’s not just the price of admission. When I ask the students about barriers, there’s a number of things, but the one that makes me sad is time. I don’t know if it’s just the third places that have disappeared, it’s the time to engage with them. Basically, the time to think about concerns other than home, other than work. When do we do that? The library can create the space, but who creates the time?

I don’t think we’re anhedonic at least

I presented today to the library staff about the current academic program review the Librarians’ Departmet is embroiled in. The requirement for an academic program review came down from above in an already tumultuous year (but aren’t they all), and we are making our way through it, steadily.

I have to admit, there have been some outcomes to this review process so far that I’m very pleased with. We developed Program Learning Outcomes (PLOs) for our department for the first time, aligned with the 6 frames from the ACRL Framework for Information Literacy. I know there are varying opinions of how much these high-level goals documents matter, but I sincerely do love finding the points where these goals and ideals interact with our day to day. It made it clearer for me when designing the ELIP program to see how what we were doing there factored into our outcomes, and in discussing where we wanted to go with outreach the other day, I suggested going back to the PLOs to figure out ultimately what we wanted to acheive. It’s the same principle as backwards design for courses.

Of course, the major problem with mapping library services directly to PLOs is that it glosses over the massive amount of infrastructure support work necessary to providing these services. Unlike other academic departments on campus, we largely support our own technology and infrastrucure, they are essential to the services and resources we provide, and the amount of labour required needs to account for this support. Technological infrastructure can’t directly support learning outcomes but it can support the programs and services that do. So, my favourite innovation we’ve made on the standard process was to design an infrastructure support crosswalk, mapping all of our systems and software infrastructure to our programs and services. One of the goals of this process should be to communicate the work we do and the work required; I’ve been fairly relentless in attempting to communicate to academic leadership what it means as faculty to support infrastructure, so I feel really satisfied with how the pieces fell into place on this one.

The other perpetual challenge that we (both us specifically and librarians as a whole, I think) face is connecting directly with students and faculty as a pan-institutional program. Surveys, focus groups and the like typically target library frequent and habitual users, so that there is never an opportunity to hear from who we don’t hear from already. This year, rather than attempt to send out our own surveys, we’ve developed some standard survey questions that will go out in other program review cohorts’ surveys to be answered by their students, alumni, and faculty. We developed 3 questions for each survey, and I’m honestly excited to see the responses.

The current major project is the self-study report, which the Program Review team will work on over the next few months in preparation for inviting 3 external reviewers to campus. I can’t think of an external review report without hearkening back to my very favourite report, the UBC Library External Review from 2012 or 2013. I can no longer locate this report on the Internet, so instead it will live in my head forever as the most elegantly scathing report I have had the pleasure to read. I remember a well-placed use of “anhedonic” and a quote from Anna Karenina (“All happy families, etc.”) in particular.

In mentioning this to a colleague, she brought up and was able to send me a copy of the TRU Library External Review report from a 2007 Program Review. While not quite rising to the level of UBC, the 2007 report offers its own particularly delicious nuggets, including the following:

It may be that the lack of change has contributed to lower morale in the library and the perception that nothing will change. The librarians appear to be suffering from “learned helplessness”; ie their belief in their ability to lead change is almost non-existent and they are skeptical that positive change is possible.

Kimberly B. Kelley and Michael Ridley, Review of TRU Library, 2007

I imagine that would have felt like a gut punch to read back then, but from my position of distance, I don’t think the review is targeting the TRU librarians, so much as TRU’s lack of support for the library. There are recommendations in the 2007 report that didn’t happen until long after I arrived. There are some recommendations that I don’t believe have happened yet. I don’t know that any “learned helplessness” has entirely gone away; I catch myself at times wondering why I should bother with certain things or even when we think of goal assessment, realism and achievability are touted as important factors. That being said, I do think we’ve made changes and I think we want to make more. I think we’ve at least made it to a place where our anxieties have not prevented us from having aspirations, which is a starting place if nothing else.

How do we measure, measure our years?

It is annual statistics time for libraries, which requires that I round up various numbers related to our collection, use, facilities, and personnel and report these numbers to the consortia that compile them. On its face, it is the most tedious kind of work that I do, steeped as it is in quantatative metrics and tweaking reports, but it also provides an opportunity to check in and reflect on our reporting infrastructure, to ensure it’s still working and that the procedures make sense.

I’ve always said that being an effective assessment librarian requires holding on to two contrasting beliefs: first, that every single number that you ever report will be wrong; and second, that it absolutely must be wrong in a methodically consistent way that represents the highest level of accuracy that we can aspire to. It hurts me in a way, that my most fundamental professional duty is to be wrong, but meticulously so. When I wish feel more noble about it, I can hearken back to Socrates’ knowing that he knew nothing. We’re basically the same, he and I.

The interesting part of the stats process is after it’s all done, to find the most compelling stories the numbers are telling and to inquire further why they should be this way. I’ve seen a lot of our traditional metrics starting to rebound this year after COVID, which leaves me with a lot of complicated feelings. At the British Columbia Library Conference in 2021, I said the following about usage stats as indicators of value:

Let’s start from a very broad shared ground of believing that libraries have some sort of value to some set of people. And the way we have traditionally measured that value is primarily through quantitative usage metrics…gate counts, head counts, workshop attendance, circulation, reference questions, etc. And for over a year, most of that was just…gone. Gate count: 0. Circulation…even when our curbside service started, the numbers were a fraction of our regular circulation…And yet, I’m completely confident that if I asked you all if libraries largely lost value this past year, I would hear a resounding NO. If I asked you if there was less work put in to these virtual and curbside services than the in-person one, I’d also hear a lot of NOs.

And I think it’s very easy for us to say that our traditional metrics did not work this year, but I don’t think we can just say “well that doesn’t describe our value this year” about gate counts and info lit sessions and physical checkouts without making the next logical hop to…they never actually did.

Paterson, Amy. “How do we measure, measure this year.” BCLA, 2021.

To see our metrics bounce back, for me, is partially to see a grand opportunity for change slipping away, that opportunity to rewrite the library narrative closing. But I suppose libraries alone cannot collapse neoliberalism, and defining our value by our values within neoliberal academia is always going to be Sisyphean.

There are some changes, though. The shift from print to electronic collections stands out in both our collection numbers and circulation figures. While this change might seem semantic, it requires a major shift in thinking about not only the library as place, but who our users are; how, when, and where they access us; and ultimately what the library is even for, which is the question all our metrics should seek to answer. Gate count numbers I still report, but we haven’t tracked our overall gate count figures at TRU since we moved into the new building with the Tim Horton’s in the lobby. I say that last a little facetiously, but I have a glint of hope that we’re moving beyond seeing anything signifcant that the gate count number has to say.

The other number I always like to check on is the librarian consultation statistics that have increased almost every year since I implemented the new booking system in 2017-18. This is one of the figures that for me come closest to getting at a library value story: building relationships and cultivating care for our communities. This past year, the numbers increased by over 200, which is the most significant increase since the 2017-18 year. This is to me in part a story about potential; we’re limiting in how much we can see students by our capacity. How much more could we do if had the capacity?

When I interviewed librarians for my research on academic librarian work during COVID, one of the biggest changes seemed to be the shift from forward-facing (countable) services to infrastructure support (a lot less readily countable). That’s a challenge we’re facing in our current program review work, and it’s a challenge that less flashy services will always face. This would seem to feed into the unfortunate trend of gradually offloading infrastucture onto vendors, which I of course have a lot more to say about but will not follow that snake hand right now.

My BCLA talk from 2 years ago ended with choices:

We make choices all the time, and sometimes we don’t like to admit that’s what we’re doing. If we just sit back and let decisions make us by doing what we’ve always done, we lose our value by losing ourselves.

We are still making choices, and I like to think we (here at the TRU library at least) are doing that a little more deliberately than we used to.