“How do we carry out a useful qualitative and interpersonal approach to assessment of student learning in our programs, while we know we bring our individuality to the setting?”
Recently a Student Affairs assessment leader at my university asked me this question while we were discussing methods. Our conversation continued that we all want to write student learning outcomes using our best understanding and judgment. We want to collect and analyze the data fairly and inclusively, without making assumptions. We want to tell student stories well. To the goal of fairness, though, “Will I have to hide my feelings, and maybe even my perspectives, in the name of objectivity? Will I have to change who I am or how I communicate? What’s so wrong about my assumptions, anyway?”
How can one or a few educators who implement formative assessment make substantive, data-based changes in programs and services when it remains uncertain whether the outputs address the needs of a host of diverse learners? Do the research questions created by one individual meet the needs of the students served? These are the kinds of questions that current inquiry into learning and assessment epistemologies seeks to address.
Each research decision that an assessment leader or researcher makes, involves one’s individual interests’ attitudes, and habits, (i.e., schema). All of our decisions are infused in the process of the project that brings about an observed situation. Consider how we recruit assessment project teams, articulate a project purpose, write student learning outcomes, schedule the work, gather data, interpret results, etc.
In the qualitative tradition, strategies to address some of the (for lack of a better name) investigator effects have filled chapters and articles on credibility, integrity, and trustworthiness in the approach to gathering information from students in order to learn what they take away from our programs. In fact, the issue of separating science from individual ideology was raised back in the previous century about some of the work of the anthropologist, Margaret Mead. Anthropology often is considered the grandparent of qualitative inquiry. And with “researcher involvement” or using “the researcher as an instrument,” in the current paradigm, it remains that the inquirer’s ideology and “credibility… affect the way findings are received” (Patton, 2002. p. 566). So, respected practice is for the researcher to responsibly and ethically note the existing professional or personal situations that might influence decisions, most often in a section of the report unto itself. But is this enough, or fair, or even useful to address student learning today?
Coming to terms with the reflective practitioner
The reflective assessment practitioner may ask questions such as “To what extent is my personal and emotional situation affecting my considerations of the data? Will my assumptions about the subject(s) affect the success and funding of the project? Do I mark as “uncodable” a particular comment so critical about an aspect of the project that stakeholders have invested in?
Ethical checks and balances are critical to any investigation, from an informal program survey to a large research study. I, as well as my colleagues, strongly believe in and rigorously implement these checks and balances:
- Member checking. The “member” check” is vital and part of the canon of qualitative methodology. Notes taken during a focus group or interview are read back to the subject(s) for their approval, edits, and additions.
- Triangulation of observers. Allow more than one assessment practitioner or researcher observer to be present in the research setting as a notetaker or recorder in order to avoid the biases of one person alone.
- Triangulation to strengthen analysis. Triangulate data analysis by recruiting other assessment practitioners to analyze the data according to their understandings. Then discuss and work through discrepancies (Patton, 1987).
The above list is not exhaustive, but articulates that multiple perspectives are critical in the quest to understand the meaning of data.
How well do we do all of this? Even if we have been practicing qualitative research for a long time and/or are highly self-reflective, with strong teams of co-researchers: Does our transparency do enough to allow us to claim a degree of credibility and trust about the results?
I am not submitting that ethics needs a greater emphasis in the field of assessment (although that may be the case), but am offering that the current set of ethical standards of our practice is not enough to reflect the rich demographics of students assessed and served in our universities today. Specifically, “Uniformity of educational approach is not synchronous with equity” (McCarty, 2018, p. 271). Unconscious bias pervades our everyday experience to varying degrees.
It would be difficult to find someone to disagree with the statement that we’re making assumptions and judgements all the time, every day, based on what people are wearing, how they walk, speak, appear, etc. How do we turn off the “assumption and judging machine” in the name of trustworthy research? That sounds like an impossibility. No wonder assessment leaders get pushback when they talk about assessment and rigor to others.
All that we do as professionals involves ethical responsibilities and judgements
The Student Affairs educator and assessment leader has profound ethical responsibilities to students. Through the programs and services that we offer to students, we can encourage “higher-level thinking on the basis of sensitive and respectful consideration of others” (Hamrick, Evans & Schuh, 2002). We can do this not only by engaging students in our decision-making, but also in acting ethically in our assessment work. Fundamental principles to apply when performing assessment in student affairs include respecting autonomy (no coercion); doing no harm (protecting anonymity); benefiting others (a culture of improvement, not retaliation); being just (fair treatment of individuals and resources); and being faithful (scrupulous attention to finding the truth) (Upcraft & Schuh, 1996). Arguably, the two aspects of assessment most affected by ethical decisions are data gathering and reporting of results. Our work is trustworthy to the extent that we can “bracket or suspend one’s own belief in reality” or reflexivity, not only facing our choices about process, but “with the multiple identities that represent the fluid self in the research setting” (Alcoff and Potter, 1993). That is asking a lot.
How do I “bracket” my assumptions when I’m analyzing data about a program that is special to me? On a deeper level, can I discern if my assumptions, at their unconscious core, contribute to “interlocking forms of oppression? (Zamora, 2018).
Does the belief that you CAN bracket your assumptions ignore the fact that they may need dismantling instead? Have we made credible an ethnocentric habit of mind such as those that have perpetuated concepts of racism, i.e., “I don’t care what color your skin is,” “I see everyone the same,” “I don’t see color,” etc. Coke (2020) explains that the danger of the “I don’t see color” mentality is that it allows individuals to ignore complex racial issues; minimizes struggles of people of color in society; and contributes to the problem because “you can’t fix something you can’t see.” Has the concept of a single researcher bracketing their assumptions become obsolete in an academic society that strives to connect with students in authentic ways that resonate with and honor their individuality?
If all this is true, what do we do?
Phenomenology posits that the researcher enters into the subject’s perspective. Methods for doing this, such as in-depth interviewing, focus groups, even narrative inquiry and storytelling, are taught as ways to delve into lived experience. The phenomenologist approaches a problem with a type of radical empathy toward participants in the study to reflect how they make sense of what they know (Denzin & Lincoln, 2003).
I do not believe this is actually possible with a majority of human beings, no matter their educational background and training. This fissure in my devotion to trustworthiness in the qualitative approach was prompted as I read articles on researcher involvement and bracketing, alongside articles on “Developing Pluralistic Skills and Disposition in College” (Engberg & Hurtado, 2011). Perhaps the more racially literate the research community becomes, the more the impossibility of bracketing is realized. Our research courses may need either to redefine bracketing or use more dynamic ways of trying to get at the student experience.
I am offering that in the current era of research and human rights, the socially literate researcher has more potential than ever to raise awareness of injustice and transform social understandings of who individuals are and why we do what we do. Toward that goal, the impossibility of any one researcher being able to bracket their assumptions and provide the kind of insight the qualitative canon has claimed becomes clear. I think, most often, that bracketing is actually code for “creating my own attempted version of fairness and objectivity as best as I can.”
Engberg and Hurtado (2011) state that studies in diverse institutional learning environments are “made more complex by inequality and intergroup conflict” (p. 417). Perhaps it follows that we optimize our methods and acknowledge the complexities and interdependence of the social situations in which we and our students find themselves today. Maybe this is a route that more effectively would reflect the students we say we are describing. Reflecting the experiences and perspectives of our rich multilingual, multi-cultural, and multi-literate students requires not only a “radical pedagogy” (McCarty, 2018, p. 276), but an equally radical approach to research. I think it is time to propose that assessment and research reframe researcher involvement and what makes assessment information trustworthy.
https://live.staticflickr.com/4599/27601654399_9768efd066_b.jpg
It’s time to move from bracketing our assumptions to dismantling them.
The dismantling can come in various forms, depending on the researcher or assessment leader, the study or setting, and the purpose of the project. Rich, critical psychological research aims to empower groups who have traditionally been marginalized (Evans, Duckett, Lawthom, & Kivell, 2017). Participatory Action Research (PAR) specifically allows participants to be part of the study from its inception (Hope, Brugh, Nance, 2019), which certainly would contribute to minimizing the investigator effect. Daniels and Varghese (2020) argue for the field of teacher education to make central the issue of teacher subjectivity via a constantly changing understanding of self.
Perhaps an assessment or research team could engage in synchronous “memoing” or intentional, mutual journaling during the process of the study. Researcher introspection could become an important element of serious assessment and research. It is feasible that using intentionally diverse teams of researchers could create more trustworthy data (as determined by the subjects of the study) than teams that are organized by convenience, etc.
I believe we are at an important juncture with issues of subjectivity, trustworthiness, and being true to our assessment participants and other stakeholders in higher education. It is time that our theories and models seek to approach our students where they are. And the closer we get to that place, the more effectively our work serves students and opens the road for future assessment and inquiry.
Have you worked on an intentionally diverse team? How have you reframed assessment for inclusivity?
References
- Coke, D. (2020, February 12). The dangers of the “I don’t see color” mentality [Instagram post]. Retrieved from https://www.instagram.com/p/B8e6ZfLlnGU/?igshid=5sq0esn49eb3
- Daniels, J.R., Varghese, M. (2020). Troubling practice: Exploring the relationship between whiteness and practice-based teacher education in considering a raciolinguicized teacher subjectivity. Educational Researcher, 49(1), pp. 56-63.
- Denzin, N. K. & Lincoln, Y. S. (2003). The Landscape of Qualitative Research. Thousand Oaks, CA: Sage Publications.
- Engberg, M.E., & Hurtado, S. (2011). Developing pluralistic skills and dispositions in college: Examining racial/ethnic group differences. The Journal of Higher Education, 82(4), pp. 416-443.
- Evans, S.D., Duckett, P., Lawthom, R., & Kivell, N. (2017). Positioning the critical in community psychology. In M.A. Bond, I. Serrano-García, & C.B. Keys (Eds.), APA handbook of community psychology (volume 1): Theoretical foundations, core concepts, and emerging challenges (pp. 107-128). Washington, DC: American Psychological Association.
- Hamrick, F.A., Evans, N.J., & Schuh, J. H. (2002). Foundations of student affairs practice. San Francisco: Jossey-Bass.
- Hope , E.C., Brugh, C.S., and Nance, A (2019). Community Psychology in Global Perspective CPGP, Community Psychological Global Perspectives, 5, (2), pp. 63 – 69.
- McCarty. T.L. (2018). So that any child may succeed: Indigenous pathways toward justice and the promise of Brown. Educational Researcher 47(5), pp. 271-283.
- Upcraft, M. L. and Schuh, J. H. (1996). Assessment in student affairs. San Francisco: Jossey-Bass.
- Zerquera, D., Reyes, K.A., et al. (2018). Understanding practitioner-driven assessment and evaluation efforts for social justice. New Directions for Institutional Research 177, pp. 15-40 DOI: 10.10002/ir/20254
Sandra Mahoney, University of the Pacific