For graduate school I was asked to write a fake grant proposal as if I were going to apply for one. I struggled with the assignment, did anything except it, and tried to hide from the deadline. Any time any one ever mentions grant writing it is nearly always followed by “Ugh my life” so I was afraid of it as well–untill now.
There is never an end to paperwork (screenwork?) in the Assessment Office. This isn’t so much a bad thing as it is just the nature of the metaphorical beast. Each semester needs certain reports created, documents drawn up, and files stored safely away. Because unlike a business model where projects open and then close, education is a revolving door of (hopefully) never ending service to the student body.
The real problem is keeping up with all those different exams, surveys, and day-to-day work sheets. Enter the Assessment Matrix! Now when I first learned about the assessment matrix I was a little let down that I didn’t learn kung fu from it, however it has a similar effect on your workload.
I have finished compiling the official enrollment figures for the Spring 2015 trimester and have created a graph to summarize the results.
I have spent some time considering how I should characterize these latest figures without over-editorializing. I am sad to say I have not been all that successful in that regard, so I am left with a bit of disappointment at these results. Our current headcount represents an 8% decline from last spring’s numbers. This decline is relatively small, but when put into the context of the past few years it seems to add up. For instance, our currently enrollment figures are a 15% decrease from our 5-year running average and 26% lower than our 10-year running average. Any way you slice it, our student population has shrunk over time and that means that fewer people get to experience a community I have come to value.
I am not currently able to speak to the potential causes for these enrollment trends. Though I could likely speculate on a wide number of factors, about all I can say with any confidence is that change is about the only constant in the human experience. However, what I can do is offer just a bit more detail about the nature of the changes to the USAO student body over time.
First off, I know that the correct demographic term for this variable is “sex” as opposed to gender. However, the correct term sounds a bit out of place in this type of report and further I am not entirely certain that what I report is actually sex. At any rate, it looks like our gender demographics are relatively unchanged. At the moment, our student body is about 64% percent female. While this proportion changes a bit year over year, the longer term pattern reveals little change. In other words, while we have lost plenty of both over the years, we have lost each gender at about the same rate.
Other Demographic Variables
By contrast to this relative stability in gender demographics, the racial/ethnic/cultural composition of the USAO student body does seem to be changing steadily. Over the past ten years, enrollments have decreased the most among Caucasian and African American students (-35% and -24% respectively). In fact, enrollments have only grown for two categories of students. USAO currently enrolls about 6% more Latino students on average than it has over the past 10 years, while the international student category boasts a dramatic 64% increase in the same time period.
The USAO Office of Institutional Research and Office of Assessment asked for the student body’s thoughts on their overall satisfcation with the university. After collecting our responses and picking a random winner we presented Leerisa with a gift certificate to the USAO Book Store for $300.00!
I am subjected to stories and articles about testing on a nearly weekly basis. Given my work, this might not be particularly surprising. What is surprising is that I run across the subject in non-specialist media. It is not like I get into my car and tune into the Journal of Applied Psychology radio station (from a public safety standpoint, I am glad such a thing does not exist- drowsy driving is dangerous driving). The point of all this is to say that testing, particularly of the standardized variety, is very much a part of public discourse. More to the point, nobody seems to have anything positive to say about it.
I am not a legislator, charged with developing public policy. I am not a standardized test publisher, risking unemployment if the industry changes. What I am is a person with many years of formal training in psychology, much of it in psychometrics specifically. So, I know a thing or twelve about testing and after reading an article in the most recent Chronicle of Higher Education, I felt the need to clear the air about the topic. In all fairness I should point out that the article in question mostly addresses how these tests are used, but they pull readers in with an image of a standardized test score form (sneaky).
Nature of the Test
Psychologists and other professionals seeking to evaluate different aspects of being human find themselves at a disadvantage relative to other scientists. Where a chemist can directly observe the volume of a liquid, or a biologist can directly observe the number of organisms in a specific habitat a psychologist is almost never able to directly measure the objects of her study. We cannot weigh happiness, nor can we count the number of extraversion in someone’s brain. Even ignoring the ethical problems involved in rooting around in someone’s skull, the physical essence of psychological phenomenon still eludes us. We know it has something to do with neural connections and can even point to certain areas of the central nervous system that work harder in certain situations, but we have yet to isolate the physical components of a memory or an attitude. This inability to directly observe the phenomenon of our primary interest can be a bit of a problem, but one that we train hard to overcome.
Much like not knowing about deoxyribonucleic acid (DNA) did not stop geneticists from making discoveries, psychologists continue to advance human understanding in spite of the very real limitations we face. We do this by operationalizing, which is a fancy way of saying we find a way to estimate. For example, while we can’t directly measure how angry someone might be, we can count the number of times that person yells, how hard they punch, or even measure the amount of cortisol in their blood streams. We then combine these direct observations to make inferences about the angry person’s emotional state. Even if I can’t say anything about the person’s amount of angry in some absolute sense, I can absolutely say something about that person’s anger levels compared to some other person. When you think about it, the whole process mirrors what humans do intuitively on a daily basis, only psychologists tend to be more systematic and objective (at least when conducting research).
What does any of this have to do with standardized testing? Each of these instruments are designed to estimate something. In a broad sense, they attempt to estimate the amount of an individual’s knowledge about something. Much like with previous examples, we are not able to measure knowledge in any direct way, but we can operationalize it and measure phenomenon associated with it. In this case, we might give two people the same math test. One of these participants is a math expert and the other is a math novice. It is reasonable to expect that these two test takers will vary in their performance on this test. Maybe the expert will answer more items correctly, or perhaps the novice will quit trying after a certain number of questions. Either way, we can observe the difference in their response patterns and make inferences about their knowledge in that particular domain. This is the basic premise of standardized testing.
Why Use Standardized Testing?
Once we accept that the basic premise behind testing is a sound one (and it is accepted, isn’t it?), we have to understand why they have become so prevalent in the world of education. At least in higher education, I can think of a few reasons, but at a fundamental level it is an issue of scarcity. As the educational system currently exists, we cannot educate everyone that wants an education. Education takes time and money. It is a cost born by bother the individual seeking education and the society providing it. Since scarcity exists, we need some method of determining who gets and education and who does not. Standardized tests give universities a tool to identify individuals that are better prepared for the rigors they will face, so that resources may be better allocated towards them. Standardized test scores allow a university an objective basis for selection decisions; every dollar we spend on a student that did not graduate is a dollar we did not spend on one that might have (see also opportunity cost).
One might reasonably argue that everyone should have the opportunity to get a higher education. I am the last person to disagree with such a notion. Unfortunately, universities do not make those decisions. As a society, we have decided that we cannot or will not spend the resources needed to provide a universal education at the post-secondary level. Whether of not we agree with our national and societal priorities is beside the point. Universities operate in the real system, not the ideal one.
In the interest of a balanced discussion on the use of standardized testing in the higher education context, I should point out that it is not a given that these tests predict college success. It is what my graduate school professors would call “an empirical question.” In other words, it is not something we necessarily know, but we can find the answer with some research. In my own experience, I have found that standardized tests do predict college success, but they aren’t particularly good at it. The specifics are rather too technical for the current piece, but revolves around the difference between statistical and practical significance. Universities currently use the best instruments available, even if the best aren’t particularly good. I have heard arguments that a more holistic approach would be better, but I vehemently disagree. Holistic approaches have a number of serious disadvantages- they are subjective, resource intensive and quite frankly they do not lend themselves to empirical validation. In other words, we have no way of knowing whether or not they work. A holistic approach would have universities replace a system that we know works, if not particularly well, with one we can’t tell works at all.
So where does all this leave us? Well, we know that standardized testing is a reliable and valid tool in differentiating between prospective university students. Further, we know that alternatives to these instruments are impractical, at least as they are likely to be implemented. Standardized test scores are verifiably better than random selection, if not by much, more egalitarian than subjective approaches and much more practical to use. Personally, I find it useless to freak out about these tools when we could use that energy into either improving their utility and/or overhauling our educational system to eliminate scarcity from playing a role.
I am often asked to report on the price of attending USAO. I am taking a break from just that as I write this piece. Normally, tuition and fees take center stage in price comparisons. But today, I was struck by the price of living at USAO. How expensive is it really to eat and sleep while you are a student here.
The simple answer, of course, is too much. I was an undergraduate student too and remember full well what it felt like to be broke constantly. This feeling might be particularly acute for first-time freshman- for the first time in their lives they are being asked to pay for stuff that was always (probably) free. In the minds of many students (I am sure there are some exceptions), the price went up from zero to whatever these things cost now. Zero being what it is, I am unable to compute a percentage increase, but psychologically the increase is potentially enormous.
Of course, room and board were never actually free. They felt free because someone else was paying for it. Unfortunately for us all, at some point every adult begins to pay for all that stuff, so the real price comparison should account for that. In other words, comparing the cost of USAO room and board against free is misleading. You would still have to pay for all that stuff, even if you decided not to live at USAO. So, in order to evaluate the value, we must consider the price of alternatives to USAO room and board.
USAO Dining Value
Looking over the various meal plan options available this academic year (2014), I found that these range in price from $940 to $1435 each term. On a per meal basis, students pay between $6 and $8.70, such that larger meal plans are less expensive per meal. Additionally, I simplified the comparison by eliminating the cash equivalent portion of any meal plan including one. Let’s just call this an average of $7.35 per meal.
To determine whether this is a good deal, we should look at what that amount would purchase elsewhere. As a connoisseur of fast food, I can confidently say that this would purchase a medium meal at only 2 of the places I frequent. Everywhere else in town, I will easily spend closer to $10 per meal (assuming I don’t have to include a tip).
So, if you are ok with eating at the same 2 places every day, then a USAO meal plan is just about even. If you want a bit more variety, while still eating at the less expensive places, USAO will usually cost less. I should point out that price is not the only consideration here. In order to estimate value, one must weigh how much one pays against what one gets in return. It is here that the value proposition swings in the favor of USAO’s meal plans. All of the off-campus meals I considered in this comparison are of the fast food variety (think burgers, fried chicken, etc.). Off-campus meals that are a bit healthier or offer more by way of variety involves increasing our price range. Overall, USAO meal plans offer students a tremendous variety, generous serving sizes (you can stuff your face at our buffet if you really want to), healthy preparation and desert- for a price that can’t be matched. If you can find a better deal in town, please let me know. I will go with you.
USAO Room Value
Just like your food was never really free, the roof over your head wasn’t either. Someone was paying for it and it is usually not cheap. As a USAO student, there is a good chance you are staying on campus (about half of our students do, and that proportion is much greater when we consider just freshman students). Examining the published housing prices for the current term, I see a few options ranging from $1380 to $2630 per term. On a monthly basis, this means $394 to 751 per month. On the surface, this seems like a bit much, even I can admit that. In order to craft a decent comparison, I called around for prices on 1 bedroom apartments in the local area. This was by no means a comprehensive search. I limited myself to commercial rental companies with multiple unit facilities with comparable amenities. Additionally, I only called those rental companies I have been able to discover while driving around town these past few years. On average, students can expect to pay about $410 per month on living arrangements. Once you add the cost of power, cable and internet, the monthly bill swells to $590.
Depending on which living arrangement a student selects, they might find themselves saving almost $200 per month by living on campus. Of course, there is also the possibility that they could be saving about the same amount by living off campus, so the advantage could go either way. When we expand our analysis beyond cost and explore the value proposition, it seems that it all boils down to personal preferences. For instance, living on campus allows you access to our internet connection which is 2 to 3 times faster than normal residential access. I know that students living in Lawson Court have had problems accessing the internet for a while now (I read your student satisfaction surveys). My pal over at information services indicates that they will soon complete a MASSIVE upgrade before classes begin this spring. In fact, that connection will be of the “gigabit” variety, so about 200 times faster than even my own fast home connection. I have seen this thing in action. It is scary fast. How much is this lightning fast connection worth? Since you can’t buy residential access like that off campus, it can be tough to monetize.
Another thing to consider is relative freedom offered by these options. Clearly, living off campus offers students a bit more freedom. You can choose your own roommates, you don’t have to answer to any resident assistants and any number of your rowdy friends can come over for a visit. This can seem like a no-brainer in favor of off-campus living, but it has its drawbacks too. If your roommate bails on you, you still have to pay for the full rent. If your rowdy friends get you into trouble, it is not they who get evicted. With freedom comes responsibility and I know I have gotten burned by horrible roommates more than once.
Wrapping It Up
When all is said and done, I think on-campus living is a very competitive option. Depending on your circumstances, you could be saving a decent amount of money, could be eating better, or at minimum could be saving a few considerable headaches. Personally, I try to get to the dining hall as often as I can. The food is good, plentiful and healthier than the stuff I normally eat. As for living on campus, I suppose I am a bit biased- I had a roommate stick me with a $300 phone bill and skip out on a month’s rent. In that situation, I would have preferred someone else take care of that issue for me. Given the new tech upgrade, I may have to find a way to sneak into Lawson Court.
This morning I remembered something I had not thought about in many years. It is relevant to my approach to institutional research, so I thought I would share. I must have been six (6) or seven (7) years old when I had an argument with one of my slightly younger cousins. He was convinced that firefighters set people on fire, police killed people and “ambulance drivers” made people sick. We had the childhood version of a heated debate about the nature of these professions and no amount of reasoning would convince him he was wrong. At the time I had no idea why I couldn’t win what should have been a very winnable argument and it is only now, viewing this phenomenon through the eyes of a psychologist do I begin to understand what happened.
I should mention that this particular cousin is now very successful and probably no longer thinks that firefighters set people on fire. Given the national news over the past few months, I won’t focus on police shooting people, but I suspect that he understands the normal chain of events here as well.
I share this story because this specific example is amusing and as adults strikes us as an odd way of seeing the world. However, this type of error is remarkably common in the human experience and is not limited to the less experienced minds of an elementary school student. In fact, this type of reasoning is among the major reasons that scientists rely on math to provide evidence for our findings. You see, humans are remarkable at pattern recognition. See see patterns everywhere; we seemed to be wired to do so even with near-random stimuli (see pareidolia). Unfortunately, humans are pretty horrible statisticians. In this case, my young cousin saw the relationship between paramedics and illness, a relationship we might call a correlation. What he got wrong was the normative causal relationship between these two (2) concepts. In other words, firefighters and fire often occurred together, but firefighters did not normally cause the fires.
As a psychologist, these types of errors in reasoning are particularly salient. This is not because I was born better at thinking about probability and causal relationships, but because I was trained over the course of many years to find ways to combat this type of thinking from infecting my professional output (I still occasionally mess this up in my personal life). This is why it is often so hard to get a simple yes or no answer from me, or why my reports are littered with terms like “may, might” and “imply.” It is also why I prepare so many graphs when talking about trends or relationships, rather than discuss raw figures (these require very specific wording that may discourage those without extensive statistical training).
So next time you sit down for a chat with your friendly neighborhood institutional researcher, keep this in mind. We are not trying to be obtuse, we are trying to be accurate.
The average class size at USAO for the Fall 2014 trimester is 13.4 students.
Even better, the median class size for the same term is 10.5 students.
Does anyone know why the average and median are different in this case and what said difference implies about course sizes are USAO?
One major issue facing the field of assessment is the lack of consistency. Many people begin this job (like myself) never intending to arrive in it in the first place. Psychology being such a board brush for careers it is hard to know what to study, research, or think about before starting. As I encounter problems I want to share them here to help others who might have the same questions I did when I started. To start things off, here is Sign-In Consent Form:
I spent yesterday visiting the University of Oklahoma main campus for the annual conference on legal issues. While there, I had an opportunity to refresh my understanding of a few issues I confront in the institutional research trenches. I attended a session on FERPA issues, some basic issues surrounding intellectual property, as well as some developments on immigration law as it impacts admissions.