The Division
of Undergraduate Affairs
Office of
Assessment
Bi-Monthly
Newsletter
Volume 2, Issue 7
December 2002
Why Do I Have to Document My Assessment Findings and Decisions?
By Marilee J. Bresciani, Ph. D.
The Issue
The
actual steps of engaging in assessment are not new to many higher education
faculty and administrators. Often
times, it is the terminology used in assessment that is new. Furthermore, assessment is not typically
implemented in a systematic way, (Maki, 2001; Banta et al., 2001; Anderson,
2001) which would result in documented outcomes, findings, and decisions made
from those findings. Thus, when one
goes to demonstrate that they have engaged in assessment by articulating
outcomes, gathering data to measure those outcomes, and making decisions for
continuous improvement, there is no evidence to reveal. There is no evidence not because the faculty
member or administrator has not done the work of assessment; there is no
evidence because none of the steps in the process were documented … in any
manner. So, why should assessment be
documented? This article will attempt
to illustrate why providing such documentation of the assessment process is
important not just for accountability, but it is meaningful in learning even
more about how well your program or course meets its intended end result.
The Introduction
Why do
we have to be able to present evidence that we are engaged in assessment? Isn’t it enough to go about the process of
thinking and doing assessment so that improvements in education can be
made? I once answered the latter question
with a resounding yes. As a matter of
fact, it was not until I started reading the works of the great assessment
experts that I realized I was engaged in assessment but was not familiar with
the terminology, nor did I understand the systematic process that should
accompany assessment. It was in reading
about the systematic process of assessment that I realized documentation is a
necessary part of the entire process.
One
reason to document the assessment process - - and one that we are all aware of
- - is that higher education, in all its aspects, has undergone increasing
scrutiny. One only needs to pick up a
local newspaper to read of how the public does not understand why X was spent
on Y, why there even needs to be some sort of transition program for any given
population, and why the graduate they just hired can’t follow simple directions
in his/her new task. Higher education
undergoes increasing inspection about the learning it can produce; (Ewell,
2002) yet we still refuse to understand why we have to document the intended
results of that which we do. In the
past, and some would argue the present, we have been able to thrive and rely on
our own beliefs, traditions, and our rich institutional history, which included
our own undocumented assessment of our work.
Now,
however, we are at a different age of increased accountability, and we have
been here for a while, yet still, we seem to be dragging our feet in the
accountability arena. We need to
document the effectiveness or lack thereof for all of our constituents - - that
much is true. The accountability reason
is the one we most often hear in regards to why we should document our
assessment process. It is logical and
it is obvious. Yet, what do you do when
faculty and administration could still care less about this reasoning?
I have
always found greater motivation in doing something when I can see its direct
benefit to me. In this case,
documenting my assessment process has had great benefits in the accountability
arena and in the budget arena, but before that happened, there were great
rewards for me in learning more about my program. Simply put, I have learned
more about my program and courses because I had to document the assessment
process. I believe that we need to go
through the process of documenting our own assessment work for ourselves, as
well as for the direct benefactors of our programs and classes. What do I mean? I will attempt to illustrate this point with an example.
The Example
When I
first began to get serious about assessment, I approached it with a bit of
arrogance. How hard could this be, I
thought. I believed assessment was
common sense and I felt I had been doing it for years - - at least the part
where I asked if the program had produced what I expected it to upon its
completion and why or why not? I even
administered satisfaction questionnaires to program participants and used that
information, along with my observations to make improvements in the next
cycle. Throughout this process of
asking if the program “worked”, I never took the time to document the intended
outcomes of the program prior to the start of the program. Nor did I record the findings in a manner
that could tie to any sort of outcomes.
Furthermore I didn’t write down the decisions I made to improve the
program. I didn’t even document and
celebrate program successes. No one
even asked me for the information. I
just made changes in each program cycle.
It was
not until I started to attempt to record the entire process that I realized
that the documentation process itself provided incredible learning
opportunities about how my program or course worked. Writing down what it was that I wanted a program or a course to
accomplish - - what learning and development I wanted to see in the students
and participants - -before I even delivered the program or course caused great
reflection as to why I was even offering the program or the course. For example, when I taught an enrollment
management course, one of the learning outcomes was that the students would
be able to interpret financial aid yield analysis in order to determine how to
influence yield through the controllable aspects of their institution’s
financial aid. In articulating this
as an outcome, I had to ask myself why I thought this was an important outcome
– given all that students need to learn about enrollment management – why
this? I had an answer and the answer
resulted in a refinement of this outcome, allowing me to use the classroom and
assignment time in a more productive manner.
Implementing the Outcome
There
was still more learning for me in the articulation of an outcome. After articulating the aforementioned
outcome, I realized that in the course I had offered in the past, there was no
learning component for the student to understand what portions of a financial
aid package could be influenced by his/her institution so that he/she could
then learn what aid could be adjusted in order to attempt to influence
yield. I had other learning
opportunities for students for the other steps that led to the fulfillment of
this outcome, but I had missed this very important piece. Articulating this outcome and changing the
course syllabus saved the students and me from unnecessary frustrations during
the course. In addition, just writing
this assessment process down helped me to ensure that I would better meet this
particular outcome.
Recording the Assessment Methods
As I
continued in the assessment planning and the recording of each assessment step,
I realized that I was not giving the student any opportunity to demonstrate
that he or she had learned how to interpret the financial aid yield
analysis. I was not providing an
opportunity for the student to demonstrate the application of this information
in their final project. Furthermore, I
was missing an opportunity to check in on the student’s understanding of this
aspect after the portion of this learning had occurred or was suppose to
occur. It was not until I tried to
document where I would gather evidence for this outcome that this realization
occurred - - I now understood that I
had no opportunity to capture the evidence in place, yet since this realization
occurred during the assessment planning and recording of the assessment plan, I
had time to implement the remedy, and I did.
In
documenting how I would measure each outcome, I also discovered that some of
the methods I had used in the past weren’t really gathering evidence for any
outcome. Instead I was gathering
evidence that the students were satisfied with how the group projects were
presented or that they liked my power point slides. Neither piece of evidence gave me to much information about what
the students were learning; all I knew was that they liked the group projects
and the power point slides. Interesting
information, but I could save the students and me time by asking questions that
were going to provide me with more meaningful information on which to improve
my course. Thus, I could improve my
assessment efforts by re-allocating time from earlier efforts that were less
productive.
Recording the Findings
As you may know from your own experiences, it is one thing to look
at data and make decisions, and quite another to commit to writing down one’s
interpretation of the data. Recording the findings from my assessment methods
as they relate to whether an outcome had been achieved provided me with two
learning opportunities. One was in the
viewing of the data and the consideration of how well the chosen assessment
method measured the outcome. This documentation step provided me with
opportunities to improve the assessment method, the implementation of the
outcome, and the articulation of the outcome itself.
The other learning came in gauging the extent to which the outcome
was assessed and met. In recording the
findings, I had visibly committed myself to making a decision to improvement,
in knowing that something either had gone really well or not so well. In this step, I felt I was saying to the
world, now I know the end result and now I need to do something about it. The first time I set out to write down my
first assessment finding, I literally felt afraid. For some reason, I felt that writing the results down made me
more vulnerable, as if someone would see my inner faults and blame me for being
a bad teacher or program administrator.
Now, the documenting comes with much less fear and much more liberation
as I find this step helpful in simply reminding me of what I need to change the
next go around and exactly why.
Recording the Decisions Made
Finally,
the process of recording the decision made; and yes, maintaining the status quo
is a decision and in this day and age of tightening institutional budgets, it
is not necessarily an easy one to make.
Again as you know, most of us do make decisions based on our assessment
results, but we just may not write them down. Why should we? Well, of course to demonstrate
accountability, that we did some thing with the information we collected. Recording decisions may also serve as
helpful reminders to ourselves and to others about why we did make the decision(s)
we did. (Don’t laugh too hard. I need all the reminders I can get.) For example, with tightening budgets, I only
get to teach the enrollment management course once every three or four
years. This time frame does require a
dramatic update in the course readings and assignments. This time frame also means that I cannot
remember the assessment outcomes of the previous course and the decisions I
made to improve that course when I teach it next time unless I write them
down. If this example sounds weak, let
me pose another.
Two weeks ago, we had our program retreat so that we could go over our assessment findings, outcomes, and record interpretation of results and decisions made. As we poured through the assessment plan and the data, a previous year’ s particular outcome, findings, and decision-made struck me. Just moments before seeing this “gem”, I h ad made a sarcastic comment about the “idiot” who attempted to assess the outcome of a particular program without going back to check and see if the participants even understood the overall concept. Well, you can probably guess who the idiot was - - it was I. Having recorded the decision made though, reminded me that there had been no time for that particular “check” and I had recorded some of what I thought, of course, were pretty good reminders so that I wouldn’t make that mistake again. Not only could I not remember that had happened, I would not have remembered the recommendations that I had made in the midst of it all to ensure I wouldn’t repeat that error.
Now,
while I may have convinced you that I am the most absent-minded person alive, I
hope that I have made a little bit of sense regarding the value to you of
documenting your assessment work. If
you are still reading this, then you may be interested in learning about
various ways of documenting your assessment process. I don’t know of any wrong way to do this. If you have discovered one, please let me
know so I can avoid it and tell others to do the same. The following lists some methods of
documentation. All of them vary in benefits
and time committed to that particular process.
At NC State, we are committed to finding diverse ways in which to
document our findings and so I have included a few examples from colleagues who
gave me permission to share their fine work.
I have also listed a few from some other universities.
Ways to Document Assessment Processes
1)
Word
Processing Programs (See Appendix A)
2) Spreadsheets (See Appendix B)
3) Relational Databases (See
Appendix C for and Appendix D for an example
of a homegrown one)
4) Diagrams (See Appendix E)
5) Web (See Appendix F and G)
Conclusion
Documenting
the assessment process is valuable for demonstrating accountability and for
personal learning about one’s program and/or course outcomes. There are several ways in which to document
the process; no one is more “right” than the other. There is however, no way around the time that it takes to record
assessment plans and results in a meaningful way. Time is a valuable commodity and it is needed in order to make
this all work. I continue to hope that
more will see the value of investing the time meaningful and manageable
assessment, which requires documenting the process. And it is my hope that the investment in assessment will pay off
unimaginable dividends.
Appendices
Appendix
A – North Carolina State University New Student Orientation - http://www.ncsu.edu/undergrad_affairs/assessment/files/projects/units/nso_impact_0102.pdf
Appendix B – North Georgia
State College and University Core of Cadets
http://www.ncsu.edu/provost/academic_programs/uapr/resources/ngcsu_plan.pdf
Appendix C– North Carolina
State University Office of Student Conduct
http://www.ncsu.edu/undergrad_affairs/assessment/files/projects/projects.htm#student
Appendix D – North Carolina State University College of Engineering
-
http://www.engr.ncsu.edu/assessment
Appendix
E – North Carolina State University College of Design School of Architecture
http://www.ncsu.edu/provost/academic_programs/uapr/assess/restricted/graphic_design_map.pdf
Appendix
F – Indiana University and Purdue University -
Appendix
G – North Carolina State University Undergraduate Academic Program Review
–
http://www.ncsu.edu/provost/academic_programs/uapr/UAPRindx.html
References
Anderson,
J.A. (August 7, 2001). Why Assessment is Important to Student Affairs. NASPA
NetResults. http://www.naspa.org/netresults/article.cfm?ID=393
Banta,
T. W., Black, K. E., & Kline, K.A. (August
28, 2001). The Challenge to Assess Outcomes in Student Affairs. NASPA NetResults.
http://www.naspa.org/netresults/article.cfm?ID=428
Ewell,
P. T. (2002). Grading student learning: You have to start somewhere.
Maki,
P. (January 15, 2002). Using Multiple Assessment Methods to Explore Student
Learning and Development Inside and Outside of the Classroom. NASPA
NetResults. http://www.naspa.org/netresults/article.cfm?ID=558
Contact
for Questions
Marilee J. Bresciani, Ph.D.
Director of Assessment
Division of Undergraduate Affairs
North Carolina State University
126 Leazar Hall, Box 7105
Raleigh, North Carolina 27695-7105
Phone: 919-515-6433
Fax 919-515-4416