Many files on this site require Acrobat Reader. To get free Reader, click above.

Data Analysis / Evaluation Techniques
Frequently Asked Questions

How do I determine my analysis capabilities?

• What you may want to do is choose a particular assessment method and then check your understanding of the analysis that will be required in order to use that method. You will most likely know fairly quickly whether you are capable of conducting and understanding the required analysis.

• This answer varies depending on your particular initiatives and evidence that you require to assess particular outcomes. Examples of indirect evidence can include
  • Student work samples
  • Collections of student work (e.g. Portfolios)
  • Capstone projects
  • Project-embedded assessment
  • Observations of student behavior
  • Internal juried review of student projects
  • Evaluations of performance
  • External juried review of student projects
  • Externally reviewed internship
  • Performance on a case study/problem
  • Performance on problem and analysis (Student explains how he or she solved a problem)
  • Performance on national licensure examinations
  • Locally developed tests
  • Standardized tests
  • Pre-and post-tests
  • Essay tests blind scored across units

How much and how often do I need to collect data to make it useful?

• This is a very good question and its answer depends on how your program is run. In other words, how often do you need to gather evidence of how well your program is working in order for you to make changes to improve your program? For some, this may be every semester. For others it may be at the end of an academic year or at the send of a summer. Still others may find it meaningful to gather evidence every other year, as it may take them a while to implement the changes they need in order to see if any improvements from those changes have been made.
• In regards to how much, I use a simple gauge for that. The gauge is the answer to this question, how much information do I need to gather in order to know if X is working well? I often also want to know about y and z, but time never seems to allow me to obtain all of the information I want. Quite honestly, as much time as it takes to gather the data, analyze it, and interpret it, it seems to take even more time to make the changes that the data calls on to be made. Thus, I recommend not gathering more data than one can reasonably respond to, unless you are gathering benchmark data to use for later comparisons. How much data is no more data than one can reasonably respond to? Again, it varies; you will need to find out for yourself.

I would like to see software or coding techniques that would allow analysis for the data.

• We use SAS software to analyze quite a bit of our data. You can purchase books that have analysis code already written in them. See www.sas.com.
• If you are interested in qualitative analysis, I highly recommend Denzin and Lincoln's Handbook of Qualitative Research, second edition, (2002) by Sage Publications.
  

• Seeing what others are doing is certainly helpful when choosing appropriate assessment methods, tools, and criteria. You can get a lot of information from http://www2.acs.ncsu.edu/UPA/assmt/resource.htm. Most importantly however, is your ability to answer the question, "will the evidence that I expect to collect from this method, tool, and/or criteria lead me to making decisions for continuous improvement in regard to the outcome I am intending to assess"? While many attempts may seem like "trial and error", the more astute you are at articulating the intended result of your program and how you know if you have reached it, the more likely you will choose the appropriate assessment method and thus, the less you will feel you are engaging in a "trial and error" process.

• I can't answer this quickly because there are several steps involved and several key things to consider. Thus, I am referring you to a couple of websites that will answer this question.
• http://www.lwc.edu/staff/dkelley/Socl345/validity/
• http://trochim.human.cornell.edu/kb/constval.htm
• http://trochim.human.cornell.edu/kb/reliable.htm
• http://seamonkey.ed.asu.edu/~alex/teaching/assessment/reliability.html
  

• See http://www.ncsu.edu/undergrad_affairs/assessment/evaluation/uapr_process_evaluation_rubric.htm

What are some cause and effect examples of assessment?

• No, I cannot. Cause and effect assumes that there are no other explanations for the relationship you are seeing. In most assessment work, those potential influencing variables or influences cannot be controlled and thus, there is no true cause and effect. For more information on this, see http://trochim.human.cornell.edu/kb/causeeff.htm. And finally remember, the purpose of assessment is to gather information that will assist you in making decisions for continuous improvement.

• As you know, formative assessment means to form or shape the program or performance while summative means to make judgments about the result. Formative and summative is more about the context in which you approach your assessment and how it influences the formation of particular tools rather than influencing the actual methods chosen.
• For example, if you are conducting a semester-long program and you are engaging in formative assessment, you will want to assess your program at points in which you can still influence the end product by making changes in the program. You may have already chosen the assessment methods of surveys, observations, and interviews for your summative assessment. You can use those methods for formative assessment as well; you will just want to pay attention to the timing of the assessment and change what you are looking for. In other words, you won't be looking for your end result half way through the semester. You will be looking to see if your students are on track for that end result.

• Great question. See Denzin and Lincoln's Handbook of Qualitative Research, second edition, (2002) by Sage Publications.

Are multiple t-tests used to determine significance?

• I am not sure I understand the question. There are several ways to determine significance in quantitative data analysis. A t-test is one of those methods and is used to determine whether the means of two groups are statistically different from each other. See http://trochim.human.cornell.edu/kb/stat_t.htm for more information about t-tests.