Print

Print


Sara,

Well, I think that we are approaching "dead-horse" territory here, so this is probably my last take on this (especially as the semester started today!).

You write: "An evaluation's means to an end is to make recommendations for continuous program improvement."

Yes. But in order to do that, your assessment must be valid; in particular, what you think you are measuring--program impact on student outcomes--must be closely related to what you actually are measuring--e.g., comparative outcomes of participants and non-participants.

But what if those comparative outcome differences are *really* caused by lurking variables related to self-selection, rather than any impact of your program? My claim is that that would erode the validity of your assessments, and they would be of little help to you in making real improvements to your program. Or at least, they would be of considerably less help than if you were able somehow to account for those lurking variables.

Jered Wasburn-Moses
Math Center Coordinator
Success Skills Coordinator
Learning Assistance Programs
Northern Kentucky University
http://lap.nku.edu
University Center 170F
(859) 572-5779


# -----Original Message-----
# From: Open Forum for Learning Assistance Professionals [mailto:LRNASST-
# [log in to unmask]] On Behalf Of Sara Weertz
# Sent: Friday, January 10, 2014 6:43 PM
# To: [log in to unmask]
# Subject: Re: Motivation vs Academic Success
# 
# Jered,
# 
# I think we're both saying the same thing but in different ways. As a program
# analyst by trade, however, I would never use the terms "believe" or for that
# matter "prove our worth" when talking about evaluating the efforts of
# academic support. And I'm not so sure that our mission should be to
# "convincingly document and, ideally, quantify the help we provide...." End-
# of-the-year evaluations are, more or less, summative evaluations that
# analyze expected program outcomes in an effort to "sum up" program
# results. An evaluation's means to an end is to make recommendations for
# continuous program improvement. If you feel you did not meet your
# program goals, recommendations should indicate how you're going to try to
# do better.
# 
# Second, I didn't raise the question about motivation.  Marcia did, implying
# that motivation is part and parcel of student academic success.  Though I do
# believe that students who self-select have a greater sense of self-efficacy,
# which can be construed as more motivated to learn and thus more apt to be
# academically successful, as a learning specialist, I can't do much with that.
# 
# I don't rule out motivation, but even if we were to discover certain traits
# complement higher levels of motivation, how does this help me or the
# student? On one hand, motivation is not part of our admissions standards; on
# the other, I can't change the student who is lacking. I can, however, train SI
# leaders and tutors to be role models and to offer the best possible
# assistance. I then have to sit back and hope the student takes me up on the
# offer. Change comes from within...and it's not an overnight process.
# 
# sal
# 
# 
# Sara Weertz, M.Ed.
# Executive Director, First Year Experience ASU Station #10915 Angelo State
# University San Angelo, TX  76909
# (325) 942-2595
# [log in to unmask]
# 
# CRLA President-Elect 2013-2014
# www.crla.net
# 
# 
# 
# 
# -----Original Message-----
# From: Open Forum for Learning Assistance Professionals [mailto:LRNASST-
# [log in to unmask]] On Behalf Of Jered Wasburn-Moses
# Sent: Thursday, January 09, 2014 10:46 AM
# To: [log in to unmask]
# Subject: Re: Motivation vs Academic Success
# 
# Sara,
# 
# First, let's be clear: I'm pretty sure that all of us here *believe* that academic
# supports help students, or we wouldn't be here. (Not that you were
# asserting otherwise--I just want to make sure we make that baseline explicit,
# because there have been misinterpretations on this listserv in the past!) The
# question is how to convincingly document and, ideally, quantify the help that
# we provide, so that we can "prove our worth" to skeptical penny-pinching
# administrators. ;-)
# 
# Second, I'm wondering if we are talking past each other a little bit. You ask if
# the goal of academic support is to motivate students or to help them
# achieve. But I (and I assume Marcia) am not talking about any increase in
# motivation that might result from our interventions. Rather, I am talking
# about the fact that there probably is an important characteristic difference
# between students who use opt-in services and those who do not; for lack of
# a better term, I'll call it "motivation," though "self-advocacy" or "self-efficacy"
# might be equally appropriate.
# 
# Students who opt-in to academic services probably have greater
# "motivation" than those who do not. It is *also* reasonable to believe that
# "motivation" is strongly correlated to student academic success. Thus, it is a
# reasonable hypothesis that any greater success demonstrated by
# participants in opt-in programs versus non-participants is really due to higher
# "motivation" in those students, rather than to any particular effectiveness of
# the services. (In statistics they sometimes call this a "lurking variable": some
# unmeasured entity affecting both the independent and dependent
# variables, creating correlation without direct causation between them.)
# 
# In other words, maybe it is the case that an SI program (or tutoring program,
# or Early Alert program, or...) is really very good at attracting students who
# ultimately will be successful in college, due to personal characteristics and
# possibly in spite of their group characteristics. Then students who participate
# in this program will obviously do better than other students, regardless of
# the intervention itself!
# 
# So, the question still remains, how do we--or, indeed, can we--rule out this
# "motivation" hypothesis, or account for it in some way, to measure the actual
# effectiveness of a particular intervention?
# 
# Jered Wasburn-Moses
# Math Center Coordinator
# Success Skills Coordinator
# Learning Assistance Programs
# Northern Kentucky University
# http://lap.nku.edu
# University Center 170F
# (859) 572-5779
# 
# 
# # -----Original Message-----
# # From: Open Forum for Learning Assistance Professionals [mailto:LRNASST-
# # [log in to unmask]] On Behalf Of Sara Weertz # Sent: Thursday, January 09,
# 2014 11:12 AM # To: [log in to unmask] # Subject: Motivation vs
# Academic Success # # Jered and Marcia, # # Yes, you're right.  My approach
# doesn't answer the question about # motivation or even self-selection for
# that matter. It does, however, show # that academic support can help the
# less proficient student. My take on the # difference between the two has
# more to do with the goal of academic # support: Is it to motivate students or
# help them achieve academic success? I # think it's the latter.
# #
# # I'm not sold on the idea that personality-inventory-type assessments such
# as # the MSLQ will help to answer the age-old question regarding motivation
# and # self-selection. Creating a baseline score for participants/non-
# participants is # akin to chasing a ghost; there are so many variables to
# consider when you're # talking about students, particularly first-year
# students who first need to learn # how to learn. Much of this initial process is
# learning to self-regulate, ask for # help, manage time....
# #
# # Students who fail will often say, in one way or another, if we know #
# something is "good" for them, why don't we make them do it. Though, #
# forcing students to attend tutoring or SI raises the argument regarding #
# prescriptive vs optional. Give students the option to work with a tutor and #
# they will opt out. Make students work with a tutor and they're not only #
# resentful but resistant to any help the tutor tries to offer.
# #
# # sal
# #
# # Sara Weertz, M.Ed.
# # Executive Director, First Year Experience ASU Station #10915 Angelo State #
# University San Angelo, TX  76909 # (325) 942-2595 # [log in to unmask]
# # # CRLA President-Elect 2013-2014 # www.crla.net # # #
# **********************************************************
# #
# # -----Original Message-----
# # From: Open Forum for Learning Assistance Professionals [mailto:LRNASST-
# # [log in to unmask]] On Behalf Of Jered Wasburn-Moses # Sent: Thursday,
# January 09, 2014 9:34 AM # To: [log in to unmask] # Subject: Re: ROI
# on Academic Support Services? -- Different Take # # This is a great approach,
# Sara, but I am not sure that it addresses the core # issue raised by Marcia.
# After all, of all the data you cite below, which of them # is supposed to serve
# as a proxy measure for personal motivation or stick-to- # it-ive-ness or
# gumption or whatever? Because attendance rate at tutoring # (including SI)
# is, at least in part, a proxy measure of such traits/skills.
# #
# # The difficult question remains, how much of the effect from ANY "opt-in"
# # intervention is due to the intervention itself, and how much is due to the #
# proxy measurement of certain psychological traits captured by the act (and #
# rate) of opting in?
# #
# # I believe that I have seen some people (maybe even you?) try to address
# this # in the past using personality-inventory-type assessments. This might
# work, if # you had a baseline score for non-participants as well as
# participants...
# #
# # Jered Wasburn-Moses
# # Math Center Coordinator
# # Success Skills Coordinator
# # Learning Assistance Programs
# # Northern Kentucky University
# # http://lap.nku.edu
# # University Center 170F
# # (859) 572-5779
# #
# #
# # # -----Original Message-----
# # # From: Open Forum for Learning Assistance Professionals
# [mailto:LRNASST- # # [log in to unmask]] On Behalf Of Sara Weertz # Sent:
# Thursday, January 09, # 2014 10:15 AM # To: [log in to unmask] #
# Subject: Re: ROI on # Academic Support Services? -- Different Take # # Ah, I
# love this # question...one I think I can answer because this used to be a #
# typical # response to Supplemental Instruction (SI) which has a history of #
# empirical # evidence indicating that students who use SI on a regular basis
# get # better # grades. Faculty often opine that SI students would have gotten
# # successful # grades no matter what; they argue that SI students (or those
# who
# # # self-select) are already the "good" students.   Faculty continued their
# # # criticism of the numbers even after I added qualitative data--feedback #
# from # the students themselves, in their own words, saying they excelled in
# # their # coursework because of SI.
# # #
# # # It was, however, more difficult to be critical of my interpretive report, #
# which # pulls the following data on students enrolled in SI-supported classes:
# # #
# # # * GPA (at the beginning of the term) # # * ACT/SAT scores # # *
# Classification # # * Ethnicity # # * Residency (on/off campus) # # *
# Major/Minor # # * Academic Standing # # * Cohort attributes such as
# athletics, provisional status, international # student, # etc.
# # #
# # # If I run the interpretive reports at the beginning of the term, I get a bird's-
# # # eye view of the class, which allows me to also create individual student #
# # profiles.
# # #
# # # The beauty of the interpretive report is its use as a tool to make
# predictions # # about the students in our SI-supported classes. An example
# would be to # # examine how a freshman with several at-risk factors and low
# ACT scores # # (which tests science acumen) might fare in a traditionally
# difficult biology # # class. Since our SI support focuses on traditionally difficult
# classes where # # many students struggle, we then make predictions on
# success (A, B, or C) # # depending on whether the less proficient students
# and those considered at- # # risk attend SI, how often they attend, and when
# they attend. The # # interpretive report allows us to compile some
# fascinating reports for variety # # of departments and student services. Our
# measurements consistently show # # that no matter how many at-risk
# factors a student may have, the more SI # # visits, the higher the final grade.
# # #
# # # While something like an interpretive report is more difficult to generate #
# with # tutoring, it can be done.
# # #
# # # sal
# # #
# # #
# # # Sara Weertz, M.Ed.
# # # Executive Director, First Year Experience ASU Station #10915 Angelo State
# # # University San Angelo, TX  76909 # (325) 942-2595 #
# [log in to unmask] # # # CRLA President-Elect 2013-2014 #
# www.crla.net # # #
# ****************************************************
# # #
# # #
# # # -----Original Message-----
# # # From: Open Forum for Learning Assistance Professionals
# [mailto:LRNASST- # # [log in to unmask]] On Behalf Of Marcia Toms # Sent:
# Thursday, January 09, # 2014 8:32 AM # To: [log in to unmask] #
# Subject: Re: ROI on # Academic Support Services? -- Different Take # # That is
# great, Leonard.
# # #
# # # One question, though: Do students voluntarily come to your center?  If
# so, # # how do you address the motivation issue?  In other words, who is to
# say # that # these students wouldn't have higher retention rates anyway?
# # #
# # # Best,
# # # -Marcia
# # #
# # #
# # # On Thu, Jan 9, 2014 at 9:06 AM, Roberta Schotka # #
# <[log in to unmask]>wrote:
# # #
# # # > Leonard,
# # # >
# # # > That is brilliant, especially since it is so difficult to link grades # > directly
# to # tutoring, given all of the other contributing factors.
# # # >
# # # > -Roberta
# # # >
# # # >
# # # > On Wed, Jan 8, 2014 at 4:52 PM, Geddes, Leonard G.
# # # > <[log in to unmask]
# # # > >wrote:
# # # >
# # # > > Melissa and any others who are interested, # > > # > > I have
# attached # part of a report that I sent up to the "powers  that be"
# # # > > about the influence our services are having on the bottom line -- # > >
# # retention. In the past, we communicated how we were affecting # > > #
# academic performance.  However, when it seemed like reporting how # > > #
# students were improving academically was not generating the traction # > >
# # that we thought it deserved, I decided to speak the administration's # > > #
# language by adding a retention element to the report.  In short, we # > > #
# compared the re-enrollment rates of students using our services to # > > #
# general student retention, # > athletic # > > teams, etc.  Our numbers #
# rocked!  (I've attached an abbreviated # > > report since I don't think the #
# administration would like us to share # > > financial info publically.) # > > # > >
# # In the actual report, we put figures to the report by factoring in # > > the #
# "real" revenue that is generated per student. For example, # > #
# hypothetically, # > > if the overall retention rate was 70%, but our numbers !
#  # were 86%, # > > then we sh!
# #  owed numerically how much revenue 16% more students # added # > > to
# # the bottom line, thus showing that we are revenue generating.
# # # > >
# # # > > As a result of changing to reporting this way, our reports have been #
# > # going # > > all the way up the chain to the Board.  Recently, they
# specifically # # > > referenced our center and services in the new strategic
# plan!  We # > > # are now preparing for a significant budget increase as well --
# yay!
# # # > >
# # # > > I hope this is useful.
# # # > >
# # # > > Leonard Geddes
# # # > > Associate Dean of Co-Curricular Programs Director of the Learning # >
# > # Commons Division of Student Life Lenoir-Rhyne University www.lr.edu #
# > > # [log in to unmask] # > > (828) 328-7024 # > > (828) 328-7702 (fax)
# # > > # # > > The LearnWell Projects Blog:
# # # > http://www.thelearnwellprojects.com/thewell/
# # # > >
# # # > >
# # # >
# # # > ~~~~~~~~~~~~~~~
# # # > To access the LRNASST-L archives or User Guide, or to change your # > #
# subscription options (including subscribe/unsubscribe), point your web # > #
# browser to http://www.lists.ufl.edu/archives/lrnasst-l.html
# # # >
# # # > To contact the LRNASST-L owner, email [log in to unmask] # > # # # # -- #
# # Marcia Toms, Ph.D.
# # # Associate Director
# # # Undergraduate Tutorial Center
# # # Division of Academic and Student Affairs North Carolina State University #
# # Campus Box 7118 / 101 Park Shops Raleigh, NC 27695-7118 # 919.513.7829 #
# # http://www.ncsu.edu/tutorial_center/
# # #
# # # Public Record Reminder: All electronic mail messages in connection with #
# # State business that are sent to or received by this account are subject to
# the # # NC Public Records Law.  They are retained and may be disclosed to
# third # # parties.
# # #
# # # Confidentiality: Nothing in the NC Public Records Law diminishes the #
# privacy # protections afforded by federal law (e.g., FERPA, HIPAA, etc.) # # #
# ~~~~~~~~~~~~~~~ # To access the LRNASST-L archives or User Guide, or to
# # change your # subscription options (including subscribe/unsubscribe), point
# # your web # browser to http://www.lists.ufl.edu/archives/lrnasst-l.html
# # #
# # # To contact the LRNASST-L owner, email [log in to unmask] # # #
# ~~~~~~~~~~~~~~~ # To access the LRNASST-L archives or User Guide, or to
# # change your # subscription options (including subscribe/unsubscribe), point
# # your web # browser to http://www.lists.ufl.edu/archives/lrnasst-l.html
# # #
# # # To contact the LRNASST-L owner, email [log in to unmask] # #
# ~~~~~~~~~~~~~~~ # To access the LRNASST-L archives or User Guide, or to
# change your # subscription options (including subscribe/unsubscribe), point
# your web # browser to http://www.lists.ufl.edu/archives/lrnasst-l.html
# #
# # To contact the LRNASST-L owner, email [log in to unmask] # #
# ~~~~~~~~~~~~~~~ # To access the LRNASST-L archives or User Guide, or to
# change your # subscription options (including subscribe/unsubscribe), point
# your web # browser to http://www.lists.ufl.edu/archives/lrnasst-l.html
# #
# # To contact the LRNASST-L owner, email [log in to unmask]
# 
# ~~~~~~~~~~~~~~~
# To access the LRNASST-L archives or User Guide, or to change your
# subscription options (including subscribe/unsubscribe), point your web
# browser to http://www.lists.ufl.edu/archives/lrnasst-l.html
# 
# To contact the LRNASST-L owner, email [log in to unmask]
# 
# ~~~~~~~~~~~~~~~
# To access the LRNASST-L archives or User Guide, or to change your
# subscription options (including subscribe/unsubscribe), point your web
# browser to http://www.lists.ufl.edu/archives/lrnasst-l.html
# 
# To contact the LRNASST-L owner, email [log in to unmask]

~~~~~~~~~~~~~~~
To access the LRNASST-L archives or User Guide, or to change your
subscription options (including subscribe/unsubscribe), point your web browser to
http://www.lists.ufl.edu/archives/lrnasst-l.html

To contact the LRNASST-L owner, email [log in to unmask]