Evidence Collection

All data collection must be preceded with consideration of how it will be analyzed and how well it can answer the question of service efficacy.  Before a data collection scheme is implemented, an overall evaluation method must be chosen to guide what type of data to collect.  Some examples are shown in the next table.

Example Method	Strength of Evidence	Ethical Consideration
Randomly assign students to receive or not receive a service	Can state that a service does or does not cause an outcome	Denies access to a service that may or may not be effective for some students
Randomly assign students to receive directed information about services	Weaker causality claim	All students have access but some receive less information
Track student use and correlate with performance or skills measures; Dose-Response approach	Causality cannot be claimed, evidence is suggestive and should be accompanied by other data such as surveys	No restriction of access or information
Survey to collect self-reported impact of services	Causality cannot be claimed, useful only in conjunction with other information or to assess satisfaction	No restriction of access or information, use student time to complete survey
Case study and journals	Causality cannot be claimed but complex and difficult to measure effects can be noted	No restriction of access or information, confidentiality most important here as case study consists of much detailed personal information

Prudence suggests the collection a few well-chosen pieces of evidence rather than obtaining all possible data.  Observations should consist of quantitative and qualitative data consistently collected over time resulting in a suite of convergent evidence relating services to learning outcomes.  Some types of evidence to collect include:

Quantitative

• Database records such as enrollment, grades, assessment scores, number of uses of a service
• Surveys assessing student satisfaction and self-reported influence of student services

Qualitative

• Journal kept by staff with at least one entry per month
• Case studies of selected students
• Documents such as brochures, flyers, letters, testimonials

Tracking student use of services is the most important quantitative data.  Collecting usage data requires resources for hand collection and entry of data or an automated system as well as labor for support, data extracts, and analysis.  Usage data combined with enrollment, grade, and skill assessment data can connect use of services to performance. 

Qualitative data will also be critical to gather.  For instance, if every worker (staff, faculty, and administrators) recorded at least one relevant observation per month for the years between accreditation reports, a collective body of insight would begin to develop.  Such narratives would become a compelling addition to the suite of evidence.  Workers should be knowledgeable about learning outcomes and record not only their observations but also how they used their insights to improve their services for students. 

As students are exposed to many influences on learning, partialling out the effects of one service from other services or instruction or life events proves difficult.  For quantitative data, standard statistical methods that can help but qualitative data may have an advantage.  Surveys, interviews, and journals can detail specific effects and interactions of one service with another in ways that may be difficult to track with a database or account for with statistical controls.  In addition, some effects will not be immediately apparent but will take time to manifest.  One method to help capture the overall impact of the college over time would be to give exiting students the same assessment tests they are given at the beginning.  As defining or tracking student departure is problematic in open admission institutions, the post-test would probably only be applied to students earning a degree or certificate and perhaps transfer students.

The demands of the new standards will likely exceed the resources of a research office even if a college has one.  The task of evidence compilation is an institutional responsibility and must be distributed across the institution.  Researchers will act as valuable consultants and analysts in the process but every worker at the college has a part to play not just in enhancing learner outcomes but recording their successes and failures.

Concerns about Learning Outcomes

Some suggest that it is inappropriate to relate student services to learning outcomes (Collins 2002).  The concern is that students have a fundamental right to services and if these services are not or cannot be related to learning outcomes, then their existence would be unfairly jeopardized.  On the other hand, with limited resources, it does make sense to direct efforts towards services with proven efficacy.  Nevertheless, some services exist on the basis of ethics rather than profitability and so deserve an unyielding commitment even while we seek to improve them.

Another concern is the uncritical application of business models and concepts to education.  Ideas such as the Baldridge model, benchmarking, standardization, hierarchies, and process control can be useful in many endeavors from making widgets to conducting an orchestra to providing quality education.  However, the primary goal of private enterprise is to make a profit, especially in the short term, while the primary goal of education is to develop human potential.  These fundamentally different goals may require fundamentally different strategies.

Finally, some will worry that learning outcomes represent another fad that will invoke hypnotic tendrils of smoke and little flame.  Perhaps all the measurement and data collection will take valuable resources and not impact student achievement.  In response, one could challenge the new standards even to the point of seeking a new accrediting agency.  Alternatively, the approach taken in this paper has been to take the mandate and try to implement it in a way we are comfortable with that benefits our students by providing them feedback on their progress and assurance that our services are effective. 

Conclusion

Can student services adjust to these new standards?  Yes.  Student service professionals have often been leaders resulting in better services to students.  Consider the waves of new community college students:  re-entry students, women and men in nontraditional disciplines, historically underrepresented students, immigrants, refugees, students with disabilities, and so on.  The learning outcomes challenge to student services can be another opportunity.  Every staff person will focus on assisting student learning and be responsible for collecting data, recording information, and improving themselves based on their findings.  Through the stories of success told both in the precise language of numbers and the complexity of the narrative, we can make our colleges more effective learning environments. 

---

References