Key questions in creating an effective assessment effort on each campus include:

1. What does the faculty expect students to learn?
2. Can it be demonstrated that students have learned what the faculty expects?
3. How can the results of assessment be used to improve student learning?

These must be answered if a college is to adequately address what the Accrediting Commission for Community and Junior Colleges (ACCJC) calls for in the new accreditation standards, in particular:

• Student Learning Outcomes (SLO's): Knowledge, skills, abilities and attitudes that a student has attained at the end (or as a result) of his or her engagement in a particular set of collegiate experiences.
  
  
• Assessment:  the systematic collection of data and information across courses, programs and the institution, an integral part of teaching and learning used to help both, and an essential component of a college's mission.

Assessment is NOT the same as placement testing, classroom assessment techniques (CAT’s), or institutional effectiveness, all of which are sometimes used interchangeably with the term “assessment”.  Placement testing is assessing students’ basic skills at college entry; classroom assessment techniques are specific methods instructors can use to improve teaching. Institutional effectiveness is too often confused with assessment.  While the latter focuses on student learning outcomes, institutional effectiveness focuses on the other aspects of an institution including: effectiveness of mission, governance, facilities, finances, and so on.

The function of assessment is: to focus on student learning outcomes (SLO’s), but also includes process, especially in seeking ongoing improvement; to demonstrate and improve student learning and student success; and to facilitate accreditation, accountability and institutional effectiveness.  Accreditation should not serve as an alternate form of personnel evaluation.


MODES OF ASSESSMENT

A comprehensive and effective assessment program should include both direct and indirect measures.

• Direct assessment: the measurement of actual student learning, competency or performance.  Examples of such measurement can include: essays, tests, speeches, recitals, capstone experiences and portfolios.
  
  
• Indirect assessment: the measurement of variables that assume student learning such as retention/persistence, transfer and graduation rates, and surveys.

FOUR PRINCIPAL AREAS OF LEARNING TO ASSESS

1. General Education: may include such skills as writing, critical thinking, problem solving and quantitative analysis as well as such content areas as: arts and humanities, mathematics, science and social science.
2. Program/Major: includes the knowledge and competencies expected of students in achieving a certificate/degree beyond basic skills and in addition to general education.
3. Basic Skills: reading, writing, ESL and mathematics below the college level.
4. Student Services/Academic Support Services: includes both the affective outcomes defined by a college and the expected outcomes of student programs including EOPS, CARE, DSPS, counseling, tutoring, learning center, etc.

EFFECTIVE ASSESSMENT PROGRAMS

An effective assessment program should include a component addressing both access (how well a college is serving its defined community) and equity (how well a college is achieving successful outcomes for different groups of students such as race/ethnicity, gender and age).

Assessment should be implemented at the institution, program and course level.  Each institution needs to decide for itself where and how to begin. Some institutions will benefit from beginning at the course level, while most will probably be better served by beginning at the program or institution level. The assessment process must include significant faculty involvement and leadership and be strongly supported by administration at all levels.  Assessment should be institutionalized, therefore ongoing and cyclical in particular aspects.  The use of a coordinating committee is important, as is the use and integration with such ongoing college efforts as: program review, matriculation, institutional research, Partnership For Excellence (California), and accreditation (both institutional and program).  Creating an assessment effort devoid of other ongoing programs and institutional efforts is counterproductive and expensive in both time and money.

To be effective at the institution level, assessment must provide ongoing feedback to help improve student learning.  Embedding assessment efforts into courses assists in student motivation and performance.  Utilizing existing data bases, tests (both standardized and home grown), sampling techniques and local expertise (e.g. the institutional researcher) are important aspects of a successful assessment effort.  Taking advantage of what has been attempted, learned and accomplished by others is very helpful.

Like instruction itself, assessment is never ending hard work that is an essential component of student learning and student success; it requires extra effort campus-wide.  Accepting this fact is helpful and motivating.  And, once assessment activities are in motion, it is crucial to use the results to improve student learning.

STUDENT SERVICES AND ACADEMIC SUPPORT SERVICES

Definition

Student Services includes such programs as: Counseling, EOPS, CARE, DSPS, PUENTE, MESA, international students, athletes, matriculation, student activities and student development courses.

Academic Support Services includes such services as: tutoring, learning center (writing center, math lab, etc), supplemental instruction, and computer-assisted instruction.

Of course, there are other areas, services and programs that are traditional included in student services, from admissions and records to financial aid to sometimes security and food services.  All of these are, or should be important to institutional effectiveness, but they do not usually relate directly to student learning outcomes (SLO’s).  An institution that tries to define all areas equally, for whatever reason, is more likely to get bogged down in minutiae and overwork, increasing the likelihood of missing the improvement of student learning. 

Assessment

Astin (1991) of UCLA proposed a model of program evaluation that still seems appropriate today in developing an assessment system for student services and academic support services.  His I-E-O model looks at Input, Environment and Output.  Input refers to what the students bring with them as they enter a program or institution: demographics, past academic achievement, basic skills proficiencies, learning styles, study skills, and various affective factors (e.g., motivation, etc.).  The Environment includes the programs and services an institution provides for students including such components as: counseling, instruction, matriculation, tutoring, mentoring, computer assisted learning, and so on.  Output includes the outcomes/results of what has happened to the students who were served by the program/service/institution. 

Assessment of student services/academic support services could begin with Astin’s I-E-O model of evaluation as a theoretical framework and then explore combining traditional program review processes with additional focus on the outcomes of the effort(s).  Some or all of the following might be used in examining the process variables traditionally included in an effective program review:

• Leadership
• Lines of authority
• Staffing
  • number
  • credentials

• Number of students served
• Facilities
• Budget
• Climate
• Satisfaction
• Services offered

An effective assessment design for student services/academic support services would add measurable outcomes to these traditional program review/process variables. As is true for the other areas, there are two major directions in assessing the efforts of Student Services/Academic Support Services: direct assessment and indirect assessment of students served by a program or service. The former is difficult to accomplish, especially in a two year college.  The latter follows a pattern similar to effective program evaluation, with the addition of student outcomes.  More specifically:

A.        Direct Student Learning.  This is an assessment program for student services similar in process and design to one for instructional programs.

1. Begin by defining the area of learning expected, including the definition of the learning outcomes expected in that area.  Possible example areas include: self-esteem, self-confidence, leadership, etc.
2. Define both which students are expected to achieve the outcomes in the area(s) selected and the criteria for achieving success for each outcome expected.  The students selected become the cohort to be assessed, and may include students in a single course or a program or, less likely, all students enrolled at the college.  The cohort might then be further defined by time limits, units enrolled, etc.
3. Select appropriate tools for assessment.  A pre- and post-testing design might be employed to assess where the students are at the beginning of the process and where they are at the end of the process/course/service provided.  Without assessing the students’ beginning level on a trait, it is not possible to determine the impact of the program or services on students’ learning/development of that trait.  The post-test assesses whether students have achieved the level expected on the trait(s) being assessed. The post-test can also provide information on the change (or “value added”) when compared to the pre-test. (A “test” is used here, especially with student services, to include a survey or other assessment tool.)
4. Collect and analyze the results of assessment.
5. Use the results to improve a program/service and its outcomes.

Caution: all of the complexities of experimental design, and the multitude of possible impacting variables, are beyond the scope of this article.  However, it is suggested that in assessing impact, do not get mired in debates over “cause and effect”.  For example, suppose an institution wants to assess its impact on student self-esteem or leadership.  Also, suppose the entering students (in a program or institution) generally score low on a pre-test of this trait.  On the post-test, however, the students generally score high on this same trait.  A reasonable person might conclude that the students improved or/achieved success on this trait because of what the program, service or college had done to improve student performance.  Critics, on the other hand, might warn that other factors led to the changes/improvements found, such as maturation of the student or other factors or programs, on or off campus.  While these critics are not wrong, educational programs/services do not permit the controlling of variables that pharmaceutical companies, for example, use to test the impact of drugs (so-called “double-blind” experiments where neither the patient nor the person who administers the drug knows whether an actual medicine or a placebo is given). 

We do not and cannot control all the possible variables (“intervening variables”) in education.  The trap is to conclude that we cannot, therefore, make any reasonable conclusions.  Not true!  The use of multiple variables and the replication of studies are powerful tools to use in addressing these intervening issues. Other traditional methods such as random selection, use of control groups or, more likely, comparison groups, also can contribute significantly to addressing extraneous variables.

B. Indirect Measures.  In this assessment model, a cohort of students is identified, selected and then followed over different time periods.  In this method, called longitudinal cohort analysis, it is important to emphasize that these cohorts do not change.  A student may drop out of the program being assessed, or even out of the college, but will still remain in the cohort.  If they re-enroll in the college or program, they continue to remain in the originally defined cohort, although they may become part of another cohort. For example, a cohort of 100 students enters a program and, one year later, 62 of them are still enrolled in the program.  The persistence rate would be 62% regardless of the number of students who dropped out during the year only to reenroll prior to the beginning of the second year, when the follow-up is made. This procedure permits taking a picture or snapshot of what is and has been happening to the cohort at various time intervals on any of a variety of variables.

An example of such a cohort would be the selection all EOPS students who began at the college during a particular semester.  Let’s assume that the cohort consisted of 100 new EOPS students who began in the EOPS Program in the fall of 2000.  We then take a picture of what has happened to those same 100 students one, two or more semesters or years later.  We might look at how many and what percent of these students are still in the program, are still in the college, achieved a degree, transferred to a university, achieved a cumulative GPA of 2.0 or higher, and so on.  This would give us feedback on what has happened to these students and can be used not only to assess the impact of the program, but also to examine the impact of the program on different cohorts, as well as provide information needed to improve the program. 

Please note that this kind longitudinal cohort analysis provides more valid information on the impact of a program on student outcomes and success than looking at the total student population of a program.  In particular, sometimes a practice is followed where students who do not perform well in a program are replaced by new students.  This practice changes the cohort and thus the assessment of the cohort/program. A change in the cohort in this way is very likely to give a distorted (and inaccurate) view of the outcomes of a program because it is not known whether the outcomes are a result of the program or the change in students (cohort).

OUTCOME VARIABLES

Once the cohort is defined, the same indirect outcome variables used with other areas of learning can help assess how effective the program or service is over different time periods: 

• Course completion rates
• Basic skills completion rates
• Retention rates
• Persistence rates
• GPA (semester and cumulative) (mean or percent who achieve a certain level)
• Graduation rates
• Transfer rates
• Success after transfer (GPA, Persistence, Graduation)
• Job Placement rates
• Job/Employer satisfaction

Comparisons using each of these variables can be made with each cohort and other cohorts in the same program, in different programs, or with the student population at large.  These variables can also be used to make comparisons over time to see if program changes are actually demonstrating improvements in outcomes.   Warning: this list is provided as examples of possibilities; it is counterproductive to attempt to use all of them, especially at the same time.  Different programs at different colleges may choose which particular ones to select from the list.

Direct assessment of SLO’s for each area of student services could also be carried out.  However, the impact of the student services program on the achievement of these SLO’s may well be marginal because they have not traditionally been responsible for achieving SLO’s in basic skills, general education or major.  For example, unless a program or service includes direct instruction in its area of responsibility, like teaching reading, it cannot be held directly accountable for whether reading is achieved.

CONCLUSION

Assessment is not simple but neither should it be made overly complex.  The following helpful hints are provided to facilitate implementation:

1. Build on what you are already doing.
2. Don’t try to do everything at once; use progressive development and cycling.
3. In general, start with the easy stuff.
4. Begin with enthusiastic faculty.
5. Build good models.
6. Learn from others.

---

(This article is derived from a larger Handbook on Assessment for Two Year Colleges written by the author.  For a free copy or to make comments or pose questions, please contact: emorante@collegeofthedesert.edu)