Legislators and policy-makers seem to have two salient information needs when they think of public programs. They need to know which program has done a good job (or a bad job). This is the need for information regarding program accountability. They also need to know how programs work to achieve (or fail to achieve) designated outcomes or outputs. Understanding how a program works enables them to make necessary changes to programs, to design new programs, or to estimate budgets. This is the need for information that explains the process of a program (that is, the causal model behind it).

Student services in the California community college system is one area that could benefit from research that produces both kinds of information. However, such research will require time, resources, and planning to succeed. Besides the size of the community college system, the set of outputs (or services) have qualities that make them difficult to analyze. Given the magnitude of the research effort that would be needed, it is necessary to formulate a long-term plan to coordinate and support the increments of research activity that will cumulatively meet the information needs for accountability and explanation (or causal analysis).

An important step in planning a research strategy (or research agenda) is the planning of methods to apply in these research efforts. The following paragraphs tackle this planning step by discussing various methods that could contribute to the achievement of the research strategy or agenda.

A Suite of Research Tools to Consider

A research strategy will naturally consider alternative ways to study a subject area such as student services. Any strategy that we suggest should consider at least the seven general research tools listed below.

• Literature review
  
• Data validation
  
• Survey research
  
• Case study
  
• Quasi-experiment
  
• Meta-analysis
  
• Statistical modeling
  
  

Obviously, the above list does not name all research tools that researchers could use in analyzing student services. But the list names general tools that at this point seem to have especially high potential for the advancement of our knowledge in student services. Of course, to appreciate any proposed strategy, we need a brief, summary explanation of these tools. We provide this below with the aid of language from various references in research.

Literature review: 𠇊 systematic survey and interpretation of the research findings (the “literature”) on a particular topic, usually designed to prepare for undertaking further research on the subject…”[Vogt, 1993, p.130]

• Researchers do literature reviews for several reasons. Most frequently, they do it to see what prior studies of the topic have found. An effective review will help a researcher to design a new study that will add to the existing knowledge base by testing a hypothesis or collecting new information that previously was unavailable. Although some researchers may not have much concern about testing a new theory, they will still do a literature review to understand how to do a study, that is, to learn about which data to collect and which analyses to employ (such as various statistical tests). Researchers also do literature reviews because they use summaries of their reviews in their report (or paper) to provide readers a context or background to the subject. In some instances, an extensive literature review can stand alone as the researcher’s product, especially when policy makers or other analysts need just a current summary of prior work on a topic.
  
  
  
  

Data validation: A process for investigating the quality of a set of measurements. Generally, this involves some estimation of the amount of error that the set of measurements may contain.

• This is a test for the usefulness of the data in terms of their accuracy in representing what they are purported to be. This process is necessary to avoid erroneous findings that may stem from the analysis of bad data. The analyst may test a data file by comparing it with a matching data file (as in large key entry operations where the same data are entered twice by different key entry staff). When data about the same student differ between the two files, some suspicion about the quality of the data may arise. In some instances, staff may check data accuracy by auditing source files for their data (i.e., checking electronic records against paper records). A difficult, but critical, test in data validation is the verification of the theoretical relevance of measurements (which researchers have labeled as “construct validity”). Ideally, validation of a data set will precede analyses that may use that data set.
  
  
  

Survey research: “…Quantitative social research in which one systematically asks many people the same questions, then records and analyzes their answers…” [Neuman, 2000, p.520] and “…relatively systematic, standardized approaches to the collection of information on individuals, households, or larger organized entities through the questioning of systematically identified samples of individuals.” [Rossi, Wright, & Anderson, 1983, p.1]

• This tool probably could benefit the research of any process or element in student services. How students, counselors (or other direct service providers), and administrators view things, how they behave, or the circumstances in effect (as measured by responses to survey questions) could really help explain outcomes in student services and reveal areas of opportunity. Obviously, survey data for this research will cost so much that we cannot expect the state’s huge management information system to collect such specialized data on an ongoing basis. It is prudent to expect survey research to occur on an adhoc basis to accommodate the specialized data needs of researchers.
  
  
  

Case study: “Gathering and analyzing data about an individual example as a way of studying a broader phenomenon…The case may be an individual, a city, an event, a society, or any other possible object of analysis…allows more intensive analyses of specific empirical details…The disadvantage is that it is hard to use the results to generalize to other cases…”[Vogt, 1993, p.30].

• This tool incorporates elements of qualitative research. In the context of this paper, a case study approach allows the researcher to do an in-depth analysis of one institution, one counselor, one student, or one family. In that analysis, the researcher gathers information through interviews, observation, participation, and records (such as documents). Although the researcher may include numeric data in the case study, the bulk of the analysis will deal with the interpretation of the unique factors or a pattern of things involving the case in order to develop a theory (or simply a conclusion) about a process or situation. This tool can advance our knowledge about student services explicitly by unveiling factors or patterns that researchers are not considering and by verifying prior inferences from quantitative analyses. The completion of multiple case studies may also provide us with the building blocks for generalizing a unique finding to a broader population. Because case study research (like many qualitative methods) takes significant time and resources (largely from the extensive field work involved) and is not a standard method of research for accountability purposes, the tool has historically had limited use. (Yin, 1989)
  
  
  

Quasi-experiment: “A type of research design for conducting studies in field or real-life situations where the researcher may be able to manipulate some independent variables but cannot randomly assign subjects to control and experimental groups…”[Vogt, 1993, p.184] and “…studies that resemble experiments but are weak on some of the characteristics, particularly the allocation of subjects to groups is not under the investigator’s control…[Everitt, 2002, p.306]

• Although scientists agree that the experimental research design provides the strongest evidence of a cause and effect relationship, educational researchers have traditionally had trouble with that design’s requirement of random assignment of people to either a control group or a treatment group. As a result, research on causal relationships (typical in program evaluation) has tried to estimate program effects with quasi-experiments, a design that lacks random assignment of people to the treatment and control groups. The common pre/post analysis is a familiar type of quasi-experimental design. (Campbell & Stanley, 1963; Cook & Campbell, 1979).
  
  
  
  

Meta-analysis: “quantitative procedures for summarizing or integrating the findings obtained from a literature review of a subject. Meta-analysis is, strictly speaking, more a kind of synthesis than analysis. The meta-analyst uses the results of individual research projects on the same topic…as data points for a statistical study of the topic.” [Vogt, 1993, p.138]

• This tool has largely come to prominence in the past two decades, especially in education and psychology. It basically gives us a statistical way to find a summary estimate of some theoretical relationship that multiple independent studies, using different data, have investigated in a comparable fashion. It essentially is a statistical upgrade of the literature review tool mentioned above. In fact, published meta-analyses generally rely upon a literature review as a necessary antecedent to the statistical estimate of a summary level of some theoretical relationship. The utility of meta-analysis grows as the number of comparable studies grows. Thus, if researchers cannot locate many comparable studies to use in this research synthesis (another name for meta-analysis), its value in advancing our knowledge about student services or in informing public policy is limited. On the other hand, the availability of meta-analysis makes it feasible to conduct a series of independent studies, in a piece-meal fashion if necessary, in order to get an overall estimate of a program’s effect or of some other theoretical relationship. Thus, the potential for using meta-analysis can allow a research program to proceed with a series of small studies instead of one super study (that may never get funded). Lastly, well-done meta-analyses can clearly have more credibility and persuasiveness than the finding from a single study (or a single site). (Cooper & Hedges, 1994;Cook, et al., 1992)
  
  
  

Statistical modeling: “…a description of the assumed structure of a set of observations…[Everitt, 2002, p.247]

• This tool may occur in the use of four other tools (analysis of a quasi-experiment, meta-analysis, survey research, or data validation). However, we list it separately to highlight that statistical modeling is a critical tool in its own right, with many uses that are not part of a causal analysis. For example, statistical modeling (via an extrapolation model) can provide a forecast of student needs, a diagnostic tool for identifying the most needy students, or an accountability mechanism for institutions. The Chancellor’s Office analysis for persistently low transfer colleges is a recent example of this tool in a non-causal, accountability context. (Bahr, Hom, & Perry)
  
  
  
  

Part of this strategy involves an assessment of the need for researchers to collect new data (labeled as primary data in research circles) for a study. It would be naïve to believe that every important variable, that may explain the process and outcomes of student services, already exists in some MIS or in some other researcher’s prior research. Nor should we expect every population of interest to be included in pre-existing data (labeled as secondary data in research circles). Secondary data often have incomplete “coverage” in terms of variables and populations. Unfortunately, even where secondary data are comprehensive in coverage (as when a special study on the topic has already occurred), they may quickly become obsolete. Moreover, until researchers perform data validation on secondary data, they should remember the phrase “handle with caution” and consider the option of gathering new data from a random sample.

In this paper, quantitative research may seem to have a monopoly in the allocation of focus. The neglect of qualitative methods in social research has long sparked debates from advocates and practitioners of qualitative research, and the recent emphasis on quantitative research in federal policy (No Child Left Behind) has re-ignited this challenge. In any case, the strategy in this article exploits the advantages of qualitative research methods but in the context of case study methods. The strategy downplays any heavy reliance upon qualitative research methods because of two major factors: (1) the interest of legislators in strong accountability (which usually requires numerical comparisons) and (2) the limitations of qualitative findings as stand-alone information for policy-making discussions. (Weiss, 1998; Cook, 1997; Rossi & Freeman, 1993). Nonetheless, researchers of specific projects may find opportunities to integrate qualitative methods by means of the recent approach of mixed methods. (Teddlie & Tashakkori, 2003; Weiss, 1998; Cresswell, 1998).

Now that we have described the tools, we can consider the context of each tool for a research strategy/agenda. Three important issues in a research agenda would be the place of a tool in a sequence of efforts, the value of a tool for providing program accountability, and the value of a tool for explaining causal processes. Conventionally, the largest concerns among evaluation researchers have focused upon the goal of accountability versus the goal of knowledge (explanatory value in this article), but we place the concern of stage in agenda as an important qualifying factor for our planning strategy. (Chelmisky, 1997). Figure 1 displays in our judgment how the seven tools stand in terms of the three important issues mentioned above.

Although literature review, case study, and data validation appear in Figure 1 to lack special value for either accountability or causal explanation, each of these tools can substantially increase the chance of success with the other tools in their resulting value for accountability or causal explanation. Meta-analysis has low accountability value in Figure 1 because meta-analysis often demonstrates the general linkage of a policy to an outcome. Unless the meta-analysis is limited to the institutions or programs for which the public demands accountability (such as studies only about a single state’s institutions), the value for accountability purposes is not high.

Hopefully, a data validation study would precede the use of secondary data, but we realize the tentativeness of that expectation, given the pressures that often act to expedite analyses. In the same token, we hope researchers include a data validation step within a study that uses primary data. From a realistic viewpoint, we should expect an iterative process with loops backward or links between tools to signify the interactive behavior that a research agenda can have over time. In any case, the sequence with which any of the other tools will be used is uncertain, given the unique environment that may exist when a particular topic in student services is selected as a research topic. Finally, a particular topic may not require more than a couple of these tools, and the budget/staffing/administrative environment may not allow more than that.

This discussion should clarify the selection process for a research tool to address a potential topic in student services. To clarify further the possible scenarios for future applications of these tools, we turn to how the tools could help in the study of a particular part of student services, outreach to increase student usage of financial aid.

An Example: Outreach Program for Financial Aid

As a demonstration of how the general strategy may apply to a concrete situation, we can use a hypothetical scenario. In this hypothetical scenario, the state is planning to study a proposed outreach program that is intended to increase the use of financial aid by target populations. To make this example meaningful, we will first review some specific issues that distinguish financial aid outreach from other programs so that the link to the choice of research tools has more realism.

The process of outreach generally could follow a basic model like that shown in Figure 2. This model is critical in highlighting explicit steps in the outreach process that otherwise would resemble the proverbial “black box.” Although a full literature review would generate a model that would use extensive documentation, we offer this model based upon a cursory extraction of concepts from several disciplines (communications, marketing research, and health education). (Simmons, 1990; Littlejohn, 1992; Solomon, 1994; Fishbein & Ajzen, 1975; Renger, Steinfelt, & Lazarus, 2002)

One immediate issue that a research agenda would address is the following question: Is the purpose of the research program development/design or program accountability? This issue will largely determine which research tool among those in Figure 1 has priority in a schedule of efforts.

If there is a priority for accountability, then the kinds of questions for the immediate phase of analysis could include the following, for example:

1. What explicit student-level result should be a program objective? In other words, is awareness, acceptance (of message), intent, or behavior (actual receipt of financial aid) the proper level of “success?”
   
   
2. How does the program benefit the program’s target group? Does receipt of financial aid contribute to student outcomes, such as retention, persistence, transfer, employment, or certificates/degrees?
   
   
3. Does the program affect disadvantaged special populations differently from other populations? Does a financial aid program benefit individuals who have less need than individuals who have more need for aid, thus creating social inequity?
   
   
4. Are resources efficiently used in the program? Is a particular mode of program operation or a particular phase in the service process obtaining less output per dollar than some other mode or phase?
   
   
   

In contrast, if priority lies in understanding how the program operates to benefit students, then the kinds of questions to address at the start could include the following, for example:

1. Is there a sequence of states or events that promote the use of financial aid? Do students need to crystallize educational plans; experience some coursework at the community college; appreciate the long-term benefits of financial aid; and accept aid’s consequences (either as an obligation or as a grant) before they obtain financial aid?
   
   
2. What factors influence student use of financial aid beyond financial need? Do things like student aspirations, family context, job market, demographic factors, risk aversion attitude, and academic preparation moderate the likelihood of using financial aid?
   
   
3. Does a particular channel of communication work better than others? Are word-of-mouth from peers, notices in student newspapers, and campus websites equally effective in promoting the use of financial aid by target populations?
   
   
4. Does a particular message content and format work better than others? Does a simple slogan repeated often work better than a single detailed description of the financial aid process?
   
   
5. Does a particular message source work better than others? Do a college financial aid officer, a college student leader, and a state official have equal effectiveness as a message source?
   
   
6. Does a particular target audience vary on the above factors? Do immigrant populations; working students; and disabled students, among others, have different reactions to a certain type of outreach effort?
   
   
   

For each of these questions, one or more research tools may be more effective than the other tools. A research strategy should consider the matching of tools to a particular question. As a demonstration of this concept, we present one such matching of tools to a research question (what factors influence use of financial aid) in Figure 3. The process begins with a literature review because the results of this step can improve all ensuing steps. Next, we might use the case study tool to study several community college students to obtain an in-depth picture of (1) how students learn about financial aid and other college-related matters; (2) what their attitudes toward financial aid are; and (3) what their college-related behavior patterns are. The information from the case studies would then help researchers to test their theories about student beliefs and behaviors with the use of survey research and/or quasi-experiments. The survey research would help researchers estimate the relative influence of various factors, especially attitudinal ones. The quasi-experiments would go further in verifying that certain types of outreach effort work at certain levels with certain student populations. The segment for statistical modeling would involve the analysis for producing performance indicators and/or so-called adjustment performance measures (as in the analysis for persistently low transfer colleges).

A different matching of tools could easily arise, given different circumstances. For example, we present in Figure 4 an alternative plan for matching tools. If researchers can locate a good set of relevant studies (that deal with financial aid outreach), then the researchers could use a meta-analysis (or research synthesis) to obtain quantitative estimates of relationships (like the “success rate” of a practice in moving students to apply for financial aid) in lieu of the detailed data-gathering that would come from the case studies, surveys, and quasi-experiments. So the ability to use meta-analysis could save money and time, but the researchers’ access to usable studies on the topic is a necessary element here.

Conclusion

We should recognize the various research tools that could help us study student services. That step will help the system to develop a coherent and effective, multi-year strategy for developing knowledge about student services. But a system strategy for knowledge development can do more than just make appropriate use of research tools. Although the following points deserve their own paper, we briefly mention them here to help us realize that there are some other things that the system could do to enhance its research efforts in student services. These points were also made years ago for the improvement of research in clinical psychiatry. (Linden & Wen, 1998)

1. We need to understand that a piece of research, a study, gains value when we can link it to other similar studies. This argues for a system level understanding for the concepts of replication in research and of study comparability. A series of small studies that replicate an initial study can be both more feasible and more valid than a single large study. (Rosnow & Rosenthal, 1984) At the same time, the ability to link findings from different studies depends heavily upon the ability of researchers to know how similar, or dissimilar, the studies are in terms of their definitions of outcomes and their data collection/analysis protocols (such as sampling). The documentation and dissemination of research is critical.
   
   
2. We also need to understand that the research will have increased value if we can use it to help program staff in the field. This means that program staff needs to have involvement in studies even if they may lack the experience and duties of a full-time researcher. Not only can program staff help in the logistics of potential studies, they can also help researchers recognize important practical implications of theories.
   
   
3. Finally, we need to incorporate the concept of cost in the research program. Obviously, this need has two significant ramifications that have peeved educational researchers. It may divert the focus of research away from student-centered processes and towards budgeting and institutional accountability. Second, it requires the use of expertise in economics, a field that most educational researchers would not claim as their own. Nevertheless, the analysis of benefits and costs of programs is an undeniable need in the system, and the Chancellor’s Office project on the “real cost of providing quality community college education for all students” is just one effort to work in this area.
   
   

Admittedly, the subject of a long-range research plan for student services deserves much more discussion and attention than it has received in the past, or in this paper for that matter. The stakes are high today and throwing up our hands in despair (because of inadequate resources or lack of interest) will not serve the system well during the next generation.

References

Bahr, P.R., Hom, W.C., & Perry, P. [unpublished manuscript]. Low Transfer Colleges: A Methodology for Equitable Identification.

Campbell, D.T., & Stanley, J. C. (1963). Experimental and Quasi-experimental Designs for Research. Boston: Houghton-Mifflin.

Chelimsky,E. (1997). The coming transformations in evaluation. In E. Chelimsky, & W.R.Shadish. Evaluation for the 21st Century: A Handbook. (pp. 1-26). Thousand Oaks: Sage.

Cook, T. D., & Campbell, D. T. (1979). (Eds.) Quasi-Experimentation: Design & Analysis Issues for Field Settings. Boston: Houghton-Mifflin.

Cook, T.D., et al. (1992). Meta-Analysis for Explanation. New York: Russell Sage.

Cook, T. D. (1997). Lessons learned in evaluation over the past 25 years. In E. Chelimsky, & W.R.Shadish. Evaluation for the 21st Century: A Handbook. (pp. 30-52). Thousand Oaks: Sage.

Cooper, H., & Hedges, L.V. (Eds.). (1994). The Handbook of Research Synthesis. New York: Russell Sage.

Cresswell, J.W. (1998). Mixed-method research: introduction and application. In G.J. Cizek. (Ed.) Handbook of Educational Policy (pp.455-472). San Diego: Academic Press.

Everitt, B.S. (2002). The Cambridge Dictionary of Statistics, Second Edition. Cambridge, UK: Cambridge University Press.

Fishbein, M., & Ajzen, I. )1975). Belief, Attitude, Intention and Behavior: An Introduction to Theory and Research. Reading, Massachusetts: Addison-Wesley.

Linden, W. & Wen, F. K. (1998). Therapy outcome research, health care policy, and the continuing lack of accumulated knowledge. In A.E. Kazdin. Methodological Issues & Strategies in Clinical Research, Second Edition (pp.561-575). Washington, D.C.: American Psychological Association.

Littlejohn, S.W. (1992). Theories of Human Communication. Belmont, California: Wadsworth.

Neuman, W. L. (2000). Social Research Methods: Qualitative and Quantitative Approaches, Fourth Ed. Boston: Allyn & Bacon.

Renger, R., Steinfelt, V., & Lazarus, S. (2002). Assessing the effectiveness of a community-based media campaign targeting physical inactivity. Family Community Health, 25, 18-30.

Rosnow, R. L., & Rosenthal, R. (1984). Understanding Behavorial Science. New York: McGraw-Hill.

Rossi, P. H., & Freeman, H.E. (1993). Evaluation: A Systematic Approach, Fifth Ed. Thousand Oaks: Sage.

Rossi, P.H.,Wright, J.D., & Anderson, A.B. (1983). Sample surveys: history, current practice, and future prospects. In P.H. Rossi, J.D.Wright, & A.B. Anderson. (Eds.) Handbook of Survey Research(pp. 1-20). New York: Academic Press.

Simmons, R. E. (1990). Communication Campaign Management: A Systems Approach. New York: Longman.

Solomon, M.R. (1994). Consumer Behavior: Buying, Having, and Being. Boston: Allyn & Bacon.

Vogt, W. P. (1993). Dictionary of Statistics and Methodology: A Nontechnical Guide for the Social Sciences. Thousand Oaks: Sage.

Weiss, C.H. (1998). Evaluation:Methods for Studying Programs and Policies. Upper Saddle River, New Jersey: Prentice-Hall.

Yin, R.K. (1989). Case Study Research: Design and Methods. Thousand Oaks: Sage