|
A comprehensive assessment of the effects of performance-based funding is not yet available. Yet according to early national survey results, the impact of performance-based funding on overall campus outcomes has been moderate at best (Burke and Minassians, 2002). These national findings may mirror experiences within California. In 1998, the State of California and the California Community Colleges inaugurated a performance-based funding program called the Partnership for Excellence (PFE). The program launch represented a commitment by the state legislature to earmark additional funding for the community colleges, in exchange for the colleges’ agreement to develop, track, and achieve, by 2005, system-wide performance goals to improve student learning and success.
Through 2000-01, the state appropriation to the community colleges for the PFE program was allocated by the Chancellor’s Office to local college districts based on enrollment. There were conditions in place for moving from an enrollment-based to a more performance-driven formula, but before those conditions were fully triggered, the program was not re-funded. The Legislative Analyst’s Office issued an analysis that was critical of the effects of the PFE program on system performance in the community colleges. In its analysis of the 2002-03 state budget, it stated that the program was “failing to meet objectives” and that the measurement of results were “hindered by methodological disagreement and conceptual vagueness” (Legislative Analyst’s Office, 2002).
During the time period when the Partnership for Excellence (PFE) program was still being funded by the Legislature, the Institute for the Study of Knowledge Management in Education (ISKME) conducted two district-wide studies, representing eleven colleges, that examined the needs and patterns of data and information access, sharing, and use in two community college districts in California. As part of these studies, a wide range of administrators, faculty, and staff were interviewed about their use of information in decision-making on campus. Given the state’s continuing interest in developing an accountability system for the California Community Colleges (Office of the Chancellor, 2005), these studies provide illuminating insights into the complex relationship between external mandates for accountability and internal practices of information and knowledge retrieval, use, and management at the district level.
Research Setting
The first study was a three-year research project that began in 2000 at a community college district in California—a large, multi-campus, suburban district with a combined enrollment of over 40,000 students. The purpose of the study was to examine how administrators, faculty, and staff used student outcome data for decision-making for administrative purposes (such as enrollment management) as well as for improvements in teaching and learning. During this time, over 70 interviews with a cross-section of administrators, faculty, and staff involved at various levels of organizational decision-making were conducted. Critical incident interviews focused on organizational structures and culture that supported data and information gathering, analysis, and sharing throughout the district. A data use survey was also completed by 220 administrators, faculty, and staff.
The second study took place in early 2002, and was conducted at a large, urban community college district serving over 95,000 students annually. In 1998, this district had invested in a new information system that gave individuals the ability to access data from their desktops via a new college intranet. In 2001, the institutional research office implemented a new decision support system (DSS), a web-based interface that linked directly to a data warehouse that contained student data from spring 1998 to the present. The purpose of this new system was to provide a more streamlined process for campus users to directly access student data, without having to request data from the institutional research office. This second study, which focused on the shift in data retrieval since the DSS had been introduced in 2001, was based on a series of in-depth critical incident interviews with 27 administrators, faculty, and staff who had used the district’s DSS between June 2001 and February 2002.
Key Findings
The following findings, drawn from the studies cited above, are pertinent in considering efforts—both internal and external—for improving information use within the California Community Colleges.
1. External accountability mandates do not necessarily create incentives for improved data use. How data are used in an environment of accountability—such as using data to explore program improvements as opposed to using data to justify cutbacks—can affect people’s willingness to participate in data gathering and analysis.
The first study found that perceptions about the way data had historically been used had a significant effect on individuals’ willingness to collect, share, and use data in decision-making. For example, during the three years of this study, it was observed that the district was actively engaged in continuous learning efforts geared toward the improvement of programs and services for students. There was much evidence that despite the significant shortcomings of the central information system, people had managed to create an active culture of internal research and inquiry, a culture that asked complex questions about student needs and that explored better ways to meet those needs.
However, even in this environment that supported continuous learning efforts, there was evidence that externally mandated performance measures (i.e., in response to the Partnership for Excellence program) created an evaluative environment that reinforced non-cooperative information-sharing behaviors among individuals (Petrides, McClelland, and Nodine, 2004b). This was observed even though district and college leadership sought to implement the Partnership for Excellence (PFE) program in a positive way as an incentive-based program. For example, campus leaders conducted a series of meetings with administrators, faculty, and staff to inform them about the goals of the PFE initiative and to set forth a plan for how the district would respond to the new state mandates. The district developed a proposal process through which campus programs could apply for the new funding. This process required programs to set measurable goals for student learning outcomes and to design evaluations to measure whether those outcomes had been reached. The district also provided training to assist administrators, faculty, and staff in developing effective evaluation methods for their programs.
Many people publicly voiced support for efforts to set up evaluative frameworks for their PFE-funded programs. However, the research found that many of those who publicly voiced their commitment to improved outcomes also sought to buffer themselves from the possible consequences of the emerging internal evaluative environment. This ranged from engaging in rationalizing behaviors to deflect attention from their own program’s possible substandard performance, to resisting attempts to improve the institutional research function on campus. Publicly criticizing the internal research function, while at the same time impeding efforts to improve it, served those who sought to prevent internal efforts to meet the external accountability requirements.
These efforts found allies in those whose own power was enhanced by maintaining limited access to data—and this may be where the lasting, unforeseen damage of the PFE mandates lay for this community college district. There were those who resisted the creation of a new campus-wide integrated information system that could have improved access to data. In addition, during our research, the restructuring of the institutional research office was redefined and renegotiated by those who stood to directly or indirectly lose control of data and information that the new structure would impose. The irony is that these types of self-serving behaviors are perhaps most common in a climate of external pressure for accountability.
2. Where there are information bottlenecks on campus, there may also be “workarounds.”
In the first study, we also found that one possible consequence of inadequate information flow on campus was that enterprising individuals devised methods to “work around” their lack of data by creating or participating in idiosyncratic methods of data collection or management. These informal practices—called workarounds—can include a wide range of low-tech and high-tech solutions. Our study found three primary kinds of workarounds:
1. Manual data collection: Participants manually gathered data, physically delving into day-to-day operations of programs and departments.
2. Manual data manipulation: People described efforts to manipulate or re-key in data that had already been gathered by the central information system, so that they could use it for their own needs.
3. Local database creation: People routinely created, maintained, and used local databases because they could not access data or manipulate data in any other way. These databases, found throughout the institution, consisted of data that had been compiled over several quarters or even years, though often in inconsistent ways. Examples: creating a database to track specific cohorts of students over time; creating a database to enable comparisons of fiscal data and student outcomes; creating a database to examine persistence rates associated with curricular revisions.
We found that more than half of the survey respondents in this district reported that they participated in localized efforts for gathering or compiling data that were consistent with workarounds. No particular group—administrators, faculty, or staff—relied on workarounds more so than others.
Community colleges across California are at various stages of information system development in seeking to support data and information analysis within the district. The presence of workarounds within a college district may offer a promising glimpse of where some of the key pockets of innovation and self-reflection exist. It is reasonable to suggest that those employees who have not created workarounds may indeed be willing to engage in self-reflective organizational improvement through effective use of data and information, if given the organizational support to do so. Those who have employed workarounds appear to be willing, at least for a limited time, to engage in such efforts without organizational support. For organizations interested in creating a culture of inquiry on campus, the presence of employees who are already engaged in workarounds suggests that this might be a promising group of individuals to get on board.
The existence of workarounds can also reveal where information technology gaps may exist on campus (Petrides, McClelland and Nodine, 2004a). In higher education environments where there are internal and external pressures for organizational improvement, there are many temporary and local needs that cannot and should not be met by formal centralized information systems, primarily because it is more cost-effective and flexible to meet these kinds of needs at the program level rather than with a college-wide solution. Workarounds will always, to some extent and for a limited time, be necessary within a vibrant, self-reflective organization. However, they can also serve to highlight where the gaps in information flow exist—as was the case in this first study. As colleges and universities work toward improving their information systems, the existing clusters of workarounds may suggest promising areas of attention.
3. Even within a district that is actively seeking to improve data access and use, significant barriers can prevent people from changing their practice.
The district in the second study we conducted was seeking to improve the access of administrators, faculty, and staff to data and information through the development of a new decision support system (DSS). We found that some of the primary barriers that hindered widespread use of the new system stemmed from perceptions and attitudes based on past practices that impeded current efforts. For example:
- Data were sometimes perceived as threatening. Some research participants spoke about their mistrust of data and the “hidden agenda” implicit in data gathering efforts in the past. One of the hidden agendas that participants spoke about was how data could be used as a mechanism to “spy” on the performance of others and to discipline faculty.
- Data were perceived as inherently unreliable. Even though “poor data” appeared to be largely a thing of the past, there was an institutional mindset, based on a historical legacy, that data were not to be trusted. For example, participants described tendencies to support data use when the data were in line with their perceptions, and to point out the flaws in the data and possible errors in the results when the data were not in line with their point of view. Some participants suggested that the historical legacy around data reliability exerted so much influence that there was little hope of breaking through this barrier, even with perfect and persuasive data, while others suggested that this could be changed over time.
Other barriers focused on issues related to technology and information system use:
- Concerns and fears about technology and “numbers.” Many participants expressed concerns about their own ability—or the ability of their colleagues—to use data, numbers, and technology effectively. People mentioned that those they worked with did not see the potential of the new decision support system because they had not had much experience with data-driven forms of inquiry. There appeared to be a generational gap with regard to technological dexterity—and there was a widespread perception that this was the case.
- Information overload. Several people mentioned that they were already overloaded by information, and that they did not have adequate tools or training to sift through even the most basic things, such as email. Without a system in place to “weed through” the existing information, access to even more data was perceived by many as an undesirable overload.
4. The information culture within each district has implicit incentives and disincentives for data use. People’s use of data and information in decision-making may be significantly altered by whether or not vice presidents, managers, deans, senior faculty, coordinators, and other campus leaders support processes for data gathering and analysis.
Technological systems and the organizational processes associated with the internal research function are embedded within an organizational structure and culture that influence data use and information sharing in decision-making. Therefore, understanding and improving information use on campus requires looking beyond formal information systems and research offices, to also emphasize the less formal organizational structures and values that create incentives and disincentives for data use.
For example, survey results from the first study indicated that those who were encouraged to use data in their job were more likely to do so. Nearly three-quarters of the respondents (74 percent) who reported that they used data “always” or “frequently” agreed or strongly agreed that “I am encouraged to use data in my job” (see Figure 1). On the other hand, less than half (41 percent) of those who said they used data “occasionally” agreed or strongly agreed that “I am encouraged to use data in my job.” And only one-quarter (24 percent) of those who said that they “rarely” or “never” used data agreed or strongly agreed that they were encouraged to use data.
These findings are significant in dispelling the notion that data use is solely dependent on such issues as hardware availability, an individual’s technical ability, or an individual’s attitudes about technology. Data use may be affected by these issues as well, but use appears to also be shaped by the individual’s environment. If this is the case, then data use can be encouraged through effective motivation and support in the workplace.
We also found differences in the way that administrators and faculty were supported in regard to data use for decision-making. For example, we found that 64 percent of all respondents agreed or strongly agreed that “I am encouraged to use data in my job,” while 10 percent disagreed or strongly disagreed with this statement (see Figure 2). However, whereas 82 percent of administrators agreed or strongly agreed with this statement, only 41 percent of faculty members did so.
These differences between administrators and faculty members may reflect the fact that administrators have long been required to collect and analyze data as part of their work, while faculty involvement in demographic, fiscal, and other data outside the classroom may be a newer phenomenon stemming, at least in part, from increased calls for greater accountability in higher education. Whatever the causes of these differences, however, faculty are, in fact, being asked to become more involved in these kinds of issues, and it appears from our research that many are willing to do so. However, it appears that many are not being encouraged and supported in their use of data and information in decision-making.
Policy Implications
Although the research summarized in this paper represents only two of the seventy-two districts in California, and is therefore limited in scope, it suggests that promoting effective use of data within community college districts may be enhanced by understanding informal processes, attitudes, and perceptions within the districts, and taking action to promote those patterns and behaviors associated with more effective information use. These kinds of management actions can be taken without the purchase or development of new technological systems, though such developments may further enhance access to and use of data.
The studies also suggest that the creation of external mandates for performance-based information may create behavior patterns that exacerbate, rather than break down, the barriers of effective information use on campus. Further studies need to be made to develop a more extensive topography of how external demands for information affect decision-making within community colleges, particularly in relation to student success. The accountability mechanism for the California Community Colleges currently under discussion as a result of the requirements in AB 1417 presents an exciting possibility for more extensive study of this issue.
In the meantime, for those interested in improving the use of data and information for decision-making with the community colleges, the following implications arise from these two studies.
Implications for Those at Community College Districts:
1. Understand the barriers to and history of data use within the district, and create ways to break through those barriers. Each district already has a context and history of perceptions about data collection, dissemination, and use, and these can vary by position within the organization. It is important to know what these perceptions are, to understand the extent to which these perceptions are accurate or out of date, and to take steps to train staff members to collect and use data effectively in their jobs. An information audit can help to pinpoint the factors that are preventing effective information access or use on campus: Are the factors related to fears about technology? Are they grounded in past practices about how data have been used? Are they based on current limitations in the information system? Knowing which factors are at play will affect the kinds of training, communications activities, or implementations that are needed.
2. Establish a clear tradition of and processes for using information in decision-making. Organizational leaders that create policies and processes that value the use of data in decision-making instill and support a culture of inquiry throughout the institution. For example, do deans and upper management ask for data support for budget requests? Are there institutional policies that support the sharing of data and information? Does upper management support efforts to investigate ways to improve student achievement, or does it seek to penalize programs that may not be meeting objectives? Are faculty, staff, and administrators engaged in efforts to improve the accuracy of data? These are the kinds of everyday practices that can encourage—or discourage—the creation and support of a culture of inquiry on campus.
3. Identify those who are using data and information in decision-making, get them on board in overall improvement efforts, and encourage them to share what they know. At many colleges, there already are faculty, staff, and administrators with reputations for gathering and using information to improve teaching or other services to students. For those institutions committed to improving student achievement, it is important to support the work of these individuals, enlist them as allies, and bring them together to formalize their knowledge processes and resources. In this way, existing workarounds can be identified and used to help plan improvements in information systems and information flow—with the overall aim of understanding and improving student success.
4. Respond to external mandates in proactive internal ways. More research needs to be conducted to understand the complex ways that individuals react within an evaluative climate. It is appears that for some colleges, external accountability mandates can be used as leverage to assist in moving the institution forward in adopting performance-driven practices. Responding to accountability requirements in proactive internal ways requires, first, assessing the appropriateness of the external demands and reinterpreting them in terms of the organization’s mission and goals, across all levels of the organization. The second step in the process, which is often overlooked, requires the organization to internalize the incentive structure for fulfilling the new demands. Moving forward proactively can include, for instance: generating and seeking support for the internal as well as external reasons why a new process or initiative should take place; providing training focused on new ways to gather data and discuss findings; and insisting upon accessing and understanding the outcomes internally. Knowing how far and how quickly to press for change in the use of performance data within the organization, and understanding how to use external mandates to assist in this effort at a specific college, requires already having made some progress in the steps listed above.
Implications for State Policymakers
Over the last several years, the California State Legislature has continued to examine new approaches for establishing an accountability mechanism for the community colleges. Most recently, the state legislature, through AB 1417, has been considering an approach for establishing an accountability mechanism for the community colleges. The California Community College Chancellor’s Office has been developing recommendations for the Legislature and the Governor for “a workable structure of the annual evaluation of district-level performance in meeting statewide educational outcome priorities”(Office of the Chancellor, 2005). The recommendations, which are to include a district-level performance evaluation system, are due to be submitted to the Legislature and Governor in early 2005. In light of these efforts, the following are recommended:
1. Create real and on-going opportunities for bottom-up involvement in the development and modification of a statewide accountability system. It is important for external oversight bodies to provide opportunities for colleges or districts to participate in the creation of performance-based incentives or other mandates. Because each district often has its own groups or individuals working to improve student success and organizational effectiveness, it may be that the most effective performance-based measures serve to empower those already on campus who are working for change. The least effective measures, conversely, may be those that create a “circling the wagons” attitude on campus, whereby even those who had been working for change perceive the state mandates as so threatening that they align with those who are seeking to maintain the status quo.
2. Be precise about outcome measurement. At the same time that state leaders and other external bodies are involving colleges in the process of developing and modifying accountability systems, state leaders should insist on agreement at a level of specificity that can adequately address the expected overall goals. From the perspective of the Legislative Analyst’s Office, one of the key problems the PFE program was that the measurement of results was “hindered by methodological disagreement and conceptual vagueness.” It is important to attain agreement not just on general goals, but on specific measurement indicators that can stand as baseline measures for progress toward more general goals.
3. Support a wide range of means to increase access and use of data on campus. The community colleges are diverse in the development of their information policies and systems and their proclivity for and sophistication in examining the effectiveness of their own practices. In developing opportunities for bottom-up involvement in the creation of a statewide system of accountability, the state can at the same time explore ways to establish a baseline for the kinds of access to data and information that should be available within districts. For some colleges, this might mean increased training and professional development opportunities, or increased support for supporting information systems. Perhaps the most effective way to achieve greater accountability in higher education is to empower stakeholders on and off campus to have greater access to data and information regarding multiple measures of student achievement.
Conclusion
There are growing national and statewide trends that shed light on new ways to develop and understand how we might better assess the performance of higher education. There is no doubt that colleges and universities, as institutions of public trust, are increasingly being held accountable for their outcomes. However, it is unclear what the overall effect of this trend is on the institutions themselves and the students they serve. Our research has shown that institutions on the front line, such as community colleges, are affected by external drivers in ways that are not yet fully understood. Therefore, more research is needed to develop a topography for understanding how external demand for information affects decision-making within community colleges, particularly how it relates to student success. What we do know, however, is that an organization’s internal culture and processes affect decision-making patterns within it. As a result, there are steps that those within the community colleges and those at the state level can take to enhance the use of data and information to improve practices and policies that will ultimately contribute to improved performance within community colleges.
References
Burke, J.C., and Minassians, H.P. (December, 2002). “Reporting Higher Education Results: Missing Links in Performance.” New Directions in Institutional Research, Vol. 116. San Francisco: Jossey-Bass.
Legislative Analyst’s Office. (2002). California State Assembly. Analysis of the 2002-03 Budget Bill. Available at http://www.lao.ca.gov/analysis_2002/education/ed_30_6870_anl02.htm (visited February 24, 2005).
Office of the Chancellor, California Community Colleges (2005). “Draft #3.2: Performance Frameworks for California Community College Districts.” Research and Planning Group, Center for Student Success Panel, January 19, 2005. Available at: http://www.cccco.edu/divisions/tris/accountability/accountability.htm (visited February 21, 2005).
Petrides, L., McClelland, S., and Nodine, T. (2004a). “Costs and Benefits of the Workaround: Inventive Solution or Costly Alternative?” Educational Management, (18)2, pp.100-108.
Petrides, L., McClelland, S., and Nodine, T. (2004b). “Using External Accountability Mandates to Create Internal Change.” Planning for Higher Education, (33)1, pp. 44-50.
Wells, J., Silk, E., & Torres, D. (1999). “Accountability, Technology, and External Access to Information: Implications for IR.” In L. Sanders (Ed.), How Technology is Changing Institutional Research (pp. 23–29). New Directions for Institutional Research, Vol.103. San Francisco: Jossey-Bass.
.
|
