What the Russian Survey of Performance of Higher Education Institutions Actually Assessed

Annual Survey of Performance of Higher Education Institutions, conducted in Russia since 2012, was used to identify “inefficient” universities, subject to measures including closure or merging with other schools. As a result of these policies, the number of Russian universities has decreased more than 1.5 times since 2013. In this article, we analyze the consequences of implementing the appraisal system invented by the Russian Ministry. We argue that the use of the Survey reflects a conceptual confusion between effectiveness (the organization’s ability to achieve socially significant goals) and efficiency (the ability to achieve goals with minimal cost). The Ministry has made managerial decisions based on the Survey results (like merging a public university with an allegedly better performing HEI), which indicates understanding of the Survey as an assessment of efficiency. At the same time, statistical analysis of the performance indicators demonstrates that structural characteristics of universities (region and belonging to an institutionalized category, e. g. a pedagogical or an agrarian university) explain the significant part of the variance in the university’s performance. This led to discrimination against certain “unlucky” categories of universities. The methods used include logistic regression to estimate the odds of being labeled as an efficient organization in 2014 and the Cox proportional hazards model to estimate the university’s chances of survival between 2013 and 2017.

(the organization's ability to achieve socially significant goals) and efficiency (the ability to achieve goals with minimal cost). The Ministry has made managerial decisions based on the Survey results (like merging a public university with an allegedly better performing HEI), which indicates understanding of the Survey as an assessment of efficiency. At the same time, statistical analysis of the performance indicators demonstrates that structural characteristics of universities (region and belonging to an institutionalized category, e. g. a pedagogical or an agrarian university) explain the significant part of the variance in the university's performance. This led to discrimination against certain "unlucky" categories of universities. The methods used include logistic regression to estimate the odds of being labeled as an efficient organization in 2014 and the Cox proportional hazards model to estimate the university's chances of survival between 2013 and 2017. Keywords: performance assessment, organizational efficiency, higher education in Russia, Survey of Performance of Higher Education Institutions. DOI: 10.17323/1814-9545-2020-1-97-125 The number of Russian universities decreased more than 1.5 times between 2012 and 2017 (from 2,130 to 1,314), and that of branch cam-puses reduced by half, from 1,229 to 5831. Most often, the universities that disappear either have their licenses revoked as a result of an unscheduled inspection by the Federal Service for Supervision in Education and Science (Rosobrnadzor) (in case they are private) or merged with other institutions (in case they are public). What can be described, in terms of the population ecology of organizations [Hannan, Freeman 1977] as "organizational extinction" [Marion, Bacon 1999] has been an outcome of deliberate policy aimed at reducing the number of universities offering ersatz education. The Survey of Performance of Higher Education Institutions was supposed to become the main tool to identify such universities. The Survey was designed as a guidance for "optimizing the university network" as it was explicitly stated in a governmental resolution elaborating the Presidential decrees of May 20122. This article is an attempt to find out how well the survey actually solved the task it had been supposed to solve.
Our main thesis is that a conceptual error was committed by the survey designers, who confused effectiveness with efficiency3. As a result, while being a relatively meaningful, though questionable, measure of effectiveness, the Survey was used as a measure of efficiencyinadequately so, as it measured "ascriptive" competitive advantages of universities instead. Below, we are going to show that rejection of "inefficient" universities on the basis of Survey results was in fact the policy of punishing the unlucky institutions which had no required "ascriptive" properties.
In organizational analysis, effectiveness is doing the right things, measured by the quality of output. Efficiency is doing things right. An organization producing more with the same costs is considered more efficient [Cameron, 1983;Witte, López-Torres, 2017]. Many successful organizations are efficient and effective at the same time, but this is far from always being the case [Ostroff, Schmitt 1993]. Universities' efficiency has been studied by education economists focusing on the methods and statistical analysis as a measurement instrument4. Ef- 3 TN: Both "efficiency" and "effectiveness" are translated into Russian as эффективность, which is the word used in the name of the Survey.
4 Data envelopment analysis (DEA) is the most popular method of studying universities' efficiency. For a review of input-and output-oriented models used in different countries and a description of two possible models for Russia, see [Abankina et al. 2013]. Other studies in this field explore the impact of the excellence initiative "5-100 Project" on universities' efficiency [Agasisti et al. 2018b] and the impact of universities on regional economic growth [Agasisti et al. 2018a].

Effectiveness vs. Efficiency
fectiveness has been explored within the framework of organizational theory [Cameron 1986]. Few studies investigate both efficiency and effectiveness [Powell, Gilleland, Pearson 2012;Ostroff, Schmitt 1993].
Depending on whether an instrument should measure efficiency or effectiveness, it is designed to include inputs or outputs. Inputs are resources that are put into the university, such as average score of admitted applicants, tuition fees, or publications of newly-recruited researchers. Outputs are outcomes of the university's performance, such as graduate employment rate or publications per faculty member. It is not always possible to distinguish between the input and output indicators. Even the most straightforward metrics appear to be misleading: the quality of students is sometimes treated as an input, but the ability to attract better students may be regarded as an outcome of effective recruiting, which makes it an output [Edvardsen, Førsund, Kittelsen 2017].
Meanwhile, distinguishing between inputs and outputs and using both types of indicators in analysis is critical for measuring efficiency, as outputs alone are not enough to evaluate the university performance. Let us say, we have measured average salaries of university graduates. Can this data be used to assess university's efficiency? No. The university might have recruited gifted students who took care of their education themselves, being left to their own devices5. Where inputs are hard to measure directly, they can be measured through comparison with other universities in the same category.
Measurement of effectiveness relies more on outputs, as they allow to see whether an organization fulfills its mission. Inputs may be used as well, but only as an indirect indicator of outputs. Say, being unable to assess the quality of university research, an assessor will turn to R&D spending, reasoning that good research costs a lot, so cost-intensive research is more likely to be of high quality.
Application of measurement results is also determined by whether efficiency or effectiveness is assessed. Low efficiency requires punitive measures against administrators misusing the funds. Low effectiveness, however, does not imply finding a scapegoat, but it rather provides an opportunity to identify the centers of excellence to which heavier investments should be directed. Academic Ranking of World Universities, also known as Shanghai Ranking, is an example of a purely effectiveness-oriented instrument that uses the number of articles published and cited as an indicator but does not control for faculty size, thus ignoring average productivity [Kincharova 2014]. A ranking designed this way detects the most important centers of ex-5 A review of studies showing that the fact of graduation from a top U.S. university adds nearly nothing to starting salaries, as Harvard recruits the best of the best who will succeed in life anyway, is presented in [Gerber, Cheung 2008]. cellence but does not show whether their success results from efficient use of the resources at hand.
The process of developing the Survey of Performance of Higher Education Institutions is shrouded in mystery. As far as we know, the names of its authors have never been publicized. Consequently, it remains unclear whether it was intended to measure efficiency or effectiveness. The internal logic of the survey can be analyzed by examining its structure and methods of implementation -but these two avenues lead to contradictory inferences. The way the Survey results were actually used indicates that it had been designed as a measure of efficiency. Indeed, merging with an allegedly better managed institution was the most widespread response to low university performancewhich is only reasonable if low performance is interpreted as a result of poor management6. However, the next section demonstrates that structurally, the Survey consists predominantly of input-based characteristics, and thus could measure only effectiveness.
The numbers of organizations that participated in the Survey of Performance of Higher Education Institutions are given in Table 1. The number of participating universities differs from year to year, mainly due to organizational mortality. For instance, mergers affected 18 universities in 20147, 40 in 2015, 28 in 2016, and only 10 in 2017. Meanwhile, not all the inefficient universities were closed or reorganized. Some of them were left out from the Survey for a year and then materialized again.
The Survey collects information on a few dozens of indicators, grouped into eight categories, (1) education, (2) research and development, (3) international activities, (4) financial and economic activities, (5) infrastructure, (6) employment, (7) faculty, and (8) additional characteristics. Each group is represented by one key indicator (changes in the number of variables within each group from year to year are reflected in Table 2). The key indicators are not integrated indexes; they are based on a single variable from the relevant group. Mean USE (Unified State Exam) score of enrolled applicants, for example, is the key indicator of education8. Somewhat unexpectedly, the indicator of research and development does not make allowance for publication productivity and only considers R&D spending per faculty member (R&D and creative project spending in art schools). Financial and economic activities are described as revenue from all sources per faculty member, international activities as the percentage of inter-6 In [Guba, Zavadskaya 2017], it is shown that dissolution has not always followed "inefficiency" automatically; below, however, we are going to see that organizations labelled as inefficient had fewer chances of survival. 7 The number of surviving entities. 8 Art schools also take into account creative competition results, and military and sports universities consider the results of aptitude/fitness tests.

How the Survey Is
Designed national students, infrastructure as total floor area of laboratories per student, and employment as the percentage of students who did not apply for employment assistance. Additional indicators are estimated depending on university specialization. For non-specialized universities (all except art, sports, and military schools), the additional indicator is defined as the population of doctoral faculty per 100 students.
Most key indicators do not describe what could be unequivocally regarded as independent achievements, i. e. output-or outcome-based variables assessing the university's contribution to science, regional and national wellbeing. Exceptions include the number of international students, the attraction of which can be valued as a form of education export, and graduate employment (however, there is considerable doubt about assessment validity here). The Survey relies rather on inputs, which is always fraught with the risk of (i) overestimating the organizations that accumulate a lot of resources but cannot use them efficiently, (ii) overestimating the resources that have, in effect, low significance for organization success (is lab floor area really key to achieving the university's mission?), and (iii) (in case it was designed to measure efficiency) failing to distinguish between the university's own achievements and what came as a result of enjoying a favorable set of external parameters. Indeed, by the time the ineffi-  cient university shutdown initiative was launched, Russian universities had had a long evolutionary history and occupied certain niches that had not been chosen by the then effective administrators. Moreover, those niches were largely predetermined by universities' "ascriptive" characteristics -similar to those ascribed to individuals -that affected their status in university stratification [Sokolov 2017].
The past decade has seen ample literature on the factors affecting the trajectories of university development [Ramsden 1999;Warning 2004;Shin 2009;Zhang, Patton, Kenney 2013;Cataneo, Meoli, Signori 2016;Boliver 2015;Gómez et al. 2009]. Most of the studies examine the influence of various characteristics on university research performance, while paying little attention to the education aspect. Accordingly, researchers mostly use bibliometric indicators as dependent variables. Table 3 presents a few studies, specifying the country of origin and the factors that they regard as determinants of university performance. As seen from Table 3, researchers used different approaches to university classification. However, the majority of the studies use, in some form or other, geographic and economic determinants as well as the effects of various national initiatives9. Let us dwell on the studies of Italian universities as an example. How do structural determinants affect (if at all) performance indicators? Significance of the historic economic divide between northern and southern Italy was tested in [Mateos-González, Boliver 2019]. The authors suggest analyzing the structural predictors of organizational performance in the way that sociologists analyze the effects of economic, social, and cultural capital on individual achievement. The Italian system of higher education turned out to be an important case for structural determinant analysis, as the authors managed to demonstrate the impact of the regional factor, reflecting the difference in the socioeconomic status between northern and southern universities, on their research and education performance. For instance, it was found that students in northern Italy tend to sign up for more courses and spend less time on completing their programs, northern universities have a higher doctoral student enrollment, and publications of professors affiliated with northern universities are more likely to be found in international journals and cited.
Mattia Cattaneo and his colleagues [Cattaneo, Meoli, Signori 2016] investigate the impact of university characteristics on the number of publications produced by the faculty. As performance-based funding was introduced, an increase in the number of publications 9 In those countries, just as in Russia, universities are subject to a number of national excellence initiatives (see the far-right column of Table 3). When analyzing university performance, researchers use data obtained as a result of those initiatives, which is also true for this article.

Trajectory of University Development
was more perceptible in the more popular universities -probably because it did not require much effort from them. Popularity was measured as a number of mass media articles mentioning the university. As in the previous study, universities in northern Italy were found to be more productive. In addition, big universities produced more publications, whereas private schools were less research-oriented. A research team from the National Research University Higher School of Economics puts forward a few assumptions on how belonging to a certain "family" (institutionalized category) of Soviet universities (pedagogical, (poly)technic, etc. -see below) determined the choice of a specific trajectory of development [Kuzminov, Semenov,  Froumin 2013]. Another study links the ability of universities in a certain region to attract high school graduates to the region's migration attractiveness but never brings analysis to the level of individual universities [Abankina, Abankina, Filatova 2016]. One of the authors of the present study [Sokolov 2017] attempted to use logistic regression to evaluate significance of the "family" factor for the emergence of one of the types of university economies.
If we ignore such contextual factors, we will be unable to explain why this or that university occupied a certain niche in the ecology of higher education and, in particular, to what extent it used all of its growth opportunities. Consequently, we will be unable to say how efficiently it was managed. For instance, to assess the progress of an institution on its way to becoming a leading research university, it is not only managerial decisions that should be taken into account but also a number of objective "innate" characteristics that define the university's competitive status. Such characteristics include being located in a large city with a stimulating academic environment, region's wealth, university's age, specialization ensuring an inflow of financially reliable students and/or connections with growing industries, and being a historical monopolist in the local market. If the role of those factors in university performance is not considered, performance-based distribution of funding will promote further polarization in higher education [Abankina, Abankina, Filatova 2016;Talovskaya, Lisyutkin 2018] as well as degradation of institutions that lack the "innate" characteristics required to be high performers.
It was not that the Ministry did not see the pitfalls of using the Survey as a metric of efficiency. Allowance being made for the influence of university size on simple quantitative performance indicators, which had been proven for U.S. universities [Dundar, Lewis 1998], most indicators were standardized by being divided by the number of faculty members10. A specific indicator of "weighted enrollment" was used instead of employment for branch campuses. Besides, the baselines used by the Ministry to measure efficiency along the key indicators change as a function of geographic location (e. g. education baseline values differ between St. Petersburg and Moscow, and employment baselines were estimated for federal districts). However, the method of identifying those baselines is not transparent, so the justification behind them remains unclear. Meanwhile, their effects are rather dubious, as we are going to see below.
10 In this regard, the Survey designers were not too inventive in developing the key indicators and followed the established procedure. However, different approaches are possible. For example, [Calabrese et al. 2018] suggest using a "power law"-based method instead of linear dependence on size. Using the case of Italian universities, they demonstrate that this approach to performance indicator design reduces the effect of size-related bias. Now, we are going to assess the extent to which performance indicators of individual universities in the Survey could be explained by efficient management or the effects of structural factors that made some key resources more or less accessible. We can analyze to what extent the assessment of university management efficiency was biased by the Survey's inability to control for the ascriptive factorshence, it will become clear how good the Survey-based purging campaign was at detecting actually worse-performing institutions.
The study is based on an analysis of key performance indicators of the universities participating in the Survey11 in 2014. The number of participants was the highest that year, and certain subpopulations had not been reduced yet as a result of mergers and acquisitions. That is why the 2014 data is the most suitable for describing the institutional ecology of post-Soviet universities in all its diversity12. Originally, the sample consisted of 1,801 universities13, 822 main campuses and 979 branch campuses. However, primary data analysis revealed statistically significant differences between main and branch campuses. For each indicator, less than half of the branch campuses were found to be efficient, so only one quarter of branch campuses were ranked as efficient overall (Table 4). At the same time, the impact of some independent variables could not be assessed for branch campuses (in particular, some "families" had no branch campuses at all). Obviously, it would be reasonable to analyze main and branch campuses separately. In this article, we are zeroing in on main campuses.
The variables used in the Survey to rank universities as efficient or inefficient (described above) were used in this study as dependent variables. We also used overall efficiency as an aggregate of all the indicators (a university had to score above the baseline in any four indicators to be acknowledged as efficient). Independent variables were mostly ascriptive characteristics of an organization, i. e. the properties that it could not get rid of and that determined the amount of resources available. Two groups of those properties, geographic location and 11 Data was obtained from the website of the Main Data Processing Center of the Ministry of Education and Science, which contains detailed information on every university including branch campuses. The information requested was submitted by universities via a special form called "Monitoring in the Core Activities of Higher Education Institutions (Form No. 1-Monitoring)." 12 There is a 1.1% disagreement between the number of main campuses listed as participants in the 2014 Survey and the data from the Regions of Russia statistical bulletin.
13 We excluded universities under reorganization (no performance indicators were available for them) and those which recently absorbed other schools, as mergers could have affected their performance; therefore, the final sample is smaller than the number of universities covered by the 2014 Survey.

Variables
http://vo.hse.ru/en/ K. Guba, M. Sokolov, A. Tsivinskaya Fictitious Efficiency belonging to a "family", have been widely discussed in literature on higher education.
Geographic location is naturally an ascriptive variable -Adyghe State University would find it difficult to move from the Republic of Adyghe. The concept of "geographic location as an ascriptive variable" also involves region-specific characteristics. We used the socioeconomic development taxonomy of subjects of the Russian Federation [Federation Council of the Federal Assembly of Russia 2007] to classify regions depending on their relative wealth, economic structure, and migration attractiveness. This taxonomy divides all the regions into seven categories: A. Locomotives of growth 1) Global cities 2) Centers of federal importance (federal centers) B. "Backbone" regions 3) Commodity-producing regions 4) Old industrial regions C. Depressed regions 5) Stagnating regions 6) Regions in crisis 7) Special regions (the special territories of Chechnya and Ingushetia) Belonging to one of the "families"-pedagogical universities, schools of arts and culture, etc. [Sokolov 2017-requires some explanation. In the Soviet world picture, universities were producing staff for specific economic sectors. As a rule, they were directly subordinate to the relevant government department, their names containing a relevant reference. In 1991, belonging to a "family" determined the university's future in at least two senses. First, the relation to an economic sector determined prestige and attractiveness for prospective students. Second, it procured the numbers of state-funded places. In a strict sense, the word "family" is only applicable to about 500 state universities that existed in the Soviet era. New universities, private and municipal in the first place, were not obliged to take names by the same template and usually ignored it. In addition, some old "families" consisted of only one university that had to supply workers for a small department. For instance, the Russian State University for the Humanities -during its time as Moscow State Institute for History and Archives -prepared archivists, and Moscow State Institute of International Relations trained diplomats. Such universities could not be used as a categorical variable in statistical analysis.
Eventually, we came up with a taxonomy based on the characteristics that have a critical impact on how the university is perceived by the Ministry and prospective students. The very basic characteristic is the founder, i. e. whether the university was state, private, or municipal. Next, we grouped state universities into categories by the major founders that had governed universities of a certain type since the Soviet times -the Ministries of Culture, Agriculture, and Health. Within the group of universities subordinate to the Ministry of Education, we identified two major categories -pedagogical and classicalthat had standardized and common names (<City> State University). All the other state universities were divided into the broad categories of social sciences and humanities (if their names had a relevant reference), technical, and sports/military. The resulting taxonomy looks as follows: 1) Agrarian universities 2) Schools of arts and culture 3) Medical schools 4) Pedagogical universities 5) Classical universities 6) Sports and military universities 7) Technical universities 8) Universities for social sciences and humanities 9) Municipal universities 10) Private universities First thing, we are going to show how being located in a regional capital affects university's key performance indicators (Table 5).
Universities located in regional capitals differ from those in regular cities in all the indicators except financial and economic activities, but Results http://vo.hse.ru/en/ K. Guba, M. Sokolov, A. Tsivinskaya Fictitious Efficiency only differences in education and infrastructure are statistically significant. Universities in regular cities show lower performance in education (probably because education on the periphery is less prestigious than in a central city), yet they are more likely to be efficient infrastructurally (probably due to a greater shortage of physical space in regional capitals).
Data provided in Table 6 allows tracing the correlations between the type of region and the percentage of universities efficient in some or other key indicators. At first glimpse, the "global cities" of Moscow and St. Petersburg demonstrate the lowest performance, unexpectedly. We assume, however, that the baselines for those cities were set unreasonably high. Other regions classified as "locomotives of growth" showed high performance. The lowest education-related characteristics (average USE scores) were observed in the rich commodity-producing regions, perhaps due to migration to global cities. Commodity-producing regions failed the internationalization indicators but took the first place in financial and economic activities, which largely depend on regional economic health, being naturally low in the regions in crisis14. A similar situation is observed with employment, and only infrastructure is the best in the regions in crisis, probably as a result of lower demand for real estate.
14 A comprehensive analysis of universities' research and economic performance indicators should also make allowance for differences in the level of prices and purchasing power across the regions, which sometimes amount to two-fold [Litvintseva, Voronkova, Stukalenko 2007]. We are grateful to the reviewer from Voprosy Obrazovaniya / Educational Studies Moscow for drawing our attention to this circumstance. At this moment, however, consideration of such differences is a direction for further research. The number of universities efficient in education differs from "family" to "family" (Table 7), the highest performance being demonstrated by medical schools, schools of arts and culture, universities for social sciences and humanities, and pedagogical universities. The quality of students is lower in agrarian and private universities (but some of private universities scored zero on this indicator for not admitting can- didates to state-funded places based on their USE scores). Different patterns are observed for performance in research and development, which is assessed as R&D spending. While agrarian universities rarely demonstrate high USE scores and, as a consequence, are rarely acknowledged as efficient in education, they are much more likely to score high in R&D spending. Meanwhile, medical schools and schools of arts and culture are highly efficient in education but less so in research and development.
No "family" of universities is perfectly efficient in international activities. Low performers can be found in every "family", but their percentage varies from 17% among schools of arts and culture to 70% in the category of municipal institutions. Sports and military universities attract international students and faculty as rarely as agrarian ones, half of the institutions in both groups scoring under the baseline in international relations. Similar results are shown by classical, pedagogical, and technical universities, of which about one quarter are recognized as inefficient in this aspect.
Sports and military universities were found to be the most financially successful, no more than 15% of them being ranked as inefficient in financial and economic activities. In the rest of the categories, universities are much more likely to underscore, schools of arts and culture, private, and pedagogical universities being the least successful. In terms of infrastructural efficiency, the smallest lab floor area per student is observed for schools of social sciences and humanities and pedagogical universities, which are not actually in need of laboratories, given their specialization. The highest performance is demonstrated by sports and military universities (probably thanks to gym and stadium floor area), which are often found to be inefficient in other indicators. Only medical schools prove to be 100% efficient in employment. As for the other "families", the inefficiency rate is rather high, ranging from 20% among sports and military universiteis to 50% among schools for social sciences and humanities, technical, and private universities. Agrarian institutions are most likely to perform below the baseline level, 65% of them being acknoweldegd as inefficient. In overall efficiency, private universities are the worst performers, only 47% of them being efficient, while the highest efficiency of 95% is observed among sports and military universities.
The next step in this study consisted in assessing the impact of various structural factors on the university performance indicators, while controlling for the rest of the factors. Binary logistic regression was used, with being ranked as efficient as the dependent variable and structural determinants as independent variables (Table 8).
On the whole, the binary logistic regression results are consistent with the descriptive statistics presented above. The probability of university being ranked as efficient along the key indicators varies across the types of region as well as across the "families". For some "families", low performance in one indicator is partially compensated   for by advantage in others. However, private, municipal, agrarian universities, and most probably universities for social sciences and humanities are definitely in less favorable positions than classical universities -the most advantaged category. Within regions, surprisingly, universities in regional capitals turn out to be the most disadvantaged, their efficiency being lower than in federal centers, old industrial regions, and stagnating regions.
Therefore, being acknowledged as efficient or inefficient in conformity with the Survey of Performance of Higher Education Institutions was largely a function of university's ascriptive characteristics -and it was also the question of survival (avoiding closure or acquisition in 2013-2017) or organizational death. Cox proportional hazards model was used to assess that risk. Unlike logistic regression, the proportional hazards model includes a temporal component, i. e. how the influence of the factors changes over time. The event of university closing, merger, or acquisition was used as a dependent variable. Independent variables were represented by structural factors, and the model also included a time-varying covariate of efficiency (Table 9). Besides, we added an ecological variable that described competitiveness in the local education market, proceeding from an intuitively plausible assumption that universities facing fierce competition and a massive inflow of students from other cities would evolve differently from monopolists in a stagnating education market of a depressed region. Being ranked as efficient is the decisive predictor of university survival. The risk of being closed15 is seven times lower for efficient universities than for inefficient ones. The other factors remain significant even when the effects of efficiency are controlled for.
The impact of "family" decreases as efficiency is added to the model but remains significant at this stage -for instance, the risk of being closed is higher for private universities than for public ones even if private universities are ranked as efficient. On the contrary, medical schools, agrarian universities, and schools of arts and culture have a nearly zero risk of being closed even if they are recognized as inefficient (probably because closure for a public university normally implies a merger, and merging universities subordinate to different min-15 One should discriminate between risk, hazard, and odds. Risk, or hazard, is the ratio of chances of being closed to the probability of all possible scenarios; odds are ratios of chances of being closed to chances of not being closed. The difference between risk and hazard is that risk is cumulative over a time span, and hazard is instantaneous. The Cox model estimates the hazard ratio between the groups at a particular time. Even though the hazard of being closed may change for each group over time, the ratio remains constant. That is, any time that we observe the groups, the hazard of being closed is seven times higher for private universities than for public ones, although the hazard for a specific group may change over time.

Where the Reforms Have Led
istries is bureaucratically challenging). Socioeconomic development of the region did not affect university closure rates, except the special regions, in which no universities had been closed at all. The level of competition in the education market was also found to have no significant effects on university survivability. As we can see, the immediate outcome of the university performance assessment reforms was the reduction in the number of universities belonging to "unlucky" categories. Figure 1 shows statistically significant differences between the survival curves of private and public universities. Between 2013 and 2017, the risk of being closed was constantly growing for private universities.    *** p < 0,01; ** p < 0,05; * p < 0,1.
As can be seen from the above, the concerns associated with using inputs to measure performance of Russian universities are quite legitimate. A university's chances of scoring above the baseline of efficiency in certain aspects are largely contingent on its ascriptive characteristics. While "innate" strengths and weaknesses of public universities and main campuses balanced one another to some extent, private universities and branch campuses were doomed to lose. Such determinism could have been justified if performance measured by the Survey had been interpreted as effectiveness and the Ministry's goal had been simply to close weak institutions, leaving the strong ones.
In that case, implementation of the Survey results would have boiled down to weeding out the universities that conformed the least to the standard of a model research university attracting students from Russia and abroad, getting its research heavily funded, boasting generous lab floor areas, and paying high salaries to its faculty -regardless of why exactly they did not conform. Leaving aside discussion of whether such an approach to assessment would have been justified,16 we 16 In our mind, it would not be justified at all, as such an initiative would deny the plurality of modern universities' missions, in the light of which a particular indicator may be more or less relevant. Take graduate salaries, for example. A pedagogical university whose graduates all get employed as school teachers may be recognized as extremely successful in fulfilling its mission. It cultivates in students a strong belief in the importance of their vocation, Public can conclude that the Survey results were definitely not used that way. Low-performing public universities were not closed, but merged -together with all of their problems -with strong institutions. Again, this measure would have been justified if the Survey had measured efficiency and it could have been hoped that more efficient managers of surviving universities would untangle all the mess in the weak institutions. However, our findings show that efficient management could have played a much lesser role in the leadership of some universities than their administrators probably wanted to think. Some institutions had more favorable conditions from the very beginning, and their high performance reflected the advantages of their "family" and geographic location rather than successful leadership. What could be changed in the Survey if the experiment was started over again and the goal was to measure efficiency, i. e. how well a university has been using its resources to survive and develop? Naturally, the main challenge with efficiency assessment is that both resources (inputs) and outcomes (outputs) should be measured. Two solutions are possible here. The first one implies comparing the indicators longitudinally. A university may be compared to itself (provided that it has not been merged or otherwise reorganized), where a 20% increase in a specific indicator over a certain period of time will mean a significant improvement. Or, comparison can be made to the progress of other universities over the same period (precautions being taken against time lag bias, indicator volatility for small universi-ties17, etc.). Such metrics can serve as a tool for evaluating university management.

Discussion and Conclusion
The second solution is about introducing a more effective system of baselines to make allowance for competitive advantages of some university categories. Instead of applying baselines for capital cities, which are in place today, expected outcomes could be estimated for universities possessing a particular set of characteristics (main or branch campus, type of region, "family", etc.) to assess institutions by the extent to which their performance is better or worse than expected. By way of experiment, we applied the obtained logistic regression coefficients to the 2014 data, estimated the universities' chances of which makes them agree to work as teachers despite low pay. However, if 100% of the graduates of an institute of finance end up as teachers of social sciences, that institute may be suspected of preparing graduates unable to find a job matching their qualifications. At the moment, the Survey requires that universities perform well in at least four indicators. It would be more rational, however, to determine a list of missions corresponding to university specialization and resources -and specify the target performance indicators for each mission.
17 Volatility of Survey indicators for small universities is a major challenge for interpretation. Many indicators, particularly those related to the means of the Pareto distributed variables (e. g. citations per 100 faculty members), often turn out to be extremely unreliable for small universities.
being ranked as efficient, and singled out those that succeeded despite low chances and those that failed despite having all the chances. Among the universities that crossed the baseline, the lowest chances (33.4%) of doing so had been observed for 45 private universities in Moscow and St. Petersburg. Most of them (29) were still active in 2017, so from this point of view, they should be regarded as the most efficiently managed universities in Russia18.