“Forming expert environment for accreditation of educational programs: A case of Ukraine”

ARTICLE INFO Oleksandr Dluhopolskyi, Olena Knysh, Ihor Oleksiv, Lesia Smyrna and Oksana Panchenko (2021). Forming expert environment for accreditation of educational programs: A case of Ukraine. Knowledge and Performance Management, 5(1), 68-82. doi:10.21511/kpm.05(1).2021.06 DOI http://dx.doi.org/10.21511/kpm.05(1).2021.06 RELEASED ON Thursday, 30 December 2021 RECEIVED ON Saturday, 16 October 2021 ACCEPTED ON Monday, 27 December 2021


INTRODUCTION
The forming of an expert environment for assessing the quality of higher education has been relevant throughout the history of independent Ukraine. On the one hand, expert activity was carried out, forming certain national traditions, and on the other hand, these traditions were associated with corruption risks, formalism, bureaucracy, and agreements between participants in the educational process. Analytical studies of the state of higher education in Ukraine (Finikov & Sharov, 2014) demonstrated the post-Soviet "priority of state interests over public ones". Therefore, the key role of state authorities and management in the functioning of the National Quality Assurance System was evidenced. According to Kvit (2020), "Ukraine needs a decisive break with Soviet political culture and practices of social interaction. It is possible only if the approaches to national education change: from the ideal of a competent task performer to a leader who can develop the culture of an organization, field, nation-state, and globally". That is why the transition from the "old" to the "new" system of accreditation of study programs caused a full range of emotions Since November 2019, Ukraine has started the first accreditation expertise according to European standards, introduced by the newly formed institution of the National Agency for Higher Education Quality Assurance (NAQA). Due to some different external and internal reasons (Luhovyi, 2020;ENQA, 2015;Government Portal, 2017), the previous composition of the Agency had not started the work. Therefore, the newly elected configuration headed by a former Minister of Education and Science Kvit (Artiukhov et al., 2021) was required the entire legal framework to launch a new model of accreditation of study programs in a reasonably short time.
To conduct real, rather than formal, examinations of study programs in Ukraine, it was necessary to form an expert environment of pedagogical staff and students who wanted to join the reform of higher education in Ukraine to improve its quality and retrain those who had already participated in examinations before but had experience in formal expertise.

LITERATURE REVIEW
In the European educational space, considerable attention is paid to ensuring a sufficient level of competence of experts and their proper preparation for the accreditation process. Recommendations of the International Network for Quality Assurance Agencies in Higher Education (2018) and European Network for Quality Assurance in Higher Education (2015) determine the need to organize proper training of experts for accreditation and ensure education quality.
They are launching the European accreditation agencies' work primarily involved searching for best practices for expert groups training for the accreditation process (Frederiks et al., 2012). In particular, the development of training methodology, identification of key components of expert training, development of accreditation agencies' approaches to ensuring the necessary level of competence of expert groups. The authors of the E-TRAIN project are convinced that "no matter how competently the quality standards, frameworks, and procedures are composed -if the experts are not conversant with the application of these standards, the quality of the procedure might suffer. Adequate human resource development is central to the development of the external quality assurance agencies" (Cheung, 2015).
It is also noted that "for example, only carefully trained and highly qualified staff, as well as peers, can carry out such activities successfully.
Moreover, fundamental knowledge about communication, conflict management, project management, socio-psychological issues of group interaction, to name just a few, is necessary as well as competencies like discourse moderation and the ability to write reports in a clear, binding and fair manner" (Damian et al., 2015).
Preliminary training of experts is particularly important in the context of large extensive higher education systems with a significant number of higher education institutions and implemented study programs. In particular, Martin and Stella (2007) emphasized that under such conditions, accreditation agencies are obliged to ensure maximum professionalism and autonomy of expert groups and exclude potential discrepancies in approaches to accreditation procedures. Consequently, the launch of accreditation agencies is often accompanied by the implementation of large-scale projects for expert training (Silva-Triviño & Ramírez-Gatica, 2004) and the involvement of foreign partner organizations (Trifiro, 2018).
Mandatory training is provided in the recommendations of the European Association of Quality Assurance Agencies (ENQA, 2012). ENQA was eventually developed by the quality assurance professional competencies framework (ENQA, 2016), which includes: knowledge (higher education sector knowledge, legal framework, agency procedures); systematic/technical competencies (analytical, problem-solving, project management); social skills (communication, teamwork, report writing, etc.). Today, accreditation agencies of the European Higher Education Area apply various approaches to ensuring the readiness of experts for work. Appropriate forms and scope of works depend on several factors: the procedures implemented by various agencies; the context of the agencies' activities, national higher education, or the field context that the accreditation is primarily aimed at.
European accreditation agencies (EQAR, 2021; Schleicher, 2020; ENQA, 2021b) focus on improving approaches to training work with experts because this work is a prerequisite for building confidence in the activities of expert groups and decisions taken by agencies. Dunn et al. (2021) summarized the approaches to quality assurance, identifying similarities and differences, and made recommendations for quality assurance in the EU.
The study aims to analyze the peculiarities of an expert environment for the accreditation of educational programs in Ukraine through the assessment of the quality of training for experts, conducted during 2020-2021.

METHODS AND DATA
The paper uses analytical data and statistical information obtained from scientific articles on expert assessments and accreditation, as well as research of international institutions. A significant amount of information is obtained from the available reports and surveys of the NAQA respondents. NAQA conducts regular surveys among experts to obtain information about the quality of the accreditation procedure, the quality of expert training, their assessment of the expert procedure, support from various departments of the NAQA, and the perception of the accreditation procedure by the institutions of higher education. For this study, the method of correlation analysis was used, due to which the results of four surveys conducted among experts for January-February 2020, March-August 2020, September-December 2020, and January-April 2021 were analyzed. In general, the number of respondents (experts) increased permanently during 2020-2021 (from 67 to 1296), which is due to the constant updating and expansion of the NAQA's expert pool through training.

RESULTS AND DISCUSSION
At the end of 2019, training of the experts for accreditation of study programs was launched throughout Ukraine. The training was preceded by the selection and certification of 36 future trainers to train accreditation experts. The expert's training program was developed with the support of the British Council and specialists from the Quality Assurance Agency for Higher Education (QAA, UK). These pieces of training have become a good start for motivated experts, SEC and NAQA members, who have started largescale training of accreditation experts in Ukraine since the end of 2019.
Thanks to the established network of certified trainers and coordination of the work department with experts of the NAQA Secretariat, a total of 95 pieces of training were organized for future experts in six months -from September 2019 to February 2020. Pieces of training were held in almost all oblasts of the country, primarily depending on the number of higher education institutions in different regions and, accordingly, the number of potential accreditation experts. Thus, effective training of experts pool necessary to launch the accreditation process was organized -in 2019, 1,806 experts were included in the Register of experts, including 1,423 academic and teaching staff and 383 students ( Table 1).
The COVID-19 pandemic has suspended training activities for several months. However, with the forced transition of NAQA expert site visits to the online format in the spring of 2020 (Stukalo, 2020;Stukalo & Dluhopolskyi, 2020), the need has increased not only to expand the Register of experts further but also to review the format and content of the training. Therefore, training resumed in the fall of 2020 in an online format.
At the same time, systematic internal monitoring of the quality of training for future experts was carried out -both in the context of analyzing the feedback of training participants and the needs of NAQA (launching accreditations of study programs of the third level of higher education). In February 2020, an additional seminar for trainers was held, and a training program was developed for accreditation of Ph.D. study programs.
In addition, several online events were organized for trainers with the participation of leading Ukrainian and international experts on modern approaches to ensuring the quality of higher education in the world, academic integrity, etc.
In addition, in the fall of 2020, NAQA, together with QAA, certified 20 more trainers who were the most experienced and members of SEC. In December 2020, additional online training for trainers was held to train experts to strengthen the competencies of existing trainers (the socalled advanced level).
Such support made it possible to conduct 63 more online pieces of training in a relatively short time (August 2020-January 2021) (out of a total of 158 pieces of training) and add more than 1,000 new ex-perts to the Register. As of January 2021, the Register totals 3,028 experts (2,201 -academic and teaching staff, 827 -postgraduates and doctoral students), which is quite a significant result for 1.5 years of active work of the team of trainers and the NAQA Secretariat in various formats ( Table 2).
It should be noted that during 2019-2021, thanks to the well-coordinated work of experts, SEC and NAQA members, more than a thousand study programs were accredited, detailed information on which is given in Table 3. Thus, the decisions of the National Agency regarding Ph.D. programs are not so critical since accreditation of Ph.D. programs requires the development of new approaches to methodological tools for conducting accreditation of the actual scientific component of such programs (Kvit, 2021). Training structure developed by the NAQA Secretariat ( Figure 1) is aimed at reproducing the procedure for conducting accreditation expertise (the stage of studying information about self-assessment and preparation for a site-visit; planning and organizing a visit; writing a report by an expert group; providing advice to the HEI, etc.). The training includes real cases of European and Ukrainian accreditations that have already been conducted and allows a flexible combination of procedural and substantive parts of an expert visit to an HEI.
One of the key organizational features of NAQA training for future accreditation experts is that students and academic and teaching staff are equal participants. This decision is connected primarily with the need for value acceptance of the student's personality as an equal member of the expert group and the main stakeholders of the educational process in HEIs.
It is worth noting that from the very beginning of the NAQA expert training campaign, a virtually mixed training format was chosen, which allowed optimal use of available resources. It also provided the necessary level of incoming knowledge for ad-mission to full-time training (in particular, understanding of national legislation in the field of higher education, the main trends in the development of European Higher Education Area, reading and understanding the regulation on accreditation of study programs as a key regulatory document of the accreditation process, etc.).
In particular, first future experts complete the Expert on Accreditation of Study Programmes course on the Prometheus platform (massive open online courses). Doing well the final test is admission to further training. The total number of users who completed this course is 6,534, and more than 85% or 5,574 participants received certificates of successful pass. In addition, starting from 2020, a course on the ways to write a study program accreditation report is available on the Prometheus platform, which 5,826 people have completed (Kvit, 2021).
The Certificate of Expert of Accreditation of Study Programmes course by Prometheus is a preparatory stage for the inclusion of applicants in the Register of accreditation experts. The main stage is training with the participation of certified trainers from NAQA, including both Agency's mem-  bers, SEC members, and experts with experience and a good reputation, which cover the required material for newly selected experts from different points of view. Therefore, during several days of intensive training, future NAQA experts master the following skills: 1) to review self-assessment information provided by HEI; 2) to plan a site visit; 3) to identify problematic moments (conflict situations) that may arise during a site visit; 4) to summarize the self-assessment findings of the study programs in the accreditation expertise report and provide an assessment following the Standards; 5) to provide advice to representatives of HEI on improving the quality of the study programs; 6) to follow the ethics during a site visit.
Considering certain expected results, there is a variant in procedures. The training has a practical focus. This approach allows updating the skills of participants necessary for successful work as a member of the expert group and as objectively as possible assess their readiness for accreditation expertise, as well as their understanding of NAQA value standards. In particular, this is: 1) case analysis, which covers the potential most challenging situations that may unroll to the expert group at different stages of the accreditation expertise. The cases explain both the ethical norms of experts' behavior during the site and online expertise and aspects of communication between the experts and HEI, the distribution of responsibility, and the specifics of the visit planning, the final briefing, and the expert group report.
At experts' training, it is repeatedly noted that the paradigm of conducting accreditation expertise has changed ( Figure 2)  2) analysis of self-assessment information of a study program during the training sessions begins with homework, which must be completed in advance by the selected training participants. They identify aspects that need to be clarified, outline the list of documents that may be required to confirm the information in the self-assessment report, as well as the representatives of an HEI with whom it is necessary to meet for confirmation and indicate the content of a study program that can be recognized as best practices. During group work, participants develop collective decision-making skills on the results of accreditation expertise.
The online format of the training sessions has significantly influenced the methodological approaches to the arrangement of work. Apart from skills development, in fact, there has been Source: Own elaboration.

SITE VISIT
Meeting with the management of an HEI and the head of a study program Meeting with students pursuing a program, preferably of different study years Meeting with a teaching staff of a study program

Meeting with HEI' representatives
Open meeting Final briefing to provide the institution with recommendations for improving a study program, highlight its strengths, and note of acknowledgment for collaborative work Meeting of an expert group to summarise the results and distribute the work on the report writing (without the participation of an HEI) Other activities necessary to gain a full understanding of the quality of the study program, its compliance with the standard, as well as criteria (meeting with graduates, administrative staff, backup meeting, etc.) an urgent need to develop the skills of experts' interaction in the virtual space generated by the online format of accreditation expertise. Therefore, the preliminary study of the training on self-assessment was supplemented by the work with a specially created preliminary analysis form, which must be completed 24 hours before the training. Participants could receive feedback from their trainers even before the start of the training session. Participants' activities in training focus on the teamwork of joint documents. This approach was subsequently elaborated in the development and gradual introduction into the activities of NAQA expert groups in the Application of preliminary expert results (PER).
3) simulation of meetings. Simulation meetings with academic and teaching staff, students, Ph.D. students, and the administration of higher educational institutions are an integral part of the training of experts on accreditation of study programs. At such meetings, experts learn how to prepare and put questions to different focus groups, practice timing for offline and online meetings, try themselves as teachers and students, answer random questions from colleagues, and develop their soft skills.
According to the agreement, experts are required to provide HEI with advice and recommendations for improving the quality of a study program, but experts are prohibited from voicing their assessment to the representatives of a higher education institution during a site visit since it is not final. The role of an expert is crucial in the first two stages of accreditation ( Figure 3). In comparison, the following two stages are independent since both the Sectoral expert council (SEC) and NAQA evaluate the work of experts and form the final assessment.
The online training format has become perhaps the biggest challenge for this form of work because team building and live exchange of emotions are somewhat complicated during an online conference. At the same time, the understanding of the ethics of online communication, as well as the skills of moderating meetings during remote examinations, is being updated.
4) the main element of the training is the preparation of a draft report by an expert group based on the results of the accreditation examination. Certified trainers analyze in detail the key stages of writing a report, allowing participants to make a collective decision on the results of accreditation examination, compliance levels, and justifications for criteria and a whole program. The most interesting part of the training is the analysis of the training project in an expert group report and the study of a training review of the accreditation department, as well as the comparison of grades given in the expert group report and the expert opinion of the SEC (NAQA, 2020). As a rule, it is at this stage of the training that future experts learn critical thinking, realizing the importance of the objective assessment of the study program.
Since the opinions of the expert group and the SEC on the quality of a study program may differ, it becomes important to understand that the NAQA can support any of the positions at its regular meeting. The panorama of judgments of NAQA members is much broader and all cases in which the opinions of experts and the SEC do not agree are subject to additional study. Since HEIs can also provide information that changes the idea of a program in one direction or another at the end of accreditation, therefore, there is a procedure for returning the case to the SEC to clarify all the details. At the concili-Source: Own elaboration.

Site visit
Expert report Conclusion of the SEC NAQA meeting and decision ation boards, NAQA members study all disputed cases in detail and form a consolidated opinion for the next meeting. Finally, the members of the National Agency for Higher Education Quality Assurance take into account either the view of an expert group or the SEC in case of discrepancy in their assessments. They may also lower the assessment by one or another criterion if significant problematic aspects of the program are identified that were not taken into consideration by expert groups and members of the SEC. Based on the results of NAQA's work in 2019-2021, only in 3% of all cases, the decisions of the National Agency did not coincide with the previous decisions of the SEC -in such situations, NAQA sided with the expert group. All these aspects are discussed in training sessions for future experts to understand their role in the accreditation process better and realize their responsibility.
The attributes of a high-quality report of the expert group, the results of which form the reputation capital of an expert, are also noted at the experts' training sessions on accreditation of study programs (NAQA, 2020; Vorobiova et al., 2020):

1) completeness;
2) compliance with the regulatory framework and respect for the autonomy of the HEI; 3) argumentation and evidentiary Standards; 4) optimal involvement of the documentation of the HEI;

5) correlation of comments and ratings;
6) objectivity (lack of value judgments); 7) originality of the text.
It is easier for future experts to learn materials on the preparation of a visit plan, ethical aspects of accreditation, and working with various focus groups, while difficulties often arise when analyzing self-assessment information of HEIs and formulating the consolidated position of an expert group. There is an option of expressing a so-called "separate opinion" to help future experts.
To study the impact of training on the quality of accreditation visits and the quality of expert group reports, an anonymous online survey of experts was conducted. This study considered the answers to the following questions posed to the experts: • please evaluate the new accreditation process of a study program in general (Q1); • please rank how beneficial the training for experts was (Q2); • please evaluate the benefits of the guidelines that were prepared for your case (Q3); • please evaluate the support by NAQA expert department (Q4); • please evaluate the support by NAQA accreditation department (Q5); • please evaluate the support by NAQA other departments (Q6).
It should be emphasized that in 2020 the accreditation procedure was slightly changed. Accordingly, the September-December 2020 survey was supplemented with the question, "Please evaluate the benefits of the review of an expert group's draft report, if any (Q7)." In the January-April 2021 survey, this question replaced the question on the quality of methodological recommendations for experts, which were removed from the accreditation procedure (Q8). The experts answered the questions using a 10-point scale. However, the responses mostly ranged from 6 to 10. Accordingly, the responses from this range are presented in Figure 4.
The results of the assessment of the accreditation procedure by experts for all four periods are shown in Figure 4, which allows concluding that the assessment of the accreditation procedure by experts has constantly increased. In particular, if at the beginning of 2020 only 11% of experts gave the maximum score (10) to the accreditation procedure, then in 2021, they accounted for more than 34%. Figure 5 shows that experts consistently highly rated the quality of training provided by the NAQA. At the same time, it is assumed that the results of the first survey, which was conducted among only 68 experts, do not fully reflect reality. However, later, the number of experts who evaluated the quality of training for the maximum rating constantly increased and reached 57% in the recent survey. At the same time, the proportion of experts who rated the usefulness of training as not high (6 points) was constantly falling and in 2021 was less than 1%.
Correlation matrices were calculated for the entire list of questions to track causal relationships (Tables 4-7).
This approach did not reveal significant relationships between the experts' responses. In particular, there is no correlation between the quality of the accreditation procedure and the usefulness of expert training. There is also no correlation between the perception of the accreditation procedure of an HEI and the training of experts. Correlation coefficients with the average bond strength are observed only in part of the assistance of different NAQA departments.
The lack of live communication caused by the remote training format, at the same time, gave rise to the search for new interaction formats. These new formats have demonstrated their real advantages in some places. In particular, numerous interactive surveys with instant visualization of the results allowed hearing the opinion and position of each individual participant, identifying problematic issues faced by many participants, which they did not dare to voice in public.
Source: Own calculation.   Another benefit of the online format became the possibility of the increased participation of the guests in training sessions. Indeed, the first wave of (offline) training was accompanied by round tables and meetings with NAQA members in various higher educational institutions of Ukraine. However, it should be noted that such meetings were primarily aimed at communicating the principles of the new accreditation procedure to the academic environment of an HEI, the management, and guarantors of study programs. Communication with future experts was not always possible due to the extremely high frequency (up to 25 per month) and the wide geography of training sessions (held simultaneously in different regions of Ukraine Comparing the effectiveness of expert training sessions in online and offline formats allows concluding the following. First, the disadvantages of one training format automatically become the advantages of another (Table 8). Thus, the narrow regional coverage of the offline format, when the 2019 training sessions were held based on one university, and teachers, although from higher education institutions, but mainly from the same region, came for two full days, investing their time and money, is leveled by the online training format. The online format gives the possibility to reach the optimal number of participants from different regions of the country and minimize business travel expenses for participants. Also, the online format makes it possible to track the completion of homework more clearly by participants, their actual, rather than formal, participation in the training (Stavytskyy et al., 2019;

CONCLUSION
According to the aim of the study, the formation of the expert environment for the accreditation of educational programs in Ukraine was analyzed. In general, the experience gained in the expert training on the accreditation of study programs in Ukraine and the NAQA accreditation expertise, which were conducted by mid-2021, allow concluding the following.
First, the accreditation system of study programs has been working in a new way since the end of 2019without bureaucracy, corruption, through mechanisms of transparency of procedures and expert advice. It is really one of the main achievements of the European Union-Ukraine Association Agreement (part 23).
Second, the emergence of the accreditation model according to the ESG-2015 standards made it possible to create an expert environment in Ukraine in a relatively short time and encourage changes in the HEIs aimed at student-centered learning, attracting key stakeholders to the educational process, and improving the quality of study programs and services.
Third, the study did not reveal a significant correlation between the training of experts and the quality of the accreditation procedure. Unfortunately, despite many training sessions and consultations, many experts and representatives of the SEC in assessing the study programs speak from the standpoint of criticism, bias, rather than advice and assistance. Quite often, experts compare study programs of their universities with those of other HEIs and try to eliminate undesirable programs using unfair competition. The other extreme is the granting of exemplary accreditation to the programs that are not such a priori, which generally indicates an insufficient expert level of the individual experts. Conflicts of interest are also difficult to avoid in many areas.
To minimize such negative practices, NAQA regularly conducts online webinars, "National Agency's schools of quality", meetings with NAQA members, briefings for expert groups, and guarantors of study programs in the process of remote examination, as well as rotation of experts and the SEC members. In the future, NAQA is planned to be more selective in the process of retaking new experts for the educational program's accreditation.