Share on
INTRODUCTION
Audit and feedback (A&F) interventions are used to monitor and improve professional practice and quality of assistance [1, 2]. In the audit phase, key performance indicators (KPIs) measure the performance of health care professionals and compare it to specific targets. In the feedback phase, professionals receive the measurement and benchmarking results [3]. A&F usually produces little but beneficial improvement, allows cumulative incremental gains through repeated cycles [4, 5], and is potentially cost-effective [6, 7].
When professional practice needs to be changed, interventions should be tailored considering all possible barriers and enablers [8].
A number of factors are related to recipients: the different health care professionals [5, 9, 10]; their reactions and perspectives [10], specific skills and education on data interpretation [5, 11]; attitude, motivation and intention [5, 9, 10]; capability for change and improvement [5]; competing priorities [9-12]; awareness of being self-effective [10]; the degree their attention is engaged [10] and their defensive reactions are avoided [10]; engagement in A&F design [4, 5, 10, 11, 13-15].
Specific characteristics of the A&F content are also paramount: its cognitive load should not be excessive [4, 9, 10, 13, 15]; the comparators should be appropriate [4, 5, 10, 13, 15, 16], shared [17], and encourage motivation to change [5, 16, 18]; the identification of well-defined goals should be challenging [13] and perceived as relevant [9, 10, 14], achievable and under the recipient’s control [4, 9, 11, 13, 14]; actions and coping strategies to achieve the goals should be well described [10]; a justification for the need of change should be provided [10]; what was achieved rather than what is still missing should be emphasized (cognitive influence) [10]. The nature of data shown also plays an important role [10], since A&F effectiveness depends on the characteristics of those available for feedback [5, 13, 19], especially reliability [5, 9, 10, 20, 21]. Indicators should be valid, accurate [9, 11] and tailored to the context [5, 15, 22].
Effectiveness is also influenced by feedback timing and frequency [4, 5, 10, 13, 17, 21], being more effective when provided multiple times [10]. The way it is delivered and displayed is also crucial [4, 5, 9, 10, 13, 15, 21], since the credibility of the sender affects the results [4, 5, 9, 10, 13, 14]. Effectiveness also increases when it creates learning opportunities, encouraging reflection and improving memory [10]; when information is structured to allow the recipient to eventually get more afterward [10]; if recipients are engaged in discussions about the A&F. Functional clinical and managerial relationships are also facilitating factors [4, 17], especially when they allow supportive top-down feedback [4], avoiding performance anxiety [18].
Other determinants are costs [10] and an encouraging environment [10].
In the framework of the EASY-NET Project (NET-2016-02364191), we developed and calculated 61 KPIs on trauma, acute myocardial infarction, stroke to describe the Emergency Medical System (EMS) performance in Friuli Venezia Giulia (FVG; 1.2 million inhabitants), Italy [23]. We disseminated the results through: a report (November 2019; 2018 data); a report and a webinar (December 2020; 2019 data); a report and a workshop (April 2022; 2020 data). We e-mailed doctors and ward managers of two local Health Care Authorities (HCAs) a first survey in February 2020 (n=457) and a second in May 2021 (n=497) to explore their attitude towards the KPI implementation and use in the emergency area. We asked for top-down dissemination and collected 71 and 34 answers, respectively. Among the responders to the first survey, 60/71 consider feedback on their performance beneficial.
Engagement varied according to different ways to provide feedback, with multimodality proving better than the text only (1st feedback, only text report: 21/71 responders knew the report, 4 had read it; 2nd feedback, text report and webinar: up to 40 attendees; 3rd feedback, text report and workshop: 100 participants) [24].
This mixed methods study aims to describe the attitudes of health care professionals towards KPIs as quality improvement tools and to analyze the determinants of their implementation and use through an online survey, a focus group (FG), and ten interviews.
MATERIALS AND METHODS
The mixed-methods design
We used a convergent parallel design, comprising a parallel collection of quantitative and qualitative data, an analysis conducted after data collection, and a final comparison of the results. Quantitative data was collected through an online questionnaire, while qualitative data originated from one FG and ten interviews.
The rationale for integration was complementarity, since we aimed to provide a comprehensive description of the research problem through the analysis of multiple sources of evidence. At the interpretation and results level, integration was performed through narrative, both with a contiguous approach, presenting the qualitative and quantitative findings separately, and using a joint display where data is brought together and organised to facilitate comparison.
The reports
The 2020 KPI Report (2020 data; published and presented during the Workshop in 2022) describes the KPIs regarding the EMS (n=6), Emergency Room (ER; n=12), cardiology (n=20), neurology (n=4), major traumas (n=12), and Emergency Operation Centre (n=7).
The KPI Comparison Report (published in 2022) compares the second semesters of 2019 and 2020.
All sections of both reports consist of a description of the KPI calculation method (datasheets), a brief plain explanation of the main results, and a variable number of tables and graphs.
The online survey
In May 2023, we contacted the three local HCA, two Scientific Institutes for Research, Hospitalization and Health care, and the Regional Health Coordination Authority of the FVG Region, asking to send to all health care professionals a predefined brief e-mail illustrating the project and asking for collaboration, and including links to both reports and to an online survey about the attitude towards those reports and KPIs. The email reached all the employees through the generic “Mailbox Users” recipient, including all employees with an institutional email address. No reminders were sent.
To define its questions, we conducted a literature search focusing on the barriers and facilitators encountered during A&F programs using performance indicators [25-32].
The survey (estimated compilation time: 7-10 minutes) consisted of 50 multiple-choice and 23 open questions (D1-D73): D1 asked whether the compiler had read the report or not; the readers had to define if it was engaging, easy to read, complete, understandable, beneficial, and to express their opinion about the number of indicators, tables, and graphs (D2-D30). D31 asked whether the compiler had read the KPI Comparison Report and readers could compile the related questions (D32-D36). All compilers could answer questions on the mode the indicators’ results were provided (D37-D38) and their attitudes and opinions towards indicators (D39-D58). D59-D67 focused on the EASY-NET Project, while D68-D73 collected general data about the compiler.
We created the questionnaire using the online survey-management system EUSurvey (https://ec.europa.eu/eusurvey/), pre-tested it on a small sample of the final recipients, modified it according to their suggestions and kept online for 40 days (May-June 2023). The survey was anonymous, but compilers could choose to leave their name when they finished. The translation of this tool is available online as Supplementary Material.
The focus group
In June 2023, we emailed all 33 physicians who led the units involved in the emergency health care services, both in-hospital and out-of-hospital, in the FVG Region (ERs, EMS, cardiology, and neurology units), asking them to provide the names of a physician and a nurse who were both working in their unit and interested. We sent an invitation reminder one week apart. We used a multiple-choice doodle to schedule the FG session among twenty session choices (6 am, 2 pm) in 16 possible dates (14 weekdays, 2 Saturdays), from July 10th to August 1st. We asked professionals to select all the sessions they were available to attend, and we scheduled the FG to balance two objectives: ensuring the best representativeness of the emergency-urgency disciplines and geographic areas, and maximizing attendance. Six head physicians of the following units from two out of three of the FVG HCA participated in the FG: the Helicopter Rescue Unit (CL1), a spoke hospital ER (CL2), a local EMS (CL3), two cardiology units (CL4, CL5), and a neurology unit (CL6). Two researchers of the EASY-NET Project conducted the FG. CL2 could not participate in the FG and was interviewed separately.
We conducted both the FG and the interview according to Krueger and Casey [33]. The FG and the interview lasted about 60 and 50 minutes, respectively, and were all recorded and transcribed.
The interviews
Between April and June 2023, two researchers conducted, recorded and transcribed ten interviews with all the staff involved in the study: seven research team members, five co-workers of the EASY-NET Project, and two administrative assistants from the Azienda Sanitaria Universitaria Friuli Centrale (ASUFC) Research Office. The mean interview duration was 43 minutes.
Statistical analysis
Survey data: we described categorical variables as absolute frequencies and percentages.
FG and interviews: after repeatedly listening to the recordings and reading the transcriptions, we performed a thematic analysis, as described by Braun and Clarke [34].
RESULTS
The online survey
In the study period, 316 professionals compiled the questionnaire (Table 1): 22 knew the EASY-NET Project, and 10 had received at least one of its reports. Five professionals had received the invitation to join the 2020 webinar where we presented the 2019 KPI results but none of them participated due to shift overlap, day off or sick leave; 145 had not received the invitation (102 would have liked to, 43 were not interested). When asked whether they had received the invitation to join the 2022 workshop where we presented the overall project, including the 2020 KPI results, eight answered they did (one of them participated), 159 answered they did not (111 would have liked to, 48 were not interested), 141 did not remember.
When asked if they read the reports received in this study framework, 30 (9.5%) answered they read only the 2020 KPI report, 6 (1.9%) only the KPI Comparison Report, and 49 (15.5%) both. The majority (73.1%; n=231) reported having read none. When asked about the 2020 KPI report, only 23 answered they knew it before receiving our e-mail (17 being interested, eight being not), 283 they did not (232 being interested), and eight did not answer.
Readers were 42.7% among males, 21.8% among females, none among binary persons, while according to the age group, readers ranged from 22.0% (<34 years) to 30.4% (45-54 years). Among physicians, nurses and other professionals, 37.7%, 23.2% and 25.9%, respectively, had read the report. The readers were 30.3% of those who ever worked in the emergency-urgency setting (23.9% of past workers, 36.9% of current workers). The percentage of readers according to the local HCA was very similar, ranging from 25.0% (13/52) among the employees of the Azienda Sanitaria Friuli Occidentale (ASFO) to 27.3% (45/165); the only employee of the Azienda Regionale di Coordinamento per la Salute (ARCS) has read the report, as well.
Report readers
Among those who read the 2020 KPI Report (n=79), 81.0% found it very easy or easy enough to read and 86.1% found its subjects very/enough interesting.
The numbers of indicators, tables and graphs included in the 2020 KPI Report were considered “fair”. Table 2 reports a more detailed analysis.
We obtained similar results from those who read the KPI Comparison Report (n=55; data not shown).
All respondents
The majority of respondents (67.1%; n=212) reported they would be more interested if the presentation modality were different: 126 suggested an interactive presentation (education courses, congresses, conferences), 12 non-interactive tools (e-mail updates, n=5; audio or video recordings, n=7), and one a daily update on social networks. Twenty-two respondents expressed the need to receive a summary of the results.
The majority of respondents reported indicators improve their performance (very, 26%; enough, 57.3%) and 72.8% think that receiving feedback through indicators increases their job quality. Professionals declared performance indicators show professionals the improvable aspects in everyday tasks (73.7%), are useful to improve pathways (55.4%), allow comparisons with other organizations (52.8%), are objective (21.5%), increase weaknesses and strengths awareness (2.8%), reveal the need to introduce improvement strategies and allow to measure their effect (3.2%), help to quantify the resources amount (0.3%). The respondents also reported several negative aspects: they are calculated by people who are not directly involved in the care pathways and may not know essential practical details (45.9%), they’re not used to improve pathways (36.4%), they’re often calculated on old data (18.0%), and they are perceived as somehow judging their job (3.5%), they need to be interpreted (1.9%).
Most respondents (82.6%) believe discussing KPI results is beneficial, especially at the unit (59%), department (61.1%), and local HCA (61.7%) level (Table 3).
Most respondents reported they would find it convenient to have an update of KPI results annually (42.4%) or biannually (36.7%), but some of them would require it quarterly (13.9%) or monthly (3.2%). Consistently, the majority would find it beneficial to discuss the KPI results yearly (50.3%) or every six months (34.8%), while some of them would require a higher frequency (every three months: 9.8%; every month: 1.6%).
The large majority of respondents think the usefulness of KPI depends on whether data is recent (yes: 59.5%; sometimes: 27.2%); data is considered as suitable when not older than 3, 6, 12 and 24 months by 36 (11.4%), 100 (31.6%), 143 (45.3%) and 29 (9.2%) respondents, respectively.
The focus group (clinicians, CL)
Four main themes emerged from the thematic analysis: perception, communication, change, and data.
CL2/CL3/CL5 expressed their perception, one of them considering KPIs as essential, especially when dealing with time-dependent pathologies in the pre-hospital setting, where collecting data is not easy, and tools are less available than in the hospital setting (CL3). On the one hand, CL2/CL3 identify KPIs as a starting point for enhancement, allowing one to take standardized pictures of reality and to improve it. On the other hand, CL5 declared feeling disheartened, thinking KPIs are not useful for adjusting behaviors. CL2 also described the negative perception of the physicians and nurses working in the same unit: they think KPIs have no value, sometimes being felt as a school report that may be important to the head physician and it is only necessary to achieve the budget’s objectives and earn benefits.
All clinicians discussed their points of view about communication; CL2/CL3 believe a close clinicians-researchers collaboration would allow clinicians to know which KPIs researchers can calculate, and researchers to better understand data and its limits and to identify KPIs that are relevant for clinicians and that may have an impact on the patients’ pathways.
CL2/CL4/CL5/CL6 think that meetings could be beneficial for understanding data, should be organized at a regional and at local level and include both the head physicians and their collaborators, both doctors and nurses. CL2 also described the need for education on KPIs, which is paramount to learning how to contextualize KPIs and appreciate their value.
As for change, CL3 described the hard past unification of the provincial EMSs in a Regional Operations Centre, while CL2/CL5 are more future-oriented, with a strong willingness to change, CL2 believing the identification of barriers and hurdles should not prevent from changing actively, step by step, CL5 asking how to improve the quality of data and to obtain KPIs clinicians can rely on.
Data was deeply discussed: all clinicians described data collection as inadequate, since it sometimes derives from hardcopy archives or is input by health care personnel for specific purposes when information technology is insufficient; moreover, KPIs show inconsistencies when resulting from different databases or analyses. CL1/CL3/CL4 reported difficult or impossible access to data due to privacy and data ownership issues.
In addition, CL2/CL3/CL4 believe data is underused by organizations and policymakers, although adopting an A&F strategy using KPIs to measure clinical systems could help improvement; CL3/CL4 read this attitude as a lack of interest, preventing specific ventures from being sporadic and leaving clinicians isolated, while CL2 suggests that urgency outclasses planning and KPIs do not represent a priority for policymakers.
Of note, CL3 appreciated the novelty of analyzing and interpreting emergency KPIs in FVG and declared to be positively astonished, interested and enthusiastic about it.
The interviews (researchers)
Researchers debated the themes of novelty, perception, communication, change, data and pandemics.
Having calculated KPIs on the FVG emergency activities for the first time led to the enthusiastic participation of several health care professionals (novelty). Nevertheless, KPIs were sometimes interpreted as marks or judgements (perception).
KPIs interpretation was difficult for health care professionals not used to them; the lack of mediators between clinicians and researchers caused misunderstandings and hindered the comprehension of KPIs and their use for improvement, at times. In addition, a laboured collaboration between researchers and institutions did not help the project’s success. Researchers found it hard to give feedback to health care professionals, especially when results had to be shared hierarchically; in particular, the unit’s heads hardly involved their collaborators (communication).
Moreover, the context where researchers conducted the project was not stable in time (change): FVG political government and health care top management changed; modifications in health care districts and local HCA occurred both geographically and organizationally; the Regional Committee for Urgency and Emergency was dismissed; several EMS heads succeeded one another; the regional triage system and the urgency-emergency software changed. Therefore, it was not feasible for researchers to have stable institutional and technical reference figures aware of the system and trusted by professionals; moreover, systematically monitoring indicators in different periods was demanding.
Change also affected data, since the database used changed during the study period (number of records; name, number and coding of variables), often making reproducibility difficult when not impossible; moreover, a manual with metadata was not available, sometimes making not even possible to define whether changes were retrospective or prospective. However, data available in the FVG Region, although difficult to obtain, is considered a treasure by the researchers and the collaboration with colleagues working in institutional units represented a value.
Last but not least, pandemics could have been a barrier which may have lowered the interest in KPIs.
Table 4 shows a joint display which summarizes the barriers and facilitators emerging from the online survey, FG, and interviews we conducted.
DISCUSSION
This mixed-methods study investigated the barriers and facilitators of using KPIs, as reported by health care professionals and researchers. Six themes emerged from the online survey, FG and interviews: a) novelty: describing for the first time the emergency setting in FVG through KPIs led to the enthusiastic participation of health care professionals; b) perception: although KPIs are described as objective tools allowing improvement through comparison, they are also perceived as a judging method; c) communication: a fluid information flow, a close clinicians-researchers collaboration, dedicated meetings to discuss KPIs and specific education allowing their correct interpretation are crucial; d) change: many organizational changes were described as barriers but clinicians also showed a strong willingness to change; e) data: although much data is collected, data and IT infrastructure are often described as old, inadequate and hardly available due to privacy policies; f) pandemics may have lowered the interest on KPIs.
Among recipient-related factors, specific skills and education on data interpretation are described as facilitators [5, 11, 35]; consistently, in our study, clinicians reported the need for education on KPIs, which would help health care professionals to interpret their significance and appreciate their value. This is also in line with the suggestion, provided by some authors, to provide simplified key messages to minimize cognitive overload [2, 10, 13], and with the need for support in understanding and interpreting data [36].
Several authors reported that engaging the recipients when designing A&F interventions may improve effectiveness [4, 5, 10, 11, 13, 15]. Similarly, during the FG and the interviews both clinicians and researchers expressed the need for collaboration and debate to exchange information and build appropriate and meaningful indicators, also suggesting structured and well-timed meetings involving all the emergency actors, in line with the suggested participatory feedback construction [2, 13] and social engagement and interaction [10, 37]; similarly, Snider et al. reported the success of collaboration, allowing to tailor the report’s contents and structure [20].
Consistently with both our previous survey [24] and the FG, the online survey showed that receiving feedback on performance was perceived as beneficial because indicators increase awareness of both weaknesses and strengths, they reveal improvable aspects and allow to measure the effect of improvement strategies and to estimate the necessary amount of resources. Moreover, they allow comparisons with other organizations. However, we recorded contrasting opinions among the interviewees and the FG clinicians about the KPIs’ benefit in improving pathways. Moreover, some respondents considered KPIs objective, while others underlined that their interpretation needed contextualization. This is consistent with previous literature, suggesting the effectiveness of A&F varies according to data credibility and trustworthiness, as perceived by recipients [10].
Despite an overall positive attitude towards KPIs, a few interviewees declared they perceive their job being judged by KPIs, consistently with the researchers and one of the FG clinicians. This is in line with other studies, reporting that clinicians may suffer from performance anxiety [18] and feel threatened rather than supported by top-down feedback [4]. Some authors similarly described the role of attitude, motivation and intention of the recipients [5, 9, 10].
In our organization, this perception can be explained because KPIs are almost always linked to budget goals, thus KPIs may be interpreted as tools used to force behaviours.
As in other studies [10, 38], data quality was of concern. Most interviewees reported that the usefulness of KPIs depends on whether the data is recent, being less beneficial when calculated using old data. Data was also one of the themes widely discussed during the FG, when clinicians agreed that the quality of emergency data is not satisfactory, possibly causing inconsistencies when using different sources and databases. This confirms that data available for feedback should be appropriate [5, 13, 19] and tailored on the specific context [5, 15, 22], other than reliable [5, 9, 10, 20, 21], valid and accurate [9, 11]. Almost half of the interviewees reported that researchers who calculate KPIs do not have knowledge of the context they describe; for this reason, health care professionals often disagree with calculation methods and suggest clinicians and researchers should collaborate to increase the acceptance of KPIs results through the trust in data sources [10, 14, 20]. However, in our study, data represented not only a treasure but also a troublesome problem for the researchers, due to database changes and because they are difficult to obtain. Clinicians appreciated data value, as well, and some of them believe data is underused by organizations and policymakers, probably because KPIs do not represent a priority.
Our study revealed the difficulty of communicating the KPIs results to our target recipients. On the one hand, sending personal e-mails to each health care professional allowed us to enrol many more subjects than in the previous surveys [24]; on the other hand, only 2.5% of the interviewees reported they received the invitation to the workshop where we presented the KPIs results. This issue was unsurprisingly due to top-down dissemination and in line with the scientific literature, reporting that the effectiveness is influenced by the way feedback is delivered and displayed [4, 5, 9, 10, 13, 15, 21]. Unfortunately, a different dissemination method was not permitted, preventing a better engagement through multimodality feedback [2, 13, 20].
Another issue concerned the expected low percentage of the interviewees who read at least one of the KPI Reports (26.9%). This was not unexpected, since some Canadian authors reported that opening feedback reports only slightly improved even with different behavioral nudges [39].
In our study, the poor result could possibly be explained because most respondents would have leaned towards a different presentation modality, preferably interactive. Moreover, they would have preferred a summary of the results, since other studies already identified time [35, 38, 40] and competing priorities [9-12] as the most important barriers. Unfortunately, even though our reports consisted of text, tables and graphs, as suggested by the literature [2], they did not include short key messages that could encourage a first reading [2, 13].
Most online survey respondents reported they would find it convenient to have an update of KPI results and discuss them annually or biannually, especially at the unit, department, and local HCA level. Similarly, the FG clinicians requested meetings to discuss KPIs. This is in line with the scientific literature affirming A&F is more effective if recipients are engaged in discussions about the A&F [10] and that supervisors or colleagues are more reliable than other sources [13]. Moreover, it is consistent with the role of facilitator played by feedback timing and frequency [4, 5, 10, 13, 17, 21], suggesting A&F is more effective when delivery is provided multiple times [10].
An encouraging environment affects the effectiveness of A&F interventions [10], functional clinical and managerial relationships being needed [4]; consistently, we found as a major hurdle the political and organizational changes occurred in recent years, which prevented stable and informed institutional and technical reference figures professionals could trust.
Strengths and limitations
The main strength of this study is novelty: although KPIs are regularly used in FVG to monitor health care performance, this was the first research conducted to explore the point of view of health care professionals and to investigate the barriers and facilitators when using these tools. Another strength is the choice of a mixed-methods design, which allows us to provide a comprehensive description of the research problem through the analysis and integration of multiple sources of evidence. This allowed us to collect the viewpoints of: a) the health care professionals, who are those asked to change their behaviour to improve KPI estimates, b) the emergency-urgency area head physicians, who are directly involved in the budgeting process, where KPIs are used to measure the performance of each clinical unit, and c) the researchers, who are often asked to calculate KPIs but just as often are not involved in the clinical pathways. In addition, the survey’s open questions allowed health care professionals to express their point of view without constraint, not being forced to choose among predefined answers. Moreover, although the FG engaged only 6 out of the 33 invited physicians, these represent all the emergency-urgency disciplines and work in two of the three local HCA. This should also be considered a strength because the choice is derived from the predefined objective of reaching the best representativeness of disciplines and geographical areas.
However, this study also suffers from several limitations. First of all, only about three hundred health care professionals filled out the online survey, with response rates across the Authorities ranging from 0% (IRCCS Burlo Garofolo) to 3.1% (IRCCS Centro di Riferimento Oncologico, CRO, Oncology Reference Centre). One possible explanation is that the only possible way to reach health care professionals was a massive email dissemination from the Directorate of each local HCA, and for this reason, no reminders could be sent. Moreover, the email itself could have been considered as routine communication, and possibly ignored. Last but not least, the intended survey recipients were all health care professionals, but the email explicitly mentioned the emergency-urgency KPIs report; thus, it is likely that many health care professionals not engaged in that field simply ignored the invitation. This, possibly, led to a selection bias, where only more engaged and interested professionals, especially if working in the emergency-urgency field, decided to participate and provided their point of view and opinions. A similar limitation may have affected the FG results; in fact, although we asked each head physician to provide the names of a physician and a nurse, only leading physicians joined the FG, and we missed the point of view of other physicians and nurses, although the head physicians often reported their collaborators’ perceptions. However, the results are both internally consistent and in line with the evidence from the literature; thus, the sample is likely representative of the entire population and, accordingly, the study results reasonably reflect the actual attitudes of health care professionals.
CONCLUSIONS
Novelty, perception, communication, change, data, and pandemics are the primary themes describing the attitude of health care professionals towards an A&F intervention using KPIs. The facilitators and barriers identified, consistent with the scientific literature, are primarily related to education, feedback strategies and the relationships among management, researchers and recipients. Middle and top management should treasure this evidence to reinforce facilitators and remove or minimize hurdles, thus leading to a better acceptance of the KPIs as a method to measure and improve the efficiency of the system. First of all, all health care professionals should be given specific basic tools which allow them: to discriminate when data is reliable, valid and accurate, and when they’re not; to understand what numerator and denominator are used to calculate a specific KPI; to read a KPI estimate and its variability; to interpret simple tables and graphs; to figure out when different KPIs and data can be compared and when comparison is not appropriate. Education would allow health care professionals to adopt a common language and shared knowledge would lead them to be engaged and to play on a common ground with researchers, as well.
Close collaboration between researchers and health care professionals should be ensured when calculation methods are decided, in order to build appropriate and meaningful KPIs, to contextualize them, to understand their significance beyond the numeric value, and to interpret them correctly. Moreover, KPIs should be mostly used to measure the efficiency of processes, procedures and pathways with a planned frequency and only as an afterthought as a budgeting tool. Last but not least, KPIs results should be presented in dedicated interactive meetings where all the actors are invited and involved, and the moderators are preferably colleagues or supervisors; health care professionals should be actively engaged in discussions where data is presented and support for interpretation is given, allowing for discussion and debate. When published materials are also provided, they should be short, easy to read and understand, focused on clear key messages.
Other Information
Funding
This work was funded by the Italian Ministry of Health and by the Region Friuli Venezia Giulia, in the framework of the EASY-NET research program, under Grant NET-2016-02364191.
Conflict of interest statement
The Authors have no conflicts of interest to declare.
Ethical statement
Ethical approval was not required for this study because the online survey collected anonymous data and the entire research collected opinions about organizational issues.
Authors’ contributions
LD: conceptualization, methodology, formal analysis, investigation, data curation, writing – original draft, writing – review and editing, visualization, supervision, project administration; AT: formal analysis, investigation, writing – review and editing; SL: methodology, validation, investigation, writing – review and editing; FV: conceptualization, resources, writing – review and editing, project administration, funding acquisition.
Acknowledgements
We would like to express our sincere gratitude to the clinicians who accepted to participate in the FG or to be interviewed to share their ideas, perceptions and suggestions. We are also grateful to Cinzia Maria Bellettato, Francesco Carli, Letizia Cialli, Paola Corazza, Jessica De Biasi, Antonio Di Capua, Gennaro Di Ruberto, Valentino Facca, Elisa Franco, Nicola Leonardo Galizia, Federica Maria Celeste Gallizia, Elisabetta Gasparotto, Roberta Grando, Cristina Grilli, Luca Giovanni Locatello, Maristella Colette Lo Cicero, Sergio Muiesan, Consuelo Nadalin, Mario D’Oria, Belinda Prelli, Donatella Righini, Enrico Scarpis, Andrea Tandelle, Raffaello Tosolini, and all the health care professionals who decided to help our research by compiling the online questionnaire.
Address for correspondence: Laura Deroma, Dipartimento di Prevenzione, SOC Igiene e Sanità Pubblica, Azienda Sanitaria Universitaria Friuli Centrale, Via Chiusaforte 2, 33100 Udine, Italy. laura.deroma@asufc.sanita.fvg.it
Figures and tables
| # | Question | n | % |
|---|---|---|---|
| D68 | Sex (n=307) | ||
| Male | 89 | 29.0 | |
| Female | 216 | 70.4 | |
| Not binary | 2 | 0.7 | |
| D69 | Age (n=310) | ||
| <25 | 2 | 0.7 | |
| 25-34 | 48 | 15.5 | |
| 35-44 | 77 | 24.8 | |
| 45-54 | 102 | 32.9 | |
| 55-64 | 74 | 23.9 | |
| >=65 | 7 | 2.3 | |
| D70 | Profession (n=311) | ||
| Physician | 61 | 19.6 | |
| Nurse | 142 | 45.7 | |
| Social worker in public health care service | 23 | 7.4 | |
| Clerk | 12 | 3.9 | |
| Other* | 73 | 23.5 | |
| D72 | Do you work in the emergency-urgency setting? (n=307) | ||
| Yes | 65 | 21.2 | |
| No | 175 | 57.0 | |
| No, but I used to | 67 | 21.8 | |
| D73 | What local health care authority do you belong to? | ||
| ASUGI (5954 employees)** | 71 | 22.8 | |
| ASUFC (8500 employees)** | 165 | 52.9 | |
| ASFO (3537 employees)** | 52 | 16.7 | |
| ARCS (182 employees)** | 1 | 0.3 | |
| IRCCS CRO (744 employees)** | 23 | 7.4 | |
| IRCCS Burlo Garofolo (753 employees)** | - | - | |
| *9 laboratory technicians, 5 radiology technicians, 2 physical therapists, 1 biologist, 1 midwife, 1 professional educator, 1 psychologist, 1 speech therapist; **number of employees on 31st December 2022; ASUGI: Azienda Sanitaria Universitaria Giuliano Isontina; ASUFC: Azienda Sanitaria Universitaria Friuli Centrale; ASFO: Azienda Sanitaria Friuli Occidentale; ARCS: Azienda Regionale di Coordinamento per la Salute; IRCCS CRO: Centro di Riferimento Oncologico, Oncology Reference Centre. | |||
| READABILITY | Not at all | Not very | Enough | Very | |
|---|---|---|---|---|---|
| How easy do you find it to…? | |||||
| D2 | Read a report like this (n=79) | 4 (5.1) | 11 (13.9) | 63 (79.8) | 1 (1.3) |
| How clear did you find the…? | |||||
| D6 | Datasheets describing the calculation of the indicators (n=77)D7 | 2 (2.6) | 9 (11.7) | 57 (74.0) | 9 (11.7) |
| D8 | Text (n=78) | 2 (2.6) | 5 (6.4) | 54 (69.2) | 17 (21.8) |
| D10 | Tables (n=75)D11 | 2 (2.7) | 10 (13.3) | 51 (68.0) | 12 (16.0) |
| D12 | Graphs (n=77)D13 | 2 (2.6) | 13 (16.9) | 40 (52.0) | 22 (28.6) |
| INTEREST | Not at all | Not very | Enough | Very | |
| How interesting did you find…? | |||||
| D3 | The subjects it describes (n=79) | 1 (1.3) | 10 (12.7) | 46 (58.2) | 22 (27.9) |
| USEFULNESS | Not at all | Not very | Enough | Very | |
| How useful did you find the…? | |||||
| D14 | Datasheets describing the calculation of the indicators (n=77) | 1 (1.3) | 13 (16.9) | 41 (53.3) | 22 (28.6) |
| D16 | Text (n=78) | - | 8 (10.3) | 45 (57.7) | 25 (32.1) |
| D18 | Tables (n=76) | 2 (2.6) | 11 (14.5) | 41 (54.0) | 22 (29.0) |
| D20 | Graphs (n=76) | - | 14 (18.4) | 37 (48.7) | 25 (32.9) |
| NUMBER OF KPIs | Scarce | Fair | Excessive | ||
| How did you find the number of …? | |||||
| D22 | EMS KPIs (n=76) | 9 (11.8) | 62 (81.6) | 5 (6.6) | |
| D23 | Hospital Emergency System KPIs (n=76) | 8 (10.5) | 65 (85.5) | 3 (4.0) | |
| D24 | Cardiological KPIs (n=76) | 4 (5.3) | 66 (86.8) | 6 (7.9) | |
| D25 | Neurological KPIs (n=75) | 8 (10.7) | 64 (85.3) | 3 (4.0) | |
| D26 | Major traumas KPIs (n=73) | 8 (11.0) | 64 (87.7) | 1 (1.4) | |
| D27 | EMS regional operating centre KPIs (n=74) | 8 (10.8) | 59 (79.7) | 7 (9.5) | |
| NUMBER OF TABLES | Scarce | Fair | Excessive | ||
| D29 | Tables (n=76) | 2 (2.6) | 56 (73.7) | 18 (23.7) | |
| NUMBER OF GRAPHS | Scarce | Fair | Excessive | ||
| D30 | Graphs (n=75) | 3 (4.0) | 59 (78.7) | 13 (17.3) | |
| Additional hints from open questions; D7: better explain captions and acronyms; D11: use ascending/descending rather than alphabetical order; better explain captions and acronyms; highlight significant results; D13: use more contrasting colours, number graphs; KPI: key performance indicator; EMS: Emergency Medical System. | |||||
| # | Question | n | % |
|---|---|---|---|
| D44 | What doctors should be engaged in KPI discussion? (Multiple choice is allowed)D45 | ||
| Department leads | 191 | 60.4 | |
| Unit leads | 222 | 70.3 | |
| Physicians | 224 | 70.9 | |
| None | 20 | 6.3 | |
| D46 | What health care professionals other than doctors should be engaged in KPI discussion? (Multiple choice is allowed)D47 | ||
| Department nurse leads | 178 | 56.3 | |
| Unit nurse leads | 240 | 75.9 | |
| Nurses | 275 | 87.0 | |
| Professionals other than nurses | 147 | 46.5 | |
| None | 6 | 1.9 | |
| D48 | What administrative/organizational professionals should be engaged in KPI discussion? (Multiple choice is allowed)D49 | ||
| Professionals from the Medical Directorate – hospital level | 191 | 60.4 | |
| Professionals from the Medical Directorate – health care service level | 206 | 65.2 | |
| Professionals from the Regional Health Directorate | 168 | 53.2 | |
| Professionals who identified indicators and defined how to calculate them | 193 | 61.1 | |
| Professionals who calculated the indicators | 136 | 43.0 | |
| None | 12 | 3.8 | |
| D50 | At what level would KPI discussion be beneficial? (Multiple choice is allowed) | ||
| Unit | 185 | 58.5 | |
| Department | 193 | 61.1 | |
| Local health care authority | 195 | 61.7 | |
| Region | 134 | 42.4 | |
| National | 70 | 22.2 | |
| International | 25 | 7.9 | |
| Additional hints from open questions; D45: KPI discussion should include doctors working in health care districts and/or General Practitioners (n=2); D47: other professional figures cited are health care assistants (n=1) and nurses involved in quality and risk management (n=1); one respondent also suggested multidisciplinary audits that go beyond hierarchical concepts, also taking into account the patients’, families’ and social environment’s position; in this context, a psychologist could be useful in settling relational dynamics among professionals; D49: management engineers (n=1), the Chief medical officer (n=1), local and regional health care authorities (n=1); KPI: key performance indicator. | |||
| Theme | Online survey Quantitative results | Online survey Open questions (excerpts) | Focus group | Interviews |
|---|---|---|---|---|
| 1 NOVELTY | ||||
| Barriers | ||||
| Facilitators | First experience with kpis in emergency setting | Enthusiastic participation of several health professionals emergency kpis are calculated for the first time | ||
| 2 PERCEPTION | ||||
| Barriers | D52. What do you think are the negative aspects of the indicators? Indicators are a form of judgement of my job 3.5% Indicators are calculated by external people who don’t know reality 45.9% Indicators are not used to improve pathways 36.4% | Healthcare and everyday tasks are much more complex than simple a priori indicators (D42) [Indicators] could be only useful if they were contextualized and the results were used for a serious project for improvement (unfortunately, this is not what I saw in my experience) (D42) I don’t think [indicators] are useful because most of those who should interpret them are not even basically educated to be able to modify them (D42) Those who judge do not work and those who judge work in an office (D53) | KPIs interpreted as marks or judgements kpis are not useful to adjust behaviors KPIs are only necessary to achieve the budget’s objectives and earn benefits | KPIs interpreted as marks or judgements |
| Facilitators | D39. Do you think indicators could be beneficial to improve performance? very much 26.6% somewhat 57.3% D40. Do you think receiving feedback such as the indicators results in your working environment is beneficial to improve the quality of your job? Yes 72.8% D41. How do you think performance indicators could be beneficial? Indicators reveal the aspects that need improvement in everyday work 73.7% Indicators are useful for top and middle management to improve pathways 55.4% Indicators allow comparison with other organisations 52.9% Indicators are objective, thus non-interpretable 21.5% | It should be asked those working on the field what are the actually useful indicators (D42) [Indicators] increase awareness and, thus, the motivation for change (D42) [Indicators are useful because] measuring performance means not only where we can improve but also where we stand out (D42) [Indicators are useful to] guide our own professional behaviour (D42) [Indicators are useful to] align operating modes (D42) [Indicators are useful] to commit appropriate human and instrumental resources (D42) [Indicators] allow me to share possible criticalities to improve performance (D42) | KPIs are essential, especially in time-dependent pathologies in the emergency setting KPIs are a starting point for improvement | |
| 3 COMMUNICATION | ||||
| Barriers | D37. Would you be more interested if the results presentation were different from a written document (e.g., a meeting of professionals, a conference, a training course)? Yes 67.1% | I have the problem of finding the time to read a long and complex text; I am not used to reading such texts, and my reading is slower than it should be (D38) | Hard to give feedback hierarchically Sharing from leads to collaborators was difficult Lack of mediators between clinicians and researchers Labored collaboration between researchers and institutions KPIs interpretation was difficult due to scarce education | |
| Facilitators | D43. Do you think it is useful to discuss the results of the indicators? Yes 82.6% D38. What KPI results sharing mode would you appreciate? Can you explain why? Interactive presentation (education courses, congresses, conferences) 39.9% Summary of the results 7.0% | Dialogue between those professionals who prepared the report/analysed the data and the report recipients could be a great opportunity for discussion (D38) Sharing data makes data knowledge less boring (D38) | A close collaboration between clinicians and researchers is paramount Meetings are beneficial for understanding data, discussing and working together for improvement Education on kpis is beneficial | |
| 4 CHANGE | ||||
| Barriers | Change of the EMS organizational assessment | Change of political government Geographical and organizational modifications in healthcare districts Dismission of the regional committee for urgency and emergency EMS heads turnover Change of the regional triage system Change of the urgency-emergency software Absence of stable institutional and technical reference figures | ||
| Facilitators | [Indicators] are feedbacks that allow to guide necessary changes (D42) If interpreted and tackled with intellectual honesty, KPIs allow us to overcome self-referential and localist reasoning, to offer a service much more oriented to the actual needs of the “stakeholders” (D42) | Willingness to change | ||
| 5 DATA | ||||
| Barriers | D52. What do you think are the negative aspects of the indicators? They are often calculated using too old data 18.0% D56. Do you think the usefulness of the indicators also depends on how old the data used to calculate them is? Yes 59.5% Sometimes 27.2% | Without interpretation, the data is for the sake of it (D53) The data need to be explained (D53) Limitations: actual reliability of data sources, confounding factors, personal interpretation of specific indicators from uneducated professionals, et cetera (D53) The data is often not easily classified in specific categories (D53) | Inadequate data collection Insufficient information technology Privacy and data ownership issues Data is underused by organizations and policymakers | Database change (number of records; name, number, and coding of variables) Absence of a metadata manual Data is difficult to obtain |
| Facilitators | The data in itself is neutral. They need to be collected, analyzed, and interpreted by highly qualified professionals. After publication, dissemination and discussion with collaborators, a debate should origin, open to criticism, motivated doubts, suggestions [omissis]. This data should also be divulged to the general population because, in the end, society itself benefits from the performance improvement. (D53) | Data is a treasure Collaboration with colleagues working in institutional units | ||
| 6 PANDEMICS | ||||
| Barriers | The negative aspect is that the compared data also describes a pandemic year and the emergency-urgency system had this additional load (D53) [Indicators] suffer from bias needs to take into account (e.g., comparison between 2019, before pandemics, and 2020, pandemic year) (D53) Pandemics made data irregular and caused several outliers in many research fields; thus, it is paramount to gradually start collecting data regularly to obtain a picture of the new “norm” (D58) | Pandemics may have lowered the interest in KPIs | ||
| KPI: key performance indicator; EMS: Emergency Medical System. | ||||
pdf
