Share on
INTRODUCTION
“Medical research is subject to ethical standards that promote and ensure respect for all human subjects and protect their health and rights” [1].
The need of evaluation to always guarantee ethically justified research, in accordance to the Declaration of Helsinki [1], applies also to observational studies, in which subject may in any case be at risk of physical or psychological harm; even research limited to an examination of existing records, in fact, may entail a risk for the group under investigation (such as stigmatization) or it may harm people by making use of information that they regard as private. Therefore, study proposal involving human subjects must be submitted to at least one Ethics Committee (EC), and the investigators need their approval or clearance before starting the research [2].
Moreover, a valid and robust study protocol is the basis for reliable research, in particular it has to respect epidemiological principles of study design and it allows to guarantee transparency regarding methodologies used: bad science is at least poor if not bad ethics [3].
Considering all this, the value of ethical evaluation before entering into any administrative agreement is undeniable, and, on top of that, the importance of careful planning, open discussions with all concerned parties and vigorous efforts to protect confidential data, as part of good study design. But how to merge an ethical guarantee and protection of subjects’ privacy with an effective and efficient authorisation process? Excessive bureaucracy, in fact, can potentially turn unethical as well as detrimental to the competitiveness of research [4-6].
In this context, the recent publication of Italian Medicines Agency (Agenzia Italiana del Farmaco, AIFA) guidelines on observational studies [7] stimulates a broader reflection on its real impact, also considering that observational studies – according to the increasing interest for real world data and realworld evidence – represent a significant and continuously increasing fraction of clinical research in Italy. A survey conducted in 2019 by the National Coordination Center for Ethics Committees (Centro di Coordinamento Nazionale dei Comitati Etici, CCNCE) showed that out of approximately 14,800 studies examined in one year by 74 ECs, approximately 50% were observational studies (7,400 studies). Among observational studies, retrospective studies represented on average more than 50% of the studies submitted to the opinion of the ECs. Out of any doubt, observational research strongly impact on ECs’ activity and EC’s activities affect a large proportion of clinical research in Italy. During the last 20 years the Italian legislation regarding observational studies has significantly changed: from a dramatic gap at the end of the nineties to an increasing attention to regulate both ethical evaluations and data privacy management [8-16]. However, no evidence is available regarding real world impact of new regulations on the ethical evaluation processes nor on factors which can impact EC’s evaluation time.
The aim of the present work is to investigate, by means of a “before and after” study design, the impact of two distinct important regulations affecting observational studies in Italy entered in force in 2008 and 2018, respectively the AIFA guidelines for the classification and conduct of pharmaceutical observational studies [9] (AIFA guideline) and the European Union General Data Protection Regulation (GDPR) [15] adopted in Italy as for the Legislative Decree 101 of August 10th 2018 [16] (GDPR regulation). Main investigated outcomes were: 1) the duration of the ethical evaluation processes and 2) their complexity, measured using the surrogate end point of the number of suspensions observed in each single evaluation process. As a secondary outcome we aimed to investigate whether the duration and the probability of suspensions were somehow influenced by certain intrinsic characteristics of the evaluated studies.
To our knowledge, this is the first study investigating the impact of a new legislation on a large sample of ethical evaluations. This study was conducted with the aim to deriving valuable information about the evolution of ethical evaluation of observational studies in Italy and drawing insights for how to better evolve the current legislation.
MATERIALS
In this study we analysed two proprietary databases of Medineos srl, a company subject to the direction and coordination of IQVIA Solutions, an Italian contract research organization specialized in the design and executing of clinical observational studies. The first database contained qualitative data related to all studies conducted since 2002; the second database contained quantitative data on the ethical evaluation processes carried out by ECs on each single study. Therefore, we were able to evaluate 2,875 ethical submissions related to 112 different observational studies submitted over a period of 20 years (2002-2022) considering all the following informative contents:
- “project”: each study protocol submitted for evaluation to the Italian ECs;
- “EC”: the body that performed the evaluation of the project in that specific period of time. Some ECs may no longer exist when this paper is written, as a consequence of new regulations in Italy;
- “evaluation”: the process executed by each single EC to evaluate a project. During the examined period (2002-2022), very often the same study protocol was evaluated by the EC of each single clinical site participating to the project;
- “date of submission”: date in which the study protocol of a project was submitted for evaluation to one or more ECs. This milestone was used as the start date of the evaluation process;
- “suspension”: it represents the dichotomous variable defining whether or not the evaluation of an EC undergone a suspension due to a request for further information. The variable “suspension” was used as a surrogate endpoint to define the “complexity” of the individual ethical evaluation process. Indeed, we have assumed that the lack of suspension is equivalent to a more linear evaluation process, while the presence of at least one suspension represents a sign of greater complexity in the evaluation of the single study protocol. For the purposes of our study, complexity does not have a negative meaning, as the absence of a suspension could theoretically also mean greater approximation or superficiality in evaluating the study;
- “date of response”: the date of the final evaluation by the single EC, being this “positive” or “negative”. This variable was used as the end date of the evaluation process in order to measure its duration.
Moreover, we examined some additional qualitative information about specific characteristics of the projects to investigate their role as potential “risk factors” for more complexity in the evaluation process. The intrinsic, qualitative factors examined were:
- “study design”: according to the methodological classification of observational studies, (i) retrospective/primary data collection; (ii) prospective/secondary use of data or hybrid;
- “rare disease”: representing the dichotomous variable defining whether the project was on a rare disease or not;
- “genetic data”: representing the dichotomous variable defining if the project collects genetic data or not;
- “PASS study”: representing the dichotomous variable defining whether the project consists of a so-called post authorization safety study (PASS) or not.
Finally, to answer the research questions of this study and to stratify the total sample in the two arms “before and after”, we calculated the following derived variables:
- “total evaluation time”: calculated as the number of days between the “date of submission” and the “date of response”;
- “pre/post-AIFA”: which represents the dichotomous variable defining whether the project was submitted before or after the publication of the AIFA “guidelines for observational studies on drugs” on March 30th 2008;
- “pre/post-GDPR”: representing the dichotomous variable defining whether the project was submitted before or after the “GDPR” on May 25th 2018.
Only the submissions with all the listed above available and reliable information were considered for the analyses.
METHODS
We performed a descriptive analysis using absolute and relative frequencies for categorical variables, and mean, standard deviation, median, 25th and 75th percentiles, minimum, and maximum for continuous variables. We then analyzed the “total evaluation time” as a time-to-event variable within the context of survival analysis modeling. The database comprised various “projects”, mainly multicentric studies, each with multiple submissions of the same study protocol to different ECs.
Subsequently, we analyzed the probability of receiving a suspension from an EC using a univariate linear logit mixed effects model. The variables hypothesized to impact this probability included “Ethics Committee”, “study design”, “rare disease”, and “genetic data” as project characteristics, and “post-AIFA” and “post-GDPR” as indicators of legislative changes. We hypothesized that both the project characteristics and the legislative changes could affect the probability of receiving a suspension (yes/no). By including all submissions of the same project as a mixed effect variable in the model, we accounted for potential intercorrelation within the same project.
RESULTS
In total, we analysed 112 different observational studies (“projects”), with 2,875 different individual submissions to Italian ECs, over a period from October 2002 to April 2022.
Projects characteristics
Of these projects, 84 (75%) had a prospective or hybrid study design and 28 (25%) retrospective; 7 (6%) were studies on a rare disease, 2 (2%) collected genetic data and 19 (17%) were PASS (Table 1). The mean number of sites per single project was 12 (Standard Deviation, SD 8.3) with a median value of 11. Overall, 87 (77.7%) studies had at least one submission “post-AIFA” and 25 (22.3%) with at least one submission “post-GDPR”.
Submissions characteristics
Considering the submission processes, 2,075 (72.2%) were performed “post-AIFA” and 416 (14.4%) submissions were performed “post-GDPR”.
Total evaluation time
The median (and interquartile range) of “total evaluation time” for submissions that did not experience any suspension was 68 (41-113) days whereas it was 127 (84-180) days for the submissions that underwent a suspension (Table 2).
The median (and interquartile range) for submissions “pre-AIFA” was 70 (41-133) days whereas it was 75 (45-122) days for “post-AIFA” (Table 2). The median (and interquartile range) for submissions “pre-GDPR” was 70 (42-123) days whereas it was 90 (63-140) for “post-GDPR” (Table 2).
When time to evaluation was assessed by means of a Kaplan-Meier analysis, this difference was confirmed both when taking into account pre/post-AIFA (Figure 1) and pre/post-GDPR (Figure 2). Namely, the time to evaluation for 50% of responses was shorter after AIFA guidelines and longer after GDPR.
Probability for suspension of the evaluation process
Only the factor “post-AIFA” was recognized as influential in increasing the likelihood of receiving a suspension (OR=21; 95% CI 7; 60): the odds to receive a suspension for a “post-AIFA” study was on average between 7 and 60 times higher than the odds for a “pre-AIFA” study.
Any of the other pre-identified intrinsic, qualitative factors (“study design”, “PASS study”, etc.) does not seem to be associated to the occurrence of suspension (Table 3).
DISCUSSION
Before considering the results related to evaluation times, it is important to highlight that 67% of local ECs that received at least one submission during the observed timeframe no longer exist today, having been changed or replaced. This percentage rises to 81% when focusing on submissions prior to the 2008 AIFA guidelines. These figures provide remarkable evidence of the structural changes that have occurred in recent years, in terms of EC reorganization and procedural updates. It is also noteworthy that a transformative phase is still ongoing, with new AIFA guidelines, including instructions for different observational study designs, anticipated as per the Ministerial Decree of 30 November 2021 [13].
Impact of AIFA guideline
When considering our sample in its entirety – both suspended and non-suspended protocols – the AIFA guidelines led to a slight increase in the median number of days for evaluation (from 70 to 75 days).
On the other hand, the adoption of the AIFA guidelines was recognized as highly influential in increasing the likelihood of receiving a suspension. Our interpretation is that the guidelines provided clear rules and requirements for observational studies, leading ECs to exercise a higher level of control, thereby increasing suspensions. However, suspensions can be seen as indicative of ECs needing to request additional information when evaluating submissions that did not fully meet AIFA specifics, and suspension itself can be viewed as a “positive” benchmark for improving the quality of observational studies.
As a consequence, the increase in suspensions could also explain why the expected 60-day evaluation time is not usually respected. This issue is not limited to the immediate period following the introduction of the AIFA guidelines but persists over a longer timeframe (15 years).
According to the AIFA guidelines, the maximum expected evaluation time for a satellite center’s EC evaluation should be no more than 75 days (45 days waiting for the coordinating center’s EC opinion plus 30 days for the local evaluation in the satellite center). However, even for studies that did not receive a suspension, we note that both before and after the publication of the AIFA guidelines, 45% and 43% of submissions, respectively, exceeded this maximum limit of 75 days. Data related to the AIFA guidelines can be interpreted as evidence of the resolution transposition by ECs in terms of the technical evaluation of observational studies. The AIFA guidelines introduced a set of rules on the classification, planning, and conduct of observational studies in pharmacological research, aiming to make a significant contribution to improving the quality assurance of all observational studies [13]. According to the AIFA guidelines, each observational study must be based on a defined protocol, which must include: the research hypothesis, expected results, type of observational study, choice of sample size, information to be collected, possible involvement of the facility and/or healthcare professionals, required resources, origin of funding, modalities of participation, and information addressed to the patients [4]. Despite these widely agreed-upon and accepted rules, a survey conducted by Gregori et al. [17] on 6 ECs right after the publication of the AIFA guidelines, for a post-hoc comparison of 364 protocols presented as observational before March 2008, revealed that a fairly high percentage (20-40%) did not comply with the new specifics introduced by the AIFA guidelines. This may be interpreted as proof of the past need for enhancement in protocol quality, which the AIFA sought to address by defining a minimum set of common requirements for all ECs.
Impact of GDPR legislation
Considering the introduction of GDPR legislation, we observed a significant increase in evaluation time (from 70 to 90 days) without a corresponding increase in the probability of suspensions. This controversial result could be interpreted as the need for privacy evaluations by other competent bodies at the local level, impacting the evaluation duration but not resulting in suspensions solely due to privacy issues. In other words, the examination of privacy concerns appears to be a bureaucratic activity leading to minimal adjustments (e.g., modifications to informed consent at the local level), but not necessitating the suspension of the study due to the intrinsic characteristics of the study protocol.
Contrary to our observations, Benfatto et al. [18] highlighted the valuable contribution of GDPR in reducing the number of changes necessary for final submission approval and saving time in 822 clinical trial protocols. It is worth noting that the cited review focused on experimental clinical trials, whereas our analysis specifically refers to observational study protocols, with more than 25% being retrospective studies. This essential difference suggests that the observed increase in evaluation time might be linked to the observational nature of the studies, including retrospective designs. Retrospective studies require a higher effort to evaluate privacy aspects, such as data collection from deceased or untraceable patients for whom consent is not achievable.
In this regard, the Italian National Coordination Centre for Ethics Committees (CCNCE) recently released a document specifically addressing the ethical and regulatory issues in handling personal health data in observational research [19], further supporting our hypothesis.
Impact of intrinsic study factors
Regarding the intrinsic factors of the study, we did not observe any impact on evaluation duration or the probability of suspension. This suggests that our findings on the timing and complexity of ethical assessments are generally linked to the observational nature of the protocols analyzed, rather than the inherent characteristics of individual studies.
It is surprising that the inclusion of genetic data in the protocol did not result in an increased probability of suspension, considering that the AIFA guideline states: “Observational studies are not considered to be those in which the examinations are aimed at pharmacogenetic and/or pharmacogenomic studies” [9]. This peculiarity highlights the gap that still exists between a methodologically inadequate regulatory definition of observational studies and the practical approach applied by both researchers and evaluators.
Our study has some limitations: (i) we used a surrogate outcome (suspension) to define evaluation complexity; (ii) we assumed that suspensions indicate a deeper evaluation of study protocols by ECs, implying that an increase in suspensions represents an enhanced ability of ECs to evaluate observational study protocols. Additionally, our study was conducted using a database of sponsored observational studies from a single contract research organization, representing a partial view of Italian practice. However, we believe that the large dataset provides a good approximation of the trends in observational research in Italy over the past 20 years.
In addition, the replicability of these results strongly depends on legislative conditions [20]. A study proposal involving human subjects must be submitted to at least one EC for approval before research can begin. A valid and robust study protocol, adhering to epidemiological principles, ensures transparency and reliability. We believe that having study designs and protocols reviewed by peers is a recommended practice to identify potential flaws or areas for improvement. It is beneficial to have peers review the study from various perspectives, such as statistical, operational, methodological, and quality aspects. This multi-faceted review process can help identify potential issues and improve the overall robustness of the research. The new AIFA guidelines could potentially alter the timelines of the authorization process, which will be further investigated in the future.
CONCLUSIONS
Our findings show that changes in the regulatory environment significantly impact the conduct of clinical research. Specifically, we observed that both the AIFA guideline and GDPR legislation have controversial effects, increasing evaluation times but also apparently enhancing the thoroughness of ethical evaluations.
The results of this analysis serve as valuable indicators for assessing the effectiveness of past Italian regulations on observational studies, particularly in terms of ethical authorization timing and its consequential effect on the competitiveness of Italian research globally. These findings also provide a useful reference for measuring the impact of the new AIFA guidelines in the near future. Indeed, these new guidelines have been eagerly awaited by the local scientific community, with the hope that they will enhance the competitiveness of Italian clinical research in the field of real-world evidence generation while ensuring high standards of ethics and scientific quality.
Ethical evaluations naturally require time. To expedite the process, it might be beneficial to organize more frequent meetings of ethics committees. While this could increase costs, as committee members need to be compensated, it could ultimately speed up the evaluation process. We are confident that in the future, cost reductions can be achieved through the use of remote meetings and technological systems. Our research does not imply that the work of ethics committees delays research.
In particular, scientific societies highlighted to the legislator the main priorities: the need for updated methodological references and the establishment of standard processes with clearly defined rules and timelines. This would promote uniformity not only for observational studies on drugs but also for all types of studies (medical devices, other therapies, disease, and epidemiology) using observational methodology.
The results of this study can therefore be used to help evaluate the impact of the new guidelines in the coming months, considering the recent major transformation in the organization of Territorial Ethics Committees (TECs). This evaluation will help determine the need for further actions to harmonize clinical research based on observational methods in Italy. No more time can be wasted if Italy aims to secure a leading role in European research based on real-world data.
Other Information
Conflict of interest statement
The Authors declare no conflict of interest.
Address for correspondence: Alessandra Ori, Real-World Solution, IQVIA Solutions, Via Ferrari 143, 41121 Modena, Italy. E-mail: alessandra.ori@iqvia.com.
Figures and tables
Figure 1. Kaplan-Meyer curve of time of evaluation according to the Italian Medicines Agency (Agenzia Italiana del Farmaco, AIFA) guideline emission.
Figure 2. Kaplan-Meyer curve of time of evaluation according to General Data Protection Regulation (GDPR) legislation application.
| Projects analysed (N=112) | |
|---|---|
| Study design | |
| Prospective/hybrid | 84 (75%) |
| Retrospective | 28 (25%) |
| Characteristics | |
| Project on rare disease | 7 (6%) |
| Project with genetic data collection | 2 (2%) |
| Post-authorisation safety studies | 19 (17%) |
| Factors | N | Median (25-75 percentile) | Min-Max | |
|---|---|---|---|---|
| Suspensions during EC evaluation | No | 2,503 | 68.0 (41.0-113.0) | 1.0-558.0 |
| Yes | 372 | 127.0 (84.0-180.0) | 15.0-629.0 | |
| 2008 AIFA guideline | Before | 800 | 70.0 (41.0-133.0) | 1.0-494.0 |
| After | 2,075 | 75.0 (45.0-122.0) | 4.0 629.0 | |
| GDPR legislation | Before | 2,459 | 70.0 (42.0-123.0) | 1.0-629.0 |
| After | 416 | 90.5 (63.0-140.0) | 10.0-619.0 | |
| EC: Ethical Committee; AIFA: Italian Medicines Agency (Agenzia Italiana del Farmaco); GDPR: General Data Protection Regulation. | ||||
| Parameter | OR | 95% CI | ||
|---|---|---|---|---|
| AIFA guideline | after vs before | 20.69 | 7.05 | 60.75 |
| GDPR legislation | after vs before | 0.98 | 0.38 | 2.53 |
| Study design | Prospective/hybrid vs retrospective | 0.80 | 0.32 | 1.98 |
| Genetic data collection | Yes vs No | 1.00 | 0.07 | 14.74 |
| Rare disease study | Yes vs No | 0.52 | 0.11 | 2.54 |
| CI: confidence interval; AIFA: Italian Medicines Agency (Agenzia Italiana del Farmaco); GDPR: General Data Protection Regulation. | ||||
PDF
