Why Pragmatic Free Trial Meta May Be More Risky Than You Think
Pragmatic Free Trial Meta
Pragmatic Free Trail Meta is an open data platform that allows research into pragmatic trials. It gathers and distributes clean trial data, ratings and evaluations using PRECIS-2. This allows for a variety of meta-epidemiological analyses that examine the effect of treatment across trials of various levels of pragmatism.
Background
Pragmatic trials are increasingly recognized as providing real-world evidence for clinical decision making. The term "pragmatic" however, is used inconsistently and its definition and measurement need further clarification. Pragmatic trials must be designed to inform policy and clinical practice decisions, not to confirm a physiological or clinical hypothesis. A pragmatic trial should try to be as close as is possible to the real-world clinical practice which include the recruitment of participants, setting up, delivery and execution of interventions, determination and analysis outcomes, and primary analyses. This is a major distinction from explanatory trials (as described by Schwartz and Lellouch1) which are designed to provide more complete confirmation of the hypothesis.
The trials that are truly practical should not attempt to blind participants or clinicians as this could lead to distortions in estimates of the effects of treatment. Pragmatic trials will also recruit patients from different health care settings to ensure that the results can be generalized to the real world.
Additionally, clinical trials should focus on outcomes that matter to patients, like quality of life and functional recovery. This is particularly important for trials that involve surgical procedures that are invasive or may have dangerous adverse effects. The CRASH trial29 compared a 2 page report with an electronic monitoring system for hospitalized patients with chronic heart failure. The catheter trial28 on the other hand was based on symptomatic catheter-related urinary tract infection as the primary outcome.
In addition to these features, pragmatic trials should minimize the trial procedures and data collection requirements in order to reduce costs. Furthermore pragmatic trials should try to make their results as applicable to clinical practice as is possible by making sure that their primary method of analysis is the intention-to-treat approach (as described in CONSORT extensions for pragmatic trials).
Many RCTs that do not meet the criteria for pragmatism, but have features that are contrary to pragmatism have been published in journals of various types and incorrectly labeled pragmatic. This could lead to false claims of pragmatism, and the use of the term should be standardized. The development of a PRECIS-2 tool that offers an objective, standardized assessment of pragmatic features is a good start.
In a practical study it is the intention to inform clinical or policy decisions by showing how an intervention could be integrated into routine care in real-world contexts. Explanatory trials test hypotheses about the cause-effect relation within idealized settings. In this way, pragmatic trials can have less internal validity than explanation studies and be more susceptible to biases in their design analysis, conduct, and design. Despite their limitations, pragmatic studies can be a valuable source of information to make decisions in the healthcare context.
The PRECIS-2 tool scores an RCT on 9 domains, with scores ranging between 1 and 5 (very pragmatist). In this study the areas of recruitment, organization, flexibility in delivery, flexibility in adherence, and follow-up received high scores. However, the primary outcome and the method of missing data was scored below the pragmatic limit. This suggests that a trial could be designed with good practical features, but without compromising its quality.
It is difficult to determine the degree of pragmatism that is present in a trial because pragmatism does not have a binary characteristic. Certain aspects of a research study can be more pragmatic than others. A trial's pragmatism can be affected by modifications to the protocol or logistics during the trial. Koppenaal and colleagues found that 36% of 89 pragmatic studies were placebo-controlled or conducted prior to licensing. They also found that the majority were single-center. They are not close to the norm and are only referred to as pragmatic if their sponsors agree that such trials are not blinded.
Another common aspect of pragmatic trials is that the researchers try to make their results more valuable by studying subgroups of the trial. However, this can lead to unbalanced comparisons with a lower statistical power, increasing the likelihood of missing or misinterpreting the results of the primary outcome. This was a problem in the meta-analysis of pragmatic trials because secondary outcomes were not corrected for covariates that differed at the baseline.
Furthermore practical trials can present challenges in the collection and interpretation of safety data. This is due to the fact that adverse events are usually self-reported, and therefore are prone to errors, delays or coding errors. Therefore, it is crucial to improve the quality of outcome for these trials, ideally by using national registries instead of relying on participants to report adverse events on the trial's own database.
Results
Although the definition of pragmatism does not require that all trials be 100% pragmatic, there are advantages to including pragmatic components in clinical trials. These include:
Incorporating routine patients, the results of trials can be more quickly translated into clinical practice. But pragmatic trials can have disadvantages. For instance, the right type of heterogeneity can help the trial to apply its findings to a variety of patients and settings; however, the wrong type of heterogeneity could reduce assay sensitivity and therefore reduce the power of a study to detect small treatment effects.
Several studies have attempted to classify pragmatic trials using different definitions and scoring methods. Schwartz and Lellouch1 have developed a framework that can distinguish between explanatory studies that confirm a physiological hypothesis or clinical hypothesis and pragmatic studies that inform the choice for appropriate therapies in the real-world clinical practice. The framework was comprised of nine domains that were scored on a scale ranging from 1-5, with 1 indicating more explanatory and 5 suggesting more pragmatic. The domains included recruitment, setting up, delivery of intervention, flex adherence and primary analysis.
The original PRECIS tool3 had similar domains and scales from 1 to 5. Koppenaal and colleagues10 developed an adaptation of this assessment called the Pragmascope that was simpler to use in systematic reviews. They discovered that pragmatic reviews scored higher on average in all domains, but scored lower in the primary analysis domain.
The difference in the primary analysis domains could be explained by the way that most pragmatic trials approach data. Some explanatory trials, however, do not. The overall score for pragmatic systematic reviews was lower when the domains of management, flexible delivery and following-up were combined.
It is crucial to keep in mind that a pragmatic study should not mean a low-quality trial. In fact, there are a growing number of clinical trials that employ the word 'pragmatic,' either in their abstract or title (as defined by MEDLINE, but that is neither precise nor sensitive). These terms may indicate that there is a greater appreciation of pragmatism in abstracts and titles, but it's unclear whether this is reflected in the content.
Conclusions
As the value of real-world evidence grows commonplace and pragmatic trials have gained momentum in research. They are randomized clinical trials that evaluate real-world alternatives to care instead of experimental treatments in development, they involve patient populations which are more closely resembling the patients who receive routine medical care, they utilize comparators that are used in routine practice (e.g. existing drugs), and they depend on the self-reporting of participants about outcomes. This method can help overcome the limitations of observational studies, such as the limitations of relying on volunteers and limited accessibility and coding flexibility in national registries.
Pragmatic trials have other advantages, such as the ability to draw on existing data sources and a higher probability of detecting meaningful differences than traditional trials. However, they may be prone to limitations that compromise their validity and generalizability. The participation rates in certain trials could be lower than expected because of the healthy-volunteering effect, financial incentives or competition from other research studies. The need to recruit individuals in a timely manner also restricts the sample size and the impact of many practical trials. In addition certain pragmatic trials do not have controls to ensure that the observed differences aren't due to biases in the conduct of trials.
The authors of the Pragmatic Free Trial Meta identified 48 RCTs self-labeled as pragmatic and were published until 2022. The PRECIS-2 tool was employed to assess the pragmatism of these trials. It includes areas such as eligibility criteria as well as recruitment flexibility as well as adherence to interventions and follow-up. They discovered 14 trials scored highly pragmatic or pragmatic (i.e. scoring 5 or higher) in at least one of these domains.
Trials that have a high pragmatism score tend to have higher eligibility criteria than traditional RCTs which have very specific criteria that aren't likely to be found in the clinical setting, and include populations from a wide range of hospitals. The authors claim that these characteristics could make the pragmatic trials more relevant and relevant to daily practice, but they do not guarantee that a pragmatic trial is free of bias. In addition, the pragmatism that is present in trials is not a fixed attribute; a pragmatic trial that doesn't have all the characteristics of a explanatory trial may yield reliable and relevant results.