• Clinical Practice and Guidance Center

    landing-header-kc-guides1_37748_17213
  • Summary: Methodology for Updates to the ACOEM Practice Guidelines 

    As part of the update process, ACOEM adopted a new more meticulous strength-of-evidence rating methodology. The enhanced methodology incorporates the highest scientific standards for reviewing evidence-based literature, thus ensuring the most rigorous, reproducible, and transparent occupational health guidelines available.

    Under the new methodology, the process begins with the systematic identification of high-quality original research studies on a topic. References are identified from a number of national and international databases of original research. Additional references are identified through an exhaustive “hand search” of the literature by trained health science researchers. Studies are then graded for actual design and for execution of that design and the subsequent analyses of results. Evidence with the highest available ranking—e.g., all randomized controlled trials (RCTs) or randomized crossover trials for treatment studies—is selected. Each article that meets inclusion criteria is reviewed and critically appraised.

    As an example, RCTs that meet inclusion criteria are scored on 11 criteria. Each criterion is scored 0.0, 0.5 or 1.0. These individual ratings are summed up, resulting in an overall rating that ranges from 0 to 11 

    Criteria Rating Description

    Randomization

    Assessment of the degree that randomization was both reported to have been performed and successfully achieved through analyses of comparisons of variables between the two groups.

    Treatment Allocation Concealed

    Concealment of the allocation scheme from all involved, not just the patient.

    Baseline Comparability

    Measurement of how well the baseline groups are comparable (e.g., age, gender, prior treatment).

    Patient Blinded

    Blinding of the patient/subject to the treatment administered.

    Provider Blinded

    Blinding of the provider to the treatment administered.

    Assessor Blinded

    Blinding of the assessor to the treatment administered.

    Co-interventions Avoided

    Assessment of the degree to which the study design avoided multiple treatments. This includes either a study design that includes combinations of interventions (e.g., a combination of exercise and anti-inflammatory medication) or patient self-administration of other treatments that may plausibly alter the results.

    Compliance Acceptable

    Measurement of the degree of non-compliance.

    Dropout Rate

    Measurement of the drop-out rate.

    Timing of Assessments

    Assessment of whether the timing of measurements of effects is the same between treatment groups.

    Analyzed by Intention to Treat

    Ascertainment of whether the study was analyzed with an intent-to-treat analysis.

     

    The rating for each article is then converted into a quality grade—low quality (0-3.5), moderate quality (4.0-7.5), or high quality (8.0-11.0).

    While literature searches also seek systematic reviews and meta-analyses, on critical appraisal, very few of these secondary studies are truly systematic as the term is used in the evidence-based medicine literature. Most typically, there are errors in analyses or interpretation. For this reason, ACOEM relies primarily on the original literature as the source for its evidence syntheses and recommendations.

    Acceptable studies are abstracted into evidence tables that include details of study methods, outcomes, and statistical analyses. Research staff then use the tables to grade the strength of evidence in order to draft specific clinical practice recommendations that will be combined into collective evidence-based guidelines. Evidence is drawn almost entirely from original research studies. Panels of experts (Evidence-based Practice Panels) then review the draft strength of evidence ratings and recommendations, modify them within the rules of this methodology, and develop final recommendations. 

    Strength-of-evidence ratings are categorized as A, B, C, or I:

    A

    Strong Evidence-base: Two or more high-quality studies.1 

    B

    Moderate evidence-base: At least one high-quality study or multiple moderate-quality studies2 relevant to the topic and the working population.

    C

    Limited evidence-base: At least one study of moderate quality.

    I

    Insufficient Evidence: Evidence is insufficient or irreconcilable.


    DEVELOPMENT OF RECOMMENDATIONS 

    In reviewing or revising recommendations, the expert Panels review the articles, evidence tables, and strength-of-evidence ratings (A, B, C, or I). Panels discuss recommendations for diagnosis or treatment based on the critically appraised body of evidence using a “best evidence” approach.  

    In addition to critically appraised evidence, “first principles” of medical logic and ethics are observed in formulating recommendations:

    • Imaging or testing should generally be done to confirm a clinical impression.
    • Tests should affect the course of treatment.
    • Treatments should improve on the natural history of the disorder, which in many cases is recovery without treatment.
    • Invasive treatment should be preceded by adequate conservative treatment and may be performed if conservative treatment does not improve the health problem.
    • The more invasive and permanent, the more caution should be exerted in considering invasive tests or treatments and the stronger should be the evidence of efficacy.
    • The more costly the test or intervention, the more caution should be generally exerted prior to ordering the test or treatment and the stronger should be the evidence of efficacy.
    • Testing/treatment decisions should be a collaboration between the clinician and patient with full disclosure of benefits and risks.
    • Treatment should not create dependence or functional disability. 

    Health benefits, side effects, and risks are explicitly considered and discussed in formulating recommendations. Benefits should significantly exceed risks. Each recommendation specifies the clinical problem to which it relates and is linked to the relevant higher quality available evidence. Consensus recommendations, following the first principles above, are formulated when there is either a lack of quality evidence or the available evidence substantially conflicts. The ACOEM evidence-based recommendations are explicitly classified as follows:

    Recommendation Category Evidence Rating Description of Category

    Strongly Recommended 

    A 

    The intervention is strongly recommended for appropriate patients.

    The intervention improves important health and functional outcomes based on high quality evidence, and the Evidence-based Practice Panel (EBPP) concludes that benefits substantially outweigh harms and costs. 

    Moderately Recommended 

    B 

    The intervention is recommended for appropriate patients.

    The intervention improves important health and functional outcomes based on moderate quality evidence that benefits substantially outweigh harms and costs. 

    Recommended 

    C 

    The intervention is recommended for appropriate patients.

    There is limited evidence that the intervention may improve important health and functional benefits.

    Insufficient Recommended (Consensus-based) 

    I 

    The intervention is recommended for appropriate patients and has nominal costs and essentially no potential for harm. The EBPP feels that the intervention constitutes best medical practice to acquire or provide information in order to best diagnose and treat a health condition and restore function in an expeditious manner. The EBPP believes based on the body of evidence, first principles, and/or collective experience that patients are best served by these practices, although the evidence is insufficient for an evidence-based recommendation. 

    InsufficientNo Recommendation (Consensus-based) 

    I 

    The evidence is insufficient to recommend for or against routinely providing the intervention. The EBPP makes no recommendation. 

    Evidence that the intervention is effective is lacking, of poor quality, or conflicting and the balance of benefits, harms, and costs cannot be determined. 

    Insufficient
    Not Recommended
    (Consensus-based) 

    I 

    The evidence is insufficient for an evidence-based recommendation. 

    The intervention is not recommended for appropriate patients because of high costs/high potential for harm to the patient. 

    Not Recommended 

    C 

    Recommendation against routinely providing the intervention.

    The EBPP found at least moderate evidence that harms and costs exceed benefits based on limited evidence.

    Moderately
    Not Recommended
     

    B 

    Recommendation against routinely providing the intervention to eligible patients. 

    The EBPP found at least moderate evidence that the intervention is ineffective, or that harms or costs outweigh benefits. 

    Strongly Not Recommended 

    A 

    Strong recommendation against providing the intervention to eligible patients. 

    The EBPP found high quality evidence that the intervention is ineffective, or that harms or costs outweigh benefits. 

     


    1For therapy and prevention, randomized controlled trials (RCTs) with narrow confidence intervals and minimal heterogeneity.  For diagnosis and screening, cross sectional studies using independent gold standards.  For prognosis, etiology or harms, prospective cohort studies with minimal heterogeneity.
    2For therapy and prevention, a well-conducted review of cohort studies. For prognosis, etiology or harms, a well-conducted review of retrospective cohort studies or untreated control arms of RCTs.