• Promoting a Healthier Workforce

    Health and Productivity Management

    landing-header-kc-hpm1_37748_17213
  • Health and Productivity Management Center

    Impact of Illness and Its Treatment on Workplace Costs: Regulatory and Measurement Issues

    JOEM/January 2001

    Paul E. Greenberg, MA, MS; Howard G. Birnbaum, PhD; Ronald C. Kessler, PhD, Michael Morgan, MS; Paul Stang, PhD

    In an attempt to document a broader spectrum of the benefits of their pharmaceutical products, drug companies increasingly seek to include productivity claims in their promotional campaigns. We describe the existing regulatory framework of the Food and Drug Administration (FDA) for considering productivity claims, distinguishing between the traditional “substantial evidence” standard and the “competent and reliable scientific evidence” standard. But the notion of competent and reliable scientific evidence may itself be problematic, even when it is the appropriate regulatory standard, because there exists no consistent measurement approach across diseases, workplaces, jobs, and worker capabilities that are widely accepted in this emerging area of health outcomes research. We examine the various measurement approaches that have been used to quantify the impact of illness and its treatment on workplace productivity, and we describe some of the shortcomings associated with each alternative. This discussion highlights the possible difficulties faced by the FDA in reviewing productivity-based promotional claims. Finally, we suggest possible strategies for furthering this field of investigation. (J Occup Environ Med. 2001;43:56 – 63)

    In an attempt to document a broader spectrum of the benefits of their pharmaceutical products, drug companies increasingly seek to include productivity claims in their promotional campaigns. Some investigators have developed evidence of the impact of treatment on work performance, such as how much an employer can save in reduced absenteeism after the use of one product compared with another (eg, “Drug X will get your employees back to work faster compared with drug Y”). Others have aimed their productivity claims directly at employees, who are now being seen as advocates for their own health (eg,”In my company, I can't afford a week away from work” or “Less drowsiness will mean I can be more productive”).

    Given these initiatives, the Food and Drug Administration (FDA) increasingly must regulate productivity claims for use in labeling and advertising materials, even though the scientific standards for such claims are still evolving. The objective of the FDA is to ensure that the promotional materials on safety and efficacy that are transmitted from manufacturers to prescribers, payers, and patients are neither false nor misleading. Even if the intended claim appears to be based on empirical evidence (eg, from clinical trials or administrative claims data), the soundness of the underlying methodologies and the generalizability of the results must be considered.

    In this article, we describe the FDA’s existing regulatory framework for considering productivity claims, distinguishing between the traditional "substantial evidence" standard and the more recent "competent and reliable scientific evidence" standard. Which of these regulatory requirements is operative for any specific productivity claim depends on the intended use of the information generated (eg, for clinical effectiveness claims versus economic support) and the target audience (eg, formulary committee or consumers in general). But the notion of competent and reliable scientific evidence may itself be problematic, even when it is the appropriate regulatory standard, because there exists no consistent, widely accepted measurement approach across diseases, workplaces, jobs, and worker capabilities in this emerging area of health outcomes research. Given these uncertainties, we examine the various measurement approaches for quantifying the impact of illness and its treatment on workplace productivity, and we describe some of the shortcomings associated with each alternative. The discussion highlights some of the possible difficulties faced by the FDA when reviewing productivity-based promotional claims. Finally, we suggest possible strategies for furthering this field of investigation.

    Regulation of Clinical and Health Economic Claims in Prescription Drug Labeling and Advertising

    Two Existing Evidentiary Standards

    The 1962 Drug Amendments to the Federal Food, Drug, and Cosmetic Act include a provision requiring manufacturers of drug products to establish a drug's effectiveness by presenting substantial evidence.1 The law requires well-controlled investigations by scientific experts to determine whether substantial evidence exists to support a particular claim of a drug’s clinical effectiveness, generally in at least two well-controlled clinical trials, although there are circumstances in which a single study will provide adequate support,2 and in some cases multicenter trials have been accepted,2 (see also pages 12 to 15 of Ref. 2). In fact, Section 115 of the Food, and Drug Modernization Act of 1997 amended Section 505(d) (21 USC 355(d)) by adding that: "If the Secretary determines, based on relevant science, that data from one adequate and well-controlled clinical investigation and confirmatory evidence (obtained prior to or after such investigation) are sufficient to establish effectiveness, the Secretary may consider such data and evidence to constitute substantial evidence". In brief, evidence relied on to support a claim of clinical benefit in terms of safety or effectiveness must meet the requirements for adequate and well-controlled studies (ie, in 21 CFR 314.126). Furthermore, promotional materials may not be “false, lacking in fair balance, or otherwise misleading” (ie, as defined in 21CFR 202.1(3), (6), and (7)). This means that the evidentiary requirement for a claim in prescription drug advertising is the same as for a claim in approved labeling.

    In response to concerns regarding the economic and quality-of-life impact of medications and the increasing desire of manufacturers to present data regarding such effects, the Food and Drug Modernization Act of 1997 offered specific guidance explicitly regarding the use of health care economic information in promotional claims (ie, Section 114).3 It stated that economic information would not be considered false or misleading if it is based on competent and reliable scientific evidence (as opposed to substantial evidence). This implies that economic evidence submitted as a basis for a promotional claim must meet the condition that it is based on standards that are widely used by economic experts. Under certain circumstances, this standard allows evidence to be based on outcomes studies that mirror real-world resource utilization patterns more clearly than is usually the case in clinical trial settings. Under the Food and Drug Modernization Act of 1997, for economic information to be eligible for this evidence standard when disseminated, even to audiences with the skills necessary to interpret it properly (eg, formulary committees), the information must be directly related to the approved indication(s) for the product. For example, the FDA would object to a claim that a drug approved to lower lipids is cost-effective when used to prevent cardiovascular events unless the drug is also approved for the prevention of cardiovascular events. For a discussion of the complex questions raised by Section 114, see Neumann et al.4

    Application of Evidentiary Standards to Workplace Productivity Claims

    The distinction between a clinical and an economic claim is not always clear, and there can be uncertainty as to which standard of evidence applies in a specific case. In practice, the categorization of an intended claim as clinical or economic may also be determined in part by the intended audience. For example, if the promotional materials imply that the drug has the clinical benefit of improving task speed among arthritis patients, the clinical benefit standard of substantial evidence would be appropriate, in light of the fact that all of the concerns for scientific validity (found in Section 314.126) would apply. In contrast, if the cost of work cutback among these individuals is presented as one component of a comprehensive resource utilization estimate, the competent and reliable scientific evidence standard could apply.

    Attention to the specific use of the information developed for a promotional claim applies in numerous other contexts as well. For example, an employee benefits manager at a large company may be interested in the potential impact of a disease (eg, the common cold) on sporadic absenteeism, the ability to operate heavy equipment, or concentration while on the job. In response, manufacturers might develop productivity outcomes information for dissemination to this audience as follows: "Typical US citizens lose an average of 4 school days or workdays annually because of the common cold, which results in $10 billion worth of lost wages annually."

    Although this kind of descriptive cost-of-illness information may be acceptable on its own, any suggestion that a particular manufacturer's treatment can reverse any of these productivity decrements would result in a clinical benefit claim requiring supportive evidence. In addition, the substantial evidence standard requires that clinical claims of benefit be based on outcomes that are both statistically significant and clinically meaningful.

    Even if the intended use of promotional data is economic support, and the intended audience is skilled in interpreting economic information, it may still be difficult to present competent and reliable scientific evidence in support of a desired productivity claim. This is because the productivity measurement literature has not yet advanced to the point at which all of the important measurement issues are fully resolved. Concerns may arise about the quality of the scientific evidence relied on for some intended productivity-based promotional claims.

    Productivity Models Used in Health Outcomes Research

    The ability to apply the competent and reliable scientific evidence standard would still be limited by the state-of-the-art measurement techniques, even if the distinction were always clear between productivity information that is intended to support a clinical promotional claim compared with information supporting an economic promotional claim. Therefore, it is useful to review these widely used techniques and to describe some of the limitations associated with each of them.

    At the outset, it is important to recognize that a number of different population subgroups are adversely affected by illness in ways that can influence their productive capacity, including employees, homemakers, retirees, students, and caregivers of those directly affected by illness. Although each subgroup is a stakeholder in this discussion, there are a number of reasons why the literature has focused particular attention on worker productivity. First, in their role as payers, employers have moved to the center of many health care benefit decisions, with the productivity of their workforces occupying growing attention. Second, employees are beginning to understand how treatment decisions affect not only their survival and quality of life, but also their on-the-job productivity and long-term labor market outcomes. These outcomes can include their job status (eg, full-time, part-time, unemployed, or out of the labor market) and their position and compensation in a particular organization, given their specific training and experience. This growing realization recognizes that because employee promotional paths and salary trajectories are affected by individual health status, treatment decisions can have profound economic consequences on workers. Third, as the effectiveness of available health interventions improves, illnesses will less likely result in a worker's complete withdrawal from the labor market. Instead, an increasing number of workers will be faced with the difficulty of managing their chronic illnesses while still employed. Indeed, for many individuals with chronic illnesses, eligibility for health insurance can be one important motivation for employment. An additional factor involves the Americans for Disability Act requirement that an employee's health profile not be improperly used for performance assessment. Finally, productivity as a health outcome can be denominated in terms of dollars.

    The effect of illness on productivity is just one part of a more intricate set of influences, as shown in Fig. 1. In this depiction, treatment affects illness symptoms, which affect work capabilities, which affect work performance or productivity, which affect long-term labor market outcomes (ie, A = B = C = D = E). Within this framework, different strands of health outcomes literature have focused on different pieces of these relationships. For example, if a cost-of-illness study includes an indirect cost component, it would consider the effect of B on D. In contrast, cost-effectiveness analyses often emphasize the relationship between A and D. From a public policy perspective, attention to longer-term labor-market outcomes can underscore the need to sometimes examine the entire chain of influence.

    Reg Fig 1

    The large and growing medical, economics, managed care, and managerial literature that has developed around these themes often focuses on measuring the productivity consequences of a specific illness or particular type of treatment, often in one carefully chosen work setting. In most instances, the underlying model used to calibrate a productivity change from month 0 to month 1 takes the following form: [(w+(xxy)) xz],where W = number of days missed from work due to illness in the past month, X = number of work cutback days in the past month due to illness, Y = 1 – average productivity on work cutback days in the past month, and Z = average compensation per day in dollars.

    This approach is equivalent to measuring the value of time saved at work following the onset of illness or intervention, and it can be used for comparison within groups of patients, diseases, or workplaces. Two different research strategies have been used to gather data on each of the parameters measured in the traditional workplace productivity model of the impact of illness. The first strategy relies on self-report instruments; the second uses archival records that quantify employee performance. In each case, measurement concerns arise, as described in detail below.

    Days Missed From Work and Work Cutback Days (“W” and “X”)

    Self-report Instruments. Self-report measures usually are compiled from responses to questions such as: “How many days in the past month did you miss from work due to physical or emotional problems?” This research approach can be implemented broadly in many different subpopulations and for a variety of disease states, as illustrated in several epidemiological studies and clinical trials.5-7

    Methodological concerns often arise with respect to assessments of days missed from work and work cutback days when a self-report approach is used. In addition to the usual recall issues associated with self-reporting, the ability to make causal inferences about the role of health status as a determinant of these workplace outcomes may be questionable.

    Archival Data Assessment. A second research strategy focuses on those employers with access to administrative databases that describe patterns of missed work and work cutback for targeted groups of individuals. This alternative strategy is not based on recollections and perceptions but rather on archival data. A number of analyses provide examples regarding the effects of illness on productivity.8,9 Because of the difficulty of controlling for all possible influences on productivity, no study using this methodology has been done to date that allows treatment-specific effects to be highlighted in a drug manufacturer's promotional claims.

    A variety of factors may underlie a variation in employee work loss for reasons other than illness or treatment. For example, companies often track long periods of consecutive days missed (ie, disability days) by their employees far more carefully compared with their workers' sporadic sick leave. In addition, in many companies, there are norms surrounding the use of sporadic sick leave for personal days to be taken as extra vacation days. Furthermore, the classification and duration of chronic absenteeism, in the form of disability or workers’ compensation, often is set administratively according to company or state policies rather than only on the basis of the clinical status of the patient.

    Employee Productivity on Work Cutback Days (“Y”)

    Self-report Approach. Asking people to estimate numerically their productivity on work cutback days often relies on a simplified visual analog scale. Respondents can be asked how much work they accomplish on their job with reference to a scale ranging from “nothing” (ie, corresponding to a value of zero on the scale) to “the most possible” relative to a specified reference point. The reference point can vary from one study to another and can include any of the following examples:

    • your own most productive period
    • your usual level of productivity
    • your peer group's usual level of productivity
    • your supervisor's expectations

    These approaches have been used in epidemiological studies and in clinical trials in such therapeutic areas as depression10 and migraine headaches.11

    A variety of measurement issues are associated with this approach to quantifying at-work productivity loss. First, because employee-specific contributions to productivity may not be linear, the use of a linear scale can be inappropriate (eg, when a “winner take all” effect is prominent). For example, if a critical delivery arrives at its destination even slightly past an important deadline, the bidder could lose the new business prospect. Even though the loss to a company may be substantial, from an individual worker’s perspective, productivity may have been only slightly impaired on that particular day.

    A second measurement issue concerns the fact that, in the majority of work settings, each employee may not be a stand-alone resource. Where team production exists, one worker’s on-the-job impairment can spill over to affect the ability of the entire team to function properly. Third, as is the case with all self-report instruments, individuals may not be able to assess their at-work productivity reliably, and they may have recall bias across different jobs and workplace environments. Fourth, the symptoms of illness for a variety of health disorders may affect self-report answers in different ways. For example, depressed patients may overstate their self-assessed impairment at the beginning of a clinical trial, whereas alcoholics may understate it.

    Archival Data Assessment (or “Counting Widgets”). As an alternative to self-report data, a growing number of productivity-based investigations rely on archival data gathered at the employee level. In occupations in which the number of widgets produced can be enumerated on a daily basis (eg, amount of data entered into a computer, number of packages delivered), measures of on-the-job performance can be established that overcome some of the problems associated with reliance on subjective instruments. These measures might be linked across administrative claims systems, so that utilization patterns can be assessed by combining information from various types of records regarding a patient's medical, prescription drug, absenteeism, and work performance.

    This approach has a number of desirable features. The measures are employee-specific, comprehensive, and frequently available over long time periods, thereby allowing identification of the effect of periodic changes in health status on workplace productivity. They may also incorporate a quality dimension (eg, error rate, defect rate).

    Because of the challenges associated with obtaining piece-rate data for this type of analysis, only a limited number of studies have been published that use archival data to measure productivity while at work.12 Other research has used archival data to investigate the impact of illness on work loss due to disability and sporadic absenteeism.13,14

    As with self-report methods, a number of measurement issues are associated with assessing at-work productivity by using archival data. One of the most important is that it is very difficult to identify occupations in which on-the-job performance is routinely measured in a way that incorporates the desirable features associated with piece-rate output. In addition, the accuracy of the recorded data may be questionable in some cases, even though its archival nature seems to offer comfort regarding accuracy. Interdependencies among employees raise further concerns the analysis contains underlying assumptions that each individual's on-the-job productivity is determined in isolation. In fact, some "teammates" may step up their contribution in the short run to offset an illness-related productivity decline by a coworker, whereas others may suffer a productivity decline with sufficiently widespread disease within a work group. Furthermore, there may be medical interdependence of employees that affects the objective measurement of productivity (eg, influenza patients may be encouraged to stay away from work because of contagion concerns).

    Another disadvantage of these counting approaches is that, in many cases, the workers know that their piece-rate productivity is being monitored. For example, workers may stay late and work on their own time to achieve their well-publicized sales quota. In such an environment, because the minimum amount of output realized is organizationally determined, job dissatisfaction and job turnover may be the true adverse outcomes of illness.

    Compensation (“Z”)

    The traditional approach to quantifying productivity costs involves multiplying the quantity of work-loss days (ie, the sum of total-day plus partial-day equivalents) by the employee’s wage rate in the case of sporadic sick time, or when relevant, by disability payments, which are typically less than wages. If actual wage data are unavailable, it may be possible to approximate this information by using disability payment rates or employer job classification averages.

    However, productivity might not be realistically measured by salary for a variety of reasons. For example, as noted above, the output of each worker could have spillover effects on the performance of others nearby. In addition, in the case of a long-term absence, the use of an employee's wage rate – or per diem disability reimbursement rate, which often is lower can substantially understate the true disruption cost to the employer associated with absence from work. In fact, if the chronically absent worker must be replaced, additional costs will be incurred for recruiting and training a new employee.15

    A Gold Standard for Productivity Measurement: Validating a Self-Report Instrument Using Archival Productivity Data

    There may be a trade-off between (1) the ability to measure productivity effects precisely among a specific set of employees in a particular work setting, and (2) the generalizeability of the results to other work settings, with employees in other kinds of jobs, and for other illnesses. These measurement issues raise additional concerns from an FDA perspective about the ability to marshal appropriate evidence to support promotional claims. Of course, as continued progress is made in the health outcomes literature to resolve these measurement issues, the FDA will be able to base its consideration of promotional claims on improved standards of scientific evidence. One helpful advance would be the development of a gold standard for productivity measurement, mirroring the work that has been done to measure aspects of health-related quality of life at the disease-specific level with increasing precision.

    Development of a Productivity Instrument Gold Standard

    A gold-standard instrument in the area of productivity research could serve the needs of a broad base of constituencies, each with a different perspective and application of the results. Establishing the perspective from which this information is interpreted (ie, the employer, the payer, or, in some countries, society as a whole) provides the context of the analysis. Because the perspective also represents the consumer of these data, those involved in the research must ensure that the data they collect and interpret have the requisite demonstrable external and internal validity and that the results are meaningful to the audience.

    Although productivity is often assessed by using self-report measures, archival data of worker contributions over time may supplement employee self-report information and can be used as the metric to validate the self-report data. Thus, workplace productivity offers the opportunity for systematic measurement in a fairly controlled environment. Ideally, the accuracy and validity of self-report data can be assessed in the face of consistent and repeated input or output measurements in the workplace.

    Two main circumstances dictate the use of self-report data in productivity measurement:

    • when other data exist but are too complex or difficult to obtain
    • when no data exist other than that obtained directly from the subject (eg, pain intensity).16

    However, the development of a gold-standard self-report measurement, which would be validated by and directly related to objective productivity measures, may allow us (eventually) to efficiently address productivity issues purely on the basis of self-report.

    Who Would Use This Information?

    Faced with choices regarding alternative interventions, corporate health managers, employees, health care organizations/payers, and insurers must make intervention decisions. If a gold-standard measure of productivity were developed and validated, the individuals responsible for making critical choices could more easily compare alternatives. For example, with regard to drug therapy, the FDA would be better able to ensure that productivity claims made by drug companies were not false or misleading, which would provide a more level playing field across therapies. In this scenario, it would be the responsibility of the manufacturers to prove that their drug had a positive effect on productivity, or in some cases, that a more favorable side-effect profile had a substantive workplace benefit. Although studies often take the perspective of a payer or an employer, the use of a standardized instrument would better assure stakeholders that the collected data provided a common metric that would allow reasonable comparison across diseases and interventions.

    What Would Be the Components of a Gold Standard?

    A gold standard must be anchored to some external measure of similar elements. Ideally, the standard would produce a response that corresponds to some workplace measurement that is meaningful, in that changes in one are reflective of changes in the other at a quantifiable level of precision that is acceptable. Although there are self-report instruments that have been “validated” in the absence of external objective data, it is possible in the case of productivity measurement to assess whether a direct relationship exists between a self-report instrument and actual measures of worker performance. Although it may seem relatively easy to administer a questionnaire to a group of employees, there are many practical issues involving both the execution and interpretation of the results.

    There may be considerable skepticism about the accuracy and precision of self-reported information, perhaps stemming from difficulties in articulating the concepts to be measured and in generalizing the findings. The long list of biases that must be appreciated in the development and interpretation of self-report instruments is well beyond the scope of this article.17 We hope that answers to questionnaires are truthful, but it is human nature for respondents to want to appear fully productive, and this may distort responses. This bias alone demands calibration against more objective criteria.

    The development of a gold standard productivity instrument should be undertaken at a series of sites and occupations where self-report data can be integrated with objective measures, health care records (including prescription drug information), and attendance. Each of these critical pieces of information plays a role in codifying aspects of the self-report instrument relating to the different measurements that may reflect the impact on productivity in the workplace.

    Several methodologies have evolved in their application to evaluating the impact of disease and its treatment on the workplace. One approach involves the experience sampling method, which randomly samples employees throughout the day to determine, for given moments in time, exactly the nature and extent of the work that they are performing.18,19 Another approach involves simulations that create hypothetical circumstances and assesses the employee's reaction to them. Simulations include formal computerized simulators to measure responses and experimental “tests” in which subjects are asked to perform a task and are “graded” objectively to determine whether they manifest decrements that can be attributed to their disease state or treatment. When these tests are exactly what the worker or student would be doing on the job or at school, they are called simulations (eg, typing tests given by temporary agencies to screen applicants); otherwise, they are considered psychometric, or ability, tests. These approaches contrast with the more crude assessments of simple absenteeism or self-report of productivity under various health states.20,21

    One effort currently under way that encompasses all of these metrics and a set of self-report questions is the Harvard Workplace Productivity Study.22 This extensive pilot study of more than 6000 employees in several companies and job classes incorporates numerous available methodologies for obtaining prospective data and harnesses available objective and self-report data that have been routinely collected for each subject. When all of the objective data (including health care claims records) are merged with the self-report data, this study may provide researchers and employers with a basis for a gold-standard instrument.

    Ideally, such a standard would draw on broad evidence of consistency between objective and subjective measures of workplace performance to eventually permit the widespread use of a survey-based instrument, even when no objective measures of productivity exist. Thus, a gold-standard self-report instrument could provide the basic measurement approach for intervention trials and could provide employers with a tool whose measurement will have some currency across occupational and organizational lines. The accepted standard would also define a scope of acceptable methodologies that can be effectively used in the subsequent refinement of scales and instruments.

    Conclusion

    As demand for productivity-related outcomes information grows, the FDA increasingly will review promotional claims by pharmaceutical manufacturers and will request supporting data. The evidence standard that FDA will use to determine whether the data support the claims will depend on the context and the use of the productivity-related outcomes in promotion. It will further depend on the developing state of the science of productivity measurement. There is a pressing need for progress in the health outcomes literature. Development of a gold standard, by which productivity outcomes could be grounded, would represent a valuable advancement in this area.

    References

    1. 76 Stat 780 (1962) at 781

    2. US Food and Drug Administration. Guidance for Industry. Providing Clinical Evidence of Effectiveness for Human Drugs and Biological Products. Rockville, MD: USFDA;1998:3

    3. S. Rep. No. 105-143, pp 42-43.

    4. Neumann PJ, Claxton K, Weinstein MC. The FDA's regulation of health economic information. Health Affairs 2000;19(5):129-137.

    5. Verbrugge LM, Patrick DL. Seven chronic conditions: their impact on US adults' activity levels and use of medical services. Am J Public Health. 85:173-182.

    6. Kessler RC, Mickelson KD, Barber C, Wang P. Effects of chronic medical conditions on work impairment. In: Ross, AS, ed. Caring and Doing For Others: Social Responsibility in the Domains of Family, Work, and Community. Chicago: University of Chicago Press. In press.

    7. Finkelstein SN, Berndt ER, Greenberg PE, et al. Improvement in subjective work performance after treatment of chronic depression: some preliminary results. Psychopharmacol Bull. 1996;32:34-40.

    8. Conti DJ, Burton WN. The economic impact of depression in a workplace. J Occup Environ Med. 1994;36:983-988.

    9. Barnett A, Birnbaum H, Cremieux PY, Slavin M. The costs of cancer to a major employer in the United States: a case-control analysis. Am J Managed Care. 2000;6:1243-1251.

    10. Kessler RC, Barber C, Birnbaum HG, et al. Depression in the workplace: effects on short-term disability. Health Affairs. 1999;18:163-171.

    11. Stang PE, Osterhaus JT, Impact of migraine in the United States; data from the National Health Interview Study. Headache. 1993;33:29-35.

    12. Burton WN, Conti DJ. Use of an integrated health data warehouse to measure the employer costs of five chronic disease states. Dis Manage.1999; 1; 17-26.

    13. Claxton A, Chawla AJ, Kennedy S. Absenteeism among employees treated for depression. J Occup Environ Med, 1999; 41:605-611.

    14. Birnbaum HG, Greenberg PE, Cremieux PY, Kessler RC. Management of major depression in the workplace: impact on employee work loss. Dis Manage Health Outcomes. 2000;7:163-171.

    15. Watson Wyatt. Staying @Work: Focusing on What Works-Watson Wyatt 4th Annual Employer Survey. Presented at: 13th Annual National Disability Management Conference, Washington Business Group on Health; October 27-29, 1999; Washington, DC. Boston: Watson Wyatt. Catalog #W-289.

    16. Baldwin W. Information that no one else knows: the value of self-report. In: Stone AA, Turkkan JS, Bachrach CA, et al, eds. The Science of Self Report. New Jersey: Lawrence Erlbaum, Associates; 1999:3-7.

    17. Stone AA, Turkkan JS, Bachrach CA, et al, eds. The Science of Self Report. New Jersey: Lawrence Erlbaum, Associates;1999

    18. Stone AA, Shiffman S. Ecological momentary assessment (EMA) in behavioral medicine. Ann Behav Med. 1994;16:199-202.

    19. deVries MW, Delespaul PAEG, Nicolson NA. Experience Sampling Method (ESM). In: Salek S, ed. Compendium of Quality of Life Instruments. Chichester: Wiley & Sons; 1999.

    20. Howland J, Mangione TW, Laramie A. Simulation for measurement of occupational performance. In: Kessler RC, Stang P, eds. Health and Work Productivity: Emerging issues in Research and Policy. Chicago: University of Chicago Press. In press.

    21. Howland J, Rohsenow DJ, Cote J, Siegel M, Mangione TW. Effects of low-dose alcohol exposure on simulated merchant ship handling power plant operation by maritime cadets. Addiction. 2000;95:719-726.

    22. Kessler RC, LeRoy L, PE Stang. The Harvard Health and Workplace Productivity Study. Boston: Department of Health Care Policy, Harvard Medical School; 1999.

    Go top