May 2011  << Back  

  Can't open the PDF? Click here for help.

 

 RESEARCH ARTICLE

Performance characteristics of the PTSD Checklist in retired firefighters exposed to the World Trade Center disaster

Sydney Chiu, MA

Fire Department of the City of New York, Bureau of Health Services, Brooklyn, NY, USA

Mayris P. Webber, DrPH

Fire Department of the City of New York, Bureau of Health Services, Brooklyn, NY, USA, Montefiore Medical Center and Albert Einstein, College of Medicine, Bronx, NY, USA

Rachel Zeig-Owens, MPH

Fire Department of the City of New York, Bureau of Health Services, Brooklyn, NY, USA

Jackson Gustave, MPH

Fire Department of the City of New York, Bureau of Health Services, Brooklyn, NY, USA

Roy Lee, BS

Fire Department of the City of New York, Bureau of Health Services, Brooklyn, NY, USA

Kerry J. Kelly, MD

Fire Department of the City of New York, Bureau of Health Services, Brooklyn, NY, USA

Linda Rizzotto, LCSW, CASAC

Fire Department of the City of New York, Bureau of Health Services, Brooklyn, NY, USA

Rita McWilliams, PhD

Environmental and Occupational Medicine, University of Medicine and Dentistry at New Jersey, Robert Wood Johnson Medical School, Piscataway, NJ, USA

John K. Schorr, PhD

Stetson Institute for Social Research, Stetson University, DeLand, FL, USA

Carol S. North, MD, MPE

Program in Trauma and Disaster, Dallas VA North Texas Health Care System, Department of Psychiatry, University of Texas Southwestern Medical Center, Dallas, TX, USA

David J. Prezant, MD

Fire Department of the City of New York, Bureau of Health Services, Brooklyn, NY, USA, Montefiore Medical Center and Albert Einstein, College of Medicine, Bronx, NY, USA

BACKGROUND: Since the World Trade Center (WTC) attacks on September 11, 2001, the Fire Department, City of New York Monitoring Program has provided physical and mental health screening services to rescue/recovery workers. This study evaluated performance of the self-report PTSD Checklist (PCL) as a screening tool for risk of posttraumatic stress disorder (PTSD) in firefighters who worked at Ground Zero, compared with the interviewer-administered Diagnostic Interview Schedule (DIS).

METHODS: From December 2005 to July 2007, all retired firefighter enrollees completed the PCL and DIS on the same day. Sensitivity, specificity, receiver operating characteristic (ROC) curves, and Youden index (J) were used to assess properties of the PCL and to identify an optimum cutoff score.

RESULTS: Six percent of 1,915 retired male firefighters were diagnosed with PTSD using the DIS to assess DSM-IV criteria. Depending on the PCL cutoff, the prevalence of elevated risk relative to DSM-IV criteria varied from 16% to 22%. Youden index identified an optimal cutoff score of 39, in contrast with the frequently recommended cutoff of 44. At 39, PCL sensitivity was 0.85, specificity was 0.82, and the area under the ROC curve was 0.91 relative to DIS PTSD diagnosis.

CONCLUSIONS: This is the first study to validate the PCL in retired firefighters and determine the optimal cutoff score to maximize opportunities for PTSD diagnosis and treatment.

KEYWORDS: disasters, September 11 terrorist attacks, PTSD checklist, posttraumatic stress disorder, firefighters

ANNALS OF CLINICAL PSYCHIATRY 2011;23(2):95-104

  INTRODUCTION

The terrorist attack and collapse of the World Trade Center (WTC) towers on September 11, 2001 (9/11) exposed many New York City (NYC) residents and workers to physical and emotional trauma, putting them at risk for posttraumatic stress disorder (PTSD) and other illnesses. Between October 16 and November 15, 2001, a survey given to a random sample of Manhattan residents living south of 110th Street—up to 6.75 miles north of the WTC—found that 7.5% reported PTSD-like symptoms; this increased to 20% among those living <1 mile from the site.1

Studies have indicated that rescue/recovery workers exposed to a major disaster have a higher risk for PTSD than the general population2-6 but a lower risk than survivors.7 They have reported PTSD-like symptoms in 12% to 13% of Fire Department, City of New York (FDNY) firefighters within the first 6 months after 9/11,8 12% to 17% of firefighters 2 to 3 years after 9/11,6 and 11% to 20% of non-FDNY rescue/recovery workers 2 to 5 years after 9/11.9

PTSD has been recognized as a condition that may follow an event in which an individual experienced or witnessed a life-threatening situation, loss of life, or serious injury resulting in a sense of fear, helplessness, or horror.10 Symptoms include reliving the event, avoiding situations that are reminders of the event, having trouble remembering the event, feeling indifferent toward other people, losing interest in activities that were once enjoyed, and/or irritability.11

Screening tools identify persons who reach a predetermined threshold or cutoff score for follow-up with more intensive and time-consuming tests. The PTSD Checklist (PCL) was designed to screen for PTSD in Vietnam War veterans.12 It uses 17 items corresponding to post-traumatic symptoms in DSM-IV and is scored from 17 to 85, with higher scores more likely to be associated with a diagnosis. The original work reported an optimum cutoff score of 50. Others have reported cutoff scores from 2813 to 50,12,14 with 44 being most frequently used.6,15

The Diagnostic Interview Schedule (DIS) for DSM-IV diagnosis is a structured diagnostic interview designed to assess DSM-IV criteria for a PTSD diagnosis.16 Unlike the PCL, the DIS is not a self-assessment screening tool for PTSD but rather a fully-structured diagnostic interview. A diagnosis of PTSD requires full assessment of accepted diagnostic criteria, such as accomplished with the DIS or clinician interview.

The purpose of our study was to evaluate the performance characteristics of the PCL screening tool in relation to assessment of full diagnostic criteria using a structured diagnostic interview (the DIS) in a large population of retired firefighters. Further, because the FDNY is using a PCL cutoff score of 44—which means that only persons scoring at or above that level receive a full clinical diagnostic assessment—our intent was to evaluate the screening efficiency of this cutoff for the full PTSD DSM-IV diagnostic criteria.

  METHODS

Since 1997, the FDNY Bureau of Health Services (BHS) has performed periodic health evaluations on active FDNY members approximately every 18 months, including physician examinations and—since 2001—self-administered health questionnaires. Beginning in 2005, a more extensive mental health questionnaire incorporated the PCL. This occurred coincident with outreach to retired WTC-exposed firefighters, who were invited to resume participation in health screenings that previously were limited to the active work-force. During the first 19 months of the retiree screenings (December 2005 to July 2007) all retired firefighters completed both the PCL and the DIS, in that order and on the same day, regardless of whether a threshold score was attained. Participation in the research study required written informed consent and was approved by the Institutional Review Board of Montefiore Medical Center.

Participants

Eligibility for study enrollment included status as a retired FDNY employee who worked at least one shift at any of the designated WTC work sites between September 11, 2001 and July 25, 2002, when the site was formally closed. Active recruitment involved contacting retirees through invitation letters and/or by telephone, using contact information from the FDNY pension database. Passive recruitment involved self-enrollment through the FDNY Web site. A total of 4,080 retirees were eligible to enroll in the WTC Medical Monitoring and Treatment Program. We focused our study on retirees because administering the DIS—with its lengthy medical visit—to all active-duty firefighters would have been much more difficult.

By the end of the initial enrollment period, 2,574 retired firefighters (63.1%) had returned to FDNY BHS and completed a monitoring exam that included the self-administered mental health questionnaire. We excluded the following: persons who could not be classified in a 9/11 exposure group or first arrived at any of the WTC sites after September 24, 2001 (n=316); those who did not complete the PCL and the DIS on the same day (n=218); fire marshals (n=84) because of the distinct nature of their work; persons who retired with a mental health disability (n=31); and female firefighters (n=10), who represent a small proportion of the workforce. The final analytic population totaled 1,915 (46.9% of eligible).

Data sources

Age, retirement date and status (ordinary vs disability for any cause other than psychological), and rank (chiefs, captains, lieutenants, and firefighters) were obtained from the FDNY employee database. All other variables were collected from the questionnaires, including marital status on 9/11 and current marital status. Participants were identified as having a marital status change if they indicated a different marital status on the day of the exam than on 9/11.

9/11 exposure

To measure exposure to trauma, we used the FDNY-WTC Exposure Intensity Index17 that was first developed to categorize exposure to WTC respiratory contaminants. This index is based on first arrival at the WTC site as follows: “Group 1,” the most severely exposed, arriving on the morning of 9/11 and present during the tower collapses; “Group 2” arriving during the afternoon of 9/11; “Group 3” arriving the next day on September 12, 2001; and “Group 4” or least exposed, arriving any day between days 3 and 14.

Measures

The PCL was originally incorporated into the FDNY BHS mental health questionnaire to identify persons requiring further evaluation. PTSD symptoms, as defined by the DSM-IV, have been categorized as re-experience, avoidance/numbing, and hyperarousal.11 The PCL was designed using 17 items that correspond to the DSM-IV symptoms of PTSD.12 Using a Likert scale of 1 to 5 (1=“not at all” to 5=“extremely”), scores can range from 17 to 85. The PCL was modified in our questionnaire to fit the context of 9/11. For example, some questions that refer to a traumatic event were modified to refer specifically to the event of “WTC.” Each question also was specific to current symptoms (eg, “In the past month;” TABLE 1). Because the PCL does not assess full diagnostic criteria, PTSD assessments as determined by the PCL were labeled “elevated PTSD risk.” We believe this term is a more accurate description of this measure and of what other authors have termed “probable PTSD.”6,18

We calculated elevated PTSD risk by using the PCL in 2 ways: 1) fitting symptoms endorsed on the DIS to fulfill DSM-IV criteria, which require the presence of at least 1 re-experiencing, 3 avoidance, and 1 hyperarousal symptoms among its other criteria19; and 2) using various cutoff scores for the screening instrument. To estimate DSM-IV symptom criteria with the PCL, we defined the presence of PTSD symptoms as responses of “moderately” or greater (answer choices 3 to 5) when participants were asked to describe the severity of specific symptoms.20 Diagnostic assessment using the DIS module for PTSD, administered by social workers trained by a DIS-certified psychiatrist, assessed full criteria for the diagnosis of PTSD to compare with results of the PCL screening tool.


TABLE 1

PTSD Checklist questions modified to fit context of WTC

B. Re-experience
• In the past month, have you been bothered by repeated, disturbing memories, thoughts, or images of WTC?
• In the past month, have you been bothered by repeated disturbing dreams of WTC?
• In the past month, have you been suddenly acting or feeling as if WTC was happening again (as if you were reliving it)?
• In the past month, do you feel upset when something reminded you of WTC?
• In the past month, have you been having physical reactions (eg, heart pounding, trouble breathing, sweating) when something reminded you of WTC?
C. Avoidance
• In the past month, have you been avoiding thinking about or talking about WTC or avoiding having feelings related to the incident?
• In the past month, have you been avoiding activities or situations because they reminded you of WTC?
• In the past month, have you been having trouble remembering important parts of WTC?
D. Arousal
  (No questions modified)
Answer choices for all questions were: (1) Not at all, (2) A little bit, (3) Moderately, (4) Quite a bit, (5) Extremely.
PTSD: posttraumatic stress disorder; WTC: World Trade Center.
Data analyses

We used sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), overall correct classification (OCC), receiver operating characteristic (ROC) curve, and Youden index (J) to assess performance of the PCL in the retiree population relative to assessment of full diagnostic criteria for PTSD using the DIS. Sensitivity measures a screening tool’s ability to correctly identify those with an identified disorder. Specificity measures the tool’s ability to correctly identify those without a disorder. PPV is the probability of having a disease when the tool tests positive, and NPV is the probability of not having a disease when the tool tests negative. OCC is the percent of correctly identified participants with and without the disease. The Youden index (sensitivity + specificity –1) allows for the selection of a cutoff score (the cutoff score at the maximum value of J) when sensitivity and specificity are equally important. A perfect screening tool relative to a diagnostic assessment would have a Youden index of 1.21 We used the area underneath the ROC curve (AUC) to quantify the performance of the PCL. An AUC of 1.00 indicates a perfect screening tool relative to a diagnostic assessment.

We also analyzed demographic and other variables in relation to elevated PTSD risk. Bivariate analyses of categorical variables used χ2 with odds ratios (OR) and 95% confidence intervals (CI) or McNemar’s test, as appropriate. Correlation between 2 variables was measured using Pearson’s R2. Continuous variables were assessed using ANOVA. Separate multivariate logistic regression analyses were performed to compare independent predictors of PTSD prevalence using 4 different outcomes: PCL positive scored with fitting symptoms to match DSM-IV symptom criteria, PCL positive with a threshold score of 44; PCL positive with a threshold score of 39; and DSM-IV PTSD diagnosis using the PTSD section of the DIS. Backwards stepwise elimination was used to determine the final models, with variables retained if at least one of the categories was significant at P ≤ .10. Goodness-of-fit was assessed using the Hosmer-Lemeshow test. All analyses were performed using Statistical Analysis Software, version 9.1.3 (SAS Institute, Cary, NC).

  RESULTS

Characteristics of the study population

Among the 1,915 retired firefighters, average age on 9/11 was 47.0 years (SD=6.9; range 28.0 to 73.4 years). On 9/11, 83% were married; at the time of our study, 10% reported a change in marital status. Approximately one-half had some college education or an associate degree (48%), and 38% were officers (TABLE 2).

In terms of exposure, most arrived at the WTC site on 9/11 (72%), with the majority arriving that afternoon after the WTC towers collapsed (56%). More than 88% reported working at the WTC site by the end of day 2. Approximately 40% worked at the WTC site for ≥5 months.

Retirement because of WTC-related disability was granted to 64%. We observed a significant association between retirement with a disability pension and age (P < .0001). In analyzing the characteristics of these firefighters (TABLE 2), we found a 50% decrease in the probability of retirement on disability with every 10-year increase in age (OR=0.50, CI: 0.50 to 0.51).

Approximately 46% of the retirees reported having previously held jobs in the military, law enforcement, corrections, and/or a health profession. Older age was strongly correlated with previous professions: for every 10-year increase in age, individuals were 3 times more likely to report previous professions (OR=2.97, CI: 2.9 to 3.0).

We compared characteristics of the 1,915 participants with 2,127 nonparticipating retirees who met the study criteria for inclusion. The primary difference was that more participants lived in the tri-state area (New York, New Jersey, and Connecticut) (P < .01). Small, albeit significant differences also existed in age (participants mean age 47.0 vs 45.4 for nonparticipants; P < .001) and the proportion retiring as chiefs (6% vs 4%; P < .0001). These retiree groups were similar in the proportion of members who were present during the morning of the collapse (16% vs 17% P=.21) and in those retiring with disability benefits (65% vs 64% P =.81).


TABLE 2

Characteristics and average PCL scores of retired firefighters exposed to WTC collapse

  n % PCL (mean score)
Total 1915 100.00 30.44
Age on 9/11 (P < .0001)
  <40
  40 to 44
  45 to 49
  50 to 54
  55+

292
470
563
323
267

15.25
24.54
29.40
16.87
13.94

32.68
31.14
31.25
29.2
25.89
Exposure group (P < .0001)
  1 - Morning of 9/11
  2 - Afternoon of 9/11
  3 - Day 2
  4 - Day 3 to day 14

297
1077
316
225

15.51
56.24
16.50
11.75

36.38
30.30
28.38
26.12
Retirement status (P < .0001)a
  Disability
  Non-disability

1211
670

64.38
35.62

31.96
27.86
Previous professions (P < .01)b
  No other profession
  Other professions

1030
885

53.79
46.21

31.23
29.50
Number of children (NS)
  No children (0)
  1
  2
  3
  4
  5+
195
218
735
510
183
74
10.18
11.38
38.38
26.63
9.56
3.86
30.98
30.52
30.61
30.59
30.42
29.97
Education (NS)
  High school diploma
  College classes or associate degree
  Bachelor degree or RN
  Post college classes or degree
545
927
309
134
28.46
48.41
16.14
7.00
30.61
30.62
29.51
30.57
Rank (P < .001)
  Chiefs
  Captains and lieutenants
  Firefighters

122
597
1196

6.37
31.17
62.45

27.10
29.52
31.23
Marital status on 9/11 (NS)c
  Single
  Married
  Not married, living with partner
  Separated
  Divorced
  Widowed

101
1594
63
34
110
12

5.28
83.28
3.29
1.78
5.71
0.63
30.83
30.21
33.29
32.62
31.38
27.83
Marital status change since 9/11 (P < .05)
  No change
  Status change

1726
189

90.13
9.87

30.20
32.56
Years retired (P < .01)d
  Years ≤1
  <1 years ≤2
  <2 years ≤3
  <3 years ≤4
  <4 years ≤5
  years >5

247
282
289
614
275
208

12.90
14.73
15.09
32.06
14.36
10.86

30.10
31.69
32.89
29.92
29.24
28.84
Months spent at site (P < .05)e
  Months <5
  Months ≥5

580
357

61.90
38.10

30.10
32.28
aRetirement pension information (disability vs nondisability) was not available for participants who retired before 1991 (n=34).
bReferred specifically to previous professions of military, law enforcement, corrections, and health.
cOne participant did not respond.
NS: not significant; PCL: PTSD Checklist; RN: registered nurse; WTC: World Trade Center.
dNumber of years the individual had been retired on the day the questionnaire was completed.
eAmount of time spent at WTC site was available for only 937 (48.9%) of the participants.
NS: not significant; PCL: PTSD Checklist; RN: registered nurse; WTC: World Trade Center.
Elevated PTSD risk using the PCL and assessment of full DSM-IV PTSD criteria using the DIS

The average PCL score for retired firefighters was 30.4 (SD=13.3). Age on 9/11, 9/11 exposure group, retirement status, previously held professions, rank, change in marital status since 9/11, number of years retired, and number of months spent at the WTC site were all related to mean PCL scores (TABLE 2). Using a PCL cutoff score of 44 identified 16% of retired firefighters (n=306) as having elevated PTSD risk. Fitting PCL responses to approximate DSM-IV symptom criteria identified 17% of retired firefighters (n=329) as having elevated PTSD risk. In contrast, using the DIS identified 6% of retired firefighters (n=124) as fulfilling PTSD criteria.

Diagnostic value

The ROC curve for the PCL is presented in FIGURE 1. The AUC was 0.91. We examined the diagnostic value of scores from 26 to 50 in comparison to the diagnosis of PTSD determined to meet full DSM-IV criteria by DIS interview (TABLE 3). A cutoff score of 44 had a sensitivity of 0.74 and a specificity of 0.88 compared with DIS diagnosis. However, the Youden index is at its maximum for cutoff score 39 (J=0.67), which has a sensitivity of 0.85 and specificity of 0.82. Lowering the cutoff score from 44 to 39 identified elevated PTSD risk in 22% of retired firefighters (n=422); relative to full DSM-IV criteria assessed with the DIS, the PPV for the PCL screener decreases from 30% to 25%, corresponding to an increase in false positives from 214 to 317; and the NPV increases from 98% to 99%, corresponding to a decrease in false negatives from 32 to 19 (FIGURE 2).

At either cutoff point, however, the continuous PCL symptom response score outperformed our attempt to fit PCL symptom responses to DSM-IV criteria in sensitivity, which was only 0.69. Further, statistically significant differences existed between elevated PTSD risk using the method of fitting PCL symptom responses to DSM-IV criteria compared with the continuous PCL symptom response scores at cutoff of either 39 (P < .0001) or 44 (P < .05). Similarly, the Youden index for fitting PCL responses to DSM-IV symptom criteria was lower than either PCL symptom response cutoff score 44 or 39 (J=0.55 vs 0.62 and 0.67, respectively), which indicated that the method of fitting PCL symptom responses to DSM-IV criteria performed poorly as compared with using cutoff scores in this population.

FIGURE 1: ROC curve of PTSD Checklist test taken by retired firefighters exposed to WTC collapse

This study used the area underneath the ROC curve (AUC) to quantify the performance of the PTSD Checklist. An AUC of 1.00 indicates a perfect screening tool relative to a diagnostic assessment.

PTSD: posttraumatic stress disorder; ROC: receiver operating characteristic;

WTC: World Trade Center.

FIGURE 2: Distribution of PTSD Checklist cutoff score test results compared with the DIS

DIS: Diagnostic Interview Schedule; PTSD: posttraumatic stress disorder.


TABLE 3

Characteristics of PCL with different cutoff scores and DSM-IV criteria in relation to DIS

Cutoff score Sensitivity Specificity PPVa NPVb OCCc Jd
27 0.99 0.53 0.14 1.00 0.56 0.52
28 0.98 0.56 0.14 1.00 0.59 0.54
29 0.97 0.59 0.15 1.00 0.61 0.56
30 0.97 0.62 0.16 1.00 0.64 0.59
31 0.96 0.66 0.17 1.00 0.68 0.61
32 0.94 0.68 0.18 0.99 0.70 0.62
33 0.93 0.71 0.19 0.99 0.72 0.63
34 0.91 0.73 0.20 0.99 0.74 0.64
35 0.89 0.75 0.21 0.99 0.76 0.64
36 0.88 0.77 0.22 0.99 0.77 0.65
37 0.88 0.78 0.23 0.99 0.79 0.66
38 0.85 0.80 0.24 0.99 0.80 0.65
39 0.85 0.82 0.26 0.99 0.82 0.67
40 0.83 0.83 0.27 0.98 0.83 0.66
41 0.80 0.84 0.28 0.98 0.84 0.65
42 0.77 0.86 0.29 0.98 0.85 0.63
43 0.75 0.87 0.30 0.98 0.86 0.62
44 0.74 0.88 0.31 0.98 0.87 0.62
45 0.71 0.89 0.33 0.98 0.88 0.60
46 0.71 0.90 0.34 0.98 0.88 0.61
47 0.68 0.91 0.35 0.97 0.89 0.59
48 0.66 0.91 0.37 0.97 0.90 0.58
49 0.63 0.92 0.38 0.97 0.90 0.55
50 0.60 0.93 0.39 0.97 0.91 0.53
DSM-IV criteria 0.69 0.86 0.27 0.97 0.85 0.55
aPositive predictive value.
bNegative predictive value.
cOverall correct classification.
dYouden (J) index.
DIS: Diagnostic Interview Schedule; PCL: PTSD Checklist: PTSD: posttraumatic stress disorder.
Multivariate models

We used multivariate logistic regression models to estimate the independent effects of demographic and other variables on 4 outcomes: PCL positive through fitting symptom responses to DSM-IV symptom criteria, PCL positive using a cutoff score of 44; PCL positive using a cutoff score of 39; and meeting full DSM-IV criteria for PTSD using the DIS. We found that age on 9/11, exposure group, and retirement status (disability vs ordinary) were significant covariates in all final models (TABLE 4). Elevated PTSD risk was 1.38 to 4.62 times greater for younger age groups relative to the oldest retirees, and risk was 2.19 to 3.83 times greater in retired firefighters who first arrived at the WTC site during the morning of 9/11. Persons who retired with a disability were almost twice as likely to have elevated PTSD risk (OR=1.84 to 2.23) as ordinary retirees.


TABLE 4

Multivariate analysis of PTSD models using logistic regressiona

  Model 1
Elevated PTSD risk with DSM-IV criteria on the PCL
Model 2
Elevated PTSD risk with cutoff score of 44 on the PCL
Model 3
Elevated PTSD risk with cutoff score of 39 on the PCL
Model 4
PTSD on DIS Module E
Variable Odds ratio 95% CI Odds ratio 95% CI Odds ratio 95% CI Odds ratio 95% CI
Age on 9/11
  <40
  40 to 44
  45 to 49
  50 to 54
  55+

1.82
1.97
1.89
1.57
1.00

1.06 to 3.11
1.20 to 3.27
1.15 to 3.10
0.91 to 2.70
(ref)

1.70
1.66
1.85
1.38
1.00

0.99 to 2.91
1.00 to 2.77
1.12 to 3.04
0.79 to 2.40
(ref)

1.76
1.52
1.52
1.50
1.00

1.11 to 2.80
0.98 to 2.35
0.99 to 2.34
0.94 to 2.38
(ref)

3.70
4.62
3.81
3.19
1.00

1.25 to 11.00
1.62 to 13.19
1.33 to 10.89
1.01 to 9.66
(ref)
Exposure group
  1 - Morning of 9/11
  2 - Afternoon of 9/11
  3 - Day 2
  4 - Day 3 to day 14

3.58
1.85
1.44
1.00

2.06 to 6.24
1.10 to 3.10
0.80 to 2.60
(ref)

3.42
1.69
1.25
1.00

1.96 to 5.97
1.01 to 2.84
0.69 to 2.28
(ref)

3.83
1.91
1.38
1.00

2.33 to 6.29
1.21 to 3.01
0.82 to 2.33
(ref)

2.19
1.24
0.53
1.00

1.02 to 4.74
0.60 to 2.55
0.20 to 1.36
(ref)
Retirement status
  Disability
  Nondisability

1.84
1.00

1.39 to 2.44
(ref)

1.90
1.00

1.42 to 2.55
(ref)

1.75
1.00

1.36 to 2.25
(ref)

2.23
1.00

1.40 to 3.56
(ref)
aThe following covariates were included in each model: age on 9/11, exposure group, retirement status, rank, years retired, months spent at site, and change in marital status. Those presented were retained.
CI: confidence interval; DIS: Diagnostic Interview Schedule; PCL: PTSD Checklist; PTSD: posttraumatic stress disorder; ref: reference group.

  DISCUSSION

The PCL is a self-administered screening tool for identifying individuals with elevated PTSD risk. Screening tests are designed to help distribute resources efficiently so that more intensive and time-consuming diagnostic testing can be administered to persons at greater risk (ie, those exceeding a specific threshold). It can be difficult, however, to determine the most appropriate cutoff score for identifying those who require further diagnostic assessment. This study underscores a basic screening tenet: that an optimal cutoff score is population-specific and should be based on the prevalence of undiagnosed disease in the target population, the availability of interventions and resources, and the relative importance of identifying all true positives. The last becomes increasingly important as disease severity increases but is invariably balanced by the cost of evaluating greater numbers of false positives. This study is the first to validate the use of the PCL as a screening tool for PTSD in retired firefighters and to determine an optimal cutoff score for predicting a diagnosis of PTSD.

Our goal was to compare the performance of the self-report screening PCL-17 with DSM-IV diagnostic criteria—the recognized “gold standard.” DSM-IV criteria were systematically assessed in structured interviews by trained mental health professional interviewers. Several instruments, other than the DIS, could have been used to assess diagnostic criteria. These include the Clinician-Administered PTSD Scale (CAPS) and the Structured Clinical Interview for the DSM-IV Axis I Disorders (SCID). Acknowledging that no universal preference exists,22 we chose the DIS for several reasons: 1) its demonstrated effectiveness in identifying mental health disorders in a large sample,23 2) its accuracy when administered by trained mental health interviewers, which mimics the results of psychiatrist interviews,24 3) its proven validity in clinical settings,25 4) its fully-structured format (rather than semi-structured), which affords greater inter-rater reliability and validity, and 5) its ability to assess the full DSM-IV criteria for PTSD, including criterion A (exposure to a traumatic event). Compared with other assessment instruments, use of a structured interview such as the DIS to assess full diagnostic criteria permits appropriate focus on the diagnosis as defined in the DSM-IV.

We found that 6% of retirees had PTSD using the DIS to assess full DSM-IV criteria for PTSD. Consistent with other studies, the PCL had very good diagnostic value relative to a DSM-IV diagnosis, with an AUC of 0.91 out of a possible 1.0.13,19 The cutoff score of 446,15 was acceptable, identifying 16% of the retired firefighters with elevated PTSD risk, whereas lowering the score to 39 optimized sensitivity with only a minor reduction in specificity. Our data also demonstrate that either cutoff demonstrated higher sensitivity and Youden index than using a PCL score based on the DSM-IV criteria.

Our population consisted of retirees with an average age of 47.0 on 9/11. The youngest retirees had significantly higher average PCL scores than older retirees. Disability pensions also were granted to a significantly higher percentage of younger retirees. This is explained by the fact that age is positively correlated with years of service (R2=0.76). Actives typically retire after at least 20 years of service when they qualify for regular pensions, whereas those with work-related injuries/illnesses may qualify for disability pensions with fewer years of service. Those granted disability pensions had significantly higher average PCL scores (32.0) than those who did not retire on a disability pension (27.9). We found this difference in mean PCL scores between retiree groups even though—in an effort to protect against inflating the relationship between disability retirement and PTSD—we excluded from all analyses individuals who retired solely because of a mental health disability (N=31). This supports the idea that the mental health consequences of 9/11 may have been underappreciated in relation to physical illnesses and suggests the need to further investigate comorbid psychopathology and the relationship between PTSD and physical disability.

Retirees with the earliest arrival time generally suffered the greatest physical and psychological trauma. These firefighters were at the WTC site either during the collapse or shortly thereafter. We found that 9/11 exposure Group 1 had greater average PCL scores (36.4) than exposure groups with later arrival times (26.1 to 30.3), which is consistent with previous findings.1,5,8 In our multivariate models, exposure groups had a dose-response relationship with PTSD. Because 9/11 exposure group is such an important factor in predicting PTSD in this and other populations, it may be useful to consider adjusting the cutoff scores used in screening tools for those arriving earliest to a disaster to maximize identification of true positives.

This study had a few limitations. First, retiree participation was self-selected, and participants were significantly more likely to live in the NYC area than nonparticipating retired firefighters who met inclusion criteria. We cannot rule out the possibility that living in the NYC area may have been associated with PTSD because we do not have PTSD data on nonparticipants. Participants also were older and more likely to retire as officers compared with nonparticipants, both variables that were related to lower mean PCL scores in bivariate analyses. Participating and nonparticipating retiree groups were similar in their proportions of most highly exposed and disabled. Therefore, we believe it is reasonable to extrapolate our PCL threshold scores to the entire WTC-exposed retiree population. Because our study population was predominantly white men, the findings may be limited to this demographic. Others,9,26 however, did not find that sex or race were significant factors in setting cutoff scores of the PCL in Veterans Affairs primary care settings.

Second, exposure group based on earliest arrival time at the WTC site may have been influenced by recall bias, although we tried to minimize its potential impact by using exposure information from the earliest post-9/11 assessments. Third, we did not collect information on lifetime exposure to traumatic events or diagnoses of PTSD before 9/11. Therefore, some of the elevated PTSD risk we found might not have been a consequence of 9/11. We addressed this possibility in part by modifying the PCL to fit the context of 9/11. Finally, the DIS as a fully structured interview affords improved reliability but may prevent clinical flexibility.

ACKNOWLEDGEMENTS: This work was supported by the National Institute for Occupational Safety and Health (NIOSH) RO1-OH07350.

DISCLOSURES: The authors report no financial relationship with any company whose products are mentioned in this article or with manufacturers of competing products. The authors alone are responsible for the content and writing of the paper. The contents of this article do not represent the views of the Department of Veterans Affairs or the United States Government.

    REFERENCES

  1. Galea S, Ahern J, Resnick H, et al Psychological sequelae of the September 11 terrorist attacks in New York City. N Engl J Med. 2002;346:982–987.
  2. Epstein RS, Fullerton CS, Ursano RJ. Posttraumatic stress disorder following an air disaster: a prospective study. Am J Psychiatry. 1998;155:934–938.
  3. Fullerton CS, Ursano RJ, Wang L. Acute stress disorder posttraumatic stress disorder, and depression in disaster or rescue workers. Am J Psychiatry. 2004;161:1370–1376.
  4. Guo YJ, Chen CH, Lu ML, et al Posttraumatic stress disorder among professional and non-professional rescuers involved in an earthquake in Taiwan. Psychiatry Res. 2004;127:35–41.
  5. Kessler RC, Chiu WT, Demler O, et al Prevalence, severity, and comorbidity of 12-month DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. 2005;62:617–627.
  6. Perrin MA, DiGrande L, Wheeler K, et al Differences in PTSD prevalence and associated risk factors among World Trade Center disaster rescue and recovery workers. Am J Psychiatry. 2007;164:1385–1394.
  7. North CS, Tivis L, McMillen JC, et al Psychiatric disorders in rescue workers after the Oklahoma City bombing. Am J Psychiatry. 2002;159:857–859.
  8. Corrigan M, McWilliams R, Kelly KJ, et al Mental health outcomes in the FDNY firefighter World Trade Center cohort: a study based on a self-administered, computerized survey with 2.5 years of follow up. Am J Public Health. In press, 2009.
  9. Stellman JM, Smith RP, Katz CL, et al Enduring mental health morbidity and social function impairment in world trade center rescue, recovery, and cleanup workers: the psychological dimension of an environmental health disaster. Environ Health Perspect. 2008;116:1248–1253.
  10. Heinrichs M, Wagner D, Schoch W, et al Predicting posttraumatic stress symptoms from pretraumatic risk factors: a 2-year prospective follow-up study in firefighters. Am J Psychiatry. 2005;162:2276–2286.
  11.  Diagnostic and statistical manual of mental disorders. 4th ed. Washington DC: American Psychiatric Association; 1994.
  12. Weathers FW, Litz BT, Herman DS, et al The PTSD Checklist (PCL): reliability, validity, and diagnostic utility. Presented at: Annual Meeting of the International Society of Traumatic Stress Studies; San Antonio, TX; October 24-27, 1993.
  13. Lang AJ, Laffaye C, Satz LE, et al Sensitivity and specificity of the PTSD checklist in detecting PTSD in female veterans in primary care. J Trauma Stress. 2003;16:257–264.
  14. Forbes D, Creamer M, Biddle D. The validity of the PTSD checklist as a measure of symptomatic change in combat-related PTSD. Behav Res Ther. 2001;39:977–986.
  15. Blanchard EB, Jones-Alexander J, Buckley TC, et al Psychometric properties of the PTSD Checklist (PCL). Behav Res Ther. 1996;34:669–673.
  16. Robins LN, Cottler L, Bucholz K, et al Diagnostic Interview Schedule for DSM-IV. St. Louis, MO: Washington University; 1995.
  17. Prezant DJ, Weiden M, Banauch GI, et al Cough and bronchial responsiveness in firefighters at the World Trade Center site. N Engl J Med. 2002;347:806–815.
  18. North CS, Pfefferbaum B. Research on the mental health effects of terrorism. JAMA. 2002;288:633–636.
  19. Smith MY, Redd W, DuHamel K, et al Validation of the PTSD Checklist-Civilian Version in survivors of bone marrow transplantation. J Trauma Stress. 1999;12:485–499.
  20. DiGrande L, Perrin MA, Thorpe LE, et al Posttraumatic stress symptoms, PTSD, and risk factors among lower Manhattan residents 2-3 years after the September 11, 2001 terrorist attacks. J Trauma Stress. 2008;21:264–273.
  21. Bewick V, Cheek L, Ball J. Statistics review 13: receiver operating characteristic curves. Crit Care. 2004;8:508–512.
  22. Kulka RA, Schlenger WE, Fairbank JA, et al National Vietnam Veterans Readjustment Study (NVVRS): description, current status, and initial PTSD prevalence estimates. Washington, DC: Veterans Administration; 1988.
  23. Robins LN, Helzer JE. Diagnostic Interview Schedule (DIS). Version III-A. St. Louis MO: Washington University School of Medicine; 1985.
  24. Breslau N, Davis GC. Posttraumatic stress disorder: the etiologic specificity of wartime stressors. Am J Psychiatry. 1987;144:578–583.
  25. Schlenger WE, Kulka RA. Performance of the Keane-Fairbank MMPI scale and other self-reported measures in identifying post-traumatic stress disorder. Paper presented at the 95th Annual Convention of the American Psychological Association. New York NY; August 28 -September 1, 1987.
  26. Yeager DE, Magruder KM, Knapp RG, et al Performance characteristics of the posttraumatic stress disorder checklist and SPAN in Veterans Affairs primary care settings. Gen Hosp Psychiatry. 2007;29:294–301.

CORRESPONDENCE: Mayris P. Webber, DrPH New York City Fire Department, Bureau of Health Services 9 Metrotech Center Brooklyn, NY 11201 USA E-MAIL: webberm@fdny.nyc.gov