Investigative Report

Accepted May 24, 2016; Epub ahead of print May 31, 2016

Acta Derm Venereol 2017; 97: 24–31.

Corr: Prof. Dedee F. Murrell, Department of Dermatology, St George Hospital, Gray Street, Kogarah, Sydney, NSW 2217, Australia. E-mail: d.murrell@unsw.edu.au.

Bullous pemphigoid (BP) represents the commonest form of autoimmune bullous diseases (AIBDs), characterised by autoantibodies directed against two keratinocyte epitopes of the dermal-epidermal junction, BP180 and BP230 (1–4).

Although our scientific understanding of BP has progressed significantly in recent years, there is a scarcity of randomised-controlled trials to guide evidence-based management of BP (5–8). Pooling of data through meta-analysis of existing studies can provide valuable information but this is only achievable if uniform outcome measures are available. Most studies, however, employed a concoction of outcome measures based on non-validated, subjective, or non-specific ratings of disease activity (5–8). As such, a significant obstacle in comparing treatment modalities is the lack of a standardised, validated scoring system for BP. Clinical trials in BP require a standardised, well-validated scoring system.

The Bullous Pemphigoid Disease Area Index (BPDAI) (9) and the Autoimmune Bullous Skin Disorder Intensity Score (ABSIS) (10) are two independent disease severity assessments proposed to measure disease extent in BP. So far, there have been only two validation studies for BPDAI and ABSIS in patients with BP. A study conducted in Greece (11) revealed that both BPDAI and ABSIS were significantly correlated with anti-BP180 titers but not with anti-BP230 titres. Anti-BP180, but not anti-BP230, titres has been established to correlate well with disease activity (12–14). Another study, conducted in France (15), found good correlation of BPDAI with clinical parameters of disease activity and proposed a cut-off value of 56 for severe BP.

However, the intra-rater and inter-rater reliability and responsiveness to change for both tools have not been determined. These are crucial when conducting clinical trials because in actual practice, doctors assessing patients may change from visit to visit and if the outcome measure is still reliable between different doctors, it can be more easily applied in studies and in real life practice. In addition, patient-reported outcome (PRO) measures are becoming more and more relevant in clinical trials, and hence the correlation between these subjective PROs and the objective scores are important. Therefore, we evaluated the reliability, convergent validity, and responsiveness of BPDAI and ABSIS and their correlation to validated PROs for bullous pemphigoid for use in clinical and research settings.

This observational study received ethical approval from the South Eastern Sydney Local Health District Human Research Ethics Committee – Northern Network.

All patients were older than 18 years and enrolled with informed consent. Inclusion criteria included clinical examination, histologic result from skin biopsy and immunofluorescence studies consistent with the diagnosis of BP. The patient sample included a range of disease severities (Table I).

Sample size was determined in consultation with a professional statistician from the University of New South Wales, Sydney, Australia. Calculations based on a review of similar studies showed that at least 30 patients and 100 separate visits would allow a sufficient intraclass correlation coefficient.

Thirty-two patients participated in a prospective-cohort validation study that was conducted over 4.5 years from February 2010 to September 2014. Several outcome instruments were completed at each routine clinic visit (baseline, week two, month one and subsequent 3-monthly visits). The principal clinician (D.F.M.) assessed patients using BPDAI, ABSIS, and Physician’s Global Assessment of Severity (PGA). BPDAI measures skin and mucosal involvement with two activity components, one for extent of blistering, one for extent of urticarial/eczematous change and a separate damage component (9). Damage scores in BPDAI are not intended for assessment of disease severity; they are there to remind clinicians not to score post-inflammatory effects as activity (9). ABSIS assesses all lesions for extent by the rule of nines with a multiplication factor for degree of erosion; it also has a subjective component for oral mucosal involvement, which is often zero for BP patients (10). The PGA has been validated in studying pemphigus, psoriasis, dermatomyositis, and other dermatoses (16–18).

Dermatology fellows, who changed at each visit, rated each BP patient independently, using BPDAI and ABSIS, for inter-rater reliability testing. For intra-rater reliability, repeated and blinded scoring was performed randomly after a mean of 30 min, in the period of March to September 2014. None of the assessors were informed beforehand about the rescoring activity to minimise recall bias.

To determine responsiveness and minimal clinically important differences (MCIDs), the principal clinician (D.F.M.) classified disease activity as improved, stable, or deteriorated since the last visit. This method of Physician’s Subjective Assessment of Clinical Improvement (PSACI) has been used in multiple responsiveness studies of the Cutaneous Lupus Disease Area and Severity Index (CLASI) and Localised Scleroderma Cutaneous Assessment Tool (19–21).

Patients were asked to complete the BPDAI-Pruritus, validated Autoimmune Bullous Disease Quality of Life (ABQOL) (22) and Treatment of Autoimmune Bullous Disease Quality of Life (TABQOL) (23) questionnaires to measure the subjective element of disease progression. ABQOL measures AIBD-specific disease burden while TABQOL measures burden attributable to the side effects of AIBD treatment, with higher scores indicating worse QOL. Patients’ sera were collected for anti-BP180 and anti-BP230 ELISA testing when routine safety bloods were taken at these visits.

BPDAI. The BPDAI tool (9) computes 2 scores: total BPDAI activity and total BPDAI damage. The total BPDAI activity score is the arithmetic sum of the 3 subcomponents – cutaneous blisters/erosions, cutaneous urticaria/erythema, and mucosal blisters/erosions. The total BPDAI damage score is the arithmetic sum of the items rated regionally for damage caused by more permanent features such as post-inflammatory hyperpigmentation, scarring and other. BPDAI quantifies lesion number and size thresholds. Lesions are rated based on the regions affected. BPDAI gives additional weighting to areas of the skin primarily affected in BP, such as the limbs, and less emphasis to scalp and face, to better differentiate clinical response in BP. Scores can range from 0 to 360 for BPDAI total activity (maximum 240 for total skin activity and 120 for mucosal activity), and 0 to 12 for BPDAI damage, with higher scores indicating greater disease activity or damage. BPDAI also has a separate subjective measure known as BPDAI-pruritus.

ABSIS. ABSIS (10) is an instrument that is widely used for the measurement of disease activity in patients with autoimmune blistering diseases. ABSIS uses Rule of Nines and Rule of Palms methods where the palm of the patient’s hand is set as 1% of total body surface area. The estimated percentage of body surface area involved is then multiplied with a weighting factor 1.5 (erosive, exudative lesions, blisters), 1 (erosive, dry lesions), and 0.5 (re-epithelialised lesions). The total score ranges from 0 to 206, with higher scores denoting more severe disease. It consists of subsections for skin involvement (maximum score of 150), mucosal extent (maximum score of 11), and patient-reported mucosal discomfort (maximum score of 45).

PGA. PGA is a ten-point Likert scale ranging from 0–10, indicating perfect health to worst skin condition. This scoring system was used to allow rating of disease activity by general overall impression. Patients were also categorised as having mild, moderate, or severe disease by the principle clinician.

PSACI. To determine responsiveness to minimal clinically significant changes, the principle clinician (D.F.M.) also classified disease activity in each patient as improved, stable, or deteriorated since the last visit.

Serum markers. Anti-BP180 titres have been established to correlate well with disease activity while anti-BP230 levels appear to be irrelevant to the extent of disease activity (12–14). Commercial ELISA kits (MBL) were used to measure anti-BP180 by bullous experts from Kurume University, Japan. The positive index for anti-BP180 level is ≥ to 9. We compared these levels to the scores of BPDAI and ABSIS.

BPDAI-Pruritus. Pruritus is a major symptom of BP that may signal disease onset and recurrence. A separate subjective component of the BPDAI, BPDAI-pruritus, was used to measure the severity of this. The intensity of pruritus was subjectively graded using a visual analog scale where 0 represented no itch and 10 as maximal itching. Patient marked an “x” to indicate severity of itching today, in the past week, and in the past month, producing a total score out of 30. In cases where patient was unable to reliably complete the grading, pruritus was inferred from the degree of excoriations, also scored out of 30.

ABQOL. Autoimmune bullous disease-specific quality of life (QOL) was measured using the ABQOL that has been shown to have good validity and reliability (22). This questionnaire consists of 17 items, which encompasses physical burden of the disease, psychiatric effects and effects to daily life functioning. Each question ranges from 0 to 3 points, with higher scores indicating worse QOL. The maximum ABQOL score is 51.

TABQOL. Side effects associated with the treatment of autoimmune blistering diseases can have significant risk of medical complications and impairment to QoL. TABQOL questionnaire was used in this study to measure the burden attributable to the effects of treatment specific to autoimmune bullous diseases (23). Similar to ABQOL, this questionnaire consists of 17 items, with higher scores indicating worse QOL due to treatment effects.

All statistical analyses were performed using software (SPSS, Version 22.0, IBM Corp, Armonk, NY; and Excel, Microsoft, Redmond, WA).

Reliability. Both inter- and intra-rater reliabilities were assessed by intraclass correlation coefficients (ICC) with 95% confidence intervals (CI), using Type A ICC with absolute agreement definition and two-way random effect analysis. An ICC value of above 0.81 is considered excellent (24). Visits were randomly selected using the Random Number Generator function on Excel. Selections and analyses were performed 3 times with different clinic visits to verify the results obtained. Bland-Altman plots were constructed to illustrate the spread of scores and inter-rater differences for the 2 scoring instruments. Linear regression analyses were conducted to detect any proportional bias in determining the inter-rater reliability.

Convergent validity. BPDAI activity/damage and ABSIS scores were examined against global severity rating, serum markers, and QoL measures. We used Pearson’s correlation with the p-value to determine correlation coefficients between pairs of variables. t-test was performed to evaluate if positive anti-BP180 and anti-BP230 titres predict higher BPDAI and ABSIS scores. A p-value of ≤ 0.05 was considered significant. Visits were randomly selected using the Random Number Generator function and repeated three times to verify the correlation coefficients obtained.

Responsiveness. To validate and assess the responsiveness of BPDAI and ABSIS to clinically significant changes, paired t-test was conducted to determine if there was a statistically significant difference in score to the previous visit. For a tool to be responsive to clinically meaningful changes, there should be significant differences in visits with improvement and deterioration, and no significant difference in visits that remained stable.

MCIDs. Receiver Operating Characteristic (ROC) analyses were performed to determine the MCIDs of BPDAI and ABSIS. The sensitivity, specificity and percentage of patients correctly classified for each signed change were determined using this method. MCIDs were chosen based on the mean signed change in BPDAI and ABSIS scores derived by responders, which is then confirmed by assessing the ROC at and around that cut-off, focusing primarily on the highest percentage of patients correctly classified.

Of the 32 BP patients, 40.6% were male and 59.4% were female, with a mean age of 78 years (range: 52–93 years). 65.6% of patients were Caucasian, 25% were Hispanic, and 9.4% were Asian.

All 32 patients were included in the convergent validity analysis. Twenty-two patients were assessed for inter-rater reliability and 15 patients had rescoring completed for intra-rater reliability. Twenty-seven patients had at least 2 visits with completed outcome measures, totalling to 108 visits included in the responsiveness analysis. 13 patients had anti-BP180 and anti-BP230 ELISA testing results from Japan (Table I).

For inter-rater reliability of the 22 BP patients, the ICCs (95% CI) for all visits (including no activity) were: BPDAI total activity 0.957 (0.901–0.982), BPDAI damage 0.807 (0.592–0.915), and ABSIS total score 0.881 (0.736–0.949) (Table II). BPDAI was demonstrated in Bland-Altman plots to have less variability in scores compared to ABSIS, particularly as scores increase in magnitude. This is shown by the distribution of the scores close to the mean inter-assessor difference of scores (Fig. 1). Linear regression analyses for both BPDAI and ABSIS showed a T-value with no statistical significance, revealing that there was no proportional bias.

For intra-rater reliability of the 15 BP patients, the ICCs (95% CI) were: BPDAI total activity 0.993 (0.969–0.998), BPDAI damage 0.939 (0.828–0.979), and ABSIS total score 0.995 (0.987–0.998) (Table II). Analysis of BPDAI and ABSIS subsectional ICCs for various subcomponents were also calculated for inter- and intra-rater reliabilities (Table II).

Two repeat analyses demonstrated similar findings (Table III). Four patients had no visible evidence of disease activity, generating physician-assessed scores of 0. ICCs were recalculated with these 4 patients excluded to remove the floor effect. The resulting ICCs were only slightly reduced for inter-rater reliability and had close to no difference for intra-rater reliability (Table III). Perfect correlations were obtained for the subjective component of the ABSIS because these BP patients had no oral involvement and hence all scored zero for pain when eating different types of food. The same occurred with the mucosal component (extent) for BPDAI.

BPDAI activity had a substantial correlation with ABSIS total score (r = 0.864, p < 0.001) while, as expected, BPDAI damage had low correlation with ABSIS total score (r = 0.362, p < 0.05). BPDAI activity (r=0.875, p < 0.001) showed higher correlation with PGA than ABSIS total score (r = 0.752, p < 0.001) with PGA. As with the Patsatsi study, both disease activity scores were significantly correlated with anti-BP180 levels (p < 0.05) but not with anti-BP230 levels.

BPDAI activity had moderate and significant correlations with BPDAI Pruritus, ABQOL and TABQOL. BPDAI damage was not significantly correlated with any subjective measure. ABSIS total score was only significantly correlated with BPDAI pruritus (Table SI) but not with ABQOL and TABQOL scores, which often remained high even when patients had no evidence of disease activity, due to anxiety about disease recurrence and concerns about their medications. When BPDAI and ABSIS were 0, mean (standard deviation, range, interquartile range) of ABQOL was 8.4 (5.7, 0–26, 4–12) and TABQOL 12.8 (8.2, 0–31, 6.5–18), with no correlation with QOL measures (Table III). All correlations were repeated two more times with different clinic visits and demonstrated similar findings to the first analysis (Table SI).

There were 59 pairs of evaluable visits, including 18 instances (30.5%) of clinically significant improvement, 24 instances (40.7%) of remaining stable, and 17 instances (28.8%) of clinically significant deterioration.

In instances of clinically significant improvement, the mean BPDAI activity and ABSIS total scores decreased by 11.5 (p < 0.001) and 12.6 (p < 0.01) points, respectively. In instances of clinically significant deterioration, the mean BPDAI activity and ABSIS total scores increased by 11.0 (p < 0.001) and 9.1 (p < 0.05) points, accordingly (Fig. 2). There were no statistically significant differences in visits that remained stable (p = 0.21 and 0.57).

An ROC analysis indicated that a 4-point decrease in the BPDAI activity score was 82.4% sensitive and 91.7% specific for improvement, resulting in 87.8% of patients being correctly classified as having clinically improved or remained stable. The area under the curve was 0.913 (Table IV). Another ROC analysis indicated that a 3-point increase in the BPDAI activity score was 76.5% sensitive and 91.7% specific for deterioration, resulting in 85.4% of patients being correctly classified as having clinically worsened or remained stable (Table IV). Area under the curve was 0.924 (Fig. 3). These areas under the ROC curves are considered excellent, as they are greater than 0.9 (25).

In categorising improvement against remaining stable, a 4.75-point decrease in ABSIS total score was demonstrated to produce 58.7% sensitivity, 83.3% specificity, and 73.2% correct categorisation. This can be maximised by using 8.6-points to produce 78% correct categorisation. In classifying deterioration against remaining stable, a 4-point increase in ABSIS total score generated 52.9% sensitivity, 87.5% specificity, and 73.2% correct classification (Table IV). These sensitivities, specificities, and percentages of correct classifications were substantially lower than those with using the BPDAI activity score (Fig. 3). In addition, ROC analyses for ABSIS total score indicated a smaller area under the curve for both clinical improvement (0.799) and deterioration (0.729). These areas are considered fair as they lie between the range of 0.7 and 0.8 (25).

Thus far no clinical trials in BP have used a validated severity score. In order to choose a primary outcome measure for BP, the severity score needs to have content validity, demonstrate excellent inter and intra-rater reliability, and be sensitive to changes in clinical improvement and deterioration – i.e. responsiveness. This study was able to compare the reliabilities of the BPDAI and ABSIS for BP and determine the MCID to determine improvement and worsening and disease scores. The study found that the BPDAI had excellent inter-rater reliability and a superior ability to discriminate responsiveness to therapy than the ABSIS for BP.

BPDAI’s better inter-rater reliability and precision are likely because it was developed in terms of content specifically for BP, whilst ABSIS was originally developed for pemphigus and then used for other AIBDs which rarely involve the mouth. BPDAI quantifies lesion number and size threshold, which has been shown in previous studies (16, 26–28) to be a more reproducible method of cutaneous assessment than percentage estimates of body surface area involvement used in ABSIS. The Rule of Nines and Rule of Palms methods adopted in ABSIS have been frequently illustrated to have poor accuracy, particularly at quantifying small areas with a tendency to overestimate the area involved (16, 29–31). The proportion of a patient’s palm area remains relatively constant throughout life at 0.76–0.78% of total body surface area, thus not exactly equivalent to 1% (32, 33). In the original ABSIS publication (10), the authors acknowledged that estimating with these rules can be challenging, and may result in inter-rater disagreement when raters are untrained. In this study, the primary assessor is an experienced dermatologist with special interest in bullous diseases and all other assessors are clinicians training full-time in dermatology practice. Both of the measures were found to have superior inter-rater reliability than the inter-reliability of the PGA visual analogue score found by other studies examining other dermatological diseases (34).

The intra-rater reliabilities for BPDAI activity (ICC 0.993) and ABSIS total scores (0.995) in our study were equally excellent. Intra-rater testing was only performed from March to September 2014, recruiting patients with limited disease extent. Therefore, although this result represents excellent test-retest reliability for low disease activity, it cannot be extrapolated to patients with more severe disease. Compared to inter-rater reliability, intra-rater reliability may be less important in real practice in clinics, as clinicians change for the same patient.

BPDAI activity was found to have excellent convergent validity against PGA and anti-BP180 titre. ABSIS had only fair correlation with PGA, indicating that it is less accurate in measuring disease extent of BP than the BPDAI. Both BPDAI activity and ABSIS total scores were revealed to have moderate correlation with BPDAI pruritus component, implying that patients with worse disease activity are more likely to have prevalent symptoms of pruritus.

Patient-reported QOL measures correlated substantially better with BPDAI activity than BPDAI damage score. This denotes that patients experiencing severe BP activity reported worse QOL and that residual hyperpigmentation and other features were not particularly affecting their QOL. Accordingly, composite scores that included damage – ABSIS and PGA, were less likely to correlate with QOL. As the BPDAI gives greater weighting to the areas of the skin relevant to BP patients, perhaps that is an additional explanation of why the BPDAI correlated with QOL measures and yet ABSIS did not. It is also worth noting that the QOL in patients with BP remained poor despite resolving disease activity, suggesting the importance of holistic, long-term management of these patients.

Other key new findings in this study were responsiveness and MCID. Responsiveness analysis of BPDAI and ABSIS helped in defining clinical improvement or deterioration, allowing trial investigators to objectively identify patients with improvement from treatment, or patients experiencing deterioration from relapse, from those undergoing no change. A lower MCID indicates better sensitivity at detecting a smaller degree of improvement or deterioration. MCIDs for ABSIS were higher and its sensitivity, specificity and area under the curve were substantively lower than BPDAI activity, resulting in a lower percentage of patients by approximately 10% classified in each category. BPDAI activity is therefore more capable of distinguishing minor differences in severity and correctly separating true responders from non-responders than ABSIS. BPDAI damage score, as expected, was not responsive to clinical change, emphasising the importance of separating activity and damage to avoid stability, or increase of the total score, while the active disease is improving.

In the responsiveness analysis, the sensitivity for the MCID selected was lower than specificity. Optimisation of sensitivity could have been achieved with lower change scores, but this would have caused specificity to decrease. For the objectives of a clinical trial, we agree with the CLASI responsiveness study for lupus (35) to consider specificity more important than sensitivity to minimise the degree of false-positive responses. This reduces the inclusion of participants who have not experienced true clinical change. As it is critical to have high specificity, we based the analysis on the percentage of patients correctly classified.

Limitations of our study include the limited sample size, given that BP primarily occurs in older patients who may have more comorbidities and difficulties in mobilising to the outpatient clinic. To compensate for this, we scored patients longitudinally at repeated clinic visits. This sample size is larger than the number of patients recruited for other previous validation studies, the pemphigus study (15 patients) (16) and CLASI responsiveness study (8 patients) (19), and most of those patients also had limited disease extent. Moreover, the highest BPDAI total skin activity and ABSIS total scores in this study were 77/240 and 65.5/206 (32.1% and 31.8% of possible maximum scores), respectively. As with the PASI, the severe cases are generally in the 25% to 45% range of the maximum score (36). Additionally, the BPDAI and ABSIS should be validated in other settings in which it would be employed thus another limitation was that this is a single-centre study.

The comprehensively validated BPDAI and ABSIS provide simple and quantitative clinical tools that homogenise the way disease activity is quantified. Also, they provide guidelines for evaluating clinical improvement or deterioration in bullous pemphigoid. This analysis provides a foundation for the validated and practical use of BPDAI in clinical trials and routine practice.

IRB status: The study received ethical approval from the South Eastern Sydney Local Health District Human Research Ethics Committee – Northern Network (HREC 06/94).

Presentation: An oral presentation of the study findings was awarded first prize at the St George and Sutherland Medical Research Symposium in Sydney, Australia, October 2014. The study’s findings were presented at the Australasian Society for Dermatology Research Cutaneous Biology Meeting in Queensland, Australia, September 2014; the Medical Dermatology Society Annual Meeting in San Fransisco, United States of America, March 2015; and 23^rd World Congress of Dermatology in Vancouver, Canada, June 2015.

The authors thank the Australasian Blistering Diseases Foundation for supporting this project and the Independent Learning Program of UNSW Australia.