European Heart Journal Cardiovascular Imaging (2013) 14, 167174 doi:10.

1093/ehjci/jes138

Dening the magnitude of measurement variability in the virtual histology analysis of acute coronary syndrome plaques
Scott W. Murray1,2*, Rodney H. Stables 1, George Hart 2, and Nicholas D. Palmer 1
1 Institute of Cardiovascular Medicine and Science, Liverpool Heart and Chest Hospital, Liverpool, UK; and 2Department of Critical Care Research, University of Liverpool, Liverpool, UK

Downloaded from http://ehjcimaging.oxfordjournals.org/ at McGill University Libraries on January 2, 2014

Received 22 April 2012; accepted after revision 17 June 2012; online publish-ahead-of-print 10 July 2012

Background

Previous intravascular ultrasound-based virtual histology (IVUS-VH) measurement variability studies have been conned to single-frame or short-segment analysis in stable patients with minimal disease. We sought to determine the magnitude of human measurement variability in acute coronary syndrome (ACS) plaques. ..................................................................................................................................................................................... Methods Prior to percutaneous coronary intervention, we performed IVUS-VH analysis in troponin-positive ACS culprit lesions. A total of 3840 IVUS-VH frames were analysed by two operators to determine intra- and inter-observer and results variability. The plaque constituent area and volumes were compared using intra-class correlation coefcient (ICC); within-subjects standard deviation (WSSD, mm2 or mm3) and the repeatability coefcient (RCO) to quantify the magnitude of operator error that 95% of future measurements should not exceed. The majority of intra- and inter-observer measurements had ICC of . 0.92 conrming excellent agreement. Only the brous area (0.86), bro-fatty (FF) area (0.72) and FF volume (0.87) had ICC levels suggesting an operator error . 10%. However, the mean RCO and the percentage this represents in single-frame analysis (area error) varied across the plaque subtypes: brous area 1.64 mm2 (59%); FF area 0.49 mm2 (140%); necrotic core (NC) area 0.39 mm2 (21.3%); dense calcium (DC) area 0.29 mm2 (33.7%). For full lesion pullbacks (volume error): brous volume 8.14 mm3 (9.9%); FF volume 5.63 mm3 (53.8%); NC volume 3.78 mm3 (6.9%) and DC 2.4 mm3 (9.6%) ..................................................................................................................................................................................... Conclusion As in previous studies, intra- and inter-observer ICC suggests good agreement between observers. However, this can still represent large measurement error values and percentages. These ndings could impact on the interpretation of previous studies and inuence future studies using IVUS-VH measurements as endpoints.

----------------------------------------------------------------------------------------------------------------------------------------------------------Keywords
Acute coronary syndromes Intravascular ultrasound Virtual histology

Introduction
Coronary plaque imaging with virtual histology (VH) allows the estimation of the quantitative composition of atherosclerotic plaques in vivo and has already been used as a surrogate outcome measure.1 However, there is currently no consensus on appropriate VH endpoints for clinical trials or for accurate power calculations to ensure a true difference is being detected. Repeat measurements of plaque composition can be subject to both reproducibility (inter-catheter and inter-pullback) error and variability (human measurement) error introduced in the following ways:

(i) Inconsistent denition by the operator of the surfaces to be measured (lumen, vessel). (ii) Inconsistent selection of the segment to be measured (proximal and distal marker frames). (iii) Difculty in distinguishing plaque margins, for example blood from thrombus and tissue. (iv) Difculty in distinguishing plaque components such as calcication from adventitia or artefact. Recent serial clinical IVUS-VH studies, such as the IBIS-2 trial1 have revealed only small changes in the plaque volume and its relative composition following pharmacological intervention. The

* Corresponding author. Tel: + 44 7828 242328; fax: + 44 151 600 1699, Email: scottmurray@doctors.org.uk Published on behalf of the European Society of Cardiology. All rights reserved. & The Author 2012. For permissions please email: journals.permissions@oup.com

168
measurement variability of volumetric IVUS-VH analysis is therefore an important scientic issue. In the past there has been a distinct lack of published literature documenting variability or reproducibility for volumetric IVUS-VH measurements in the clinical setting.2 4 Previous validation studies have been conned to single-frame area analysis or volumetric analysis in pre-dened, short coronary segments with mild disease in stable patients. These patients were not scheduled for coronary intervention.5 To date, assessments of the level of agreement have been based on calculating the paired differences in data and applying a token rule of , 10% difference in mean values as being clinically acceptable. This does not take into account the inherent standard deviation (SD) within the data which has a dramatic effect on nal numerical calculations of individual plaque measurements. The knowledge of the observer variability of any measurement is essential for validation of scientic techniques and in this case, it is required to ensure a high predictive accuracy in serial plaque measurement and composition. Therefore, in this study, we assessed the intra- and interobserver variability of individual plaque areas, volumes and segment length selection in a single pullback of acute coronary syndrome (ACS) plaques. We utilized a selection of detailed statistical tests of measurement error including the repeatability coefcient (RCO). Our aim was to comprehensively determine the limits of human variability error for this proposed clinical tool.

S.W. Murray et al.

xed length (n 300 frames) were also individually analysed by each operator to provide comparative data on area and volume calculations from a xed length model. Following individual observer manual adjustment of borders in keeping with previous consensus statements,6 8 plaque variables and constituents were then calculated by the VH algorithm over the entire length of a lesion9 Data were collected to include the lumen area; vessel area; plaque area; media area; brous; bro-fatty (FF); necrotic core (NC); dense calcium (DC) area are as well as the total plaque burden. The total volume of the brous; FF; DC and NC plaque was also quantied in mm3. The technique and validation of IVUS-VH have been described elsewhere6,10 13 and in a single human study, histological verication has shown accuracy.10

Statistical methods
There are limited previous studies discussing the calculation of power for variability analyses. Bartlett et al.14 published a paper focused purely on measurement errors in continuous variables; we used the statistical methods from this to calculate our provisional limits of agreement for each measurement involved in this work. Using NC data from our rst ve patients, the mean of the paired difference was 0.74 mm3 with a SD of + 0.82 mm3. To calculate the limits of agreement for a Bland Altman plot of variability we used these results in the previously veried formula15,16: Limits of agreement = mean difference + 1.96 SD From these data, the worked limits of agreement are from 2 0.867 to 2.34 mm3. To calculate the standard error of these limits and the subsequent 95% condence intervals (CI), we used our SD and n 5 in the formula: Standarderror = 1.71 SD (n) This produces a standard error of 0.63 and in turn 95% CIs of ( 2 2.09, 0.36) and (1.12, 3.58), respectively. The data for ve patients produce a standard error of 0.63 which are reduced to 0.31 by increasing the sample size to 20 patients (assuming the SD remains relatively constant). A standard error of 0.31 ensures 95% CIs around our limits of agreement, which is also within the previously acceptable Bland Altman variability of 10%.

Downloaded from http://ehjcimaging.oxfordjournals.org/ at McGill University Libraries on January 2, 2014

Methods
This was a single-centre, prospective, observational study that received ethical approval from a regional National Health Service ethics committee in the UK. Patients presenting with ACS gave written informed consent and were studied prior to percutaneous revascularization. IVUS-VH imaging of the culprit lesion (CL) was undertaken using a phased array catheter (EagleEye catheter, 2.9 F/20 MHz; Volcano Corp). Marker frames were chosen in keeping with a consensus document on how to analyse both IVUS and VH.6 8 The IVUS transducer was advanced beyond the presumed CL and beyond the next distal side branch which would be used as a marker for localization during IVUS image assessment. A proximal border for image assessment was established in a similar fashion using another side branch or the mouth of the guide catheter. Following administration of 1 mg of isosorbide dinitrate intra-arterially, a continuous motorized pull-back at 0.5 mm/s was performed (Volcano R100 pullback device) until the probe had passed the proximal segment marker. The plaque composition and other analyses were performed by off-line IVUS-VH analysis (S5i Tower and PC VIAS 3.0.394 software; Volcano Corporation) following manual adjustment of border contours. The data were analysed off-line at intervals of 2 weeks by two independent operators (S.W.M. and N.D.P.), to compare for intra- (S.W.M.1 and S.W.M.2) and interobserver variability (S.W.M.1 and N.D.P.). A segment length was initially determined by the operator choice of distal and proximal VH reference frames. This decision was made after consultation of angiographic images (if required), the grey-scale IVUS runs and a written description of the pullback segment markers. Short pullbacks ( , 10 mm) or pullbacks containing signicant calcication or large side-branches (obscuring adequate vessel delineation) were not included in the nal analysis. Thrombus laden or ruptured plaques were included, as they are a common feature in high-risk ACS patients. A separate cohort of high-risk ACS lesions with a

Intra-class correlation coefcient

A 10% variability limit, as in previous studies, without considering the inherent SD and the limits of agreement, underestimates error. Therefore, we have chosen to calculate the intra- and inter-observer agreement using intra-class correlation coefcient (ICC) as well as displaying the overall within-subject SD (WSSD). ICC was used over Kappa because our data are mainly continuous. ICC is effectively equivalent to a weighted Kappa statistic.17 We have included Bland Altman plots for visual illustration of variability. Optimal sample sizes for ICC are based on the desired power level, the magnitude of the predicted ICC and the lower condence limit. Bonnett et al. 18 in 2002, investigated sample size issues for ICC, concluding that the optimum sample size is a function of the size of the ICC and the number of ratings per subject, as well as the desired signicance level (alpha) and desired width (w) of the CI. Bonnett concluded that the optimal sample size for two ratings varied from 15 for ICC 0.9 to 378 for ICC 0.1. Our sample size of 20 is therefore adequate in this context.

Measurement variability of acute coronary syndrome plaques

169

The repeatability coefcient

In order to nd the maximum magnitude of measurement error during the analysis of these cases, we expanded our analysis to include the RCO. This is calculated from the WSSD as follows: RCO = mZxw

Patient characteristics
The basic patient clinical data are documented in Table 1.

IVUS-VH analysis and data

The mean segment length analysed in this study was 27.14 + 9.1 mm (range: 13 45 mm). The results of these analyses are presented in Tables 2 4 and displayed graphically with Bland Altman plots of inter-observer variability (Figure 2). The details of the separate individual area analyses and xed length volume analyses are displayed in Table 5.

where m is the number of observations per subject, Z is a quantile from the standard normal distribution (usually taken as the 5% two tailed quantile of 1.96) and Xw is the estimated WSSD (calculated previously). The RCO is a precision measure. It can be thought of as a CI for error. It represents the value, below which, the absolute difference between the two repeated test results may be expected to lie, with a probability of 95%. This is the denition adopted by the British Standards Institution.19 We can therefore say that 95% of our potential future measurement errors, in this cohort, should not be beyond the gure calculated by the RCO. Therefore, any change greater than this level is likely to represent a true change in the measured variable of the plaque.

Segment length
Despite this being an operator-based judgement where large amounts of variability could be introduced, the ICC remained . 0.90 at 0.986 for intra-observer and 0.96 for inter-observer. However, this equated to an RCO of 2.81 mm for intra- and 4.81 mm for inter-observer. This represented 10.4 and 17.4% of the total segment length analysed. For comparative assessment, the results from a xed length assessment of 300 frames (147.9 mm) are also displayed within Table 5.
Downloaded from http://ehjcimaging.oxfordjournals.org/ at McGill University Libraries on January 2, 2014

Statistical analysis
Statistical analysis was performed using the agreement function within Stats Direct statistical software version 2.7.7. Quantitative data are expressed as the mean + 1SD or WSSD (in mm, mm2 or mm3). The RCO is presented as a calculated area or volume in mm2 or mm3 and is also expressed as a proportion of the total measured area or volume. Bland Altman plots provide a visual display of the agreement within + 2 SD of the mean between measurements. An ICC of . 0.90 is considered excellent indicating that , 10% of the error is due to operator variability.

Total plaque volume

Results for this variable (along with % plaque burden, Table 5) show how heavily diseased our analysed segments were. The actual variability between operators was small considering the amount of plaque. ICC was . 0.99 for both intra- and interobserver, with paired differences between operators of only 0.24 + 0.6 mm3. This represents only 0.1% of the total plaque volume. However, due to the wide SD and resultant 95% limits of agreement, the repeatability error between operators is calculated at 15.51 mm3 (7.3% of the total plaque volume). This error is reduced to 3.32 mm3 (1.6%) by single-operator repeat analysis.

Results
Twenty lesions from 20 patients generated 1080 frames of IVUS-VH. Each frame was analysed three times (S.W.M.1, S.W.M.2, N.D.P., n 3240). A further 300 xed frames were analysed twice (S.W.M.1, N.D.P.) to provide data on individual area and volume variability from a xed length. Therefore, a total of 3840 frames were analysed on both the S5i platform and PC VIAS 3.0.394 (Figure 1).

Composition
IVUS-VH assessments of composition were highly correlated in the majority of plaque variables with ICC of . 0.92 (i.e. , 8% error between operators). From our xed data in ACS lesions: brous area (ICC 0.86); FF area (ICC 0.72) and FF volume (ICC 0.87) show less agreement between operators and this appears to be related to the placement of luminal borders around soft

Figure 1 An example of differing intravascular ultrasound-based virtual histology border detection between operators.

170

S.W. Murray et al.

Table 1

Patient clinical characteristics

Value or percentage 61.2 ( + 9.7) 80% 42.3% 36% 21.7% 60% 16% 24% 12.14 ( + 24.4) 1.4 (0.04 13.7) 8.5 ( + 2.3) 64% 24% 36% 16% 60% 44% 4% 12% 0% 0%

Characteristic Age ( + SD) Male LAD/diagonal RCA Circumex/OM NSTEMI

................................................................................

Table 3 Paired differences between individual observers and the relative percentages of the variable total
Vessel variable S.W.M. (1 1 2), mean + SD N.D.P., mean + SD Paired difference, + SD (% total)

................................................................................
Segment length (mm) Total plaque volume (mm3) Fibrous volume (mm3) FF volume (mm3) NC volume (mm3) DC volume (mm3) 27.2 + 8.9 212.66 + 103 27.1 + 9.4 212.90 + 103.6 0.1 + 0.5 (0.36) 0.24 + 0.6 (0.11) 0.07 + 0.7 (0.09) 0.62 + 0.4 (4) 0.51 + 0.2 (1.8) 0.33 + 0.4 (2.2)

STEMI (thrombolysed) ACS Mean troponin ( + SD) Median troponin (IQR) Local modied TIMI risk score Hypertension Diabetes Current smoker Ex-smoker Family history CAD Hypercholesterolaemia CKD Previous MI Previous PCI Previous CABG

79.33 + 39.1 15.43 + 8.7 29.02 + 32.2 14.70 + 18.7

79.40 + 39.8 16.05 + 9.1 28.51 + 32 15.03 + 18.3

Downloaded from http://ehjcimaging.oxfordjournals.org/ at McGill University Libraries on January 2, 2014

Discussion
We have investigated and reported the intra- and inter-observer variability for volumetric VH analyses in coronary lesions implicated in an ACS presentation. From the results, it is clear that variability in segment length selection between operators exists. This is one of the most important limitations of IVUS technology and serial studies in the real world (as reliance is placed on local landmarks such as side branches and calcium deposits). Moreover, assessing a long segment of disease requires analysis of multiple serial tomographic slices recorded during a pullback sequence. These slices must be manually corrected by the individual and the area of the lumen and external elastic membrane must be traced accurately to ensure viable results. Although the ICC appears to state that our variability error for tracing the lumen and vessel areas is acceptable between 1 and 4% (ICC: 0.99 0.96), when calculating the more quantiable RCO, we appear to have differences between operators up to 14% for the vessel area and 31% for the lumen area. Interestingly, the analysis of 300 individual frames between operators has revealed signicant differences with regard to plaque areas (see Table 5). Although the paired difference of the mean appears very small (as documented in other studies) by calculating the WSSD and RCO, we see that maximum error levels for the FF plaque area can be greater than the measured area itself (RCO 0.49 mm2 140% of the FF area, 0.32 mm2). Although on rst consideration the inherent variability of volume measurements should be higher as subsequent area errors are multiplied by length, the opposite appears to be true. Volume measurements appear to smooth out the individual frame errors over the length of the lesion. This could be because one operator may measure a greater area than the other operator in one frame, but the reverse may be true on the next and over a large lesion this may balance out some of the volume variability. This is the rst report to highlight that

Table 2 variable
Vessel variable

Calculated observer means + SD, per

................................................................................
Segment length (mm) Total plaque volume (mm3) Fibrous volume (mm3) FF volume (mm3) NC volume (mm3) DC volume (mm3) 27.02 + 9.1 212.63 + 103 27.29 + 8.7 212.68 + 102.9 27.12 + 9.4 212.90 + 103.6

S.W.M.(1), mean + SD

S.W.M.(2), mean + SD

N.D.P., mean + SD

78.99 + 38.9 15.13 + 8.6 29.10 + 32.3 14.68 + 18.8

79.66 + 39.2 15.73 + 8.7 28.93 + 32.1 14.62 + 18.6

79.40 + 39.8 16.05 + 9.1 28.51 + 32 15.03 + 18.3

S.W.M., Observer 1; NDP, Observer 2.

plaques and thrombus. It is also clear that one of the reasons for the large limits of agreement and RCO is the large SD of the data. This is secondary to the large spread of plaque values and the large variation seen between individual plaque components. The relevant results for plaque composition are displayed in Tables 2 5 and Figures 2 4.

Measurement variability of acute coronary syndrome plaques

171

Table 4 total
Vessel

Observer measures of agreement, precision and repeatability with the relative percentages of the variable

Intra-observer ICC

................................................................
WSSD (% total) 1.01 (3.7) 1.2 (0.56) 1.85 (2.3) 1.49 (9.7) 0.37 (1.3) 0.26 (1.8) RCO (%total) 2.81 (10.3) 3.32 (1.6) 5.12 (6.5) 4.13 (26.8) 1.04 (3.6) 0.73 (5) 0.986

Inter-observer ICC 0.96 0.997 0.993 0.947 0.994 0.974

.................................................................
WSSD (% total) 1.74 (6.4) 5.59 (2.6) 3.14 (4) 1.99 (12.9) 2.42 (8.3) 2.89 (19.7) RCO (% total) 4.82 (17.7) 15.51 (7.3) 8.71 (11) 5.52 (35.8) 6.71 (23.1) 8.01 (54)

...............................................................................................................................................................................
Segment length (mm) Plaque volume (mm3) Fibrous volume (mm3) FF volume (mm3) NC volume (mm3) DC volume (mm3) 1 0.998 0.969 1 1

ICC, intra-class correlation coefcient; WSSD, within-subject standard deviation; RCO, repeatability coefcient.

Downloaded from http://ehjcimaging.oxfordjournals.org/ at McGill University Libraries on January 2, 2014

there could be a regression-type phenomenon within the values seen for volume measurement error. In our cohort, the use of xed length analysis improved the errors seen in the measurement of NC and DC components. However, the measurement of brous and FF components did not change greatly and this may be due to the severe nature of the plaques analysed with thrombus, plaque rupture and difcult luminal border interpretations. The main source of error in all of these cases is a direct reection of the difculties in selecting the true lumen-blood interface (brous and FF plaque calculations). ACS plaques have a high thrombus burden (usually coded as the FF plaque in VH) and this has affected our ability to distinguish blood speckle from true plaque or thrombus.20 Despite the above results and as in previous studies,2 5 the classical markers of agreement such as ICC and paired difference of the mean prove that basic IVUS-VH data can be well matched. Our ICC correlation is in keeping with all previously published results, including a recent multi-centre IVUS-VH reproducibility registry3 which reported values of ICC of . 0.9 for all plaque components and total plaque volume. These results were in mildmoderately diseased coronary segments without a culprit stenosis and this is clear if you compare the maximum total plaque volume and the maximum NC volume between their study (A) and ours (B): A, TPV 91.9 + 27.3; NC 9.2 + 8.8; B, TPV 212.66 + 103; NC 54.98 + 45.6. It appears from these gures that we have examined the severe end of the spectrum of coronary disease with NC and plaque volumes being up to six times greater, from relatively similar pullback lengths. There is no doubt that with increasing plaque burden comes increasing SDs and subsequently statistical variability errors. The previous benchmark reproducibility and variability study was performed in 15 patients undergoing IVUS-VH before percutaneous coronary intervention (PCI).5 In this study, RodriguezGranillo et al. used two independent observers from a well-established core lab and focused on geometrical and compositional elements of IVUS-VH output and reported acceptable reproducibility with , 10% error. They published the limits of agreement around the measurement of the NC area. An interobserver range between + 0.16 and 2 0.20 mm2 was seen, indicating a potential for maximum error values of around 0.36 mm2 for NC across individual frame analysis (between operators). This is

very similar to what we have calculated with the RCO for the NC area (0.39 mm2). Although this inter-observer variability appears reasonable and suggests an overall mean 6% change in the NC area measurement, if it were applied to a 50 frame volume analysis, then it may produce worst-case errors up to a maximum of 18 mm3 NC. However, as we have demonstrated in our volume analysis, subsequent over or under estimations can cancel each other out over the full lesion length. This may explain our relatively accurate error results of 6.7 mm3 for the NC volume in variable segment lengths and 3.78 mm3 for xed length assessments. Importantly, a recent inter-catheter reproducibility study by Prasad et al. in 20084 showed that in a subgroup of volumetric analysis, involving only 12 segments with a mean length of 11 mm, the RCO for inter-catheter error produced large compositional differences of brous of 13.89 mm3; FF of 2.95 mm3; NC of 10.46 mm3 and DC of 14.16 mm3. These results and a warning in the discussion chapter of other papers2,3,5 state that volumetric VH analysis (between catheters and pullbacks) produces high levels of measurement error. Despite these known issues, papers rarely document clearly (in mm3) the volumetric change that could be expected purely from a total measurement error.21 29 Opting to only report percentages or statistical analysis of paired differences may grossly underestimate true errors in compositional volumes. The IBIS-2 trial1 used data from the aforementioned RodriguezGranillo et al. 5 (single-frame area analysis) to justify adequate reproducibility and variability of IVUS-VH volume analysis. In this study, the change in the NC volume was used as a surrogate endpoint to evaluate pharmacotherapy. The nal absolute difference between placebo and Darapladib treatment was a 5.2 mm3 reduction in the NC volume. Unfortunately, this endpoint value lies within the inter-catheter reproducibility error for the NC in the study by Prasad et al.4 (10.46 mm3). This endpoint value of 5.2 mm3 does appear to be measurable between operators on a single pullback, as it lies above our inter-observer variability RCO of 3.78 mm3. Moreover, the power calculation for the IBIS II trial was initially based upon primary endpoint palpography and HS-CRP data. Two hundred and thirty-nine patients were analysed by VH-IVUS but if a repeat power calculation is performed based on the data from the paper (mean NC volumes around 22 26 mm3 with a SD of 25), then 12001600 patients would have

172

S.W. Murray et al.

Downloaded from http://ehjcimaging.oxfordjournals.org/ at McGill University Libraries on January 2, 2014

Figure 2 Bland Altman plots for the variables measured between different operators with solid lines representing the 95% limits of
agreement.

Measurement variability of acute coronary syndrome plaques

173

Table 5

Fixed length inter-observer measurement of 300 individual VH frames

Observer 1 (S.W.M.), + SD 6.58 + 4.1 14.99 + 6.17 8.41 + 3.94 2.95 + 0.64 56.46 + 15 2.55 + 1.46 0.32 + 0.34 1.73 + 0.62 0.86 + 0.83 82.39 + 41.3 9.65 + 5.8 54.77 + 44.8 25.87 + 18.4 Observer 2 (N.D.P.), + SD 6.24 + 3.4 14.98 + 6.13 8.75 + 4.42 2.96 + 0.59 57.39 + 14.28 2.77 + 1.69 0.35 + 0.34 1.83 + 0.74 0.84 + 0.84 83.15 + 44.05 10.45 + 6.17 54.98 + 45.6 25.22 + 17.7 Paired difference, + SD (% total) 0.34 + 0.7 0.01 + 0.04 0.34 + 0.48 0.01 + 0.05 0.93 + 0.72 0.22 + 0.23 0.03 + 0.00 0.1 + 0.12 0.02 + 0.01 0.76 + 2.75 0.80 + 0.37 0.21 + 0.8 0.65 + 0.70 ICC WSSD (% total) 0.74 (11) 0.78 (5) 1.05 (12) 0.17 (5.7% ) 3.72 (6.5) 0.59 (21.3) 0.18 (51.4) 0.18 (9.8) 0.10 (11.6) 2.94 (3.5) 2.03 (19.4) 1.36 (2.5) 0.89 (3.4) RCO (% total)

Variable, mean (mm2 or mm3) Lumen area Vessel area Plaque area Media area Plaque burden % Fibrous area FF area NC area DC area Fibrous volume FF volume NC volume DC volume

...............................................................................................................................................................................
0.96 0.99 0.94 0.92 0.94 0.86 0.72 0.96 0.98 0.99 0.87 0.99 0.99 2.06 (31.3) 2.1 (14) 2.92 (33.3) 0.48 (16.2% ) 10.31 (17.9) 1.64 (59) 0.49 (140) 0.39 (21.3) 0.29 (33.7) 8.14 (9.9) 5.63 (53.8) 3.78 (6.9) 2.49 (9.6)

Downloaded from http://ehjcimaging.oxfordjournals.org/ at McGill University Libraries on January 2, 2014

ICC, intra-class correlation coefcient; WSSD, within-subject standard deviation; RCO, repeatability coefcient.

Figure 3 The magnitude of the repeatability coefcient error

between observers.

Figure 4 The repeatability coefcient as a percentage of the total measured plaque between observers.

been required at a standard power (80%; a 0.5) to prove a real difference. Clearly, these ndings impact greatly on the validity of some serial studies in this eld.21 29 Also, a very recent variability analysis by Obaid et al.30 has shown that in comparison with core-lab analysis, there can be signicant errors in judgement when identifying plaque sub-types (VH-TCFA, etc.) between operators. This further reduces the clinical utility of this tool and has implications for studies where different observers make judgements about plaque types.

Limitations
This study was performed in a single centre, by two experienced operators, without the aid of an industry sponsored core lab. Some of our analyses were performed by an operator purely on

the S5i platform and in some cases these were compared with the results from an operator on PC VIAS 3.0.394. Potential differences in the classication tree or software may therefore have inuenced plaque components. However, it is hoped that there are not further variability errors to consider between software packages as well as operators, catheters and pullbacks. Studying high-risk ACS plaques introduces large variations between individual plaque components and therefore creates large SD values. This in turn affects the calculation of limits of agreement and the RCO. It may be that the inherent variability in these plaque types precludes them from future serial study or that the spread of this data makes the RCO less sensitive. Furthermore, our results can only be applied to this cohort of patients, examined by these operators and will therefore not be

174
representative of future measurements made by other operators in non-ACS patients. Our aim has been to provide a cautionary guide to the potential of measurement error with this technique, even in experienced hands. This will hopefully forewarn future result interpretation by operators in a clinical setting.

S.W. Murray et al.

Conclusions
As in previous studies, there appears to be good agreement between IVUS-VH observers. However, when the SD of the data is taken into account, this can occasionally represent large measurement error values and percentages. These ndings could impact on the interpretation of previous studies and inuence future studies using IVUS-VH measurements as endpoints.

Acknowledgments
The Cardiology staff and patients from the Liverpool Heart and Chest Hospital. Conict of interest: S.W.M.: Travel and accommodation from Volcano Corporation to attend an international conference. R.H.S.: none. George Hart: none. N.D.P.: Honorarium and travel from Volcano Corporation for IVUS training courses.

Funding
Liverpool Heart and Chest Hospital: The Johnson Interventional Cardiology Research Fellowship.

References
a-Garc a HM, Buszman P, Erne P; for the Integrated Biomarker 1. Serruys PW, Garc and Imaging Study-2 Investigators. Effects of the direct lipoprotein-associated phospholipase a2 inhibitor darapladib on human coronary atherosclerotic plaque. Circulation 2008;118:1172 82. 2. Hartmann M, Mattern ES, Huisman J et al. Reproducibility of volumetric intravascular ultrasound radiofrequency-based analysis of coronary plaque composition in vivo. Int J Cardiovasc Imaging 2009;25:1323. 3. Huisman J, Egede R, Rdzanek A, Bose D. Between-centre reproducibility of volumetric intravascular ultrasound radiofrequency-based analyses in mild-tomoderate coronary atherosclerosis: an international multicentre study. Eurointervention 2010;5:925 31. 4. Prasad A, Daisha J, Cipher B et al. Reproducibility of intravascular ultrasound virtual histology analysis. Cardiovasc Revasc Med 2008;9:71 7. a-Garc a HM et al. Reproducibility of intra5. Rodriguez-Granillo GA, Vaina S, Garc vascular ultrasound radiofrequency data analysis: implications for the design of longitudinal studies. Int J Cardiovasc Imaging 2006;22:621 31. a-Garc a HM, Mintz GS, Lerman A, Vince GD. Tissue characterisation using 6. Garc intravascular radiofrequency data analysis: recommendations for acquisition, analysis, interpretation and reporting. Eurointervention 2009;5:177 89. 7. Di Mario C, Gorge G, Peters R, Kearney P, Pinto F, Hausmann D. Clinical application and image interpretation in intracoronary ultrasound. Study group on intracoronary imaging of the working group of coronary circulation and of the subgroup on intravascular ultrasound of the working group of echocardiography of the European society of cardiology. Eur Heart J 1998;19:207 29. 8. Mintz GS, Nissen SE, Anderson WD, Bailey SR. American College of Cardiology Clinical Expert Consensus Document on Standards for Acquisition, Measurement and Reporting of Intravascular Ultrasound Studies (IVUS). A report of the American College of Cardiology Task Force on Clinical Expert Consensus Documents. J Am Coll Cardiol 2001;37:1478 92. 9. von Birgelen C, de Feyter PJ, de Vrey EA. Simpsons rule for the volumetric ultrasound assessment of atherosclerotic coronary arteries: a study with ECG-gated three-dimensional intravascular ultrasound. Coron Artery Dis 1997;8:363 9.

10. Nasu K, Tsuchikane E, Katoh O et al. Accuracy of in vivo coronary plaque morphology assessment: a validation study of in vivo virtual histology compared with in vitro histopathology. J Am Coll Cardiol 2006;47:2405 13. 11. Nair A, Kuban BD, Tuzcu EM, Schoenhagen P, Nissen SE, Vince DG. Coronary plaque classication with intravascular ultrasound radiofrequency data analysis. Circulation 2002;106:2200 6. 12. Nair A, Margolis P, Kuban BD, Vince DG. Automated coronary plaque characterization with intravascular ultrasound backscatter: ex vivo validation. Eurointervention 2007;3:113 30. 13. Nasu K, Tsuchikane E, Katoh O, Vince DG. Impact of intramural thrombus in coronary arteries on the accuracy of tissue characterization by in vivo intravascular ultrasound radiofrequency data analysis. Am J Cardiol 2008;101:1079 83. 14. Bartlett JW, Frost C. Reliability, repeatability and reproducibility: analysis of measurement errors in continuous variables. Ultrasound Obstet Gynaecol 2008;31: 466 75. 15. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986;1:307 10. 16. Bland JM, Altman DG. Measurement error and correlation coefcients. BMJ 1996; 313:41 2. 17. Fleiss JL. The equivalence of weighted kappa and the intraclass correlation coefcient as measures of reliability. Educ Psychol Meas 1973;33:6139. 18. Bonett DG. Sample size requirements for estimating intra-class correlations with desired precision. Statist Med 2002;21:1331 5. 19. British Standards Institution. Precision of test methods, Part 1: guide for the determination of repeatability and reproducibility for a standard test method. BS 5497. London: BSI, 1979. 20. Frutkin AD, Mehta SK, McCrary JR, Marso SP. Limitations to the use of virtual histology intravascular ultrasound to detect vulnerable plaque. Eur Heart J 2007;28:1783 4. 21. Van Mieghem CA, McFadden EP, de Feyter PJ, Bruining N. Noninvasive detection of subclinical coronary atherosclerosis coupled with assessment of changes in plaque characteristics using novel invasive imaging modalities: the Integrated Biomarker and Imaging Study (IBIS). J Am Coll Cardiol 2006;47:1134 42. 22. Kawasaki M, Sano K, Okubo M et al. Volumetric quantitative analysis of tissue characteristics of coronary plaques after statin therapy using three-dimensional integrated backscatter intravascular ultrasound. J Am Coll Cardiol 2005;45: 1946 53. 23. Yokoyama M, Komiyama N, Courtney BK et al. Plasma low-density lipoprotein reduction and structural effects on coronary atherosclerotic plaques by atorvastatin as clinically assessed with intravascular ultrasound radio-frequency signal analysis: a randomized prospective study. Am Heart J 2005;150:287. 24. Rodriguez-Granillo GA, Serruys PW, Mc Fadden E et al. First-in-man prospective evaluation of temporal changes in coronary plaque composition by in vivo ultrasound radiofrequency data analysis: an integrated biomarker and imaging study (IBIS) substudy. Eurointervention 2005;3:282 8. 25. Wei H, Schiele F, Descotes-Genon V, Oettinger J. Changes in unstable coronary atherosclerotic plaque composition after balloon angioplasty as determined by analysis of intravascular ultrasound radiofrequency. Am J Cardiol 2008;101:173 8. 26. Garcia-Garcia HM, Gonzalo N, Pawar R et al. Assessment of the absorption process following bioabsorbable everolimus-eluting stent implantation: temporal changes in strain values and tissue composition using intravascular ultrasound radiofrequency data analysis. A substudy of the ABSORB clinical trial. Eurointervention 2009;4:443 8. 27. Surmely JF, Nasu K, Fujita H et al. Coronary plaque composition of culprit/target lesions according to clinical presentation: a virtual histology intravascular ultrasound analysis. Eur Heart J 2006;27:2939 44. 28. Hong MK, Mintz GS, Lee CW, Suh J, Kim JH. Comparison of virtual histology to intravascular ultrasound of culprit coronary lesions in acute coronary syndrome and target coronary lesions in stable angina pectoris. Am J Cardiol 2007;100: 953 9. 29. Rodriguez-Granillo GA, Aoki J, Ong AT et al. Methodological considerations and approach to cross-technique comparisons using in vivo coronary plaque characterization based on intravascular ultrasound radiofrequency data analysis: insights from the Integrated Biomarker and Imaging Study (IBIS). Int J Cardiovasc Intervent 2005;7:52 8. 30. Obaid DR, Calvert PA, McNab D, West NE, Bennett MR. Identication of coronary plaque sub-types using virtual histology intravascular ultrasound is affected by inter-observer variability and differences in plaque denitions. Circ Cardiovasc Imaging 2012;5:86 93.

Downloaded from http://ehjcimaging.oxfordjournals.org/ at McGill University Libraries on January 2, 2014