Вы находитесь на странице: 1из 9

Chemico-Biological Interactions 255 (2016) 3e11

Contents lists available at ScienceDirect

Chemico-Biological Interactions
journal homepage: www.elsevier.com/locate/chembioint

Evaluation of multiple mechanism-based toxicity endpoints in

primary cultured human hepatocytes for the identification of drugs
with clinical hepatotoxicity: Results from 152 marketed drugs with
known liver injury profiles
Jie Zhang a, Utkarsh Doshi b, Ayako Suzuki c, Ching-Wei Chang a, Jürgen Borlak d, 1,
Albert P. Li b, *, 1, Weida Tong a, **, 1
Division of Bioinformatics and Biostatistics, NCTR/FDA, Jefferson, AR, USA
In Vitro ADMET Laboratories LLC, Columbia, MD, USA
Division of Gastroenterology, University of Arkansas for Medical Sciences and Central Arkansas Veterans Healthcare System, Little Rock, AR, USA
Center of Pharmacology and Toxicology, Hannover Medical School, Hannover, Germany

a r t i c l e i n f o a b s t r a c t

Article history: We report here the results of a collaborative research program to develop a robust and reliable in vitro
Received 6 September 2015 system to allow an accurate definition of the drug-induced liver injury (DILI) potential of new drug
Received in revised form entities during drug development. The in vitro hepatotoxic potential of 152 drugs with known DILI
31 October 2015
profiles were evaluated in primary cultured human hepatocytes with four mechanistically-relevant
Accepted 6 November 2015
endpoints: cellular ATP depletion, reactive oxygen species (ROS), glutathione (GSH) depletion, and
Available online 12 November 2015
caspase activation for apoptosis. The drugs, 80 in the testing set and 72 in the validation set, were
classified based on serious clinical/regulatory outcomes as defined by reported acute liver failure, black-
Drug induced liver injury
box warning, and/or withdrawal. The drugs were further sub-categorized for dominant types of liver
Acute liver failure injury. Logistic regression models were performed to calculate the area under the receiver operating
Oxidative stress characteristics curve (AUROC) and to evaluate the prediction potential of the selected endpoints for
In vitro toxicity testing serious clinical/regulatory outcomes. The ROS/ATP ratio was found to yield an excellent AUROC in both
Human hepatocytes the testing (0.8989, P < 0.0001) and validation set (0.8545, P < 0.0001), and was found to distinguish
Drug development drugs associated with severe from non-severe DILI cases (p < 0.0001). The results suggest that evaluation
of drugs in primary human hepatocytes using the ROS/ATP ratio endpoint may aid the definition of their
potential to cause severe DILI.
© 2015 Elsevier Ireland Ltd. All rights reserved.

1. Introduction inadequacy of the current testing strategy to accurately identify

perpetrators that can cause death or result in the need for liver
Drug-induced liver injury (DILI), being one of the most common transplantation. The shortfall of preclinical safety evaluations may
reasons for early termination of drug development and post- be attributed to the significant species difference in drug-
marketing regulatory actions [16,28], continues to be a major metabolizing enzyme pathways (e.g. species differences in meta-
challenge in the drug development and regulatory decision making bolic activation and detoxification) and toxicological mechanisms
processes. The occurrence of severe DILI (sDILI) events leading to [2,20]. The failure of clinical trials to identify sDILI events can be
acute liver failure (ALF) of marketed drugs illustrates the explained by their low incidence, which has been estimated to be 1
case per 100,000 patients per year [3,23].
Therefore, there is an unmet need for effective strategies at an
early stage in drug development to avoid the occurrence of sDILI.
* Corresponding author. 9221 Rumsey Road, Suite 8, Columbia, MD 21045, USA.
** Corresponding author. 3900 NCTR Rd, Jefferson, AR 72079, USA. One strategy is to identify the at-risk population. However, in spite
E-mail addresses: lialbert@invitroadmet.com (A.P. Li), Weida.Tong@fda.hhs.gov of an apparently promising finding that the major histocompati-
(W. Tong). bility complex (MHC) variant alleles HLA-B*5701 genotype is a
Senior authors contributed equally to the manuscript.

0009-2797/© 2015 Elsevier Ireland Ltd. All rights reserved.
4 J. Zhang et al. / Chemico-Biological Interactions 255 (2016) 3e11

major determinant of drug-induced liver injury due to flucloxacillin the United States Pharmacopeia (http://www.usp.org), and Sequoia
[7,8], an international genome-wide association study (GWAS) Research Products Ltd (http://www.seqchem.com/).
failed to identify common genetic variants amongst patients with DILI Classification Criteria: The 152 drugs were categorized
serious drug-induced liver injuries, thus suggesting that either based on their association with DILI events in order to assess the
genetic determinants are drug-specific or that the genetic events predictive performance of an in vitro experimental sDILI screening
are rare and are not readily identifiable [25,31]. It has been postu- approach using human hepatocytes. The drugs were classified as
lated that the individual differences in sensitivity to hepatotoxic severe DILI (sDILI), non-severe DILI (non-sDILI), general DILI (gDILI),
drugs are a result of the co-occurrence of multiple, transient, and non-DILI using the following criteria:
environmentally-related phenomena, and therefore cannot be
identified by genome characterization alone [19]. One promising 1. sDILI drugs: Drugs associated with severe DILI events leading to
ongoing focus is to develop accurate diagnostic early biomarkers one of the following:
such as microRNA-122 to complement serum transaminases in a. Drugs which were withdrawn in either the US or Europe or
order to allow early therapeutic intervention to avoid full devel- received FDA black-box warning in the US due to hepato-
opment of DILI events [29,33,34]. toxicity [6].
The study reported here is based on the hypothesis that drugs b. Drugs with ALF reported in the US and with causality adju-
found to cause idiosyncratic hepatotoxicity may have common dicated by using a standardized causality assessment tool
chemical and biological properties which can be used for their and/or presented in the LiverTox website developed at the
identification [19]. We have selected primary human hepatocytes National Institute of Diabetes and Digestive and Kidney Dis-
as the experimental system to evaluate the hypothesis for the eases (NIDDK) [14].
following reasons: 1. While DILI may involve multiple cell types, c. Drugs with ALF reported in the US and associated with an
damage to hepatocytes is a major pathological finding for acute increased reporting frequency of ALF incidents calculated as
liver failure [11,32]. 2. Primary human hepatocytes are known to Empirical Bayes Geometric Mean (EBGM) and considered as a
retain key liver functions, especially human-specific drug metab- disproportional increase when 2.0 in the World Health
olizing enzymes and cofactors, and are generally considered the Organization (WHO) individual case safety report database
“gold standard” for the evaluation of human drug metabolism and (VigiBase™) [26,27].
hepatotoxicity [12,21]. 2. Non-sDILI: Drugs with DILI but without association with the
We report here results of the evaluation of 152 marketed drugs sDILI criteria described above for sDILI drugs.
in primary human hepatocytes with well-defined clinical hepato- 3. General DILI (gDILI): This group is comprised of all drugs asso-
toxicity using cellular ATP depletion, reactive oxygen species (ROS), ciated with DILI events (both sDILI and non-sDILI drugs).
glutathione (GSH) depletion, and caspase activation for apoptosis 4. Non-DILI: Drugs with no known association with DILI events.
as endpoints. We observed that the drug-induced elevation of ROS/
ATP ratio in human hepatocytes accurately identifies drugs asso- Clinical DILI findings: The Zimmerman's Hepatotoxicity text-
ciated with severe DILI events. book [36], the NIDDK “LiverTox” website [14], and a key review
paper for classifying immune-mediated DILI drugs [30] were used
2. Materials and methods to classify DILI drugs based on clinical pathological findings (DILI
phenotypes). Drug metabolism information was obtained from the
Drug Library: The 152 drugs (Tables 1 and 2) used in the study MICROMEDEX website (micromedex.com). All the information
were commercially obtained from Sigma Chemicals (St. Louis, MO), collected is available at FDA LTKB website [6].

Table 1
Drugs associated with severe DILI events (sDILI drugs) evaluated in the study. The drugs were evaluated in the initial experiment to evaluate analytic procedures to distinguish
sDILI drugs from non-sDILI drugs (test set) and in an independent experiment to validate the selected procedures (validation set). The criteria for the annotation as sDILI drugs
(SD criteria) were: 1) Drugs which were withdrawn in either US or Europe or received FDA black-box warning in the US due to hepatotoxicity. 2) Drugs with ALF reported in the
US and with adjudicated causality by either using a standardized causality assessment tool and/or presented in the LiverTox website developed at the National Institute of
Diabetes and Digestive and Kidney Diseases (NIDDK) [14,26]. 3) Drugs with ALF reported in the US and associated with an increased reporting frequency of ALF incidents
calculated as Empirical Bayes Geometric Mean (EBGM) and considered as a disproportional increase when 2.0 in the WHO individual case safety report database (VigiBase™)
[26]. Abbreviations for the types of DILI are as follows: HCN: Hepatocellular necrosis; HT: Hypertrophy; CHL: Cholestatic, Mixed: Mixed hepatocellular cholestatic liver injury;
NM: DILI not mentioned in the publications.

Testing set (31) Validation set (34)

Drug Criteria Type Drug Criteria Type Drug Criteria Type Drug Criteria Type

Acitretin #1 HCN Ketoconazole #1,3 HCN Abacavir #1,3 HCN Methyldopa #2,3 HCN
Allopurinol #3 Mixed Labetalol #3 HCN Atorvastatin #2 Mixed Mexiletine #1,3 HCN
Amiodarone #1,3 HCN Leflunomide #1,3 HCN Azithromycin #3 CHL Minocycline #2 HCN
Azathioprine #3 HCN Nefazodone #1,2,3 HCN Benzbromarone #1,3 HCN Naproxen #2 HCN
Carbamazepine #2,3 Mixed Nimesulide #1,2,3 HCN Bicalutamide #2,3 HCN Nelfinavir #3 HCN
Cyclofenil #1 HCN Nitrofurantoin #2,3 HCN Bosentan #1,3 CHL Nevirapine #1,3 HCN
Diclofenac #2 HCN Nomifensine #1 HCN Bromfenac #1,3 HCN Paroxetine #2 HCN
Disulfiram #2,3 HCN Propylthiouracil #1,2 HCN Cerivastatin #1 Mixed Pemoline #1,3 HCN
Fipexide #1 HCN Saquinavir #1 CHL; HT Ciprofloxacin #2 Mixed Phenytoin #2,3 Mixed
Fluoxetine #2 HCN Sertraline #2 HCN Clarithromycin #3 CHL Quetiapine 31 HCN
Flutamide #1,2,3 Mixed Simvastatin #2 Mixed Deferasirox #1 HCN Ritonavir #1,3 CHL
Ibuprofen #2 Mixed Tamoxifen #2 HCN Docetaxel #1 HCN Sulfasalazine #2,3 HCN
Indomethacin #2 HCN Troglitazone #1,3 CHL Imipramine #2 CHL Sunitinib #1 HCN
Isoniazid #1,2,3 HCN Valproic acid #1,2,3 Mixed Itraconazole #2 HCN Terbinafine #2 Mixed
Isotretinoin #1 NM Verapamil #2 Mixed Lumiracoxib #1 HCN Tolcapone #1,3 HCN
Zimelidine #1 HCN Mercaptopurine #3 HCN Trovafloxacin #1,3 Mixed
Venlafaxine #2 Mixed
Zafirlukast #3 HCN
J. Zhang et al. / Chemico-Biological Interactions 255 (2016) 3e11 5

Table 2
Drugs reported to be associated with DILI events but not classified as sDILI drugs (non-sDILI drugs) and drugs not associated with DILI events (non-DILI) evaluated in the study.
The drugs were evaluated in the initial experiment to evaluate analytic procedures to distinguish sDILI drugs from non-sDILI drugs (test), and in an independent experiment to
validate the selected procedures (validation). Abbreviations for the types of DILI are as follows: HCN: Hepatocellular necrosis; CHL: Cholestatic; STS: Steatosis; NM: DILI not
mentioned in the publications used for DILI classification.

Non-sDILI drugs (52)

Testing (31) Validation (21)

Drug Type Drug Type Drug Type Drug Type Drug Type

Aspirin HCN Etodolac Mixed Pioglitazone Mixed Amiloride HCN Melphalan HCN
Captopril CHL Felbamate HCN Pravastatin HCN Bleomycin HCN Methimazole CHL
Chlormezanone HCN Fenofibrate HCN RifampicinƗ HCN Cisplatin HCN Norfloxacin Mixed
Chlorpromazine HCL Haloperidol CHL Rosiglitazone Mixed Enalapril CHL Omeprazole HCN
Chlorpropamide CHL Ifosfamide NM Stavudine Mixed Ethambutol CHL Piperacillin CHL
Cimetidine Mixed Isocarboxazid HCN Sulindac CHL Famotidine HCN Piroxicam CHL
Cyclophosphamide HCN Lovastatin CHL Tamsulosin HCN Fluconazole HCN Pyrazinamide HCN
Cyclosporine CHL Methotrexate HCN Tetracycline HCN Gemfibrozil HCN Ranitidine HCN
Dacarbazine HCN Nifedipine HCN Ticlopidine CHL Griseofulvin CHL Rofecoxib CHL
Erythromycin CHL Phenobarbital Mixed Zidovudine STS Lamivudine HCN Tacrine HCN
Esmolol HCN Lisinopril HCN

Non-DILI drugs (35)

Testing (18) Validation (17)

Atropine Bupropion Cyproterone Isoproterenol Oxytetracycline Tolbutamide Acyclovir Cefepime Emtricitabine Ganciclovir
Biotin Clotrimazole Folic acid Moxisylyte Primidone Toremifene Adefovir Cromolyn Etoposide Hydrocodone
Bisoprolol Colchicine Glimepiride Naltrexone Streptozocin Trazodone Amantadine Cytarabine Fluorouracil Leuprorelin
Butorphanol Dantrolene Furosemide Moclobemide

Human hepatocytes: Cryopreserved plateable human hepato- Inc., St. Louis, MO) exclusion. Cell concentration of the hepatocyte
cytes (In Vitro ADMET Laboratories LLC, Columbia, MD) were used in suspension was adjusted to 0.5  106 per mL in CHPM. A volume of
the study. The hepatocytes had high viability (>85%) and plating 10 ml of the cell suspension (approximately 5000 cells) was added to
efficiency (>80%). Hepatocytes pooled from 10 individual donors each well of collagen-coated 384 well plates (CellaFix™; APSciences
(Table 4) were used to minimize individual variance. The hepato- Inc., Columbia, MD) and the cells were allowed to attach for 4 h
cytes were thawed and recovered in Cryopreserved Hepatocyte followed by addition of 10 ml of 2 dosing solution for 48 h.
Recovery Medium (CHRM™; APSciences Inc., Columbia, MD) and In vitro hepatotoxicity endpoints: Cellular ATP content, cas-
suspended in Cryopreserved Hepatocyte Plating Medium (CHPM; pase 3/7 activity, reduced glutathione, and ROS were quantified
APSciences Inc., Columbia, MD). The cell suspension was quantified using the ATPlite™ (Perkin Elmer), Caspase-GLO® 3/7 (Promega,
for viability and cell concentration was based on Trypan Blue (Sigma Madison, WI), GSH-Glo™ Glutathione (Promega, Madison, WI), and
the indicator H2DCFDA (Invitrogen Inc., San Diego, CA) assays, ac-
cording to the instructions given by the manufacturers. Each drug
Table 3 treatment contains 7 different concentrations starting at 250 mM
Drugs associated with DILI events reported to involve with 2-fold serial dilutions and each concentration was assayed in
immune-mediated hepatotoxicity [30] evaluated in the triplicate. The final DMSO concentration in all treatment groups
study. These drugs are annotated as sDILI or non-sDILI
was 0.2% v/v. The measurement was normalized with DMSO con-
trol and data from each triplicate was reported as mean ± SD. Dose-
Drug Type response curves were constructed by fitting a 4-parameter model
Allopurinol sDILI with the calculated mean. ROS values were corrected for cell
Azathioprine sDILI viability at each concentration as the ROS/ATP ratio prior to fitting
Carbamazepine sDILI doseeresponse curve. Caspase 3/9 values were likewise
Diclofenac sDILI
Fenofibrate Non-sDILI
Gemfibrozil Non-sDILI Study design and data analysis: The study was a collaboration
Ibuprofen sDILI between the FDA National Center for Toxicological Research
Imipramine sDILI (NCTR), Little Rock, Arkansas, and In Vitro ADMET Laboratories
Indomethacin sDILI
(IVAL), Columbia, Maryland. Researchers in NCTR were responsible
Isoniazid sDILI
Methimazole Non-sDILI for the procurement and preparation of DMSO stock solutions of
Methyldopa sDILI the 152 drugs. The drugs were coded and supplied to IVAL (assay
Minocycline sDILI team) for evaluation in human hepatocytes, with the test results
Nevirapine sDILI sent to NCTR (data analysis team) for data analysis. The identities of
Nitrofurantoin sDILI
Pemoline sDILI
the drugs assayed were blinded to the IVAL assay team. Evaluation
Phenytoin sDILI of the 152 drugs was performed in two stages. In the first stage, 80
Propylthiouracil sDILI drugs (31 drugs associated with sDILI vs. 49 non-sDILI drugs) were
Pyrazinamide Non-sDILI assayed as a testing set to identify the metrics and endpoints to
Rifampicin sDILI
predict a drug's likelihood for sDILI. Once the metrics were estab-
Simvastatin sDILI
Tetracycline Non-sDILI lished, the remaining 72 drugs were tested in the second stage as a
Troglitazone sDILI validation set to assess the predictive value of the identified metrics
Zafirlukast sDILI and endpoints.
6 J. Zhang et al. / Chemico-Biological Interactions 255 (2016) 3e11

Table 4
Donor demographic data of the hepatocytes employed in the study. The hepatocytes from ten donors were pooled and cultured for the study. The use of a pool of ten donors
was intended to minimize unique contributions from hepatocytes of a single donor, as individual variation is an inherent property of human hepatocytes (and of the human
population). The hepatocytes from all 10 donors had high viability (>85% based on trypan blue exclusion) and plating efficiency (>80%).

Donor Gender Race Age Cause of death

Donor 1 Female Caucasian 50 yrs. Anoxia

Donor 2 Female Caucasian 57 yrs. Anoxia
Donor 3 Female Caucasian 58 yrs. Aneurysm
Donor 4 Female Caucasian 20 yrs. Drug overdose
Donor 5 Female Caucasian 57 yrs. Intracranial hemorrhage (ICH)
Donor 6 Male Caucasian 19 yrs. Self-inflicted wound
Donor 7 Male African American 64 yrs. ICH/Stroke
Donor 8 Male Caucasian 59 yrs. ICH/Stroke
Donor 9 Male Caucasian 18 yrs. Anoxia/Motor Vehicle Accident
Donor 10 Male Caucasian 53 yrs. Anoxia

Ketoconazole, an sDILI drug which has received black-box statistically-significant difference between sDILI and non-sDILI for
warnings or prohibitions against clinical use as an oral drug due to the prevalence of each clinical phenotype was found (Chi-square;
the risk of liver injury leading to liver transplantation or death [13], p ¼ 0.285) (Fig. 1).
was used to derive normalized ROS/ATP ratio in order to minimize
inter-assay variance between testing and validation sets, allowing
3.1. Comparison of different quantitative metrics and mechanistic
data from the two independent experiments performed in this
study to be combined for analysis. The normalized ROS/ATP ratio
was calculated using the following equation.
1. Results with Drugs in the Testing Set: Primary human he-
Normalized ratio ¼ [AUC (drug)/AUC(ketoconazole)] x 100%;
patocytes were initially treated with 80 drugs to identify the most
where AUC is that from the plot of ROS/ATP ratio versus drug
appropriate endpoints for the distinction of sDILI and non-sDILI
drugs. Trapezoidal AUC for each of the doseeresponse curves was
Logarithmic values of normalized ROS/ATP ratios were used to
computed. Receiver operating characteristic (ROC) of the AUC
evaluate the performance of the endpoints chosen to identify
curves was employed to evaluate performance (Fig. 2). The sensi-
hepatotoxic drugs. Logistic regression with receiver operating
tivity and specificity values for the various endpoints were 64.52%
characteristic (ROC) curve (area under the ROC curve or AUROC)
and 89.8%, respectively, for ATP; 45.16% and 85.71%, respectively, for
was used to assess the accuracy (sensitivity and specificity) in
ROS; 51.61% and 87.76%, respectively, for GSH; 45.16% and 87.76%,
predicting sDILI [9]. Sensitivity was defined as percentage of posi-
respectively, for caspase activation; 83.87% and 91.84%, respec-
tive test substances found to be positive in the assay, and specificity
tively, for ROS/ATP ratio; 77.42% and 75.51%, respectively, for GSH/
was defined as the percentage of negative test substances found to
ATP ratio; and 64.52% and 83.67%, respectively, for caspase/ATP
be negative. 95% confidence limit (CI) was computed for AUROC and
ratio. ROS/ATP ratio therefore was found to yield the highest
compared among the different metrics. Statistical analysis was
sensitivity and specificity.
performed with the KruskaleWallis test to compare quantitative
2. Validation Set Results: The sensitivity and specificity of the
metrics among different clinical phenotypes and the Wilcoxon rank
ROS/ATP ratio as an endpoint to identify sDILI drugs was evaluated
sum test and Bonferroni adjustment for post-hoc pair-wise group
in an independent experiment of validation set with 34 sDILI and
comparisons. Statistical analyses were performed using the
38 non-sDILI drugs. Results with the validation experiment were
GraphPad Prism Software (version 6.00 for Windows; GraphPad
similar to that in the initial experiment with the Test Set (Table 5),
Software, La Jolla California USA, www.graphpad.com) and differ-
with ROS/ATP ratio yielding higher sensitivity and specificity than
ences considered statistically significant when the p-value(s) were
ATP as an endpoint alone.
less than 0.05. Due to the descriptive nature of this study, p-values
3. sDILI versus gDILI. ROC analysis of ROS/ATP AUC for sDILI
have not been adjusted for multiple comparisons.
drugs versus non-sDILI drugs was compared to that for gDILI versus
non-DILI using the combined data of all 152 drugs from both the
3. Results test and validation experiments (Fig. 3). AUROC analysis yielded
higher sensitivity and specificity values for sDILI vs non-sDILI than
DILI classification: 65 drugs were classified as sDILI drugs the values for g-DILI vs non-DILI. The results suggest that the ROS/
(Table 1). Of these sDILI drugs, 14 were withdrawn from the US or ATP endpoint was more effective in the identification of sDILI than
Europe (N ¼ 14) and 17 drugs received black-box warnings, with gDILI drugs.
the remaining 34 sDILI drugs known to be associated with ALF. 52 4. Comparison of Different Clinical Classifications of DILI
drugs were associated with DILI events but did not fulfill the sDILI Drugs. Log drug-induced normalized ROS/ATP AUC was plotted
criteria and were classified as non-sDILI drugs (Table 2). 35 were versus clinical phenotypes defined by DILI pathology: hepato-
not associated with DILI events and were classified as non-DILI cellular necrosis (H); cholestasis (C); immune-mediated DILI
drugs (Table 2). sDILI and non-sDILI drugs together were classi- (IM); mixed liver injury (M) or undefined (NM: DILI type not
fied as general DILI (gDILI) drugs. Subsequent data evaluations were mentioned in FDA label), clinical outcome (acute liver failure
performed based on these 4 DILI classifications (sDILI, non-sDILI, (ALF); vanishing bile duct syndrome (VBDS); sinusoidal
gDILI, non-DILI). Drugs associated with DILI events and reported obstruction syndrome (SOS); steatosis (STS); cholestatic hepatitis
to involve immune-mediated hepatotoxicity are shown in Table 3. (CHLH); serum enzyme elevation (SEE) or not applicable due to
Clinical phenotypes vs. DILI classification: The reported clin- lack of DILI (NA). Drugs associated with ALF (sDILI drugs) were
ical pathological findings (phenotypes) of drugs associated with found to be mainly associated with H, C/M, and IM DILI type,
DILI events are shown in Tables 1 and 2; those reported to involve while VBDS, SOS, STS, CHLH, SEE, and NA were associated with
immune-mediated mechanisms are shown in Table 3. No non-sDILI drugs. The highest mean ROS/ATP ratio was observed
J. Zhang et al. / Chemico-Biological Interactions 255 (2016) 3e11 7

Fig. 1. The distribution of clinical histopathological DILI findings of the sDILI and non-sDILI drugs. No statistically significant difference was observed between the two types of DILI

Fig. 2. Receiving operating characteristics (ROC) curves for the evaluation of the performance of the different biological endpoints used in human hepatocytes for the identification
of sDILI drugs. ROC curves for individual endpoints (left) as well as the ratio of the AUC (area under the doseeresponse curve) of each endpoint to that of cellular ATP contents. ROC
analysis results are shown in the accompanied table. ROS/ATP was found to be the most accurate endpoint for the identification of sDILI drugs (highest sensitivity and specificity

Table 5
Summary on the performance of ATP and ROS/ATP ratio in the test and validation studies. The results show that, in both studies, ROS/ATP ratio was superior to ATP in sensitivity
while providing similar specificity.


Test Validation Testing Validation

Cut-off value <129 <129 >0.62 >0.62

Sensitivity% 64.52 58.82 80.65 79.41
95% CI (Sensitivity) 45.37%e80.77% 40.70%e75.35% 62.53%e92.55% 62.10%e91.30%
Specificity% 89.8 89.47 91.84 89.47
95% CI (Specificity) 77.77%e96.60% 75.20%e97.06% 80.40%e97.73% 75.20%e97.06%
Likelihood ratio 6.323 5.588 9.879 7.544

Likelihood ratio ¼ Sensitivity/(1 e Specificity); Cl: confidence limit.

for sDILI drugs with H DILI type, followed by that with C/M DILI 5. ROS/ATP Ratio to Distinguish sDILI vs. non-sDILI Drugs by
type. Drugs with H and C/M DILI types had ROS/ATP ratios higher Therapeutic Categories. A total of 152 drugs were mapped to 14
than IM sDILI drugs and all non-sDILI drugs. Statistically signif- categories defined by the WHO's Anatomical Therapeutic Chemical
icant difference (p < 0.0001) was observed between sDILI and (ATC) classification system. Four major therapeutic categories,
non-DILI drugs (Fig. 4). namely, antidepressant, antifungal, antilipidemic, and antidiabetic
8 J. Zhang et al. / Chemico-Biological Interactions 255 (2016) 3e11

Fig. 3. AUROC analysis of the identification of sDILI versus non-sDILI (left) and gDILI versus nonDILI drugs. gDILI drugs are defined as all drugs with DILI properties, including both
sDILI and non-sDILI drugs. The results show that ROS/ATP was more effective (higher sensitivity and specificity values) in the identification of sDILI than gDILI drugs. ROC analysis
results are shown in the accompanied table.

drugs, have a higher number of sDILI drugs (Fig. 5a). ROS/ATP ratio sensitivity (Table 6). We further examined whether the ROS/ATP
in human hepatocytes clearly identified sDILI from non-sDILI drugs ratio of the 140 orally administered drugs in the 152 drugs eval-
in these four therapeutic categories (Fig. 5b). uated in this study would add value to the existing models (Models
6. ROS/ATP Ratio Compared to Other Prediction Models. The 2 and 4, Table 7) by using the non-parametric approach, as sug-
performance of the ROS/ATP ratio was compared with two previ- gested by others [9]. The addition of Models 1 and 3 enhanced the
ously reported DILI prediction models: The RO2 rule model with AUROC of ROS/ATP alone for both testing and validation drugs
daily dose (100 mg) and lipophilicity (log P  3) as risk factors (Table 7).
[5], and the model of Lammert et al. with suggested daily dose
(50 mg) and extensive drug metabolism as risk factors [18]. ROS/
4. Discussion
ATP ratio in human hepatocytes yielded higher specificity and
Accurate identification of drug candidates with sDILI properties
is an ongoing challenge in drug development. The current status
quo of identification of sDILI drugs by post-marketing survey is
unacceptable due to the serious consequences in patients with
these events. Our study was performed to test the “common
property hypothesis” of Li that drugs with idiosyncratic drug hep-
atotoxicity could be identified based on their biological activities
[19]. We have chosen human hepatocytes as the experimental
model due to the following key properties: 1. Human-specific
metabolic competence: human hepatocytes are well-established
to retain in vivo hepatic xenobiotic metabolic enzymes and co-
factors. 2. Human-specific and physiologically-relevant toxicolog-
ical targets: Hepatocytes are the major cell type injured during DILI
events. Hepatocellular injuries have been postulated to be the
initiating event which, upon the concurrence of multiple risk fac-
tors, would lead to a cascade of subsequent escalating toxicological
reactions in the patient, culminating in liver failure [19].
We report here the successful development of a novel in vitro
approach that accurately identifies drugs associated with severe
DILI events, namely, treatment of primary cultured human hepa-
tocytes with the drugs in question followed by quantification of the
Fig. 4. ROS/ATP ROC identification of sDILI drugs based on reported DILI type. Log elevation of ROS/ATP AUC ratio. The performance of the human
normalized ROS/ATP AUC values were plotted versus the multiple DILI types (H: he- hepatocyte ROS/ATP assay (85% sensitivity and 87% specificity)
patocellular necrosis; C: cholestasis; IM: immune-mediated DILI; M: Mixed liver represents an improvement to results from other models reported
injury; NM: DILI type not mentioned in FDA label). The drugs were also classified based
on clinical outcome (ALF: acute liver failure; VBDS: vanishing bile duct syndrome; SOS:
by others. For instance, an in silico approach based on ligand-based
sinusoidal obstruction syndrome; STS: steatosis, CHLH: cholestatic hepatitis, SEE: Bayesian modeling, with a training set of 295 compounds and a
serum enzyme elevation, NA: not applicable due to lack of DILI), and DILI classification validation set of 237 compounds, results in concordance of 60%,
(sDILI and non-sDILI) are also shown. Mean (horizontal line inside the rectangle), sensitivity of 56%, and specificity of 67% [10]. High content cellular
standard error of the mean (rectangle containing the mean), and maximum and
imaging of primary cultured human hepatocytes treated with 300
minimum values (upper and lower bars connected to the rectangle) of log ROS/ATP
AUC are shown. The number of drugs in each category is also shown at the bottom X- drugs, measuring mitochondrial damage, oxidative stress, and
axis. Pair-wide comparisons via Wilcoxon rank sum test and Bonferroni adjustment intracellular glutathione as endpoints, is reported to have a true
indicate that the differences of ROS/ATP ratios between three sDILI types (hepatocel- positive rate of 50e60%, and a false-positive rate of 0e5% [35].
lular (H), cholestasis and mixed injuries (C/M), and immune-mediated (IM)) and non- Evaluation of 45 drugs in micropatterned co-cultures (MPCC) of
DILI drugs (NM) are statistically significant (p value < 0.0001). However, those between
non-sDILI types (VBDS, SOS, steatosis (STS), cholestatic hepatitis (CHLH), and serum
human hepatocytes and mouse 3T3 fibroblasts is reported to have a
enzyme elevation (SEE)) and NM are not (p value ¼ 1). ****: p value < 0.0001, ns: not sensitivity of 65.7% and a false positive rate of 10% for the identi-
significant. fication of DILI drugs [17].
J. Zhang et al. / Chemico-Biological Interactions 255 (2016) 3e11 9

Fig. 5. Rank order of Log (normalized ROS/ATP AUC) of different clinical DILI types and adverse outcomes. (a): Number drugs in each therapeutic category. (b): Log (normalized ROS/
ATP AUC) values for drugs in the 4 major therapeutic categories (antidepressant, antifungal, antilipidemic, and antidiabetic). The Log (normalized ROS/ATP AUC) value of 0.61 (cut-
off value) allowed distinction of sDILI and non-sDILI drugs in each of the 4 major therapeutic categories. *: statistically significant (p < 0.05) to be different between sDILI and
nonDILI drugs.

Table 6
A comparison of the human hepatocyte ROS/ATP assay to two previously reported in silico prediction models in the identification of sDILI drugs. All three approaches were
performed using the same 152 sDILI, non-sDILI, and non-DILI drugs. The two in silico models are the RO2 rule with daily dose (100 mg) and lipophilicity (log P  3) as risk
factors [5] and the Lammert et al. model with suggested daily dose (50 mg) and extensive drug metabolism as risk factors [18]. The 152 drugs were classified as sDILI or non-
sDILI drugs based on clinical findings and compared to their classification by each of the three models. The human hepatocyte ROS/ATP ratio model is superior to the two
previous models in both sensitivity and specificity. sDILI: severe DILI drugs; non-sDILI: non-severe DILI drugs; PPV: positive predictive value calculated by true positives
divided by sum of all positives; NPV: negative predictive value calculated by true negatives divided by sum of all negatives.

Model Classification Classification by Model Odds ratio PPV% NPV% Sensitivity Specificity

sDILI Non-sDILI (95% confidence)

(n ¼ 64) (n ¼ 76)

RO2 sDILI 27 10 4.816 72.97% 64.08% 42.19% 86.84%

non-sDILI 37 66 (2.101e11.042)
Lammert et al. sDILI 47 33 3.603 58.75% 71.67% 73.44% 56.58%
non-sDILI 17 43 (1.760e7.376)
Human Hepatocytes ROS/ATP ratio sDILI 53 9 35.869 85.48% 85.90% 82.81% 88.16%
non-sDILI 11 67 (13.847e92.912)

We attribute our success to the following: 2. Human-specific, metabolic competent experimental system:
Primary human hepatocytes, the generally-accepted gold stan-
1. Accurate DILI annotation: The drugs evaluated in the study were dard for in vitro human drug metabolism and hepatotoxicity
classified as sDILI, non-sDILI, gDILI, and non-DILI drugs based on studies, were used in the study rather than hepatic cell lines,
a comprehensive review of literature. Emphasis was placed on which are known to exhibit attenuated drug metabolism ca-
drugs associated with hepatotoxicity reviewed by multiple pacity. The use of primary human hepatocytes would ensure
study groups/registries with comprehensive information on that the drugs evaluated would be subjected to human hepatic
hepatotoxicity (i.e., causality level, severity, reporting frequency, metabolism followed by the manifestation of toxicity via
and regional difference) based on post-marketing safety expe- perturbation of human cellular and biochemical pathways.
rience in different countries, and multiple databases for drug- Furthermore, hepatocytes pooled from multiple donors (5 male
induced acute liver failure. and 5 female donors) were used to avoid results that may be

Table 7
Enhanced performance in the identification of sDILI drugs via combination of in silico models and the human hepatocyte ROS/ATP model with the same 152 sDILI, non-sDILI,
and non-DILI drugs evaluated in this study. Model 1: The RO2 rule (with daily dose (100 mg) and lipophilicity (log P  3) as risk factors [5]; Model 2: Combination of Model 1
and the human hepatocyte ROS/ATP AUC ratio; Model 3: Lammert model (with suggested daily dose (50 mg) and extensive drug metabolism as risk factors [18]), Model 4:
Combination of Model 3 and the human hepatocyte ROS/ATP AUC ratio. AUROC of 1.0 represents perfect performance. Models 2 and 4 (combination of the in silico models and
the human hepatocyte model) yielded higher AUROC than each individual model alone.

Model Variables AUROC 95% Confidence interval Comparison P-value

Testing Validation Testing Validation

1 RO2 0.717 0.758 (0.608, 0.826) (0.650, 0.866)

2 RO2 þ ROS/ATP 0.912 0.912 (0.845, 0.978) (0.840, 0.983) Model 2 vs. 1 0.0001
3 Daily dose þ Metabolism 0.639 0.725 (0.532, 0.747) (0.610, 0.840)
4 Daily dose þ Metabolism þ ROS/ATP 0.906 0.914 (0.839, 0.973) (0.827, 0.980) Model 4 vs. 3 0.0001
10 J. Zhang et al. / Chemico-Biological Interactions 255 (2016) 3e11

biased to the properties of a specific individual and may not be lipophilicity (log P  3) as risk factors [5] and the Lammert et al.
universally applicable. model with suggested daily dose (50 mg) and extensive drug
3. Mechanistically-relevant toxicological endpoints: The end- metabolism as risk factors [18]. In this study, we found that the
points chosen for the study, reactive oxygen species (ROS) for- accuracy of sDILI prediction was further enhanced by combining
mation, ATP depletion, caspase activation, and glutathione ROS/ATP ratio findings in human hepatocytes with each of the two
depletion, allow the quantification of oxidative stress, hepato- in silico models (Table 7). This finding underscores the merits of
cellular damage, apoptosis, and reactive metabolite formation, combining both clinically- and mechanistically-relevant common
respectively. These events are generally believed to be key DILI drug properties to identify DILI drugs.
toxicological mechanisms associated with DILI events. The most significant contribution of our findings to the overall
4. Objective, quantitative data analysis: The identities of the drugs goal of accurate identification of sDILI drugs is that these hepato-
evaluated were blinded to the technical team performing the toxic drugs share a common inherent biological property, namely,
in vitro study and to the computational team carrying out the ROS/ATP ratio, which can be experimentally evaluated in human
quantification and analysis of the doseeresponse curves in or- hepatocytes. It is interesting to note that our evaluation was per-
der to remove the possibility of objective bias in data collection formed with hepatocytes pooled from multiple donors: 5 male and
and analysis. 5 female. The ROS/ATP ratio therefore represents the response of a
5. Trapezoidal AUC approach: The data analysis approach of our “generalized” human to sDILI drugs. Our results therefore suggest
study is based on a recent proposal that drug safety prediction that the sDILI drugs would illicit similar response in human pa-
and evaluation needs to evolve from qualitative to quantitative tients in general. Specific individuals may develop sDILI events due
approaches, with a point estimate of risk bounded by confidence to their specific genetic and/or environmental conditions at the
intervals, especially the capacity to determine both sensitivity time of drug exposure as described in Li's “multiple determinant
and specificity [1]. In this study, we have evaluated several hypothesis” of idiosyncratic drug toxicity [19]. We believe that the
quantitative matrices for their utility in predicting severe DILI. human hepatocyte ROS/ATP ratio can be used as a biomarker in
Similar to what was previously reported [15], we found that conjunction with routine preclinical and clinical safety evaluations
trapezoidal AUC, which provides systematic and cumulative to identify sDILI drugs. In our laboratories, we will continue to build
information across the entire dose range, represents a superior a database with additional marketed drugs with clear definition of
approach to quantify the doseeresponse relationship than the clinical hepatotoxicity, with an ultimate goal of accurately identi-
commonly used EC50 and Emax approaches. Moreover, trape- fying sDILI, gDILI, and non-DILI drugs. Our research findings are
zoidal AUC is a model-free matrix that does not require a perfect complementary to ongoing efforts to evaluate DILI drug properties
doseeresponse curve, which is a challenge for most drugs with [4,24].
relatively low toxicological effects. Our findings may aid mechanistic elucidation of the manifes-
tation of sDILI. The elevated ROS/ATP ratio in sDILI drugs suggests
A valid and often-raised concern with the evaluation of DILI that oxidative stress, exacerbated by cellular damage in hepato-
using in vitro systems is that the clinical manifestation of DILI are cytes, may be a key initiating event culminating in liver failure in
known to involve complex in vivo interactions of cell types from susceptible human populations. A practical application of the hu-
hepatic and nonhepatic organs, which is difficult to adequately man hepatocyte ROS/ATP assay is screening of new drug entities to
model using in vitro systems. This concern is supported by our minimize sDILI liability in drug development. We are also hopeful
observation that drugs associated with hepatotoxicity involving that our discovery of ROS/ATP ratio as an endpoint for the identi-
non-hepatocytes, such as vanishing bile duct syndrome, sinusoidal fication of sDILI drugs may allow for the discovery of risk factors
obstruction syndrome, cholestatic hepatitis, and immune- and the identification of at-risk human populations for sDILI.
mediated DILI events, could not be readily identified in the hu-
man hepatocyte assay. Our data also suggest that the ROS/ATP ratio Disclaimer
is not appropriate for identifying hepatotoxic drugs with steatosis
and cholestasis and immune-related DILI as the major mode of The views presented in this article do not necessarily reflect
action. Our success in the accurate identification of sDILI drugs is current or future opinion or policy of the US Food and Drug
consistent with the finding that direct induction of hepatocellular Administration. Any mention of commercial products is for clari-
injuries is the predominant toxicological effect of such drugs, and it fication and not intended as endorsement.
could thereby be readily identified in an experimental system with
human hepatocytes [32]. In our laboratories, efforts are underway
to evaluate additional hepatocyte-based mechanistic endpoints, Acknowledgment
such as BSEP inhibition for cholestasis, as well as the application of
co-culture systems such as the IdMOC™ [22], to evaluate hepato- Jürgen Borlak is supported by the German Federal Ministry for
toxicity involving interaction between hepatocytes and non- Education and Research as part of the Virtual Liver Network
hepatocytes. Initiative (grant 031 6154). He is also a recipient of an ORISE stipend
Our study demonstrated that the quantitative measurements of from the US Food and Drug Administration. We appreciated Ke Yu
mechanistically-relevant endpoints successfully identified sDILI and Yuri An for collecting data from LiverTox, Minjun Chen and
drugs within our collection of 152 drugs, representing a wide range Qiang Shi for providing information on FDA Drug Labeling, Wei-
of molecular structures and therapeutic indications. The findings gong Ge and Zhichao Liu for making the figures, and Mikyung Lee
support the previous hypothesis of Li [19] that DILI drugs, including for analysis on Toxicogenomics Project data. We also thank Hong
those causing idiosyncratic events and thereby not readily identi- Fang and Reagan Kelly for comments and suggestions, and Yumiko
fied in clinical trials, could be identified by their common drug LaForge for technical support on experiments.
properties using a physiologically-relevant experimental system
such as primary cultured human hepatocytes. Transparency document
The human hepatocyte ROS/ATP AUC ratio as a predictive tool
appears to be superior to the two previously reported in silico Transparency document related to this article can be found at
models, namely, the RO2 rule with daily dose (100 mg) and http://dx.doi.org/10.1016/j.cbi.2015.11.008.
J. Zhang et al. / Chemico-Biological Interactions 255 (2016) 3e11 11

References hepatotoxicity and the “multiple determinant hypothesis” for the manifes-
tation of idiosyncratic drug toxicity, Chem. Biol. Interact. 142 (2002) 7e23.
[20] A.P. Li, Human-based in vitro experimental systems for the evaluation of
[1] D.R. Abernethy, J. Woodcock, L.J. Lesko, Pharmacological mechanism-based
human drug safety, Curr. Drug Saf. 2 (2007) 193e199.
drug safety assessment and prediction, Clin. Pharmacol. Ther. 89 (2011)
[21] A.P. Li, Human hepatocytes as an effective alternative experimental system for
the evaluation of human drug properties: general concepts and assay pro-
[2] D.E. Amacher, The primary role of hepatic metabolism in idiosyncratic drug-
cedures, ALTEX 25 (2008) 33e42.
induced liver injury, Expert Opin. Drug Metab. Toxicol. 8 (2012) 335e347.
[22] A.P. Li, Evaluation of adverse drug properties with cryopreserved human
[3] L.N. Bell, N. Chalasani, Epidemiology of idiosyncratic drug-induced liver injury,
hepatocytes and the integrated discrete multiple organ co-culture
Semin. Liver Dis. 29 (2009) 337e347.
(IdMOC(TM)) system, Toxicol. Res. 31 (2015) 137e149.
[4] S. Bhattacharya, et al., Modeling drug- and chemical-induced hepatotoxicity
[23] A.S. Rangnekar, R.J. Fontana, An update on drug induced liver injury, Minerva
with systems biology approaches, Front. Physiol. 3 (2012) 462.
Gastroenterol. Dietol. 57 (2011) 213e229.
[5] M. Chen, J. Borlak, W. Tong, High lipophilicity and high daily dose of oral
[24] M. Robles-Diaz, et al., Use of Hy's law and a new composite algorithm to
medications are associated with significant risk for drug-induced liver injury,
predict acute liver failure in patients with drug-induced liver injury, Gastro-
Hepatology 58 (2013) 388e396.
enterology 147 (2014) 109e118 e105.
[6] M. Chen, et al., The liver toxicity knowledge base: a systems approach to a
[25] S. Russmann, A. Jetter, G.A. Kullak-Ublick, Pharmacogenetics of drug-induced
complex end point, Clin. Pharmacol. Ther. 93 (2013) 409e412.
liver injury, Hepatology 52 (2010) 748e761.
[7] A.K. Daly, C.P. Day, Genetic association studies in drug-induced liver injury,
[26] A. Suzuki, et al., Drugs associated with hepatotoxicity and their reporting
Drug Metab. Rev. 44 (2012) 116e126.
frequency of liver adverse events in VigiBase: unified list based on interna-
[8] A.K. Daly, et al., HLA-B*5701 genotype is a major determinant of drug-induced
tional collaborative work, Drug Saf. 33 (2010) 503e522.
liver injury due to flucloxacillin, Nat. Genet. 41 (2009) 816e819.
[27] A. Suzuki, et al., Comedications alter drug-induced liver injury reporting fre-
[9] E.R. DeLong, D.M. DeLong, D.L. Clarke-Pearson, Comparing the areas under
quency: Data mining in the WHO VigiBase, Regul. Toxicol. Pharmacol. 72
two or more correlated receiver operating characteristic curves: a nonpara-
(2015) 481e490.
metric approach, Biometrics 44 (1988) 837e845.
[28] G. Tarantino, M.N. Di Minno, D. Capone, Drug-induced liver injury: is it
[10] S. Ekins, A.J. Williams, J.J. Xu, A predictive ligand-based Bayesian model for
somehow foreseeable? World J. Gastroenterol. 15 (2009) 2817e2833.
human drug-induced liver injury, Drug Metab. Dispos. 38 (2010) 2302e2308.
[29] P. Thulin, et al., Keratin-18 and microRNA-122 complement alanine amino-
[11] R.J. Fontana, Pathogenesis of idiosyncratic drug-induced liver injury and
transferase as novel safety biomarkers for drug-induced liver injury in two
clinical perspectives, Gastroenterology 146 (2014) 914e928.
human cohorts, Liver Int. 34 (2014) 367e378.
[12] M.J. Gomez-Lechon, et al., Human hepatocytes as a tool for studying toxicity
[30] J. Uetrecht, Immunoallergic drug-induced liver injury in humans, Semin. Liver
and drug metabolism, Curr. Drug Metab. 4 (2003) 292e312.
Dis. 29 (2009) 383e392.
[13] H.K. Greenblatt, D.J. Greenblatt, Liver injury associated with ketoconazole:
[31] T.J. Urban, D.B. Goldstein, P.B. Watkins, Genetic basis of susceptibility to drug-
review of the published evidence, J. Clin. Pharmacol. 54 (2014) 1321e1329.
induced liver injury: what have we learned and where do we go from here?
[14] J.H. Hoofnagle, et al., LiverTox: a website on drug-induced liver injury, Hep-
Pharmacogenomics 13 (2012) 735e738.
atology 57 (2013) 873e874.
[32] P.B. Watkins, Idiosyncratic liver injury: challenges and approaches, Toxicol.
[15] S. Huang, L. Pang, Comparing statistical methods for quantifying drug sensi-
Pathol. 33 (2005) 1e5.
tivity based on in vitro dose-response assays, Assay. Drug Dev. Technol. 10
[33] P.B. Watkins, Biomarkers for the diagnosis and management of drug-induced
(2012) 88e96.
liver injury, Semin. Liver Dis. 29 (2009) 393e399.
[16] N. Kaplowitz, Drug-induced liver injury, Clin. Infect Dis. 38 (Suppl 2) (2004)
[34] P.B. Watkins, Drug safety sciences and the bottleneck in drug development,
Clin. Pharmacol. Ther. 89 (2011) 788e790.
[17] S.R. Khetani, et al., Use of micropatterned cocultures to detect compounds that
[35] J.J. Xu, et al., Cellular imaging predictions of clinical drug-induced liver injury,
cause drug-induced liver injury in humans, Toxicol. Sci. 132 (2013) 107e117.
Toxicol. Sci. 105 (2008) 97e105.
[18] C. Lammert, et al., Oral medications with significant hepatic metabolism at
[36] H.J. Zimmerman, Hepatotoxicity: the Adverse Effects of Drugs and Other
higher risk for hepatic adverse events, Hepatology 51 (2010) 615e620.
Chemicals on the Liver, Lippincott Williams & Wilkins, Philadelphia, 1999.
[19] A.P. Li, A review of the common properties of drugs with idiosyncratic