Pattern Recognition 67 (2017) 97–109

Contents lists available at ScienceDirect

Pattern Recognition
journal homepage: www.elsevier.com/locate/patcog

Hierarchical learning of multi-task sparse metrics for large-scale

image classification
Yu Zheng a, Jianping Fan b, Ji Zhang c, Xinbo Gao a,∗
a
Department of Electronic Engineering, Xidian University, Xi’an 710071, PR China
b
Department of Computer Science, University of North Carolina, Charlotte, NC 28223, USA
c
Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an 710049, PR China

a r t i c l e i n f o a b s t r a c t

Article history: In this paper, a novel approach is developed to learn a tree of multi-task sparse metrics hierarchically
Received 20 July 2016 over a visual tree to achieve a fast solution to large-scale image classification, where an enhanced visual
Revised 18 December 2016
tree is first learned to organize large numbers of image categories hierarchically in a coarse-to-fine fash-
Accepted 24 January 2017
ion. Over the visual tree, a tree of multi-task sparse metrics is learned hierarchically by: (a) performing
Available online 3 February 2017
multi-task sparse metric learning over the sibling child nodes under the same parent node to explicitly
Keywords: separate their commonly-shared metric from their node-specific metrics; and (b) propagating the node-
Hierarchical multi-task sparse metric specific metric for the parent node to its sibling child nodes (at the next level of the visual tree), so that
learning more discriminative metrics can be learned for controlling inter-level error propagation effectively. We
Visual tree have evaluated our hierarchical multi-task sparse metric learning algorithm over three different image
Large-scale image classification sets and the experimental results demonstrated that our hierarchical multi-task sparse metric learning
algorithm can obtain better performance than the state-of-the-art algorithms on large-scale image classi-
fication.
© 2017 Elsevier Ltd. All rights reserved.

1. Introduction any sense to ignore their strong inter-category visual correlations

Large-scale image classification (i.e., classifying millions of im-
ages into thousands or even tens of thousands of categories) is a
fundamental research issue for the communities of computer vi-
sion, machine learning and multimedia computing, and it has re-
ceived extensive attentions recently [1–4]. In spite of recent sig-
nificant progresses on recognizing hundreds of image categories,
large-scale image classification is still a challenging task because:
(1) distinguishing large numbers of image categories (i.e., thou-
sands or even tens of thousands of image categories) is inher-
ently more complex than distinguishing among just a few, thus the
accuracy rates for the state-of-the-arts methods are usually very
low; (2) the computational cost at test time is a critical issue (i.e.,
the computational cost for a flat approach grows linearly with the
number of image categories, and it is unacceptable for large-scale
image classification application); (3) some image categories may
have strong inter-category visual correlations, and it does not make
∗
Corresponding author.
E-mail addresses: yuzheng.xidian@gmail.com (Y. Zheng), jfan@uncc.edu
(J. Fan), zhang_ji@stu.xjtu.edu.cn (J. Zhang), xbgao@mail.xidian.edu.cn,
xbgao.xidian@gmail.com (X. Gao).
completely and learn their inter-related classifiers independently.
One promising way to address these issues is to organize large
numbers of image categories hierarchically through a tree struc-
ture by exploiting their inter-category correlations. Some previous
works [2,3,5] have leveraged the semantic ontologies (taxonomies)
to organize large number of image categories hierarchically ac-
cording to their inter-category semantic relationships. On the
other hand, it is very attractive to learn a visual hierarchy directly
from large amounts of training images, and some researches
[6–8] have recently been done on learning label trees to organize
large numbers of image categories hierarchically according to
their inter-category visual correlations. To learn a label tree, a
confusion matrix for N image categories is first obtained from
N one versus rest (OVR) binary classifiers. Such an approach to
label tree learning may seriously suffer from two problems: (a)
huge computational cost: it could be very expensive to learn N
OVR binary classifiers independently, especially when N is very
large (N is typically very large for large-scale image classification);
(b) huge sample imbalance: for a given image category (positive
class), the negative samples from other N − 1 image categories
(negative classes) heavily outnumber its positive samples, thus
huge numbers of negative samples may easily mislead the process

http://dx.doi.org/10.1016/j.patcog.2017.01.029
0031-3203/© 2017 Elsevier Ltd. All rights reserved.
98 Y. Zheng et al. / Pattern Recognition 67 (2017) 97–109
Fig. 1. The flowchart of our hierarchical multi-task sparse metric learning algorithm.
for training the OVR binary classifiers and result in incorrect
OVR binary classifiers (with very low accuracy rates), which
may further produce an incorrect confusion matrix for label tree
construction. Therefore, other researchers have proposed visual
trees learning [9,10] to organize large number of image categories
hierarchically. Unlike label tree learning requires a pre-calculated
confusion matrix, visual tree learning can be achieved by using
the inter-category visual similarities directly, which may be able to
deal with the issues of huge computational cost and huge sample
imbalance effectively. However, one open problem of visual tree
learning is how to provide more effective representations of image
categories (which should be able to cover huge intra-category
visual diversity sufficiently) and characterize the inter-category vi-
sual similarities accurately. Based on these observations, it is very
attractive to develop new algorithms that are able to characterize
the inter-category visual similarities accurately while reducing the
computational cost dramatically.
It is also worth noting that the performance of all the algo-
rithms for image classification crucially depends on the discrim-
ination ability of the underlying metrics for similarity characteri-
zation. For examples, the distance metric plays an important role
in the performance of kNN (k-nearest neighbor) classifiers, and
the performance of the kernel-based algorithms (such as SVMs)
can be improved significantly by adopting a proper metric to
achieve more accurate similarity characterization [11–16]. Recently,
many researches focus on learning large-margin metrics to achieve
more accurate similarity characterization [17–20]. Even such large-
margin metric learning algorithms have provided a good solution
for enhancing the discrimination power of the metrics, they are
not scalable for large-scale image classification application due to
their huge computational complexity.
Based on these observations, a hierarchical multi-task sparse
metric learning algorithm is developed to learn a tree of multi-task
sparse metrics over an enhanced visual tree to achieve more accu-
rate similarity characterization and support more effective solution
to large-scale image classification. As shown in Fig. 1, our hierar-
chical multi-task sparse metric learning algorithm contains three
key components: (a) An enhanced visual tree is first learned to or-
ganize large numbers of image categories hierarchically, and such
enhanced visual tree can provide a good environment to automat-
ically identify the inter-related tasks for multi-task sparse metric
learning, e.g., the tasks for learning the large-margin metrics for
the sibling child nodes under the same parent node are strongly
inter-related; (b) For the root node at the first level of the visual
tree, a multi-task sparse metric learning algorithm is developed
to learn a multi-task sparse metric for separating its sibling child
nodes more accurately; (c) For the non-root node of the visual tree,
both the inter-node visual correlations (among the sibling non-root
nodes under the same parent node) and the inter-level visual cor-
relations (between the parent node and its sibling child nodes at
the next level of the visual tree) are leveraged to learn a tree of
multi-task sparse metrics, so that more discriminative metrics can
be learned for controlling inter-level error propagation effectively.
The rest of the paper is organized as follows. A brief review of
related work is presented in Section 2. In Section 3, we introduce
our work on constructing the enhanced visual tree. In Section 4,
we present our work on hierarchical learning of a tree of multi-
task sparse metrics over the visual tree. Section 5 demonstrates
our experimental results, followed by conclusions.
2. Related work
Recently, metric learning [15–17,19,21–23] has received exten-
sive attentions in communities of computer vision, machine learn-
ing and multimedia computing, especially Mahalanobis metric
learning. Weinberger et al. have developed a metric learning ap-
proach to support nearest neighbor classification (LMNN) [19],
which aims to learn a Mahanalobis distance by forcing the samples
from the same class to be closer while the samples from differ-
ent classes are far away. Parameswaran et al. have extended such
LMNN approach to a multi-task setting and a multi-task metric
learning approach [24] is proposed. The multi-task metric learning
approach aims to learn one common metric shared among multi-
ple inter-related tasks and multiple task-specific metrics simulta-
neously. By separating the commonly-shared metric from the task-
specific metrics explicitly, the multi-task metric learning approach
can obtain more discriminative metrics and achieve higher classi-
fication accuracy rates.
 Y. Zheng et al. / Pattern Recognition 67 (2017) 97–109
The problem of hierarchical classification [2,8,9,25,26] has been
investigated extensively in the past decades. The hierarchical clas-
sification approaches can significantly reduce the computational
cost at test time by organizing large numbers of image categories
hierarchically in a coarse-to-fine fashion[10]. Some researchers
have integrated the semantic ontology to support hierarchical clas-
sifier training [2,27,28]. Because the feature space is the common
space for classifier training and image classification, it is more
attractive to learn a visual hierarchy to organize large numbers
of image categories directly in the feature space. The visual tree
[9,10] is developed to organize large number of image categories by
characterizing the inter-category visual similarities directly, where
the mean features are employed for category representations and
the inter-category visual similarities are simply characterized by
the distances between their mean features. Because of huge intra-
category visual diversity, such mean features may not have suf-
ficient capacity for category representations (i.e., such mean fea-
tures may not be able to cover huge intra-category visual diversity
sufficiently). In addition, the hierarchical approaches may seriously
suffer from the problem of inter-level error propagation (i.e., mis-
classification at a high-level node will pass to its child nodes and
propagate along the tree structure until the leaf nodes).
To control the inter-level error propagation, some researchers
have used a hierarchy to regularize the child nodes towards their
parent node [10,29,30]. Xiao et al. have developed an interesting
approach by adding an orthogonal regularization term into the ob-
jective function for hierarchical classifier training [30], so that the
classifier learned for the current node could be orthogonal with
the classifier for its parent node. It is worth noting that orthog-
onality cannot guarantee to prevent inter-level error propagation
effectively because the orthogonal classifiers cannot ensure that:
(a) the classifier for the parent node may not make mistakes; and
(b) the mistakes for the parent node may not be propagated to its
child nodes.
Rather than enforcing orthogonality between the classifiers
for the parent node and its child nodes, some interesting ap-
proaches have been developed to learn a tree of disjoint metrics
by adding a trace-norm regularization term into the objective
function [21,31,32], so that the classifiers at different levels can
select disjoint metrics or feature subsets. By focusing on the
inter-level (parent-child) correlations, such algorithms can learn
disjoint and sparse metrics for different nodes at different levels
of a pre-defined tree structure, but the inter-node correlations
among the sibling child nodes under the same parent node are
completely ignored. It is worth noting that the discrimination
power of the metrics may largely depend on their ability on dis-
criminating the sibling child nodes under the same parent node. A
good solution to hierarchical classifier training has to accomplish
two inter-related requirements simultaneously: (a) making better
inter-level (parent-child) decision along the tree structure, so that
the inter-level error propagation can be controlled effectively; and
(b) achieving more accurate separation of the sibling child nodes
under the same parent node, so that the corresponding node
classifiers (especially the classifiers for high-level nodes) can make
less misclassification decisions.
3. Visual tree learning
Because the feature space is the common space for classifier
training and image classification, it is more attractive for us to
learn a hierarchical tree structure directly in the feature space.
In this section, we are interested in learning an enhanced visual
tree. Given M image categories, we learn a visual tree VT = (V, E ),
which comprises a set of tree nodes V and a set of edges E. Our
algorithm for visual tree learning contains two key steps: (a) Ac-
tive Sampling for Category Representation: One open problem of vi-
99
sual tree learning is how to extract an effective representation for
each image category and how to characterize the inter-category vi-
sual similarities more accurately. For two given image categories,
one straightforward way for characterizing their inter-category vi-
sual similarity is to compute the pairwise distances among all their
relevant training samples. But the computational cost for such pair-
wise sample-oriented method is very heavy because the numbers
of training samples could be very large. Some researchers employ
mean features for category representations and the inter-category
visual similarities are simply calculated by using the distances be-
tween their mean features. Because of huge intra-category visual
diversity, such mean features may not be able to cover the intra-
category visual diversity sufficiently and thus they are ineffective
for category representation. Based on this understanding, we uti-
lize active sampling to find multiple representative samples for
each image category, and using multiple representative samples for
category representation can allow us to cover the intra-category
visual diversity at a certain sufficiency level while significantly re-
ducing the computational cost for calculating the inter-category
visual similarities. It’s worth noting that yen et al. proposed an
advanced scalable exemplar clustering [33] method can also se-
lect representative subset for each image category. (b) Hierarchical
Affinity Propagation Clustering for Node Partitioning: A top-down ap-
proach to hierarchical affinity propagation (AP) clustering is devel-
oped for visual tree learning. Such hierarchical AP clustering pro-
cess for visual tree construction starts from the root node (that
contains all the image categories) and ends at the leaf nodes (each
leaf node contains only one single image category).
3.1. Active sampling for category representation
Inspired by a recent work [34], we utilize active sampling to
select multiple informative samples for category representation.
In order to select the most informative samples for covering the
intra-category visual diversity at certain sufficiency level, active
sampling considers two criteria: (a) representativeness: the samples
at high density regions should be selected, (b) diversity: the sam-
ples with high diversity should be selected. For a sample xi ∈ X, its
representativeness is defined as:
1   2
R ( xi ) = exp(−xi − x j  /2σR2 ) (1)
|Ni | j∈Ni
2
where R(xi ) is the representativeness of xi , Ni is the set of neigh-
bors of xi , Gaussian distance is employed to measure the distance
between xi and rest samples, then select the 2 ∗ n_s/n − 1 sam-
ples with the smallest distance as neighbor set, ns is number of
training samples, and n is number of active samples [34]. σ R is the
bandwidth of Gaussian kernel.
The diversity is defined as:
 2
D(xi ) = min[− exp(−xi − x j  /2σR2 )] (2)
x j ∈S 2
where D(xi ) is the diversity of xi and S is the set of selected infor-
mative samples.
Active sampling considers both representativeness and diversity,
thus the combined objective function define as:
arg max(λR(xi ) + (1 − λ )D(xi )) (3)
whereλ ∈ (0, 1) is the tradeoff between representativeness and di-
versity. Considering the balance, we choice λ = 0.5 in this paper.
Active sampling allows us to select an informative subset from
the sample set, so that the computational cost for calculating the
inter-category visual similarities can be reduced dramatically while
covering the intra-category visual diversity at a certain sufficiency
level.
100 Y. Zheng et al. / Pattern Recognition 67 (2017) 97–109
Fig. 2. Results of two different hierarchical AP clustering approaches on synthetic data. Left subfigures show the clustering results of high-level nodes; right subfigures show
the clustering results of low-level nodes.
3.2. Hierarchical affinity propagation clustering for node partitioning
After the most representative samples are selected for each im-
age category, hierarchical clustering is employed to build the en-
hanced visual tree. Due to the number of cluster centers for affinity
propagation (AP) clustering [35] does not require to be pre-defined,
the AP clustering allows us to obtain a better structure of visual
tree, and a hierarchical AP clustering algorithm is used to construct
the enhanced visual tree. Givoni et al. have proposed a bottom-up
approach to support hierarchical AP clustering [36], but it may lead
to higher accuracy rate for the bottom level but lower accuracy
rate for the top level. To control the inter-level error propagation,
the metrics for high-level nodes are required to be more discrimi-
native than the metrics for low-level nodes, so that the inter-level
error propagation can be controlled effectively. Based on this un-
derstanding, a top-down approach to hierarchical AP clustering is
developed to learn an enhanced visual tree. Fig. 2 shows a clus-
tering result on a group of synthetic data by using two different
hierarchical AP clustering methods. One can observe that our top-
down hierarchical AP algorithm achieves better clustering accuracy
rates for the high-level nodes.
In our top-down hierarchical AP algorithm, the Hausdorff dis-
tance [37] is used to characterize the inter-category visual simi-
larities. Let A = {a1 , . . . , am } and B = {b1 , . . . , bn } denote two point
sets. Inter-category visual similarities should be characterized
through measures how far two point sets are from each other. We
employ Hausdorff distance to characterize the inter-category visual
similarities, since the Hausdorff distance can utilize the max-min
characteristic of the set, calculates the maximum value in the near-
est neighbor set to measure the maximum mismatch between two
point sets. The Hausdorff distance of point sets A and B is defined
as:
dH (A, B ) = max(dh (A, B ), dh (B, A )) (4)
where dh (A, B) donates the one-sided Hausdorff distance, defined as:
dh (A, B ) = sup inf x − y (5)
x∈A y∈B
Hereby  ·  is Euclidean norm. By definition, the Hausdorff dis-
tance between two point sets is the greatest of all the distances of
one set to the closest point from the other set.
 Y. Zheng et al. / Pattern Recognition 67 (2017) 97–109
Based on this understanding, we use the Hausdorff distance to
calculate the distance between the most representative samples for
two image categories, so that we can obtain more accurate simi-
larity matrix for visual tree construction (via hierarchical AP clus-
tering over the similarity matrix). Such hierarchical AP clustering
process for visual tree construction starts from the root node (that
contains all the image categories) and ends at the leaf nodes (each
leaf node contains only one single image category).
For the root node R, all the image categories L(R ) ⊆ {1, . . . , L}
are partitioned into B subsets by using AP clustering. Then, AP
clustering are employed to partition these B subsets simultane-
ously. Note that the number of cluster center B is learned from
training samples automatically. During the learning process, we
calculate within-class distance dw and between-class distance db
under the same parent node. It is assumed that one super-category
C has n samples and has m sibling super-categories which share a
same parent node, the within-class distance dw and the between-
class distance db are defined as:

n
dw = tr ( (xi − μyi )(xi − μyi )T )
i=1

m task sparse metric, the distance function for the tth sibling child
db = tr ( n k ( μk − μ ) ( μk − μ ) )
T
k=1
where xi indicates one sample in one super-category C and μyi in-
dicates the mean sample of this super-category. μk indicates one
mean sample of one super-category and μ indicates mean of all
mean samples of sibling super-categories under the same parent
node. nk is the number of samples in the kth super-categories. dw
indicates the degree of aggregation within the most representa-
tive samples for the super-category C, db indicates the degree of
separation between the most representative samples for the super-
category C and its sibling super-categories under the same parent
node. After the dw and db are obtained, we calculate their ratio
rd = ddw . Note that if rd is greater than a presetting threshold θ ,
b +
then only |L(C)| leaf nodes are generated directly. This hierarchical
AP clustering process for node partition is applied recursively until
a complete tree is created, each leaf node contains one single im-
age category. For a given leaf node, its depth is defined as the path
length from itself to the root node, e.g., Fig. 3 shows the enhanced
visual tree for CIFAR-100 image set, we can observe that the depth
of the enhanced visual tree is 4 (from the root node to the leaf
nodes). Such visual hierarchy is an imbalanced tree, and different
high-level nodes are related with different numbers of fine-grained
image categories.
4. Hierarchical multi-task sparse metric learning
In this section, we introduce our hierarchical multi-task sparse
metric learning algorithm that can learn a tree of multi-task sparse
metrics hierarchically over the visual tree, where both the inter-
node visual correlations (among the sibling child nodes under the
same parent node) and the inter-level visual correlations (between
the parent node and its sibling child nodes at the next level of the
visual tree) are leveraged to learn more discriminative metrics to
enhance the separability.
4.1. Learning multi-task sparse metric for root node
After the visual tree is available, a top-down approach is used
to learn a tree of multi-task sparse metrics hierarchically over the
visual tree. For the root node at the first level of the visual tree, a
multi-task sparse metric learning algorithm is developed to learn
a multi-task sparse metric to achieve more accurate separation
of its sibling child nodes at the second level of the visual tree.
101
One open issue for multi-task learning [38,39] is how to automat-
ically identify the inter-related tasks. It is worth noting that our
visual tree has provided a good environment to automatically iden-
tify the inter-related tasks for multi-task metric learning, e.g., the
tasks for learning the metrics for the sibling child nodes under the
same parent node are strongly inter-related and such inter-related
metrics should be learned jointly to enhance their discrimination
power.
For the root node at the first level of the visual tree, its Maha-
lanobis distance is defined as:

dW (xi , x j ) = (xi − x j )T W (xi − x j ) (8)
where xi , xj are two data points, W is the Mahalanobis metric ma-
trix. If the weight matrix W reduces to the identity matrix I, the
distance dW (., .) is identical to the Euclidean distance.
Our goal for learning a multi-task sparse metric for the root
node is to enhance the separability of its sibling child nodes ef-
fectively [24]. Thus the metric for each sibling child node under
the same root node (each task) consists of two parts: the common
(6) metric shared among multiple sibling child nodes under the root
node and the node-specific individual metric. By using such multi-
(7) node is defined as:

dt (xi , x j ) = (xi − x j )T (W0 + Wt )(xi − x j ) (9)
where W0 is the common metric W0 0, which is shared among
all the sibling child nodes under the root node and Wt is the node-
specific individual metric Wt 0 for the tth child node. Both W0 and
Wt are symmetric positive semidefinite matrices. The joint objec-
tive function for multi-task sparse metric learning is then defined
as:

B
min γoW0 − I2F + αt tr[W0 + Wt ]
W0 ,...,WB
 t=1  (10)
B  2 
γt Wt F + dt (xi , x j ) + ξi, j,k
2
t=1 i, j i, j,k
subject to:
dt2 (xi , xk ) − dt2 (xi , x j ) ≥ 1 − ξi jk
ξi jk ≥ 0
W0 , W1 , . . . , WB  0.
where the trade-off parameters γ 0 and γ t are used to adjust the
relative importance between W0 and Wt , t = 1, . . . , B, xi is an input
exemplar, xj is the target neighbor, xk is the impostor, and ξ ijk is
the slack variable. In the extreme cases, if the γ 0 → ∞, one single
metric W0 is learned for all the sibling child nodes under the root
node, on the other hand, if γ t → ∞, B independent metrics are
learned. By adding the trace-norm into multi-task sparse metric
learning, the multi-task sparse metric (that is learned for the root
node) can have higher discrimination power on distinguishing its
sibling child nodes accurately.
By embedding the inter-node visual similarities (i.e., inter-task
relationships) into a structure regularization term, our multi-task
sparse metric learning algorithm can learn more discriminative
metrics for separating the sibling child nodes under the same par-
ent node. By focusing on the sibling child nodes under the same
parent node, our algorithm can effectively control the complexity
for multi-task sparse metric learning. Our multi-task sparse metric
learning algorithm is able to establish two separable metrics for
characterizing: (1) the common visual similarity shared among the
sibling child nodes under the same parent node; and (2) the node-
specific visual difference for each sibling child node. By separat-
ing the common metric from the node-specific metrics explicitly,
our multi-task sparse metric learning algorithm can have higher
102 Y. Zheng et al. / Pattern Recognition 67 (2017) 97–109

Fig. 3. The enhanced visual tree for CIFAR-100 image set with 100 categories. The depth of the enhanced visual tree is 4 (from the root node to the leaf nodes). The visual
hierarchy is an imbalanced tree, and different high-level nodes are related with different numbers of fine-grained image categories.

discrimination power on distinguishing the sibling child nodes un-
der the same parent node, which may share some common visual
properties significantly and are usually hard to be separated from
each other.
4.2. Hierarchical learning of multi-task sparse metrics for non-root
nodes
For a non-root node at the mid-level of the visual tree, a hierar-
chical approach is developed to leverage both the inter-node visual
correlations (among the sibling child nodes under the same parent
node) and the inter-level visual correlations (between the parent
node and its sibling child nodes at the next level of the visual tree)
for learning its multi-task sparse metric hierarchically. For the sib-
ling non-root nodes, their parent node is used to characterize their
common visual properties, this is the reason why such sibling non-
root nodes and their associated image categories are assigned into
the same parent node. By borrowing the node-specific metric from
the parent node to set a prior regularization term for its sibling
child nodes at the next level of the visual tree, our hierarchical
multi-task sparse metric learning algorithm is able to leverage both
the inter-level visual correlations and the inter-node visual corre-
lations to train more discriminative metrics for the sibling child
nodes under the same parent node. We define the inter-level reg-
ularization term as:
W0 − Wp 2 (11)
where W0 is the common metric shared among the sibling child
nodes under the same parent node p and the Wp is the node-
specific metric for the parent node p at the upper level of the
visual tree. For the current non-root node, we make the node-
specific individual metric for its parent node at the upper level
of the visual tree and its commonly-shared metric to be similar
or closer. Thus the node-specific individual metric learned for the
parent node at the upper level of the visual tree is propagated to
its sibling child nodes. For the sibling child nodes under the same
parent node, the joint objective function for hierarchical learning
 Y. Zheng et al. / Pattern Recognition 67 (2017) 97–109
of their multi-task sparse metrics is defined as:

B
min γoW0 − I2F + αt tr[W0 + Wt ]
W0 ,...,WB
 t=1 
B  2  (12)
+ γt Wt F + dt (xi , x j ) + ξi, j,k
2 5.2. Experimental settings
t=1 i, j i, j,k
+βW0 − Wp 2
subject to:
dt2 (xi , xk ) − dt2 (xi , x j ) ≥ 1 − ξi jk
ξi jk ≥ 0
W0 , W1 , . . . , WB  0.
where α t is used to control the sparse regularization of W0 + Wt
for t = 1, . . . , B and β is used to adjust the inter-level regulariza-
tion. When the multi-task sparse metrics are available for the sib-
ling non-root nodes at the current level, a level-by-level process is
adopted to learn the multi-task sparse metrics at the next level of
the visual tree recursively until the leaf nodes are reached.
By leveraging the visual tree to generate subtrees (each subtree
contains one parent node and its sibling child nodes) and identify
the inter-related tasks for multi-task sparse metric learning, our hi-
erarchical multi-task sparse metric learning algorithm can provide
an iterative solution for large-scale metric learning, so that learning
a tree of multi-task sparse metrics over the enhanced visual tree
becomes computationally tractable. By leveraging more discrimi-
native metrics for classifier training, the corresponding tree clas-
sifiers become more discriminative, which can accurately rule out
unlikely coarse-grained groups of image categories (i.e., irrelevant
high-level nodes on the visual tree) at an early stage and signifi-
cantly reduce the computational cost for large-scale image classifi-
cation. By propagating the node-specific metric for the parent node
to its sibling child nodes at the next level of the visual tree, our hi-
erarchical multi-task sparse metric learning algorithm can control
the inter-level error propagation effectively and achieve higher ac-
curacy rates for large-scale image classification.
5. Experimental results and analysis
In this section, we evaluate the performance of the proposed
approach on three public image sets. We first investigate the ef-
fect of different parameters on the classification performance. Then
we validate the effectiveness of the proposed enhanced visual tree
through compared with other hierarchical structure. We finally
evaluate our hierarchical multi-task sparse metrics classifier by
comparing with flat classifiers and other state-of-the-art hierarchi-
cal approaches.
5.1. Image sets
In this paper, three well-known image sets are used to evaluate
the performance of our method, including the CIFAR-100 [40], the
Caltech-256 [41], and the ILSVRC-2012 [5]. Each of them is briefly
introduced as follows.
CIFAR-100 has 100 image categories and each category contains
600 images, where 500 are the training images and 100 are the
testing images. The size of each image is 32 × 32 pixels.
Caltech-256 contains 256 image categories, and each contains
more than 80 images. The size of each image is about 300 × 200
pixels. The object’s category label as well as its bounding box are
manually annotated. In our experiment, we set 50 images per class
for training and the rest are for testing.
ILSVRC-2012 contains 10 0 0 image categories and it is a subset
of ImageNet, where each category has over 1,0 0 0 images. Our algo-
rithm is trained based on ILSVRC2014 (the latest databases remain
unchanged from ILSVRC2012), which has over 1.2 million training
103
images, 50 thousands validation images, and 150 thousands testing
images. We utilize all training images to train our classifier and use
all validation images as our testing data.
The DeCAF features [42] are extracted for image representation
in our experiments. The DeCAF features are extracted by using a
deep convolutional network [43], where the 7th layer (fully con-
nected layer) is treated as the discriminative features for image
representation. The dimensionality of the original DeCAF features
is 4096, PCA is used to reduce the dimensionality of the DeCAF
features to 128 for speeding up the process for image classifica-
tion. The Mean Accuracy (%) is used as the criterion to evaluate
the performance of all these approaches. A single machine with
4 3.20 GHz cores and 32GB memory is utilized to run all experi-
ments. There are many parameter settings in this paper. We inves-
tigate the effect of different parameters on the classification per-
formance at different stages.
5.2.1. Visual tree learning
In Section 3, for the balance of representativeness and diversity
of active samples, we set the tradeoff parameter λ = 0.5. In active
sampling, the number of active samples will affect the structure
of enhanced visual tree, and different tree structures will result
in different classification accuracy, in addition, more samples will
lead to more computational cost for constructing the enhanced vi-
sual tree. Therefore, we determine the number of active samples
per category by experiment. The effect of active samples per cat-
egory is shown in Fig. 4. The x-axis indicates the number of ac-
tive samples we select from each category. The left y-axis indi-
cates the classification accuracy rates (%). The right y-axis indicates
the computational cost (s) for constructing the enhanced visual
tree. The red curve shows the classification accuracy with changes
in the number of active samples while the blue curve shows the
computational cost for constructing the enhanced visual tree with
changes in the number of active samples. One can observe that
with the growth in the number of active samples, the classifica-
tion accuracy is not necessarily increased. The reason is only the
active samples reflect the distribution of the original data, a rea-
sonable tree structure can be achieved to obtain better classifica-
tion accuracy, however, more active samples do not guarantee that
we can approximate the real distribution of the original data accu-
rately. On the other hand, one can observe that the computational
cost increase linearly with the number of active samples. Accord-
ing to the results in Fig. 4, we set the number of active samples
per category Nas = 30 for CIFAR-100 image set, Nas = 5 for Caltech-
256 image set and Nas = 15 for ILSVRC-2012 image set. Moreover,
the threshold θ affect the structure of enhanced visual tree. Ac-
cording to our experience, we set θ = 4.0, 5.0, 1.5 for CIFAR-100,
Caltech-256 and ILSVRC-2012 image set respectively.
5.2.2. Hierarchical learning
In Section 4, there are many parameters affect our hierarchi-
cal classifier: trade-off parameters γ 0 and γ t , sparse regularization
parameter α t and inter-level regularization parameter β . Note that
we consider the importance of all the tasks are the same, so we
set all the γ t t = 1, . . . , B are equal, so does the α t . We evaluate
the effect of these parameters as shown in Fig. 5. Fig. 5(a), (b) and
(c) correspond CIFAR-100, Caltech-256 and ILSVRC-2012 image set
respectively. Since all γt , t = 1 . . . T are equal, we set a parameter
ψ = γ0 /γt to verify effect of trade-off parameters. The first col-
umn shows the evaluation for the effect of trade-off parameters for
three image sets, the second column shows the evaluation for the
sparse regularization parameter and the third column shows the
evaluation for the inter-level regularization parameter. The x-axis
104 Y. Zheng et al. / Pattern Recognition 67 (2017) 97–109
Fig. 4. The effect of active samples to enhanced visual tree. The red curve shows the classification accuracy with changes in the number of active samples while the blue
curve shows the computational cost for building the enhanced visual tree with changes in the number of active samples. (For interpretation of the references to color in this
figure legend, the reader is referred to the web version of this article.)
Table 1
Comparison of classification accuracy (%) of the different hierarchical
structure and proposed enhanced visual tree on three image sets.
Approaches CIFAR-100 Caltech-256 ILSVRC-2012
Semantic ontology [45] 48.95 57.78 44.47
Label tree [6] 52.04 61.62 49.64
Visual tree [10] 51.30 62.13 51.03
Enhanced visual tree 54.33 64.06 58.76
indicates the value of the different parameters. The y-axis indicates
the classification accuracy (%). We choose the trade-off parameters
ψ from [0.005 0.01 0.05 0.1 0.5 1 5 10 50 100 500] and choose
the sparse regularization parameter α t and the inter-level regular-
ization parameter β from [0.01 0.05 0.1 0.5 1 5 10 50 100]. One
can observe that these parameters do not affect the classification
accuracy seriously. It shows that the proposed hierarchical learning
approach is robust to parameter setting. According to the results
in Fig. 5, we set γ0 = 1, γt = 100, αt = 0.5 and β = 0.1 for CIFAR-
100 image set, γ0 = 1, γt = 50, αt = 1 and β = 0.5 for Caltech-256
image set and γ0 = 1, γt = 10, αt = 1 and β = 10 for ILSVRC-2012
image set.
5.3. Experiments on different hierarchical structures
For the semantic ontology (taxonomies), some databases pro-
vide semantic ontology, such as: CIFAR-100 image set. The reason
that we construct enhanced visual tree rather than available se-
mantic ontology is shown in Fig. 6. We employ the t-SNE toolkit
[44] to visualize the partition of root node of semantic ontol-
ogy and enhanced visual tree respectively in CIFAR-100 image set.
Fig. 6(a) illustrates the partition of root node of semantic ontol-
ogy. One can observe that the data points are confusing in the fea-
ture space, because semantic similar categories does not necessar-
ily close in the feature space. Because the feature space is the com-
mon space for classifier training and image classification, it is more
attractive to learn a better visual hierarchy to organize large num-
bers of image categories directly in the feature space. As shown in
Fig. 6(b), the distinction degree of enhanced visual tree is clearly
higher than semantic ontology.
In this experiment, we compare the proposed enhanced visual
tree (EVT) structure with other hierarchical structures on three im-
age sets as shown in Table 1. For those image sets with no seman-
tic ontology, such as Caltech-256 image set, a semantic hierarchy is
constructed manually by employing WordNet [45]. We employ the
approach [6] to construct label tree and the approach [10] to con-
struct visual tree. For semantic ontology, its hierarchical structure
is constructed by semantic meaning and we perform classification
in the feature space rather than semantic space, so it achieves bad
performance in all three image sets. For label tree and visual tree,
these two approaches learn hierarchical structure in the feature
space and achieve good performance. However, for the label tree,
a confusion matrix is first obtained from one-versus-rest (OVR) bi-
nary classifiers. Such approach for label tree learning may seriously
suffer from huge computational cost. Visual tree approach employs
mean features for category representations, however, such mean
features may not have sufficient capacity for category representa-
tions. Our proposed enhanced visual tree employs representative
active samples for category representations, so it can obtain a bet-
ter hierarchical structure and achieve best performance on all three
image sets.
5.4. Experiments on different method
In this experiments, we compare the proposed method with
state-of-the-art approaches on three image sets respectively. To il-
lustrate our experimental results, our hierarchical multi-task met-
ric learning algorithm is referred as HMML, and it has been com-
pared with other two types of classifiers: flat classifiers and hi-
erarchical classifiers. The baseline kNN classifier with Euclid dis-
tance is referred as Euclid and the state-of-the-art metric learn-
ing algorithm is referred as LMNN [19]. Lei et al. have developed
an hierarchical large-margin metrics learning algorithm referred
as HLMM [46], which learns the metric automatically from the
training samples. The hierarchical multi-task learning algorithm is
referred as HMTL [24]which performs multi-task metric learning
(without inter-level metric propagation). Grauman et al. have de-
veloped an interesting approach to learn a tree of disjoint met-
rics by adding a trace-norm regularization term into the objective
function, so that the classifiers at different levels can select disjoint
metrics or feature subsets. The idea of this approach has some sim-
ilarities with proposed method, so we should compare with it and
it is referred as ToM [31]. Another type of inter-level constraint for
SVM classifiers is also included for comparison and it is referred as
HTC [10]. Deep learning has demonstrated its outstanding perfor-
mance on extracting discriminative features to significantly boost
the accuracy rates for large scale image classification. Therefore,
on large-scale image dataset ILSVRC-2012, we compare our algo-
rithm with convolutional neural networks (referred as CNN) with
Alex model [43] which is the same model with our feature extrac-
tion. In this paper, we employ the Matconvnet [47] deep learning
toolbox1 to complete the experiment about CNN.
5.4.1. CIFAR-100
We first investigate the performance of the proposed HMML
method in the CIFAR-100 image set. The experimental results are
1
http://www.vlfeat.org/matconvnet/
 Y. Zheng et al. / Pattern Recognition 67 (2017) 97–109 105

Fig. 5. The effect of parameters to classification accuracy. The first column shows the effect of trade-off parameters, the second column shows the effect of sparse regular-
ization parameter and the third column shows the effect of inter-level regularization parameter.

shown in Table 2. From these results, one can observe that: (a) All
the hierarchical approaches can achieve higher accuracy rates than
the flat approaches because the flat approaches completely ignore
the inter-category visual correlations and learn the metrics for all
the categories independently, on the other hand, the hierarchical
approaches can effectively leverage the inter-category visual simi-
larities for enhancing the separability of the sibling child nodes un-
der the same parent node; (b) By learning large-margin metrics to
make the exemplars in the same category to be closer and the ex-
emplars from different categories to be far away, both the flat and
hierarchical approaches can achieve higher performances than the
traditional nearest neighbor approach by using Euclidean metric
directly; (c) Our hierarchical multi-task metric learning algorithm
(HMML) can have better performance than the traditional HLMM
approaches, because our algorithm can leverage multi-task learn-
ing and inter-level metric propagation to learn more discriminative
metrics. In addition, our visual hierarchy has provided a good envi-
ronment to automatically identify the inter-related tasks for multi-
task metric learning. By supporting inter-level metric propagation,
our hierarchical multi-task metric learning algorithm (HMML) can
learn more discriminative metrics and control the inter-level er-
ror propagation effectively. The experimental results show that our
106 Y. Zheng et al. / Pattern Recognition 67 (2017) 97–109

Fig. 6. Visualization of semantic ontology and enhanced visual tree in CIFAR-100 image set.

Table 2 sual hierarchy plays an essential role in hierarchical learning of a

Comparison on CIFAR-100 image set.
tree of multi-task sparse metrics.
Structure Approaches Mean accuracy (%)

Flat Euclid 46.03

LMNN [19] 48.84 5.4.3. ILSVRC-2012
Hierarchical HLMM [46] 50.16 We investigate the performance of the proposed HMML method
HMTL [24] 52.78 in the ILSVRC-2012 image set. An enhanced visual tree is learned
ToM [31] 53.02 automatically, and Fig. 7 illustrates the enhanced visual tree with
HTC [10] 53.53
5 levels, where the icon images are used to illustrate the image
HMML 54.33
categories and one icon image is used to represent one particular
image category. The icon images for the visually-similar image cat-
Table 3 egories, which are assigned into the same non-leaf node, are in-
Comparison on Caltech-256 image set.
tegrated to illustrate that non-leaf node jointly. Due to the space
Structure Approaches Mean accuracy (%) limitation, only three subtrees are illustrated.
Flat Euclid 50.36 Table 4 shows the experimental results. One can observe that
LMNN [19] 54.05 the improvement on the accuracy rates of LMNN is not obvious as
Hierarchical HLMM [46] 58.17 compared with the traditional kNN classifiers by using Euclidean
HMTL [24] 60.22 distance directly. Our hierarchical multi-task metric learning algo-
ToM [31] 61.52 rithm can achieve higher accuracy rates than the ToM and HTC
HTC [10] 60.78
approaches. Deep learning method have achieved reasonable suc-
HMML 64.06
cesses on large-scale image classification because of its outstand-
ing performance on extracting discriminative features. However,it
utilizes softmax regression as last layer of network and it will
proposed method achieves better performance than ToM because lead to ignore the inter-category correlations. Thus, one can ob-
the ToM approach concentrate on utilizing trace norm to control serve that our hierarchical multi-task metric learning algorithm
the inter-level error propagation and ignore the inter-category vi- can achieve better performance than the CNN methods. This evi-
sual correlations. Moreover, the proposed method also achieves dence has shown that our hierarchical multi-task metric learning
better performance than HTC because our enhanced visual tree can algorithm can accurately learn a tree of multi-task sparse met-
characterize the inter-category visual correlations accurately in the rics over the enhanced visual tree and control the inter-level error
feature space. propagation effectively.
The detailed comparison on all 10 0 0 image categories in the
5.4.2. Caltech-256 ILSVRC-2012 image set is illustrated in Fig. 8. Each method is ar-
We investigate the performance of the proposed HMML method ranged in ascending order of the category accuracy. Obviously, our
in the Caltech-256 image set. The experimental results are shown hierarchical multi-task metric learning algorithm can achieve bet-
in Table 3. One can observe that similar conclusions are reached ter accuracy rates for most of 10 0 0 image categories. Fig. 9 il-
as we have obtained from the CIFAR-100 image set. In addition, lustrates the comparison of all approaches. The x-axis indicates
the hierarchical approaches with inter-level regularization (inter- the classification accuracy rates (%), and the y-axis indicates the
level metric propagation) can learn more discriminative metrics percentage of the image categories whose accuracy rates exceeds
and control the inter-level error propagation effectively, and thus the corresponding threshold. One can observe that our hierarchi-
it can achieve better performance on large-scale image classifica- cal multi-task metric learning algorithm can achieve better perfor-
tion. Our experimental results have also demonstrated that the vi- mance on large-scale image classification.
 Y. Zheng et al. / Pattern Recognition 67 (2017) 97–109 107

Fig. 7. Visualization of three different paths of our enhanced visual tree for ILSVRC-2012 image set with 10 0 0 categories. The icon images are used to illustrate the image
categories and one icon image is used to represent one particular image category.

Fig. 8. Accuracy comparison on the ILSVRC-2012 image set with 10 0 0 image categories.

Table 4
Comparison on ILSVRC-2012 image set.

Structure Approaches Mean accuracy (%)

Flat Euclid 44.09

LMNN [19] 45.72
Hierarchical HLMM [46] 51.67
HMTL [24] 52.94
ToM [31] 56.62
HTC [10] 53.27
CNN [47] 56.69
HMML 58.76

5.5. Experiments on computational efficiency

The computational cost at test time is evaluated for all the
approaches on 3 image sets. Table 5 illustrates the experimental
results. One can observe that all the hierarchical approaches are
much faster than the flat approaches which computational cost in-
creased linearly with the increasing of the numbers of image cate-
gories. It is because the hierarchical approaches allows us to search
partial categories at testing phase (i.e., each test sample is matched
with partial nodes over our enhanced visual tree and node clas-
sifiers), thus they can achieve high efficiency. Different hierarchi-
Fig. 9. Comparison on accuracy percentage. The x-axis indicates the classification
accuracy rates (%), and the y-axis indicates the percentage of the image categories
whose accuracy rates exceeds the corresponding threshold.
cal approaches have little difference on testing efficiency because
their efficiency depends mainly on the depth of the enhanced vi-
sual tree, and their tree structure is uniform in this experiment.
One can also note that the testing efficiency of deep learning is
not high.
108 Y. Zheng et al. / Pattern Recognition 67 (2017) 97–109

Table 5 [8] B. Liu, F. Sadeghi, M. Tappen, O. Shamir, C. Liu, Probabilistic label trees for
Comparison of testing time (Milliseconds/image). efficient large scale image classification, in: CVPR, 2013, pp. 843–850.
[9] J. Fan, X. He, N. Zhou, J. Peng, R. Jain, Quantitative characterization of semantic
Approaches CIFAR-100 Caltech-256 ILSVRC-2012 gaps for learning complexity estimation and inference model selection, IEEE
Euclid 25.86 115.59 1309.15 Trans. Multimedia 14 (5) (2012) 1414–1428.
LMNN 23.07 127.08 1233.94 [10] J. Fan, N. Zhou, J. Peng, L. Gao, Hierarchical learning of tree classifiers for
large-scale plant species identification, IEEE Trans. Image Process. 24 (11)
HLMM 9.27 6.65 30.1
(2015) 4172–4184.
HMTL 9.89 7.01 33.1
[11] E.P. Xing, M.I. Jordan, S. Russell, A.Y. Ng, Distance metric learning with appli-
ToM 9.18 5.77 28.49 cation to clustering with side-information, in: NIPS, 2003, pp. 505–512.
HTC 21.46 18.25 55.95 [12] A. Bar-Hillel, T. Hertz, N. Shental, D. Weinshall, Learning a mahalanobis metric
CNN – – 65.80 from equivalence constraints, J. Mach. Learn. Res. 6 (6) (2005) 937–965.
HMML 9.93 6.98 33.4 [13] A. Globerson, S.T. Roweis, Metric learning by collapsing classes, in: NIPS, 2005,
pp. 451–458.
[14] C.J. Veenman, Statistical disk cluster classification for file carving, in: IAS, 2007,
6. Conclusion pp. 393–398.
[15] B. Kulis, Metric learning: a survey, Found. Trends Mach. Learn. 5 (4) (2012)
287–364.
In this paper, a hierarchical multi-task sparse metric learning [16] Q. Qian, R. Jin, S. Zhu, Y. Lin, Fine-grained visual categorization via multi-stage
algorithm is developed to learn a tree of multi-task sparse met- metric learning, in: CVPR, 2015, pp. 3716–3724.
rics hierarchically over an enhanced visual tree to achieve a fast [17] M.P. Kumar, P.H. Torr, A. Zisserman, An invariant large margin nearest neigh-
bour classifier, in: ICCV, 2007, pp. 1–8.
solution for large-scale image classification, where both the inter- [18] L. Torresani, K.-c. Lee, Large margin component analysis, in: NIPS, 2006,
level visual correlations (between the parent node and its sibling pp. 1385–1392.
child nodes at the next level of the enhanced visual tree) and the [19] K.Q. Weinberger, L.K. Saul, Distance metric learning for large margin nearest
neighbor classification, J. Mach. Learn. Res. 10 (2009) 207–244.
inter-node visual correlations (among the sibling child nodes under [20] J. Goldberger, G.E. Hinton, S.T. Roweis, R. Salakhutdinov, Neighbourhood com-
the same parent node) are exploited to learn a tree of multi-task ponents analysis, in: NIPS, 2004, pp. 513–520.
sparse metrics over the enhanced visual tree. By explicitly sepa- [21] N. Verma, D. Mahajan, S. Sellamanickam, V. Nair, Learning hierarchical similar-
ity metrics, in: CVPR, 2012, pp. 2280–2287.
rating the node-specific metrics from the commonly-shared met- [22] Y. Ying, K. Huang, C. Campbell, Sparse metric learning via smooth optimization,
ric, our hierarchical multi-task metric learning algorithm can learn in: NIPS, 2009, pp. 2214–2222.
more discriminative metrics for large-scale image classification ap- [23] Y. Zhang, D.-Y. Yeung, Transfer metric learning by learning task relationships,
in: ACM SIGKDD, 2010, pp. 1199–1208.
plication. The experimental results have demonstrated that our al-
[24] S. Parameswaran, K.Q. Weinberger, Large margin multi-task metric learning,
gorithm has superior performance as compared with other state- in: NIPS, 2010, pp. 1867–1875.
of-the-art techniques on both the classification accuracy and the [25] N. Zhou, J. Fan, Automatic image–text alignment for large-scale web image in-
computational efficiency. dexing and retrieval, Pattern Recognit. 48 (1) (2015) 205–219.
[26] L. Cai, T. Hofmann, Hierarchical document categorization with support vector
The contributions of this paper can be summarized as: (a) An machines, in: ICIKM, 2004, pp. 78–87.
enhanced visual tree is constructed to organize large numbers [27] M. Marszalek, C. Schmid, Constructing category hierarchies for visual recogni-
of image categories in a coarse-to-fine fashion and automatically tion, in: ECCV, 2008, pp. 479–491.
[28] E. Bart, M. Welling, P. Perona, Unsupervised organization of image collections:
identify the inter-related tasks for multi-task sparse metric learn- taxonomies and beyond, IEEE Trans. Pattern Anal. Mach. Intell. 33 (11) (2011)
ing; (b) A new objective function is developed for multi-task sparse 2302–2315.
metric learning; (c) A top-down approach is developed for sup- [29] S. Gopal, Y. Yang, Recursive regularization for large-scale classification with
hierarchical and graphical dependencies, in: ACM SIGKDD, 2013, pp.
porting hierarchical learning of a tree of multi-task sparse metrics 257–265.
over the enhanced visual tree. [30] L. Xiao, D. Zhou, M. Wu, Hierarchical classification via orthogonal transfer, in:
ICML, 2011, pp. 801–808.
Acknowledgments [31] K. Grauman, F. Sha, S.J. Hwang, Learning a tree of metrics with disjoint visual
features, in: NIPS, 2011, pp. 621–629.
[32] B. Babenko, S. Branson, S. Belongie, Similarity metrics for categorization: from
The authors would like to thank the editor and the anonymous monolithic to category specific, in: ICCV, 2009, pp. 293–300.
reviewers for their critical and constructive comments and sug- [33] I.E. Yen, D. Malioutov, A. Kumar, Scalable exemplar clustering and facility lo-
gestions. This work was supported in part by the National Natu- cation via augmented block coordinate descent with column generation, in:
AISTATS, 2016, pp. 1260–1269.
ral Science Foundation of China under Grant 61432014, U1605252, [34] H. Wang, X. Gao, K. Zhang, J. Li, Single image super-resolution using ac-
61571347, 61672402 and 61603233, in part by Key Industrial In- tive-sampling gaussian process regression, IEEE Trans. Image Process. 25 (2)
novation Chain in Industrial Domain under Grant 2016KTZDGY-02, (2016) 935–948.
[35] B.J. Frey, D. Dueck, Clustering by passing messages between data points, Sci-
in part by the Fundamental Research Funds for the Central Uni-
ence 315 (5814) (2007) 972–976.
versities under Grant BDZ021403 and Grant JB149901, in part by [36] I. Givoni, C. Chung, B.J. Frey, Hierarchical affinity propagation, UAI (2011)
National High-Level Talents Special Support Program of China un- 238–246.
[37] R.T. Rockafellar, R.J.-B. Wets, Variational Analysis, 317, Springer Science & Busi-
der Grant CS3111720 0 0 01, in part by the Program for Changjiang
ness Media, 2009.
Scholars and Innovative Research Team in University of China un- [38] F. Cai, V. Cherkassky, Generalized smo algorithm for svm-based multitask
der Grant IRT13088. learning, IEEE Trans. Neural Netw. Learn. Syst. 23 (6) (2012) 997–1003.
[39] T. Evgeniou, C.A. Micchelli, M. Pontil, Learning multiple tasks with kernel
References methods, J. Mach. Learn. Res. (2005) 615–637.
[40] A. Krizhevsky, G. Hinton, Learning multiple layers of features from tiny images,
[1] Y. Li, Q. Huang, W. Xie, X. Li, A novel visual codebook model based on fuzzy Technical report, U. Toronto (2009).
geometry for large-scale image classification, Pattern Recognit. 48 (10) (2015) [41] G. Griffin, A. Holub, P. Perona, Caltech-256 object category dataset(2007).
3125–3134. [42] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama,
[2] L.-J. Li, C. Wang, Y. Lim, D.M. Blei, L. Fei-Fei, Building and using a semantivisual T. Darrell, Caffe: convolutional architecture for fast feature embedding, arXiv
image hierarchy, in: CVPR, 2010, pp. 3336–3343. preprint arXiv:1408.5093(2014).
[3] B. Zhao, F. Li, E.P. Xing, Large-scale category structure aware image categoriza- [43] A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep con-
tion, in: NIPS, 2011, pp. 1251–1259. volutional neural networks, in: NIPS, 2012, pp. 1097–1105.
[4] S. Gopal, Large-scale Structured Learning, Carnegie Mellon University, 2014 [44] L.v.d. Maaten, G. Hinton, Visualizing data using t-sne, J. Mach. Learn. Res. 9
Ph.D. thesis. (11) (2008) 2579–2605.
[5] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: a large-scale [45] G.A. Miller, Wordnet: a lexical database for english, Commun. ACM 38 (11)
hierarchical image database, in: CVPR, 2009, pp. 248–255. (1995) 39–41.
[6] S. Bengio, J. Weston, D. Grangier, Label embedding trees for large multi-class [46] H. Lei, K. Mei, J. Xin, P. Dong, J. Fan, Hierarchical learning of large-margin met-
tasks, in: NIPS, 2010, pp. 163–171. rics for large-scale image classification, Neurocomputing 208 (2016) 46–58.
[7] J. Deng, S. Satheesh, A.C. Berg, F. Li, Fast and balanced: efficient label tree [47] A. Vedaldi, K. Lenc, Matconvnet: convolutional neural networks for matlab, in:
learning for large scale object recognition, in: NIPS, 2011, pp. 567–575. ACM MM, 2015, pp. 689–692.
 Y. Zheng et al. / Pattern Recognition 67 (2017) 97–109 109

Yu Zheng received the B.Eng. degree in electronic information engineering from Xidian University, Xi’an, China, in 2012. He is currently pursuing the Ph.D. degree in
intelligent information processing with the VIPS Laboratory, School of Electronic Engineering, Xidian University. His current research interests include machine learning and
computer vision.

Jianping Fan received the M.S. degree in theory physics from Northwestern University, Xi’an, China, in 1994, and the Ph.D. degree in optical storage and computer science
from the Shanghai Institute of Optics and Fine Mechanics, Chinese Academy of Sciences, Shanghai, China, in 1997. He was a Post-Doctoral Researcher with Fudan University,
Shanghai, in 1998. From 1998 to 1999, he was a Researcher with the Japan Society of Promotion of Science, and the Department of Information System Engineering, Osaka
University, Osaka, Japan. From 1999 to 2001, he was a Post-Doctoral Researcher with the Department of Computer Science, Purdue University, West Lafayette, IN. In 2001, he
joined the Department of Computer Science, University of North Carolina at Charlotte. His research interests include automatic image/video analysis, semantic image/video
classification, personalized image/video recommendation, surveillance videos, and statistical machine learning.

Ji Zhang received the B.E. and M.S. degrees in Electronics Engineering and in 2011, and 2014, respectively, Xi’an Jiaotong University, Xi’an, China. He is now pursuing his
Ph.D. degree on Pattern Recognition and Intelligence Systems at the Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University. His research interests include
image analysis and embedded vision computing. He is now a visiting student at UNC-Charlotte.

Xinbo Gao received the B.Eng., M.Sc., and Ph.D. degrees in signal and information processing from Xidian University, Xi’an, China, in 1994, 1997, and 1999, respectively. From
1997 to 1998, he was a Research Fellow at the Department of Computer Science, Shizuoka University, Shizuoka, Japan. From 20 0 0 to 2001, he was a Post-Doctoral Research
Fellow at the Department of Information Engineering, the Chinese University of Hong Kong, Hong Kong. Since 2001, he has been at the School of Electronic Engineering,
Xidian University. He is currently a Cheung Kong Professor of Ministry of Education, a Professor of Pattern Recognition and Intelligent System, and the Director of the
State Key Laboratory of Integrated Services Networks, Xi’an, China. His current research interests include multimedia analysis, computer vision, pattern recognition, machine
learning, and wireless communications. He has published five books and around 200 technical articles in refereed journals and proceedings. Prof. Gao is on the Editorial
Boards of several journals, including Signal Processing (Elsevier), and Neurocomputing (Elsevier). He served as the General Chair/Co-Chair, Program Committee Chair/Co-Chair,
or PC Member for around 30 major international conferences. He is currently a fellow of the Institution of Engineering and Technology.