A New Machine Learning Method based on PCA and SVM*

Rongyong Zhao CIMS Research Center, Tongji University Shanghai, 200092, China zhaorongyong Hao Zhang Department of Information and Controlling Engineering, Shanghai University of Electric Power, Shanghai, 200090, China hzhangk(yaho o.com.cn

@mail.tongji.e
du.cn

Jiangfeng Lu CIMS Research Center, Tongji University Shanghai, 200092, China lujianfeng(ma il.tongji.edu.cn

Cuiling Li Department of Electrical Automation Shanghai Maritime University, Shanghai, 200135,China licuiling88@so hu.com

Hui Zhang CIMS Research Center, Tongji University Shanghai, 200092, China lujianfeng(ma il.tongji.edu.cn

Abstract
In fault pattern recognition field, the real-time onlinefault diagnosis is a new requirement especially from the high-speed machines, and also the

wavelet packet analysis, PCA, and SVM is proved a new, effective and practical approach especially for the real-time onlinefault diagnosis exactly.

magnificent researching direction. The precision and speed of the classification are important research issues in fault pattern recognition for this kind of intelligent fault diagnosis. Although many improved ANN (Artificial Neural Net) methods have been proposedfor this purpose, most approaches focus only on the classification precision, instead of the computing speed. In this paper, a SVM modelforfault diagnosis is introduced and analyzed about its limitation from sensibility to noisy data. To address this problem, we introduce the PCA (Principal Component Analysis) method to reduce the dimension of the sample set and de-noise data sampledfrom the machine. Furthermore, for the whole real-time data processing and fault pattern recognition, a kind of wavelet packet analysis is applied to translate the field sensor signal in time-spectrum into the energy value in frequency-spectrum correspondingfrequency segments real-timely. Therefore we present a new machine learning method: PSVM (Primary component analysis Support Vector Machine) method based on wavelet packet analysis in the fault diagnosis field. And this improved method particularly betters the precision and computing speed of the classification then other typical networks. And also the whole processing workflow is illustrated in details. Finally the diagnosis result of the CNC grinding machine demonstrates this method with both high classification precision and quick computational speed And the whole integration of
*

1. Introduction
Based on statistics learning theory, SVM (Support Vector Machine) is a relatively new method of machine learning method by Vapnik [1-5]. According to the structural risk minimization (SRM) [6] principle, SVM focuses the generalization of learning machine, therefore has good classifying precision and good solution to over-learning problem. Mathematically it is its advantage that it is a kind of convex optimization problems, and its local optimization vale must be whole optimization value, which the typical ANN and other machine learning methods don't have.Recently typical SVM for classification is used for fault pattern recognition successfully [13]. In this paper, our application object is a CNC grinding machine(in this paper, we select MK215 as a typical type). For fault diagnosis, Wavelet Packet Decomposition is used firstly to transfer the real-time vibrating signal into energy distribution in the spectrum directly. Secondly, the PCA (primary components analysis, PCA) approach is applied to preprocess the noisy data and make them de-noised. Thirdly we classify the sampled data set for different typical fault with SVM. Finally we get a satisfying result: very high accuracy for the CNC grinding machine fault diagnosis in very short computation time.

The Sino-German Government Cooperation Project (NO.20002DFG00027) supported this research work.

1-4244-0605-6/06/$20.00 C2006 IEEE.

187

2. Classification Principle of SVM

Based on the computational efficiency without depend on the number of features of classified entities, SVM classifier has higher generalization ability than general neural network [5], which is a good performance in fault diagnostics, because the number of chosen features as the base of fault classification is often limited factually.
Wx+b=+1

/ , x+b.
0

/Vx+b=O

(Cortes and Vapnik, 1995). The detailed solution is similar to linear separable case, which can be made refer to Cortes and Vapnik's paper [2,12]. The selection of kernel function should match Mercer's conditions. And from linear separable, linear non-separable and nonlinear separable cases, it is very obvious that the typical SVM can only classify the data with relatively distance effectively, which make it disable to classify the data mixed with noise from the factual sensors.

3. New Machine Learning Method-PSVM

It is unavoidable that the fault sample data set of the vibrating signal contains noise on some degree. Especially when these data with noise are the very support vectors to decide the classification border, the generalization ability and classification ability will reduce greatly. Therefore original and single SVM cannot guarantee the high accuracy to the classification of fault data with noise. Considering the SVM's sensitivity to noise, we should find a new solution to minimize the effect of noise disturbance in the actual fault diagnosis. PCA(principal components analysis, PCA)[7,8] with its clear computational methodology, does not suffer from the arbitrariness of the Fourier filter where choice of the frequency cutoff is subjective by the experts. Therefore, PCA is a very appropriate approach to noise removal. We integrate the PCA and standard SVM in to a new method-PSVM (Primary component analysis Support Vector Machine). In detail, we apply the PCA to preprocess the sample data with noise from different faults, then use SVM to process the data de-noised. And by this new combination of the PCA and SVM, the negative effect of noise to classification accuracy of SVM is removed completely. PCA preprocess steps to sample data sets are shown as follows: a: compute covariance matrix of sample data points vectors S = (s, ) , and sample correlative matrix ;b: In order to eliminate measurement R = (r ) unitage effect, normalize the sample data, and then compute normalized primary components; c: select t ( t<n ) sample primary components in front, until the accumulative contributing rate reaches to some percentage (80 90 percent normally);d: reduce the sample data set matrix dimension from m x n to m x t , ( t<n ) ,and input the small matrix into SVM;e: execute SVM classification algorithm.

0
I\Mrgi

Fig.l. Optimal classification hyperplane sketch map

In Figure. 1 above, the typical SVM, we can find that when we use SVM for classification, we just depend on a small part of sample data for training (also called support vectors) to decide the decision-making border. These support vectors are very important to the final result of classification hyperplane. In the application to fault diagnosis, in fact, the classification arithmetic of SVM is mainly used. As to two-class classification, we assume the sample set as = n I x E= xi (xi, y1) (Xiii Yi) 1,- , n,x E R d

belongs to class I or II, here Yi E {+ 1,- 1}

We suppose Y i will be 1 ,if x i belongs class I, and -1 for class II. As to linear separable classification case, the training data set is divided by a hyperplane that is markedas w x + b = 0 . Here w is thenormal of the hyperplane, and b marks the relative distance from the origin. A optimal hyperplane can separate the two classes with widest margin. And we can transfer the optimal hyperplane into computing the minimum of Eq. (1) in the constraint of Eq. (2):
.

~~every

(( W) =

=W i 1,12
,- ,n,

(2)

s.t. yi (xi w +b)21i=

(2)

As to linear non-separable case, the non-negative slack variables are introduced into inequation (2)

1-4244-0605-6/06/$20.00 C2006 IEEE.

188

4. Case Study
4.1 Fault diagnosis process The whole fault diagnosis program flow is shown in Fig. 2. The actual fault signal is sampled from the running grinding machine in the workshop. When the judgment result is true, the signal with noise on some degree in input into wavelet packet analysis program, after which, characteristics of the sampled fault signal are computed out in the energy vector form simultaneously and directly. And then, the vector is entered into the PSVM program module, from which we can draw the conclusion about the existence and the degree of fault.

As to one of typical faults the spindle collision and abrasion fault, the character vector can also be obtained. The result is shown in the Fig.3 below. With the same method, other fault signal data, and the detailed character vector data are shown in Tablet.
-

(a). Sensor signal with noise

(b). Character vector

Fig.3. Spindle Collision and abrasion fault signal character

Table 1. Typical fault data with noise

Fau1t
062.5-

62.5fSHz

25Hz~

187.5fSHz
0.0082751
0.0051332

125-

187.5-

250Hz~

312.5fSHz
0.047812
0.015901 0.043578 004231

250-

312.5-

375Hz~

437.5fi
O.OOSIS
0.660123 0.05812

375-

437.5-

50SOHz~

Fl,l
F2,1 F3,1

0.004246
0.0160120 0.0033012

0.0070021 0.0042892
0.0035035

0.91391
0.28234

0.004289
0.011559
0.001612

0.0077776
0.0080602

0.0076734
0.0068156

0.87538

0.0067012

F4,1

0.0047012
0.0015013 0.0008145
0.020146

0.0050123

0.7412 0.017001
0.001601
0.06012

0004212

0.21037
0.0015024

0.0048962 0.0012981
0.0011246
0.0036016

F5,1
F6,1
F7,1

0.029023 0.028914
0.0025602

0.93901 0.97012
0.0032573

0.007812 0.005619
0.006331

0.0017045 0.0007217
0.013913

0.0006731
0.88145

Fig. 2. Fault analysis program flow. Note: A: the average of the absolute value of signal amplitude; Max: the maximum data (assumed with experience and also adjusted according to the actual running conditions of the machine).

In this wavelet packet decomposition, dmey wavelet is selected, 3 levels wavelet packet decomposition, 8 sub-space in the bottom level (the third level). And the energy in every sub-space is computed with the sum of square of the signal data in the sub-spaces. The grinding spindle's rotating speed is 9300 r/min correspondingly, 155 Hz in frequency (the monitored frequency). Considering the Shannon sampling theorem and actual noisy disturbance, we set sampling frequency more than 5 times of the rotating frequency, 1OOOHz. And then with the wavelet packet decomposition ,the whole cared frequency domain is divided into 8 segments: 0-62.5Hz 62.5-125Hz 1 25'- 187.5Hz- 187.5 250Hz- 250 312.5Hz312.5-375Hz 375-437.5Hz 437.5-500Hz.
-

Note:F1,1: imbalance grinding wheel fault; F2, 1: noncentered spindle fault ; F3,1 : off-centering bearing fault; F4,1: spindle crack fault; F5,1: spindle looseness fault; F6,1: oil film oscillation F7,1 : imbalance spindle rigidity fault.

4.2 Application of PSVM

In the same grinding machine running conditions (spindle rotating speed: 9300r/min; workpiece rotating speed: 150 r/min), the same environmental conditions (temperature humidity, surrounding noise, etc.) ,7 typical grinding machine's faults are recorded altogether ,as the sample set in the ARM intelligent diagnoser And corresponding every typical fault, there are 50 recorded sample vectors. In the data table
,
.

as

different faults. The values among the table 1 mean the normalized energy in different frequency segment of

folloDws, f

lFi j (i = I1,...,17; j = 1,---.50) t

means

the

1-4244-0605-6/06/$20.00 C2006 IEEE.

189

the typical faults respectively. This application is a classification question for the faults, and can be simplified into the multi-classification question of SVM. As a one of recent research focuses, SVM multiclassification method includes two main solutions: one-to-one solution [9] and one-to-multi solution[10]. In this application, we adopted one-to-multi solution, and we applied the Libsvm [11] toolbox, - 3 SVC, and RBF(Radial Basis Function) as the kernel function, coefficient C= 1000, gamma=0.6,coef= 1,and other coefficients are set as default values. And here we constructed k (k - 1) / 2 two-classification sorters to match the k classes to be classified, and with Max Wins Vote strategy, to decide which class the data set belongs to. With PCA, sample data dimension is reduced from 8 down to 6 ,also the de-noised effect are shown in the table2. And the reduced result is imported into SVM model.
Table 2. The result data after PCA process
I1
12 13 14

6. References
[1] Kui Wu; Kim-Hui Yap.Fuzzy SVM for content-based image retrieval: a pseudo-label support vector machine framework. Computational Intelligence Magazine, IEEE, Volume 1, Issue 2, May 2006, pp. 10 16 [2] Vladimir N .Vapnik, Xue-gong Zhang.The Nature of Statistical Learning Theory[M].Beijing: Tsinghua University Press, 2000(in Chinese). [3] G. Betta, C. Liguori, A. Paolillo, A. Pietrosanto, A., A DSP-based FFT analyzer for the fault diagnosis of rotating machine based on vibration analysis [A], Proc. IEEE Conf. on Instrumentation and Measurement Technology,Budabest, Hungary, 2001, pp. 572-577. [4] Wang, Z.; Marek-Sadowska, M.; Tsai, K.-H.; Rajski, J.Analysis and Methodology for Multiple-Fault Diagnosis,Computer-Aided Design of Integrated Circuits and Systems, IEEE Transactions [J],Volume 25, Issue 3, March 2006, pp. 558 575 [5] N. Cristianini, J. Shawe-Taylor, Support vector machines and other kernel-based learning methods[M] (Cambridge University Press, 2000). [6] Land, W.H., Jr.; Wong, L.; Mckee, D.W.; Masters, T.; Anderson, F.R." Breast cancer computer aided diagnosis (CAD) using a recently developed SVM/GRNN Oracle hybrid", Systems, Man and Cybernetics, 2003. IEEE International Conference on ,Volume: 5 ,5-8 Oct. 2003 pp. 4705 4711 vol.5 [7] Jin-Cheng Fan,Chang-Lin Mei. Data Analysis [M]. Beijing, Science Press, 2002 [8] Sastry, Murali. Application of principal component the role analysis to X-ray photoelectron spectroscopy of noise in the spectra [J], Journal of Electron Spectroscopy and Related Phenomena Volume: 83, Issue: 2-3, February 1, 1997, pp. 143-150 [9] KreBel. U. Pairwise classification and support vector machines[M]. In B. Sch"olkopf, C. J. C. Burges, and A. J. Smola (Eds.), Advances in Kernel Methods Support Vector Learning. Cambridge[M], MAT: Press, 1999. pp. 255-268. [10] Weston, J. and C. Watkins. Multi-class support vector machines[R]. Technical Report CSD-TR-98-04. England: Department of Computer Science Egham surrey. 1998 [11] Chih-Chung Chang, Chih-Jen Lin. LIBSVM: a library for vector machines[EB/OL]. support http://www.csie.ntu.edu.tw/-cjlin/libsvm, 2001-12-04 [12] 0. Mangasarian. Arbitrary-norm separating plane[R]. TR-97-07.University of Wisconsin, 1997. [13] Shui Yu; Fan-Yuan Ma; Jian-Xue Chen; Xing-Guo Yin; Hong-Bo Shi,Unsteady fault diagnosis method for chemical process based on SVM[A], Machine Learning and Cybernetics, 2002. Proceedings. 2002 International Conference on ,Volume: 2 ,4-5 Nov. 2002, pp. 772 775 vol.2
-

I5 -0.68014 0.46503 0.46752 0.46502 -0.62375 1.4331 -1.2078

16

Fl,l F2,1 F3,1 F4,1 F5,1 F6,1 F7,1

-1.0341 1.0054 1.3596 1.0021

1.3958

-0.9012 -1.05021 -0.8235 -1.0521 1.6323 2.1342 -1.3987

1.8014 -0.67069 0.98125 -0.63324 0.16246 -0.17254 -1.3498

-0.23015 0.076589 -0.090232 0.068075 1.6146 -0.1192 0.45983

0.80283 0.03692 -0.08025 0.03298 -0.00924 -0.47005 -1.3201

-1.8021 -3.6456

In detail, we took 10 data from every typical fault sample set as testing data set, and 40 remained from every typical fault sample set as training set. After normalization process to the sample set which would be reduced in dimension and de-noised, trained with - SVC, tested with testing data set, and finally, the classification precision (also the veracity) reached to 95 %, and the whole computing time was only about 1.5s. For example, the same computing task with FFT normally needs 10.52 seconds, and ANN(BP net) 132.61 seconds in Matlab6.5 software.

5. Conclusion
SVM is limited to application because of its sensitivity to the noisy data. Therefore, in this paper, we introduce the PCA to reduce the data set dimension and de-noise the data, which makes SVM an obvious improvement in the ability of the anti-noise and a good classification precision. Therefore we integrate the PCA with typical SVM into a new method-PSVM in the factual application.

1-4244-0605-6/06/$20.00 C2006 IEEE.

190