SSHSPRODADILITY AND STATISTICS {sclished by WALTER A. SHEWHART and SAMUEL S, WILKS. fuer Vie Barer Ralph A. Brady Nicholas. Fisher, J. Sar Hm, inj Tees Goer § Watson fh isn hs ses appeal ei hi aa Matrix Analysis for Statistics JAMES R. SCHOTT ® A Wiley Intercienee Publication JOHN WILEY & SONS, INC. New York * Chichester © Brisbane + Toronto * Singapore © Wenhsin‘io © 197 By Jt Wie Son [ips Reene, Pb Sims In Cont, epducsn tsa fay a of is wok Beyond ‘Sted Copy Ac wit perma of apy owe rane Ron or rn na ‘ooo hats belo the ers Depicte Som ya Some, nee Teno New Yr ‘onsen. brary Of Congres Catling I Publication Das: Se, eres R185 c= (ey Series Protably Ad Sis. Aged Baby ka Se "Wier iencace iblin” fees bhai! oon on de aN o-r3 a pe) Sivonen sen To Susan, Adam, and SarahContents Proface ‘A Review of Elementary Matrix Algebra 1 Tnodueton, 1 Defiitions and Notation, 1 3. Matix Adon and Mulpication, 2 4 The Transpo, 3 5 TheTrace, 4 6 The Determinant. 5 7. ThelImenc, 8 % Parttoned Mates, 11 9. The Rank of = Matin, 12 10. Oxtogonal Mauices, 14 1) Quadratic Forms, 15 12. Complex matress, 10 13 Random Vertes and Some Relsted Problem, 28, 2 Vector Spaces 2 Invodvction. 32 Definitions, 32 Linear Independence and Dependence, 38 Bases and Dimension, 41 “Matrix Rank and Linear Indeendence, 49 Onthonormal Bases and Pojoctions, 48 Projection Matrices, 32 Linear Tinnstormatons and Sytsms of Linear Batons, 60 eae weepsvi conrents 9. The Intersection and Sum of Vector Spaces, 67 10. Convex Sets, 70 Probiems, 14 3, igenvalues and Higenvectors ry 1 troduction, 84 Eigenvalues, Eigenvectors, and kigenspaces, 84 ‘Some Base Propertise of igeavalus and Eigenvectors, 68 ‘Summetic Matrices, 98 ‘Continuity of Eigenvalues and Eigenpojectons, 102 Extremal Properties of Eigenvalues, 108 Psbleme, 122 x Factoriations a Maer Nore wt 1. intoduetion, 131 2 Th Singlar Value Decomposition. 131 3, Te Spectral Decomposition and Square Root Mates of a Symmetie Marat, 138 “The Diagonaliontion of « Square Matis, 144 ‘Te Jordan Decomposition, 147 ‘The Scher Decomposition, 149 ‘The Simutaneous Dagonalization of Two Symmetric Matices, isa 8. Matix Norms, 157 Problems, 162 5. Conerllad Inverse 10 Introduction, 170 ‘The Moore-Penose Generalized Inverse, 171 Some Basie Properties of the Moore-Pencose Inverse, 174 ‘The Mowe-Tesoaelaverc of « Matix Product, 180 “The Moore-Penrose Inverse of Paitoned Matices, 185 “The Moore-Penrose Inverse of @ Sum, 186 ‘The Contnaity ofthe Moore-Penrose Inverse 188 ‘Some Omer Generalized Inverses, 190 9. Comping Generalized Inverse, 107 Problems, 204 6. Systems of Linear Equations 210 Introduction, 210 ‘Consintency ofa System of Equations, 210 Saloon to a Consistent System of Equations, 213, Homogeneous Systems of Equations, 219 Least Squares Solutions to System of Linear :quasons, 222 ‘Least Squtes Estinutin Tou Lose Tn Fall Rank Models, 778 Systeme of Viner Eqvations andthe Singular Value Decomposition, 233, Sparse Linear Systems of Bquations, 259 Probiens, 241 "Special Matrices and Matrix Operators a 1 Introdvtion, 247 Partiuoned Masices, 247 “The Kroneskar Product, 25% ‘The Direct Sum, 260, The Vee Operator, 261 ‘The Hadamard Product, 206 “The Commutation Matrix, 276 Some Other Matrices Associated with the Vee Operator, 283 Nonnegatve Matrices, 288 Ciutat and Toeplitz Matrices, 300, ‘adamard and Vatntcmonde Mace, 305 ‘cbems, 200 8. Matrix Derivatives and Kelated Topics a3 Introduction, 323 ‘Mulvaiable Differin! Calls, 32% \ecine and Matzx Functions, 326 Some Useful Matrix Derivatives, 382 Derivatives of Functions of Paserned Matnces, $39 ‘me Perurbation Medion 337 ‘Maxima and Minis, 442 ‘Convex and Concave Functions, 349 “The Method of Lagrange Maltpis, 253, Problems, 3609. Some Special Topics Related to Quadratic Forms 370 Invoduction, 370 2. Some Results on Idempotent Mace, 370 3. Cochran's Theorem, 374 44 Distituion of Qusorate Forms i Normal Varies, 378 5 Intsonence of Qradratic Rome, 384 6. Expected Values of Quaatie Forms, 390 7. The Wishart Distribution, 398 Proieas 309 References 416 Index or List of Series Tikes Preface ‘As the fld of statistics tas developed over the years, the ole of mats ‘methods has evolved from a tool through which sasical problems could be ‘ore conveniently exprevied tan aeahtly eset prt the developmen Understanding, and we ofthe more complicated saisical analyses that have appeared in recent yeas. As suc, a background in mas analysis has tecome 2 Vital pat ofa grate edcatin in tists, To often. the statistics sad $e stant ges sor her matrix background in is and peces though various courses on topics sucha egriion ali, makvarite analy linear tod. ts, stochastic processes, and so on_An alternative o this fragmented approach [San entre course devoted t matrix methods useful in statis. This ox has een write th sch comme mine Wane be mae a 2 et foe an «advanced undergraduate course wit an unusaly bright group of students and Should prove w be useful asa refereace for Dom apple ana reeareh sats ‘Students beginning a gradate program in statis often have ter revi- us egress in ther Held, such as aes a oily thei sana backgrounds may not be all ha extensive. With hs in mind, T Have Wed 0 ‘ake the statistical topics presented as examples inthis text at se conaied [ose Thie has hee accomplished hy ining» ection i the st chapter that coves some basic statistical concepts and by having most ofthe stat ‘ial examples deal wt applications at ae fay simple to understand fr instance, many of these examales involve leas souaesreresion of applica ‘Thus an inroductory sttstescoure should provige th reader with a sufient background in satsics An additonal prerequisite fan unde ‘radate course 1m matrices or liner algebra, While a calculus background is cestary for smn ptone ofthe hook, oe! nah Chap By selectively omitting some sections, al nine chapters of his book canbe covered ina oe-semeste course, For instance in burs targeted a ste ‘who end tee educational carers withthe master’s degree | ypcally omit Seutons 210, 31537, 48, 487, and 86, alongwith 3 Tew oler sectionsyore wing book on sis for which ter ex ave aad ben te tans teen taser nak sta cra ts bene se he ny Rt) CO) Hy 198 and ae (98D, books on ues fr ais, have Bled ein vaya ges to fomulte my tess on ras, rns tn hasten ue ay atta se hee Bok tat Ieee exes, Ait "Jade sent ad then ine ea age of my research eer. tet srr ete hye an a bo OP aur and Nendo (188), inthe wring of Caper 8, and Magnus (1988). . “ne Tra totnsk scet anonymous evo whofe may vey en genom and Mal ohor fr hi spp and eacouragement Gout sept am ds pal ioe aaron stents wba have seed © ee Satins cho tao eta dy igen osx prseading ended) tome sakes rea, a rad appre Dig tres fy ae Mt sin Senor CHAPTER ONE ‘A Review of Elementary Matrix Algebra 1, INTRODUCTION In this chapter we review some of the basic operation and fundamental proper I seer ute, Tn cates properie wil Be stated without Sout in some cases when instuctive, proofs wil be presented. We end chapter wit a brie dscmsion of random variables and random yest epee tus of rio yrabes, and some important dsiutions encour fered elewere inthe book. 2, DEFINITIONS AND NOTATION excep when sated otherwise a scalar such asa will represent are umber eee nA ok aze m Xn isthe mX m rectangular aay Of Salar ven by yan On 2) Souelines it uo wil be cone ‘and sometimes simply denied as & (aly m= aeit afer tothe (Jn element of A as (A hi is (hen Ain ealled a aquare marx of order m. An mx | a‘sallow column vero ply vec. The een refered oa he Ith component of a. A x mats i called a row veto. The ith row and th co of the mati A somtimes wil be refered toby (A) and (A), espe Uvely. We wil ually wo apa Hetlts Wy epson ules and Lowerene bold eters for vector. ‘Te diagonal element ofthe mx-m mat Aare ay032--\dnm Hall ther clement of Aare equal #0, Ai called a digonal mae and tan be Hen Vedas A = diag, .sdu)- I im ation oy = 1 for f= Iy..ym $0 tata diag. 1) the the maton A sealed the ety ma Of Order ‘mand will be writen at A'= Ty oe simply A = Tf the onder obvious ‘Ae dial.) ad bia scala, then we wil use ° i denote the diago- ‘mart with diagonal cements equal othe diagonal element of A and, for ay ssl Vector @ ora lower angola matin. An upper angular mati se which hs ll Fs element below the diagonal equal to 0, while a lower tangular matic hav all fi elements above the diagonal egal t 0, “The th slum of the m>m dents at ll be dnote by a hat isthe mT vector whit has ts th component equal © 1 and al of ts other ‘components equal 1 0. When Be value of m1 Pot Obvious, We vl Make ieimore exit by wrtng@, a8 e,m The m xm mati whose only nonzero clement | inthe (sh positon wil be deat as By. “The wear aro writ, wile vector of zo, alle ll vec, wl be dented by O, anda matrix of zeros, called a all matrix, wil be denoted by (0). The mI vector having each component equal to I wil be denoted cor imply Twhon the sae of the vetr i obvious 2 MATRIX ADDITION AND MULTIPLICATION ‘he sum of to matnces A and a dene they nave te sme pumber ot true an the same ter af ea, the eae Ae Beta r6i) “The product of scale and aman Ait nb A= Cony) “Te premukiplcaton of the marx B by the matin A is defied nly i the rnber of clans ofA eal the umber of vows of B. Ths fA ie mp Ind B is px m then C'~ AB willbe the m x n mars, which a is (jh se mAsrose 3 eae ey ven by anim, = maby ‘There is similar definition fr BA, th postmaipication of B by A. When bth products are defined we will nt have in genera, AB ~ BA. Ifthe mate ‘Ais sgn, then the preictA4, ors 42 ie tein In theca, fe have A? =A, then is sai oe an idempotent mati The following baie properes of mun atin and mliplication ar easy to veri Theorem 11. Le oad fhe salts and AR and Che matics The, when the operations involved are defined, the following popes hold. (o) A+B=B+A. ©) A+B)+C244B+0). (0) a{A+B)=aA+ a8. (os BA =a + a. @)AWASAGCA 4. THE TRANSPOSE, “The tanepose of an mm atic Ae the mm mnie A ose hy ine changin the ows and columns of A, Thus, the (ith elemeat of A’ isd IF [Avs an Bs pm then te kre of (AB) ean expec sh (Ah =D 8D ~ FWA = A “Thus evidently (ABY ~ BA. This along ith some oer ee nvaving the transpose ae summarized below. Theorem 1.2. Unt a fe clare sad A and Re ratrions Then, when defined, the following ol (9 (AY = 0X, our(6) (aA BBY = a’ 660 ) (aBy = BH As mer, hats 8a square max, then A$ 0 mm. In hs case if A=A then A iccalled seme mars while A celled a sew-symmetic matin fA = =A The tanspose of a com vector isa ww Yet, an a sone siuations we may write a matrix as a column veeoe tesa row vector. For insane the matrix Ey defined inthe previous tection can be expressed as Ey = ee. More tetas sun ite an mutes ving Ia He only naar ment, tn the (jh Oona WA is an mx mati then YY aeitin S. THE TRACE, ‘Te trace i a function tat is defined only on square mates IC is an mm mats, then the tage of A, denoted by tA) s defined tobe the eum of the Giagoal elements ofA; that is, wa Now if A is mm and Bis nm, them AB is mom and wane Soa S,asion $$. a8 3, Sh = Za -Y ooy=ecn ‘spp eta nh ne dn peg econ ‘Theorem 13. Leta be a salar and A and B be maces. Then, when the sppropite operations ate deine we have TE DETERMINANT . 60) WA + B)= WA) +0, (ed) tAB)= BA), (@) tiA’A) = 0 if and only fA = ©). 6, THE DETERMINANT “The determinant is another function defined on square matrices It Aas an mx ‘abn, ten Hs dexrinant denoted by [lis given by Lal = 1a ag rin LC teagan where the summation is taken over ll peMUtabons i...) wf ie wt wf Mess Cleves; and the fanction fl) equals the number of tras: Fosklonsnecasary to change (ye yim)t (hsm). A Wanspsition isthe Peattgaage of two ofthe integer, Although 8 wot unigoe itis wigs even eases that [ali uiguely defied. Note thatthe determinant proces ll frodvet of m ems of the elements ofthe marx A soch that exactly o0e Feats Selected from each row and each column of A ‘Ae stemative expression fr lA can be given in terms ofthe cofactors of 4 ‘The minor ofthe element a, dened by my, the determinant ofthe n= 11% Cr Dinu aed afer removing the throw and jt columa from A, The oa iter fa, dente by Ay then Siena y= (17° responding oa amino canbe oe oy exyadig sone fearon tal= Y avAin up cor expanding along the th eoluma, Seay ny ‘On the ots ban if the cofactors of # row or coum ae matched with the ements fom a diffrent row or column, the expansion reuces (00; thats Fd hen Fou D ato a3)6 | REVIEW OF ELEMENTARY MATRIX ALOESRA ‘We wil find the determinant of the 5x 5 marx given by ‘ning Whe ctacoreapusion formule on te fst claus of A, we aia jo 3 02 o1 12 lala sad then ning the same expansion formals on the ret colin ofthis 4 & mark we et 309 wea-en:[2 2 9 iid Since the determinant ofthe 3x 3 matrix above ie6 we have ll-2-@p-6--2 ‘The following properties ofthe determinant are fry srsihtforward to verify sng the defston of «determinant or the expansion forse given in (1) and (12), Theorem 14 Wve cal and A ian mm ati, then the following properties hold (a) U1 1A (0) Jal = 2A (©) IFA isa diagonal mais, then JA] = 2° ay = FI () Il elements of aso (o olan of A ae seve A 0 {@) If two rows (or columns of Aare proportional one another. [A = 0 (©) Theitrchange of two rows (or columns) ofA changes the sign of A. (g) Fall he element of a row (or column) of Aare malpid bya, then the determinant pid by (0) The determinant of A is unchanged whea a matpe of one tow (oo: uma) is added to anther row (oF clus). ‘Consider the m x m matrix C whose columas are given by the vectors heart that is we can write C= (c,.--.e)- Suppose ha or some vector BGT aay and mate A fa) oe have Seo8(Em)e Marah $o( See) Sone “a 0 Ghat dhe determinant of Cis Hina combination of m etrminans, 1 fan mx matrix and we now define C= AB, then by applying th derivation shove on each column of C we ind hat gly whee this ial sum i oly over ll permutations of (1,8 since Theorem Te) implies dat No for any jk Finally, oeering he columns in [yah5 1 REVIEW OF ELEMENTARY MATRIX ALGEBRA using Theorem 14(9, we have HL= J by bil ial. “This very wtf ee ¢ sma Below ‘Theorem 13. bo Aad 8 ae agus wales uf he sme xd sen (4B) =| 18 7. THE INVERSE ‘An x m mati A for which [A] 40s said tobe a nonsingular mati. In this ee mong mat ety Aa eal ee ANT AA= as ‘is vee sigue ins i Bis ante mm matic ashing he ive formula (1.4) for A, then BA = Iq, and so ‘ins Bas ta" ‘The ovina propre he mi re a ect verie utilizing (1.4). i" : Theorem 1.6, Wax a unceso waa A an ae sonsingla nm mates, hen to (wart ata, ar eany, aya, (a a= ar, (0) A Sasa, ok, (8) fA=A, thea 4 @ apr Ac withthe determinant of A the inverse of A. can be expressed in terms ‘ofthe coactrs of 4. Let Ay cled the aint fA, be the wanspose of the fons of eofatns of A, iat the Ge lament of Ay tA, the cofactor ‘He vent ° ef ay Ten (Au lA [Ate sine hd =A tn or (1a 2. ant Fn aT enon svete Ys be ass al lim atari 2 “IAA “The retaissip beeen the inveree of «matric point ad the product of the inverses, ive in Theorem 16() i very wseful property. Unforuttly ae ne cladonahip dos not exist beeen he inverse axa andths som oa. ‘Ue at near ve the following veut which somesimes ‘ful ‘Theorem 1.7. Suppose A and B are nonsingular maces, with A bing Imomand B being nme For any > ati € and say mx mars Di {allows that i» CBD is nonsingular hen (as eppy! =A a1" + DA*CY"DA ‘Prof, The pfoof simply involves verifying that (A+ CBDYA+CBD)= ly for (+ CBD)" given above. We have (as capa a's + DAC" DA") sly = 0(8"! # DAC) "DAT + CBDA™ "capae'cye! + DAC DAT ag C(W" +A)! By ADA! +A ICy DL t= C{B1@" + DANE» DALEY! BDA oty-c(@ BDA ~L o fm em and C and D te identity mates, ten we obtain de owing special cae of Theor 17 Corollary 1.7.1, Suppose that A, Hand A +B a snatices The as ‘We obtain anther special ease of Theorem 1.7 when m= 10 ‘a REMIEW of PLeMENTARY MaTRDK ALOEORA Corollary 1.7.2._Let A bean mm nonsingular marx. and da Se seco ad oi nosing hen set 2a ated tye da'e Example 1.2, Theor 17 can he paricurty uf when i lpr than andthe inverse of Al fai easy to compute For inane, suppose we have 8 from which we obtain G=AsceD: Ie is somewhat tedious to compute the inverse ofthis 5x 5 matrix drety However the salons fn Theoremn 17 a fly straightforunrd, Cleary, weow'ey-[ 3 renner [2% 8 Patino ATC n ‘Ths, we find that oh QW. Dwey'p 115-05 -25 33 1 4 =] 325 15 35 22 9 3 05-05 -15 8. PARTITIONED MATRICES Occatonally we wll dt usfl to partons given mati int submatces, Forinstanc, suppose is mx andthe postive integers mm, mm a sch {at m= msm and n= my +m. Then one Way of wring A 2's Prtitoned rman is acf4n 4 [ha Aa her A im 2a, A m9 Ay Xm and A mn Thal 5, Au is the matx consisting ofthe fst my rows and my solu of A, Are fs he main consisting of he fst my rows ad Hs my columns of A” and $0 on. Matrix operations can be expresied in terms of the erhatrce of the paritioned matin. For example, suppose Bis an nx p tai priioned sk hee Bim pi Bi sm x pa, Bay im * PBs m % Pan py + fe p. Than the promulpiation of By A sau Ue eapressed in pte form a8 [AnBi + AnB AuBoAndee [asi ras, Anis Anta DMatices can be paitond ino submatices in oer ways besides he 2x 2 aionig given above For instance, we coal partition ony the ean of Ayia te exrenionRn | REVIEW OF BLEMENTARY MATRIX ALGEBRA ASIA Ash where Ay fs mx and As 8 mm. A wore gens ston it one i whieh {cows of Awe porttoned ino 7 aroups and the columas of 4 sre parttoned inva e group so tat Aca Be writen 8 Ay Aso Ae fy Aa oA here the submatx Ay s mya he integers mio abd mye ee such that Deen at Dee “The matrix A above ssid wo be in block agonal form ifr =, Ay square Tras fr each i and Ay i ll ait for all ane for which 4-H is (ise we wl wee A= dag. Sat Ay OO An O agi Aone OO Ae Example 1, Suppose we wish compute the transpose product AA, where the 3% 3 mann A Is ven by 10 1 aloo a4 “The computation can be simplified by observing that A may be writen Tie KAN OF A MAR B ‘Asa result, we bave vareh 2, BY he B)-[ SER ahaa atin 3) [i Ge") =| ens enn ga22nt nome oye )/2 3244 wee test) |? 275 3 rita 9, THE RANK OF A MATRIX ‘Our ial defirtion of the rank of am mx mates A is given in tems of oa general, any tants fine by deleting rows ox cobs of A BeEIuE sakmande of A, The determinant of an rr submatrix of Ai led 1 Sonora order, For instance, for an m x. mak A, we nave PEVIOU) SRI Shae we cite the minor of yi an example fa mint of ore Now the rank ofa moanall mx mai A sr, weten rnk(A) = if ics ais minors of err rissomzero wile si ior of oder r+ 1 ic are any) are seo. HA ell max then rank) The rank of 2 mati A i unchanged by any ofthe following operation, called elementary transformations (a) The interchange of to rows (r columns) ofA {@) The mulation of 3 row (or cours) of A By «nonzero salar (0) Tie aidon of ealar makiple af 2 row (or cola) of A fo anther row (or column) of A “Any elementary tensormotion of A can be expres athe mutton of dU) sm fered to as an elemenartansforation mai AM leer 24 ee ssormaon ore rw of Awl be given by the premio of A eermnary wantormaion matt, wl a elementary transformation 4 as coesponds 10a posmulpiaon. Eleenary eesformatve oa cocina and any nonsingular mat canbe expressed the rock of elementary tanfomation matrices. Consoquety, w= have the fl Towing very weful esl ‘Theorem 1. Let bean mx mati, B an mx m mati, and Can "2 matin, Than if B and ae nonsingular nes fellows that ank(BAC) = rank(BA) = rnk(AC) = anki)“ A REVIEW OF ELEMENTARY MATRIX ALGEBRA By sing clementagy trnaformaion sates, any mati A can be tne formed to another mats of simpler form having the same rank a8 A. ‘Theorem 1.8. 6A san min matin of tak 70, then ther exit nosine ular mim and nx matices B and C, soc that H = BAC and A= BHC", ‘where 5 piven by OL O if o[g] monn off 2 “Te following isan immediate consequence of Theotem 1.9. >0. Then here ank(@) =r @L it men, Corollary 1.9.1. Let Abe an mn mati with rata) exist an mir mata Fad an cm mate such Dat ak Sua" FG. 10, ORTHOGONAL MATRICES ‘Ana vector eat be «normalized vector of awit vectr if = ‘The mx vet yp when om are sid be etapa p/p =O forall) ifm aon ach, io a nomaiaed veto hen be veto ‘Bide batho. An mm ate Peto sama form anton “tot vectos allan eopoa at ned Tolows ht pepe “Ting the determinant of both des, we Se that lerPl=\P'|1Pl= iP ‘Thus [P| ~ 11 or 1, 20 that Pie nonsingular, Po! = BY, and PPP = Tin tudtion to P'P = I tha is the rows of also form an orthonormal st of rnb sectors. Some base prope ot orthogonal mates fe Semmes the following theoeem. Theorem 140. Lot P and Q be mm thogonl matines ant A be any mcm mati, Then 1 (©) [PAP = Al, (©) PQ isan onhogonal msi ee TPR ‘quapeaic roms 1s ‘An m2 aauia Ps called pecan sain If ea sow ad ea column of P his a single element, while all the remaining element ae zeros. AS real the courant of Pw bee. the colimns Of ly in some order. Note then that the (hth clement of BP wil be ee, ~"Y for some f and the (hth element of P’P wil be (¢, = O for sme | 4 j Wh thats, 2 permutauon marx isa spect erbogonal mai Since there ae ml ways of permiting thr lone of. there a mt iferet per rmataton mates of order fA is alo m % m, then PA creates an mm Sai ype ow of AP psc ai ty ping 11, QUADRATIC FORMS etx be an mx I vector. an 9x 1 vector and A an ae matin, Then the function of x and y given by ea-3 Sms soni le ier fm nad. We wl mot irs in the special casein which m= ro that Ai 8m andy In the sa, he function above redvees tothe Fancion of x, eS sn, fas wg is alle a quate or iA rere a he ma of itis not, A may be replaced by B= S(A'+A), whichis symmetri, without slering a) that is, neh eaen’) vm he asae 1 icareataty)! 7 JO Ae ewA's) =} fae An aaa since 2A = (A w/a. "Even symmete matnx A and ts asocited quadratic form i clased into ‘one af the fllowing five eateries. 0 IA #0 far alk £0, hen Ais postive definite (0) IFAx 20 forall 40 sndx’Ax = 0 for some £10, then Ais positive seaideiite,16 | REVIEW OF ELEMENTARY MATREKALOEBRA (tev Ar<0 to all £40, hn Ain wegative defi (a) trate 2 foe all 4 and Ax = 0 for Some x 40, then A is negative semideint (oan > 0 for some x and Ax <0 fa san yn A det. ‘Not that the ll maxis actully both postive semiefinite and negate ‘emit Prt ifn and negative definite matrices ae nonsingular, wheres pos iting Somidefinite an negative semiefinte matnces are sngus. Sometines ie ee dtc wil be no enero a syste matrix th 1 fot poate definite or postive semifinte. An mx me mat Bis called ister mie en pacman nic aD Sones iF done onrh = mati B a A. If Bi alo symmetic 0 tht. then Bis ale the symmetic quae oot of A tne forms yj a prominent ol i inferential aii. In Chap 9, sve uit vel tome of he most important results involving quadratic forms {har me of patclar terest in statistics. 12, COMPLEX MATRICES “Theoughout this ete text we wil be dealing with the analysis of vetrs and eae ect lel numbers or arales However there ae ocasons search an payaso real atx, sach asthe decomposition of mat i Fae Sra prado tier macs, eas 10 matrices tat conan complex (Minbors Por te acon, wll ely summarize i tis section some of the te soation and terminology regarding complex oumbers “any comple numer cca Ue wate ne Frm e-atib, where a and bar eal numbers and represents the imaginary number V1 Pee arbor ais called te ral pat ofc, wile D seated to as the eeeant me re Te, the mbes cis eal umber ony if is 0 If me Hee cemplex numbers, c= ai # ib) and = a2 +a then their im 8 geen ty exten (ay 4) +b #20), oiler pc given by exer an Lib ands 226) Cconesponding to each complex number ¢= + (bs another complex mamber ‘COMPLEX MATRICES " noted by @ and called the complex conjugate ofc. The complex conjugate eer pea by Zsa 10 and sxnen c= + Uy so that the pros of = cept number ant its conjugate results ina real number. “empl mumber can be represented geometrically By a pout te com- ee pla ie one ue sce th roa si ad thee ai isthe Pimple or mina axis, Tus, he complex number c= a+ ib, woud be ‘Gyrsemd by te pont (8) im ths complex plane. Actes we a8 arama ortnaes (78), whore rue lena ofthe ine from the ot ‘Sy ee poi (a,b) and 0s the angle Between thie line and the, poste Baran Teal axis Ue relay Between snd & and rand @ is thon geen hy cosa, be rand ‘tng cin tems of the polar coordinates, we have = eon 8 +i x: fer using Euler's formula, simply ¢= re, The absolute value, lo some- Te seed ihe modulus of te wmplex umber ¢i defined to ber. THs is. Pour always nonnegative real number, and since a+ 6” =r we have el = lav ible Veo We alo find that leyeal = VGaran DiB Garba ab = Vee e+ =a le ‘sing te vent above repel, we io hat or any compe saber and any psi inet mel = 1 "Raffa vlan x comple umber and sts conjugate we ab me vale of cis “Arlying this othe sum of complex numbers 1+ an ming ‘pBher saad we set8 REVIEW of LEMESTARY MATRIX ALOEDRA levee =e eenlerval= ler tenk@s #29) 21 +eita +e + ef Sle? +2lel lel +f? lal + lead? From tis we ge the important neat ly +l e+ less known a the langle inequality, ‘Neomplex mati ie simply a mati whose elerent re complex ubere ‘Ac anita complex mati canbe writen as te sum ofa real mati and ‘3 mapinary marx ta C6 an mx m eomplex eA he Han De xpress as Coan, shore both A and B ace mx real maces. The complex conjugate of C, ‘noid Cis simply the eatin containing the compl conjugates ofthe ee tens of C; that, <8 The cope taeps of Cis C¥ = the caper mat Ci aque ns Elam 5. ben cm said to be Herman ete tC Hein and C4 na ain tee C's aymeie The m> m matt © iS tbe utr 'C*C'= Thi he pnerizaton of comet ‘hitless ea an C= 12, RANDOM VECTORS AND SOME RELATED STATISTICAL CONCEPTS: In this ection, we envio ome of the hae defintione and ment in ie tbtion theo which wil be needed ltr inthis text. A more comprehensive ‘temmen of this subject ean be found ia books ow sais Dey sch os (Casela and Berger (1980) or Lindgren (1993). To be consitem with or nota thon, which uses a captal Iter to dene 2 matin, a bold lowercase leer fora vecen and Jowsrease Iter fora sla ve wl ase a lomereace let ter instead of te more conventional capital eter to deaote a sclar random ‘arible. ‘Arodoe vrishle is i tn he net fit collection of posible values ist countable set In this case, x asa probability function p()saisying pl) = Poe =, fore Ry at p= 0, for € Ry. A continuous exndom ‘isle the other hand has fo is ange Ran uncountbly infinite st [RANDOM VECION AND SOME HEL ATED STATISTICAL CONCEPTS » Associated with each continuous random variable x isa density fncion f(t) saying (0 0, for © Rand J0)= 0, tort & Re Probables fr ¥ we ‘obtained by integration: i is a set of thera ine then roe aie f now For both dice and continous x, we have ce R= 1 ‘tne expectes vale of areal unetion of), gves the ave observe ae fon) Phe nepran ng tes Me aera staan Sioa, if isso and sof atorane {ts contiuows. Properties of the expectation ope fallow diet from ropes of un and negra Por astance and are rondo waka Sn and ae constants, then he expectation arte sas te properties, Eta)= a, Elagy (s+ (991 = aFlesa) + 69), sete ad ae may cava uneons Ihe eof expected ales of ao aa phen by i = 12 at he ment of 1 "Thee ae imporaat cine and theoreti purposes. The fst feu mews an be ued to desde ssn eaten of dios Fo insane, the fst moment or mean of, = tn) loa ena vac of we tibaon te vanance of, denod oor vas defined ss oF vat) = Ble w= Bte) 92, ‘0 that itis a ncton of he st and second moments of x The vane gives ‘measure ofthe disersion ofthe observe values of + about the seal luc2» |, REVIEW OF ELEMENTARY MATRIX ALOEBRA sna Using propectes of expectation, i easily verified that vasta+ B= Bars) ‘of the moment of a random virile «are imbedded in a fnction called ‘he moment generating fanction of Tas Tunchon 8 Gene ay a pute Sms tye ane rin sno mh % mite), provided this expectation exists fr values oft Ainete momen gnc Semin fonction of des exists then we ea obain any momeat fom i ie eighborhood of 0. Ome Smal, oe) More imgorandy te moment generating function characte the distibaton tr in tat noo ifleentGsmbtions ave he sae aanmene geoeetog, PFE focus on two particular families of distributions that we will nop ner se Asano variable «sad o have» nora se ‘etter nth mean and nrance neat By x ~ NG a") the density ols given by o- } WO- pe, “The conmeping moment generating funtion is mgs ere ‘paca member of tis amily of normal istebutins is the standard normal dination NO. 1). The importance of ths distribution follows fom the fact that t Ngso*) den he salascing eansformaton = (e—p0/0 yields a ‘nor vatiabies which as he Standard normal dsrbaton. By dereniating ‘he moment generating function of ¢ ~ NC, 1) 1 easy 10 very that te fs ix moments which we wll ned Tater chapter ore, 1,0, 3.0, and 15, respectively ees pode intege, then a random vanable his x cirausie ds RANDOM VECTUKS AND SOME RELATED STATICNEAE CONCEITS u ution with degrees of feedom, writen =x fs density fonction ote way 0" aT” see 1/2) ihe gana fntion ene at 2 The momen ener Heer Ei pees by mine = 2° for. The mporace of he fection ot ne roma omen to somal dion SEN ten tne? Puteri, independent andor vals Sih NO, I) for = Tyne en xe “The chi-squared distribution mentioned above is sometimes feted 0 8 Fee \ *)‘The importance of this dsrbution arses from the fat that if wy andy ae ‘dependent rndow variable ih fj and 8s - xh then te alo ‘as the F ditribtion with r and rs depress of feedom. “The conseptof «rand vatable ca be xine Wal of «rar ves tor A sequence of rete random varibles myx ie modeled by jin. ex maivarite probability faction, pif al ofthe dom vals we is Pare a) [oJrmea ‘wile the expected aloe ofthe real-valued finction a) of is given by suff sr ay “The ran variables... a ad tobe independent,» concept we have lready refered o,f and oly ifthe joint probability funetion ot density fone “ion fats ito the prot of the magn pabubiiy or density fotos, that sin he continuous ese x... af Independent and only if Sa -ff0)- Fags forall “The mean yet of 4 dente by pis Ue vee uf expect values ue (is! = BY = UBL A measre ofthe liner eationhip between x and. given by the covariance (of and, which is denoted cows, x) oF oy and is defined by y= cor) = Xt ~ wy ~ ANUUB VECIONS AND SOME KELL SHAISHICAL-CONEREIS » ‘When j= j this covariance reduces tothe varitnce ofthat i, oy = of = asta). When 24 j and 4) andy we lnepeaent dhe carla) = © save team) = in. Ie, 3, By and are sales, then eons + B+ ay a eO¥ 3) ‘The marx. which has oy a its (eh clement is called the variance-covari- ance mati, of simply the covariance ratix, of x. This matrix willbe also encted sometines by vats) evt42)- Clay oy ~ oy 9 at 2 Isa ye ‘merc mats, Using (1.7) we obtain the mat formulation foe 2, > vant) = Ble — wr w= Bear’) ww Wari an mt vector of constants and we define the random variable y= a, then wenct(S or) -S aten- Sa 1, in ado, is another mx veto of constants and w =x hen wir toeie( San$ se) ¥ asoortn.s1= 3 ¥ aes ca it omy. In parca, vel) cou. 3) — We Sine thi de For any eine of snl ice he vara anys sone, QS Ne A KORBAIE Dee-apes More seer fA ps 1 mann oP caesar SEBO AE then By) = BUA) = AB(e) = An ay vari) = BL(y~ BLP) EL9)'] = ENA AAs ~ Ag)on A REVIEW OF ELEMENTARY MATRIX ALGEBRA ‘nus, the mea veto aed ovate at of the transformed vector Ax it [Ajeand AGA Ifo and aze random vectors, then the max of covariances fttween components of © and compoaents of W 8 even By In partial, p= Ax and w= Be, en ovlegm) = Acoviea)B =A varie) = ADB ‘A measure of he tvcat Winship between sand a hat anaffoeted hy the measurement scales of x and, clled the corelton coefficient, (sfined by ye emt) 9 Fear vartgy— Jents when ~ jspy =! The coslation mat P. wich has 9 ats (th lerent For any m4 Yee a, we fave ef Par= a5" OD5!"a= BOB, here B = Doe, and 50. must be nuuegative dfn Decsze O ite mal i ie the (corn ofthe mx Mety mai, then le+eyPete) (Py shy Per ey 2 ree. (ei ey Pee 60 Phy Phy Oey = id>9. {oom which we obtain the inequabiy. -1 aml 2 Since Ele) = 0 and var) = hy tis partials mincssional malate wo Sa eion Ie domed 2s No, lq) If isan mx 1 vector of constants Te Pip gaem aoninglar mary hen x = ps 125 said 10 five he aa oa ulnar norma isebuion with moan vce wand Cova ream De TT" This i indiaed by x ~ Na(ysQ) For instance if wee ihe vector 2 = (et has a bivariate norma sation stl em ip ladaced ty the eanformation = p+ 7, cn be shown t0 be6 | REVIEW OF ELEMENTARY MATE At ea {acer foe all x € , whver 9 = pig ic the combation coefcent When p = 0, "his density factors into the product of the marginal deste, x0 x; and £3 sie lndejemlca apd only i p=, The txt cumbersome looking density faneton piven in (1-11) can be more conveniently exoressed ying mare oan, Is saigtforward to verify tht this deny iental to Sa) mire teween} un “The densi fnction of an m-vanate ngemal random vector i very similar to ‘he function given in (1.12) If ~ Na), then dency ie 1 1° Gerry forall Re TE is postive semidett, then x ~ Na.) is id to have a ig sora detbtion In this acs Q" doe tl eto and othe aves ‘eral dest cannot be writen in he frm given in (L13). However he fandom vetorx can sl exes im terme of independent andar poral fandom variables Suppose ha ak(0) =r at cn mr ma ing G2 a Then = ~ Nap) if is itbited the wane oo yes Ue whee rows Na@i ‘he imparan propery ofthe mlivarite nora dstibaton that cattiastomation of muerte noma veto yells ulate Sel Sache ht iF = Nn 0) and Aten mae ants ee Tas avai nema sun. npr rom (18) snd (19) we know Oily Naga). ‘Gar of Te mot willy used procedure in ities segreion analy sis, We wll ie esrb ths tals ee sn subsequently ne repression Sats oils some ofthe sala nao evlope inne Some god references om regression are Neter,Wasseman, and Kotnt (195) and Sen and Srivastava (1990) Inthe piel regen polem, one wishes Sy the relomaip heron some tepose waa tyy, and eaplane, {or variables. Fo insta, y might Be the el of Some rd of [RaNnoM VECTORS AY SOME RELATED TareTCaL COMETS a ssmmutacunng proces, while the explanatory variables ae conditions fect ing the prodcton process, auch ar tempest, rem, an om ‘Aro lating thes 10 yi given by Posfircs-4 Boxes 8) wer By... are unknown parameters and «is random ent, ht i & random viable, with EG) ~ 0. In what i known as unnuy leat ues Fearsson, we also have the errs being independent random arabes nih ‘un variance 0; that and at rabdom errs aseoited wie Tesponses yan then vate) = wart) = an on ven in (114) san example of linear model since its liner fancon of he Parameters, It med not be linear ia the au ta, ft isance, We mig ave 422 af Since the parameters are unknown, they must be estimated and this Wi be posable if we have some obrerved wales of 9 andthe conesponding 4h Thus for the th obsertion suppove tht the explanatory vanscs oe ett the vale ok yielding the response, and thi Is done for aN where > +1 ede! (1.14) olds, ten We should have, appro imately, DB Bn 14 Bake foreach This canbe writen asthe matrix equation rx on * B) fifas (One method of estimating thes, which we will discuss from time to tne in hie tent sell the cto of least wuten IB = (Brecht an estimate ofthe parameter vector B, then j= XB the vector of fied values hile $= gives the vector of errs of deviations of he scunl responce row the comesponding fied vate, nd 1@)= (9X80 ~X6) ‘eS the sum of squares ofthese eros, The method of lest squares west2” | REVIEW OF ELEMENTARY MATRIX ALOEBRA “eany vstr that minimizes the fnetion fi) We wil se Inter that any ‘Sch sector satisfies the system of linear eaiatos, sometimes refered fo 38 the ormal equations, xx xy 1 sf column tht can(X) = then ORY exit and 90 the least squares eximstr of Bi unique and is given by b-comey 1. Prove Theorem 1c) that i A 8 an mma, sow da AA) = 0 itand nly FAO 2. Show that fx andy are mx 1 vets, ay) —29. Show tha FA and B Se mem matices and B is nonsingular, (BAB!) = WA). 4 Show that any square mtx can be writen a te sum of &symameuic 5, Define he mx m mates, A,B, and Cas woe oS] f 4 og fn en oe | on 2a) Oat os ta 6 ©. Verify the esas of Theorem 1), Consider the 4% 4 matrix roatens » 120d o 120 1230 ot] rind the detennant of A by ating the cofactor expansion formula onthe first coma ofA 5 Using the ati A fiom the previous problem, veify equation (13) when ‘ad teen bea variable and consider the determinant of ~My, where Asan Tt as fncion off WAAL tye of Furation oF ee? 10. ind the adjoin max ofthe max A given in Problem 7. Use ths © bain the averse of 4 11, Using elementary wansforatons, determine mates Band Co a ing ct Tacs piven in Probl 7. Use B ad Cto comput ie aac tha take the inverse of bo sides ofthe equation BAC = Is tnd then solve for A" 12. Show tat the detrnat of anole mati isthe product fs dag Sin ta ton, show ttc iverse ofa owerWangulr MAT The loves tong mai. 13 Leta and b be mx 1 vectors and D be an mx diagonal ti Use cae a Pan toad an expression fo the inverse of D+ aa, where isa sean 14, Consider the mm partion mati. an @ Alan and here the mm, mat Ayan the mma ua Ay ace nog Ra aprenon for A! ia terms of An Ar, and A 15. Let0 sree Au sm Xm Ag 18 ma X may AA my Xm. Show tha HCA is postive definite hem Ay and Agr ae alo posve dete, Fin Ge mk of es mae 2 01-1 rai bor2 0 200-2, i i ye pone ing the form piven in Theorem 1.9. Conseasely. ‘ermine matnces and C 49 BACH Prove pats (b) and (c) of Theorem 1.10. [322 aM 44 ve Px Pa 2] Fi aus fh hy and 70 tat Pian ogo mai f yor Suppose the mx m onbogonal mati P is paitioned as P= {Py tee xin mm and mm =m Show tat FP PR RIE Tce Bd Ply] {et Alea mx mma an suppone thee xis eal AX mR T ‘sich thal 4” Show that A must he nonoegsoe define AIST Ta) 30 a Prove tht a nounegative dette matrix mast hive nonnegative diagonal elements; hats show tf asymette mat hac a) apse Sgr ements then iis not nonnegative definite. Show tat the onverse fs Not tro: hats, fds ymmctre mat that has ugav Jaga ema but snot soonegaive define LetA bean mcm nanneatve definite mati, while Bi an mm mattis Show that BAB" is @ nonnegative definite mat, | montanes o 25, Use the standard normal moment-geerting funtion, m,(0)= & to show that the Get six moments ofthe standard normal dition ace 0, 1,9, 3,0, and 15, 26, Ulu properties of expectation how tht fr random vail r anv and sears a, 2, Bi, and Be cov(ay + 81,03 + Bata) = BiB cov.) N42), whsse 27. Suppore x (6) Find the corrlition mats of. (b) Determine the isbution of w = Ix (6) Determine the distibution of = Az. (© Determine te astibation of [= (© Whi ay edits tain in nd we in macs 28, Suppose xis an mx {random vector with mean vestry and covariance mathe tI As an nm matic of constants and e ie an mI vector oF Constants pve expeesins for @ Hae re () vatate +eCHAPTER TWO Vector Spaces Lo INrRODUC HUN In statics, observations typically tke the form of vetors of valves afferent vara tor coon foreach sbject ina sample, one might record hei ‘WEight ae, and soon In estimation and hypotheses testing situations, we ae CREIy Mckned in inferences regarding 2 Yecwr of puncte. As 6 rem, {be tops of ths chapter vector spact, has important applications statics, Tn aldtion, te concept of linearly dependent and dependent vectors, which ese Seaton 5 vey veal he understanding and determination tf he rank of 2 atx 2. DEFINITIONS ‘A vector space it collection of vectors tat satisfies some special properties {h panelr, tbe collection v closed under the ae of veces nd under ation nf» wetne hy scala Definiis 2,Lat $e 5 coletion of m | vectors evn the fol. lowing () Weve Sond ny 6 S,thenx 435 € 5S. (@) fae S and cis any real scala, then a 6 S. “Then $e called a vector space in dimensional pace. If Ss a subset of T, “ohh is aother vector space in mdimensioal spe, then 38 called ecm UiShae uf 7 Ths wil be indised by wring ST “The choice of =U im Denon 2.) nies tha se walk veto 0 $5 that sexy vac space rot contain the ull vector. In fact, these = (0), “oussing oft ul ecto only, eel a vector space. Not so tha he £40 ‘Saat a) and (0) are einen the ane condition hat aye i 5, ermians B sy 5, and ey al eye any sears, thn (ot mtn) € STi can FE Sasy generalized to more can wo, sa m, vector; hati if ay. a Peiealfe andi vstors suc aC 3, € 5 lea for Sto be 3 ecto space ae mat ave Yanes en ‘The lefehand side of (2.1) is ctled a linear combination ot the vey Fe ee ot chor spe i led unde the famaton of near comb “ion vector spaces ate sometimes also refered tas linear spas Example 2.1. Consider the ses of vectors given by (ahar bi: 0 1. Yet another example of estar nom swash nity n0mm teint nm, i sven y Bate = mag al2 voto seaces Although we have been confining attention 1 eal vectors. the noe defined shoveals serve a norms frcomplex vectors, However, inthis case, the absolute ‘alas appearing inthe expression fot fay ate mvenaty even heap even. In tcl the Evclidean nor, valid for complex ae well a eal vector is 3. LINEAR INDEPENDENCE AND DEPENDENCE We have seen that the formation of linear combinations of vectors is fun- amenal operation of vector spaces. This operation is what exalishes link beunen spinning stad it vector space. In many situations, or invests tio of vector space canbe reduced simply to an investigation of spaming set formar vce space Inti cas, wil be advantageous To rake the panting Set as small as posible. In order to doth. efit necesy to tnderstand the concepts of linea Independence and liner dependence. Definition 26. The st of m1 vectors f,..¥) ie said tobe lineal independent set ifthe only Slatin othe eit Yoxeo ss given by ay » = = ay = 0. I thee are oer solutions, the set is called & Tiwary det ct [Example 2. Conse the thee vector y= (111/89 =(1,0)-1Y and 22 G2 IV. To determine whether ther vector at line’ independent. solve the system of equations aia) an + ayn) 0 of equivalently, arta +3an=0, a +203 ‘These equations yield the contin a: any scalar a, 4 solution wil be given by a Sa, ay =~0.5a, and ‘othe vectors ae lineal dependent. On the ot and any pat ofthe tice ‘tors ae linearly independent thats, (3a), 8,23} and 5) each = nearly independent eof Yor, LUNEAR INDEPENDENCE AND DEPENDENCE w ‘The proof ofthe following eas we Belt de reader Theorem 23. Let (21...) deasetof mX | vectors Thea the following statements hol (@) The sets hearty dependent if the nll vector sin the set. (0) this et of west Hineary depen way wowing subst of 8 also lnesy independent (6) eth se of vectors is inary dependent, any oter set containing this set ne eset ale inary dependent ‘Sore mar im Denton 26, Af m= 1, that i there is only one vector inthe ‘at then th ets liearly independent lee tha ecto Hem 2s the st linearly independent unless one of the vectors isthe ll vector, o each vector fs nonzero scalar multiple uf dhe cter vector that aS of two Nett linearly dependent if and ony if at est one ofthe vectors is a sealer mutile ‘ofthe othe. In geaeral, we have the following. Theorem 24. ‘The set of mx 1 vectors (),.-s) where n> I is i erly dependent set if anc ony atlas oneveor inthe set canbe expressed 88 linear combination of the remaining vector Proof. The cet is ubvou ine of me vectors the et thee vector since then the set mast be linearly dependent, andthe m x 1 ill vector linear combination of any staf m1 vectors Now rome the St docs ot Incl he ml vector. Ft suppers one of the vector yay va be eNpeed 8 linear combination ofthe others; thats, We can find scalars i. eh that y= uy ++ Bu hs implies hat Lon ff we define ap — yo the vecio ,--te ae iealy dependent. Con ersely suppose that he veetor 2.» ae linet dependat so that (2) holds Yor some choice of a... ay, wih af at ont ofthe, 58) ay 000 eal io rm Ts, can cove (3) fray in which ease ne ge 2 E(2)» 0 that x i liner combination of 2... 4 Thi completes the prot o” WeCTUR SPACES ‘We ed this section by proving to ational ests that we wil ned later ‘oe tha the it uf nse Wott llnough sated la terms of he oles (ofa oti, applies aswell 0 the rows ofa mats ‘hcrem 2.5. Conder the mm notre X with alma |x| 40 iF and only if the vectors. are linearly independent. Prof IK] = 0, then rnktX) = F< m and 0 it follows from Theorem 1.9 tat there ae onsinglar m Xm maces U and V'= (Vy Vale mith Vi Lo rwey[ Ot sun ttn pe cn ie obit ate act Use fe ein 2 ct a ln sei en ert ge care a ae Se Leet aan Oo SOE ache eae ank(X) «anki U)=rank((W)'Sm—1 CConsequny iX] 40, s tha eal) =m then ta mst e inealy independent a hear 26. These yout) of mx 1 vectrs is Tinerly dependent fam Proof Consider the subset of vectors (2,.--.tq) I this is tnealy epedent sett follows Geom Theorem 2.36) that 501 the 30 (y-- Fo) ‘Ths te poof wil be compl f we cam eho that when rt ae ine Carly indepenien. then one of he ober vectrs, 52 Xn AN Be expressed a8 ‘Minar combination of. 9. WBeR "yf fe ily indepen, Fotos rom he prvi there hat if we define X as the mm mati with Thon ote ait columns, then (| #0 and so XT exists. Leta = Xan ‘nk ha 0 wens nyt Oy which cave the thcorcm ily Be de to Theorem 23). Tha we ave Saas and 50 the Set (Ro tng 4) ANE Bence ASO Che 9 GoeoAn) I Hinaly Sepenteat a BASES Ato DBUENSION 4 4. BASES AND DIMENSION “The concept of dimension isa familiar one from geometry. For example, we Feopuie' line ass oneinvaiond gion and plan 2 2 eo dimensiont ‘olan thi seton we teeralie this nasi o ay vector space. The dimen ‘ion ot vector space ean be determined by folang at spanning se for hat sera apace. te pices eed to he able of the minima numberof ‘recor necesary for spanning St Definition 27. Let 'S. This sets ealled a basis of. vues ae linearly independent ste) bea et of m1 vetoes veer spare i spans the veto space Sand the vectors vers vector soace, cxcet the vestor space consisting ony ofthe ml vetor (has basis. Although a basi fora veto space i nt uniuely denned, ne umber of yecor in baie wique and this x what gives us the dimension of a vector space Defintion 2.8. If the vector space 5 i (0), then the dimension ofS. eae by din) i defines to be are. Otherwise te dimension ofthe estos Space $i the numberof vector in any basis for 5 Example 24, Consider the st of m x 1 yeetrs (ei... where for cach defined to be the Vector whose only nonzero Component th ith ‘Smpceent wtih eons, Now he Lnear combination of the 6 Fees tenne will equal @ aly if Independent Alea, i= (2 = 0, 30 he vx ey-evty a Healy “Ey is am abitary veto in Ry then Sewn so tat (ent) spas A. Ths (e:--s6n) i a basis forthe mien onl space and in fat any neat independent set of m mx 1 vectors ‘Mb 8 Basi for For instanoe i them >t veto has 8 a eon esc to one wile th eat at all eo, then o,f) ie also a basis one,a ‘vector seaces Baample 25. Cuusides the vec space $ speed by he vestry = (eleteas © (L0.-1)5 tad ay = G,2.1Y. We saw in Example 23, that (si,knas) isa linearly dependent Set of Vectors so tht tis set nota bis for However, the set (sons) i lnc independent and = 23) 1829 ha (sts) and (ej. mst span the same vector space, Ths, (3) 15a bans for and so 916 a two-dmensona subspace that 2 plane In Spf inert indepenont verne nw hace fo for example {133} an (ras) ar als bases for S. vey vcr i vectors can becca coinaton this i pot the ese when the spanning set sa bass Theorem 27. Supe the set of m1 ects (xh, ta) 8 ae for ‘he vector space S. Then any vector ¥ Shas a unigue representation 2s "ined cominion ofthe vetors Proof Since (ei..-au) spans $ and x © S, thre must exist salars eye sh hat dan “Ts, wr only nee to prove that the rapeeseniation shove it onique. Suppose itis not unique so thee exists anothe sof Seals B.- By for which wSon But this then implies that Site toee Saar S aneres Sine 4.4) 1.8 bass the vetoes. US be nearly independent snd the ation ahs feted oly a, = forall Pee ie ave = Bi for! I,m and so the representation is unigue o Some additonal usefil results reaarding vector spaces and thee bases are summarized below: The proofs ae eft othe reader. amo RANE AND LBSEAR BDRRERDENCE “ Theorem 28 (@) Any two bases ofa vector space § must have the sme numberof vector. (0) If xy. opt) i sof Lncatly independent vectors in ayeetor space S and the dimension of $2 m thea fay svat) ta baat (6) the et (ry..t) sans the vector spice Sand the dimension ofS ism then the set {f,..-te) ust be linanly independent and this a Ite for (@) It the vector space Shas dimension n and he st of linear independent weston (Fire) i in Ss where r on nse (Or nex result gives useful inequality forthe rank of the produto thre Theorem 212. LAA, 8, s9d C6 prim xn and me mates apes tively. Then * = sets ana re -ro Pr om Tee 2146) ma([ ])2mmanemwo as sine La lls Ne ty ¢ ol -LF Mlle Stell €]: hr ley esta st ie eights re neasigue, ‘we must also have " mens wou([ie tol) mel Ste] Combining and (24) we btsn the dese ces ‘A special case of Theorem 2.12 is obtained when n= m and B is the mx Wentiy mats. The cating inequality gives «lower bound forthe tak ot ‘matrix product complementing the upper bound given in Theorem 2.108, Cortlry 2.12.1. EA eam msm tis and Ban np i,t nk) 2 rank) + rank(B)— ns ‘vector SPACES 6 ORTHONORMAL BASES AND PROJECTIONS iach vctorin a bass fra vector pace Ss orthogonal to evry ote vet rea ee stm in vale am otagansl hase Ta tis case the vectors vratne viewed as et of coordinate axes forthe vector space S. We wil find ‘PeyBil to w have each veetot nour basis sale uit eat in which tence would have an oethonnel ass “Suppor the set lars) forms a bss for the vector space, and we iaeee tits an orhoormal bean fut 5. Unss r ~ 1 am ethanol Ce Shite so there are many diferent orthonomal bases that we can sees lc One tchod ofeaining an omhonormal aks tom = given bess Ser Od Cram Schmit onbonrrmalonton, Fst we const the Sein etegonal veers ven by yen en By eee yn 2M Yeo apo Pky, PE as) yin” Sra 8) oh ef entonoma ve fee whe fr ech one Noe thatthe Hina independence of xs. guarantees the ines indepen ence of yee ade Thus we ave he fling esl. ‘Theorem 213. Every rfimensional sector space, exept the terodimen sont space (0) as an orbonorma ms. ley. 4.) ies fais forthe vet space $ and x ¢ She fom Theorem 1270S no thas an be uniques expessed i the tome = a8" Reena uudenormal ba, each of he ese ny BAS 8 vi Lito fms pemulipicadon of te equation for x above by 2 yields the ety a= 2 ‘Peample 27. We wll nd an orthonormal basis forthe three-dimensonsl vector spre § which ha a basis (3,220), Whee ‘ORTHONORMAL BASES AND PROJECTIONS ” i]. lil wey pt [a] fs S| 28\| = spl Normalizing these vector ys the orthonormal has (25.23), where 1 we 12 ae 2 2 | "lin | yap 8°] -u2 12 17, cA “Ths, foc any x€ S,2= ait tate ast, where a= 2% Fristoe sce Tn ages e Dotty = 2 we have ns = 36,126 +28. Now if $ sa vector subspace of RY and x © 1 he following inlets roc esas scan be desomnpesed ino the um of vector in Sand another Theorem 2.14 Vet (ejn-te) be an orthanomal basis for som vector sabepces 3 of R Then echt RP can be expressed uniquely 8 where we $ and w is vector tats onogonl to every vector iS. ‘Prof, itons fom Theorem? (8) that wn ca nd VECO 2 so that the 61 yee) is ab ortonoral bass forthe m-linensionl ‘Bhctukan spice Ri ago fol from theorem 27 da dee is 3 eigne et fsa ont 500 tha” ecton sence any #24 ty ad B= aye #294 igs ME VE, feat Sand eal be omhogomal o very veto iS due 3 ‘Thus, if we Het angel to the ombogonaity ofthe vectors 2.24 ‘Ts vectra the theorem above is known a the enon peeton fom S When m= 3, te oxhogonsl projection has a simple geomet GeESptba nat aos tor mivaatn 1 or ne, ep in me Sinema pace and Sis a twodimensioal subspace then the othogotal ein ag moet of De Sn ow Than PARGBERED o's ana poses through = “Tuccmporance ofthe othopoal projection 0 many applications arses cut fais esis sat ane pm sen gs te octane Tanton hen an) 5 (9). Te iy tml very, Since w andy aren fallow fom he desomposton POU'B im te secwe wy b ogoal t 8 =v ad hoes tay) 0. Consent, (= y)= (ew) +--+) We wee +206 wye—D) oe mem la =p) = (ACE P+ COUP, from which da) $6.9) follows since (a y)) 20 vample 2. Simple linear rearssion lates a response variable t0 one cexplanatry variable Ovough the mevel y=BeBere: hate ts mde comet hen serve adeed pi 4) shod Be ‘steed abot some line inthe x,y pine. Suppose we have N observations, Geonhis tect and we form the 8% I ¥ectoe 7= yee 9N aNd BE W3 amain ‘ORLHUNOUUAL BASES AND PROJECTIONS st ‘De east sures ema of = (1 nines te sum of aed -HO-D= -XB'O~ XB). In Chapter 8 we will see how ttf cing sitferntn methods Here we il ‘ge the geometrical properties of projections to termine For any choice of 8,9 ~ XB gives «point in the suispace of” spanned bythe columns of X. that the plane spanned by the two vectors Au and x. Ths the pt § minimizes te dtc from y wil be given be ok a tw both ty and x, This leads y= See jews Bx i Se Boo ie I we want ts te het hat 8 = 0, we Holl cone the eet ‘model me yore sete natn i a itn Ig ‘vector | lues satisfies § = Bol, vo for any choice of By, § wil be avon by apn the line png heh the eign hy. Tes 9 {0 minimize the sum of squared eo and hence sstace fom he "a be gen ye rags! proton ay om th ne. Comey. 08 (B'Av = (9~Botyy'tn © or simply2 ‘vecroR SPACES ne vecur& la Thre 2.14 called the component of x orthogonal oS ee Aonping to what krowa 25 the onogonal complement of 5 Definition 210, Lat Sbe a vector subspace of The rg com pee Ss densed by St te cleo of all ett Rta Pe erg every vector in; tha i, Se = (ara RY and xy = O forall yesh ‘Theorem 21S. If a vector subspace of "thea i orogvnal wpe rent Se alo vet subspace of Proof, Soppore tat x, € 5 anlaz = St so that xy — ey = 9 fOr my OE tis tee any ve and any Seaars ay and ca, We BAVC inns candy = esi tent =O. an 30 (aya) +a) € Stand hus 383 vest Spe ° ‘rconseqaene ofthe following theorem i that iia vector sbipace of eo Aa the tension of, te Ue dmeaaion of SH m— 7 Theorem 2.16. Soppose (@).:-s2a) {8 an omhonoral Basis for andere 27%, Suthonomel toss forthe vector subspace. Then ieee vea) ean onthonoral basis fr S*, roof, Let Tbe the vector sen anges BY (Ete ata We must show has yestr space the sae ae Sot © T andy € 3y then there exist Cha ee nay eae rs" ke ANE a bak see a nomiy of he ga, 9 = 060 Stand tus T SS" Pee cee suppore that fe ST. Since x i alo un ere eit sala ca at Ag, te: Now Wwe EY gl og Ai since 4 SL we must have y= a +--+ a = 0 Bat this can ve Appel cn ssn say, Win cake wakes Sake ad 2e"5 oa Pe ee also have ST and 50 this establishes tht 17. PROJECTION MATRICES “Tne orogoalprojeton of en m1 vector ont wetr space Scan becom aa ary cksed in matrix frm. Let (.) be any orthonormal basis Teo een sec) an othonomal bai fA. Supe isu A the constants aiaying the elainehip fay #-F a2) aye TT id rutnction MATRICES 3 ejay mdz = 12, Zh Meee a= (ene (Gynt and Zs = Tee St. Then ine expression for given above can be writen a5 Za Zoe +203; thats, a= Zjy and v= Za, Duet the omonormity of he cs we Base Zz, ly and 225), 0 20 aiaha = Bith2a= 2.22 [eee orf] -20ree “Thos, we have the following esl ‘Theorem 2.17. Suppose the colomne of the m xr mati Zs frm a0 cnhomoreal tas forthe vector space which ia subspace of RY. IC € RY, he orthogonal projection of ont Si given By ZiZix “The matrix 2)Z; apeatng in Theorem 2.17 is elle the projection mati oe nt Shace and someies wit be dence ty Simi, 2:2: avptaeton matric for Sand ZZ" = ky is the projection marx for R Sate Bes zizy + Zaza, we have the simple equation 2:25 = In = 2121 se ene prton macs of vector sabpace and ie ohogona cm rae Atboorh 2 vector space does ot have a unig erthonormal bass, {be projection max formed rom thse ortonoamal ses is uni, ‘Theorem 2.18. Suppose the columas ofthe m x r maces Z) and Wi cach font athonoral basis forthe r-cimensional ver spss S. Then aah ~ A Proof, Bach coluna of W canbe women as tines combination of he oe en ee coms af 7, span S and each columa of Wi iS seers exists an rx rea aac tat Wy = Z:P. Bat 32) = 501 = TSnce each mat bas eronral clans, Thus Le Wii = PZ;ZP= PLP = PP. so tht P is an orogonal max. Conequct, Pls stses P=. an WW, = Z0PPZ) = ZahZ = 212)We will take another lok at the Gram-Schmidtorthonormalizaton pro seduce, this ae aiing projection mes, The prover takes an inl linearly independent set of vectors (x...) whichis wansformed 10 an ltboponal st (y).--0J), which i then tanfonmed to an orthonormal set (eu). Rie vay ear tery that fort yor Tete veto yyy cat be expressed ae that i, ay = le = ZerZi bate where Zip = Gt) Tha, the (1 ‘orhogondl vector yy 8 obtuned as te projection of the (i+ i} oiinl ‘Sestor om the orhogonal complement o the vet spece yn by te fs Frtogonal vectors, 0. “The Gram-Schmidt‘ortionomalization process represents one method of huang an ortboonrmal basis for a vesior space § fom a. given bos (ivory) In genera if we define the mx r mat X) — eat) the Zeki o6, ill form an onthonnemal hase for Ait any rma for which ZL AX XA |, “The matrix A mathe nontnglie since we mast have rank(X,) = snk) =r SOA" exists and XX) = (A-IVA™ of OG)" = AA that's Ais square Foot mais oF OFA) + Consequently, we can obtain an expression fot the Projection matx Ps ent the vector space Sin tm of ¥y Pe 212) = AA f= KONG en Noe that the Gram-Schmid equations piven in (2.5) cane writen i mae form as Yu = XiTe where ¥4= (ip .en¥9 X= (joke and T isan ror upper tingoler mate wth cach digoval leat cua oe The moi ‘ion to prodce 2; can thea be writen at Z; = X\7D" where D i the dia ‘nal mat with the psiove square rot of a teh diagonal element. CConceqely the mse A'= "ie upper tiangulor th postive diagonal elements, Thus the Gram- Schmid othonmalzation tthe particular cae of ‘uation (2.6) in which te mae has ben coset be te Uppet tingle square root matrix of (X(X;)" having positive diagonal clement PROTECTION MATRICES 55 Bxample29, Using the basis (1,223) from Example 27, we frm the xy maint aii cy Lo vy that sag 8 xae[fit 4]. ox 4] tt} . ‘Tas, the projection matrix for the vector space spanned by {x33} siven by tot ta 31 13 ‘Tis of course, is the same as Z;Z}, where Zs = (209) and eto) ate the vests obtained by the Gram-Sehmidt ontonarmaizaton inthe prev us example, Now if r = (1,2,-1,1 hen the projection af one i XiXGR) Xx = Be projection ofx i equal tox since r=ay—m ms € S (Om te ober ani CL. 17, hen the pajenion of = piven By w= XOX N a = G4, 5,3. The componeatof x onhoronal 8. orn ‘other words, the orthogonal projection of x onto Sis (I= XyXK XG j= FOMORN Neen hy This gives ue the decomposition (Hd Example 210, We wll gress some ofthe iss of Example 2.8 0 the smuliple regression model Pox XXX of Theorem 2.14 YoBes Bane Bierce& ‘VECTOR SPACES reaing a epome vavable 10 kcapantory vrialee n,n TF we have Ni obwervatons, this model can be writen as xRre saber y 8 NC 1X18 Gy in + 1 Lane NT while the oro ied! values is even bY 5x8, where (is an estimate of. Cle, for any iy 8 a pointe sume of Fey tc calms of To be a eae sqaares estimate of, B mos Fe seNtiatg iB yields the poit in this subspace closest to the Yetor Since his wl ave the Sum OF squared em, (y= x89 -XB, nisiiced. Thue X(1 must he the atnogonalproetin of y onto the space ‘enoe bythe elurns of I has ull cola an then this pe as Paarron masix XUCX) "2 and 0 the reuied projection it xe xor' xy Premuiplying this emiation by XXX, we obtain the east squares estimator Boom ty In adon, we in tht th sum of squared errors (SSE) forthe fited model XB canbe weten a5 8B) = (y= XBL XB)= (9 = KOCK N= KOC" Fly XOX P= ly OTD nd on thie eam squares represents he squared distance ofthe projection of [Fam the ortogonal complement of he cluin spice of X. Suppo mW at [pana Nise prion os (BB and X= Xo) whet the number lore of Kis the stme asthe numberof element in Bi, and we wish aaa ether x mt Br sO. Ifthe cOLUARS Of axe vtiogoral 0 the Shs of Be hen 245) and ys [aK O eo osxD! PROJECTION MATRICES av so canbe paritoned as = (BBY, where By = (KML. and Bee ge prune she som of seed enor forte fed madd PE Xp. can be decomposed as (y= ABI = AB = Vy ~ AOTAY Clb KRG RYN = NOY (nthe oter hand he least squares estimator of i the reduces mane) 1s B= ean UX we se of aun eres given By S88; yy = Xi R y= KiB Xi “thus the tem SSE5—SSE, = X9EH)*Nay gives the eduetion i the sum Teeter Sto stbuble to the inclason of he erm Xe model age typ, eTsbrre, an soa relative sor wile ell in deci Pee nats = OTF By = 0, then the W observations of y should be geiy eatred about the cumin space of Xin A” wid wo tendeny 0 ar ie sbapce in one dreton more than i any oer direction. se ed 0, we would expect larger deviations in dcetons wit the ile Ba 2 ay tan in dons ondingonal tthe ol spe of X Soha othe dimension ofthe column space of X i k+ 1, SSE i the su wees devatons in N ~~ | orbogonsl deco, wile SSE of eget of square deviations ink onbogonaldecions, where A aesper of component in Bz. Thus, SSE;/(N ~ k~ 1) and (SSE>— SSEn/A: aa ot rar magnites ify ~O wie the bat should Be aan han cao lt BZ 0. Consequently decison about fy can be based on the ‘ale ofthe statistic (88 ~ SSE:V/ks a: 'SSE/N- k= 1) es ing reste that we wil develop in Chane 9, scan be shown tht F = Fanyreken if'e~No(Q.07) and Bs "dada ats Pk de expacion for (SSE: — SSF) i oot eal © _YRLOERF Ry since, in ths ease, ji atthe sum ofthe projection of Y Fes ee am apace of X andthe projection of y ty he oko ast a eases th fr fh incuson of he term XB ie rc, we amet decompose j nto te sum of the projection of yon he coy rae of and te potion of om ie atapuc of the clam rae nn ae eaaleethecolmn szceof Ki Tis ar subspace it spanned BYo vere spaces Kae = (ly HVC, sine ly (8421) he penton wan de athogowat complement ofthe column space of Thus the vector of fte values » ean be writen ARK + Noel aNaad a9 une, the sum of squared errors sive By fly ~ XVOEIAKY = Kaa OKaNaad MI, snd the rection inthe rum of squared errors atiibtable to the inlarion of ‘he term Ys in the model = XB is PX aN) Nay Least squares estimators are not lvays unig as they have ben thoughout this example Por iatance, let us ctr to the lett square etiution of Bia the model = XB+e, where now X doesnot have full column rank. As before {= XB wil be given by the onbogonal projection of y onto the space spanned ty the column of, tthe necessary projection mari canbe expesied NOCH), sce X°X is sngular. Ifthe projection marx of the column space deed Pay en est esi of 5 ay Yee ‘ativan X= Fan Since X does not have fll colamn rank, the mension ofits nll space is at Jen onc a] we wil beable o find a nal yer salting Xa = 8 In this case, Bas also least squares estimator since XGra)-Pany and so the least squares estimator snot unique. ‘We have sen that if the columns ofan. mx mati Z fom an orthonormal basis fora vector space 5, then the projection matrix ofS is given by 2:2. Chey this poetion mati symm and cine 7,7) =I it aan dem potent that is every projection mat is symetric and idempotent. Our next FROECTION MATRICES » ‘aut roves he cunvene Brety symnelc Wenyteat mai na projection ‘matrix for some vector space. ‘Theorem 2.19. Let Pbe an mm syrmeine wempotet mate ot rank Then thee is an rlimensional vector space whch has at is pojection Proof, From Corllacy 191, ther exe an mr mate Fan an ro ‘mati G such that rank(F) = ank(G) =r and P= FG Since P is idempotent, wwebave FOFG= FG, hich implies that FFGFGG =F FG es) Since F and Gare ull cola rank. the matics FF and GG’ sre nonsingla. Premutiplying 29) by (FF) and posumatplying by (GO), we obiain OF =, Using thi end the symmetry of P= FG, we Bo ha Fs FGF o(FGYF< GFF, which leads to G = FUE). Thuy, P= FO= FE AY-'F Compan tis © custo (27), we Se hat P must be the potion mati for he vector ace Spaned by the columns of F. This completes the proof. 3 Example 2.11. Consider the 333 matrix ‘testy, Pi syne and itis easly vite dat Fix dempen,x0.F 6 2 projection matrix. We wl ind the vector space S associated with his poison matrix. Fst note that he fist two columasof Pare ieal independent while {he third column is he average of heft two cama, Thus, an) and 50 the dimension ofthe vector space associated with P i 2 For any x © By ‘Px yields vector mS. In pata, ey and Pey ae in, These two vecore form a basis for $ since they are inal independent andthe drension of § 12, Consequently, S contains al vectors ofthe form (Sa, Sb~a,24 925o vector spaces , LINPAR TRANSFORMATIONS AND SYSTEMS OF LINEAR EQUATIONS, 115 is «vector subspace of A, with projection matrix Ps, then we have seen Tea Manatee eeu ude) © ax tte orthogonal rojo of» onto Si That cache i rancor into awe 8. The function w(x) = Paris tm exampe of linear ansformation of R nto 5. Definition 2.11. Let we a function defined for ll x in the veto space 1 anh tha for any 2.6 T, als) © 5, where Sis also a vector space. hen ‘he eastormation deed by w nee transformation of Tinto if for any fears ac and pan any two vectors #; = T and 2 € Ty Wie wil be intrested in matrix transformations of the form u =A, where -xisinne bapa of Re denoted by 7 iin the srbspace of R" denoted by J idd is do men matrn This deincsatansformation of 7 into S, and the itantormstion linear since foe Salars ya and <1 vex sd Ht follow immediatly tat Atay 0) = aay eA e210) Infact every linear tansformiton ca be expressed asa matrix transformation a er deal projection desebed ate begioning ofthis section, A~ Px Eohte rs mand thoy we have a linear tansfomation of Rito R°, of {0 Te Ror specie linear tansformaton of "into S. fo parla, for he vaste genio blem dines in Example 210, we iw that for any N i Vector of observations y the vector of estimated or ited values was given S EHOERY TY Tine ye a RO, we vee ea tiasformation of nto RUT). ‘t poul te sbvious fom (2.10) that if Sis actually defined to be the set us aggre 1), ten T being « vests space grantees that § wil len Te Neto pee ation i the vetoes yy span Then the vector® we Rar apan Sn parla, i 5 hen ste SPAM we Tener ae lapan 5 at ys the com space or rane of A sinc tis spanned by the columns of A ‘Nose ue mate A does et have ll comm rank dn hte wil be veto 1 Mtns nl rcon,ttfyng Ax = 0 These ofall sich Vestn ‘Lua the ull pace ofthe wansormation Ax or simply the nal space of 6 rate A ‘Theorem 220, Lette nea transformation of Rat S be given by = ‘Anvunere re and Asan m ae mauin The the ell space of 4, eve _ncan TRANSFORMATIONS AND SYSTEMS OF LINEAR EQUATIONS on bythe set NOA)= (& Are 0.26), 3 vector spec. Tre ot ay ears Proof, Letxy andy bein N(A) so that x= 482 4 and a, we hve Alay +0363)» Any + apts = 160) +420) ~ 8 se that (ayn snes) € MAD and ence, (A) is 8 veto sass a op af etnogni com sa sla isthe same ash etogonal complement ofthe clu eee ofA The following ress an immediate consequence of Theorem 210 ‘Theorem 221, Let be an mn mai, Ihe dimension ofthe row #856 onan the dimension of he ol space Of Ais Fy then n #72 =" ‘Since the rank of the marie A ie equ 0 the dimension ofthe ow space cof Ate real above can be equivalently expressed 8 ranktA) =n din (W1A)) en ‘tis coanection tween the rank uf « main andthe dimension ofthe rll TReStnt marx can be very usefl in tering the ak of 8 mati 8 certain sitions, Example 212. Toillsrte the wy of (211), we will ive an stemative cad deity KA) = at) which as ven Teen 320) ro ey ll pce ofA zo tht Av =O. Ten, clay; we most ave Sapo ch implics that ix also in the null pace of AA, 30 it follows tbat ei (A) 5 dim( NAD}, oF equivalent rank(A) 2 ean’) en) onthe her band sis inthe all space uf A'A thon 4°42 - 0 Premubiying De els 4/Ax ~ 0, which is satisfied only if Ax = 0. Thus 8 as0 in ‘eral space of Aso that eim(N(A)) 2 dim(M(A'AD, oF@ vecrom snare rank(A) Sean’) a Combining (2.12) and (219), we got mak (A) = rank When Ais an mx m nonsingular mane and x © A, en a= Ax defines 1 anencone tantormation of "nto R. One way of viewing this tans fonmation is asthe movement ofeach point in R* to another point in R°. ‘Ateratvely, me en view the Uanafruation ara change of coordinate es fwe stat with the standard coordinate axes which are given by ofthe idenity mats HER ine for any x € Ky ‘Stsnganets nf lu th sonia ofthe point x feat to these Sard coordinate aes. On the oer hand i ohm 32 er tais fr fs thes from Theorem 2.7 there exist eal y= vg 20 that th a= (nt aBd X= Gyo tse HAVE Soon that i. a= (ess Eves the coordinates of the point x relative 1 the Coveinate ate nicer The transformation frm the standard coordinate Stem to the one with anes, ay then given by the marx ranforma- onus Ax, where A =X"! No tat the squared Euclidean distance of from the oii, Wu=(Axy(Ax) = Ax, willbe the same a the squared Bulidean distance of from the fig for every Choke of ifand only if Asan hence aso Xy 8 ocogopal matt. To ti che ty forme an eathonrmal ui for Rand the tansformaton has eplacd he standard coordinate axes by anew set of orthogonal ats given Example 213 Orogonltanformatons ae of wo gps aceding 1 whee fie deteuniaut of Ais of 1 Hf A= Ly then the ew SHER SRRST "ae sandr ats Fr example fore fed age tet soe coed of cos 8 sin H 50 that [Al = win? @ esa? — 1. The wanaformatin given by # = A tan forms the standard axes ¢)2,€) 10 the new axes x1 = (Cs 0,~sin €,0), LINEAR TRANSFORMATIONS AND SYSTEMS OF LINEAR EQUATIONS e 5. = (ia 8208 8,07, 5 ~ ey and this simply rpresnts a roton of ey ad though an angle f 6. If instead we have tne ene 8 cose —ane ‘ 0 0 co? 6+ in? 0) Now the transformation piven hy transforms the sandad axes tothe new ates y= (e088, 80 8,0). 21 = la oon 0,07, and ay = ~ey eae ance ve bla by 4 saan of 6 andes through an angle of @ followed by aeteton of es sbou the, #2 plane. Although othogonal transformations are very common, thee are stuations In whieh onsingular nonorthogonal tasformatons ie Rrample 214 Suppote we have several thee dimension vector -xjy-yay that ace observations fom diselbution, cach having he ste yo tive dette covariance marx. I we are meee in how thet vectors ie ffom one another, then spot ofthe points in may be usfl. However, 38 Ascvsed in Example 22,60 ie ot the Melty mat, then the Euclioan fiance isnot appropiate, and st becomes diffclt to compare and interpret ‘he observed erences among the r pias. This diiculy ean be reslved by 1m apropite transformation. We wil se in alter cigtr that singe 0 it posive definite, there exiss a nasinglarmatix T ssisfying Q = TTI we Ket) = Tz, then the Mahalanobis stance, which was éefined ln Eaanple 22, between ands ages) = =)" "ea" 2 STE 9"? 2M Me Ye, Th yt = us mu WW = a) ile he vance of I given by atu) = vara) = Tyas) 7! TOT ts the ansormaton = ln, produce vectors fr which the Bucliean yy © Shots € Sa) cove For any st, the set (S) defined as the intersection ofall convex ses onaming 9 calle he convex hal oS, Cotsequeny, eto 3 generalization ‘€ Theorem 224, C15) the smallest conve et contain {A point as limit or acumolaton point of a set Sc Ri Tor any 8> 0, the 60 5p (an © Rye aV(a a) ~ 5) eon a Beast one pout of S iinet fom a. A closed set sone that contain al of limit point If Ss & Set then § wll eaot ts closure; that sf Sis the et fall iit pins ofS, then B= SUSy In ur nen theorem, we nee that he convent of Sptarantes the conveniy ofS Theorem 114 16S & R*is convex st, ten 1 closures also a convex Proof. easly venhed that te st By = (xe RE S WY conver set. whens a pestv integer Coascquenyi follows rom hearer 2230) that C, = $+ By is also conves. Italo follows from «generalization ofthe result given in Theseus 223) ta eset “Ae is convex The ret now follow by oberving that A = B ° ‘One ofthe ost important results regarding convex ses ia theorem krown the separating hyperplane theorem A hyperplane nie 4 et ofthe for, Te (exe Rare c), shew ais an mv | vector and c isa scala. Thus ite = 27 represen ine ln and if m = 3, T isa plane In. We wil see that the separating hyperplane theorem sats thal to convex se and ‘Sh ae scpaated by 2 hyperplane if tee itersection ls empy tat fy ete le a byperlane which pations R into two parts eo that fe contained in ‘one par, while Sy i contained inthe ace. Before proving this rest we wil ‘eed to Oba some preliminary resus. Ov Hist reali a speial case of the separating hyperplane theorem in which one ofthe es consi the ingle point 0 Theorem 2.25. Let Sbe a nonempty closed cover suse of Rand sp pose that O 5. Then thre exist an mI vecor a such tht a's > O forall Proof. Leta be a pint in S satisfying here inf denotes the infimum or greats lower bound. Ii a consequence of te fat that 5 closed and nonemry tha such ana € S exis. In Addon, 124 O sioce 0 € S. Now lec be an abirary scalar any vector in except or a, and consider the vector cx + (1 ca. The squared length of his vest 2s faut of cs given by (or +(1~ da’ fer + (1 ~ a) = eer - a) +0 (eer = ee ~ axa) + 2ea(e 0) +a fo)Since the second deivative ofthis quarati unstio ) > pasties we Sd that ic hs oique minim athe point au-e) =a Note that since Sis convex, = er +(1 =a $ when 065 1 and 0 we Mis huveaen “JO >/0) A/a for 0S.e 1 de tothe way a was defined. ‘Br because ofthe quadrature of /(0 this implies tht fl) > (0) for eS In oer words cw 50, and His ads 20 ae-a20, ae2aa>0 o [A point x, is an inci point of $ fuer exists 2B > 0 such thatthe set syn Genre We tye 4s) 0, the se Sy comtais atleast one pont ins and tka one point not nS. Our next resol shows tat the ets Sand $ave de see ites pot 5S i conven. ‘Theorem 2.26. Suppose tit $6 convex subse of, while T sa. open whol BUTS ten TCS. Proc, Lett bean atiaty point in T and dene the sete Seafexsy-myesh Te aay eT Ie ftiows fom the conditions ofthe theorem that Sy convex, Ti opm and eae pro! wil be complete if we can show tat Oe Sy Smce as wii Miavahans eg, Sine Oc Te and Pe ie an open or, we can find an >0 ich thal ech ofthe vesor,¢),--éq.~el ar it Ty Sine thes vector: SED gUN bein Sor we can hd Sognces, tay fo = Tsou be Mar eschiny e Seand ry 046 for = Nenad xy > thy OF Petpet any oo Define the mene rats Xj = yy) 80th X) eon ej ose rolows tha ne ents an igor Mi 20ch at Xe mrmsingaar fava > Ny. Bor i>, define 78) "emense ew sothat Ne yy 2a “Thos there exists some integer Na = Noy such that for al > 8a all 9 he ompones uf ate negative But from (214) we have saoiy My esa] 1 by the wnt vector Gis e eteyy TV. Thos Oss convex ciao of he clamas of fest of whieh ein Sus since Se is conven, Se “The next result i sometime called te supporting hyperplane thre tt acer tran har ot of cOENeX eS there exists a hyperplane ‘Sig trugh int ptt ch at one of the ots ofS a8 oR one sew! {ne hyperplane ‘Theorem 227. Soppose that $63 convex subset of "ae tht ether ara boundary pnt of § itis in S. Ten there exists a9 1% t Nectar B40 sch dat Dix 2 Bix, for all 5: roof, follows fom the previous theorem tha also notin Sox mas 16 eat pin of Si itis nS. Consequny,Wete exist sequen of be. a boundary old eau a; € Ssich that yx» ae on, Comesponding Meee Shine eet §)= G2 year © S)vand note ha Og 8, since 19 Thue ine 8s closed and conves by Thcorem 2.2, i fellows om 5, Ts hes dare cents am mom vec, such tha 9 > 0 forall FES ay equivalent afer =x) >0 forall x S.Aleratively, we can wre Jhis Bray) >0, where B= a) Now since By ~ the sequence Here nThonded sequence tnd 0 it has & convergent absence; tat betas ple igers € oad ome mL unit veer b Balog Soy orm comscguenty yen) = Mer ras) 0 ae mi have Bate) 20 forall x S since Bj (xx) >0 forall x 5. ‘Tis completes the proot 5 We ae now ready to rove the separating hyperplane theorem. ‘Theorem 228. Let 5; ond S be convex subsets of R" with 1 15: = 2. steam inse exit an mx Leste DO sch that Bx; 2 625 forall x1 © 3s and all © 3 Proof. Clay the set Sy, = (&\* € S:) “rus tom Theorem 2.23 we ao ha the 2 conver since Sy eomves” ‘VecToR SPAces S950, feenn mene Se) is also conves. Inston, 0 € 5 since $) 1S: = ©. Consequsty, using “Theorem 227 end that there san Toner B © for which Bi > for ail 8. Bu this implies hat Be, 23) 20 forall x, € 5; and all #2 € Sa, 386 requied 6 ‘Suppose ht) i a pomnegative funtion which is symmetric about x = 0 so har ony one maimam, occuring t ster words fs) =f) forall x and fs) © fles) 0S c <1. Clay, the integral of f(e) over an Interval of fied length will be maximized when the interval i centred at 0, fl osonee2 ff sesnae for any », a> 0, and 0c 1, This result has some important applications regarding prbabliies of radom variablee The flowing ral which ie geteralizaton toa function (x of the m x 1 vector replaces the interval in Fi'by esymmerc convex sein A". Ths generalization is dae w Anderson (41955. For simple apications ofthe rel o probabilities of random vectrs, fee Problem 2 Theorem 229, Let S be a convex subset of R", symmetic about 0,20 hat, steve Sas. LtjQ) 20 bea fonction fr we fa) = Ce J) 2a is convex fr any piv aa fx} Find eapesion in terms of mr and foe the smallest and ages), posible valves of dims +82). (&) Find the smallest an largest posible values of dim 1) Let be the vector pace spanned by the vectors (11 1/2} Find 2 vecorspee Sy sugh that R= TQS}, Ptpd another veto space 2 ouch fa FS, ad 0S 0) Les; be the vector space spa by (Ls 1.~25 072,01, YI whl Sie spanned by (11. ~3Y< (Ly ll) ) Show that R= 5) +S Ts this ‘inet sum? Thats, cin We wete 5) @'5;? Are Sy and Sy oriogons! Nector spaces? Let 5; and Sz be vector subspaces of KT and let T= $i +8 Show tht ths sme» et eum, thats. T= 5: Seif and only if hing) ~ iets) + iS) “The concept of orthogonal projections and their associated projection mat. Frere Stead to projections that se 90 ohogonal Tn the se of ‘Sthoponal projections onto the vector space Sc R, we decompose R™ SpE SOS. The projection matsx tat projets orogoally onto S i the matin Psaisfying Py @ Sand (9 Pv) € S* for ally © R" and Pena forall xe SIF Si the column space ofthe fll rank mati X, then S ml Bethe nll space of X, andthe projection mai deserted shove ic piven by P= XOCXY'X. Suppose now that we decompose R" Sree SOT, whee 51s as before and Tis the ml space ofthe full an hatin Note tha Sand Tae m0 neces ohogensl vector spares ‘We wish to fnd a projection mati O satstying Qy € Sand (y~ Qs) T for ally ¢ "and Qr= forall € 5 (a) Show tat Qs pejeston matin i and only if isan idempotent (©) Show that Q can be expressed as Q= KOI" Prove Theowem 2.23, Show that if nd Ss ae comers shuts of KY. then Si U Ss need not be Show that for any postive salar, these By = Grae ROS) 46, Toray se 5 A" show ta ts canven Ill C(S) consis ofall convex combinations ofthe vectors in, 17 Sue at op a o£. Sho hk ty vet conver bl of $ can be epesed as comnex cmon fm To ‘fewer vectors in S. " won fim) ‘© fen edt Joy.) eos rah pouse or epee ‘Sis & convex subset of R", symmetric about 0. pose a: Seppene at (©) Show that Pic € 5) =r y 6 8) for any conta vestry € § (© Show hat Be aegstty in alo ol iy an aden vstor ‘tte independent of (© Stow at t= Na.) 18 enyfucton sss the condions tis exer. (@ Show that if andy ae independently dtd wih x ~ sey" Naf) st 8) By mega dans ten Reessitye 3)CHAPTER THREE Eigenvalues and Eigenvectors INTRODUCTION Eigenvalues and eigenvectors ate speci implicitly defined functions of the eiecent ofa agate matin. In many applications involving the analysis of (era 6 RP and Ax = 2), Theorem 3.1. 1 5h) ste eigenspace of the mx matnx A corespond- Ing tothe rot A then R) isa vector subspace of Proof. By deition ix € 34(A) then AX = Ax. Thus, x © 34(A) and ye S10). we have forany salah and Mex + By) ax + Bay = a(x) BOAN)= Nar BY) ‘Consequently, (ax + y) « $40), and so S,() isa vector space a Baample 33, The matrix 2-0 [3 ‘9 tas the characteristic equation 50 the eigenvalues of A ae, with mulipicity 2, and 2 To Bd (1). he igensgacecoresponding othe eigenvalue I, we solve the equation Ax = for 5, We leave 10 he reader © very tat hs leads ovo nearly independent Solutions: any solution to Ax = willbe a linear combination of he two vectors 2515 (00,17 andy = (11,07. Ths, (1) ste sabe panned bythe sss (eyes that S(1) a plane ine a smir fbi, we i find the clpenagace 5,2), Solving Ax = 2x, we find that x must bea scalar mull of (0,07. Tus, 542) Is he hne in #* given by (@,0,0):-~ acm} In the preceding example, for each vale of, we hae dimiS{Q)) being ‘equal tothe multiplicity of This soe always the case; he following example ‘Master that tt possible for aim (SCA) t Bele than the mle of the eigenvaluets IGENVALUES AND EIGENVECTORS FFeample 34. Consider the 33 matrix sven by Ae i i 4] oat Since [A =Dl|=(1=), Aas the eigen 1 repeat ee ines. The Sine I Mtn egaton AX =e yk ec uals equations aye2g edn en which have a solution only vetoes ofthe form x= (20,07. mus, snow ecdunipicly ofthe eigenvalue 1s 3, the aeoeatd eigenspace Sal) ((a.0,0/2=» <<} is oly one-dimensional, ‘3. SOME BASIC PROPERTIES OF EIGENVAL ‘AND EIGENVECTORS. les Io this seaton, we establish some very seul cesults regarding clgenvalues “The powfs of the ress in our fst theorem, which ae et 10 the reader is an exercise, ae eal obuned Dy using the characterise equation or the Ceemvale-cigonvestor equation Theorem 3.2, Le A be an mx matix, Then (a) The eigenvalues of A’ are the same athe eigenvaes ofA () As singular if and ly if a eat oe cigenratc of Ais equal 1 0 {c) The diagonal cloments o ae the eigenvalues ofA fs tango (a) The eigenvalues of HAM ' ze the sre a he elgsavacs of Ay 3. {tnoninglar nm mat (e) Bach ofthe eigemales of Ais ether +1 or ~1, 14 is an onthogonal ‘We sa in Example 3.4 that tis possible fo the dimension ofan eigenspace soomised wi a egenvlae) te lee than the mili of. The ok Towing theorem shows that fdim{S,(9))¥r, wherer denotes the mulpicity cof then dim (S4000) <7: ‘Theorem 3.3. Suppose an eigenvalue, with mulipicy r2 1, ofthe xm mani A. The Jes An FLOENVECTORS » SOME BASH: PROPLLIES OF COE Isdimisanns Prat Ith isan eigenvalue ofA by defniton there exists an + # Daisy ing igenaeeigeetor equation A¥ = Ax aso, sf din{ S00) = ine era gca(SaC)). and lf Be lnerly independent egen- | Now ponding to om nonsingular m maui X whieh has Freer SSitoe as tnt columns, hat 1X faethe form X= (Xe Xl Siecle tr, craband X ism (mW), Since each cola of Xy isa SESS ter FA conesponding te egevae A oows ma A, ~ im follows fom the fact hat X12 = I Asa result Wend eat SAX CHAR, AX= AIK, AN [is 4] “uhere By and Bs represent a paring ofthe max X-'ANs 1 isan cen alte of VAX, then cptancutal<|°O 8 0 AK | gy ta == HPD Alaa where he lst equality canbe obtained by repeated use ofthe cofactor expan ae tn detrmunat. Thus, rus be an cgchvabe of LAX with Sofa lean FT rest now follows snc, fom Theorem 3.24 the eenvaues of XK are te sme 2 thas of A. 5 ‘now prove he flowing theorem involving both the egenvaes znd the cigenvector of « mate ‘Theorem 34, Lst be a eigenvalue of the mx m mats A and etx a conesponding eigenvector. The, ay Tr mis an integer 21, 0 isan eigenvale of A” comesponding fo the cdigenvedtor x (HH ning, 1 ea eigemaln of AT corresponding tothe daenvestor =% ‘IGFAALUES AND EIGERVECTORS Prony Yaa 8 proven by repel wing estoy Ax a areata “To prove putt), premuliply the eigenalue-igeavector equation Arede by A, yeting te equation reo 6 Sine Ae soning we know fom Theorem 3.20) that #0, and so divi ing bth sides of (3.3) by yields dren's Which the eigenvalue-eigenvector equition for A, wi eigenvalue Wand eigenvector x 3 ‘The detent and tae of a mati have very simple and woeful ration ships with the eigenvalues ofthat matrix. These eaionshps ar established in the ext theorem ‘Theorem 3S. Let be an mm max with eigenvalues hiy..yhqe Then Proof Recall hat the charictrtc equation, [A~AII =O, an be expressed inthe polynomial form Oy ene OI a mA a= 6a We wil ist tentty the coetcens ag and aig. We can determine an by valinting the eb tke of equation (34) af =O: thus a= A ~ CO = [nism cedr to find i. all at, from its definition, the determinant i fxprese exam of tis overall penutatons of he integer (1. Since oy i the coefficient of (-AJ*", (0 evaluate this tem we only need to consider te tere inthe sum wich valve atleast m ~ 1 of he cagocal ‘lemons of (AAD). But each tem nthe sm tthe product of m elements from the matrbx (A~ ND, mpl by the appropiate sign, with one element ‘SOME BASIC PROPERTIES OF EIGENVALUES AND EIGENVECTORS o hoten from each rou and ach colame of (4 ~D. Consequenty the oely term inthe sum involving atleast m1 ofthe diagonal elemeats of (A ~ XI) {ste term hat involves the product ofl ofthe ago cements Sac his term involves an even permutation he sen term wil eqal #1, abd 0 e+ ‘wl be the coefficient of (AJP it fen Me 9a hich leary 4034-4. or simpy (A). Now 0 ete ay = [Aland fant = tA} to the eigenvaice OFA, te A SIGE. ATE the th chester uation. which ean mth deeeeoolomial nit allows that y=) WHO Mahiplyng ou the left-hand side of this equation and then matching coe sgonding terms wit hose i 4), we find that waef]a wo-Sn 2 “The following theorem gives scent conion fora set of eigenvectors 1o be nearly independent. Theorem 3: Soppoie 30-0, ae ebgenvertrs of the mm mate A, where r mf the eoresponding eigenvalues hyy.y)y ate such Mat # all £2 then the vector: ys ate Bnesly indent Proof. Ou proofs by conraicto: that i, we begin by assuming that the estos sieve te hotly dependent Lt be the largest integer for which nese a linearly independent Sacha st can be found since x, being a eigenvector, cannot equ 0 an 20H inert naependea. ne vectors ecersesmnst belinesrly dependent, soho vit ealare ay ith leas wo not equal ero since no eigenvector canbe the ml vector, such ane ete mere Premuiplyng the left-hand side of his equation by (A ~ AyD, we nd tat