Академический Документы
Профессиональный Документы
Культура Документы
CoolSolutions:RecoveringaLostLVMVolumeDisk
NovellisnowapartofMicroFocus
Products
Services&Support
Partners
LOGIN
Communities
AboutUs
UnitedStates,English
CHANGE
HowtoBuy
RecoveringaLostLVMVolumeDisk
CoolSolutionsHome(New)
NovellCoolSolutions:AppNote
ByJasonRecord
ClassicCoolSolutions
Home
DiggThisSlashdotThis
Authors
CoolBlogs
CoolSolutionsWiki
CoolTools
GetInvolved>
OpenAudio(podcasts)
Posted:27Jul2007
Contents:
Overview
ServerConfiguration
DiskBelongingtoaVolumeGroupRemoved
CorruptedLVMMetaData
DiskPermanentlyRemoved
Conclusion
Overview
LogicalVolumeManagement(LVM)providesahighlevel,flexibleviewofaserver'sdiskstorage.Thoughrobust,problemscan
occur.Thepurposeofthisdocumentistoreviewtherecoveryprocesswhenadiskismissingordamaged,andthenapplythat
processtoplausibleexamples.Whenadiskisaccidentallyremovedordamagedinsomewaythatadverselyaffectsthelogical
volume,thegeneralrecoveryprocessis:
1.Replacethefailedormissingdisk
2.Restorethemissingdisk'sUUID
3.RestoretheLVMmetadata
4.RepairthefilesystemontheLVMdevice
Therecoveryprocesswillbedemonstratedinthreespecificcases:
1.Adiskbelongingtoalogicalvolumegroupisremovedfromtheserver
2.TheLVMmetadataisdamagedorcorrupted
3.Onediskinamultidiskvolumegrouphasbeenpermanentlyremoved
ThisarticlediscusseshowtorestoretheLVMmetadata.Thisisariskyproposition.Ifyourestoreinvalidinformation,youcan
looseallthedataontheLVMdevice.AnimportantpartofLVMrecoveryishavingbackupsofthemetadatatobeginwith,and
knowinghowit'ssupposedtolookwheneverythingisrunningsmoothly.LVMkeepsbackupandarchivecopiesofit'smeta
datain/etc/lvm/backupand/etc/lvm/archive.Backupthesedirectoriesregularly,andbefamiliarwiththeircontents.Youshould
alsomanuallybackuptheLVMmetadatawithvgcfgbackupbeforestartinganymaintenanceprojectsonyourLVMvolumes.
Ifyouareplanningonremovingadiskfromtheserverthatbelongstoavolumegroup,youshouldrefertotheLVMHOWTO
beforedoingso.
ServerConfiguration
Inallthreeexamples,aserverwithSUSELinuxEnterpriseServer10withServicePack1(SLES10SP1)willbeusedwithLVM
version2.Theexampleswilluseavolumegroupcalled"sales"withalinearlogicalvolumecalled"reports".Thelogicalvolume
andit'smountpointareshownbelow.Youwillneedtosubstituteyourmountpointsandvolumenamesasneededtomatch
yourspecificenvironment.
lslvm:~#cat/proc/partitions
majorminor#blocksname
804194304sda
81514048sda1
821052257sda2
831sda3
85248976sda5
816524288sdb
832524288sdc
848524288sdd
lslvm:~#pvcreate/dev/sda5/dev/sd[bd]
Physicalvolume"/dev/sda5"successfullycreated
Physicalvolume"/dev/sdb"successfullycreated
Physicalvolume"/dev/sdc"successfullycreated
Physicalvolume"/dev/sdd"successfullycreated
lslvm:~#vgcreatesales/dev/sda5/dev/sd[bd]
Volumegroup"sales"successfullycreated
lslvm:~#lvcreatenreportsL+1Gsales
Logicalvolume"reports"created
lslvm:~#pvscan
PV/dev/sda5VGsaleslvm2[240.00MB/240.00MBfree]
PV/dev/sdbVGsaleslvm2[508.00MB/0free]
PV/dev/sdcVGsaleslvm2[508.00MB/0free]
https://www.novell.com/coolsolutions/appnote/19386.html
1/6
8/9/2016
CoolSolutions:RecoveringaLostLVMVolumeDisk
PV/dev/sddVGsaleslvm2[508.00MB/500.00MBfree]
Total:4[1.72GB]/inuse:4[1.72GB]/innoVG:0[0]
lslvm:~#vgs
VG#PV#LV#SNAttrVSizeVFree
sales410wzn1.72G740.00M
lslvm:~#lvs
LVVGAttrLSizeOriginSnap%MoveLogCopy%
reportssaleswiao1.00G
lslvm:~#mount|grepsales
/dev/mapper/salesreportson/sales/reportstypeext3(rw)
lslvm:~#dfh/sales/reports
FilesystemSizeUsedAvailUse%Mountedon
/dev/mapper/salesreports
1008M33M925M4%/sales/reports
DiskBelongingtoaVolumeGroupRemoved
Removingadisk,belongingtoalogicalvolumegroup,fromtheservermaysoundabitstrange,butwithStorageArea
Networks(SAN)orfastpacedschedules,ithappens.
Symptom:
Thefirstthingyoumaynoticewhentheserverbootsaremessageslike:
"Couldn'tfindallphysicalvolumesforvolumegroupsales."
"Couldn'tfinddevicewithuuid'56pgEk0zLScKBcz9vJkP65DUBIhwZPSu'."
'Volumegroup"sales"notfound'
Ifyouareautomaticallymounting/dev/sales/reports,thentheserverwillfailtobootandpromptyoutologinasroottofixthe
problem.
1.Typeroot'spassword.
2.Editthe/etc/fstabfile.
3.Commentoutthelinewith/dev/sales/report
4.Reboot
TheLVMsymptomisamissingsalesvolumegroup.Typingcat/proc/partitionsconfirmstheserverismissingoneof
it'sdisks.
lslvm:~#cat/proc/partitions
majorminor#blocksname
804194304sda
81514048sda1
821052257sda2
831sda3
85248976sda5
816524288sdb
832524288sdc
lslvm:~#pvscan
https://www.novell.com/coolsolutions/appnote/19386.html
2/6
8/9/2016
CoolSolutions:RecoveringaLostLVMVolumeDisk
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
PV/dev/sda5VGsaleslvm2[240.00MB/240.00MBfree]
PV/dev/sdbVGsaleslvm2[508.00MB/0free]
PVunknowndeviceVGsaleslvm2[508.00MB/0free]
PV/dev/sdcVGsaleslvm2[508.00MB/500.00MBfree]
Total:4[1.72GB]/inuse:4[1.72GB]/innoVG:0[0]
Solution:
1.Fortunately,themetadataandfilesystemonthediskthatwas/dev/sdcareintact.
2.Sotherecoveryistojustputthediskback.
3.Reboottheserver.
4.The/etc/init.d/boot.lvmstartscriptwillscanandactivatethevolumegroupatboottime.
5.Don'tforgettouncommentthe/dev/sales/reportsdeviceinthe/etc/fstabfile.
Ifthisproceduredoesnotwork,thenyoumayhavecorruptLVMmetadata.
CorruptedLVMMetaData
TheLVMmetadatadoesnotgetcorruptedveryoftenbutwhenitdoes,thefilesystemontheLVMlogicalvolumeshouldalso
beconsideredunstable.ThegoalistorecovertheLVMvolume,andthencheckfilesystemintegrity.
Symptom1:
Attemptingtoactivatethevolumegroupgivesthefollowing:
lslvm:~#vgchangeaysales
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
Couldn'treadvolumegroupmetadata.
Volumegroupsalesmetadataisinconsistent
Volumegroupforuuidnotfound:m4Cg2vkBVSGe1qSMNDf63v3fDHqN4uEkmWoTq5TpHpRQwmnAGD18r44OshLdHj05
0logicalvolume(s)involumegroup"sales"nowactive
Thissymptomistheresultofaminorchangeinthemetadata.Infact,onlythreebyteswereoverwritten.Sinceonlyaportionof
themetadatawasdamaged,LVMcancompareit'sinternalchecksumagainstthemetadataonthedeviceandknowit's
wrong.ThereisenoughmetadataforLVMtoknowthatthe"sales"volumegroupanddevicesexit,butareunreadable.
lslvm:~#pvscan
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
PV/dev/sda5VGsaleslvm2[240.00MB/240.00MBfree]
PV/dev/sdbVGsaleslvm2[508.00MB/0free]
PV/dev/sdcVGsaleslvm2[508.00MB/0free]
PV/dev/sddVGsaleslvm2[508.00MB/500.00MBfree]
Total:4[1.72GB]/inuse:4[1.72GB]/innoVG:0[0]
Noticepvscanshowsalldevicespresentandassociatedwiththesalesvolumegroup.It'snotthedeviceUUIDthatisnot
found,butthevolumegroupUUID.
Solution1:
1.Sincethediskwasneverremoved,leaveitasis.
2.TherewerenodeviceUUIDerrors,sodon'tattempttorestoretheUUIDs.
3.ThisisagoodcandidatetojusttryrestoringtheLVMmetadata.
lslvm:~#vgcfgrestoresales
/dev/sdc:Checksumerror
/dev/sdc:Checksumerror
Restoredvolumegroupsales
lslvm:~#vgchangeaysales
https://www.novell.com/coolsolutions/appnote/19386.html
3/6
8/9/2016
CoolSolutions:RecoveringaLostLVMVolumeDisk
1logicalvolume(s)involumegroup"sales"nowactive
lslvm:~#pvscan
PV/dev/sda5VGsaleslvm2[240.00MB/240.00MBfree]
PV/dev/sdbVGsaleslvm2[508.00MB/0free]
PV/dev/sdcVGsaleslvm2[508.00MB/0free]
PV/dev/sddVGsaleslvm2[508.00MB/500.00MBfree]
Total:4[1.72GB]/inuse:4[1.72GB]/innoVG:0[0]
4.Runafilesystemcheckon/dev/sales/reports.
FollowUs
RequestaCall
18883214272
Feedback
lslvm:~#e2fsck/dev/sales/reports
e2fsck1.38(30Jun2005)
/dev/sales/reports:clean,961/131072files,257431/262144blocks
lslvm:~#mount/dev/sales/reports/sales/reports/
lslvm:~#dfh/sales/reports/
FilesystemSizeUsedAvailUse%Mountedon
/dev/mapper/salesreports
1008M990M0100%/sales/reports
Symptom2:
MinordamagetotheLVMmetadataiseasilyfixedwithvgcfgrestore.Ifthemetadataisgone,orseverelydamaged,thenLVM
willconsiderthatdiskasan"unknowndevice."Ifthevolumegroupcontainsonlyonedisk,thenthevolumegroupandit's
logicalvolumeswillsimplybegone.Inthiscasethesymptomisthesameasifthediskwasaccidentallyremoved,withthe
exceptionofthedevicename.Since/dev/sdcwasnotactuallyremovedfromtheserver,thedevicesarestilllabeledathrough
d.
lslvm:~#pvscan
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
PV/dev/sda5VGsaleslvm2[240.00MB/240.00MBfree]
PV/dev/sdbVGsaleslvm2[508.00MB/0free]
PVunknowndeviceVGsaleslvm2[508.00MB/0free]
PV/dev/sddVGsaleslvm2[508.00MB/500.00MBfree]
Total:4[1.72GB]/inuse:4[1.72GB]/innoVG:0[0]
Solution2:
1.First,replacethedisk.Mostlikelythediskisalreadythere,justdamaged.
2.SincetheUUIDon/dev/sdcisnotthere,avgcfgrestorewillnotwork.
lslvm:~#vgcfgrestoresales
Couldn'tfinddevicewithuuid'56ogEkOzLScKBcz9vJkP65DUBIhwZPSu'.
Couldn'tfindallphysicalvolumesforvolumegroupsales.
Restorefailed.
3.Comparingtheoutputofcat/proc/partitionsandpvscanshowsthemissingdeviceis/dev/sdc,and
pvscanshowswhichUUIDitneedsforthatdevice.So,copyandpastetheUUIDthatpvscanshowsfor/dev/sdc.
lslvm:~#pvcreateuuid56ogEkOzLScKBcz9vJkP65DUBIhwZPSu/dev/sdc
Physicalvolume"/dev/sdc"successfullycreated
4.RestoretheLVMmetadata
lslvm:~#vgcfgrestoresales
Restoredvolumegroupsales
lslvm:~#vgscan
Readingallphysicalvolumes.Thismaytakeawhile...
Foundvolumegroup"sales"usingmetadatatypelvm2
lslvm:~#vgchangeaysales
1logicalvolume(s)involumegroup"sales"nowactive
5.Runafilesystemcheckon/dev/sales/reports.
lslvm:~#e2fsck/dev/sales/reports
e2fsck1.38(30Jun2005)
/dev/sales/reports:clean,961/131072files,257431/262144blocks
lslvm:~#mount/dev/sales/reports/sales/reports/
https://www.novell.com/coolsolutions/appnote/19386.html
4/6
8/9/2016
CoolSolutions:RecoveringaLostLVMVolumeDisk
lslvm:~#dfh/sales/reports
FilesystemSizeUsedAvailUse%Mountedon
/dev/mapper/salesreports
1008M990M0100%/sales/reports
DiskPermanentlyRemoved
Thisisthemostseverecase.Obviouslyifthediskisgoneandunrecoverable,thedataonthatdiskislikewiseunrecoverable.
Thisisagreattimetofeelgoodknowingyouhaveasolidbackuptorelyon.However,ifthegoodfeelingsaregone,andthere
isnobackup,howdoyourecoverasmuchdataaspossiblefromtheremainingdisksinthevolumegroup?Noattemptwillbe
madetoaddressthedataontheunrecoverablediskthistopicwillbelefttothedatarecoveryexperts.
FollowUs
RequestaCall
18883214272
Print
Feedback
Symptom:
ThesymptomwillbethesameasSymptom2intheCorruptedLVMMetaDatasectionabove.Youwillseeerrorsaboutan
"unknowndevice"andmissingdevicewithUUID.
Solution:
1.Addareplacementdisktotheserver.Makesurethediskisempty.
2.CreatetheLVMmetadataonthenewdiskusingtheolddisk'sUUIDthatpvscandisplays.
lslvm:~#pvcreateuuid56ogEkOzLScKBcz9vJkP65DUBIhwZPSu/dev/sdc
Physicalvolume"/dev/sdc"successfullycreated
3.RestorethebackupcopyoftheLVMmetadataforthesalesvolumegroup.
lslvm:~#vgcfgrestoresales
Restoredvolumegroupsales
lslvm:~#vgscan
Readingallphysicalvolumes.Thismaytakeawhile...
Foundvolumegroup"sales"usingmetadatatypelvm2
lslvm:~#vgchangeaysales
1logicalvolume(s)involumegroup"sales"nowactive
4.Runafilesystemchecktorebuildthefilesystem.
lslvm:~#e2fscky/dev/sales/reports
e2fsck1.38(30Jun2005)
snip
Freeinodescountwrongforgroup#5(16258,counted=16384).
Fix?yes
Freeinodescountwrong(130111,counted=130237).
Fix?yes
/dev/sales/reports:*****FILESYSTEMWASMODIFIED*****
/dev/sales/reports:835/131072files(5.7%noncontiguous),137213/262144blocks
5.Mountthefilesystemandrecoverasmuchdataaspossible.
6.NOTE:Ifthemissingdiskcontainsthebeginningofthefilesystem,thenthefilesystem'ssuperblockwillbemissing.
Youwillneedtorebuildoruseanalternatesuperblock.Restoringafilesystemsuperblockisoutsidethescopeofthis
article,pleaserefertoyourfilesystem'sdocumentation.
Conclusion
LVMbydefaultkeepsbackupcopiesofit'smetadataforallLVMdevices.Thesebackupfilesarestoredin/etc/lvm/backupand
/etc/lvm/archive.Ifadiskisremovedorthemetadatagetsdamagedinsomeway,itcanbeeasilyrestored,ifyouhave
backupsofthemetadata.ThisiswhyitishighlyrecommendedtoneverturnoffLVM'sautobackupfeature.Evenifadiskis
permanentlyremovedfromthevolumegroup,itcanbereconstructed,andoftentimestheremainingdataonthefilesystem
recovered.
AdditionalReferences
SLES10InstallationandAdministrationGuide,7.1LVMConfiguration
http://www.novell.com/documentation/sles10/sles_admin/data/sec_yast2_system_lvm.html
LVMHOWTO
http://www.tldp.org/HOWTO/LVMHOWTO/index.html
Likewhatyousee?
Signupforourweekly
newsletter.
Wanttocontribute?
Itcouldearnyouanano!
Learnmore.
LikeWikis?
JointheCoolSolutions
Wiki.
Interested?
Requestasales
call
NovellCoolSolutions(corporatewebcommunities)areproducedbyWebWiseSolutions.www.webwiseone.com
https://www.novell.com/coolsolutions/appnote/19386.html
5/6
8/9/2016
2016MicroFocus Careers
CoolSolutions:RecoveringaLostLVMVolumeDisk
Legal
FollowUs
https://www.novell.com/coolsolutions/appnote/19386.html
RequestaCall
18883214272
Feedback
6/6