DesignVerificationTheCaseforVerification,NotValidation

Overview: TheFDArequiresmedicaldevicecompaniestoverifythatallthedesignoutputsmeetthedesigninputs. TheFDAalsorequiresthatthefinalmedicaldevicemustbevalidatedtotheuserneeds.Whenthereare somanymoredesigninputsandoutputsthanspecificuserneeds,whydocompaniesspendsolittle timeverifyingthedeviceandsomuchtimeandmoneyonvalidation?Andwhatistheroleofrisk managementindeterminingtheamountoftestingrequired?Thispresentationwilldemonstratethatif developersconductmorecompleteverificationsofdesignoutputsandriskmitigations,validationscan becompletedinashortertime,morereliably,andmoresuccessfully. Introduction: InourexperienceworkingwithmedicaldevicemanufacturerstoimprovetheirQualityManagement Systemsandtogainregulatoryclearanceofnewdevices,wehavefoundthatDesignVerificationisan oftenunderutilizedtoolforensuringsuccessduringthelatterstagesofdevicedevelopmentefforts.In manycases,limitedverificationeffortsrepresentalostopportunitytosignificantlyreducethescopeof validationeffortsandtherebyreducetimetomarketanddevelopmentcosts. ThispaperexplorestheuseandmisuseofDesignVerificationandhowdevicemanufacturescangetthe mostoutoftheirverificationefforts.Butfirst,somebackground... Background: Medicaldevicemanufacturershavebeenworkingtoaligntheirdesignanddevelopmentsystemswith theFDAsdesigncontrolregulations(21CFR820.30)sincetheirreleasein1996.Theregulationsare structuredtoensurethatdevicemanufacturersmaintaincontrolovertheirdesignsthroughoutthe developmentprocessandthatthemarketeddeviceissafeandeffectiveforitsintendeduse.Attheir mostbasiclevel,theregulationsrequirethatmanufacturers: Clearlystatewhattheyintendtoproduce(PlanningandDesignInputs); Developadesignthatmeetsthoseneeds(DesignOutputsandDesignReview); Confirmthatthedesignmeetstheoriginalintent(Verification); Confirmthatmanufacturedproductcanbeproducedreliablyandachievethedesiredresult (TransferandValidation);and

Page1of12
MEDIcept,Inc. 200HomerAvenue Ashland,MA01721
11/2010 MaynotbereprintedorcopiedwithoutexpressedpermissionfromMEDIcept

DesignVerificationTheCaseforVerification,NotValidation
Maintainrecordsofkeydevelopmentactivitiesanddecisions(DesignChangesandDesign HistoryFile)

Itwouldbehardtoarguethatanyofthesestepsshouldbeexcludedfromanydevelopmenteffort. Whetheryouarebuildingabridge,acellphone,orasurgicaltool,thiscombinationofstepshasalogical flowthatmostengineerswouldsimplyconsidertobegoodpractice. Thechallengeariseswhenyoumovefromtheconceptualworld,whereproductdevelopmentcanbe seenassinglestreamofwaterflowingdownahill:steadilymovingfromonepointtothenextwithout interuption;totherealworld,fullofrapids,eddies,branchingstreams,and(occasionally)deeppoolsof waterthatdontseemtobemovingatall.Inthisturbulentenvironment,itscanbedifficulttomaintain adisciplineddesignapproachparticularlywhenbusinessschedulesdemandrapidprogressandany delaysinmovingontothenextdevelopmentphaseriskaffectingprojectmilestonesanddevelopment staffdeploymentplans.Thesechallengesaremultipliedwhenthedevicehasmultiplecomponentsand integrationpointsthatmustbemanagedthroughoutthedevelopmentprocess. Muchhasbeenwrittenabouttheimportanceofearlystageplanningandproblemsolvingtospeed timetomarketandreducedevelopmentcostsandthevalueofclearDesignandDevelopmentPlans andwelldefinedDesignInputscannotbeoverstated.Anequalamountofattentionhasbeenfocused onDesignTransferandValidationactivities.Atthislatestageofadevelopmenteffort,theprojectscope willhaveexpandedtoincludemoreactiveinvolvementofClinical,OperationsandMarketingstaff.In addition,thecostofvalidationstudiesandthemakeorbreakaspectofthesestudiesrequireagreat dealofattentionandcompanyresources.Problemsatthevalidationstagewillhaveseriousimplications forthesuccessoftheprojectandespeciallyforsmallcompaniescoulddeterminethefateofthe wholecompany. Comparedtotheseearlyandlatestageactivities,theimportanceofDesignVerificationtendstobe overlooked.Thereareseveralreasonswhy: Definitions:Manypractitioners(andestablishedQualitySystems)continuetohavetrouble definingexactlywhatverificationis,andhowitfitsintothedesigncontrolprocess.Thetermis oftenconsideredtobesynonymouswithvalidationandthedevelopmentofcombined V&Vplansthatdonotestablishacleardistinctionbetweentheobjectivesofthetwoefforts donthelp.[Note:whendiscussingverificationwiththeFDA,itsbestnottotalkaboutyour V&Vplanincreasingly,inspectorspreferencesaretoaddressthetwoactivitiesseparately.] Timeframe:Verificationisoftenanongoingeffortconductedthroughoutthedevelopmentof outputs.So,agreatdealoftimemaypassbetweenwhenthefirstandlastoutputisverified.

Page2of12
MEDIcept,Inc. 200HomerAvenue Ashland,MA01721
11/2010 MaynotbereprintedorcopiedwithoutexpressedpermissionfromMEDIcept

DesignVerificationTheCaseforVerification,NotValidation
Withthisextendedtimeframe,verificationactivitiescangetlostwithinallofthechurning associatedwithdevelopingfinaloutputs. Approaches:Thereareavarietyofwaystocompletetheverificationforanoutput,soitcanbe difficulttocommunicatethatalloftheseactivities,asagroup,representthedesignverification.

Theproblemwiththislackofattentionisthatitweakensacriticallinkinthedesigncontrolchain, affectingthestrengthoftheoveralldesigncontroleffort.Poordesignverificationcanleadtoproblems duringdesigntransferandvalidation,andcanreduceyourabilitytotrackdownandcorrectproblemsif (andwhen)theyoccur.Justasimportantly,itmayrepresentalostopportunitytooptimizevalidation activities,reducetimetomarket,andincreaseyouroverallconfidenceinthesafetyandefficacyofyour products. KeyConcepts: Tomakesurethatwerealignedontheproperuseofthetermverification,hereareafewkeypoints fromtheFDAsDesignControlGuidanceForMedicalDeviceManufacturers: Definition:Verificationmeansconfirmationbyexaminationandprovisionofobjectiveevidence thatspecifiedrequirementshavebeenfulfilled[820.3(aa)]. TypesofVerificationActivities:Verificationactivitiesareconductedatallstagesandlevelsof devicedesign.Thebasisofverificationisathreeprongedapproachinvolving:tests,inspections, andanalyses. Anyapproachwhichestablishesconformancewithadesigninputrequirementisanacceptable meansofverifyingthedesignwithrespecttothatrequirement.Inmanycases,avarietyof approachesarepossible...themanufacturershouldselectandapplyappropriateverification techniquesbasedonthegenerallyacceptedpracticesforthetechnologiesemployedintheir products. Table1providessomeexamplesofthetypesofverificationactivitiesthatdevicemanufacturersoften employ.

Page3of12
MEDIcept,Inc. 200HomerAvenue Ashland,MA01721
11/2010 MaynotbereprintedorcopiedwithoutexpressedpermissionfromMEDIcept

DesignVerificationTheCaseforVerification,NotValidation
Table 1: Types of Verification Activities
o o o o Tests Material performance / fatigue tests Package integrity tests* Biocompatibility testing of materials* Bioburden testing of products to be sterilized* o o Inspections First article inspection Comparison of a design to a previous product having an established history of successful use* o Analyses Worst case analysis of an assembly to verify that components are derated properly and not subject to overstress during handling and use* Thermal analysis of an assembly to assure that internal or surface temperatures do not exceed specified limits* Fault tree analysis of a process or design* Failure modes and effects analysis* Engineering analyses: o Finite Element Analysis (FEA) o Computational Fluid Dynamics (CFD) o Tolerance stack-up

o o o

*Source:DesignControlGuidanceForMedicalDeviceManufacturers(1997) Akeydistinctionbetweendesignverificationanddesignvalidationactivitiesisthatverificationonly requiresthatasingleunitbeassessed.Whatconstitutesthatsingleunitwillvarydependingonthe intentoftheverification.Itmightbeonebatchofrawmaterial(formaterialperformancetests),one machinedpart(forfirstarticleinspection),ononepackagesample(forintegritytests).Theintentof verificationistoconfirmthatthedesignoutputs(i.e.,thematerialsorcomponentsspecifiedindesign documents)meetthedesigninputrequirements. Validations,ontheotherhand,requirethatstudiesbeconductedtoensurethatthedevicecanbe manufacturedtomeetdesignspecificationsonaconsistentbasis(i.e.,processvalidation),andtoensure thatthefinisheddeviceissafeandeffectiveforitsintendedpurpose(i.e.,designvalidation).Forthe processvalidation,multipledevicesmustbemanufacturedandevaluatedtoconfirmthatthe productionprocessiscapableofproducingdeviceswithinspecificationsonaconsistentbasis.Forthe designvalidation,multipledevicesmustbeusedtotreatmultiplepatientstoconfirmthatthetreatment issafeandeffective. Page4of12
MEDIcept,Inc. 200HomerAvenue Ashland,MA01721
11/2010 MaynotbereprintedorcopiedwithoutexpressedpermissionfromMEDIcept

DesignVerificationTheCaseforVerification,NotValidation
Thenumberofdevicesthatneedtobeproducedandthenumberofpatientsthatneedtobetreatedis afunctionoftheriskassociatedwiththeparticularaspectofthedeviceandtheneedtoestablish confidenceinthestudyresults.Aswewillsee,theeffectiveuseofriskanalysistoolsandsolid verificationresultscanhelptoreducethesizeofthevalidationstudieswithoutnegativelyaffectingthe confidenceinthestudyresults. WhatGoesWrong? Asdescribedabove,toooftendesignverificationdoesnotreceivetheamountofattentionneededto ensuresuccessinthelatterstagesofadevicedevelopmentprogram.Whilenotaperfectrepresentation ofwhatgoesoninalldevicecompanies,itisinterestingtolookattheFDAsdesigncontrolaudit findingstoseewhatproblemstheyhavefoundwithcompaniesdesignverificationprograms.Figure1, showsthenumberofwarninglettersintheFDAsdatabasethatincludefindingsrelatedtospecific sectionsoftheDesignControlregulation(21CFR820.30). Figure1:DesignControlWarningLetters

Page5of12
MEDIcept,Inc. 200HomerAvenue Ashland,MA01721
11/2010 MaynotbereprintedorcopiedwithoutexpressedpermissionfromMEDIcept

DesignVerificationTheCaseforVerification,NotValidation
Interestingly,themostobservationswerefoundwithregardtotheDesignChange,DesignValidation, andtheGeneraldesigncontrolsubheadings(theGeneralfindingssuggestanoverallfailureofthe companysdesigncontrolprocess).Wearguethatfailuresinthesethreeareasarelargelytheresultof poorperformanceintheearlierstagesoftheprogram(e.g.,validationfailuresduetopoorlystructured inputrequirements,outputsandincompleteverification;anddesignchangefailuresduetoan ineffectivesystemtoupdateverificationsandvalidationswhenchangesaremade). Oftheremainingcategories,mostwarningletterscite21CFR820.30(f)DesignVerification.Inthese64 warningletters,theFDAidentifiedthatthemanufacturerdidnotcompleteallrequiredverificationsin nearly45%ofthecases.Alackof(orsignificantgapsin)verificationprocedureswereidentifiedinabout 25%ofthewarningletters.Inaddition,outofspecificationresults,alackofrecordsintheDHF,anda lackofacceptancecriteriawerefoundin16%,14%,and14%ofthesecompanies,respectively.Figure2 providesabreakdownofallofthetypesofviolationsidentifiedinthesewarningletters. Figure2:TypesofDesignVerificationViolations

Page6of12
MEDIcept,Inc. 200HomerAvenue Ashland,MA01721
11/2010 MaynotbereprintedorcopiedwithoutexpressedpermissionfromMEDIcept

DesignVerificationTheCaseforVerification,NotValidation
Note:92separateviolationswereidentifiedinthe64warningletters.

Whyarethesefailuresoccurring?Onepossibilityisthatbyviewingverificationasaregulation,device developersarelosingsightofthefactthatconductingverificationsissimplygoodengineering.For example,itmakesnosensetomoveforwardwiththedevelopmentofacomponentbeforeyouaresure thatthematerialpropertiesmeettheperformanceandsafetyrequirementsestablishedinthedesign inputs. Thismeetingtheregulationsmindsetcanleadengineerstoviewdesignverificationsimplyasatask thatmustbecheckedoffbyQAinsteadofvaluabletoolforbuildingknowledgeabouttheirdevice. Failuresoccurbecausesometimesthecheckoffsgetmissed. Evenwhenalltheboxesarechecked,theregulationmindsetcandriveengineerstofocusonlythe minimumnumberofverificationactivitiesneededtosatisfythedesigncontrolrequirements.Whilethis approachmayhelpsatisfyshorttermbudgetorscheduleconstraints,thedevelopmentteamwilllose theopportunitytolearnpotentiallyimportantinformationabouttheperformanceandbehaviorofits deviceanditscomponents.Ifthatlearningisputofftothevalidationstage,thecostoffailurecangrow dramatically. WhatToDo? Thereareavarietyoftoolsthatdeveloperscanusetoimprovetheeffectivenessoftheirdesign verificationprocess.Inthispaperwediscussthreetoolsthatcanhelpensurethatverificationactivities areappropriateandcompleteand,ifwelldocumented,canprovideregulatorswithsufficient confidenceinthedesignthatexcessivevalidationcanbeavoided.Thethreetoolsare TraceabilityMatrix:Thetraceability(trace)matrixisabasicdesigncontroltoolthatalldevice developersshouldusethroughoutthedesigncontrolprocesstoestablishclearlinkagesbetweenDesign Inputs,Outputs,Verifications,ValidationsandRiskAnalyses.ToooftenthetracematrixisleftforQAto completejustintimeforthefinaldesignreview.However,ifitisbegunassoonasDesignInputsare approvedandmanagedthroughoutthedevelopmenteffort,thetracematrixprovidedanexcellent roadmapguidingdevelopersthroughkeystepsofthedesigncontrolprocessandensuringthat requireddocumentationiscreatedandcontrolled. TraceabiltyMatrix FailureModesandEffectsAnalysis(FMEA) FaultTreeofDesignInputs

Page7of12
MEDIcept,Inc. 200HomerAvenue Ashland,MA01721
11/2010 MaynotbereprintedorcopiedwithoutexpressedpermissionfromMEDIcept

DesignVerificationTheCaseforVerification,NotValidation
Toillustratehowthetracematrixcanbeused,Table2showswhatonerowinatracematrixmightlook likewhenthematrixisfirstdevelopedandapproved.Atthisstageofthedevelopmenteffort,thematrix issolelyalistofthedesigninputs(thereferencenumberfromtheProductRequirementsDocument, andadescription).Atthisstagethereisnoinformationabouthowthatdesigninputwillbesatisfied,but itprovidesaroadmapforwhatquestionsneedtobeaskedandwhatdocumentsneedtobedeveloped. Astheprojectproceedsthroughthedevelopstages,subsequentcellswillbefilledin,identifyinghow thedesigninputisbeingmet. Table2:TraceabilityMatrixatDesignInputsStage TraceabilityMatrix Design DesignInput Design Risk Verification Process Design Req. (Requirement) Output Analyses Validation Validation No. (Specification) 1.1.1 CanbeETO sterilized

Comments

Table3illustrateshowarowofatracematrixmightlookwhenbeingreviewedattheFinalDesign Review.Atthispointthedesignoutputshavebeenapprovedandtherequirementthatthematerialin questionisappropriateforETOsterilizationisestablishedinDrawingNo.1.2.Thematrixalsoidentifies thatidentifiedrisksassociatedwiththedevicehavebeenmitigated(inpart)throughtheuseofthis material.ThereferencestodesignFMEA1.3andprocessFMEA5.4identifytwoplaceswherethis materialisaddressed.Verificationwasachievedbyconfirmingthatthespecifiedmaterialisincludedin thepurchasingbillofmaterial(BOM2.3).TheProcessValidationcolumnidentifiesthatproductionof thematerialwasassessedinanoperationqualification(OQStudy3.5)andaperformancequalification (PQStudy4.3).Finally,itshowsthatadesignvalidation(SterilizationStudy2.1)wasconductedto confirmthatthefinalmanufactureddevicecouldinfactbeeffectivelysterilizedusingETO. Table3:TraceabilityMatrixatFinalDesignReview TraceabilityMatrix Design DesignInput (Requirement) Req. No. 1.1.1 CanbeETO sterilized

Design Output (Specification) MaterialA (Dwg1.2)

Risk Verification Process Design Analyses Validation Validation dFMEA 1.3 pFMEA 5.4 BOM2.3 OQStudy 3.5 PQStudy 4.3

Comments

Sterilization None Study2.1

Page8of12
MEDIcept,Inc. 200HomerAvenue Ashland,MA01721
11/2010 MaynotbereprintedorcopiedwithoutexpressedpermissionfromMEDIcept

DesignVerificationTheCaseforVerification,NotValidation
Thetracematrixnowservesasaclearrecordofthechainofanalysesandstudiesusedtoensurethat thespecificdesigninputismet.Allofthedocumentsreferencedinthetracematrixareincludedinyour designhistoryfile(DHF),soifaquestionarisesregardinganystageintheprocess,thetracematrix pointstothekeydocumentationdevelopedatthatstage. Whilethetracematrixisanecessaryandvaluabletool,itlargelyservesjustarecordkeepingrole.Itdoes nothingtoinformtheteamabouthowtoprioritizeverificationandvalidationefforts.Forthatinput,risk analysesareneeded. FMEA:Likethetracematrix,riskanalysesneedtobestartedearlyinthedesignanddevelopment processandfilledinandenhancedasprojectproceeds.Whilethispaperwillnotdiscussthedetailsof theapplicationofriskanalyses,andFMEAsinparticular,wewilladdressthelinkbetweenFMEAsand verification. Toooften,riskanalysesareconductedinisolationfromtheotherdesigncontrolactivities.Asdiscussed earlier,theregulationmindsetcanleadtotheperspectivethatriskanalysesarejustarequiredtaskto becompletedandnotastoolstosupportdecisionmakingwhichtheyare. FMEAsareconductedtoprovidedeveloperswithasharedunderstandingoftherisksassociatedwith theirdeviceusuallyfromthreeperspectives:design,use/application,andprocessing.Theresultofthis tool,asillustratedinFigure1,istheclassificationofriskstopeople,property,andtheenvironmentinto threecategories:BroadlyAcceptable,AsLowasReasonablyPracticle(ALARP),andIntolerable.Typically, applicationofthistoolfocusesontheIntolerablerisksandthedevelopmentofmitigationstrategiesto reducetheprobabilityofoccurrenceforthoserisks,bringingthemintotheALARPregion.

Page9of12
MEDIcept,Inc. 200HomerAvenue Ashland,MA01721
11/2010 MaynotbereprintedorcopiedwithoutexpressedpermissionfromMEDIcept

DesignVerificationTheCaseforVerification,NotValidation
Figure2:RiskClassification

Whileriskidentificationandmitigationisclearlytheprimaryobjectiveoftheriskmanagementactivity, theseanalysescanalsobeusedtofocusverificationandvalidationefforts.Forexample,considera devicethatincludestwocomponents,oneaffectshowthedeviceisheldbytheuserandtheother comesintocontactwiththepatient.Bothcomponentshaveasimilarnumberofphysicaldimensionsto verify,butrisksassociatedwiththeuserfacingcomponentareBroadlyAcceptable,whilethepatient facingcomponentrisksareALARP.WhileaFirstArticleinspectionoftheuserfacingcomponentmay beappropriate,amorethoroughanalysisofthepatientfacingcomponentmaybewarranted.This analysiscouldincludeatoleranceanalysisortheinspectionofmultiplepiecestoprovideabetter indicationofthepotentialcapabilitytoproducethecomponentwithinspecification. Whiletheadditionaleffortsdescribedabovemaynotberequiredattheverificationstage(conductinga FirstArticlewouldbesufficienttoclaimthattheverificationiscomplete),theALARPriskclassificationis anindicationthatvalidationofthiscomponentmaybechallenging.Themoreyoulearnaboutthis componentduringverification,thebetterpreparedyouwillbeonceyougettothevalidation.Iffact,if theverificationisthorough,itmayeliminatetheneedforsomeaspectsofvalidationsavingtimeand moneylaterinthedevelopmenteffort. Thekeypointisthatincompleteorminimalverificationsmaynotprovideasufficientunderstandingof thelikelyperformanceofadeviceduringvalidation,andexcessiveverificationcanbeawasteof resources.Toeffectivefocusyourverificationefforts,riskanalyses,andFMEAsinparticular,canprovide Page10of12
MEDIcept,Inc. 200HomerAvenue Ashland,MA01721
11/2010 MaynotbereprintedorcopiedwithoutexpressedpermissionfromMEDIcept

DesignVerificationTheCaseforVerification,NotValidation
aclearrationaleforhowtofocusyourtimeandresourcespotentiallyreducingvalidation requirements. FaultTreeAnalysis:WhileFMEAsareawellacceptedriskanalysistool,oneweaknessoftheapproachis thateachriskisconsideredinisolation.Thereisnoabilitytoassesstheriskoftwoindependentfailures (i.e.,dimensionAistooshortandtheuserappliestoomuchforce).AsillustratedinFigure3,Faulttree analysis(FTA)allowsyoutoconsidertheriskoftwoindependentfailuresoccurringinparallel(the AND),andtheriskofanyoneofaseriesoffailures(theOR).Forexample,inthisBadCoffee example,addingOldCreamrequiresthecreamtobepastitsexpirationdateANDforthecoffee drinkernottoreadthedate.Unsatisfactoryservingtemperaturecouldbeduetothecoffeebeingeither toohotORtoocold. Figure3:FaultTreeAnalysis

WhilemorechallengingtostructureandassessthananFMEA,theFTAprovidesabetterunderstanding oftherisksassociatedwiththesystem.Sincetheperformanceofthesystemistheobjectiveof validationactivities,FTAisanidealtooltoprepareforvalidations.Withthetopofthefaulttree representingtheintendeduseofthedevice,thistoolallowsdevicedeveloperstofilloutthetree

Page11of12
MEDIcept,Inc. 200HomerAvenue Ashland,MA01721
11/2010 MaynotbereprintedorcopiedwithoutexpressedpermissionfromMEDIcept

DesignVerificationTheCaseforVerification,NotValidation
identifyingthatrangeofdesignorusagefaultsthatcouldleadtoaproblem.JustastheFMEAhelpedto focusattentiononindependentrisks,theFTAhelpstofocusonsystemrisks. Oncethebranchesofatreeassociatedwithasignificantriskareidentified,additionalverification resourcescanbefocusedonverifyingthedesignofcomponentslinkedtothatrisk.Again,theobjective ofverificationandvalidationactivitiesistobuildconfidencewithinyourorganizationandwith regulatorsthatyourdeviceisgoingtobesafeandeffective.AwellstructureFTAcombinedwith thoroughverificationsofkeycomponentscanbeinstrumentalintheidentificationandresolutionof problemsearlyinthedevelopmentprocess,allowingyourteamtoenterintothevalidationphaseofthe processwithconfidenceintheperformanceofthedeviceandtofocusitsvalidationonthosesystem elementsthatmostdirectlyaffectuserneeds. WhataretheBenefits? Whileitcansometimesgetlostinthechurningofengineeringprocess,verificationisacriticalelement ofthedesigncontrolprocess.Whilemeetingthethresholdrequirementofdocumentingaverification foreachdesigninputmayhelptomovethedesignthroughthedevelopmentstages,suchanapproach doesnotprovidethevaluethatastrongverificationeffortcouldprovide.Bydevelopingatracematrix toensurethatalldesigninputsareproperlyaddressedandleveragingriskanalysistools,medicaldevice developerswouldbebetterabletofocusscarceresourcesonthoseverificationactivitiesthatwill providethegreatestbenefit.Wellstructuredverificationactivitiesprovidethefoundationfor validations.Iftheverificationsaresound,validationscanbebetterfocused,helpingtoreducethescope oftheseactivitiesreducingvalidationcostsandacceleratingtimetomarket.

Page12of12
MEDIcept,Inc. 200HomerAvenue Ashland,MA01721
11/2010 MaynotbereprintedorcopiedwithoutexpressedpermissionfromMEDIcept