Вы находитесь на странице: 1из 72

02Feb2010

WhatisRedis?
Presentedby,

SearleOliveira

DatabaseAdministrator Globo.com

Aboutme
DBAatGlobo.com CertifiedMySQLAssociateID222572032 LongtimeLinux/MySQLuser PHPOOPDeveloper BeginnerCprogramming RedisAdminProjectOwner

Schedule
WhatisRedis? RedisxMemcached RedisPersistence RedisReplication Sharding Benchmark SupportedLanguages QuickStart Commands StudyCase:twitter RedisAdmin

WhatisRedis?
Redisisaadvancedpersistentkeyvaluedatabase,bySalvatore AntirezSanfilippo,whereeverykeyisassociatedwithavalue. Forexample,Setthekey"surname_1992"tothestring "Smith".
> SET surname_1992 Smith

Inotherwords,youcanlookatRedisasadatastructures server. Theessenceofakeyvaluestoreistheabilitytostoresome data,calledvalue,insideakey.


RedisxMemcached
Itissimilartomemcachedbutthedatasetisnotvolatile. Valuescanbestrings,exactlylikeinmemcached,butalso lists,sets,andorderedsets. Alldatatypescanbemanipulatedwithatomicoperations.

RedisisfreesoftwarereleasedundertheveryliberalBSD license.

RedisxConventionalKV
1. Redisencouragesacolumnorientedstyleof programmingdatastorage. NoTransactions NoACID 2. MostDataGridsencourageanentityorientedstyle: Transational ACID AMapusuallyisabusinessobjectordatabasetable. ConstrainedTreeSchemasaretypical.

DataTypes
WhatmakesRedisdifferentfrommanyotherkeyvalue storages,isthateverysinglevaluehasatype.The followingtypesaresupported: Strings Lists Sets SortedSets

DataTypes
Thetypeofavaluedetermineswhatoperations (commands)areavailableforthevalueitself. Forexample,youcanappendelementstoaliststoredat thekey"mypostlist"usingtheLPUSHorRPUSH command: > LPUSH mypostlist Hello World! Eachcommandisperformedthroughserversideatomic operations.

Datainmemory,butsavedondisk
Redisloadsandmantainsthewholedatasetintomemory, butthedatasetispersistent,sinceatthesametimeitis savedondisk,sothatwhentheserverisrestarteddata canbeloadedbackinmemory. Therearetwokindofpersistencesupported:semi persistentmode(Snapshotting)andfullypersistentmode (AppendOnlyFile).

RedisPersistences
Redisisveryfastbutatthesametimepersistentthe wholedatasetistakeninmemory.

Semipersistentmode

timetotimesavedataondiscasynchronously. Fullypersistentmode alternativelyeverychangeiswrittenintoanappendonly file.


SemiPersistentMode
InthismodeRedis,fromtimetotime,writesadumpon diskasynchronously.Thedatasetisloadedfromthe dumpeverytimetheserveris(re)started. Rediscanbeconfiguredtosavethedatasetwhena certainnumberofchangesisreachedandafteragiven numberofsecondselapses. Becausedataiswrittenasynchronously,whenasystem crashoccurs,thelastfewqueriescangetlost(thatis acceptableinmanyapplicationsbutnotinall).

FullyPersistentMode
ThismodeiscalledAppendOnlyFile,whereevery commandreceivedalteringthedataset(sonotareadonly command,butawritecommand)iswrittenonanappend onlyfileASAP. Thiscommandsarereplayedwhentheserverisrestarted inordertorebuildthedatasetinmemory. AppendOnlyFilesupportsaveryhandyfeature:theserver isabletosafelyrebuildtheappendonlyfileinbackground inanonblockingfashionwhenitgetstoolong.

It'spersistentbutsupportsexpires
Rediscanbeusedasamemcachedonsteroidsbecauseis asfastasmemcachedbutwithanumberoffeatures more. Likememcached,Redisalsosupportssettingtimeouts tokeyssothatthiskeywillbeautomaticallyremoved whenagivenamountoftimepasses.

anumberoffeaturesmore

RedisReplication
Whateverwillbethepersistencemodeyou'lluseRedis supportsmasterslavereplicationsifyouwanttostay reallysafeorifyouneedtoscaletohugeamountsofreads. RedisReplicationistrivialtosetup.Sotrivialthatallyou needtodoinordertoconfigureaRedisservertobeaslave ofanotheronewithautomaticsynchronization. slaveof 192.168.1.100 6379

Sharding
DistributingthedatasetacrossmultipleRedisinstancesiseasy inRedis,asinanyotherkeyvaluestorage.Andthisdepends basicallyontheLanguagesclientlibrariesbeingabletodoso. Ashardisamethodofhorizontalpartitioninginadatabase orsearchengine. Horizontalpartitioningisadesignprinciplewherebyrows ofadatabasetableareheldseparately,ratherthansplitting bycolumns(asfornormalization).Eachpartitionformspart ofashard,whichmayinturnbelocatedonaseparate databaseserverorphysicallocation.

Multipledatabasessupport
Redissupportsmultipledatabaseswithcommandsto atomicallymovekeysfromonedatabasetotheother. BydefaultDB0isselectedforeverynewconnection,but usingtheSELECTcommanditispossibletoselect/create adifferentdatabase. TheMOVEoperationcanmoveanitemfromoneDBto anotheratomically.

HowFastisRedis?
Redisincludestheredisbenchmarkutilitythatsimulates SETs/GETsdonebyNclientsatthesametimesendingMtotal queries(itissimilartotheApache'sabutility). Belowyou'llfindthefulloutputofthebenchmarkexecuted againstaLinuxbox. 50simultaneousclientsperforming100.000requests. SETandGETisa256bytesstring. RunningLinux2.6,it'sXeonX33202.5Ghz. Textexecutedusingtheloopbackinterface(127.0.0.1).

BenchmarkResults

About110.000SETspersecond About81.000GETspersecond.
======SET====== 100.007requestscompletedin0.88seconds 50parallelclients 3bytespayload keepalive:1 58.50%<=0milliseconds 99.17%<=1milliseconds 99.58%<=2milliseconds 99.85%<=3milliseconds 99.90%<=6milliseconds 100.00%<=9milliseconds 114.293.71requestspersecond

======GET====== 100.000requestscompletedin1.23seconds 50parallelclients 3bytespayload keepalive:1 43.12%<=0milliseconds 96.82%<=1milliseconds 98.62%<=2milliseconds 100.00%<=3milliseconds 81.234.77requestspersecond

Supportedlanguages
Ruby Python PHP Erlang Tcl Perl Lua Java Scala Clojure C# C Javascript

WhoisusingRedis?
EngineYard Github Vidiowiki WishInternetConsulting RubyMinds Boxcar(iphoneapplicationforpushnotifications) LLOOGG VirgilioFilm(Italianmoviescommunity)

Globo.comisnotusingRedis,yet!

QuickStart
Thisquickstartisafiveminuteshowtoonhowtoget startedwithRedis. ThelateststablesourcedistributionofRedis:
$ wget http://redis.googlecode.com/files/redis-1.02.tar.gz

RediscanbecompiledinmostPOSIXsystems.To compileRedisjustuntarthetar.gz,enterthedirectlyand type'make'.


$ tar xvzf redis-1.02.tar.gz $ cd redis-1.02 $ make

Runtheserver
Rediscanrunjustfinewithoutaconfigurationfile(when executedwithoutaconfigfileastandardconfigurationis used).TorunRedisjusttypethefollowingcommand:
$ ./redis-server

WiththedefaultconfigurationRediswilllogtothe standardoutputsoyoucancheckwhathappens.

Playwiththebuiltinclient
Redisshipswithacommandlineclientthatisautomatically compiledwhenyouranmakeanditiscalledrediscli. Forinstancetosetakeyandreadbackthevalueusethe following:
$ ./redis-cli set mykey somevalue OK $ ./redis-cli get mykey somevalue

RedisCommandReference
1. 2. 3. 4. 5. 6. 7. 8. 9.

Connectionhandling Commandsoperatingonallthekindofvalues Commandsoperatingonstringvalues Commandsoperatingonlists Commandsoperatingonsets Commandsoperatingonsortedsets Sorting Persistencecontrolcommands Remoteservercontrolcommands

1.Connectionhandling
QUITclosetheconnection
Ask the server to silently close the connection.

AUTHsimplepasswordauthenticationifenabled
Request for authentication in a password protected Redis server. A Redis server can be instructed to require a password before to allow clients to issue commands. This is done using the requirepass directive in the Redis configuration file.

2.CommandsoperatingonAll
EXISTS keyTestifakeyexists DEL keyDeleteakey TYPE keyReturnthetypeofthevaluestoredatkey KEYS patternReturnallthekeysmatchingagivenpattern RANDOMKEYReturnarandomkeyfromthekeyspace
EXPIRESetatimetoliveinsecondsonakey TTLGetthetimetoliveinsecondsofakey

2.1CommandsoperatingonAll
RENAME oldname newnameRenametheoldkeyinthenew one,destroingthenewnamekeyifitalreadyexists RENAMENX oldname newnameRenametheoldkeyinthe newone,ifthenewnamekeydoesnotalreadyexist

2.2CommandsoperatingonAll
DBSIZEReturnthenumberofkeysinthecurrentdb SELECT indexSelecttheDBhavingthespecifiedindex MOVE key dbindexMovethekeyfromthecurrentlyselected DBtotheDBhavingasindexdbindex FLUSHDBRemoveallthekeysofthecurrentlyselectedDB FLUSHALLRemoveallthekeysfromallthedatabases

3.CommandsoperatingonString
SET key value Setakeytoastringvalue GET key Returnthestringvalueofthekey GETSET key valueSetakeytoastringreturningtheold valueofthekey MGET key1 key2 ... keyN Multiget,returnthestrings valuesofthekeys SETNX key valueSetakeytoastringvalueifthekeydoes notexist

3.1CommandsoperatingonString
MSET key1 value1 key2 value2 ... keyN valueN Setamultiplekeystomultiplevaluesinasingleatomicoperation
MSETNX key1 value1 key2 value2 ... keyN valueN Setamultiplekeystomultiplevaluesinasingleatomicoperation ifnoneofthekeysalreadyexist

3.2CommandsoperatingonString
INCR keyIncrementtheintegervalueofkey INCRBY key integerIncrementtheintegervalueofkeyby integer DECR keyDecrementtheintegervalueofkey DECRBY key integerDecrementtheintegervalueofkeyby integer

4.CommandsoperatingonLists
RPUSH key valueAppendanelementtothetailoftheList valueatkey LPUSH key valueAppendanelementtotheheadoftheList valueatkey LLEN keyReturnthelengthoftheListvalueatkey LRANGE key start endReturnarangeofelementsfromthe Listatkey LTRIM key start endTrimthelistatkeytothespecified rangeofelements

4.1CommandsoperatingonLists
LINDEX key indexReturntheelementatindexpositionfrom theListatkey LSET key index valueSetanewvalueastheelementat indexpositionoftheListatkey LREM key count valueRemovethefirstN,lastN,orall theelementsmatchingvaluefromtheListatkey LPOP keyReturnandremove(atomically)thefirstelementof theListatkey

4.2CommandsoperatingonLists
RPOP keyReturnandremove(atomically)thelastelementof theListatkey BLPOP key1 key2 ... keyN timeoutBlockingLPOP BRPOP key1 key2 ... keyNtimeoutBlockingRPOP RPOPLPUSH srckey dstkeyReturnandremove (atomically)thelastelementofthesourceListstoredat_srckey_ andpushthesameelementtothedestinationListstoredat _dstkey_

5.CommandsoperatingonSets
SADD key member AddthespecifiedmembertotheSetvalue atkey SREM key memberRemovethespecifiedmemberfromtheSet valueatkey SPOP keyRemoveandreturn(pop)arandomelementfromthe Setvalueatkey SMOVE srckey dstkey member Movethespecified memberfromoneSettoanotheratomically

5.1CommandsoperatingonSets
SCARD keyReturnthenumberofelements(thecardinality)of theSetatkey SISMEMBER key memberTestifthespecifiedvalueisa memberoftheSetatkey SINTER key1 key2 ... keyNReturntheintersection betweentheSetsstoredatkey1,key2,...,keyN SINTERSTORE dstkey key1 key2 ... keyN Compute theintersectionbetweentheSetsstoredatkey1,key2,...,keyN, andstoretheresultingSetatdstkey

5.2CommandsoperatingonSets
SUNION key1 key2 ... keyNReturntheunionbetween theSetsstoredatkey1,key2,...,keyN SUNIONSTORE dstkey key1 key2 ... keyNCompute theunionbetweentheSetsstoredatkey1,key2,...,keyN,and storetheresultingSetatdstkey SMEMBERS keyReturnallthemembersoftheSetvalueatkey SRANDMEMBER keyReturnarandommemberoftheSetvalue atkey

5.3CommandsoperatingonSets
SDIFF key1 key2 ... keyNReturnthedifferencebetween theSetstoredatkey1andalltheSetskey2,...,keyN SDIFFSTORE dstkey key1 key2 ... keyNCompute thedifferencebetweentheSetkey1andalltheSetskey2,..., keyN,andstoretheresultingSetatdstkey

6.CommandsoperatingonSortedSets
ZADD key score memberAddthespecifiedmembertothe SortedSetvalueatkeyorupdatethescoreifitalreadyexist ZREM key memberRemovethespecifiedmemberfromthe SortedSetvalueatkey ZINCRBY key increment memberIfthememberalready existsincrementitsscoreby_increment_,otherwiseaddthe membersetting_increment_asscore

6.1CommandsoperatingonSortedSets
ZRANGE key start endReturnarangeofelementsfromthe sortedsetatkey ZREVRANGE key start endReturnarangeofelements fromthesortedsetatkey,exactlylikeZRANGE,butthesorted setisorderedintraversedinreverseorder,fromthegreatesttothe smallestscore ZRANGEBYSCORE key min maxReturnalltheelementswith score>=minandscore<=max(arangequery)fromthesorted set

6.2CommandsoperatingonSortedSets
ZCARD keyReturnthecardinality(numberofelements)ofthe sortedsetatkey ZSCORE key elementReturnthescoreassociatedwiththe specifiedelementofthesortedsetatkey ZREMRANGEBYSCORE key min max Removeallthe elementswithscore>=minandscore<=maxfromthesortedset

7.Sorting
SORT key [BY pattern] [LIMIT start count] [GET pattern] [ASC|DESC] [ALPHA] [STORE dstkey] SorttheelementscontainedintheList,Set,orSortedSetvalueat key.Bydefaultsortingisnumericwithelementsbeing comparedasdoubleprecisionfloatingpointnumbers.Theseare thesimplestformofSORT: > SORT mylist DESC > SORT mylist LIMIT 0 10 > SORT mylist LIMIT 0 10 ALPHA DESC

8.Persistencecontrolcommands
SAVESynchronouslysavetheDBondisk BGSAVE AsynchronouslysavetheDBondisk LASTSAVE ReturntheUNIXtimestampofthelastsuccessfully savingofthedatasetondisk SHUTDOWN SynchronouslysavetheDBondisk,thenshutdown theserver BGREWRITEAOFRewritetheappendonlyfileinbackground whenitgetstoobig

9.Remoteservercontrolcommands
INFOProvideinformationandstatisticsabouttheserver MONITORDumpallthereceivedrequestsinrealtime SLAVEOFChangethereplicationsettings

StudyCase:twitter
DesignandimplementationofasimpleTwittercloneusingonly theRediskeyvaluestore. Wedon'thavetables,sowhatshouldbedesigned?Weneedto identifywhatkeysareneededtorepresentourobjectsandwhat kindofvaluesthiskeysneedtohold. Let'sstartfromUsers.Weneedtorepresentthisusersofcourse, withtheusername,userid,password,followersandfollowing users,andsoon. Thefirstquestionis,whatshouldidentifyanuserinsideour system?

StudyCase:twitter
> > > > SET global:nextUserId 1000 INCR global:nextUserId => 1001 SET uid:1001:username searleoliveira SET uid:1001:password e10adc3949ba59abbe56e057f20f883e
*atomicINCRoperation

Weusetheglobal:nextUserIdkeyinordertoalwaysgetan uniqueIDforeverynewuser.ThenweusethisuniqueIDto populatealltheotherkeysholdingouruserdata. ThisisaDesignPatternwithkeyvaluesstores! Keepitinmind.


StudyCase:twitter
SometimesitcanbeusefultobeabletogettheuserIDfromthe username,sowesetthiskeytoo: > SET username:searleoliveira:uid 1001
Thismayappearstrangeatfirst,butrememberthatweareonly abletoaccessdatabykey! It'snotpossibletotellRedistoreturnthekeythatholdsaspecific value. Thisnewparadigmisforcingustoorganizethedatasothat everythingisaccessiblebyprimarykey,speakingwithrelational DBslanguage.

StudyCase:twitter
Following,followersandupdates Everyuserhasfollowersusersandfollowingusers.Wehavea perfectdatastructureforthiswork! We'llneedtoaccessthisdatainchronologicalorderlater,from the mostrecentupdatetotheolderones,sotheperfectkindofValue forthisworkisaList.
> > > > LPUSH LPUSH LPUSH LPUSH uid:1001:followers 1002 uid:1001:followers 1003 uid:1001:followers...Setofuidsofallthefollowersusers uid:1001:following...Setofuidsofallthefollowingusers

StudyCase:twitter
Posts Anotherimportantthingweneedisaplacewerewecanaddthe poststodisplayintheuserhomepage.
> LPUSH uid:1001:posts Hey, I am Superman!

*aListofpostsids,everynewpostisLPUSHedhere.

StudyCase:twitter
Authentication We'llhandleauthenticationinasimplebutrobustway:wedon't wanttousePHPsessionsorotherthingslikethis,oursystemmust bereadyinordertobedistributedamongdifferentservers,so we'lltakethewholestateinourRedisdatabase.
> SET uid:1001:auth fea5e81ac8ca77622bed1c2132a021f9 > SET auth:fea5e81ac8ca77622bed1c2132a021f9 1001

StudyCase:twitter
Authentication Toauthenticateanuserwe'lldothissimplework: Gettheusernameandpasswordviatheloginform; Checkiftheusername:<username>:uidkeyactuallyexists; Ifitexistswehavetheuserid,(i.e.1001); Checkifuid:1001:passwordmatches,ifnot,errormessage; 5.Okauthenticated!Set"fea5e81ac8ca77622bed1c2132a021f9" (thevalueofuid:1001:auth)as"auth"cookie. Thishappenseverytimetheuserslogin.

StudyCase:twitter
Logout Theonlythingit'smissingfromalltheauthenticationisthe logout. Whatwedoonlogout? That'ssimple,we'lljustchangetherandomstringin uid:1001:auth. removetheoldauth:<oldauthstring> addanewauth:<newauthstring>.

StudyCase:twitter
Updates Updates,alsoknownasposts,areevensimpler.Inordertocreate anewpostonthedatabasewedosomethinglikethis:
> INCR global:nextUpdatesId => 10343 > SET updates:10343 "$owner_id|$time|I am Superman!"

Asyoucansetheuseridandtimeofthepostarestoreddirectlyinside thestring,wedon'tneedtolookupbytimeoruseridintheexample applicationsoitisbettertocompacteverythinginsidethepoststring. Afterwecreateapostweobtainthepostid.WeneedtoLPUSHthis postidineveryuserthat'sfollowingtheauthorofthepost,andofcourse inthelistofpostsoftheauthor.

StudyCase:twitter
Paginatingupdates NowitshouldbeprettyclearhowwecanuserLRANGEinorder togetrangesofposts,andrenderthispostsonthescreen:
> LRANGE uid:1001:posts 0 10 > LRANGE uid:1001:posts 10 20 > LRANGE uid:1001:posts 0 -1

ColumnOrientedStyle
Oneglobalmapwithacommonkeyspace Redisapplicationsstoreanentityusingattributes: R.set(U:123:firstname,Searle) R.set(U:123:lastname,Oliveira) R.set(U:123:password,AdE4Erf) R.set(U:123:username,searleoliveira) R.set(U:uid:searleoliveira,123) Eachnamedattributeoftheentitycombinedwiththeentity keybecomesakeyfortheentriesforthecorresponding values. Spaceconsumedisaconcernthough!

Makingithorizontallyscalable
Probablyyouwillnotneedmorethanoneserverforalotof applications,evenwhenyouhavealotofusers. Butlet'sassumeweareTwitterandneedtohandleahugeamount oftraffic. Whattodo? Thefirstthingtodoistohashthekeyandissuetherequeston differentserversbasedonthekeyhash.Thegeneralideaisthat youcanturnyourkeyintoanumber,andthantakethereminder of thedivisionofthisnumberbythenumberofserversyouhave:

server_id = crc32(key) % number_of_servers

Specialkeys
Forexampleeverytimewepostanewmessage,weneedto incrementtheglobal:nextPostIdkey.Howtofixthisproblem?A Singleserverwillgetalotifincrements.Thesimplestwayto handlethisistohaveadedicatedserverjustforincrements.This isprobablyanoverkillbtwunlessyouhavereallyalotoftraffic. Thereisanothertrick.TheIDdoesnotreallyneedtobean incrementalnumber,butjustitneedstobeunique.Soyoucan getarandomstringlongenoughtobeunlikely(almost impossible, ifit'smd5size)tocollide,andyouaredone.Wesuccessfully eliminatedourmainproblemtomakeitreallyhorizontally scalable!

Features

Speed

Persistence SupportforDataStructures AtomicOperations VarietyofSupportedLanguages Master/SlaveReplication Sharding SimpletoInstall,SetupandManage Portable LiberalLicensing

WhatisRedisAdmin?
RedisAdmin,orReAdmin,isaopensource webinterfacetotheAdministrationofRedis. ReAdminisfullywritteninPHPusingRedis, ofcourse. Thecurrentversionsupportscreateschemas, addkeysstrings,listsandsetsonly.Sorted setsisnotsupportedyet. Supportspersistencecontrol,commandquery editorandaserverstatusondashboard. ReAdminwascreatedbySearleOliveira.

Requirements
WebServerAwebserver,withRewritemodule,isneededtoinstallReAdmin on(e.g.,Apache,IISetc.) .htaccessRuleThismoduleusesarulebasedrewritingengine(basedona regularexpressionparser)torewriterequestedURLsonthefly.Availablein Apache1.2andlater. PHP5.2.0,ornewer,withtheStandardPHPLibrary(SPL)extensionenabled. Redis1.2.0ornewer WebbrowserAnywebbrowserwithcookiesenabled.

HowtoInstall?
DownloadChoosealaststableversionfromthedownloadspage. ExtractfilesUntarthestablepackageinyourwebserver's documentroot(e.g.,/var/www/). ConfigurationSimplyuseaplaintexteditortoeditafilenamed config.phpinthemain(toplevel)ReAdmindirectory(theone that containsindex.php). FinishForApacheyoucanusesupplied.htaccessfileinthat folder,forotherwebservers,youshouldconfigurethisyourself.

WhatisSchemas?
RedisdonotusetheSchemasconcept. RedisusesDBinstances(0,1,2,3...15). So,toorganizethisinstancesandmakemoreeasyforhuman,we createdanaliastothem.ThesealiasareSchemas. Db0 Db1 Db2 Db3

=> => => =>

facebook youtube twitter google

KnowLimitations
SortedSetscommands SortedSetsisnotsupportedyet. DBinstances TheDBinstances,inRedis,arelimitedtonumbersbecausethey arejustanaccessorytoworkwithasingledataset.Wecancreate only15DBinstances(from0to14),currently.TheDB15 instance isreservedtoReAdmininfo_schema.

Screenshot:Login

Screenshot:Dashboard

Screenshot:ShowKeys

Screenshot:AddKeyString

Screenshot:PersistenceControl

Screenshot:Command

TwitterUpdates
@phpae#ec2:redisadmin http://phpappengine.com/2010/opensource/redisadmin @nginx_tipsRedisAdminDownload:readmin0.0.6test.tar.gz(103KB) http://goo.gl/fb/KsMB @abrdevopensourcewebinterfaceExcellentOpensourcewebadminpanel toredis!http://code.google.com/p/redisadmin @antirezThiscouldbeaninterestingproject,Redisadmin: http://code.google.com/p/redisadmin/

ReadMore
RedisProject
http://code.google.com/p/redis

ReAdminProject

http://code.google.com/p/redisadmin

Shard(databasearchitecture)

http://en.wikipedia.org/wiki/Shard_(database_architecture)

QConSanFrancisco2009,BillyNewport

http://www.infoq.com/presentations/newportevolvingkeyvalueprogrammingmodel

Q&A
SearleOliveira
searle@corp.globo.com @searleoliveira Y!searleoliveira

Вам также может понравиться