Академический Документы
Профессиональный Документы
Культура Документы
WhatisRedis?
Presentedby,
SearleOliveira
DatabaseAdministrator Globo.com
Aboutme
DBAatGlobo.com CertifiedMySQLAssociateID222572032 LongtimeLinux/MySQLuser PHPOOPDeveloper BeginnerCprogramming RedisAdminProjectOwner
Schedule
WhatisRedis? RedisxMemcached RedisPersistence RedisReplication Sharding Benchmark SupportedLanguages QuickStart Commands StudyCase:twitter RedisAdmin
WhatisRedis?
Redisisaadvancedpersistentkeyvaluedatabase,bySalvatore AntirezSanfilippo,whereeverykeyisassociatedwithavalue. Forexample,Setthekey"surname_1992"tothestring "Smith".
> SET surname_1992 Smith
RedisxMemcached
Itissimilartomemcachedbutthedatasetisnotvolatile. Valuescanbestrings,exactlylikeinmemcached,butalso lists,sets,andorderedsets. Alldatatypescanbemanipulatedwithatomicoperations.
RedisisfreesoftwarereleasedundertheveryliberalBSD license.
RedisxConventionalKV
1. Redisencouragesacolumnorientedstyleof programmingdatastorage. NoTransactions NoACID 2. MostDataGridsencourageanentityorientedstyle: Transational ACID AMapusuallyisabusinessobjectordatabasetable. ConstrainedTreeSchemasaretypical.
DataTypes
WhatmakesRedisdifferentfrommanyotherkeyvalue storages,isthateverysinglevaluehasatype.The followingtypesaresupported: Strings Lists Sets SortedSets
DataTypes
Thetypeofavaluedetermineswhatoperations (commands)areavailableforthevalueitself. Forexample,youcanappendelementstoaliststoredat thekey"mypostlist"usingtheLPUSHorRPUSH command: > LPUSH mypostlist Hello World! Eachcommandisperformedthroughserversideatomic operations.
Datainmemory,butsavedondisk
Redisloadsandmantainsthewholedatasetintomemory, butthedatasetispersistent,sinceatthesametimeitis savedondisk,sothatwhentheserverisrestarteddata canbeloadedbackinmemory. Therearetwokindofpersistencesupported:semi persistentmode(Snapshotting)andfullypersistentmode (AppendOnlyFile).
RedisPersistences
Redisisveryfastbutatthesametimepersistentthe wholedatasetistakeninmemory.
Semipersistentmode
SemiPersistentMode
InthismodeRedis,fromtimetotime,writesadumpon diskasynchronously.Thedatasetisloadedfromthe dumpeverytimetheserveris(re)started. Rediscanbeconfiguredtosavethedatasetwhena certainnumberofchangesisreachedandafteragiven numberofsecondselapses. Becausedataiswrittenasynchronously,whenasystem crashoccurs,thelastfewqueriescangetlost(thatis acceptableinmanyapplicationsbutnotinall).
FullyPersistentMode
ThismodeiscalledAppendOnlyFile,whereevery commandreceivedalteringthedataset(sonotareadonly command,butawritecommand)iswrittenonanappend onlyfileASAP. Thiscommandsarereplayedwhentheserverisrestarted inordertorebuildthedatasetinmemory. AppendOnlyFilesupportsaveryhandyfeature:theserver isabletosafelyrebuildtheappendonlyfileinbackground inanonblockingfashionwhenitgetstoolong.
It'spersistentbutsupportsexpires
Rediscanbeusedasamemcachedonsteroidsbecauseis asfastasmemcachedbutwithanumberoffeatures more. Likememcached,Redisalsosupportssettingtimeouts tokeyssothatthiskeywillbeautomaticallyremoved whenagivenamountoftimepasses.
anumberoffeaturesmore
RedisReplication
Whateverwillbethepersistencemodeyou'lluseRedis supportsmasterslavereplicationsifyouwanttostay reallysafeorifyouneedtoscaletohugeamountsofreads. RedisReplicationistrivialtosetup.Sotrivialthatallyou needtodoinordertoconfigureaRedisservertobeaslave ofanotheronewithautomaticsynchronization. slaveof 192.168.1.100 6379
Sharding
DistributingthedatasetacrossmultipleRedisinstancesiseasy inRedis,asinanyotherkeyvaluestorage.Andthisdepends basicallyontheLanguagesclientlibrariesbeingabletodoso. Ashardisamethodofhorizontalpartitioninginadatabase orsearchengine. Horizontalpartitioningisadesignprinciplewherebyrows ofadatabasetableareheldseparately,ratherthansplitting bycolumns(asfornormalization).Eachpartitionformspart ofashard,whichmayinturnbelocatedonaseparate databaseserverorphysicallocation.
Multipledatabasessupport
Redissupportsmultipledatabaseswithcommandsto atomicallymovekeysfromonedatabasetotheother. BydefaultDB0isselectedforeverynewconnection,but usingtheSELECTcommanditispossibletoselect/create adifferentdatabase. TheMOVEoperationcanmoveanitemfromoneDBto anotheratomically.
HowFastisRedis?
Redisincludestheredisbenchmarkutilitythatsimulates SETs/GETsdonebyNclientsatthesametimesendingMtotal queries(itissimilartotheApache'sabutility). Belowyou'llfindthefulloutputofthebenchmarkexecuted againstaLinuxbox. 50simultaneousclientsperforming100.000requests. SETandGETisa256bytesstring. RunningLinux2.6,it'sXeonX33202.5Ghz. Textexecutedusingtheloopbackinterface(127.0.0.1).
BenchmarkResults
About110.000SETspersecond About81.000GETspersecond.
======SET====== 100.007requestscompletedin0.88seconds 50parallelclients 3bytespayload keepalive:1 58.50%<=0milliseconds 99.17%<=1milliseconds 99.58%<=2milliseconds 99.85%<=3milliseconds 99.90%<=6milliseconds 100.00%<=9milliseconds 114.293.71requestspersecond
======GET====== 100.000requestscompletedin1.23seconds 50parallelclients 3bytespayload keepalive:1 43.12%<=0milliseconds 96.82%<=1milliseconds 98.62%<=2milliseconds 100.00%<=3milliseconds 81.234.77requestspersecond
Supportedlanguages
Ruby Python PHP Erlang Tcl Perl Lua Java Scala Clojure C# C Javascript
WhoisusingRedis?
EngineYard Github Vidiowiki WishInternetConsulting RubyMinds Boxcar(iphoneapplicationforpushnotifications) LLOOGG VirgilioFilm(Italianmoviescommunity)
Globo.comisnotusingRedis,yet!
QuickStart
Thisquickstartisafiveminuteshowtoonhowtoget startedwithRedis. ThelateststablesourcedistributionofRedis:
$ wget http://redis.googlecode.com/files/redis-1.02.tar.gz
Runtheserver
Rediscanrunjustfinewithoutaconfigurationfile(when executedwithoutaconfigfileastandardconfigurationis used).TorunRedisjusttypethefollowingcommand:
$ ./redis-server
WiththedefaultconfigurationRediswilllogtothe standardoutputsoyoucancheckwhathappens.
Playwiththebuiltinclient
Redisshipswithacommandlineclientthatisautomatically compiledwhenyouranmakeanditiscalledrediscli. Forinstancetosetakeyandreadbackthevalueusethe following:
$ ./redis-cli set mykey somevalue OK $ ./redis-cli get mykey somevalue
RedisCommandReference
1. 2. 3. 4. 5. 6. 7. 8. 9.
1.Connectionhandling
QUITclosetheconnection
Ask the server to silently close the connection.
AUTHsimplepasswordauthenticationifenabled
Request for authentication in a password protected Redis server. A Redis server can be instructed to require a password before to allow clients to issue commands. This is done using the requirepass directive in the Redis configuration file.
2.CommandsoperatingonAll
EXISTS keyTestifakeyexists DEL keyDeleteakey TYPE keyReturnthetypeofthevaluestoredatkey KEYS patternReturnallthekeysmatchingagivenpattern RANDOMKEYReturnarandomkeyfromthekeyspace
EXPIRESetatimetoliveinsecondsonakey TTLGetthetimetoliveinsecondsofakey
2.1CommandsoperatingonAll
RENAME oldname newnameRenametheoldkeyinthenew one,destroingthenewnamekeyifitalreadyexists RENAMENX oldname newnameRenametheoldkeyinthe newone,ifthenewnamekeydoesnotalreadyexist
2.2CommandsoperatingonAll
DBSIZEReturnthenumberofkeysinthecurrentdb SELECT indexSelecttheDBhavingthespecifiedindex MOVE key dbindexMovethekeyfromthecurrentlyselected DBtotheDBhavingasindexdbindex FLUSHDBRemoveallthekeysofthecurrentlyselectedDB FLUSHALLRemoveallthekeysfromallthedatabases
3.CommandsoperatingonString
SET key value Setakeytoastringvalue GET key Returnthestringvalueofthekey GETSET key valueSetakeytoastringreturningtheold valueofthekey MGET key1 key2 ... keyN Multiget,returnthestrings valuesofthekeys SETNX key valueSetakeytoastringvalueifthekeydoes notexist
3.1CommandsoperatingonString
MSET key1 value1 key2 value2 ... keyN valueN Setamultiplekeystomultiplevaluesinasingleatomicoperation
MSETNX key1 value1 key2 value2 ... keyN valueN Setamultiplekeystomultiplevaluesinasingleatomicoperation ifnoneofthekeysalreadyexist
3.2CommandsoperatingonString
INCR keyIncrementtheintegervalueofkey INCRBY key integerIncrementtheintegervalueofkeyby integer DECR keyDecrementtheintegervalueofkey DECRBY key integerDecrementtheintegervalueofkeyby integer
4.CommandsoperatingonLists
RPUSH key valueAppendanelementtothetailoftheList valueatkey LPUSH key valueAppendanelementtotheheadoftheList valueatkey LLEN keyReturnthelengthoftheListvalueatkey LRANGE key start endReturnarangeofelementsfromthe Listatkey LTRIM key start endTrimthelistatkeytothespecified rangeofelements
4.1CommandsoperatingonLists
LINDEX key indexReturntheelementatindexpositionfrom theListatkey LSET key index valueSetanewvalueastheelementat indexpositionoftheListatkey LREM key count valueRemovethefirstN,lastN,orall theelementsmatchingvaluefromtheListatkey LPOP keyReturnandremove(atomically)thefirstelementof theListatkey
4.2CommandsoperatingonLists
RPOP keyReturnandremove(atomically)thelastelementof theListatkey BLPOP key1 key2 ... keyN timeoutBlockingLPOP BRPOP key1 key2 ... keyNtimeoutBlockingRPOP RPOPLPUSH srckey dstkeyReturnandremove (atomically)thelastelementofthesourceListstoredat_srckey_ andpushthesameelementtothedestinationListstoredat _dstkey_
5.CommandsoperatingonSets
SADD key member AddthespecifiedmembertotheSetvalue atkey SREM key memberRemovethespecifiedmemberfromtheSet valueatkey SPOP keyRemoveandreturn(pop)arandomelementfromthe Setvalueatkey SMOVE srckey dstkey member Movethespecified memberfromoneSettoanotheratomically
5.1CommandsoperatingonSets
SCARD keyReturnthenumberofelements(thecardinality)of theSetatkey SISMEMBER key memberTestifthespecifiedvalueisa memberoftheSetatkey SINTER key1 key2 ... keyNReturntheintersection betweentheSetsstoredatkey1,key2,...,keyN SINTERSTORE dstkey key1 key2 ... keyN Compute theintersectionbetweentheSetsstoredatkey1,key2,...,keyN, andstoretheresultingSetatdstkey
5.2CommandsoperatingonSets
SUNION key1 key2 ... keyNReturntheunionbetween theSetsstoredatkey1,key2,...,keyN SUNIONSTORE dstkey key1 key2 ... keyNCompute theunionbetweentheSetsstoredatkey1,key2,...,keyN,and storetheresultingSetatdstkey SMEMBERS keyReturnallthemembersoftheSetvalueatkey SRANDMEMBER keyReturnarandommemberoftheSetvalue atkey
5.3CommandsoperatingonSets
SDIFF key1 key2 ... keyNReturnthedifferencebetween theSetstoredatkey1andalltheSetskey2,...,keyN SDIFFSTORE dstkey key1 key2 ... keyNCompute thedifferencebetweentheSetkey1andalltheSetskey2,..., keyN,andstoretheresultingSetatdstkey
6.CommandsoperatingonSortedSets
ZADD key score memberAddthespecifiedmembertothe SortedSetvalueatkeyorupdatethescoreifitalreadyexist ZREM key memberRemovethespecifiedmemberfromthe SortedSetvalueatkey ZINCRBY key increment memberIfthememberalready existsincrementitsscoreby_increment_,otherwiseaddthe membersetting_increment_asscore
6.1CommandsoperatingonSortedSets
ZRANGE key start endReturnarangeofelementsfromthe sortedsetatkey ZREVRANGE key start endReturnarangeofelements fromthesortedsetatkey,exactlylikeZRANGE,butthesorted setisorderedintraversedinreverseorder,fromthegreatesttothe smallestscore ZRANGEBYSCORE key min maxReturnalltheelementswith score>=minandscore<=max(arangequery)fromthesorted set
6.2CommandsoperatingonSortedSets
ZCARD keyReturnthecardinality(numberofelements)ofthe sortedsetatkey ZSCORE key elementReturnthescoreassociatedwiththe specifiedelementofthesortedsetatkey ZREMRANGEBYSCORE key min max Removeallthe elementswithscore>=minandscore<=maxfromthesortedset
7.Sorting
SORT key [BY pattern] [LIMIT start count] [GET pattern] [ASC|DESC] [ALPHA] [STORE dstkey] SorttheelementscontainedintheList,Set,orSortedSetvalueat key.Bydefaultsortingisnumericwithelementsbeing comparedasdoubleprecisionfloatingpointnumbers.Theseare thesimplestformofSORT: > SORT mylist DESC > SORT mylist LIMIT 0 10 > SORT mylist LIMIT 0 10 ALPHA DESC
8.Persistencecontrolcommands
SAVESynchronouslysavetheDBondisk BGSAVE AsynchronouslysavetheDBondisk LASTSAVE ReturntheUNIXtimestampofthelastsuccessfully savingofthedatasetondisk SHUTDOWN SynchronouslysavetheDBondisk,thenshutdown theserver BGREWRITEAOFRewritetheappendonlyfileinbackground whenitgetstoobig
9.Remoteservercontrolcommands
INFOProvideinformationandstatisticsabouttheserver MONITORDumpallthereceivedrequestsinrealtime SLAVEOFChangethereplicationsettings
StudyCase:twitter
DesignandimplementationofasimpleTwittercloneusingonly theRediskeyvaluestore. Wedon'thavetables,sowhatshouldbedesigned?Weneedto identifywhatkeysareneededtorepresentourobjectsandwhat kindofvaluesthiskeysneedtohold. Let'sstartfromUsers.Weneedtorepresentthisusersofcourse, withtheusername,userid,password,followersandfollowing users,andsoon. Thefirstquestionis,whatshouldidentifyanuserinsideour system?
StudyCase:twitter
> > > > SET global:nextUserId 1000 INCR global:nextUserId => 1001 SET uid:1001:username searleoliveira SET uid:1001:password e10adc3949ba59abbe56e057f20f883e
*atomicINCRoperation
StudyCase:twitter
SometimesitcanbeusefultobeabletogettheuserIDfromthe username,sowesetthiskeytoo: > SET username:searleoliveira:uid 1001
Thismayappearstrangeatfirst,butrememberthatweareonly abletoaccessdatabykey! It'snotpossibletotellRedistoreturnthekeythatholdsaspecific value. Thisnewparadigmisforcingustoorganizethedatasothat everythingisaccessiblebyprimarykey,speakingwithrelational DBslanguage.
StudyCase:twitter
Following,followersandupdates Everyuserhasfollowersusersandfollowingusers.Wehavea perfectdatastructureforthiswork! We'llneedtoaccessthisdatainchronologicalorderlater,from the mostrecentupdatetotheolderones,sotheperfectkindofValue forthisworkisaList.
> > > > LPUSH LPUSH LPUSH LPUSH uid:1001:followers 1002 uid:1001:followers 1003 uid:1001:followers...Setofuidsofallthefollowersusers uid:1001:following...Setofuidsofallthefollowingusers
StudyCase:twitter
Posts Anotherimportantthingweneedisaplacewerewecanaddthe poststodisplayintheuserhomepage.
> LPUSH uid:1001:posts Hey, I am Superman!
*aListofpostsids,everynewpostisLPUSHedhere.
StudyCase:twitter
Authentication We'llhandleauthenticationinasimplebutrobustway:wedon't wanttousePHPsessionsorotherthingslikethis,oursystemmust bereadyinordertobedistributedamongdifferentservers,so we'lltakethewholestateinourRedisdatabase.
> SET uid:1001:auth fea5e81ac8ca77622bed1c2132a021f9 > SET auth:fea5e81ac8ca77622bed1c2132a021f9 1001
StudyCase:twitter
Authentication Toauthenticateanuserwe'lldothissimplework: Gettheusernameandpasswordviatheloginform; Checkiftheusername:<username>:uidkeyactuallyexists; Ifitexistswehavetheuserid,(i.e.1001); Checkifuid:1001:passwordmatches,ifnot,errormessage; 5.Okauthenticated!Set"fea5e81ac8ca77622bed1c2132a021f9" (thevalueofuid:1001:auth)as"auth"cookie. Thishappenseverytimetheuserslogin.
StudyCase:twitter
Logout Theonlythingit'smissingfromalltheauthenticationisthe logout. Whatwedoonlogout? That'ssimple,we'lljustchangetherandomstringin uid:1001:auth. removetheoldauth:<oldauthstring> addanewauth:<newauthstring>.
StudyCase:twitter
Updates Updates,alsoknownasposts,areevensimpler.Inordertocreate anewpostonthedatabasewedosomethinglikethis:
> INCR global:nextUpdatesId => 10343 > SET updates:10343 "$owner_id|$time|I am Superman!"
StudyCase:twitter
Paginatingupdates NowitshouldbeprettyclearhowwecanuserLRANGEinorder togetrangesofposts,andrenderthispostsonthescreen:
> LRANGE uid:1001:posts 0 10 > LRANGE uid:1001:posts 10 20 > LRANGE uid:1001:posts 0 -1
ColumnOrientedStyle
Oneglobalmapwithacommonkeyspace Redisapplicationsstoreanentityusingattributes: R.set(U:123:firstname,Searle) R.set(U:123:lastname,Oliveira) R.set(U:123:password,AdE4Erf) R.set(U:123:username,searleoliveira) R.set(U:uid:searleoliveira,123) Eachnamedattributeoftheentitycombinedwiththeentity keybecomesakeyfortheentriesforthecorresponding values. Spaceconsumedisaconcernthough!
Makingithorizontallyscalable
Probablyyouwillnotneedmorethanoneserverforalotof applications,evenwhenyouhavealotofusers. Butlet'sassumeweareTwitterandneedtohandleahugeamount oftraffic. Whattodo? Thefirstthingtodoistohashthekeyandissuetherequeston differentserversbasedonthekeyhash.Thegeneralideaisthat youcanturnyourkeyintoanumber,andthantakethereminder of thedivisionofthisnumberbythenumberofserversyouhave:
Specialkeys
Forexampleeverytimewepostanewmessage,weneedto incrementtheglobal:nextPostIdkey.Howtofixthisproblem?A Singleserverwillgetalotifincrements.Thesimplestwayto handlethisistohaveadedicatedserverjustforincrements.This isprobablyanoverkillbtwunlessyouhavereallyalotoftraffic. Thereisanothertrick.TheIDdoesnotreallyneedtobean incrementalnumber,butjustitneedstobeunique.Soyoucan getarandomstringlongenoughtobeunlikely(almost impossible, ifit'smd5size)tocollide,andyouaredone.Wesuccessfully eliminatedourmainproblemtomakeitreallyhorizontally scalable!
Features
Speed
WhatisRedisAdmin?
RedisAdmin,orReAdmin,isaopensource webinterfacetotheAdministrationofRedis. ReAdminisfullywritteninPHPusingRedis, ofcourse. Thecurrentversionsupportscreateschemas, addkeysstrings,listsandsetsonly.Sorted setsisnotsupportedyet. Supportspersistencecontrol,commandquery editorandaserverstatusondashboard. ReAdminwascreatedbySearleOliveira.
Requirements
WebServerAwebserver,withRewritemodule,isneededtoinstallReAdmin on(e.g.,Apache,IISetc.) .htaccessRuleThismoduleusesarulebasedrewritingengine(basedona regularexpressionparser)torewriterequestedURLsonthefly.Availablein Apache1.2andlater. PHP5.2.0,ornewer,withtheStandardPHPLibrary(SPL)extensionenabled. Redis1.2.0ornewer WebbrowserAnywebbrowserwithcookiesenabled.
HowtoInstall?
DownloadChoosealaststableversionfromthedownloadspage. ExtractfilesUntarthestablepackageinyourwebserver's documentroot(e.g.,/var/www/). ConfigurationSimplyuseaplaintexteditortoeditafilenamed config.phpinthemain(toplevel)ReAdmindirectory(theone that containsindex.php). FinishForApacheyoucanusesupplied.htaccessfileinthat folder,forotherwebservers,youshouldconfigurethisyourself.
WhatisSchemas?
RedisdonotusetheSchemasconcept. RedisusesDBinstances(0,1,2,3...15). So,toorganizethisinstancesandmakemoreeasyforhuman,we createdanaliastothem.ThesealiasareSchemas. Db0 Db1 Db2 Db3
KnowLimitations
SortedSetscommands SortedSetsisnotsupportedyet. DBinstances TheDBinstances,inRedis,arelimitedtonumbersbecausethey arejustanaccessorytoworkwithasingledataset.Wecancreate only15DBinstances(from0to14),currently.TheDB15 instance isreservedtoReAdmininfo_schema.
Screenshot:Login
Screenshot:Dashboard
Screenshot:ShowKeys
Screenshot:AddKeyString
Screenshot:PersistenceControl
Screenshot:Command
TwitterUpdates
@phpae#ec2:redisadmin http://phpappengine.com/2010/opensource/redisadmin @nginx_tipsRedisAdminDownload:readmin0.0.6test.tar.gz(103KB) http://goo.gl/fb/KsMB @abrdevopensourcewebinterfaceExcellentOpensourcewebadminpanel toredis!http://code.google.com/p/redisadmin @antirezThiscouldbeaninterestingproject,Redisadmin: http://code.google.com/p/redisadmin/
ReadMore
RedisProject
http://code.google.com/p/redis
ReAdminProject
http://code.google.com/p/redisadmin
Shard(databasearchitecture)
http://en.wikipedia.org/wiki/Shard_(database_architecture)
QConSanFrancisco2009,BillyNewport
http://www.infoq.com/presentations/newportevolvingkeyvalueprogrammingmodel
Q&A
SearleOliveira
searle@corp.globo.com @searleoliveira Y!searleoliveira