Вы находитесь на странице: 1из 233

PostgreSQL:

. . aka leopard
Creative Commons Attribution-Noncommercial 4.0 International
20102014

2
2.1 . . . . . . . . . . . . . . . . . . . . . . .
. . . .
. . . . .
.
2.2 . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . .
. . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
2.3 . . . . . . . . . . . .

CLUSTER . . . . . . . . . . . . . . . . . . . . . . .
2.4 PostgreSQL . . . . . . . . .
Pgtune . . . . . . . . . . . . . . . . . . . . . . . . .
pg_buffercache . . . . . . . . . . . . . . . . . . . .
2.5 . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . .
2.6 . . . . . . . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

7
7
8
8
9
10
10
14
17
18
18
19
19
20
20
20
24
24
25
28
28
30
34

3
3.1 . . . . . . . . .
3.2 . . . . . . . . . . .
3.3
. . . . . . . . .

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

35
35
36
37
37

.
.
.
.

.
.
.
.
1

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

3.4

. . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
constraint_exclusion
. . . . . . . . . . . . . . . . . . . . . . . . . . .

4
4.1 . . . . . . . . . . . . . . . . . . . . .
4.2 (Streaming Replication)
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
4.3 PostgreSQL Bi-Directional Replication (BDR) .
4.4 Slony-I . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
. . . . . . . . . . .
4.5 Londiste . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
. . . . . . . . . . .
4.6 Bucardo . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
. . . .
4.7 RubyRep . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . .
4.8 . . . . . . . . . . . . . . . . . . . .
5
5.1 . . . . .
5.2 PL/Proxy . . . . .
. . . . .
. . . . .
?
5.3 Postgres-XC . . . .

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.
2

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.

39
41
41
43

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

45
45
47
47
48
48
55
56
57
57
57
58
63
65
68
68
69
70
80
81
82
82
82
83
86
86
88
88
89
89
91
91

.
.
.
.
.
.

93
93
94
95
95
99
100

5.4

5.5

. . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .
(HA) . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
HadoopDB . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.
.
.

6 PgPool-II
6.1 . . . . . . . . . . . . . . . . . . . . .
6.2 ! . . . . . . . . . . . . . . . . .
pgpool-II . . . . . . . . . . . . . . .
. . . . . . . . . . . . . .
PCP . . . . . . . . . . . . .
. . . . . . . . .
/ pgpool-II . . . . . . . . . .
6.3 . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . . .
6.4 . . . . . . .
. . . . . . .
SystemDB . . . . . . . . . . . . . .
. . .
. . . . . . . . .
. . . . . . .
6.5 Master-slave . . . . . . . . . . . . . . .
Streaming Replication ( )
6.6 . . . . . . . . . . . . .
Streaming Replication ( )
6.7 . . . . . . . . . . . . . . . . . . . .
7
7.1 .
7.2 PgBouncer .
7.3 PgPool-II vs

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

100
101
102
103
109
110
110
110
114
127
128

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

129
129
130
130
131
132
133
133
134
134
135
136
136
137
140
141
142
143
143
144
145
146

147
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
PgBouncer . . . . . . . . . . . . . . . . . . . . . . 149

8 PostgreSQL
8.1 . . . . . . .
8.2 Pgmemcache . . . . . .
. . . . . . .
. . . . . . .

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.
3

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

150
150
151
151
152

8.3

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157

9
9.1 . . . . . . . . . . . . . . . .
9.2 PostGIS . . . . . . . . . . . . . . . . .
9.3 pgSphere . . . . . . . . . . . . . . . . .
9.4 HStore . . . . . . . . . . . . . . . . . .
. . . . . . . .
. . . . . . . . . . . . . . .
9.5 PLV8 . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . . . .
9.6 Pg_repack . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
9.7 Pg_prewarm . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
9.8 Smlar . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
. . . . . . . . . . .
Smlar . . . . . . . . . . . . . . . . . .
:
. . . . . . . . . . . . . . .
9.9 Multicorn . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
PostgreSQL 9.3+ . . . . . . . . . . . .
. . . . . . . . . . . . . . .
9.10 Pgaudit . . . . . . . . . . . . . . . . .
9.11 Ltree . . . . . . . . . . . . . . . . . . .
Ltree? . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
9.12 PostPic . . . . . . . . . . . . . . . . .
9.13 Fuzzystrmatch . . . . . . . . . . . . . .
9.14 Tsearch2 . . . . . . . . . . . . . . . . .
9.15 OpenFTS . . . . . . . . . . . . . . . .
9.16 PL/Proxy . . . . . . . . . . . . . . . .
9.17 Texcaller . . . . . . . . . . . . . . . . .
9.18 Pgmemcache . . . . . . . . . . . . . . .
9.19 Prefix . . . . . . . . . . . . . . . . . .
9.20 Dblink . . . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

158
158
158
158
159
159
160
161
161
163
166
166
167
168
168
169
169
169
169
171
173
176
176
176
183
183
183
184
184
185
189
189
190
192
192
193
193
193
193
193


9.21 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
10 PostgreSQL
10.1 . . . . . . . . . . . . . . . . .
10.2 SQL . . . . . . . . . . . . . . . . .
SQL . . . .
10.3 . . .
10.4 .
. . . . . . . . . . . . . . . . .
10.5
WAL-E . . . . . . . . . . . . . . . . . .
Barman . . . . . . . . . . . . . . . . . .
10.6 . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

195
195
196
197
198
199
199
200
200
206
213

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

214
214
215
215
215
216
216
216

12 (Performance Snippets)
12.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
count . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . .
. . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . .
Quine . . . . . . . . . . .
LIKE . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . .
(bloat)

217
217
217
217
218
219
220
220
221
224
224
224
227
227
228

231

. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .

. . . . . . . .
. . . . . . . .
. . . . . . . .

11 PostgreSQL
11.1 . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
11.2 . . . . . . . . .
. . . . . . . . . . . . . .
11.3 . . . . . . . . .
. . . . . . . . . . . . . .
11.4 . . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

.
.
.
.
.
.
.

,

,
.

PostgreSQL.
PostgreSQL, . ,
.

2


,
.

2.1
, , . ,
.
:
;
.
,
.
,
?,
,
? , .
PostgreSQL.
,
PostgreSQL , , FAQ. postgresqlperformance, .
, ,
. : - .
7

2.1.


PostgreSQL ,
. . : ,
PostgreSQL, . (
) , . PostgreSQL, ,
. , , ..
PostgreSQL :

;
, ;
;
, (, -).


PostgreSQL, , .
.
7.1 ,
;
7.2 :
VACUUM, ;
ANALYZE, ,
;
;
7.4 ( IN/NOT IN);
8.0 ,
, CHECKPOINT VACUUM ;
8.1 , MIN() MAX(),
pg_autovacuum (),
;
8.2 SQL , ;
8

2.1.
8.3 , SQL/XML
, ;
8.4 , , , EXISTS/NOT EXISTS ;

9.0

,
VACUUM/VACUUM FULL ,
;
9.1 ,
( , ), ,
, MIN/MAX;
9.2 ,
, JSON , ,
25%, COPY;
9.3 materialized view, , SysV shared memory POSIX shared
memory mmap, c ,
,
;
, 8.4.


, ,
,
. , ,
. ,
, , ? , ,
.
,
, .
, , :
;

2.2.
(
);
(, ,
);
;
(.
2.5 );

2.2
, . postgresql.conf
.


: shared_buffers
PostgreSQL
. ,
,
, .
,
. , , . , ,
.

,
, .
: , PostgreSQL, PostgreSQL ,
. , PostgreSQL
, , . ,
, , .
, shared_buffers,
, ,
, .
10

2.2.
8
2 . ,
, , , ,
. , , .
, .
:
4 (512) ;
256512 : 1632
(20484096);
14 : 64256
(819232768);
.
ipcs (, free
vmstat). 1,2 2
, . ,
,
. ,
(
PostgreSQL < 9.3). PostgreSQL ,
: www.postgresql.org
, :
Laptop, Celeron processor, 384 RAM, 25 : 12 ;
Athlon server, 1 RAM,
10 : 200 ;
Quad PIII server, 4 RAM, 40 , 150 , : 1 ;
Quad Xeon server, 8 RAM, 200 , 300 ,
: 2 .
: work_mem
sort_mem, ,
,
, .
, work_mem (
). :
( , ,
11

2.2.
, shared_buffers) ,
.
, .
, .
work_mem postgresql.conf.
1 . 1024.
24% . -
work_mem, , , 512 2048 . ,
work_mem 500
. , , ,
, . , 14 32128 MB.
VACUUM: maintenance_work_mem
PostgreSQL 7.x vacuum_mem.
, VACUUM, ANALYZE,
CREATE INDEX, . ,
, . 50 75% , , 32 256
. , work_mem. . ,
14 128512 MB.
Free Space Map: VACUUM FULL
( PostgreSQL) :
, , , , ( );
( UPDATE
DELETE) 1 ( );
,
PostgreSQL VACUUM (
3.1.1).
1

12

2.2.
7.2 VACUUM . 7.2, VACUUM , SELECT, INSERT,
UPDATE DELETE . VACUUM FULL.
, , , ,
.
:
max_fsm_relations ,
. VACUUM.
max_fsm_relations ( );
max_fsm_pages ,
, . , ,
VACUUM. ,
VACUUM VERBOSE ANALYZE
, . max_fsm_pages ,
.
FSM, VACUUM
, VACUUM FULL,
.
! 8.4 fsm , Free Space Map , .

temp_buffers , . 16 ;
max_prepared_transactions (PREPARE TRANSACTION).
5;
vacuum_cost_delay , , , I/O VACUUM, . ,
vacuum_cost_delay 0.
13

2.2.
50 200 . vacuum_cost_page_hit vacuum_cost_page_limit.
VACUUM, . (Jan Wieck) ,
delay 200, page_hit 6 limit 100
VACUUM 80%, ;
max_stack_depth ,
, .
, , . 24 MB;
max_files_per_process , . , Too many open files;


PostgreSQL : ( ) , ,
.
:
. ,
:
( checkpoint_segments,
3) , ( checkpoint_timeout, ,
300).
,
, .
: checkpoint_segments
,
1 .
- .
(checkpoint_segments).
( 16 )
1

.
checkpoint_warning ( ):
, .

14

2.2.
. ,
, .
12 256 , (warning) , , .
, , (checkpoint_segments *
2 + 1) * 16 , , .
, 32, 1
.
, ,
.
fsync, synchronous_commit

off fsync.
,
. : ,
, !
,
.
.
synchronous_commit WAL .
on (). (off)
,
( wal_writer_delay * 3). fsync,
: ( ),
. synchronous_commit , , ( MongoDB:
MongoDB
).

15

2.2.

commit_delay ( , 0 ) commit_siblings
(5 ) .
commit_siblings
, commit_delay.
,
, .
, .
wal_sync_method ,
. fsync=off, . :
open_datasync open()
O_DSYNC;
fdatasync fdatasync() commit;
fsync_writethrough fsync() commit, ;
fsync fsync() commit;
open_sync open()
O_SYNC;
. , .
full_page_writes off, fsync=off.
, on, PostgreSQL
. , , . ,
.
, . full_page_writes
, (
checkpoint_interval).
wal_buffers SHARED MEMORY
1 . 256
512 , . , 14
2561024 .
1

16

2.2.




. 3 ,
:
default_statistics_target
,
ANALYZE (. 3.1.2). ,
, . ALTER TABLE ... SET STATISTICS.
effective_cache_size
PostgreSQL
,
1 .
1,5 , shared_buffers
32 , effective_cache_size 800 .
700 , PostgreSQL ,
merge joins. effective_cache_size
200 , , .
effective_cache_size
2/3 ;
RAM
, .
random_page_cost
, . 3.0, 2.5
2.0.
, .
. , , (sequential scans) (index
1

17

2.3.
scans), . , , ,
. . random_page_cost 2.0; , random_page_cost ,
.


PostgreSQL , .

, ,
. , true/false:
track_counts . ,
autovacuum . , (
autovacuum);
track_functions ;
track_activities
.
. ,
, ,
.
, , .
, : , (
autovacuum ).

2.3
, . , , , .
PostgreSQL , ,
18

2.3.
. ,
, .
, ,
, noatime1 .



, .
(
, , :
), .
, , ,
,
.
:
(!);
pg_clog pg_xlog,
, ;
;
.
, , ,
, , , .

CLUSTER
CLUSTER table [ USING index ]

, .
,
. ,
.
:
1

19

2.4. PostgreSQL
, ,
. , CLUSTER
ACCESS EXCLUSIVE ,
( )
. PostgreSQL ,
CLUSTER .

2.4 PostgreSQL
Pgtune
PostgreSQL Gregory Smith pgtune
.
Linux . ,
. :
2.1 Pgtune
Line 1

$ pgtune - i $PGDATA/ p o s t g r e s q l . c o n f - o $PGDATA/ p o s t g r e s q l .


c o n f . pgtune

- i , -- input-config postgresql.conf, -o
, -- output-config postgresql.conf.
:
-M, --memory ,
. , pgtune
;
-T, -- type . : DW, OLTP, Web,
Mixed, Desktop;
-c, -- connections . , .
pgtune.
, pgtune PostgreSQL.
, , ,
.

pg_buffercache
Pg_buffercache PostgreSQL, (shared_buffer)
20

2.4. PostgreSQL
. ,
.
:
2.2 pg_buffercache
Line 1

# CREATE EXTENSION p g _ b u f f e r c a c h e ;

pg_buffercache , :

bufferid ID ;
relfilenode , ;
reltablespace Oid ;
reldatabase Oid ;
relforknumber ;
relblocknumber ;
isdirty ?;
usagecount LRU .

ID ( bufferid ) , , .
:
. -with-blocksize . 8 ,
,
.
, PostgreSQL;
. shared_buffers
PostgreSQL .
, shared_buffers 128 8 16384 . pg_buffercache
16384. shared_buffers 256 1 262144 .
(, , ):
2.3 pg_buffercache
Line 1
5
-

# SELECT c . relname , count ( * ) AS b u f f e r s


FROM p g _ b u f f e r c a c h e b INNER JOIN p g _ c l a s s c
ON b . r e l f i l e n o d e = p g _ r e l a t i o n _ f i l e n o d e ( c . o i d ) AND
b . r e l d a t a b a s e IN ( 0 , (SELECT o i d FROM pg_database WHERE
datname = c u r r e n t _ d a t a b a s e ( ) ) )
GROUP BY c . relname
ORDER BY 2 DESC
LIMIT 1 0 ;

21

2.4. PostgreSQL
10
15
20
-

relname
| buffers
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - pgbench_accounts
|
4082
pgbench_history
|
53
pg_attribute
|
23
pg_proc
|
14
pg_operator
|
11
pg_proc_oid_index
|
9
pg_class
|
8
pg_attribute_relid_attnum_index |
7
pg_proc_proname_args_nsp_index |
6
pg_class_oid_index
|
5
( 1 0 rows )

( ) :
2.4 pg_buffercache
Line 1
5
-

# SELECT c . relname , count ( * ) AS b u f f e r s , u s a g e c o u n t


FROM p g _ c l a s s c
INNER JOIN p g _ b u f f e r c a c h e b
ON b . r e l f i l e n o d e = c . r e l f i l e n o d e
INNER JOIN pg_database d
ON ( b . r e l d a t a b a s e = d . o i d AND d . datname = c u r r e n t _ d a t a b a s e
() )
GROUP BY c . relname , u s a g e c o u n t
ORDER BY c . relname , u s a g e c o u n t ;

10
15
20
-

relname
| b u f f e r s | usagecount
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - -+ - - - - - - - - - - - pg _ r e w r i t e
|
3 |
1
pg_rewrite_rel_rulename_index
|
1 |
1
pg_rewrite_rel_rulename_index
|
1 |
2
pg_statistic
|
1 |
1
pg_statistic
|
1 |
3
pg_statistic
|
2 |
5
pg_statistic_relid_att_inh_index |
1 |
1
pg_statistic_relid_att_inh_index |
3 |
5
pgbench_accounts
|
4082 |
2
pgbench_accounts_pkey
|
1 |
1
pgbench_history
|
53 |
1
pgbench_tellers
|
1 |
1


( )
():
2.5 pg_buffercache
Line 1

# SELECT

22

2.4. PostgreSQL
5
10
15
-

c . relname ,
p g _ s i z e _ p r e t t y ( count ( * ) * 8 1 9 2 ) a s b u f f e r e d ,
round ( 1 0 0 . 0 * count ( * ) /
(SELECT s e t t i n g FROM p g _ s e t t i n g s WHERE name= s h a r e d _ b u f f e r s
) : : integer ,1)
AS b u f f e r s _ p e r c e n t ,
round ( 1 0 0 . 0 * count ( * ) * 8192 / p g _ t a b l e _ s i z e ( c . o i d ) , 1 )
AS p e r c e n t _ o f _ r e l a t i o n
FROM p g _ c l a s s c
INNER JOIN p g _ b u f f e r c a c h e b
ON b . r e l f i l e n o d e = c . r e l f i l e n o d e
INNER JOIN pg_database d
ON ( b . r e l d a t a b a s e = d . o i d AND d . datname = c u r r e n t _ d a t a b a s e
() )
GROUP BY c . oid , c . relname
ORDER BY 3 DESC
LIMIT 2 0 ;

20
25
30
35
40
-

- [ RECORD 1 ] - - - - - - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - relname
| pgbench_accounts
buffered
| 32 MB
buffers_percent
| 24.9
percent_of_relation | 99.9
- [ RECORD 2 ] - - - - - - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - relname
| pgbench_history
buffered
| 424 kB
buffers_percent
| 0.3
percent_of_relation | 94.6
- [ RECORD 3 ] - - - - - - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - relname
| pg_operator
buffered
| 88 kB
buffers_percent
| 0.1
percent_of_relation | 61.1
- [ RECORD 4 ] - - - - - - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - relname
| pg_opclass_oid_index
buffered
| 16 kB
buffers_percent
| 0.0
percent_of_relation | 100.0
- [ RECORD 5 ] - - - - - - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - relname
| pg_statistic_relid_att_inh_index
buffered
| 32 kB
buffers_percent
| 0.0
percent_of_relation | 100.0



.
shared_buffers PostgreSQL.
23

2.5.

2.5
:
1. , . :
a) . ;
b) , ;
2. ;
3. ;
4. .


,
. ( cron)
.
ANALYZE
.
.
VACUUM ANALYZE.
, ,
,
ANALYZE.
.
REINDEX
REINDEX . :
;
.
. , ,
. PostgreSQL
, . ,
.
24

2.5.
- , REINDEX. :
REINDEX, VACUUM FULL, ,
, .


, . , ,
, . .
, , :
, ,
. , ,
;
. ,
.
, , , , , .
EXPLAIN [ANALYZE]
EXPLAIN [] , PostgreSQL . EXPLAIN ANALYZE [] 1 , .
, .
:
(seq scan);

(nested loop);
EXPLAIN ANALYZE: ?
,
.
,
. , ,
, - ,
1

EXPLAIN ANALYZE DELETE . . .

25

2.5.
,
. 80%
,
.
EXPLAIN ANALYZE
, . ,
2.6 enable_seqscan
Line 1

SET e n a b l e _ s e q s c a n=f a l s e ;

,
, , .
postgresql.conf! ,
!


. :
pg_stat_user_tables
, , ,
,
;
pg_stat_user_indexes , , , (
, ,
);
pg_statio_user_tables , , ,
(. 2.1.1),
, , TOAST.
, :
(
);
26

2.5.
.
, , , ,
PRIMARY KEY UNIQUE;
.
, , .
PostgreSQL

/
, , . , ,
foo foo_name,
foo_name = , .

2.7
Line 1

CREATE INDEX foo_name_first_idx ON f o o ( ( l o w e r ( s u b s t r (


foo_name , 1 , 1 ) ) ) ) ;


2.8
Line 1

SELECT * FROM f o o WHERE l o w e r ( s u b s t r ( foo_name , 1 , 1 ) ) = ;

.
(partial indexes)
WHERE.
, , scheta uplocheno
boolean. , uplocheno = false , uplocheno
= true, .

2.9
Line 1

CREATE INDEX sc h e ta _ ne u pl o ch e no ON s c h e t a ( i d ) WHERE NOT


uplocheno ;


2.10
Line 1

SELECT * FROM s c h e t a WHERE NOT u p l o c h e n o AND . . . ;

, ,
WHERE, .

27

2.5.


PostrgeSQL , PostgreSQL ,
.
, , 1 .
, ,

;
.
, : ,
.


, , . ,

, .
SELECT count(*) FROM < >
count() : , ,
. (
!) , ,
, .

2.11 SQL
Line 1

SELECT count ( * ) FROM f o o ;

foo, .
, , .
:
1

RULE PostgreSQL SQL, ,


,

28

2.5.
1. , 1 , ,
ANALYZE:
2.12 SQL
Line 1

SELECT r e l t u p l e s FROM p g _ c l a s s WHERE relname = f o o ;

2. , , ,
. ,
. ,
;
3. , (cron).
DISTINCT
DISTINCT . GROUP BY DISTINCT. GROUP BY
, ,
DISTINCT ( 8.4 ).
2.13 DISTINCT
Line 1
5

p o s t g r e s=# s e l e c t count ( * ) from ( s e l e c t d i s t i n c t i from g ) a


;
count
-- ----19125
( 1 row )

Time : 5 8 0 , 5 5 3 ms

10
-

p o s t g r e s=# s e l e c t count ( * ) from ( s e l e c t d i s t i n c t i from g ) a


;
count
-- ----19125
( 1 row )

15
-

Time : 3 6 , 2 8 1 ms
1
10000 ,
50000 !

29

2.5.
2.14 GROUP BY
Line 1
5

p o s t g r e s=# s e l e c t count ( * ) from ( s e l e c t i from g group by i )


a;
count
-- ----19125
( 1 row )

Time : 2 6 , 5 6 2 ms

10
-

p o s t g r e s=# s e l e c t count ( * ) from ( s e l e c t i from g group by i )


a;
count
-- ----19125
( 1 row )

15
-

Time : 2 5 , 2 7 0 ms


pgFouine
pgFouine log- PostgreSQL,
log- PostgreSQL. pgFouine , . pgFouine PHP - ,
GNU General Public License.
,
log- .
pgFouine PostgreSQL
log-:
syslog
2.15 pgFouine
Line 1
-

log_destination = syslog
redirect_stderr = off
silent_mode = on

, n :
2.16 pgFouine

30

2.5.
Line 1
-

log_min_duration_statement = n
log_duration = o f f
l o g_ s t a te m e nt = none

log_min_duration_statement 0. ,

-1.
pgFouine .
HTML- :
2.17 pgFouine
Line 1

p g f o u i n e . php - f i l e your / l o g / f i l e . l o g > your - r e p o r t . html

10 :
2.18 pgFouine
Line 1

p g f o u i n e . php - f i l e your / l o g / f i l e . l o g - top 10 - format t e x t

, ,
pgfouine.projects.pgfoundry.org.
pgBadger
pgBadger , pgFouine,
Perl. ,
( pgFouine 24.02.2010, pgBadger 12.10.2012).
pgBadger :
2.19 pgBadger
$
$
- $
- $

Line 1
-

t a r x z f pgbadger - 2 . x . t a r . gz
cd pgbadger - 2 . x/
p e r l M a k e f i l e . PL
make && sudo make i n s t a l l

pgFouine PostgreSQL :
2.20 PostgreSQL
Line 1
5
-

l o g g i n g _ c o l l e c t o r = on
log_min_messages = debug1
log_min_error_statement = debug1
log_min_duration_statement = 0
l o g _ l i n e _ p r e f i x = %t [%p ] : [% l - 1 ] u s e r=%u , db=%d
l o g _ c h e c k p o i n t s = on
l o g _ c o n n e c t i o n s = on

31

2.5.
10

l o g _ d i s c o n n e c t i o n s = on
l o g _ l o c k _ w a i t s = on
log_temp_files = 0

PostgreSQL pgBadger:
2.21 pgBadger
Line 1
5
-

$ . / pgbadger ~/ p g s q l / master / pg_log / p o s t g r e s q l - 2 0 1 2 - 0 8 - 3 0 _132


*
[========================>] Parsed 10485768 b y t e s o f
10485768 ( 1 0 0 . 0 0 % )
[========================>] Parsed 10485828 b y t e s o f
10485828 ( 1 0 0 . 0 0 % )
[========================>] Parsed 10485851 b y t e s o f
10485851 ( 1 0 0 . 0 0 % )
[========================>] Parsed 10485848 b y t e s o f
10485848 ( 1 0 0 . 0 0 % )
[========================>] Parsed 10485839 b y t e s o f
10485839 ( 1 0 0 . 0 0 % )
[========================>] Parsed 982536 b y t e s o f 982536
(100.00%)

HTML ,
PostgreSQL.
dalibo.github.com/pgbadger.
pg_stat_statements
pg_stat_statements
.
, PostgreSQL,
pgFouine pgBadger. :
2.22 pg_stat_statements postgresql.conf
Line 1
-

s h a r e d _ p r e l o a d _ l i b r a r i e s = pg_stat_statements
c u s t o m _ v a r i a b l e _ c l a s s e s = pg_stat_statements #
PostgreSQL 9 . 1

pg_stat_statements . max = 10000


pg_stat_statements . t r a c k = a l l

PostgreSQL .
pg_stat_statements:
1. pg_stat_statements.max (integer) sql , (
);

32

2.5.
2. pg_stat_statements.track (enum) SQL . : top ( /), all ( , ) none ( );
3. pg_stat_statements.save (boolean)
PostgreSQL. .
:
2.23 pg_stat_statements
Line 1

# CREATE EXTENSION pg_stat_statements ;

:
2.24 pg_stat_statements
# SELECT query , c a l l s , t o t a l _ t i m e , rows , 1 0 0 . 0 *
shared_blks_hit /
n u l l i f ( s h a r e d _ b l k s _ h i t + shared_blks_read , 0 )
AS h i t _ p e r c e n t
FROM pg_stat_statements ORDER BY t o t a l _ t i m e DESC
LIMIT 1 0 ;
- - [ RECORD 1 ] - ------------------------------------------------------------------------

Line 1

5
10
-

15
-

20

query
| SELECT query , c a l l s , t o t a l _ t i m e , rows , ? *
shared_blks_hit /
|
n u l l i f ( shared_blks_hit +
shared_blks_read , ? ) AS h i t _ p e r c e n t
|
FROM pg_stat_statements ORDER BY
t o t a l _ t i m e DESC LIMIT ? ;
calls
| 3
total_time | 0.994
rows
| 7
hit_percent | 100.0000000000000000
- [ RECORD 2 ] - -----------------------------------------------------------------------query
| insert into x ( i ) select generate_series (? ,?) ;
calls
| 2
total_time | 0.591
rows
| 110
hit_percent | 100.0000000000000000
- [ RECORD 3 ] - -----------------------------------------------------------------------query
calls

| s e l e c t * from x where i = ? ;
| 2

33

2.6.
total_time | 0.157
rows
| 6
- hit_percent | 100.0000000000000000
- - [ RECORD 4 ] - ------------------------------------------------------------------------

query
calls
- total_time
- rows
- hit_percent

25
-

|
|
|
|
|

SELECT p g _ s t a t _ s t a t e m e n t s _ r e s e t ( ) ;
1
0.102
1

pg_stat_statements_reset:
2.25
# SELECT p g _ s t a t _ s t a t e m e n t s _ r e s e t ( ) ;
- [ RECORD 1 ] - - - - - - - - - - - - + - pg_stat_statements_reset |

Line 1
5
10
-

# SELECT query , c a l l s , t o t a l _ t i m e , rows , 1 0 0 . 0 *


shared_blks_hit /
n u l l i f ( s h a r e d _ b l k s _ h i t + shared_blks_read , 0 )
AS h i t _ p e r c e n t
FROM pg_stat_statements ORDER BY t o t a l _ t i m e DESC
LIMIT 1 0 ;
- [ RECORD 1 ] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - query
| SELECT p g _ s t a t _ s t a t e m e n t s _ r e s e t ( ) ;
calls
| 1
total_time | 0.175
rows
| 1
hit_percent |

, PostgreSQL
9.2 contrib SQL . 9.1 SQL
, select * from table where id = 3 select
* from table where id = 21 ,
.

2.6
, PostgreSQL .
, . .

34

- ,

.
, ,

.

3.1
(partitioning, )
(, ) .
, .
( ).
(.. ,
). ( 5. . . 10 ) 40. . . 50% ,
( ), ,
. , ( )
, ( ). , , . SELECT *
FROM articles ORDER BY id DESC LIMIT 10
, .
, :
(, ,
) ;
35

3.2.

(DROP TABLE , DELETE);
.

3.2
PostgreSQL :
(range) , ,
. , ;
(list)
.
,
:
, . .
,
;
, ;
,
. ,
. :
3.1
Line 1
-

CHECK ( o u t l e t I D BETWEEN 100 AND 200 )


CHECK ( o u t l e t I D BETWEEN 200 AND 300 )

, 200;
(
), ;
, ;
, constraint_exclusion
postgresql.conf. , .

36

3.3.

3.3
. , ,
.
. , , ()
(). ,
3 . ,
. , , , ,
( ). .
.

, :
3.2
Line 1
5
-

CREATE TABLE my_logs (


id
SERIAL PRIMARY KEY,
user_id
INT NOT NULL,
logdate
TIMESTAMP NOT NULL,
data
TEXT,
some_state
INT
);

, .
.
my_logs,
. ():
3.3
Line 1
5
10

CREATE TABLE my_logs2010m10


CHECK ( l o g d a t e >= DATE
2010 -11 -01 )
) INHERITS ( my_logs ) ;
CREATE TABLE my_logs2010m11
CHECK ( l o g d a t e >= DATE
2010 -12 -01 )
) INHERITS ( my_logs ) ;
CREATE TABLE my_logs2010m12
CHECK ( l o g d a t e >= DATE
2011 -01 -01 )
) INHERITS ( my_logs ) ;
CREATE TABLE my_logs2011m01

(
2 0 1 0 - 1 0 - 0 1 AND l o g d a t e < DATE

(
2 0 1 0 - 1 1 - 0 1 AND l o g d a t e < DATE

(
2 0 1 0 - 1 2 - 0 1 AND l o g d a t e < DATE

37

3.3.
-

CHECK ( l o g d a t e >= DATE 2 0 1 1 - 0 1 - 0 1 AND l o g d a t e < DATE


2010 -02 -01 )
) INHERITS ( my_logs ) ;

my_logs2010m10,
my_logs2010m11 ..,
( ). CHECK
, (
, !).
logdate,
:
3.4
CREATE INDEX
logdate ) ;
- CREATE INDEX
logdate ) ;
- CREATE INDEX
logdate ) ;
- CREATE INDEX
logdate ) ;

Line 1

my_logs2010m10_logdate ON my_logs2010m10 (
my_logs2010m11_logdate ON my_logs2010m11 (
my_logs2010m12_logdate ON my_logs2010m12 (
my_logs2011m01_logdate ON my_logs2011m01 (

,
.
3.5
Line 1
5
10
15
20
-

CREATE OR REPLACE FUNCTION m y _ l o g s _ i n s e r t _ t r i g g e r ( )


RETURNS TRIGGER AS $$
BEGIN
IF ( NEW. l o g d a t e >= DATE 2 0 1 0 - 1 0 - 0 1 AND
NEW. l o g d a t e < DATE 2 0 1 0 - 1 1 - 0 1 ) THEN
INSERT INTO my_logs2010m10 VALUES (NEW. * ) ;
ELSIF ( NEW. l o g d a t e >= DATE 2 0 1 0 - 1 1 - 0 1 AND
NEW. l o g d a t e < DATE 2 0 1 0 - 1 2 - 0 1 ) THEN
INSERT INTO my_logs2010m11 VALUES (NEW. * ) ;
ELSIF ( NEW. l o g d a t e >= DATE 2 0 1 0 - 1 2 - 0 1 AND
NEW. l o g d a t e < DATE 2 0 1 1 - 0 1 - 0 1 ) THEN
INSERT INTO my_logs2010m12 VALUES (NEW. * ) ;
ELSIF ( NEW. l o g d a t e >= DATE 2 0 1 1 - 0 1 - 0 1 AND
NEW. l o g d a t e < DATE 2 0 1 1 - 0 2 - 0 1 ) THEN
INSERT INTO my_logs2011m01 VALUES (NEW. * ) ;
ELSE
RAISE EXCEPTION Date out o f r a n g e . Fix t h e
my_logs_insert_trigger ( ) function ! ;
END IF ;
RETURN NULL;
END;
$$

38

3.3.
-

LANGUAGE p l p g s q l ;

: logdate,
.
.
:
3.6
CREATE TRIGGER i n s e r t _ m y _ l o g s _ t r i g g e r
BEFORE INSERT ON my_logs
FOR EACH ROW EXECUTE PROCEDURE m y _ l o g s _ i n s e r t _ t r i g g e r ( ) ;

Line 1
-

my_logs:
3.7
INSERT INTO my_logs ( user_id , l o g d a t e , data , some_state )
VALUES( 1 , 2 0 1 0 - 1 0 - 3 0 , 3 0 . 1 0 . 2 0 1 0 data , 1 ) ;
- INSERT INTO my_logs ( user_id , l o g d a t e , data , some_state )
VALUES( 2 , 2 0 1 0 - 1 1 - 1 0 , 1 0 . 1 1 . 2 0 1 0 data2 , 1 ) ;
- INSERT INTO my_logs ( user_id , l o g d a t e , data , some_state )
VALUES( 1 , 2 0 1 0 - 1 2 - 1 5 , 1 5 . 1 2 . 2 0 1 0 data3 , 1 ) ;

Line 1

:
3.8
Line 1
-

p a r t i t i o n i n g _ t e s t=# SELECT * FROM ONLY my_logs ;


i d | u s e r _ i d | l o g d a t e | data | some_state
- - - -+ - - - - - - - - -+ - - - - - - - - -+ - - - - - -+ - - - - - - - - - - - ( 0 rows )

.
:
3.9
Line 1
-

p a r t i t i o n i n g _ t e s t=# SELECT * FROM my_logs ;


id | user_id |
logdate
|
data
|
some_state
-- -+ - - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - -

1 |

2 |

1 | 2 0 1 0 - 1 0 - 3 0 0 0 : 0 0 : 0 0 | 3 0 . 1 0 . 2 0 1 0 data

1
2 | 2 0 1 0 - 1 1 - 1 0 0 0 : 0 0 : 0 0 | 1 0 . 1 1 . 2 0 1 0 data2 |
1

39

3.3.
-

3 |

1 | 2 0 1 0 - 1 2 - 1 5 0 0 : 0 0 : 0 0 | 1 5 . 1 2 . 2 0 1 0 data3 |
1

( 3 rows )

. , :
3.10
Line 1
-

p a r t i t i o n i n g _ t e s t=# S e l e c t * from my_logs2010m10 ;


id | user_id |
logdate
|
data
|
some_state
-- -+ - - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - 1 |

1 | 2 0 1 0 - 1 0 - 3 0 0 0 : 0 0 : 0 0 | 3 0 . 1 0 . 2 0 1 0 data |
1

( 1 row )

10

p a r t i t i o n i n g _ t e s t=# S e l e c t * from my_logs2010m11 ;


id | user_id |
logdate
|
data
|
some_state
-- -+ - - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - 2 |

2 | 2 0 1 0 - 1 1 - 1 0 0 0 : 0 0 : 0 0 | 1 0 . 1 1 . 2 0 1 0 data2 |
1

( 1 row )

! .
my_logs :
3.11
Line 1
-

p a r t i t i o n i n g _ t e s t=# SELECT * FROM my_logs WHERE u s e r _ i d = 2 ;


id | user_id |
logdate
|
data
|
some_state
-- -+ - - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - 2 |

2 | 2 0 1 0 - 1 1 - 1 0 0 0 : 0 0 : 0 0 | 1 0 . 1 1 . 2 0 1 0 data2 |
1

( 1 row )

p a r t i t i o n i n g _ t e s t=# SELECT * FROM my_logs WHERE data LIKE


%0.1% ;
id | user_id |
logdate
|
data
|
some_state
-- -+ - - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - -

40

3.3.
1 |

10

1 | 2 0 1 0 - 1 0 - 3 0 0 0 : 0 0 : 0 0 | 3 0 . 1 0 . 2 0 1 0 data

1
2 |

2 | 2 0 1 0 - 1 1 - 1 0 0 0 : 0 0 : 0 0 | 1 0 . 1 1 . 2 0 1 0 data2 |
1

( 2 rows )


. . ,
2008 , 10 . :
3.12
Line 1

DROP TABLE my_logs2008m10 ;

DROP TABLE , DELETE. ,


, ,
,
:
3.13
Line 1

ALTER TABLE my_logs2008m10 NO INHERIT my_logs ;

, .

constraint_exclusion
constraint_exclusion ,
. , :
3.14 constraint_exclusion OFF
Line 1
-

p a r t i t i o n i n g _ t e s t=# SET c o n s t r a i n t _ e x c l u s i o n = o f f ;
p a r t i t i o n i n g _ t e s t=# EXPLAIN SELECT * FROM my_logs WHERE
logdate > 2010 -12 -01 ;

QUERY PLAN

-------------------------------------------------------------------------

R e s u l t ( c o s t = 6 . 8 1 . . 1 0 4 . 6 6 rows =1650 width =52)


-> Append ( c o s t = 6 . 8 1 . . 1 0 4 . 6 6 rows =1650 width =52)
-> Bitmap Heap Scan on my_logs ( c o s t = 6 . 8 1 . . 2 0 . 9 3
rows =330 width =52)

41

3.3.
10
-

15
-

20
-

25
-

Recheck Cond : ( l o g d a t e > 2 0 1 0 - 1 2 - 0 1 0 0 : 0 0 : 0 0


: : timestamp w i t h o u t time zone )
-> Bitmap Index Scan on my_logs_logdate (
c o s t = 0 . 0 0 . . 6 . 7 3 rows =330 width =0)
Index Cond : ( l o g d a t e > 2 0 1 0 - 1 2 - 0 1
0 0 : 0 0 : 0 0 : : timestamp w i t h o u t time zone )
-> Bitmap Heap Scan on my_logs2010m10 my_logs (
c o s t = 6 . 8 1 . . 2 0 . 9 3 rows =330 width =52)
Recheck Cond : ( l o g d a t e > 2 0 1 0 - 1 2 - 0 1 0 0 : 0 0 : 0 0
: : timestamp w i t h o u t time zone )
-> Bitmap Index Scan on
my_logs2010m10_logdate ( c o s t = 0 . 0 0 . . 6 . 7 3 rows =330 width
=0)
Index Cond : ( l o g d a t e > 2 0 1 0 - 1 2 - 0 1
0 0 : 0 0 : 0 0 : : timestamp w i t h o u t time zone )
-> Bitmap Heap Scan on my_logs2010m11 my_logs (
c o s t = 6 . 8 1 . . 2 0 . 9 3 rows =330 width =52)
Recheck Cond : ( l o g d a t e > 2 0 1 0 - 1 2 - 0 1 0 0 : 0 0 : 0 0
: : timestamp w i t h o u t time zone )
-> Bitmap Index Scan on
my_logs2010m11_logdate ( c o s t = 0 . 0 0 . . 6 . 7 3 rows =330 width
=0)
Index Cond : ( l o g d a t e > 2 0 1 0 - 1 2 - 0 1
0 0 : 0 0 : 0 0 : : timestamp w i t h o u t time zone )
-> Bitmap Heap Scan on my_logs2010m12 my_logs (
c o s t = 6 . 8 1 . . 2 0 . 9 3 rows =330 width =52)
Recheck Cond : ( l o g d a t e > 2 0 1 0 - 1 2 - 0 1 0 0 : 0 0 : 0 0
: : timestamp w i t h o u t time zone )
-> Bitmap Index Scan on
my_logs2010m12_logdate ( c o s t = 0 . 0 0 . . 6 . 7 3 rows =330 width
=0)
Index Cond : ( l o g d a t e > 2 0 1 0 - 1 2 - 0 1
0 0 : 0 0 : 0 0 : : timestamp w i t h o u t time zone )
-> Bitmap Heap Scan on my_logs2011m01 my_logs (
c o s t = 6 . 8 1 . . 2 0 . 9 3 rows =330 width =52)
Recheck Cond : ( l o g d a t e > 2 0 1 0 - 1 2 - 0 1 0 0 : 0 0 : 0 0
: : timestamp w i t h o u t time zone )
-> Bitmap Index Scan on
my_logs2011m01_logdate ( c o s t = 0 . 0 0 . . 6 . 7 3 rows =330 width
=0)
Index Cond : ( l o g d a t e > 2 0 1 0 - 1 2 - 0 1
0 0 : 0 0 : 0 0 : : timestamp w i t h o u t time zone )
( 2 2 rows )

EXPLAIN, , , logdate > 2010-12-01 , , .


42

3.4.
constraint_exclusion:
3.15 constraint_exclusion ON
Line 1
5

10
-

15
-

p a r t i t i o n i n g _ t e s t=# SET c o n s t r a i n t _ e x c l u s i o n = on ;
SET
p a r t i t i o n i n g _ t e s t=# EXPLAIN SELECT * FROM my_logs WHERE
logdate > 2010 -12 -01 ;
QUERY PLAN
------------------------------------------------------------------------R e s u l t ( c o s t = 6 . 8 1 . . 4 1 . 8 7 rows =660 width =52)
-> Append ( c o s t = 6 . 8 1 . . 4 1 . 8 7 rows =660 width =52)
-> Bitmap Heap Scan on my_logs ( c o s t = 6 . 8 1 . . 2 0 . 9 3
rows =330 width =52)
Recheck Cond : ( l o g d a t e > 2 0 1 0 - 1 2 - 0 1 0 0 : 0 0 : 0 0
: : timestamp w i t h o u t time zone )
-> Bitmap Index Scan on my_logs_logdate (
c o s t = 0 . 0 0 . . 6 . 7 3 rows =330 width =0)
Index Cond : ( l o g d a t e > 2 0 1 0 - 1 2 - 0 1
0 0 : 0 0 : 0 0 : : timestamp w i t h o u t time zone )
-> Bitmap Heap Scan on my_logs2010m12 my_logs (
c o s t = 6 . 8 1 . . 2 0 . 9 3 rows =330 width =52)
Recheck Cond : ( l o g d a t e > 2 0 1 0 - 1 2 - 0 1 0 0 : 0 0 : 0 0
: : timestamp w i t h o u t time zone )
-> Bitmap Index Scan on
my_logs2010m12_logdate ( c o s t = 0 . 0 0 . . 6 . 7 3 rows =330 width
=0)
Index Cond : ( l o g d a t e > 2 0 1 0 - 1 2 - 0 1
0 0 : 0 0 : 0 0 : : timestamp w i t h o u t time zone )
( 1 0 rows )

, ,
, .
constraint_exclusion , , CHECK ,
, . 8.4 PostgreSQL constraint_exclusion on, off partition.
( ) constraint_exclusion on,
off, partition, CHECK .

3.4

.
43

3.4.
, . ,
, ( ) 50%
.

44

,
.
,
.

4.1
(. replication) (, ). ,
. , ,
. .
, ,
. , . (,
).
,
, - ( , , ).
, .
, , , ( ,
). , . 45

4.1.
,
,

.
,
( ). ,
, , , ,
.
(, , ). ,
, ( )
. ,
, . , ,
X, B , , Y X.
Y, Y (, - ), B
Y , , ,
.
, Y , X .
. , ,
,
( ) .

: ,
. , , .
,
. , . ,
, ,
.
,

.
PostgreSQL , , . 46

4.2. (Streaming Replication)


. :
Slony-I Master-Slave , (cascading) (failover). Slony-I
PostgreSQL
INSERT/DELETE/UPDATE ;
PGCluster Multi-Master .
;
Pgpool-I/II PostgreSQL (
II ). :
( ,
stand-by );
online-;
pooling ;
;
SELECT- postgresql;
;
Bucardo , MultiMaster Master-Slave , ;
Londiste Master-Slave .
Skytools1 . , Slony-I;
Mammoth Replicator Multi-Master ;
Postgres-R Multi-Master .
;
RubyRep Ruby, Multi-Master , PostgreSQL MySQL.
, , ,
PostgreSQL.

4.2 (Streaming Replication)

(Streaming Replication, SR)


wall xlog
. PostgreSQL 9 ( !).
, , ,
1

http://pgfoundry.org/projects/skytools/

47

4.2. (Streaming Replication)



, PostgreSQL.
:

PostgreSQL;
;
;
- .

:

PostgreSQL .

PostgreSQL 9 .
9.3 . , , Linux.

masterdb(192.168.0.10)
slavedb(192.168.0.20).


ssh. postgres . ,
:
4.1 userssh
$ sudo groupadd u s e r s s h
$ sudo u s e r a d d -m - g u s e r s s h - d /home/ u s e r s s h - s / b i n / bash \
- - c " user ssh allow " userssh

Line 1
-

(
postgres):
4.2 postgres
Line 1

$ su p o s t g r e s

RSA- :

48

4.2. (Streaming Replication)


4.3 RSA-
Line 1
5
-

$ ssh - keygen - t r s a -P " "


G e n e r a t i n g p u b l i c / p r i v a t e r s a key p a i r .
Enter f i l e i n which t o s a v e t h e key ( / var / l i b / p o s t g r e s q l / .
ssh / id_rsa ) :
Created d i r e c t o r y / var / l i b / p o s t g r e s q l / . s s h .
Your i d e n t i f i c a t i o n has been saved i n / var / l i b / p o s t g r e s q l / .
ssh / id_rsa .
Your p u b l i c key has been saved i n / var / l i b / p o s t g r e s q l / . s s h /
i d _ r s a . pub .
The key f i n g e r p r i n t i s :
1 6 : 0 8 : 2 7 : 9 7 : 2 1 : 3 9 : b5 : 7 b : 8 6 : e1 : 4 6 : 9 7 : b f : 1 2 : 3 d : 7 6
postgres@localhost

:
4.4
Line 1

$ c a t $HOME/ . s s h / i d _ r s a . pub >> $HOME/ . s s h / a u t h o r i z e d _ k e y s

. :
4.5 ssh
Line 1

$ ssh l o c a l h o s t

sshd:
4.6 sshd
Line 1

$ $/ e t c / i n i t . d/ s shd s t a r t

$HOME/.ssh
slavedb. ssh.
pg_hba.conf ,
(trust) ( replication):
4.7 pg_hba.conf
Line 1

host

replication

all

192.168.0.20/32

trust

4.8 pg_hba.conf
Line 1

host

replication

all

192.168.0.10/32

trust

postgresql .

masterdb. postgresql.conf
:
49

4.2. (Streaming Replication)


4.9
# To e n a b l e read - o n l y q u e r i e s on a standby s e r v e r , w a l _ l e v e l
must be s e t t o
- # " hot_standby " . But you can c h o o s e " a r c h i v e " i f you n e v e r
connect to the
- # s e r v e r i n standby mode .
- w a l _ l e v e l = hot_standby

Line 1

# S e t t h e maximum number o f c o n c u r r e n t c o n n e c t i o n s from t h e


standby s e r v e r s .
- max_wal_senders = 5
-

10
15
-

# To p r e v e n t t h e primary s e r v e r from removing t h e WAL


segments r e q u i r e d f o r
# t h e standby s e r v e r b e f o r e s h i p p i n g them , s e t t h e minimum
number o f segments
# r e t a i n e d i n t h e pg_xlog d i r e c t o r y . At l e a s t
wal_keep_segments s h o u l d be
# l a r g e r than t h e number o f segments g e n e r a t e d between t h e
beginning of
# o n l i n e - backup and t h e s t a r t u p o f s t r e a m i n g r e p l i c a t i o n . I f
you e n a b l e WAL
# a r c h i v i n g t o an a r c h i v e d i r e c t o r y a c c e s s i b l e from t h e
standby , t h i s may
# not be n e c e s s a r y .
wal_keep_segments = 32

20
-

# Enable WAL a r c h i v i n g on t h e primary t o an a r c h i v e


d i r e c t o r y a c c e s s i b l e from
# t h e standby . I f wal_keep_segments i s a h i g h enough number
t o r e t a i n t h e WAL
# segments r e q u i r e d f o r t h e standby s e r v e r , t h i s may not be
necessary .
archive_mode
= on
archive_command = cp %p / path_to / a r c h i v e/% f

:
wal_level = hot_standby WAL
archive, ,
( archive,
);
max_wal_senders = 5 ;
wal_keep_segments = 32 c WAL
pg_xlog ;

50

4.2. (Streaming Replication)


archive_mode = on WAL
archive_command . /path/to/archive/.
. 9.1 synchronous_standby_names, . application_name, recovery.conf:
4.10 recovery.conf
restore_command = cp /mnt/ s e r v e r / a r c h i v e d i r /% f %p
# e . g . cp /mnt/ s e r v e r / a r c h i v e d i r /% f %p
- standby_mode = on
- primary_conninfo = h o s t=masterdb p o r t =59121 u s e r=
r e p l i c a t i o n password=r e p l i c a t i o n application_name=
newcluster
# e . g . h o s t=l o c a l h o s t p o r t =5432
- t r i g g e r _ f i l e = /tmp/ t r i g _ f _ n e w c l u s t e r

Line 1

PostgreSQL .
slavedb.

slavedb
masterdb. .
masterdb . :
4.11
Line 1

$ p s q l - c "SELECT pg_start_backup ( l a b e l , t r u e ) "

.
:
4.12
$ r s y n c -C - a - - d e l e t e - e s s h - - e x c l u d e p o s t g r e s q l . c o n f - exclude postmaster . pid \
- - - e x c l u d e p o s t m a s t e r . o p t s - - e x c l u d e pg_log - - e x c l u d e pg_xlog
\
- - - e x c l u d e r e c o v e r y . c o n f master_db_datadir / s l a v e d b _ h o s t :
slave_db_datadir /

Line 1

master_db_datadir postgresql masterdb


slave_db_datadir postgresql slavedb
slavedb_host slavedb( - 192.168.1.20)
, . :
51

4.2. (Streaming Replication)


4.13
Line 1

$ p s q l - c "SELECT pg_stop_backup ( ) "

PostgreSQL 9.1+
pg_basebackup ( slavedb ):
4.14
Line 1 $ pg_basebackup -R -D / s r v / p g s q l / standby - - h o s t = 1 9 2 . 1 6 8 . 0 . 1 0
- - p o r t =5432

postgresql.conf, ( ). :
4.15
Line 1

hot_standby = on

! wal_level = archive, (hot_standby = off).


slavedb PostgreSQL
recovery.conf :
4.16 recovery.conf
# S p e c i f i e s whether t o s t a r t t h e s e r v e r a s a standby . In
streaming r e p l i c a t i o n ,
- # t h i s parameter must t o be s e t t o on .
- standby_mode
= on

Line 1

# S p e c i f i e s a c o n n e c t i o n s t r i n g which i s used f o r t h e
standby s e r v e r t o c o n n e c t
- # with t h e primary .
- primary_conninfo
= h o s t = 1 9 2 . 1 6 8 . 0 . 1 0 p o r t =5432 u s e r=
postgres

# S p e c i f i e s a t r i g g e r f i l e whose p r e s e n c e s h o u l d c a u s e
streaming r e p l i c a t i o n to
10 # end ( i . e . , f a i l o v e r ) .
- t r i g g e r _ f i l e = / path_to / t r i g g e r
-

15
-

# S p e c i f i e s a command t o l o a d a r c h i v e segments from t h e WAL


archive . I f
# wal_keep_segments i s a h i g h enough number t o r e t a i n t h e
WAL segments
# r e q u i r e d f o r t h e standby s e r v e r , t h i s may not be n e c e s s a r y
. But
# a l a r g e workload can c a u s e segments t o be r e c y c l e d b e f o r e
t h e standby
# i s f u l l y s y n c h r o n i z e d , r e q u i r i n g you t o s t a r t a g a i n from a
new b a s e backup .

52

4.2. (Streaming Replication)


-

restore_command = s c p masterdb_host : / path_to / a r c h i v e/% f "%p


"

standby_mode=on ;
primary_conninfo ;
trigger_file -,
;
restore_command , WAL
. scp masterdb (masterdb_host
- masterdb).
PostgreSQL slavedb.

:
4.17
Line 1
5

$ p s q l - c "SELECT p g _ c u r r e n t _ x l o g _ l o c a t i o n ( ) " - h192 . 1 6 8 . 0 . 1 0


( masterdb )
pg_current_xlog_location
-- -----------------------0/2000000
( 1 row )

10
-

$ p s q l - c " s e l e c t p g _ l a s t _ x l o g _ r e c e i v e _ l o c a t i o n ( ) " - h192


. 1 6 8 . 0 . 2 0 ( slavedb )
pg_last_xlog_receive_location
-- ----------------------------0/2000000
( 1 row )

15
-

$ p s q l - c " s e l e c t p g _ l a s t _ x l o g _ r e p l a y _ l o c a t i o n ( ) " - h192


. 1 6 8 . 0 . 2 0 ( slavedb )
pg_last_xlog_replay_location
-- ---------------------------0/2000000
( 1 row )

9.1 view
. master slaves:
4.18
Line 1

# SELECT * from p g _ s t a t _ r e p l i c a t i o n ;

53

4.2. (Streaming Replication)


-

procpid | usesysid |
usename
| application_name |
client_addr | client_hostname | client_port |
backend_start
|
state
| sent_location |
write_location | flush_location | replay_location |
sync_priority | sync_state
-- - - - - - -+ - - - - - - - - - -+ - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - -+ - - - - - 17135 |
16671 | r e p l i c a t i o n | n e w c l u s t e r
|
127.0.0.1
|
|
43745 | 2 0 1 1 - 0 5 - 2 2
1 8 : 1 3 : 0 4 . 1 9 2 8 3 + 0 2 | s t r e a m i n g | 1/30008750
|
1/30008750
| 1/30008750
| 1/30008750
|
1 | sync

9.1 view pg_stat_database_conflicts,


:
4.19

# SELECT * from p g _ s t a t _ d a t a b a s e _ c o n f l i c t s ;
d a t i d | datname | c o n f l _ t a b l e s p a c e | c o n f l _ l o c k |
confl_snapshot | confl_bufferpin | confl_deadlock
-- - - - -+ - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - -+ - - - - - - - - - - - - - - - -+ - - - - -

Line 1

5
-

1 | template1
0 |
11979 | t e m p l a t e 0
0 |
11987 | p o s t g r e s
0 |
16384 | marc
1 |

0 |
0 |

0 |
0

0 |
0 |

0 |
0

0 |
0 |

0 |
0

0 |
0 |

0 |
0

ps:
4.20
Line 1
-

[ masterdb ] $ ps - e f | g r e p s e n d e r
p o s t g r e s 6879 6831 0 1 0 : 3 1 ?
00:00:00 postgres :
wal s e n d e r p r o c e s s p o s t g r e s 1 2 7 . 0 . 0 . 1 ( 4 4 6 6 3 ) s t r e a m i n g
0/2000000

[ s l a v e d b ] $ ps - e f | g r e p r e c e i v e r
p o s t g r e s 6878 6872 1 1 0 : 3 1 ?
00:00:01 postgres :
wal r e c e i v e r p r o c e s s
s t r e a m i n g 0/2000000

. :
4.21

54

4.2. (Streaming Replication)


Line 1
5
-

$ p s q l test_db
test_db=# c r e a t e t a b l e t e s t 3 ( i d
name v a r c h a r ( 2 0 ) ) ;
NOTICE : CREATE TABLE / PRIMARY
i n d e x " test3_pkey " f o r t a b l e
CREATE TABLE
test_db=# i n s e r t i n t o t e s t 3 ( id ,
INSERT 0 1
test_db=#

i n t not n u l l primary key ,


KEY w i l l c r e a t e i m p l i c i t
" test3 "
name ) v a l u e s ( 1 , t e s t 1 ) ;

:
4.22
Line 1
5
-

$ p s q l test_db
test_db=# s e l e c t * from t e s t 3 ;
i d | name
- - - -+ - - - - - - 1 | test1
( 1 row )

,
.



- ( trigger_file ) , .

- ( trigger_file ) . 9.1 pg_xlog_replay_pause() pg_xlog_replay_resume() .

. , .

PostgreSQL .

55

4.3. PostgreSQL Bi-Directional Replication (BDR)



, ,
. PostgreSQL
.

4.3 PostgreSQL Bi-Directional Replication (BDR)


BDR (Bi-Directional Replication) PostgreSQL
.
. PostgreSQL 9.5. BDR
LLSR (Logical Log Streaming Replication).
BDR , ..
- . ,
. . .
, (eventually consistent).
BDR
, trigger-based (Londiste, Slony,
Bucardo).
,
BDR (LLSR) (per-database replication),
PLSR (per-cluster replication), ..
. :
INSERT/DELETE/UPDATE (TRUNCATE );
(DDL)
. DDL (
CREATE TABLE ... AS);
, , .. upstream downstream ;
WAL, (
, ..). (LLSR)

56

4.4. Slony-I
PLSR ( LLSR
PLSR).
: downstream . upstream WAL . . BDR downstream
downstream . upstream
pg_drop_replication_slot(slotname).
pg_get_replication_slots().
BDR
. PostgreSQL
BDR .

4.4 Slony-I

Slony ,
PostgreSQL . Slony Postgre
INSERT/DELETE/UPDATE .
Slony : slony slonik. slonik-,
slon .
, slon , .
slonik-e
slonik stdin.
slonik-a , , ,
slonik syntax error, .
. .

Ubuntu :

57

4.4. Slony-I
4.23
Line 1

$ sudo a p t i t u d e i n s t a l l s l o n y 1 - 2 - b i n


customers ( , ).
:

customers ;
master_host master ;
slave_host slave ;
customers_rep .

master
, Slony. , ,
slony.
4.24 master-
Line 1
-

$ c r e a t e u s e r -a -d slony
$ p s q l - d t e m p l a t e 1 - c "ALTER USER s l o n y WITH PASSWORD
slony_user_password ; "


slony, slon. , ( slon)
.
slave
,
.
PostgreSQL master , . , :
4.25 slave-
Line 1
-

anyuser@customers_slave$ p s q l - d c u s t o m e r s \
- h customers_master . com -U s l o n y

- ( , ). - , firewall-a, pg_hba.conf, $PGDATA.


slave- PostgreSQL.
, Postgres up and ready,
58

4.4. Slony-I
- , (
postmaster):
4.26 slave-
Line 1
5
-

p g s q l @ c u s t o m e r s _ s l a v e $ rm - r f $PGDATA
p g s q l @ c u s t o m e r s _ s l a v e $ mkdir $PGDATA
p g s q l @ c u s t o m e r s _ s l a v e $ i n i t d b -E UTF8 -D $PGDATA
pgsql@customers_slave$ c r e a t e u s e r - a -d slony
pgsql@customers_slave$ psql -d template1 - c " a l t e r \
u s e r s l o n y with password slony_user_password ; "

postmaster.
! . !
4.27 slave-
Line 1
-

p g s q l @ c u s t o m e r s _ s l a v e $ c r e a t e u s e r - a - d customers_owner
pgsql@customers_slave$ psql -d template1 - c " a l t e r \
u s e r customers_owner with password customers_owner_password
;"

customers_master,
-h customers_slave, slave.
slave, master, Slony.
plpgsql slave
slony.

(slony_user_password). :
4.28 plpgsql slave
Line 1
-

slony@customers_master$ c r e a t e d b -O customers_owner \
- h c u s t o m e r s _ s l a v e . com c u s t o m e r s
slony@customers_master$ c r e a t e l a n g - d c u s t o m e r s \
- h c u s t o m e r s _ s l a v e . com p l p g s q l

! , replication set
primary key. - , primary
key ALTER TABLE ADD PRIMARY KEY.
primary key ,
serial (ALTER TABLE ADD COLUMN),
. table add
key slonik-a.
. slave:
59

4.4. Slony-I
4.29 plpgsql slave
Line 1
-

slony@customers_master$ pg_dump - s c u s t o m e r s | \
p s q l -U s l o n y - h c u s t o m e r s _ s l a v e . com c u s t o m e r s
pg_dump -s .
pg_dump -s customers , psql -U slony
-h customers_slave.com customers (slony_user_pass

). : -
Slony ( make install ), sl_*,
. , :
( 5);
:-) slave :
4.30 plpgsql slave
Line 1
5
-

s l o n i k <<EOF
c l u s t e r name = c u s t o m e r s _ s l a v e ;
node Y admin c o n n i n f o = dbname=c u s t o m e r s h o s t=
customers_master . com
p o r t =5432 u s e r=s l o n y password=slony_user_pass ;
u n i n s t a l l node ( i d = Y) ;
echo okay ;
EOF

Y . . : , cluster
name - , T1 (default).
uninstall.
( ), uninstall ( master slave).

PgSQL
, - ,
.
- :
4.31
Line 1

#!/ b i n / sh

CLUSTER=customers_rep

5
-

DBNAME1=c u s t o m e r s
DBNAME2=c u s t o m e r s

60

4.4. Slony-I
-

HOST1=customers_master . com
HOST2=c u s t o m e r s _ s l a v e . com

10
-

PORT1=5432
PORT2=5432

SLONY_USER=s l o n y

15
20
-

s l o n i k <<EOF
c l u s t e r name = $CLUSTER ;
node 1 admin c o n n i n f o = dbname=$DBNAME1 h o s t=$HOST1 p o r t=
$PORT1
u s e r=s l o n y password=slony_user_password ;
node 2 admin c o n n i n f o = dbname=$DBNAME2 h o s t=$HOST2
p o r t=$PORT2 u s e r=s l o n y password=slony_user_password ;
i n i t c l u s t e r ( i d = 1 , comment = Customers DB
replication cluster ) ;

25

echo C r e a t e s e t ;

c r e a t e s e t ( i d = 1 , o r i g i n = 1 , comment = Customers
DB r e p l i c a t i o n s e t ) ;

30

echo Adding t a b l e s t o t h e s u b s c r i p t i o n s e t ;

echo Adding t a b l e p u b l i c . c u s t o m e r s _ s a l e s . . . ;
s e t add t a b l e ( s e t i d = 1 , o r i g i n = 1 , i d = 4 , f u l l
qualified
- name = p u b l i c . c u s t o m e r s _ s a l e s , comment = Table p u b l i c .
customers_sales ) ;
35 echo done ;
-

40
-

echo Adding t a b l e p u b l i c . customers_something . . . ;


s e t add t a b l e ( s e t i d = 1 , o r i g i n = 1 , i d = 5 , f u l l
qualified
name = p u b l i c . customers_something ,
comment = Table p u b l i c . customers_something ) ;
echo done ;

45
-

echo done adding ;


s t o r e node ( i d = 2 , comment = Node 2 , $HOST2 ) ;
echo s t o r e d node ;
s t o r e path ( s e r v e r = 1 , c l i e n t = 2 , c o n n i n f o = dbname=
$DBNAME1 h o s t=$HOST1
p o r t=$PORT1 u s e r=s l o n y password=slony_user_password ) ;
echo s t o r e d path ;

61

4.4. Slony-I
50

s t o r e path ( s e r v e r = 2 , c l i e n t = 1 , c o n n i n f o = dbname=
$DBNAME2 h o s t=$HOST2
p o r t=$PORT2 u s e r=s l o n y password=slony_user_password ) ;

store l i s t e n ( origin = 1 , provider = 1 , receiver = 2 ) ;


store l i s t e n ( origin = 2 , provider = 2 , receiver = 1 ) ;
- EOF
-

, ,
. : ,
, id , primary key.
: replication set .
set.
: . unsubscribe subscribe .
slave- replication set
:
4.32 slave- replication set
Line 1

#!/ b i n / sh

CLUSTER=customers_rep

5
-

DBNAME1=c u s t o m e r s
DBNAME2=c u s t o m e r s

HOST1=customers_master . com
HOST2=c u s t o m e r s _ s l a v e . com

10
-

PORT1=5432
PORT2=5432

SLONY_USER=s l o n y

15
20
-

s l o n i k <<EOF
c l u s t e r name = $CLUSTER ;
node 1 admin c o n n i n f o = dbname=$DBNAME1 h o s t=$HOST1
p o r t=$PORT1 u s e r=s l o n y password=slony_user_password ;
node 2 admin c o n n i n f o = dbname=$DBNAME2 h o s t=$HOST2
p o r t=$PORT2 u s e r=s l o n y password=slony_user_password ;

echo s u b s c r i b i n g ;
s u b s c r i b e s e t ( id = 1 , provider = 1 , r e c e i v e r = 2 , forward
= no ) ;

62

4.4. Slony-I
25
-

EOF


, .
4.33
slony@customers_master$ s l o n customers_rep \
- "dbname=c u s t o m e r s u s e r=s l o n y "

Line 1

4.34
s l o n y @ c u s t o m e r s _ s l a v e $ s l o n customers_rep \
- "dbname=c u s t o m e r s u s e r=s l o n y "

Line 1

. COPY, slave DB
.
slave-
10- . slon
, .



4.4 4.4.
id = 3. customers_slave3.com,
- PgSQL.
( 4.4.2) :
4.35
Line 1
5
-

s l o n i k <<EOF
c l u s t e r name = c u s t o m e r s _ s l a v e ;
node 3 admin c o n n i n f o = dbname=c u s t o m e r s h o s t=
c u s t o m e r s _ s l a v e 3 . com
p o r t =5432 u s e r=s l o n y password=slony_user_pass ;
u n i n s t a l l node ( i d = 3 ) ;
echo okay ;
EOF

, ,
.
. :

63

4.4. Slony-I
4.36
Line 1

#!/ b i n / sh

CLUSTER=customers_rep

5
-

DBNAME1=c u s t o m e r s
DBNAME3=c u s t o m e r s

HOST1=customers_master . com
HOST3=c u s t o m e r s _ s l a v e 3 . com

10
-

PORT1=5432
PORT2=5432

SLONY_USER=s l o n y

15
20
-

s l o n i k <<EOF
c l u s t e r name = $CLUSTER ;
node 1 admin c o n n i n f o = dbname=$DBNAME1 h o s t=$HOST1
p o r t=$PORT1 u s e r=s l o n y password=slony_user_pass ;
node 3 admin c o n n i n f o = dbname=$DBNAME3
h o s t=$HOST3 p o r t=$PORT2 u s e r=s l o n y password=slony_user_pass
;

echo done adding ;

25
30
-

s t o r e node ( i d = 3 , comment = Node 3 , $HOST3 ) ;


echo s o r e d node ;
s t o r e path ( s e r v e r = 1 , c l i e n t = 3 , c o n n i n f o = dbname=
$DBNAME1
h o s t=$HOST1 p o r t=$PORT1 u s e r=s l o n y password=slony_user_pass
);
echo s t o r e d path ;
s t o r e path ( s e r v e r = 3 , c l i e n t = 1 , c o n n i n f o = dbname=
$DBNAME3
h o s t=$HOST3 p o r t=$PORT2 u s e r=s l o n y password=slony_user_pass
);

35

echo a g a i n ;
store l i s t e n ( origin = 1 , provider = 1 , receiver = 3 ) ;
store l i s t e n ( origin = 3 , provider = 3 , receiver = 1 ) ;

EOF

id 3, 2 .
3 replication set:
4.37

64

4.4. Slony-I
Line 1

#!/ b i n / sh

CLUSTER=customers_rep

5
-

DBNAME1=c u s t o m e r s
DBNAME3=c u s t o m e r s

HOST1=customers_master . com
- HOST3=c u s t o m e r s _ s l a v e 3 . com
-

10

PORT1=5432
- PORT2=5432
-

SLONY_USER=s l o n y

15
20
-

s l o n i k <<EOF
c l u s t e r name = $CLUSTER ;
node 1 admin c o n n i n f o = dbname=$DBNAME1 h o s t=$HOST1
p o r t=$PORT1 u s e r=s l o n y password=slony_user_pass ;
node 3 admin c o n n i n f o = dbname=$DBNAME3 h o s t=$HOST3
p o r t=$PORT2 u s e r=s l o n y password=slony_user_pass ;

echo s u b s c r i b i n g ;
s u b s c r i b e s e t ( id = 1 , provider = 1 , r e c e i v e r = 3 , forward
= no ) ;

25
-

EOF

slon , . slon .
4.38
Line 1
-

s l o n y @ c u s t o m e r s _ s l a v e 3 $ s l o n customers_rep \
"dbname=c u s t o m e r s u s e r=s l o n y "



, : , :
4.39
Line 1
-

%s l o n customers_rep "dbname=c u s t o m e r s u s e r=s l o n y _ u s e r "


CONFIG main : s l o n v e r s i o n 1 . 0 . 5 s t a r t i n g up

65

4.4. Slony-I
5
10
15
20
25
30
35
40
45
-

CONFIG main : l o c a l node i d = 3


CONFIG main : l o a d i n g c u r r e n t c l u s t e r c o n f i g u r a t i o n
CONFIG s t o r e N o d e : no_id=1 no_comment= CustomersDB
replication cluster
CONFIG s t o r e N o d e : no_id=2 no_comment= Node 2 ,
node2 . example . com
CONFIG s t o r e N o d e : no_id=4 no_comment= Node 4 ,
node4 . example . com
CONFIG s t o r e P a t h : pa_server=1 p a _ c l i e n t=3
pa_conninfo="dbname=c u s t o m e r s
h o s t=mainhost . com p o r t =5432 u s e r=s l o n y _ u s e r
password=slony_user_pass " pa_connretry=10
CONFIG s t o r e L i s t e n : l i _ o r i g i n =1 l i _ r e c e i v e r =3
l i _ p r o v i d e r =1
CONFIG s t o r e S e t : s e t _ i d=1 s e t _ o r i g i n =1
set_comment= CustomersDB r e p l i c a t i o n s e t
WARN remoteWorker_wakeup : node 1 - no worker t h r e a d
CONFIG s t o r e S u b s c r i b e : sub_set=1 sub_provider=1 sub_forward=
f
WARN remoteWorker_wakeup : node 1 - no worker t h r e a d
CONFIG e n a b l e S u b s c r i p t i o n : sub_set=1
WARN remoteWorker_wakeup : node 1 - no worker t h r e a d
CONFIG main : c o n f i g u r a t i o n c o m p l e t e - s t a r t i n g t h r e a d s
CONFIG enableNode : no_id=1
CONFIG enableNode : no_id=2
CONFIG enableNode : no_id=4
ERROR remoteWorkerThread_1 : " b e g i n t r a n s a c t i o n ; s e t
transaction isolation level
s e r i a l i z a b l e ; l o c k t a b l e " _customers_rep " . s l _ c o n f i g _ l o c k ;
select
" _customers_rep " . e n a b l e S u b s c r i p t i o n ( 1 , 1 , 4 ) ;
n o t i f y " _customers_rep_Event " ; n o t i f y "
_customers_rep_Confirm " ;
i n s e r t i n t o " _customers_rep " . s l _ e v e n t ( e v _ o r i g i n , ev_seqno ,
ev_timestamp , ev_minxid , ev_maxxid , ev_xip ,
ev_type , ev_data1 , ev_data2 , ev_data3 , ev_data4 ) v a l u e s
( 1 , 219440 ,
2005 -05 -05 18:52:42.708351 , 52501283 , 52501292 ,
5 2 5 0 1 2 8 3 , ENABLE_SUBSCRIPTION ,
1 , 1 , 4 , f ) ; i n s e r t i n t o " _customers_rep " .
s l _ c o n f i r m ( c o n _ o r i g i n , con_received ,
con_seqno , con_timestamp ) v a l u e s ( 1 , 3 , 2 1 9 4 4 0 ,
CURRENT_TIMESTAMP) ; commit t r a n s a c t i o n ; "
PGRES_FATAL_ERROR ERROR: i n s e r t o r update on t a b l e
" s l _ s u b s c r i b e " v i o l a t e s f o r e i g n key
c o n s t r a i n t " s l _ s u b s c r i b e - sl_path - r e f "
DETAIL : Key ( sub_provider , s u b _ r e c e i v e r ) =(1 ,4)

66

4.4. Slony-I
i s not p r e s e n t i n t a b l e " sl_path " .
INFO remoteListenThread_1 : d i s c o n n e c t i n g from
- dbname=c u s t o m e r s h o s t=mainhost . com
50 p o r t =5432 u s e r=s l o n y _ u s e r password=slony_user_pass
- %
-

_< >.sl_path;, _customers_rep.sl_path . , id 4, (1,4)


sl_path .
, Slony.
.
, ( ,
(1,4)):
4.40
Line 1
-

slon y _ u s e r @ m a s t e r h o s t $ p s q l - d c u s t o m e r s - h
_every_one_of_slaves -U s l o n y
c u s t o m e r s=# i n s e r t i n t o _customers_rep . sl_path
v a l u e s ( 1 , 4 , dbname=c u s t o m e r s h o s t=mainhost . com
p o r t =5432 u s e r=s l o n y _ u s e r password=slony_user_password , 10 )
;

, .
_< >,
_customers_rep.


master-, SELECT-
. pg_stat_activity :
4.41
Line 1
5
-

s e l e c t e v _ o r i g i n , ev_seqno , ev_timestamp , ev_minxid ,


ev_maxxid , ev_xip ,
ev_type , ev_data1 , ev_data2 , ev_data3 , ev_data4 , ev_data5 ,
ev_data6 ,
ev_data7 , ev_data8 from " _customers_rep " . s l _ e v e n t e where
( e . e v _ o r i g i n = 2 and e . ev_seqno > 336996 ) o r
( e . e v _ o r i g i n = 3 and e . ev_seqno > 1712871 ) o r
( e . e v _ o r i g i n = 4 and e . ev_seqno > 721285 ) o r
( e . e v _ o r i g i n = 5 and e . ev_seqno > 807715 ) o r
( e . e v _ o r i g i n = 1 and e . ev_seqno > 3544763 ) o r
( e . e v _ o r i g i n = 6 and e . ev_seqno > 2529445 ) o r

67

4.5. Londiste
10
-

( e . e v _ o r i g i n = 7 and e . ev_seqno > 2512532 ) o r


( e . e v _ o r i g i n = 8 and e . ev_seqno > 2500418 ) o r
( e . e v _ o r i g i n = 10 and e . ev_seqno > 1692318 )
o r d e r by e . e v _ o r i g i n , e . ev_seqno ;

_customers_rep ,
.
sl_event - , .
:
4.42
Line 1
-

d e l e t e from _customers_rep . s l _ e v e n t where


ev_timestamp<NOW( ) - 1 DAY : : i n t e r v a l ;

.
_customers_rep.sl_log_*
, - , _customers_rep.sl_log_1
.

4.5 Londiste

Londiste , python. :
. - , Slony-I. Londiste PgQ (
,
, PostgreSQL). :

;
;
;
.

:
, .

68

4.5. Londiste

, Linux,
Ubuntu Server. , ( Windows) ,
PostgreSQL Windows, , .
Londiste Skytools,
. , Debian Ubuntu skytools
:
4.43
Line 1

% sudo a p t i t u d e i n s t a l l s k y t o o l s

2.x, , (failover) (switchover).


. . 3.2.
, :
4.44
Line 1
5
10
15
-

$ wget h t t p : / / pgfoundry . o r g / f r s / download . php /3622/ s k y t o o l s


- 3 . 2 . t a r . gz
$ t a r z x v f s k y t o o l s - 3 . 2 . t a r . gz
$ cd s k y t o o l s - 3 . 2 /
# deb
$ sudo a p t i t u d e i n s t a l l b u i l d - e s s e n t i a l a u t o c o n f \
automake a u t o t o o l s - dev dh - make \
debhelper devscripts fakeroot x u t i l s l i n t i a n pbuilder \
python - a l l - dev python - s u p p o r t xmlto a s c i i d o c \
l i b e v e n t - dev l i b p q - dev l i b t o o l
# python - psycopg L o n d i s t e
$ sudo a p t i t u d e i n s t a l l python - psycopg2 p o s t g r e s q l - s e r v e r dev - a l l
# deb
$ make deb
$ cd . . /
# s k y t o o l s
$ dpkg - i * . deb

Skytools :
4.45
$ ./ configure
$ make
- $ make i n s t a l l

Line 1
-

,
69

4.5. Londiste
4.46
Line 1
5
10
-

$ l o n d i s t e 3 -V
londiste3 , Skytools version 3.2
$ pgqd -V
bad s w i t c h : u s a g e : pgq - t i c k e r [ s w i t c h e s ] c o n f i g . f i l e
Switches :
-v
Increase verbosity
-q
No output t o c o n s o l e
-d
Daemonize
-h
Show h e l p
-V
Show v e r s i o n
-- ini
Show sample c o n f i g f i l e
-s
Stop - send SIGINT t o r u n n i n g p r o c e s s
-k
K i l l - send SIGTERM t o r u n n i n g p r o c e s s
-r
Reload - send SIGHUP t o r u n n i n g p r o c e s s

,
.

:
master-host ;
slave1-host, slave2-host, slave3-host, slave4-host ;
l3simple - ;

master (
/etc/skytools/master-londiste.ini):
4.47
Line 1
5
-

[ londiste3 ]
job_name = m a s t e r _ l 3 s i m p l e
db = dbname=l 3 s i m p l e
queue_name = r e p l i k a
l o g f i l e = / var / l o g / s k y t o o l s / m a s t e r _ l 3 s i m p l e . l o g
p i d f i l e = / var / p i d / s k y t o o l s / m a s t e r _ l 3 s i m p l e . p i d

#
# ( )
10 loop_delay = 0 . 5
-

Londiste master :
4.48 Londiste

70

4.5. Londiste
Line 1
5
10
15
-

$ l o n d i s t e 3 / e t c / s k y t o o l s / master - l o n d i s t e . i n i c r e a t e - r o o t
master - node "dbname=l 3 s i m p l e h o s t=master - h o s t "
INFO p l p g s q l i s i n s t a l l e d
INFO I n s t a l l i n g pgq
INFO
Reading from / u s r / s h a r e / s k y t o o l s 3 / pgq . s q l
INFO pgq . get_batch_cursor i s i n s t a l l e d
INFO I n s t a l l i n g pgq_ext
INFO
Reading from / u s r / s h a r e / s k y t o o l s 3 / pgq_ext . s q l
INFO I n s t a l l i n g pgq_node
INFO
Reading from / u s r / s h a r e / s k y t o o l s 3 /pgq_node . s q l
INFO I n s t a l l i n g l o n d i s t e
INFO
Reading from / u s r / s h a r e / s k y t o o l s 3 / l o n d i s t e . s q l
INFO l o n d i s t e . global_add_table i s i n s t a l l e d
INFO I n i t i a l i z i n g node
INFO L o c a t i o n r e g i s t e r e d
INFO Node " master - node " i n i t i a l i z e d f o r queue " r e p l i k a " with
type " r o o t "
INFO Done

master-server ( ).
:
4.49 master
$ l o n d i s t e 3 - d / e t c / s k y t o o l s / master - l o n d i s t e . i n i worker
- $ t a i l - f / var / l o g / s k y t o o l s / m a s t e r _ l 3 s i m p l e . l o g
- INFO { standby : 1}
- INFO { standby : 1}

Line 1

(, ), -r:
4.50
Line 1

$ l o n d i s t e 3 / e t c / s k y t o o l s / master - l o n d i s t e . i n i - r

-s:
4.51
Line 1

$ l o n d i s t e 3 / e t c / s k y t o o l s / master - l o n d i s t e . i n i - s

(kill -9) :
4.52
Line 1

$ l o n d i s t e 3 / e t c / s k y t o o l s / master - l o n d i s t e . i n i - k

skytools3 ,
/etc/skytools/. /etc/skytools.ini.
londiste :

71

4.5. Londiste
4.53 ticker
$ / e t c / i n i t . d/ s k y t o o l s 3 s t a r t
- INFO S t a r t i n g m a s t e r _ l 3 s i m p l e

Line 1

slave .
:
4.54
Line 1
-

$ p s q l - h s l a v e 1 - h o s t -U p o s t g r e s
# CREATE DATABASE l 3 s i m p l e ;

trust ( ) master slave


.
slave (/etc/skytools/slave1londiste.ini):
4.55 slave
Line 1
5
-

[ londiste3 ]
job_name = s l a v e 1 _ l 3 s i m p l e
db = dbname=l 3 s i m p l e
queue_name = r e p l i k a
l o g f i l e = / var / l o g / s k y t o o l s / s l a v e 1 _ l 3 s i m p l e . l o g
p i d f i l e = / var / p i d / s k y t o o l s / s l a v e 1 _ l 3 s i m p l e . p i d

#
- # ( )
10 loop_delay = 0 . 5
-

Londiste slave :
4.56 Londiste slave
Line 1

$ londiste3 / etc / skytools / slave1 - l o n d i s t e . i n i create - l e a f


s l a v e 1 - node "dbname=l 3 s i m p l e h o s t=s l a v e 1 - h o s t " - - p r o v i d e r
="dbname=l 3 s i m p l e h o s t=master - h o s t "

:
4.57 slave
Line 1

$ l o n d i s t e 3 - d / e t c / s k y t o o l s / s l a v e 1 - l o n d i s t e . i n i worker

:
4.58 slave
$ / e t c / i n i t . d/ s k y t o o l s 3 s t a r t
INFO S t a r t i n g m a s t e r _ l 3 s i m p l e
- INFO S t a r t i n g s l a v e 1 _ l 3 s i m p l e

Line 1
-

72

4.5. Londiste
PgQ ticker
Londiste PgQ ticker , . , ,
, master . ticker (
/etc/skytools/pgqd.ini):
4.59 PgQ ticker
Line 1
-

[ pgqd ]
l o g f i l e = / var / l o g / s k y t o o l s / pgqd . l o g
p i d f i l e = / var / p i d / s k y t o o l s / pgqd . p i d

:
4.60 PgQ ticker
$ pgqd - d / e t c / s k y t o o l s / pgqd . i n i
$ t a i l - f / var / l o g / s k y t o o l s / pgqd . l o g
- LOG S t a r t i n g pgqd 3 . 2
- LOG auto - d e t e c t i n g dbs . . .
5 LOG l 3 s i m p l e : pgq v e r s i o n ok : 3 . 2

Line 1
-

:
4.61 PgQ ticker
$ / e t c / i n i t . d/ s k y t o o l s 3 r e s t a r t
INFO S t a r t i n g m a s t e r _ l 3 s i m p l e
- INFO S t a r t i n g s l a v e 1 _ l 3 s i m p l e
- INFO S t a r t i n g pgqd
5 LOG S t a r t i n g pgqd 3 . 2

Line 1
-

:
4.62
Line 1
-

$ l o n d i s t e 3 / e t c / s k y t o o l s / master - l o n d i s t e . i n i s t a t u s
Queue : r e p l i k a
L o c a l node : s l a v e 1 - node

master - node ( r o o t )
5
|
|
+ - - : s l a v e 1 - node ( l e a f )
-

T a b l e s : 0/0/0
Lag : 44 s , Tick : 5
T a b l e s : 0/0/0
Lag : 44 s , Tick : 5

10

$ l o n d i s t e 3 / e t c / s k y t o o l s / master - l o n d i s t e . i n i members
Member i n f o on master - n o d e @ r e p l i k a :
- node_name
dead
node_location
- -- ----------------------------------------------------------

73

4.5. Londiste
master - node
master - h o s t
- s l a v e 1 - node
slave1 - host

15

False

dbname=l 3 s i m p l e h o s t=

False

dbname=l 3 s i m p l e h o s t=

: , .
add-table:
4.63
$ l o n d i s t e 3 / e t c / s k y t o o l s / master - l o n d i s t e . i n i add - t a b l e - all
- $ l o n d i s t e 3 / e t c / s k y t o o l s / s l a v e 1 - l o n d i s t e . i n i add - t a b l e - a l l - - create - f u l l

Line 1

-- all ,
, , .
master slave, -- dest-table slave . ,
master slave ,
-- create ( -- create - full ,
).
(sequences) :
4.64
Line 1
-

$ l o n d i s t e 3 / e t c / s k y t o o l s / master - l o n d i s t e . i n i add - s e q - - a l l
$ l o n d i s t e 3 / e t c / s k y t o o l s / s l a v e 1 - l o n d i s t e . i n i add - s e q - - a l l

slave (
-- create - full ). master slave:
4.65
Line 1

$ pg_dump - s - n p u b l i c l 3 s i m p l e | p s q l - h s l a v e 1 - h o s t l 3 s i m p l e

:
4.66
$ l o n d i s t e 3 / e t c / s k y t o o l s / master - l o n d i s t e . i n i s t a t u s
- Queue : r e p l i k a
L o c a l node : master - node

Line 1
-

master - node ( r o o t )
5
|
|
+ - - : s l a v e 1 - node ( l e a f )
-

T a b l e s : 4/0/0
Lag : 18 s , Tick : 12
T a b l e s : 0/4/0
Lag : 18 s , Tick : 12

74

4.5. Londiste
, Table (x/y/z).
:
x - ok (replicated). master
, , slave - master ;
y - half (initial copy, not finnished),
master 0, slave
;
z - ignored (table not replicated locally),
master 0, slave , (.. master
, slave ).
:
4.67
Line 1
-

$ l o n d i s t e 3 / e t c / s k y t o o l s / master - l o n d i s t e . i n i s t a t u s
Queue : r e p l i k a
L o c a l node : master - node

master - node ( r o o t )
|
|
+ - - : s l a v e 1 - node ( l e a f )
-

T a b l e s : 4/0/0
Lag : 31 s , Tick : 20

T a b l e s : 4/0/0
Lag : 31 s , Tick : 20

Londiste master slave :


4.68
Line 1
5
-

$ l o n d i s t e 3 / e t c / s k y t o o l s / master - l o n d i s t e . i n i t a b l e s
Tab l e s on node
table_name
merge_state
table_attrs
-- ------------------------------------------------p u b l i c . pgbench_accounts ok
p u b l i c . pgbench_branches ok
p u b l i c . pgbench_history
ok
public . pgbench_tellers
ok

10
-

$ l o n d i s t e 3 / e t c / s k y t o o l s / master - l o n d i s t e . i n i s e q s
S e q u e n c e s on node
seq_name
local
last_value
-- -------------------------------------------------------p u b l i c . pgbench_history_hid_seq True
33345

75

4.5. Londiste

pgbench .
:
4.69
Line 1
5
-

$ pgbench -T 10 - c 5 l 3 s i m p l e
$ t a i l - f / var / l o g / s k y t o o l s / s l a v e 1 _ l 3 s i m p l e . l o g
INFO { count : 1 5 0 8 , d u r a t i o n : 0 . 3 0 7 , i d l e : 0 . 0 0 2 6 }
INFO { count : 1 5 7 2 , d u r a t i o n : 0 . 3 0 8 5 , i d l e : 0 . 0 0 2 }
INFO { count : 1 6 0 0 , d u r a t i o n : 0 . 3 0 8 6 , i d l e : 0 . 0 0 2 6 }
INFO { count : 3 6 , d u r a t i o n : 0 . 0 1 5 7 , i d l e : 2 . 0 1 9 1 }

slave master .

. slave (
/etc/skytools/slave2-londiste.ini):
4.70 slave2
Line 1
5
-

[ londiste3 ]
job_name = s l a v e 2 _ l 3 s i m p l e
db = dbname=l 3 s i m p l e h o s t=s l a v e 2 - h o s t
queue_name = r e p l i k a
l o g f i l e = / var / l o g / s k y t o o l s / s l a v e 2 _ l 3 s i m p l e . l o g
p i d f i l e = / var / p i d / s k y t o o l s / s l a v e 2 _ l 3 s i m p l e . p i d

#
# ( )
10 loop_delay = 0 . 5
-

slave,
create -branch create - leaf (root,
- master , ; branch,
- , ; leaf, -
, ):
4.71 slave2
Line 1
-

$ p s q l - h s l a v e 2 - h o s t - d p o s t g r e s - c "CREATE DATABASE
l3simple ; "
$ pg_dump - s - n p u b l i c l 3 s i m p l e | p s q l - h s l a v e 2 - h o s t l 3 s i m p l e
$ l o n d i s t e 3 / e t c / s k y t o o l s / s l a v e 2 - l o n d i s t e . i n i c r e a t e - branch
s l a v e 2 - node "dbname=l 3 s i m p l e h o s t=s l a v e 2 - h o s t " - - p r o v i d e r
="dbname=l 3 s i m p l e h o s t=master - h o s t "
INFO p l p g s q l i s i n s t a l l e d
INFO I n s t a l l i n g pgq

76

4.5. Londiste
10
15
20
-

INFO
INFO
INFO
INFO
INFO
INFO
INFO
INFO
INFO
INFO
INFO
INFO
INFO
INFO
INFO
INFO
INFO

Reading from / u s r / s h a r e / s k y t o o l s 3 / pgq . s q l


pgq . get_batch_cursor i s i n s t a l l e d
I n s t a l l i n g pgq_ext
Reading from / u s r / s h a r e / s k y t o o l s 3 / pgq_ext . s q l
I n s t a l l i n g pgq_node
Reading from / u s r / s h a r e / s k y t o o l s 3 /pgq_node . s q l
Installing londiste
Reading from / u s r / s h a r e / s k y t o o l s 3 / l o n d i s t e . s q l
l o n d i s t e . global_add_table i s i n s t a l l e d
I n i t i a l i z i n g node
Location r e g i s t e r e d
Location r e g i s t e r e d
S u b s c r i b e r r e g i s t e r e d : s l a v e 2 - node
Location r e g i s t e r e d
Location r e g i s t e r e d
Location r e g i s t e r e d
Node " s l a v e 2 - node " i n i t i a l i z e d f o r queue " r e p l i k a " with
type " branch "
INFO Done

:
4.72 slave2
$ l o n d i s t e 3 / e t c / s k y t o o l s / s l a v e 2 - l o n d i s t e . i n i add - t a b l e - all
- $ l o n d i s t e 3 / e t c / s k y t o o l s / s l a v e 2 - l o n d i s t e . i n i add - s e q - - a l l

Line 1

:
4.73 slave2
Line 1
5
-

$ / e t c / i n i t . d/ s k y t o o l s 3 s t a r t
INFO S t a r t i n g m a s t e r _ l 3 s i m p l e
INFO S t a r t i n g s l a v e 1 _ l 3 s i m p l e
INFO S t a r t i n g s l a v e 2 _ l 3 s i m p l e
INFO S t a r t i n g pgqd
LOG S t a r t i n g pgqd 3 . 2

slave3 slave4, provider :


4.74 slave3 slave4
$ londiste3 / etc / skytools / slave3
s l a v e 3 - node "dbname=l 3 s i m p l e
="dbname=l 3 s i m p l e h o s t=s l a v e 2
- $ londiste3 / etc / skytools / slave4
s l a v e 4 - node "dbname=l 3 s i m p l e
="dbname=l 3 s i m p l e h o s t=s l a v e 3

Line 1

- l o n d i s t e . i n i c r e a t e - branch
h o s t=s l a v e 3 - h o s t " - - p r o v i d e r
- host "
- l o n d i s t e . i n i c r e a t e - branch
h o s t=s l a v e 4 - h o s t " - - p r o v i d e r
- host "

:
77

4.5. Londiste
4.75
Line 1
-

$ londiste3 / etc / skytools / slave4 - l o n d i s t e . i n i status


Queue : r e p l i k a
L o c a l node : s l a v e 4 - node

5
10
15
-

master - node ( r o o t )
|
|
+ - - : s l a v e 1 - node ( l e a f )
|
|
+ - - : s l a v e 2 - node ( branch )
|
|
+ - - : s l a v e 3 - node ( branch )
|
|
+ - - : s l a v e 4 - node ( branch )

T a b l e s : 4/0/0
Lag : 9 s , Tick : 49
T a b l e s : 4/0/0
Lag : 9 s , Tick : 49
T a b l e s : 4/0/0
Lag : 9 s , Tick : 49
T a b l e s : 4/0/0
Lag : 9 s , Tick : 49
T a b l e s : 4/0/0
Lag : 9 s , Tick : 49

Londiste . ,
provider slave4:
4.76
Line 1 $ l o n d i s t e 3 / e t c / s k y t o o l s / s l a v e 4 - l o n d i s t e . i n i change p r o v i d e r - - p r o v i d e r ="dbname=l 3 s i m p l e h o s t=s l a v e 2 - h o s t "
- $ londiste3 / etc / skytools / slave4 - l o n d i s t e . i n i status
- Queue : r e p l i k a
L o c a l node : s l a v e 4 - node
5
10
15
-

master - node ( r o o t )
|
|
+ - - : s l a v e 1 - node ( l e a f )
|
|
+ - - : s l a v e 2 - node ( branch )
|
|
+ - - : s l a v e 3 - node ( branch )
|
|
+ - - : s l a v e 4 - node ( branch )

T a b l e s : 4/0/0
Lag : 12 s , Tick : 56
T a b l e s : 4/0/0
Lag : 12 s , Tick : 56
T a b l e s : 4/0/0
Lag : 12 s , Tick : 56
T a b l e s : 4/0/0
Lag : 12 s , Tick : 56
T a b l e s : 4/0/0
Lag : 12 s , Tick : 56


takeover:
4.77

78

4.5. Londiste
$ londiste3 / etc / skytools / slave3 - l o n d i s t e . i n i takeover
s l a v e 4 - node
- $ londiste3 / etc / skytools / slave4 - l o n d i s t e . i n i status
- Queue : r e p l i k a
L o c a l node : s l a v e 4 - node

Line 1

5
10
15
-

master - node ( r o o t )
|
|
+ - - : s l a v e 1 - node ( l e a f )
|
|
+ - - : s l a v e 2 - node ( branch )
|
|
+ - - : s l a v e 3 - node ( branch )
|
|
+ - - : s l a v e 4 - node ( branch )

T a b l e s : 4/0/0
Lag : 9 s , Tick : 49
T a b l e s : 4/0/0
Lag : 9 s , Tick : 49
T a b l e s : 4/0/0
Lag : 9 s , Tick : 49
T a b l e s : 4/0/0
Lag : 9 s , Tick : 49
T a b l e s : 4/0/0
Lag : 9 s , Tick : 49

drop-node slave :
4.78
$ l o n d i s t e 3 / e t c / s k y t o o l s / s l a v e 4 - l o n d i s t e . i n i drop - node
s l a v e 4 - node
- $ londiste3 / etc / skytools / slave3 - l o n d i s t e . i n i status
- Queue : r e p l i k a
L o c a l node : s l a v e 3 - node

Line 1

5
10
-

master - node ( r o o t )
|
|
+ - - : s l a v e 1 - node ( l e a f )
|
|
+ - - : s l a v e 2 - node ( branch )
|
|
+ - - : s l a v e 3 - node ( branch )

T a b l e s : 4/0/0
Lag : 9 s , Tick : 49
T a b l e s : 4/0/0
Lag : 9 s , Tick : 49
T a b l e s : 4/0/0
Lag : 9 s , Tick : 49
T a b l e s : 4/0/0
Lag : 9 s , Tick : 49

15
-

tag-dead , slave
( ), tag- alive
.

79

4.5. Londiste




.
4.79
# SELECT queue_name , consumer_name , l a g , l a s t _ s e e n FROM pgq .
get_consumer_info ( ) ;
queue_name |
consumer_name
|
lag
|
last_seen
- -- - - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - -

Line 1

5
-

replika
| . global_watermark
| 00:03:37.108259 |
00:02:33.013915
replika
| slave1_l3simple
| 00:00:32.631509 |
00:00:32.533911
replika
| . s l a v e 1 - node . watermark | 0 0 : 0 3 : 3 7 . 1 0 8 2 5 9 |
00:03:05.01431
lag , last \
_seen .

, 60 .

Londiste , . PGQ, , API:
4.80
Line 1

SELECT pgq . u n r e g i s t e r _ c o n s u m e r ( queue_name , consumer_name


);


:
1.
2.
3.
4.

;
BEGIN; ;
;
COMMIT;

80

4.5. Londiste

1.
2.
3.
4.
5.

BEGIN; ;
;
COMMIT;
lag , londiste ;
.

, ,
.


Londiste lag
, , ticker. UPDATE DELETE
,
. . .
,
pgq.subscription sub_last_tick sub_next_tick.
4.81
Line 1
5
-

SELECT count ( * )
FROM pgq . event_1 ,
(SELECT t i c k _ s n a p s h o t
FROM pgq . t i c k
WHERE t i c k _ i d BETWEEN 5715138 AND 5715139
) as t ( snapshots )
WHERE t x i d _ v i s i b l e _ i n _ s n a p s h o t ( ev_txid , s n a p s h o t s ) ;

, 5 400 .
Londiste,
. Londiste
. INI PgQ
ticker :
4.82
Line 1

pgq_lazy_fetch = 500

Londiste 500 . .

81

4.6. Bucardo

4.6 Bucardo

Bucardo master-master master-slave


PostgreSQL, Perl. , .

Ubuntu Server.
DBIx::Safe Perl .
4.83
Line 1

$ apt - g e t i n s t a l l l i b d b i x - s a f e - p e r l

:
4.84
Line 1
5
-

$
$
$
$
$
$

t a r x v f z d b i x _ s a f e . t a r . gz
cd DBIx - Safe - 1 . 2 . 5
p e r l M a k e f i l e . PL
make
make t e s t
sudo make i n s t a l l

Bucardo. :
4.85
Line 1
5
-

$
$
$
$
$
$

wget h t t p : / / bucardo . o r g / downloads / Bucardo - 5 . 0 . 0 . t a r . gz


t a r x v f z Bucardo - 5 . 0 . 0 . t a r . gz
cd Bucardo - 5 . 0 . 0
p e r l M a k e f i l e . PL
make
sudo make i n s t a l l

Bucardo pl/perl
PostgreSQL.
4.86
Line 1

$ sudo a p t i t u d e i n s t a l l p o s t g r e s q l - p l p e r l - 9 . 3

Perl (DBI, DBD::Pg, Test::Simple,


boolean):
4.87
Line 1

$ sudo a p t i t u d e i n s t a l l l i b d b d - pg - p e r l l i b b o o l e a n - p e r l

82

4.6. Bucardo

Bucardo
:
4.88 Bucardo
Line 1

$ bucardo i n s t a l l

Bucardo PostgreSQL, :
4.89 Bucardo
This w i l l i n s t a l l t h e bucardo d a t a b a s e i n t o an e x i s t i n g
Postgres c l u s t e r .
- P o s t g r e s must have been c o m p i l e d with P e r l support ,
- and you must c o n n e c t a s a s u p e r u s e r

Line 1

We w i l l c r e a t e a new s u p e r u s e r named bucardo ,


- and make i t t h e owner o f a new d a t a b a s e named bucardo

5
10
-

Current c o n n e c t i o n s e t t i n g s :
1 . Host :
<none>
2 . Port :
5432
3 . User :
postgres
4 . Database :
postgres
5 . PID d i r e c t o r y : / var / run / bucardo

,
Bucardo bucardo bucardo.
Unix socket,
pg_hda.conf.

bucardo show all:
4.90 Bucardo
Line 1
5
10
-

$ bucardo show a l l
autosync_ddl
bucardo_initial_version
bucardo_vac
bucardo_version
ctl_checkonkids_time
ctl_createkid_time
ctl_sleep
default_conflict_strategy
default_email_from
default_email_host
default_email_to
...

=
=
=
=
=
=
=
=
=
=
=

newcol
5.0.0
1
5.0.0
10
0.5
0.2
bucardo_latest
nobody@example . com
localhost
nobody@example . com

83

4.6. Bucardo

,
Bucardo. master_db slave_db.
simple_database . :
4.91
$ bucardo add db master_db dbname=s im ple _d ata ba se h o s t=
master_host
- Added d a t a b a s e " master_db "

Line 1

master_db
( , master_db slave_db
simple_database Bucardo).
slave_db:
4.92
Line 1

$ bucardo add db slave_db dbname=sim pl e_d at aba se p o r t =5432


h o s t=s l a v e _ h o s t


. sync:
4.93
Line 1
5
-

$ bucardo add sync d e l t a dbs=master_db : s o u r c e , slave_db :


t a r g e t c o n f l i c t _ s t r a t e g y=b u c a r d o _ l a t e s t t a b l e s=a l l
Added sync " d e l t a "
Created a new r e l g r o u p named " d e l t a "
Created a new dbgroup named " d e l t a "
Added t a b l e " p u b l i c . pgbench_accounts "
Added t a b l e " p u b l i c . pgbench_branches "
Added t a b l e " p u b l i c . pgbench_history "
Added t a b l e " p u b l i c . p g b e n c h _ t e l l e r s "

Bucardo PostgreSQL
master-slave . :
dbs , . source target , master slave
( );
conflict_strategy master-master
Bucardo .
:
bucardo_source source;
bucardo_target target;
84

4.6. Bucardo
bucardo_skip . ;
bucardo_random ,
;
bucardo_latest , ;
bucardo_abort ;
tables , . all
;
master-master :
4.94
Line 1

$ bucardo add sync d e l t a dbs=master_db : s o u r c e , slave_db :


s o u r c e c o n f l i c t _ s t r a t e g y=b u c a r d o _ l a t e s t t a b l e s=a l l

master-master master-slave :
4.95
Line 1

$ bucardo add sync d e l t a dbs=master_db1 : s o u r c e , master_db2 :


s o u r c e , slave_db1 : t a r g e t , slave_db2 : t a r g e t
c o n f l i c t _ s t r a t e g y=b u c a r d o _ l a t e s t t a b l e s=a l l

:
4.96
$ bucardo s t a t u s
PID o f Bucardo MCP: 12122
Name
State
Last good
Time
Last I /D
Last bad
Time
- =======+========+============+========+===========+===========+=======

Line 1
-

d e l t a | Good
|

| 13:28:53

| 13m 6 s | 3685/7384 | none

/
:
4.97
Line 1

$ bucardo s t a r t

:
4.98
Line 1

$ bucardo s t o p

85

4.6. Bucardo



4.99
Line 1

$ bucardo show a l l


4.100
Line 1

$ bucardo s e t name=v a l u e

:
4.101
Line 1

$ bu c a r d o _ c t l s e t s y s l o g _ f a c i l i t y=LOG_LOCAL3


4.102
Line 1

$ bucardo r e l o a d _ c o n f i g


5.0 Bucardo : drizzle, mongo, mysql, oracle, redis sqlite ( bucardo add db type, postgres). redis.
redis Perl ( ):
4.103 redis
Line 1 $ a p t i t u d e i n s t a l l l i b r e d i s - p e r l

redis Bucardo:
4.104 redis
$ bucardo add db R dbname=s im ple _d ata ba se type=r e d i s
- Added d a t a b a s e "R"

Line 1

pg_to_redis:
4.105
$ bucardo add dbgroup pg_to_redis master_db : s o u r c e slave_db :
s o u r c e R: t a r g e t
- Created dbgroup " pg_to_redis "
- Added d a t a b a s e " master_db " t o dbgroup " pg_to_redis " a s
source

Line 1

86

4.6. Bucardo
5

Added d a t a b a s e " slave_db " t o dbgroup " pg_to_redis " a s s o u r c e


Added d a t a b a s e "R" t o dbgroup " pg_to_redis " a s t a r g e t

:
4.106 sync
Line 1
5
-

$ bucardo add sync pg_to_redis_sync t a b l e s=a l l dbs=


pg_to_redis s t a t u s=a c t i v e
Added sync " pg_to_redis_sync "
Added t a b l e " p u b l i c . pgbench_accounts "
Added t a b l e " p u b l i c . pgbench_branches "
Added t a b l e " p u b l i c . pgbench_history "
Added t a b l e " p u b l i c . p g b e n c h _ t e l l e r s "

Bucardo PostgreSQL Redis:


4.107 redis
Line 1
5

10
-

$ pgbench -T 10 - c 5 sim pl e_d at aba se


$ r e d i s - c l i monitor
"HMSET" " pgbench_history : 6 " " b i d " "2 " " a i d " " 36291 " " d e l t a "
" 3716 " "mtime" " 2 0 1 4 - 0 7 - 1 1 1 4 : 5 9 : 3 8 . 4 5 4 8 2 4 " " h i d " " 4331 "
"HMSET" " pgbench_history : 2 " " b i d " "1 " " a i d " " 65179 " " d e l t a "
" 2436 " "mtime" " 2 0 1 4 - 0 7 - 1 1 1 4 : 5 9 : 3 8 . 5 0 0 8 9 6 " " h i d " " 4332 "
"HMSET" " pgbench_history : 1 4 " " b i d " " 2" " a i d " " 153001 " " d e l t a
" " -264 " "mtime" " 2 0 1 4 - 0 7 - 1 1 1 4 : 5 9 : 3 8 . 4 7 2 7 0 6 " " h i d " " 4333
"
"HMSET" " pgbench_history : 1 5 " " b i d " " 1" " a i d " " 195747 " " d e l t a
" " -1671 " "mtime" " 2 0 1 4 - 0 7 - 1 1 1 4 : 5 9 : 3 8 . 5 0 9 8 3 9 " " h i d " "
4334 "
"HMSET" " pgbench_history : 3 " " b i d " "2 " " a i d " " 147650 " " d e l t a "
" 3237 " "mtime" " 2 0 1 4 - 0 7 - 1 1 1 4 : 5 9 : 3 8 . 4 8 9 8 7 8 " " h i d " " 4335 "
"HMSET" " pgbench_history : 1 5 " " b i d " " 1" " a i d " " 39521 " " d e l t a "
" -2125 " "mtime" " 2 0 1 4 - 0 7 - 1 1 1 4 : 5 9 : 3 8 . 5 2 6 3 1 7 " " h i d " " 4336
"
"HMSET" " pgbench_history : 1 4 " " b i d " " 2" " a i d " " 60105 " " d e l t a "
" 2555 " "mtime" " 2 0 1 4 - 0 7 - 1 1 1 4 : 5 9 : 3 8 . 6 1 6 9 3 5 " " h i d " " 4337 "
"HMSET" " pgbench_history : 1 5 " " b i d " " 2" " a i d " " 186655 " " d e l t a
" " 930 " "mtime" " 2 0 1 4 - 0 7 - 1 1 1 4 : 5 9 : 3 8 . 5 4 1 2 9 6 " " h i d " " 4338 "
"HMSET" " pgbench_history : 1 5 " " b i d " " 1" " a i d " " 101406 " " d e l t a
" " 668 " "mtime" " 2 0 1 4 - 0 7 - 1 1 1 4 : 5 9 : 3 8 . 5 6 0 9 7 1 " " h i d " " 4339 "
"HMSET" " pgbench_history : 1 5 " " b i d " " 2" " a i d " " 126329 " " d e l t a
" " -4236 " "mtime" " 2 0 1 4 - 0 7 - 1 1 1 4 : 5 9 : 3 8 . 5 9 0 7 " " h i d " " 4340 "
"DEL" " p g b e n c h _ t e l l e r s : 2 0 "

Redis :
4.108 redis
Line 1

$ r e d i s - c l i "HGETALL" " pgbench_history : 1 5 "

87

4.7. RubyRep
5
10
-

1)
2)
3)
4)
5)
6)
7)
8)
9)
10)

" bid "


" 2"
" aid "
" 126329 "
" delta "
" -4236 "
"mtime"
" 2014 -07 -11 1 4 : 5 9 : 3 8 . 5 9 0 7 "
" hid "
" 4340 "

:
4.109 redis

$ bucardo s t a t u s
PID o f Bucardo MCP: 4655
Name
State
Last good
Time
Last I /D
Last bad
Time
- ==================+========+============+========+===========+===========+==

Line 1
-

5
-

delta
| Good
| none
|
pg_to_redis_sync | Good
| 1 4 : 5 9 : 3 9 | 8m 15 s

| 14:59:39

| 8m 15 s | 0/0

| 14:59:40

| 8m 14 s | 646/2546

4.7 RubyRep

RubyRep , ruby. : . master-master,


master-slave , PostgreSQL MySQL. :
;
.
:

MySQL;
;
;
.

88

4.7. RubyRep

RubyRep : Ruby
JRuby. JRuby
.
JRuby
Java.
1. JRuby rubyrep c Rubyforge;
2. ;
Ruby
1. Ruby, Rubygems;
2. ;
MySQL:
4.110
Line 1

$ sudo gem i n s t a l l mysql2

PostgreSQL:
4.111
Line 1

$ sudo gem i n s t a l l p o s t g r e s

3. rubyrep:
4.112
Line 1

$ sudo gem i n s t a l l rubyrep


:
4.113
Line 1

$ rubyrep g e n e r a t e myrubyrep . c o n f

generate
myrubyrep.conf:

4.114
Line 1
-

RR : : I n i t i a l i z e r : : run do | c o n f i g |
config . l e f t = {

89

4.7. RubyRep
: adapter
: database
: username
: password
: host

5
-

=>
=>
=>
=>
=>

p o s t g r e s q l , # o r mysql
SCOTT ,
scott ,
tiger ,
172.16.1.1

10
15
-

config . right =
: a d a p t e r =>
: d a t a b a s e =>
: username =>
: password =>
: host
=>
}

{
postgresql ,
SCOTT ,
scott ,
tiger ,
172.16.1.2

c o n f i g . i n c l u d e _ t a b l e s dept
c o n f i g . i n c l u d e _ t a b l e s /^ e / # r e g e x p matches a l l t a b l e s
s t a r t i n g with e
20
# c o n f i g . i n c l u d e _ t a b l e s / . / # r e g e x p matches a l l t a b l e s
- end
-

. left
right. config .include_tables
( RegEx).

:
4.115
Line 1

$ rubyrep s c a n - c myrubyrep . c o n f

:
4.116
Line 1
-

dept 100% . . . . . . . . . . . . . . . . . . . . . . . . .
emp 100% . . . . . . . . . . . . . . . . . . . . . . . . .

0
1

dept , emp
.

:
4.117
Line 1

$ rubyrep sync - c myrubyrep . c o n f

:
90

4.8.
4.118
Line 1

$ rubyrep sync - c myrubyrep . c o n f dept /^ e /


. .

:
4.119
Line 1

$ rubyrep r e p l i c a t e - c myrubyrep . c o n f

( )
. ,
. ,
rubyrep. ,
.
:
4.120
Line 1

$ rubyrep u n i n s t a l l - c myrubyrep . c o n f



rubyrep Ruby :
4.121
Line 1
5
-

$ rubyrep r e p l i c a t e - c myrubyrep . c o n f
V e r i f y i n g RubyRep t a b l e s
Checking f o r and removing rubyrep t r i g g e r s from u n c o n f i g u r e d
tables
V e r i f y i n g rubyrep t r i g g e r s o f c o n f i g u r e d t a b l e s
Starting replication
E x c e p t i o n caught : Thread#j o i n : d e a d l o c k 0 xb76ee1ac - mutual
j o i n (0 xb758cfac )

Ruby.
JRuby.

4.8
, PostgreSQL.
91

4.8.
, , ..
PostgreSQL.
.
, 9.0 PostgreSQL. Slony-I
,
, , (failover) (switchover).
Londiste , . Bucardo
master-master, master-slave . RubyRep,
master-master , ,
( ).

92

,
.

5.1
. .
. , . .
,
(, 90% ). , ( !)
, ().
, , , ..
.
, , . ..
. ? . ,
. ,
10 , 10 - ..
.. - ,
, .
, ID
(user_id) -

93

5.2. PL/Proxy
. .. :
,
user_id;
;
.
:
- =. , ,
.
, .
=;
() . , ,
( , ).
.
,
.. .
, ,
, (, ,
). . , ,
, ID , 100 .
. PostgreSQL
:

Postgres-XC
PL/Proxy
HadoopDB (Shared-nothing clustering)
Greenplum Database

5.2 PL/Proxy
PL/Proxy - .

94

5.2. PL/Proxy
, , , (,
, , - ).
PL/Proxy ? .
, ,
26 . ,
-, : , , - . .
PL/Proxy
OLTP . failover-
, -, .
:
autocommit-
;
SELECT;
;
-
-; ;

-, PgBouncer;
( , )
-.

1. PL/Proxy ;
2. PL/Proxy make make install.
PL/Proxy .
Ubuntu Server PostgreSQL 9.4:
5.1
Line 1

$ sudo a p t i t u d e i n s t a l l p o s t g r e s q l - 9 . 4 - p l p r o x y

3 PostgreSQL. 2
node1 node2, ,
proxy. pl/proxy
95

5.2. PL/Proxy
.
plproxytest, users. !
node1 node2.
.
plproxytest( ):
5.2
CREATE DATABASE p l p r o x y t e s t
WITH OWNER = p o s t g r e s
ENCODING = UTF8 ;

Line 1

users:
5.3
Line 1
5
-

CREATE TABLE p u b l i c . u s e r s
(
username c h a r a c t e r v a r y i n g ( 2 5 5 ) ,
email character varying (255)
)
WITH (OIDS=FALSE) ;
ALTER TABLE p u b l i c . u s e r s OWNER TO p o s t g r e s ;

users:
5.4
Line 1
5
-

CREATE OR REPLACE FUNCTION p u b l i c . i n s e r t _ u s e r ( i_username


text ,
i_emailaddress
text )
RETURNS i n t e g e r AS
$BODY$
INSERT INTO p u b l i c . u s e r s ( username , e m a i l ) VALUES ( $1 , $2 ) ;
SELECT 1 ;
$BODY$
LANGUAGE s q l VOLATILE ;
ALTER FUNCTION p u b l i c . i n s e r t _ u s e r ( t e x t , t e x t ) OWNER TO
postgres ;

. proxy.
, (proxy) :
5.5
CREATE DATABASE p l p r o x y t e s t
WITH OWNER = p o s t g r e s
ENCODING = UTF8 ;

Line 1

pl/proxy:
96

5.2. PL/Proxy
5.6
Line 1
5
10

CREATE OR REPLACE FUNCTION p u b l i c . p l p r o x y _ c a l l _ h a n d l e r ( )


RETURNS l a n g u a g e _ h a n d l e r AS
$ l i b d i r / plproxy , plproxy_call_handler
LANGUAGE c VOLATILE
COST 1 ;
ALTER FUNCTION p u b l i c . p l p r o x y _ c a l l _ h a n d l e r ( )
OWNER TO p o s t g r e s ;
- - language
CREATE LANGUAGE p l p r o x y HANDLER p l p r o x y _ c a l l _ h a n d l e r ;
CREATE LANGUAGE p l p g s q l ;

, ,
3 , pl/proxy
. .
key-value:
5.7
Line 1
5
10
15
-

CREATE OR REPLACE FUNCTION p u b l i c . g e t _ c l u s t e r _ c o n f i g


( IN cluster_name t e x t ,
OUT " key " t e x t , OUT v a l t e x t )
RETURNS SETOF r e c o r d AS
$BODY$
BEGIN
- - l e t s u s e same c o n f i g f o r a l l c l u s t e r s
key := c o n n e c t i o n _ l i f e t i m e ;
v a l := 3 0 * 6 0 ; - - 30m
RETURN NEXT;
RETURN;
END;
$BODY$
LANGUAGE p l p g s q l VOLATILE
COST 100
ROWS 1 0 0 0 ;
ALTER FUNCTION p u b l i c . g e t _ c l u s t e r _ c o n f i g ( t e x t )
OWNER TO p o s t g r e s ;

, .
DSN :
5.8
Line 1
5
-

CREATE OR REPLACE FUNCTION


p u b l i c . g e t _ c l u s t e r _ p a r t i t i o n s ( cluster_name t e x t )
RETURNS SETOF t e x t AS
$BODY$
BEGIN
IF cluster_name = u s e r c l u s t e r THEN
RETURN NEXT dbname=p l p r o x y t e s t h o s t=node1 u s e r=p o s t g r e s
;

97

5.2. PL/Proxy
10
15
-

RETURN NEXT dbname=p l p r o x y t e s t h o s t=node2 u s e r=p o s t g r e s


;
RETURN;
END IF ;
RAISE EXCEPTION Unknown c l u s t e r ;
END;
$BODY$
LANGUAGE p l p g s q l VOLATILE
COST 100
ROWS 1 0 0 0 ;
ALTER FUNCTION p u b l i c . g e t _ c l u s t e r _ p a r t i t i o n s ( t e x t )
OWNER TO p o s t g r e s ;

:
5.9
Line 1
5
10
15

CREATE OR REPLACE FUNCTION


p u b l i c . g e t _ c l u s t e r _ v e r s i o n ( cluster_name t e x t )
RETURNS i n t e g e r AS
$BODY$
BEGIN
IF cluster_name = u s e r c l u s t e r THEN
RETURN 1 ;
END IF ;
RAISE EXCEPTION Unknown c l u s t e r ;
END;
$BODY$
LANGUAGE p l p g s q l VOLATILE
COST 1 0 0 ;
ALTER FUNCTION p u b l i c . g e t _ c l u s t e r _ v e r s i o n ( t e x t )
OWNER TO p o s t g r e s ;

,
:
5.10
Line 1
5
10
-

CREATE OR REPLACE FUNCTION


p u b l i c . i n s e r t _ u s e r ( i_username t e x t , i _ e m a i l a d d r e s s t e x t )
RETURNS i n t e g e r AS
$BODY$
CLUSTER u s e r c l u s t e r ;
RUN ON h a s h t e x t ( i_username ) ;
$BODY$
LANGUAGE p l p r o x y VOLATILE
COST 1 0 0 ;
ALTER FUNCTION p u b l i c . i n s e r t _ u s e r ( t e x t , t e x t )
OWNER TO p o s t g r e s ;

. proxy :
98

5.2. PL/Proxy
5.11
SELECT i n s e r t _ u s e r ( Sven , sven@somewhere . com ) ;
- SELECT i n s e r t _ u s e r ( Marko , marko@somewhere . com ) ;
- SELECT i n s e r t _ u s e r ( S t e v e , steve@somewhere . com ) ;

Line 1

. :
5.12
Line 1
5
10
-

CREATE OR REPLACE FUNCTION


p u b l i c . get_user_email ( i_username t e x t )
RETURNS SETOF t e x t AS
$BODY$
CLUSTER u s e r c l u s t e r ;
RUN ON h a s h t e x t ( i_username ) ;
SELECT e m a i l FROM p u b l i c . u s e r s
WHERE username = i_username ;
$BODY$
LANGUAGE p l p r o x y VOLATILE
COST 100
ROWS 1 0 0 0 ;
ALTER FUNCTION p u b l i c . get_user_email ( t e x t )
OWNER TO p o s t g r e s ;

:
5.13
Line 1

SELECT p l p r o x y . get_user_email ( S t e v e ) ;

,
, users .

?
pl/proxy
. ,
. 16 .
- . ?
Highload++ 2008,

Skype,
opensource.
, , .

99

5.3. Postgres-XC
, ,
get_cluster_partitions.

5.3 Postgres-XC
Postgres-XC - , . Postgres-XC
, : , GTM (Global Transaction
Manager).
, PostgresXC 64%, .. 10 6.4 , ( ).

PostgreSQL, ,
(
PostgreSQL API). (GTM),
, ,
(
, ).
Postgres-XC MultiMaster,
. , -
( ).

. 5.1 Postgres-XC
:
1. (GTM) Postgres-XC, ( ).
, -

100

5.3. Postgres-XC

. 5.1: Postgres-XC

. , .
2. (coordinators)
().
(
). , (datanodes)
SQL PostgreSQL . , , . ,
, ..
3. (datanodes) . Datanodes .

Postgres-XC . Ubuntu 14.04 postgres-xc :


101

5.3. Postgres-XC
5.14 Postgres-XC
Line 1

$ sudo apt - g e t i n s t a l l p o s t g r e s - xc p o s t g r e s - xc - c l i e n t
p o s t g r e s - xc - c o n t r i b p o s t g r e s - xc - s e r v e r - dev


Postgres-XC :

. 5.2:

. 5.3:

1. (distributed tables, . 5.2): (hash, round-robin, modulo). .


102

5.3. Postgres-XC
. ;
2. (replicated tables, . 5.3): () . ( ) .
,
, .
.


Postgres-XC
PostgreSQL. ( 5432).
.
5.15
CREATE TABLE
users_with_hash ( i d SERIAL , type INT , . . . )
- DISTRIBUTE by HASH( i d ) TO NODE dn1 , dn2 ;

Line 1
-

CREATE TABLE
- users_with_modulo ( i d SERIAL , type INT , . . . )
- DISTRIBUTE by MODULO( i d ) TO NODE dn1 , dn2 ;

CREATE TABLE
u s e r s _ w i t h _ r r o b i n ( i d SERIAL , type INT , . . . )
- DISTRIBUTE by ROUND ROBIN TO NODE dn1 , dn2 ;
-

10

5.15 3 :
1. users_with_hash ( id) .
15 :
5.16
#
$ psql
- # SELECT id , type from users_with_hash ORDER BY i d ;
id
| type
5 - - - - - - -+ - - - - - - 1 |
946
2 |
153
3 |
484

Line 1
-

103

5.3. Postgres-XC
10
15
20

4
5
6
7
8
9
10
11
12
13
14
15

|
|
|
|
|
|
|
|
|
|
|
|

422
785
906
973
699
434
986
135
1012
395
667
324

#
$ p s q l - p15432
- # SELECT id , type from users_with_hash ORDER BY i d ;
25
i d | type
- - - - - - -+ - - - - - - 1 |
946
2 |
153
5 |
785
30
6 |
906
8 |
699
9 |
434
12 | 1012
13 |
395
35
15 |
324
-

#
- $ p s q l - p15433
- # SELECT id , type from users_with_hash ORDER BY i d ;
40
id
| type
- - - - - - - -+ - - - - - - 3 |
484
4 |
422
7 |
973
45
10 |
986
11 |
135
14 |
667
-

2. users_with_modulo
( id)
. 15 :
5.17
#
$ psql
- # SELECT id , type from users_with_modulo ORDER BY i d ;

Line 1
-

104

5.3. Postgres-XC
5
10
15
20

id
| type
- - - - - - -+ - - - - - - 1 |
883
2 |
719
3 |
29
4 |
638
5 |
363
6 |
946
7 |
440
8 |
331
9 |
884
10 |
199
11 |
78
12 |
791
13 |
345
14 |
476
15 |
860

#
$ p s q l - p15432
- # SELECT id , type from users_with_modulo ORDER BY i d ;
25
id
| type
- - - - - - - -+ - - - - - - 2 |
719
4 |
638
6 |
946
30
8 |
331
10 |
199
12 |
791
14 |
476
-

#
$ p s q l - p15433
- # SELECT id , type from users_with_modulo ORDER BY i d ;
i d | type
- - - - - - -+ - - - - - - 40
1 |
883
3 |
29
5 |
363
7 |
440
9 |
884
45
11 |
78
13 |
345
15 |
860
35
-

3.
users_with_rrobin

(round-robin) . 15 :
105

5.3. Postgres-XC
5.18
#
- $ psql
- # SELECT id , type from u s e r s _ w i t h _ r r o b i n ORDER BY i d ;
id
| type
5 - - - - - - -+ - - - - - - 1 |
890
2 |
198
3 |
815
4 |
446
10
5 |
61
6 |
337
7 |
948
8 |
446
9 |
796
15
10 |
422
11 |
242
12 |
795
13 |
314
14 |
240
20
15 |
733

Line 1

#
- $ p s q l - p15432
- # SELECT id , type from u s e r s _ w i t h _ r r o b i n ORDER BY i d ;
25
id
| type
- - - - - - - -+ - - - - - - 2 |
198
4 |
446
6 |
337
30
8 |
446
10 |
422
12 |
795
14 |
240
-

#
- $ p s q l - p15433
- # SELECT id , type from u s e r s _ w i t h _ r r o b i n ORDER BY i d ;
i d | type
- - - - - - -+ - - - - - - 40
1 |
890
3 |
815
5 |
61
7 |
948
9 |
796
45
11 |
242
13 |
314
35

106

5.3. Postgres-XC
-

15 |

733

:
5.19
CREATE TABLE
u s e r s _ r e p l i c a t e d ( i d SERIAL , type INT , . . . )
- DISTRIBUTE by REPLICATION TO NODE dn1 , dn2 ;

Line 1
-

:
5.20
Line 1
5
10
15
-

# SELECT id , type from u s e r s _ r e p l i c a t e d


id
| type
- - - - - - -+ - - - - - - 1 |
75
2 |
262
3 |
458
4 |
779
5 |
357
6 |
51
7 |
249
8 |
444
9 |
890
10 |
810
11 |
809
12 |
166
13 |
605
14 |
401
15 |
58

ORDER BY i d ;

.
:
5.21
# EXPLAIN VERBOSE SELECT * from users_with_modulo ORDER BY
id ;
QUERY PLAN
- -------------------------------------------------------------------------

Line 1

5
-

S o r t ( c o s t = 4 9 . 8 3 . . 5 2 . 3 3 rows =1000 width =8)


Output : id , type
S o r t Key : users_with_modulo . i d
-> R e s u l t ( c o s t = 0 . 0 0 . . 0 . 0 0 rows =1000 width =8)
Output : id , type
-> Data Node Scan on users_with_modulo ( c o s t
= 0 . 0 0 . . 0 . 0 0 rows =1000 width =8)

107

5.3. Postgres-XC
10
-

Output : id , type
Node/ s : dn1 , dn2
Remote query : SELECT id , type FROM ONLY
users_with_modulo WHERE t r u e
( 9 rows )

5.21 , .
:
5.22
Line 1

# EXPLAIN VERBOSE SELECT sum ( i d ) from users_with_modulo


GROUP BY type ;

QUERY PLAN
-

-------------------------------------------------------------------------

5
10
-

HashAggregate ( c o s t = 5 . 0 0 . . 5 . 0 1 rows=1 width =8)


Output : pg_catalog . sum ( ( sum ( users_with_modulo . i d ) ) ) ,
users_with_modulo . type
-> M a t e r i a l i z e ( c o s t = 0 . 0 0 . . 0 . 0 0 rows=0 width =0)
Output : ( sum ( users_with_modulo . i d ) ) ,
users_with_modulo . type
-> Data Node Scan on "__REMOTE_GROUP_QUERY__" (
c o s t = 0 . 0 0 . . 0 . 0 0 rows =1000 width =8)
Output : sum ( users_with_modulo . i d ) ,
users_with_modulo . type
Node/ s : dn1 , dn2
Remote query : SELECT sum ( group_1 . i d ) , group_1
. type FROM (SELECT id , type FROM ONLY users_with_modulo
WHERE t r u e ) group_1 GROUP BY 2
( 8 rows )

JOIN , JOIN
. JOIN
( 5.23).
5.23
# EXPLAIN VERBOSE SELECT * from users_with_modulo ,
users_with_hash WHERE users_with_modulo . i d =
users_with_hash . i d ;
QUERY PLAN
- -------------------------------------------------------------------------

Line 1

108

5.3. Postgres-XC
5
10
15

Nested Loop ( c o s t = 0 . 0 0 . . 0 . 0 1 rows=1 width =16)


Output : users_with_modulo . id , users_with_modulo . type ,
users_with_hash . id , users_with_hash . type
J o i n F i l t e r : ( users_with_modulo . i d = users_with_hash . i d )
-> Data Node Scan on users_with_modulo ( c o s t = 0 . 0 0 . . 0 . 0 0
rows =1000 width =8)
Output : users_with_modulo . id , users_with_modulo .
type
Node/ s : dn1 , dn2
Remote query : SELECT id , type FROM ONLY
users_with_modulo WHERE t r u e
-> Data Node Scan on users_with_hash ( c o s t = 0 . 0 0 . . 0 . 0 0
rows =1000 width =8)
Output : users_with_hash . id , users_with_hash . type
Node/ s : dn1 , dn2
Remote query : SELECT id , type FROM ONLY
users_with_hash WHERE t r u e
( 1 1 rows )

:
5.24
# EXPLAIN VERBOSE SELECT * from u s e r s _ r e p l i c a t e d ;
QUERY PLAN
- -------------------------------------------------------------------------

Line 1
-

5
-

Data Node Scan on "__REMOTE_FQS_QUERY__" ( c o s t = 0 . 0 0 . . 0 . 0 0


rows=0 width =0)
Output : u s e r s _ r e p l i c a t e d . id , u s e r s _ r e p l i c a t e d . type
Node/ s : dn1
Remote query : SELECT id , type FROM u s e r s _ r e p l i c a t e d
( 4 rows )

, ( ).

(HA)
Postgres-XC .
CAP .
, . Postgres-XC
PostgreSQL (streaming) hot-standby
. , . GTM
109

5.4. HadoopDB
, GTM-standby .
.

1. Postgres-XC PostgreSQL 9.1 (9.2 );


2.
( );
3. UNIQUE ;
4. foreign keys
;
5. ( );
6. INSERT ... RETURNING ( );
7. ( ).

Postgres-XC
PostgreSQL. , ( )
,
PostgreSQL.

5.4 HadoopDB
Hadoop ,
. ,
:
: Hadoop , ;
: , .
;
: ,
;

110

5.4. HadoopDB
: , .
;
: , Java, , JVM.
HDFS

Hadoop Distributed File System.
,
:
, , ;

;
;
;
:
;
: , .
HDFS
Namenode
.
.

, . .
Datanode
.

. Rack, , , .

, . .

111

5.4. HadoopDB

. 5.4: HDFS

HDFS : . (
, ; 64 mb) ,
Datanode. , . , -
, /trash
. ,
.
- ,
Datanode Namenode .
Namenode , .
,
.

112

5.4. HadoopDB

, TCP/IP.
Namenode ClientProtocol,
DatanodeProtocol,
Remote Procedure Call (RPC).
,
DFSShell,
DFSAdmin, , -.
API : Java API,
C pipeline, WebDAV .
MapReduce
, Hadoop framework
,
. Job ()
, , :
Map
(
-) -.
.
Reduce
map .
, .
, ,
( ).
JobTracker, TaskTracker. JobTracker , ,
.
, framework,
map reduce,
.
JobTracker .

, , , .
HBase
Hadoop ,
.
113

5.4. HadoopDB
Google
BigTable.
HBase
. , .
HQL ( Hadoop Query
Language), SQL.
.
HBase
, --, . , . ,
. - , . ,
.
HQL , SQL,
help;, .
SELECT, INSERT, UPDATE, DROP ,
.
HBase Shell, HBase
API : Java, Jython, REST Thrift.
HadoopDB
HadoopDB Yale Brown
, MapReduce,
. MapReduce , , . SQL, MapReduce,
. MapReduce ,
.


Ubuntu Server.

114

5.4. HadoopDB
Hadoop
, Hadoop,
, :
ssh ,
[hadoop]:
5.25
Line 1
-

$ sudo groupadd hadoop


$ sudo u s e r a d d -m - g hadoop - d /home/ hadoop - s / b i n /
bash \
- c "Hadoop s o f t w a r e owner " hadoop

:
5.26 hadoop
Line 1

$ su hadoop

RSA- :
5.27 RSA-
Line 1
5
-

$ ssh - keygen - t r s a -P " "


G e n e r a t i n g p u b l i c / p r i v a t e r s a key p a i r .
Enter f i l e i n which t o s a v e t h e key ( / home/ hadoop / . s s h
/ id_rsa ) :
Your i d e n t i f i c a t i o n has been saved i n /home/ hadoop / .
ssh / id_rsa .
Your p u b l i c key has been saved i n /home/ hadoop / . s s h /
i d _ r s a . pub .
The key f i n g e r p r i n t i s :
7b : 5 c : c f : 7 9 : 6 b : 9 3 : d6 : d6 : 8 d : 4 1 : e3 : a6 : 9 d : 0 4 : f 9 : 8 5
hadoop@localhost

:
5.28
Line 1

$ c a t $HOME/ . s s h / i d _ r s a . pub >> $HOME/ . s s h /


authorized_keys

. :
115

5.4. HadoopDB
5.29 ssh
Line 1

$ ssh l o c a l h o s t

sshd:
5.30 sshd
Line 1

$ / e t c / i n i t . d/ ss hd s t a r t

JVM
1.5.0 .
5.31 JVM
Line 1

$ sudo a p t i t u d e i n s t a l l openjdk - 6 - j d k

Hadoop:
5.32 Hadoop
$ cd / opt
- $ sudo wget h t t p : / /www. g t l i b . g a t e c h . edu /pub/ apache / hadoop /
c o r e / hadoop - 0 . 2 0 . 2 / hadoop - 0 . 2 0 . 2 . t a r . gz
- $ t a r z x v f hadoop - 0 . 2 0 . 2 . t a r . gz
- $ l n - s / opt / hadoop - 0 . 2 0 . 2 / opt / hadoop
5 $ chown -R hadoop : hadoop / opt / hadoop / opt / hadoop - 0 . 2 0 . 2
- $ mkdir - p / opt / hadoop - data /tmp - b a s e
- $ chown -R hadoop : hadoop / opt / hadoop - data /

Line 1

/opt/hadoop/conf/hadoop-env.sh :
5.33
Line 1
5
-

$
$
$
$
$
$
$

export
export
export
export
export
export
export

JAVA_HOME=/u s r / l i b /jvm/ java - 6 - openjdk


HADOOP_HOME=/opt / hadoop
HADOOP_CONF=$HADOOP_HOME/ c o n f
HADOOP_PATH=$HADOOP_HOME/ b i n
HIVE_HOME=/opt / h i v e
HIVE_PATH=$HIVE_HOME/ b i n
PATH=$HIVE_PATH:$HADOOP_PATH:$PATH

/opt/hadoop/conf/hadoop-site.xml:
5.34 hadoop
<c o n f i g u r a t i o n>
- <p r o p e r t y>
<name>hadoop . tmp . d i r</name>
<v a l u e>/ opt / hadoop - data /tmp - b a s e</ v a l u e>

Line 1

116

5.4. HadoopDB
<d e s c r i p t i o n>A b a s e f o r o t h e r temporary d i r e c t o r i e s</
d e s c r i p t i o n>
- </ p r o p e r t y>

10
-

<p r o p e r t y>
<name> f s . d e f a u l t . name</name>
<v a l u e>l o c a l h o s t : 5 4 3 1 1</ v a l u e>
<d e s c r i p t i o n>
The name o f t h e d e f a u l t f i l e system .
</ d e s c r i p t i o n>
</ p r o p e r t y>

15
20
-

<p r o p e r t y>
<name>hadoopdb . c o n f i g . f i l e </name>
<v a l u e>HadoopDB . xml</ v a l u e>
<d e s c r i p t i o n>The name o f t h e HadoopDB
c l u s t e r c o n f i g u r a t i o n f i l e </ d e s c r i p t i o n>
</ p r o p e r t y>
</ c o n f i g u r a t i o n>

/opt/hadoop/conf/mapred-site.xml:
5.35 mapreduce
Line 1
5
10

<c o n f i g u r a t i o n>
<p r o p e r t y>
<name>mapred . j o b . t r a c k e r</name>
<v a l u e>l o c a l h o s t : 5 4 3 1 0</ v a l u e>
<d e s c r i p t i o n>
The h o s t and p o r t t h a t t h e
MapReduce j o b t r a c k e r r u n s a t .
</ d e s c r i p t i o n>
</ p r o p e r t y>
</ c o n f i g u r a t i o n>

/opt/hadoop/conf/hdfs-site.xml:
5.36 hdfs
Line 1
5
-

<c o n f i g u r a t i o n>
<p r o p e r t y>
<name>d f s . r e p l i c a t i o n</name>
<v a l u e>1</ v a l u e>
<d e s c r i p t i o n>
Default block r e p l i c a t i o n .
</ d e s c r i p t i o n>
</ p r o p e r t y>
</ c o n f i g u r a t i o n>

Namenode:

117

5.4. HadoopDB
5.37 Namenode
$ hadoop namenode - format
10/05/07 1 4 : 2 4 : 1 2 INFO namenode . NameNode : STARTUP_MSG:
- /*
***********************************************************

Line 1
-

5
10
15
20
25

STARTUP_MSG: S t a r t i n g NameNode
STARTUP_MSG:
h o s t = hadoop1 / 1 2 7 . 0 . 1 . 1
STARTUP_MSG:
a r g s = [ - format ]
STARTUP_MSG:
version = 0.20.2
STARTUP_MSG:
b u i l d = h t t p s : / / svn . apache . o r g / r e p o s
/ a s f / hadoop /common/ b r a n c h e s / branch - 0 . 2 0 - r
9 1 1 7 0 7 ; c o m p i l e d by c h r i s d o on F r i Feb 19 0 8 : 0 7 : 3 4 UTC
2010
***********************************************************
*/
10/05/07 1 4 : 2 4 : 1 2 INFO namenode . FSNamesystem :
fsOwner=hadoop , hadoop
10/05/07 1 4 : 2 4 : 1 2 INFO namenode . FSNamesystem :
s u p e r g r o u p=s u p e r g r o u p
10/05/07 1 4 : 2 4 : 1 2 INFO namenode . FSNamesystem :
i s P e r m i s s i o n E n a b l e d=t r u e
10/05/07 1 4 : 2 4 : 1 2 INFO common . S t o r a g e :
Image f i l e o f s i z e 96 saved i n 0 s e c o n d s .
10/05/07 1 4 : 2 4 : 1 2 INFO common . S t o r a g e :
S t o r a g e d i r e c t o r y / opt / hadoop - data /tmp - b a s e / d f s /name has
been
s u c c e s s f u l l y formatted .
10/05/07 1 4 : 2 4 : 1 2 INFO namenode . NameNode :
SHUTDOWN_MSG:
/*
***********************************************************

SHUTDOWN_MSG: S h u t t i n g down NameNode a t hadoop1 / 1 2 7 . 0 . 1 . 1


- ***********************************************************
*/
-

. Hadoop:
5.38 Hadoop
Line 1
5
-

$ s t a r t - a l l . sh
s t a r t i n g namenode , l o g g i n g t o / opt / hadoop / b i n / . .
/ l o g s / hadoop - hadoop - namenode - hadoop1 . out
l o c a l h o s t : s t a r t i n g datanode , l o g g i n g t o
/ opt / hadoop / b i n / . . / l o g s / hadoop - hadoop - datanode - hadoop1 . out
l o c a l h o s t : s t a r t i n g secondarynamenode , l o g g i n g t o
/ opt / hadoop / b i n / . . / l o g s / hadoop - hadoop - secondarynamenode hadoop1 . out

118

5.4. HadoopDB
s t a r t i n g jobtracker , l o g g i n g to
/ opt / hadoop / b i n / . . / l o g s / hadoop - hadoop - j o b t r a c k e r - hadoop1 . out
10 l o c a l h o s t : s t a r t i n g t a s k t r a c k e r , l o g g i n g t o
- / opt / hadoop / b i n / . . / l o g s / hadoop - hadoop - t a s k t r a c k e r - hadoop1 .
out
-

Hadoop stop-all.sh.
HadoopDB Hive

HaddopDB

hadoopdb.jar

$HADOOP_HOME/lib:
5.39 HadoopDB
Line 1

$ cp hadoopdb . j a r $HADOOP_HOME/ l i b

PostgreSQL JDBC .
$HADOOP_HOME/lib.
Hive HadoopDB SQL . HDFS Hive:
5.40 HadoopDB
$
$
- $
- $

Line 1
-

hadoop
hadoop
hadoop
hadoop

fs
fs
fs
fs

- mkdir
- mkdir
- chmod
- chmod

/tmp
/ u s e r / h i v e / warehouse
g+w /tmp
g+w / u s e r / h i v e / warehouse

HadoopDB SMS_dist. :
5.41 HadoopDB
$ t a r z x v f SMS_dist . t a r . gz
$ mv d i s t / opt / h i v e
- $ chown -R hadoop : hadoop h i v e

Line 1
-

Hadoop,
Hive :
5.42 HadoopDB
$ hive
Hive h i s t o r y f i l e =/tmp/ hadoop /
- hive_job_log_hadoop_201005081717_1990651345 . t x t
- hive >

Line 1
-

5
-

hive > q u i t ;

. :
119

5.4. HadoopDB
5.43
$ svn co h t t p : / / g r a f f i t i . c s . brown . edu / svn / benchmarks /
- $ cd benchmarks / datagen / t e r a g e n

Line 1

benchmarks/datagen/teragen/teragen.pl :
5.44
Line 1
-

use s t r i c t ;
use warnings ;

my $CUR_HOSTNAME = hostname - s ;
5 chomp ($CUR_HOSTNAME) ;
10
-

my
my
my
my
my
my
my

$NUM_OF_RECORDS_1TB
= 10000000000;
$NUM_OF_RECORDS_535MB = 1 0 0 ;
$BASE_OUTPUT_DIR
= "/ data " ;
$PATTERN_STRING
= "XYZ" ;
$PATTERN_FREQUENCY = 1 0 8 2 9 9 ;
$TERAGEN_JAR
= " teragen . jar " ;
$HADOOP_COMMAND
= $ENV{ HADOOP_HOME } . "/ b i n / hadoop " ;

15
20
-

my %f i l e s = ( " 535MB" => 1 ,


);
system ( "$HADOOP_COMMAND f s - rmr $BASE_OUTPUT_DIR" ) ;
f o r e a c h my $ t a r g e t ( k e y s %f i l e s ) {
my $output_dir = $BASE_OUTPUT_DIR. " / S o r t G r e p $ t a r g e t " ;
my $num_of_maps = $ f i l e s { $ t a r g e t } ;
my $num_of_records = ( $ t a r g e t eq " 535MB" ?
$NUM_OF_RECORDS_535MB : $NUM_OF_RECORDS_1TB) ;
p r i n t " G e n e r a t i n g $num_of_maps f i l e s i n $output_dir \ n" ;

25
30
35
-

##
## EXEC: hadoop j a r t e r a g e n . j a r 10000000000
## / data / SortGrep / XYZ 108299 100
##
my @args = ( $num_of_records ,
$output_dir ,
$PATTERN_STRING,
$PATTERN_FREQUENCY,
$num_of_maps ) ;
my $cmd = "$HADOOP_COMMAND j a r $TERAGEN_JAR " . j o i n ( " " ,
@args ) ;
p r i n t "$cmd\n" ;
system ( $cmd ) == 0 | | d i e ( "ERROR: $ ! " ) ;
} # FOR
exit (0) ;

120

5.4. HadoopDB
Perl ,
HDFS.
, HDFS.
. , ,
HDFS, :
5.45
Line 1
5
-

$ hadoop f s - g e t / data /SortGrep535MB/ part - 0 0 0 0 0 my_file


$ psql
p s q l > CREATE DATABASE g r e p 0 ;
p s q l > USE g r e p 0 ;
p s q l > CREATE TABLE g r e p (
->
key1 c h a r a c t e r v a r y i n g ( 2 5 5 ) ,
->
f i e l d character varying (255)
-> ) ;
COPY g r e p FROM my_file WITH DELIMITER | ;

HadoopDB. HadoopDB
Catalog.properties. :
5.46
Line 1
5
10
15
20
25

#P r o p e r t i e s f o r C a t a l o g G e n e r a t i o n
##################################
n o d e s _ f i l e=machines . t x t
r e l a t i o n s _ u n c h u n k e d=grep , E n t i r e R a n k i n g s
r e l a t i o n s _ c h u n k e d=Rankings , U s e r V i s i t s
c a t a l o g _ f i l e=HadoopDB . xml
##
#DB Connection Parameters
##
p o r t =5432
username=p o s t g r e s
password=password
d r i v e r=com . p o s t g r e s q l . D r i v e r
u r l _ p r e f i x=j d b c \ : p o s t g r e s q l \ : / /
##
#Chunking p r o p e r t i e s
##
chunks_per_node=0
unchunked_db_prefix=g r e p
chunked_db_prefix=cdb
##
#R e p l i c a t i o n P r o p e r t i e s
##
dump_script_prefix=/r o o t /dump_
r e p l i c a t i o n _ s c r i p t _ p r e f i x =/r o o t / l o a d _ r e p l i c a _

121

5.4. HadoopDB
30
-

dump_file_u_prefix=/mnt/dump_udb
dump_file_c_prefix=/mnt/dump_cdb
##
#C l u s t e r Connection
##
ssh_key=i d _ r s a

machines.txt localhost (
). HadoopDB HDFS:
5.47
$ j a v a - cp $HADOOP_HOME/ l i b / hadoopdb . j a r \
- > edu . y a l e . c s . hadoopdb . c a t a l o g . S i m p l e C a t a l o g G e n e r a t o r \
- > Catalog . p r o p e r t i e s
- hadoop d f s - put HadoopDB . xml HadoopDB . xml

Line 1

:
5.48
Line 1

$ j a v a - cp hadoopdb . j a r edu . y a l e . c s . hadoopdb . c a t a l o g .


SimpleRandomReplicationFactorTwo C a t a l o g . p r o p e r t i e s

HadoopDB.xml, .
HDFS:
5.49
Line 1
-

$ hadoop d f s - rmr HadoopDB . xml


$ hadoop d f s - put HadoopDB . xml HadoopDB . xml

true

hadoopdb.config.replication

HADOOP_HOME/conf/hadoop-site.xml.
HadoopDB. , HDFS:
5.50
$ j a v a - cp $CLASSPATH : hadoopdb . j a r \
- > edu . y a l e . c s . hadoopdb . benchmark . GrepTaskDB \
- > - p a t t e r n %wo% - output p a d r a i g - hadoop . c o n f i g . f i l e HadoopDB
. xml

Line 1

:
5.51
$ j a v a - cp $CLASSPATH : hadoopdb . j a r edu . y a l e . c s . hadoopdb .
benchmark . GrepTaskDB \
- > - p a t t e r n %wo% - output p a d r a i g - hadoop . c o n f i g . f i l e HadoopDB
. xml
- 1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 8 edu . y a l e . c s . hadoopdb . e x e c . DBJobBase
initConf

Line 1

122

5.4. HadoopDB
5
10
15
20
25
30
-

INFO : SELECT key1 , f i e l d FROM g r e p WHERE f i e l d LIKE %%wo%%


;
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 8 o r g . apache . hadoop . m e t r i c s . jvm . JvmMetrics
init
INFO : I n i t i a l i z i n g JVM M e t r i c s with processName=JobTracker ,
s e s s i o n I d=
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 8 o r g . apache . hadoop . mapred . J o b C l i e n t
configureCommandLineOptions
WARNING: Use G e n e r i c O p t i o n s P a r s e r f o r p a r s i n g t h e arguments .
A p p l i c a t i o n s s h o u l d implement Tool f o r t h e same .
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 8 o r g . apache . hadoop . mapred . J o b C l i e n t
monitorAndPrintJob
INFO : Running j o b : job_local_0001
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 8 edu . y a l e . c s . hadoopdb . c o n n e c t o r .
AbstractDBRecordReader g e t C o n n e c t i o n
INFO : Data l o c a l i t y f a i l e d f o r l e o - p g s q l
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 8 edu . y a l e . c s . hadoopdb . c o n n e c t o r .
AbstractDBRecordReader g e t C o n n e c t i o n
INFO : Task from l e o - p g s q l i s c o n n e c t i n g t o chunk 0 on h o s t
l o c a l h o s t with
db u r l j d b c : p o s t g r e s q l : / / l o c a l h o s t : 5 4 3 4 / g r e p 0
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 8 o r g . apache . hadoop . mapred . MapTask
runOldMapper
INFO : numReduceTasks : 0
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 8 edu . y a l e . c s . hadoopdb . c o n n e c t o r .
AbstractDBRecordReader c l o s e
INFO : DB t i m e s (ms) : c o n n e c t i o n = 1 0 4 , query e x e c u t i o n = 2 0 ,
row r e t r i e v a l = 79
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 8 edu . y a l e . c s . hadoopdb . c o n n e c t o r .
AbstractDBRecordReader c l o s e
INFO : Rows r e t r i e v e d = 3
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 8 o r g . apache . hadoop . mapred . Task done
INFO : Task : attempt_local_0001_m_000000_0 i s done . And i s i n
t h e p r o c e s s o f commiting
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 8 o r g . apache . hadoop . mapred .
LocalJobRunner$Job s t a t u s U p d a t e
INFO :
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 8 o r g . apache . hadoop . mapred . Task commit
INFO : Task attempt_local_0001_m_000000_0 i s a l l o w e d t o
commit now
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 8 o r g . apache . hadoop . mapred .
FileOutputCommitter commitTask
INFO : Saved output o f t a s k attempt_local_0001_m_000000_0
t o f i l e : / home/ l e o / p a d r a i g
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 8 o r g . apache . hadoop . mapred .
LocalJobRunner$Job s t a t u s U p d a t e
INFO :

123

5.4. HadoopDB
35
40
45
50
55
-

1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 8 o r g . apache . hadoop . mapred . Task sendDone


INFO : Task attempt_local_0001_m_000000_0 done .
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 9 o r g . apache . hadoop . mapred . J o b C l i e n t
monitorAndPrintJob
INFO : map 100% r e d u c e 0%
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 9 o r g . apache . hadoop . mapred . J o b C l i e n t
monitorAndPrintJob
INFO : Job c o m p l e t e : job_local_0001
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 9 o r g . apache . hadoop . mapred . Counters l o g
INFO : Counters : 6
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 9 o r g . apache . hadoop . mapred . Counters l o g
INFO :
FileSystemCounters
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 9 o r g . apache . hadoop . mapred . Counters l o g
INFO :
FILE_BYTES_READ=141370
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 9 o r g . apache . hadoop . mapred . Counters l o g
INFO :
FILE_BYTES_WRITTEN=153336
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 9 o r g . apache . hadoop . mapred . Counters l o g
INFO :
Map- Reduce Framework
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 9 o r g . apache . hadoop . mapred . Counters l o g
INFO :
Map i n p u t r e c o r d s =3
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 9 o r g . apache . hadoop . mapred . Counters l o g
INFO :
S p i l l e d Records=0
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 9 o r g . apache . hadoop . mapred . Counters l o g
INFO :
Map i n p u t b y t e s=3
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 9 o r g . apache . hadoop . mapred . Counters l o g
INFO :
Map output r e c o r d s =3
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 9 edu . y a l e . c s . hadoopdb . e x e c . DBJobBase run
INFO :
JOB TIME : 1828 ms .

HDFS, padraig:
5.52
$ cd p a d r a i g
$ c a t part - 0 0 0 0 0
- some data

Line 1
-

PostgreSQL:
5.53
Line 1
5
-

p s q l > s e l e c t * from g r e p where f i e l d l i k e %wo% ;


+- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - - -+
| key1
| field
|
+- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - - -+
some data

1 rows i n s e t ( 0 . 0 0 s e c )

124

5.4. HadoopDB
10

psql>

. .
. PostgreSQL:
5.54
Line 1
-

p s q l > INSERT i n t o
Maybe ) ;
p s q l > INSERT i n t o
Maybewqe ) ;
p s q l > INSERT i n t o
Maybewqesad ) ;
p s q l > INSERT i n t o
string ! ) ;

g r e p ( key1 , f i e l d ) VALUES( I am l i v e ! ,
g r e p ( key1 , f i e l d ) VALUES( I am l i v e ! ,
g r e p ( key1 , f i e l d ) VALUES( I am l i v e ! ,
g r e p ( key1 , f i e l d ) VALUES( : ) , May c o o l

HadoopDB:
5.55
Line 1

5
10
-

15

$ j a v a - cp $CLASSPATH : hadoopdb . j a r edu . y a l e . c s . hadoopdb .


benchmark . GrepTaskDB - p a t t e r n %May% - output p a d r a i g hadoopdb . c o n f i g . f i l e / opt / hadoop / c o n f /HadoopDB . xml
padraig
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 5 edu . y a l e . c s . hadoopdb . e x e c . DBJobBase
initConf
INFO : SELECT key1 , f i e l d FROM g r e p WHERE f i e l d LIKE %%May%%
;
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 6 o r g . apache . hadoop . m e t r i c s . jvm . JvmMetrics
init
INFO : I n i t i a l i z i n g JVM M e t r i c s with processName=JobTracker ,
s e s s i o n I d=
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 6 o r g . apache . hadoop . mapred . J o b C l i e n t
configureCommandLineOptions
WARNING: Use G e n e r i c O p t i o n s P a r s e r f o r p a r s i n g t h e arguments .
A p p l i c a t i o n s s h o u l d implement Tool f o r t h e same .
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 6 o r g . apache . hadoop . mapred . J o b C l i e n t
monitorAndPrintJob
INFO : Running j o b : job_local_0001
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 6 edu . y a l e . c s . hadoopdb . c o n n e c t o r .
AbstractDBRecordReader g e t C o n n e c t i o n
INFO : Data l o c a l i t y f a i l e d f o r l e o - p g s q l
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 6 edu . y a l e . c s . hadoopdb . c o n n e c t o r .
AbstractDBRecordReader g e t C o n n e c t i o n
INFO : Task from l e o - p g s q l i s c o n n e c t i n g t o chunk 0 on h o s t
l o c a l h o s t with db u r l j d b c : p o s t g r e s q l : / / l o c a l h o s t : 5 4 3 4 /
grep0
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 o r g . apache . hadoop . mapred . MapTask
runOldMapper

125

5.4. HadoopDB
20
25
30
35
40
45
50
-

INFO : numReduceTasks : 0
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 edu . y a l e . c s . hadoopdb . c o n n e c t o r .
AbstractDBRecordReader c l o s e
INFO : DB t i m e s (ms) : c o n n e c t i o n = 1 8 1 , query e x e c u t i o n = 2 2 ,
row r e t r i e v a l = 96
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 edu . y a l e . c s . hadoopdb . c o n n e c t o r .
AbstractDBRecordReader c l o s e
INFO : Rows r e t r i e v e d = 4
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 o r g . apache . hadoop . mapred . Task done
INFO : Task : attempt_local_0001_m_000000_0 i s done . And i s i n
t h e p r o c e s s o f commiting
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 o r g . apache . hadoop . mapred .
LocalJobRunner$Job s t a t u s U p d a t e
INFO :
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 o r g . apache . hadoop . mapred . Task commit
INFO : Task attempt_local_0001_m_000000_0 i s a l l o w e d t o
commit now
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 o r g . apache . hadoop . mapred .
FileOutputCommitter commitTask
INFO : Saved output o f t a s k attempt_local_0001_m_000000_0
t o f i l e : / home/ hadoop / p a d r a i g
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 o r g . apache . hadoop . mapred .
LocalJobRunner$Job s t a t u s U p d a t e
INFO :
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 o r g . apache . hadoop . mapred . Task sendDone
INFO : Task attempt_local_0001_m_000000_0 done .
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 o r g . apache . hadoop . mapred . J o b C l i e n t
monitorAndPrintJob
INFO : map 100% r e d u c e 0%
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 o r g . apache . hadoop . mapred . J o b C l i e n t
monitorAndPrintJob
INFO : Job c o m p l e t e : job_local_0001
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 o r g . apache . hadoop . mapred . Counters l o g
INFO : Counters : 6
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 o r g . apache . hadoop . mapred . Counters l o g
INFO :
FileSystemCounters
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 o r g . apache . hadoop . mapred . Counters l o g
INFO :
FILE_BYTES_READ=141345
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 o r g . apache . hadoop . mapred . Counters l o g
INFO :
FILE_BYTES_WRITTEN=153291
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 o r g . apache . hadoop . mapred . Counters l o g
INFO :
Map- Reduce Framework
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 o r g . apache . hadoop . mapred . Counters l o g
INFO :
Map i n p u t r e c o r d s =4
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 o r g . apache . hadoop . mapred . Counters l o g
INFO :
S p i l l e d Records=0
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 o r g . apache . hadoop . mapred . Counters l o g

126

5.4. HadoopDB
55
-

INFO :
Map i n p u t b y t e s=4
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 o r g . apache . hadoop . mapred . Counters l o g
INFO :
Map output r e c o r d s =4
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 edu . y a l e . c s . hadoopdb . e x e c . DBJobBase run
INFO :
JOB TIME : 2332 ms .

May. . :
5.56
Line 1
5
-

$ cd p a d r a i g
$ c a t part - 0 0 0 0 0
I am l i v e !
Maybe
I am l i v e !
Maybewqe
I am l i v e !
Maybewqesad
:)
May c o o l s t r i n g !

PostgreSQL . HadoopDB
PostgreSQL,
PostgreSQL, shared-nothing .
HadoopDB
hadoopdb.sourceforge.net.

, Hive,
HadoopDB. ,
c Hadoop.
Hadoop HaddopDB.
HadoopDB Hadoop. ,
.
HadoopDB
,
, Hadoop.
HadoopDB, , , , PostgreSQL , , PostgreSQL
. , Hadoop Hive
.
HadoopDB
Hadoop , ,
127

5.5.
, , MapReduce. HadoopDB
Hadoop ( ) HadoopDB .

5.5
.
PostgreSQL ,
, . , , .

128

6
PgPool-II

.

6.1
pgpool-II , PostgreSQL
PostgreSQL. :

Pgpool-II PostgreSQL
(.. , , ).
;

Pgpool-II PostgreSQL. 2 , ;

, SELECT
. pgpool-II
PostgreSQL SELECT
,
.
PostgreSQL. .
129

6.2. !

PostgreSQL,
. ,
, . pgpool-II
, .


,
.
.
Pgpool-II PostgreSQL
. ,
() pgpool-II PostgreSQL, () pgpool-II .
pgpool-II , ,
, , pgpoolII .

pgpool.projects.pgfoundry.org.

6.2 !

pgpool-II
.

pgpool-II
pgpool-II . , , .
6.1 pgpool-II
$ ./ configure
$ make
- $ make i n s t a l l

Line 1
-

configure
.
configure ,
130

6.2. !
, , . pgpool-II
/usr/local.
make , make install
. .
: pgpool-II libpq
PostgreSQL 7.4 (3 ).
configure ,
libpq 3 .
6.2 pgpool-II
Line 1

c o n f i g u r e : e r r o r : l i b p q i s not i n s t a l l e d o r l i b p q i s o l d

3 , -
, libpq, ,
configure.
configure libpq
/usr/local/pgsql. PostgreSQL
/usr/local/pgsql -- with-pgsql
-- with-pgsql- includedir -- with-pgsql- libdir
configure.
Linux pgpool-II
. Ubuntu Linux, , :
6.3 pgpool-II
Line 1

$ sudo a p t i t u d e i n s t a l l p g p o o l 2


pgpool-II pgpool.conf.
: = . pgpool-II pgpool.conf.sample. pgpool.conf,
.
6.4
Line 1

$ cp / u s r / l o c a l / e t c / p g p o o l . c o n f . sample / u s r / l o c a l / e t c / p g p o o l
. conf

pgpool-II localhost 9999.


, listen_addresses *.
6.5
Line 1
-

listen_addresses = localhost
p o r t = 9999

131

6.2. !
.
Ubuntu Linux /etc/pgpool.conf.

PCP
pgpool-II
, pgpool-II .. . PCP, . PostgreSQL.
pcp.conf. , (:). .
md5.
6.6 PCP
Line 1

postgres : e8a48653851e28c69d0506508fb27fc5

pgpool-II

pcp.conf.sample. pcp.conf
.
6.7 PCP
Line 1

$ cp / u s r / l o c a l / e t c / pcp . c o n f . sample / u s r / l o c a l / e t c / pcp . c o n f

Ubuntu Linux /etc/pcp.conf.


md5 pg_md5,
pgpool-II. pg_md5
md5 .
, postgres
pg_md5 md5 .
6.8 PCP
$ / u s r / b i n /pg_md5 p o s t g r e s
- e8a48653851e28c69d0506508fb27fc5

Line 1

PCP , pgpool.conf pcp_port.



pcp_port 9898 .
6.9 PCP
Line 1

pcp_port = 9898

132

6.2. !


PostgreSQL pgpoolII. pgpool-II
. ,
.
,
pgpool-II.

pgpool-II 5432, 5433, 5434 . pgpool-II pgpool.conf
.
6.10
Line 1
5
-

backend_hostname0 = l o c a l h o s t
backend_port0 = 5432
backend_weight0 = 1
backend_hostname1 = l o c a l h o s t
backend_port1 = 5433
backend_weight1 = 1
backend_hostname2 = l o c a l h o s t
backend_port2 = 5434
backend_weight2 = 1

backend_hostname, backend_port, backend_weight


, .
0 (.. 0, 1, 2).
backend_weight 1,
SELECT .

/ pgpool-II
pgpool-II .
6.11
Line 1

$ pgpool

, ,
pgpool .
pgpool, -n
pgpool. Pgpool-II -
.
6.12
Line 1

$ pgpool -n &

133

6.3.
, , , .
6.13
Line 1

$ p g p o o l - n - d > /tmp/ p g p o o l . l o g 2>&1 &

-d .

/tmp/pgpool.log. , ,
. , , cronolog.
6.14
$ p g p o o l - n 2>&1 | / u s r / s b i n / c r o n o l o g \
- - h a r d l i n k =/var / l o g / p g s q l / p g p o o l . l o g \
/ var / l o g / p g s q l/%Y-%m-%d - p g p o o l . l o g &

Line 1
-

pgpool-II, .
6.15
Line 1

$ pgpool stop

- , pgpool-II .
pgpool-II , .
6.16
Line 1

$ p g p o o l -m f a s t s t o p

6.3
.
,
6.2 !,
.
pgbench.


true replication_mode pgpool.conf.
6.17
Line 1

replication_mode = true

134

6.3.
replication_mode true, pgpool-II
.
load_balance_mode true, pgpool-II SELECT .
6.18
Line 1

load_balance_mode = t r u e

replication_mode
load_balance_mode.


, pgpool.conf, pgpool-II
. 6.2 / pgpool-II.
pgpool.conf pgpool-II,
.
, . bench_replication.
. createdb pgpool-II
.
6.19
Line 1

$ c r e a t e d b - p 9999 b e n c h _ r e p l i c a t i o n

pgbench - i . - i .
6.20
Line 1

$ pgbench - i - p 9999 b e n c h _ r e p l i c a t i o n


, pgbench -i.
,
.

branches
1
tellers
10
accounts
100000
history
0

shell.
branches, tellers, accounts history
(5432, 5433, 5434).

135

6.4.
6.21
f o r p o r t i n 5432 5433 5 4 3 4 ; do
- >
echo $ p o r t
- >
f o r table_name i n b r a n c h e s t e l l e r s a c c o u n t s h i s t o r y ;
do
- >
echo $table_name
5 >
p s q l - c "SELECT count ( * ) FROM $table_name " - p \
- >
$port bench_replication
- >
done
- > done

Line 1

6.4
. ( partitioning . ). ,
.
pgpool-II ,
(System Database) ( SystemDB).
SystemDB , .
SystemDB dblink.
,
6.2 !,
.
pgbench.


parallel_mode true pgpool.conf.
6.22
Line 1

parallel_mode = true

parallel_mode true . pgpool-II SystemDB .


SystemDB dblink pgpoolII. , listen_addresses
pgpool-II .
136

6.4.
6.23
Line 1

listen_addresses = *

: , , ,
. , - ,
bench_replication, 6.3
.
6.24
Line 1
-

replication_mode = true
load_balance_mode = f a l s e

6.25
Line 1
-

replication_mode = f a l s e
load_balance_mode = t r u e

parallel_mode
load_balance_mode
true, listen_addresses *,
replication_mode false.

SystemDB
, .
, dblink , .
dist_def . ,
, pgpool-II .
SystemDB 5432.
SystemDB
6.26 SystemDB
Line 1
5
-

system_db_hostname = l o c a l h o s t
system_db_port = 5432
system_db_dbname = p g p o o l
system_db_schema = p g p o o l _ c a t a l o g
system_db_user = p g p o o l
system_db_password =

,
pgpool.conf. pgpool pgpool
pgpool.
137

6.4.
6.27 SystemDB
Line 1
-

$ c r e a t e u s e r - p 5432 p g p o o l
$ c r e a t e d b - p 5432 -O p g p o o l p g p o o l

dblink
dblink pgpool. dblink
contrib
PostgreSQL.
dblink .
6.28 dblink
Line 1 $ USE_PGXS=1 make -C c o n t r i b / d b l i n k
- $ USE_PGXS=1 make -C c o n t r i b / d b l i n k i n s t a l l

dblink
dblink pgpool. PostgreSQL
/usr/local/pgsql, dblink.sql ( ) /usr/local/pgsql/share/contrib. dblink.
6.29 dblink
Line 1 $ p s q l - f / u s r / l o c a l / p g s q l / s h a r e / c o n t r i b / d b l i n k . s q l - p 5432
pgpool

dist_def
dist_def, . pgpoolII , system_db.sql /usr/local/share/system_db.sql ( ,

/usr/local). system_db.sql , dist_def.
dist_def.
6.30 dist_def
Line 1

$ p s q l - f / u s r / l o c a l / s h a r e / system_db . s q l - p 5432 -U p g p o o l
pgpool

system_db.sql, dist_def,
pgpool_catalog. system_db_schema
, , , system_db.sql.
dist_def .
.
138

6.4.
6.31 dist_def
Line 1
5
10
-

CREATE TABLE p g p o o l _ c a t a l o g . d i s t _ d e f (
dbname t e x t , - -
schema_name t e x t , - -
table_name t e x t , - -
col_name t e x t NOT NULL CHECK ( col_name = ANY ( c o l _ l i s t ) )
,
- -
c o l _ l i s t t e x t [ ] NOT NULL, - -
t y p e _ l i s t t e x t [ ] NOT NULL, - -
d i s t _ d e f _ f u n c t e x t NOT NULL,
- -
PRIMARY KEY ( dbname , schema_name , table_name )
);

, dist_def, :
(col_name, dist_def_func);
- (dbname, schema_name, table_name,
col_list , type_list).

.
col_name. dist_def_func ,
col_name
, ,
.
- .

, -,
.
replicate_def
,
SQL, dist_def , , ,
replicate_def. replicate_def
system_db.sql dist_def.
replicate_def .
6.32 replicate_def
CREATE TABLE p g p o o l _ c a t a l o g . r e p l i c a t e _ d e f (
dbname t e x t , - -
schema_name t e x t , - -

Line 1
-

139

6.4.
table_name t e x t , - -
c o l _ l i s t t e x t [ ] NOT NULL, - -
t y p e _ l i s t t e x t [ ] NOT NULL, - -
PRIMARY KEY ( dbname , schema_name , table_name )

5
-

);



, pgbench, . pgbench -i -s 3 (..
3). bench_parallel.
sample pgpool-II
dist_def_pgbench.sql.
pgbench. pgpool-II.
6.33
Line 1

$ p s q l - f sample / dist_def_pgbench . s q l - p 5432 p g p o o l

dist_def_pgbench.sql.
dist_def_pgbench.sql
dist_def. accounts.
- aid.
6.34
Line 1
5
10

INSERT INTO p g p o o l _ c a t a l o g . d i s t _ d e f VALUES (


bench_parallel ,
public ,
accounts ,
aid ,
ARRAY[ a i d , b i d , a b a l a n c e , f i l l e r ] ,
ARRAY[ i n t e g e r , i n t e g e r , i n t e g e r ,
character (84) ] ,
pgpool_catalog . dist_def_accounts
);

accounts. , . SQL (, PL/pgSQL, PL/Tcl, ..).


accounts
3, aid 1 300000.

.
SQL- .
140

6.4.
6.35
Line 1
5
-

CREATE OR REPLACE FUNCTION


p g p o o l _ c a t a l o g . d i s t _ d e f _ b r a n c h e s ( anyelement )
RETURNS i n t e g e r AS $$
SELECT CASE WHEN $1 > 0 AND $1 <= 1 THEN 0
WHEN $1 > 1 AND $1 <= 2 THEN 1
ELSE 2
END;
$$ LANGUAGE s q l ;



.
pgbench branches tellers.
, accounts , branches tellers.
6.36
Line 1
5
-

INSERT INTO p g p o o l _ c a t a l o g . r e p l i c a t e _ d e f VALUES (


bench_parallel ,
public ,
branches ,
ARRAY[ b i d , b b a l a n c e , f i l l e r ] ,
ARRAY[ i n t e g e r , i n t e g e r , c h a r a c t e r ( 8 8 ) ]
);

10
15

INSERT INTO p g p o o l _ c a t a l o g . r e p l i c a t e _ d e f VALUES (


bench_parallel ,
public ,
tellers ,
ARRAY[ t i d , b i d , t b a l a n c e , f i l l e r ] ,
ARRAY[ i n t e g e r , i n t e g e r , i n t e g e r , c h a r a c t e r ( 8 4 ) ]
);

Replicate_def_pgbench.sql sample. psql ,


, , .
6.37
Line 1

$ p s q l - f sample / r e p l i c a t e _ d e f _ p g b e n c h . s q l - p 5432 p g p o o l

141

6.4.


, pgpool.conf, pgpool-II
. , 6.2 / pgpool-II.
pgpool.conf pgpool-II
.
, .
bench_parallel.
. createdb pgpool-II
.
6.38
Line 1

$ c r e a t e d b - p 9999 b e n c h _ p a r a l l e l

pgbench - i -s 3. - i .
-s .
6.39
Line 1

$ pgbench - i - s 3 - p 9999 b e n c h _ p a r a l l e l

6.4
.
SELECT pgpool-II
. bench_parallel .

branches
3
tellers
30
accounts
300000
history
0
pgpool-II shell.
accounts
5432, 5433, 5434 9999.
6.40
f o r p o r t i n 5432 5433 5434 i 9 9 9 9 ; do
>
echo $ p o r t
- >
p s q l - c "SELECT min ( a i d ) , max( a i d ) FROM a c c o u n t s " \
- >
-p $port bench_parallel
5 > done

Line 1
-

142

6.5. Master-slave

6.5 Master-slave
pgpool-II ( Slony-I, Londiste).
. master_slave_mode load_balance_mode
true. pgpool-II INSERT/UPDATE/DELETE
Master DB (1 ), SELECT ,
.
, DDL DML
. SELECT ,
/*NO LOAD BALANCE*/ SELECT.
Master/Slave replication_mode false,
master_slave_mode true.

Streaming Replication ( )
master-slave ,
, pgpool-II. PostgreSQL, pgpoolII (
), ( ). , ,
( recovery.conf, trigger_file ), PostgreSQL .
:
6.41 PostgreSQL
Line 1
5
-

#! / b i n / sh
# F a i l o v e r command f o r s t r e m i n g r e p l i c a t i o n .
# This s c r i p t assumes t h a t DB node 0 i s primary , and 1 i s
standby .
#
# I f standby g o e s down , d o e s n o t h i n g . I f primary g o e s down ,
create a
# t r i g g e r f i l e s o t h a t standby t a k e o v e r primary node .
#
# Arguments : $1 : f a i l e d node i d . $2 : new master hostname . $3
: path t o
# trigger f i l e .

10

f a i l e d _ n o d e=$1
new_master=$2
- t r i g g e r _ f i l e=$3
-

# Do n o t h i n g i f standby g o e s down .
- i f [ $ f a i l e d _ n o d e = 1 ] ; then

15

143

6.6.
exit 0;

fi

20
-

# Create t r i g g e r f i l e .
/ u s r / b i n / s s h -T $new_master / b i n / touch $ t r i g g e r _ f i l e

exit 0;

: ,
.
failover_stream.sh pgpool.conf :
6.42
Line 1

failover_command = / p a th _ t o_ s c r ip t / f a i l o v e r _ s t r e a m . sh %d %H
/tmp/ t r i g g e r _ f i l e

/tmp/trigger_file , recovery.conf.
, ,
.

6.6
pgpool-II
pgpool. .
.
. pgpool, .
:

CHECKPOINT;
;
, ;
CHECKPOINT;
;
postmaster ( pgpool_remote_start);
.


:
backend_data_directory PostgreSQL
;
recovery_user PostgreSQL;
recovery_password PostgreSQL;
144

6.6.
recovery_1st_stage_command . . , recovery_1st_stage_command = some_script,
pgpool-II $PGDATA/some_script. ,
pgpool-II recovery_1st_stage;
recovery_2nd_stage_command .
- . , recovery_2st_stage_command = some_script,
pgpool-II $PGDATA/some_script. ,
pgpool-II recovery_2st_stage. , pgpool-II
.

Streaming Replication ( )
master-slave , PostgreSQL.
, . PostgreSQL
pgpool-II ( ).
:
recovery_user. postgres.
6.43 recovery_user
Line 1

recovery_user = postgres

recovery_password recovery_user
.
6.44 recovery_password
Line 1

recovery_password = some_password

recovery_1st_stage_command.
basebackup.sh ($PGDATA),
. :
6.45 basebackup.sh
Line 1
5

#! / b i n / sh
# Recovery s c r i p t f o r s t r e a m i n g r e p l i c a t i o n .
# This s c r i p t assumes t h a t DB node 0 i s primary , and 1
i s standby .
#
d a t a d i r=$1

145

6.7.
-

d e s t h o s t=$2
d e s t d i r=$3

p s q l - c "SELECT pg_start_backup ( Streaming R e p l i c a t i o n ,


true )" postgres

10
-

r s y n c -C - a - - d e l e t e - e s s h - - e x c l u d e p o s t g r e s q l . c o n f - exclude postmaster . pid \


- - e x c l u d e p o s t m a s t e r . o p t s - - e x c l u d e pg_log - - e x c l u d e
pg_xlog \
- - exclude recovery . conf $datadir / $desthost : $destdir /

15

s s h -T l o c a l h o s t mv $ d e s t d i r / r e c o v e r y . done $ d e s t d i r /
recovery . conf

p s q l - c "SELECT pg_stop_backup ( ) " p o s t g r e s

,
rsync .
SSH , recovery_user
.
:
6.46 recovery_1st_stage_command
Line 1

recovery_1st_stage_command = basebackup . sh

recovery_2nd_stage_command .
, , basebackup.sh,
WAL .
C SQL
.
6.47 C SQL
$
- $
- $
- $

Line 1

cd pgpool - I I - x . x . x/ s q l / pgpool - r e c o v e r y
make
make i n s t a l l
p s q l - f pgpool - r e c o v e r y . s q l t e m p l a t e 1

. pcp_recovery_node .

6.7
PgPool-II , PostgreSQL.
146

7


,
.
.
?

7.1
( ) , ,
PostgreSQL. Windows,
.
-, .
, :
PgBouncer
Pgpool

7.2 PgBouncer
PostgreSQL Skype.
.
Session Pooling .
;
;

147

7.2. PgBouncer
Transaction Pooling
. PgBouncer ,
, ;
Statement Pooling . .
, .
(prepared statements) .
PgBouncer :
( 2 );
;
.
:
7.1 PgBouncer
Line 1

$ pgbouncer [ - d ] [ - R ] [ - v ] [ - u u s e r ] <pgbouncer . i n i >

:
7.2 PgBouncer
Line 1
5
10

[ databases ]
t e m p l a t e 1 = h o s t = 1 2 7 . 0 . 0 . 1 p o r t =5432 dbname=t e m p l a t e 1
[ pgbouncer ]
l i s t e n _ p o r t = 6543
listen_addr = 127.0.0.1
auth_type = md5
auth_file = u s e r l i s t . txt
l o g f i l e = pgbouncer . l o g
p i d f i l e = pgbouncer . p i d
admin_users = someuser

userlist.txt : "someuser" "same_password_as_in_server"


pgbouncer:
7.3 PgBouncer
Line 1

$ p s q l - h 1 2 7 . 0 . 0 . 1 - p 6543 pgbouncer

SHOW.

148

7.3. PgPool-II vs PgBouncer

7.3 PgPool-II vs PgBouncer


. PgBouncer , PgPool-II. , PgPool-II ( ), PgBouncer.
PgBouncer , PgPool-II;
PgBouncer ;
PgBouncer (
-).
PgBouncer PgPool-II .

149

8
PostgreSQL
- ,
- .

8.1
, , . SELECT PostgreSQL.
, , , .
SQL ,
, , . PostgreSQL
. ? -,
. ? (MVCC MultiVersion Concurrency
Control) , , ,
. , ,
. , , . -, , , , .
(
), .
150

8.2. Pgmemcache

PostgreSQL:
Pgmemcache ( memcached)
Pgpool-II (query cache)

8.2 Pgmemcache
Memcached , -.
.
-.
.
,
memcached .
Pgmemcache PostgreSQL API libmemcached
memcached.
PostgreSQL , ,
memcached. , .

2.0.6
pgmemcache. Pgmemcache
PostgreSQL 9.2, Ubuntu Server 12.04. Pgmemcache , PostgreSQL PGXS
( , Linux
PGXS). memcached libmemcached
0.38.
,
Pgmemcache:

:
8.1
$ make
- $ sudo make i n s t a l l

Line 1

151

8.2. Pgmemcache
deb ( Debian, Ubuntu)
Debian Ubuntu,
deb
PostgreSQL:
8.2 deb
$ sudo apt - g e t i n s t a l l libmemcached - dev p o s t g r e s q l s e r v e r - dev - 9 . 2 l i b p q - dev d e v s c r i p t s yada f l e x b i s o n
- $ make deb92
- # deb
- $ sudo dpkg - i . . / p o s t g r e s q l - pgmemcache - 9 . 2 * . deb

Line 1

2.0.4 yada deb


:
8.3 deb
Cannot r e c o g n i z e s o u r c e name i n d e b i a n / c h a n g e l o g a t /
u s r / b i n / yada l i n e 1 4 5 , <CHANGELOG> l i n e 1 .
- make : *** [ deb92 ] 9

Line 1


debian/changelog , :
8.4 deb
$ PostgreSQL : pgmemcache/ d e b i a n / c h a n g e l o g , v 1 . 2
2010/05/05 1 9 : 5 6 : 5 0 ormod Exp $ < - - - -
- pgmemcache ( 2 . 0 . 4 ) u n s t a b l e ; urgency=low

Line 1

* v2 . 0 . 4

deb-
.

Pgmemcache
( Pgmemcache)
:
8.5
Line 1
5
-

% p s q l [ mydbname ] [ p g u s e r ]
[ mydbname]=# CREATE EXTENSION pgmemcache ;
#
# BEGIN ;
# \ i / u s r / s h a r e / p o s t g r e s q l / 9 . 2 / c o n t r i b /pgmemcache . s q l
# COMMIT;

152

8.2. Pgmemcache

memcached

memcache_server_add . . -

memcached
PostgreSQL. ,
postgresql.conf :
pgmemcache shared_preload_libraries ( pgmemcache PostgreSQL);
pgmemcache custom_variable_classes ( pgmemcache);
pgmemcache.default_servers, host:port
(port - ) . :
8.6 default_servers
Line 1

pgmemcache . d e f a u l t _ s e r v e r s = 1 2 7 . 0 . 0 . 1 ,
1 9 2 . 1 6 8 . 0 . 2 0 : 1 1 2 1 1 # memcached

pgmemcache pgmemcache.default_behavior.
libmemcached. :
8.7 pgmemcache
Line 1

pgmemcache . d e f a u l t _ b e h a v i o r= BINARY_PROTOCOL: 1

PostgreSQL
memcached.

pgmemcache, memcached .
.
memcached :
8.8 memcache_stats
pgmemcache=# SELECT memcache_stats ( ) ;
memcache_stats
- -- -------------------------

Line 1
5
10
-

Server : 1 2 7 . 0 . 0 . 1 (11211)
p i d : 1116
uptime : 70
time : 1289598098
version : 1.4.5
p o i n t e r _ s i z e : 32
rusage_user : 0.0

153

8.2. Pgmemcache
8.1: pgmemcache

15
-

memcache_server_add(hostname:port::TEXT)
memcache_server_add(hostname::TEXT)

memcached . , 11211.

memcache_add(key::TEXT, value::TEXT,
expire::TIMESTAMPTZ)
memcache_add(key::TEXT, value::TEXT,
expire::INTERVAL)
memcache_add(key::TEXT, value::TEXT)

, .

newval = memcache_decr(key::TEXT,
decrement::INT4)
newval = memcache_decr(key::TEXT)

,
( ).
.

memcache_delete(key::TEXT,
hold_timer::INTERVAL)
memcache_delete(key::TEXT)

. ,
.

memcache_flush_all()

memcached .

value = memcache_get(key::TEXT)

. NULL,
,
.

memcache_get_multi(keys::TEXT[])
memcache_get_multi(keys::BYTEA[])

.
=.

newval = memcache_incr(key::TEXT,
increment::INT4)
newval = memcache_incr(key::TEXT)

,
( ).
.

memcache_replace(key::TEXT, value::TEXT,
expire::TIMESTAMPTZ)
memcache_replace(key::TEXT, value::TEXT,
expire::INTERVAL)
memcache_replace(key::TEXT, value::TEXT)

memcache_set(key::TEXT, value::TEXT,
expire::TIMESTAMPTZ)
memcache_set(key::TEXT, value::TEXT,
expire::INTERVAL)
memcache_set(key::TEXT, value::TEXT)

.
.

stats = memcache_stats()


memcached.

rusage_system : 0 . 2 4 0 0 1
cur r _i t e m s : 0
total_items : 0
bytes : 0
curr_connections : 5
total_connections : 7
connection_structures : 6
cmd_get : 0

154

8.2. Pgmemcache
20
25
-

cmd_set : 0
get_hits : 0
get_misses : 0
evictions : 0
bytes_read : 20
b y t e s _ w r i t t e n : 782
limit_maxbytes : 67108864
threads : 4

( 1 row )

memcached :
8.9
pgmemcache=# SELECT memcache_add ( some_key , t e s t _ v a l u e ) ;
memcache_add
- -- -----------t
5 ( 1 row )

Line 1
-

pgmemcache=# SELECT memcache_get ( some_key ) ;


memcache_get
- -- -----------10
test_value
- ( 1 row )
-

memcached ( ):
8.10
pgmemcache=# SELECT memcache_add ( some_seq , 10 ) ;
memcache_add
- -- -----------t
5 ( 1 row )

Line 1
-

pgmemcache=# SELECT memcache_incr ( some_seq ) ;


memcache_incr
- -- ------------10
11
- ( 1 row )
-

15
-

pgmemcache=# SELECT memcache_incr ( some_seq ) ;


memcache_incr
-- ------------12
( 1 row )

155

8.2. Pgmemcache
pgmemcache=# SELECT memcache_incr ( some_seq , 1 0 ) ;
memcache_incr
- -- ------------22
- ( 1 row )
-

20

pgmemcache=# SELECT memcache_decr ( some_seq ) ;


memcache_decr
- -- ------------21
- ( 1 row )

25
-

30
35

pgmemcache=# SELECT memcache_decr ( some_seq ) ;


memcache_decr
-- ------------20
( 1 row )

pgmemcache=# SELECT memcache_decr ( some_seq , 6 ) ;


memcache_decr
- -- ------------40
14
- ( 1 row )
-

pgmemcache , ,
.
, memcached ( ),
, .
:
8.11
Line 1
5
-

CREATE OR REPLACE FUNCTION auth_passwd_upd ( ) RETURNS TRIGGER


AS $$
BEGIN
IF OLD. passwd != NEW. passwd THEN
PERFORM memcache_set ( user_id_ | | NEW.
u s e r _ i d | | _password , NEW. passwd ) ;
END IF ;
RETURN NEW;
END;
$$ LANGUAGE p l p g s q l ;

:
8.12
Line 1

CREATE TRIGGER auth_passwd_upd_trg AFTER UPDATE ON passwd


FOR EACH ROW EXECUTE PROCEDURE auth_passwd_upd ( ) ;

156

8.3.
(!!!) .
:
8.13
CREATE OR REPLACE FUNCTION auth_passwd_upd ( ) RETURNS TRIGGER
AS $$
- BEGIN
IF OLD. passwd != NEW. passwd THEN
PERFORM memcache_delete ( user_id_ | | NEW.
u s e r _ i d | | _password ) ;
5
END IF ;
RETURN NEW;
- END; $$ LANGUAGE p l p g s q l ;

Line 1

:
8.14
Line 1

CREATE TRIGGER auth_passwd_del_trg AFTER DELETE ON passwd


FOR EACH ROW EXECUTE PROCEDURE auth_passwd_upd ( ) ;

, memcached ( ) .

PostgreSQL Pgmemcache
memcached ,
. memcached,
SQL , PostgreSQL.

8.3
PostgreSQL . PostgreSQL, ,
, , ( , Varnish, ).

157

9.1
PostgreSQL
.
.

9.2 PostGIS
: Open Source
: www.postgis.org
PostGIS
PostgreSQL. PostGIS PostgreSQL
(), , ESRI SDE
Oracle. PostGIS OpenGIS . SQL .

9.3 pgSphere
: Open Source
: pgsphere.projects.postgresql.org
pgSphere PostgreSQL ,
.
( PostGIS)
.

158

9.4. HStore

9.4 HStore
: Open Source
HStore , / PostgreSQL (,
). , , ,
- .
.


:
9.1 hstore
Line 1

# CREATE EXTENSION h s t o r e ;

:
9.2 hstore
Line 1
5

# SELECT a=>1,a=>2 : : h s t o r e ;
hstore
-- -------"a"=>"1 "
( 1 row )

9.2 hstore . :
9.3 hstore
Line 1
5
10
15
-

CREATE TABLE p r o d u c t s (
i d s e r i a l PRIMARY KEY,
name varchar ,
attributes hstore
);
INSERT INTO p r o d u c t s ( name , a t t r i b u t e s )
VALUES (
Geek Love : A Novel ,
author
=> " K a t h e r i n e Dunn" ,
pa g e s
=> 3 6 8 ,
c a t e g o r y => f i c t i o n
),
(
L e i c a M9 ,
m a n u f a c t u r e r => L e i c a ,
type
=> camera ,
megapixels
=> 1 8 ,
sensor
=> " f u l l - frame 35mm"

159

9.4. HStore
20
25
-

),
( MacBook Air 11 ,
m a n u f a c t u r e r => Apple ,
type
=> computer ,
ram
=> 4GB,
storage
=> 256GB,
processor
=> " 1 . 8 ghz I n t e l i 7 d u e l c o r e " ,
weight
=> 2 . 3 8 l b s
);

:
9.4
Line 1
5

# SELECT name , a t t r i b u t e s -> p a g e s a s page FROM p r o d u c t s


WHERE a t t r i b u t e s ? p a g e s ;
name
| page
- - - - - - - - - - - - - - - - - - - -+ - - - - - Geek Love : A Novel | 368
( 1 row )

:
9.5
Line 1
5

# SELECT name , a t t r i b u t e s -> m a n u f a c t u r e r a s m a n u f a c t u r e r


FROM p r o d u c t s WHERE a t t r i b u t e s -> type = computer ;
name
| manufacturer
- - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - MacBook Air 11 | Apple
( 1 row )

:
9.6
# CREATE INDEX p r o d u c t s _ h s t o r e _ i n d e x ON p r o d u c t s USING GIST
( attributes ) ;
- # CREATE INDEX p r o d u c t s _ h s t o r e _ i n d e x ON p r o d u c t s USING GIN (
attributes ) ;

Line 1

c :
9.7
Line 1

# CREATE INDEX product_manufacturer ON p r o d u c t s ( ( p r o d u c t s .


a t t r i b u t e s -> m a n u f a c t u r e r ) ) ;

HStore PostgreSQL.
160

9.5. PLV8

9.5 PLV8
: Open Source
: code.google.com/p/plv8js
PLV8 , PostgreSQL V8 JavaScript.
PostgreSQL JavaScript ,
SQL.


V8 JavaScript
.
. plpgsql:
9.8 plpgsql
Line 1
5
-

CREATE OR REPLACE FUNCTION


p s q l f i b ( n i n t ) RETURNS i n t AS $$
BEGIN
IF n < 2 THEN
RETURN n ;
END IF ;
RETURN p s q l f i b ( n - 1 ) + p s q l f i b ( n - 2 ) ;
END;
$$ LANGUAGE p l p g s q l IMMUTABLE STRICT ;

:
9.9 plpgsql
Line 1
5
10
-

SELECT n , p s q l f i b ( n ) FROM g e n e r a t e _ s e r i e s ( 0 , 3 0 , 5 ) a s n ;
n | psqlfib
- - - -+ - - - - - - - - 0 |
0
5 |
5
10 |
55
15 |
610
20 |
6765
25 |
75025
30 | 832040
( 7 rows )

Time : 1 6 0 0 3 , 2 5 7 ms

, PLV8:
9.10 plv8
Line 1
-

CREATE OR REPLACE FUNCTION


f i b ( n i n t ) RETURNS i n t a s $$

161

9.5. PLV8
5
-

function fib (n) {


r e t u r n n<2 ? n : f i b ( n - 1 ) + f i b ( n - 2 )
}
return f i b (n)

$$ LANGUAGE p l v 8 IMMUTABLE STRICT ;

:
9.11 plv8
Line 1
5
10
-

SELECT n , f i b ( n ) FROM g e n e r a t e _ s e r i e s ( 0 , 3 0 , 5 ) a s n ;
n |
fib
- - - -+ - - - - - - - 0 |
0
5 |
5
10 |
55
15 |
610
20 |
6765
25 | 75025
30 | 832040
( 7 rows )

Time : 5 9 , 2 5 4 ms

PLV8 270 (16003.257/59.254)


plpgsql. PLV8
:
9.12 plv8
Line 1
5
10

CREATE OR REPLACE FUNCTION


f i b 1 ( n i n t ) RETURNS i n t a s $$
var memo = { 0 : 0 , 1 : 1 } ;
function fib (n) {
i f ( ! ( n i n memo) )
memo [ n ] = f i b ( n - 1 ) + f i b ( n - 2 )
r e t u r n memo [ n ]
}
return f i b (n) ;
$$ LANGUAGE p l v 8 IMMUTABLE STRICT ;

:
9.13 plv8
SELECT n , f i b 1 ( n ) FROM g e n e r a t e _ s e r i e s ( 0 , 3 0 , 5 ) a s n ;
n |
fib1
- - - - -+ - - - - - - - 0 |
0
5
5 |
5

Line 1
-

162

9.5. PLV8
10
-

10 |
55
15 |
610
20 |
6765
25 | 75025
30 | 832040
( 7 rows )

Time : 0 , 7 6 6 ms

PLV8 ,
plpgsql.

PLV8 PostgreSQL .
hstore,
:

( )
JSON (MongoDB, CouchDB, Couchbase ..). ,
PostgreSQL 9.2, JSON.
PostgreSQL 9.1 PLV8 DOMAIN:
9.14 JSON
Line 1
5
-

CREATE OR REPLACE FUNCTION


valid_json ( json text )
RETURNS BOOLEAN AS $$
try {
JSON . p a r s e ( j s o n ) ; r e t u r n t r u e ;
} catch ( e ) {
return f a l s e ;
}
$$ LANGUAGE p l v 8 IMMUTABLE STRICT ;

10

CREATE DOMAIN j s o n AS TEXT


- CHECK( v a l i d _ j s o n (VALUE) ) ;
-

valid_json JSON . :
9.15 JSON
$ INSERT INTO members
VALUES( not good j s o n ) ;
- ERROR:
v a l u e f o r domain j s o n

Line 1
-

163

9.5. PLV8
5
10
-

v i o l a t e s check c o n s t r a i n t " json_check "


$ INSERT INTO members
VALUES( {" good " : " j s o n " , " i s " : t r u e } ) ;
INSERT 0 1
$ s e l e c t * from members ;
profile
-- ---------------------------{" good " : " j s o n " , " i s " : t r u e }
( 1 row )

JSON PLV8
. :
9.16 JSON
$ CREATE TABLE members ( i d SERIAL , p r o f i l e j s o n ) ;
- $ SELECT count ( * ) FROM members ;
count
- -- ------5
1000000
- ( 1 row )

Line 1

Time : 2 0 1 . 1 0 9 ms

profile JSON:
9.17 JSON
Line 1

5
10
15
-

+
"name" : " L i t z y S a t t e r f i e l d " ,
+
" age " : 2 4 ,
+
"siblings": 2,
+
" faculty " : false ,
+
" numbers " : [
+
{
+
" type " :
" work " ,
+
"number" : " 6 8 4 . 5 7 3 . 3 7 8 3 x368 "+
},
+
{
+
" type " :
"home" ,
+
"number" : " 6 2 5 . 1 1 2 . 6 0 8 1 "
+
}
+
]
+

JSON (
):
9.18 JSON
Line 1

CREATE OR REPLACE FUNCTION get_numeric ( json_raw j s o n , key


text )

164

9.5. PLV8
RETURNS numeric AS $$
var o = JSON . p a r s e ( json_raw ) ;
r e t u r n o [ key ] ;
5 $$ LANGUAGE p l v 8 IMMUTABLE STRICT ;
-

, age, siblings :
9.19 JSON
$ SELECT * FROM members WHERE get_numeric ( p r o f i l e , age ) =
36;
- Time : 9 3 4 0 . 1 4 2 ms
- $ SELECT * FROM members WHERE get_numeric ( p r o f i l e , s i b l i n g s
) = 1;
- Time : 1 4 3 2 0 . 0 3 2 ms

Line 1

, . , :
9.20
$ CREATE INDEX member_age ON members ( get_numeric ( p r o f i l e ,
age ) ) ;
- $ CREATE INDEX member_siblings ON members ( get_numeric (
profile , siblings ) ) ;

Line 1

JSON :
9.21 JSON
Line 1
5
-

$ SELECT * FROM members WHERE get_numeric ( p r o f i l e , age ) =


36;
Time : 5 7 . 4 2 9 ms
$ SELECT * FROM members WHERE get_numeric ( p r o f i l e , s i b l i n g s
) = 1;
Time : 6 5 . 1 3 6 ms
$ SELECT count ( * ) from members where get_numeric ( p r o f i l e ,
age ) = 26 and get_numeric ( p r o f i l e , s i b l i n g s ) = 1 ;
Time : 1 0 6 . 4 9 2 ms


PostgreSQL.
PLV8 JavaScript
PostgreSQL. Mustache :
9.22 Mustache
Line 1
5

CREATE OR REPLACE FUNCTION mustache ( t e m p l a t e t e x t , view j s o n


)
RETURNS t e x t a s $$
// . . . 400 l i n e s o f mustache . . . . j s
r e t u r n Mustache . r e n d e r ( template , JSON . p a r s e ( view ) )
$$ LANGUAGE p l v 8 IMMUTABLE STRICT ;

165

9.6. Pg_repack
9.23
Line 1
5
-

$ SELECT mustache (
h e l l o {{# t h i n g s } } { { . } } {{/ t h i n g s } } : ) {{#data }}{{ key }}{{/
data }} ,
{" t h i n g s " : [ " world " , " from " , " p o s t g r e s q l " ] , " data " : {" key
" : "and me"}}
);
mustache
-- ------------------------------------h e l l o world from p o s t g r e s q l : ) and me
( 1 row )

10

Time : 0 . 8 3 7 ms

PLV8. Mustache PostgreSQL .

PLV8 PostgreSQL
V8 JavaScript, JavaScript
, JSON
.

9.6 Pg_repack
: Open Source
: reorg.github.io/pg_repack/
PostgreSQL , 8 ,
.
, .
DELETE UPDATE, .
autovacuum, VACUUM,
,
. autovacuum
, - , .
,
. .
, table bloat.
, autovacuum VACUUM, ,
166

9.6. Pg_repack
. PostgreSQL VACUUM FULL CLUSTER,
exclusively locks (
, ),
.
pg_repack.
VACUUM FULL CLUSTER . pg_repack
repack ( public ) .
, .
,
.
(,
100 , 40 - ,
100 + (100 - 40 ) = 160 ).
12.2 (bloat)
.
pg_repack:
;
GIST ;
DDL (Data Definition Language)
.

CLUSTER VACUUM
FULL test :

Line 1

$ pg_repack t e s t

VACUUM FULL foo bar test


( ):

Line 1

$ pg_repack - - no - o r d e r - - t a b l e f o o - - t a b l e bar t e s t

foo tbs:

Line 1

$ pg_repack - d t e s t - - t a b l e f o o - - only - i n d e x e s - - t a b l e s p a c e
tbs

167

9.7. Pg_prewarm

Pg_repack , table
bloat PostgreSQL .

9.7 Pg_prewarm
: Open Source
pg_prewarm
(, , ) PostgreSQL
. contrib
PostgreSQL 9.4.
:

Line 1

$ CREATE EXTENSION pg_prewarm ;

pg_prewarm:

Line 1
5

$ SELECT pg_prewarm ( pgbench_accounts ) ;


pg_prewarm
-- ---------4082
( 1 row )

, . ,
:
prefetch (
). . ;
read prefetch , (
). ;
buffer PostgreSQL
shared_buffers. .
fork. .
: main ( ), fsm, vm.
. , , , 1000 :

Line 1
-

$ SELECT pg_prewarm (
pgbench_accounts ,

168

9.8. Smlar
f i r s t _ b l o c k := (
SELECT p g _ r e l a t i o n _ s i z e ( pgbench_accounts ) /
c u r r e n t _ s e t t i n g ( b l o c k _ s i z e ) : : i n t 4 - 1000
)

5
-

);

Pg_prewarm , () PostgreSQL
.

9.8 Smlar
: Open Source
: sigaev.ru
( ),
- ( ), (
, ) .. PostgreSQL . , ,
.

. , , , , .. ,
, ( , n-grams).
. ?


.
, :
, ;
;
.
-

:
169

9.8. Smlar

(, ) =

( + )

(9.1)

(9.2)


(, ) =
:
;
: * log ;
.
:
(, ) =

(9.3)

:
: * log ;
.
:
, ;
: , ;
. - .
TF/IDF :
< ,< , = *
(, ) =
2
2
< *
<

(9.4)

:
= log (

+ 1)

(9.5)

:
= *

(9.6)

! smlar ,
( ) . ,
TF/IDF , .
170

9.8. Smlar

Smlar
.
PostgreSQL smlar,
( )
GIST GIN. (PostgreSQL
):
9.24 smlar
Line 1 $ g i t c l o n e g i t : / / s i g a e v . ru / s m l a r
- $ cd s m l a r
- $ USE_PGXS=1 make && make i n s t a l l

PostgreSQL 9.2 ,
PostgreSQL 9.1
. smlar_guc.c 214 :
9.25 9.1
Line 1

s e t _ c o n f i g _ o p t i o n ( " s m l ar . t h r e s h o l d " , buf , PGC_USERSET,


PGC_S_SESSION ,GUC_ACTION_SET, t r u e , 0 ) ;

( ):
9.26 9.1
Line 1

s e t _ c o n f i g _ o p t i o n ( " s m l ar . t h r e s h o l d " , buf , PGC_USERSET,


PGC_S_SESSION ,GUC_ACTION_SET, t r u e ) ;

:
9.27 smlar
$ psql
- psql ( 9 . 2 . 1 )
- Type " h e l p " f o r h e l p .

Line 1

5
-

t e s t=# CREATE EXTENSION s m l a r ;


CREATE EXTENSION

10
-

t e s t=# SELECT s m l a r ( { 1 , 4 , 6 } : : i n t [ ] ,
sm l a r
-- -------0.666667
( 1 row )

{5 ,4 ,6} : : int [ ] ) ;

t e s t=# SELECT s m l a r ( { 1 , 4 , 6 } : : i n t [ ] ,
/ s q r t (N. a * N. b ) ) ;
sm l a r
-- -------0.666667
( 1 row )

{5 ,4 ,6} : : int [ ] ,

15
-

171

N. i

9.8. Smlar
, . , :
float4 smlar(anyarray, anyarray) . ;
float4 smlar(anyarray, anyarray, bool useIntersect ) . :
9.28
Line 1

CREATE TYPE type_name AS ( element_name anytype ,


weight_name f l o a t 4 ) ;
useIntersect

;
float4 smlar( anyarray a, anyarray b, text formula ) ,
. :
N.i ();
N.a ;
N.b ;
anyarray % anyarray , , .
PostgreSQL:
9.29 Smlar
Line 1
-

custom_variable_classes = smlar
s m l a r . t h r e s h o l d = 0 . 8 # 0 1


smlar:
9.30 Smlar
Line 1
-

custom_variable_classes = smlar
s m l a r . t h r e s h o l d = 0 . 8 # 0 1
s m l a r . type = c o s i n e #
: c o s i n e , t f i d f , o v e r l a p
s m l a r . s t a t t a b l e = s t a t #
t f i d f

README .
GiST GIN %.

172

9.8. Smlar

:
.
, , , ( , , ). , ,
. ,
?
.
. , ,
, .
:

. 9.1:

(
),
15X15 (. 9.1);
( 0.299 * + 0.587 * + 0.114 * ).
, ;
173

9.8. Smlar
(. 9.2);
( + );
.

. 9.2:
, ,
:
9.31
Line 1
5
-

CREATE TABLE images (


i d s e r i a l PRIMARY KEY,
name v a r c h a r ( 5 0 ) ,
img_path v a r c h a r ( 2 5 0 ) ,
image_array i n t e g e r [ ]
);

GIN GIST :
9.32 GIN GIST
CREATE INDEX image_array_gin ON images USING GIN( image_array
_int4_sml_ops ) ;
- CREATE INDEX image_array_gist ON images USING GIST (
image_array _int4_sml_ops ) ;

Line 1

:
9.33
Line 1
-

t e s t=# SELECT count ( * ) from images ;


count
-- ------1000000

174

9.8. Smlar
5

( 1 row )

t e s t=# EXPLAIN ANALYZE SELECT count ( * ) FROM images WHERE


images . image_array %
{1010259 ,1011253 ,... ,2423253 ,2424252} : : int [ ] ;

10
-

Bitmap Heap Scan on images ( c o s t = 2 8 6 . 6 4 . . 3 9 6 9 . 4 5 rows =986


width =4) ( a c t u a l time = 5 0 4 . 3 1 2 . . 2 0 4 7 . 5 3 3 rows =200000 l o o p s
=1)
Recheck Cond : ( image_array %
{1010259 ,1011253 ,... ,2423253 ,2424252} : : integer [ ] )
-> Bitmap Index Scan on image_array_gist ( c o s t
= 0 . 0 0 . . 2 8 6 . 3 9 rows =986 width =0) ( a c t u a l time
= 4 4 6 . 1 0 9 . . 4 4 6 . 1 0 9 rows =200000 l o o p s =1)
Index Cond : ( image_array %
{1010259 ,1011253 ,... ,2423253 ,2424252} : : integer [ ] )
T o t a l runtime : 2 1 5 2 . 4 1 1 ms
( 5 rows )

{1010259,...,2424252} :: int [] ,
. smlar.threshold % (
).
:
9.34
Line 1

t e s t=# EXPLAIN ANALYZE SELECT s m l a r ( images . image_array ,


{ 1 0 1 0 2 5 9 , . . . , 2 4 2 4 2 5 2 } : : i n t [ ] ) a s s i m i l a r i t y FROM images
WHERE images . image_array % { 1 0 1 0 2 5 9 , 1 0 1 1 2 5 3 ,
. . . , 2 4 2 3 2 5 3 , 2 4 2 4 2 5 2 } : : i n t [ ] ORDER BY s i m i l a r i t y DESC;

5
-

10

S o r t ( c o s t = 4 0 2 0 . 9 4 . . 4 0 2 3 . 4 1 rows =986 width =924) ( a c t u a l


time = 2 8 8 8 . 4 7 2 . . 2 9 0 1 . 9 7 7 rows =200000 l o o p s =1)
S o r t Key : ( s m l a r ( image_array , { . . . , 2 4 2 4 2 5 2 } : : i n t e g e r [ ] )
)
S o r t Method : q u i c k s o r t Memory : 15520kB
-> Bitmap Heap Scan on images ( c o s t = 2 8 6 . 6 4 . . 3 9 7 1 . 9 1
rows =986 width =924) ( a c t u a l time = 4 7 4 . 4 3 6 . . 2 7 2 9 . 6 3 8 rows
=200000 l o o p s =1)
Recheck Cond : ( image_array % { . . . , 2 4 2 4 2 5 2 } : :
integer [ ] )
-> Bitmap Index Scan on image_array_gist ( c o s t
= 0 . 0 0 . . 2 8 6 . 3 9 rows =986 width =0) ( a c t u a l time
= 4 2 1 . 1 4 0 . . 4 2 1 . 1 4 0 rows =200000 l o o p s =1)
Index Cond : ( image_array % { . . . , 2 4 2 4 2 5 2 } : :
integer [ ] )

175

9.9. Multicorn
-

T o t a l runtime : 2 9 1 2 . 2 0 7 ms
( 8 rows )

1 (P.S.
PostgreSQL ,
,
).

Smlar ,
, : , , , , , , .

9.9 Multicorn
: Open Source
: multicorn.org
Multicorn PostgreSQL 9.1 ,
FDW (Foreign Data Wrapper)
Python. Foreign Data Wrapper ( , ,
REST API, ) PostgreSQL 9.1.

Ubuntu Linux. :
9.35 Multicorn
Line 1

$ sudo a p t i t u d e i n s t a l l b u i l d - e s s e n t i a l p o s t g r e s q l - s e r v e r dev - 9 . 3 python - dev python - s e t u p t o o l s

:
9.36 Multicorn
$ g i t c l o n e g i t @ g i t h u b . com : Kozea / M u l t i c o r n . g i t
$ cd M u l t i c o r n
- $ make && sudo make i n s t a l l

Line 1
-

:
9.37 Multicorn
Line 1
-

# CREATE EXTENSION m u l t i c o r n ;
CREATE EXTENSION

FDW Multicorn.
176

9.9. Multicorn

Multicorn SQLAlchemy . SQLite,
PostgreSQL, MySQL, Oracle, MS-SQL, Firebird, Sybase, . MySQL. :
9.38 Multicorn
Line 1 $ sudo a p t i t u d e i n s t a l l python - s q l a l c h e m y python - mysqldb

MySQL testing companies:


9.39 Multicorn
Line 1 $ mysql - u r o o t - p t e s t i n g
5
10
-

mysql> SELECT * FROM companies ;


+ - - - -+ - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - -+
| id | created_at
| updated_at
|
+ - - - -+ - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - -+
| 1 | 2013 -07 -16 1 4 : 0 6 : 0 9 | 2013 -07 -16 1 4 : 0 6 : 0 9 |
| 2 | 2013 -07 -16 1 4 : 3 0 : 0 0 | 2013 -07 -16 1 4 : 3 0 : 0 0 |
| 3 | 2013 -07 -16 1 4 : 3 3 : 4 1 | 2013 -07 -16 1 4 : 3 3 : 4 1 |
| 4 | 2013 -07 -16 1 4 : 3 8 : 4 2 | 2013 -07 -16 1 4 : 3 8 : 4 2 |
| 5 | 2013 -07 -19 1 4 : 3 8 : 2 9 | 2013 -07 -19 1 4 : 3 8 : 2 9 |
+ - - - -+ - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - -+
5 rows i n s e t ( 0 . 0 0 s e c )

PostgreSQL Multicorn:
Line 1
-

9.40 Multicorn
# CREATE SERVER alchemy_srv f o r e i g n data wrapper m u l t i c o r n
options (
wrapper m u l t i c o r n . s q l a l c h e m y f d w . SqlAlchemyFdw
);
CREATE SERVER

,
MySQL companies:
Line 1
5
-

9.41 Multicorn
# CREATE FOREIGN TABLE mysql_companies (
id integer ,
c r e a t e d _ a t timestamp w i t h o u t time zone ,
updated_at timestamp w i t h o u t time zone
) s e r v e r alchemy_srv o p t i o n s (
tablename companies ,
db_url mysql : / / r o o t : password@127 . 0 . 0 . 1 / t e s t i n g
);
CREATE FOREIGN TABLE

177

9.9. Multicorn
:
db_url (string) SQLAlchemy (: mysql://<user>:<password>@<host>/<dbname>, mssql:
mssql://<user>:<password>@<dsname>).
SQLAlchemy ;
tablename (string) .
, :

Line 1
5
-

9.42 Multicorn
# SELECT * FROM mysql_companies ;
id |
created_at
|
updated_at
- - - -+ - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - 1 | 2013 -07 -16 1 4 : 0 6 : 0 9 | 2013 -07 -16 1 4 : 0 6 : 0 9
2 | 2013 -07 -16 1 4 : 3 0 : 0 0 | 2013 -07 -16 1 4 : 3 0 : 0 0
3 | 2013 -07 -16 1 4 : 3 3 : 4 1 | 2013 -07 -16 1 4 : 3 3 : 4 1
4 | 2013 -07 -16 1 4 : 3 8 : 4 2 | 2013 -07 -16 1 4 : 3 8 : 4 2
5 | 2013 -07 -19 1 4 : 3 8 : 2 9 | 2013 -07 -19 1 4 : 3 8 : 2 9
( 5 rows )

IMAP
Multicorn IMAP . :
9.43 Multicorn
Line 1 $ sudo a p t i t u d e i n s t a l l python - p i p
- $ sudo p i p i n s t a l l i m a p c l i e n t

, IMAP :

Line 1

5
10
-

9.44 Multicorn
# CREATE SERVER multicorn_imap FOREIGN DATA WRAPPER
m u l t i c o r n o p t i o n s ( wrapper m u l t i c o r n . imapfdw . ImapFdw )
;
CREATE SERVER
# CREATE FOREIGN TABLE my_inbox (
" Message - ID" c h a r a c t e r v a r y i n g ,
"From" c h a r a c t e r v a r y i n g ,
" Subject " c h a r a c t e r varying ,
" payload " c h a r a c t e r v a r y i n g ,
" f l a g s " character varying [ ] ,
"To" c h a r a c t e r v a r y i n g ) s e r v e r multicorn_imap o p t i o n s (
h o s t imap . g m a i l . com ,
p o r t 993 ,
payload_column payload ,

178

9.9. Multicorn
flags_column f l a g s ,
s s l True ,
l o g i n example@gmail . com ,
password s u p e r s e c r e t p a s s w o r d

15
-

);
CREATE FOREIGN TABLE

host ( string ) IMAP ;


port ( string ) IMAP ;
login ( string ) IMAP ;
password (string) IMAP ;
payload_column (string) ,

;
flags_column (string) , IMAP ();
ssl (boolean) SSL ;
imap_server_charset (string) IMAP . UTF8.
my_inbox:
9.45 Multicorn
# SELECT f l a g s , " S u b j e c t " , payload FROM my_inbox LIMIT 1 0 ;
flags
|
Subject
|
payload
- -- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - -

Line 1
-

5
-

{ $ M a i l F l a g B i t 1 , " \\ Flagged " , " \\ Seen "} | Test e m a i l


Test e m a i l \ r
+
|
{" \\ Seen " }
| Test s e c o n d e m a i l
Test s e c o n d e m a i l \ r+
|
( 2 rows )

|
|
|
|

RSS
Multicorn RSS .
:
9.46 Multicorn
Line 1

$ sudo a p t i t u d e i n s t a l l python - lxml

, RSS :

179

9.9. Multicorn
9.47 Multicorn
Line 1
5
10
-

# CREATE SERVER r s s _ s r v f o r e i g n data wrapper m u l t i c o r n


options (
wrapper m u l t i c o r n . r s s f d w . RssFdw
);
CREATE SERVER
# CREATE FOREIGN TABLE my_rss (
" pubDate " timestamp ,
d e s c r i p t i o n c h a r a c t e r varying ,
t i t l e c h a r a c t e r varying ,
l i n k character varying
) server rss_srv options (
url
h t t p : / / news . yahoo . com/ r s s / e n t e r t a i n m e n t
);
CREATE FOREIGN TABLE

:
url ( string ) URL RSS .
, , PostgreSQL
UTF-8 (
). my_rss:
9.48 Multicorn
# SELECT " pubDate " , t i t l e , l i n k from my_rss ORDER BY "
pubDate " DESC LIMIT 1 0 ;
pubDate
|
title
|
link
- -- - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Line 1

2 0 1 3 - 0 9 - 2 8 1 4 : 1 1 : 5 8 | Royal Mint c o i n s t o mark P r i n c e


George c h r i s t e n i n g | h t t p : / / news . yahoo . com/ r o y a l - mint c o i n s - mark - p r i n c e - g e o r g e - c h r i s t e n i n g - 1 1 5 9 0 6 2 4 2 . html
2 0 1 3 - 0 9 - 2 8 1 1 : 4 7 : 0 3 | Miss P h i l i p p i n e s wins Miss World i n
Indonesia
| h t t p : / / news . yahoo . com/ miss - p h i l i p p i n e s wins - miss - world - i n d o n e s i a - 1 4 4 5 4 4 3 8 1 . html
2 0 1 3 - 0 9 - 2 8 1 0 : 5 9 : 1 5 | B i l l i o n a i r e s d a u g h t e r i n NJ c o u r t i n
w i l l d i s p u t e | h t t p : / / news . yahoo . com/ b i l l i o n a i r e s daughter - nj - c o u r t - d i s p u t e - 1 4 4 4 3 2 3 3 1 . html
2 0 1 3 - 0 9 - 2 8 0 8 : 4 0 : 4 2 | S e c u r i t y t i g h t a t Miss World f i n a l i n
Indonesia
| h t t p : / / news . yahoo . com/ s e c u r i t y - t i g h t - miss
- world - f i n a l - i n d o n e s i a - 1 2 3 7 1 4 0 4 1 . html
2 0 1 3 - 0 9 - 2 8 0 8 : 1 7 : 5 2 | Guest l i n e u p s f o r t h e Sunday news
shows
| h t t p : / / news . yahoo . com/ g u e s t - l i n e u p s sunday - news - shows - 1 8 3 8 1 5 6 4 3 . html

180

9.9. Multicorn
-

10

2 0 1 3 - 0 9 - 2 8 0 7 : 3 7 : 0 2 | S e c u r i t y t i g h t a t Miss World crowning


i n I n d o n e s i a | h t t p : / / news . yahoo . com/ s e c u r i t y - t i g h t - miss
- world - crowning - i n d o n e s i a - 1 1 3 6 3 4 3 1 0 . html
2 0 1 3 - 0 9 - 2 7 2 0 : 4 9 : 3 2 | Simons stamps h i s n a t u r a l mark on
Dior
| h t t p : / / news . yahoo . com/ simons - stamps n a t u r a l - mark - d i o r - 2 2 3 8 4 8 5 2 8 . html
2 0 1 3 - 0 9 - 2 7 1 9 : 5 0 : 3 0 | Jackson j u r y ends d e l i b e r a t i o n s u n t i l
Tuesday
| h t t p : / / news . yahoo . com/ j a c k s o n - j u r y - ends d e l i b e r a t i o n s - u n t i l - tuesday - 2 3 5 0 3 0 9 6 9 . html
2 0 1 3 - 0 9 - 2 7 1 9 : 2 3 : 4 0 | E r i c Clapton - owned R i c h t e r p a i n t i n g
t o s e l l i n NYC | h t t p : / / news . yahoo . com/ e r i c - c l a p t o n - owned
- r i c h t e r - p a i n t i n g - s e l l - nyc - 2 0 1 4 4 7 2 5 2 . html
2 0 1 3 - 0 9 - 2 7 1 9 : 1 4 : 1 5 | Report : Hollywood i s l e s s gay f r i e n d l y o f f - s c r e e n | h t t p : / / news . yahoo . com/ r e p o r t hollywood - l e s s - gay - f r i e n d l y - o f f - s c r e e n - 2 3 1 4 1 5 2 3 5 . html
( 1 0 rows )

CSV
Multicorn CSV .
, CSV :
9.49 Multicorn
Line 1
5
10
15
-

# CREATE SERVER csv_srv f o r e i g n data wrapper m u l t i c o r n


options (
wrapper m u l t i c o r n . csvfdw . CsvFdw
);
CREATE SERVER
# CREATE FOREIGN TABLE c s v t e s t (
s o r t _ o r d e r numeric ,
common_name c h a r a c t e r v a r y i n g ,
formal_name c h a r a c t e r v a r y i n g ,
main_type c h a r a c t e r v a r y i n g ,
sub_type c h a r a c t e r v a r y i n g ,
s o v e r e i g n t y c h a r a c t e r varying ,
c a p i t a l character varying
) s e r v e r csv_srv o p t i o n s (
f i l e n a m e / var / data / c o u n t r y l i s t . c s v ,
skip_header 1 ,
delimiter , ) ;
CREATE FOREIGN TABLE

:
filename ( string ) CSV ;
delimiter (character) CSV ( ,);
quotechar (character) CSV ;
181

9.9. Multicorn
skip_header (integer) ,
( 0).
csvtest:
9.50 Multicorn

# SELECT * FROM c s v t e s t LIMIT 1 0 ;


sort_order |
common_name
|
formal_name
|
main_type
| sub_type |
sovereignty |
capital
- -- - - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Line 1
-

10

1 | Afghanistan
| Islamic State of
Afghanistan
| Independent S t a t e |
|
| Kabul
2 | Albania
| R e p u b l i c o f Albania
| Independent S t a t e |
|
| Tirana
3 | Algeria
| People s Democratic
R e p u b l i c o f A l g e r i a | Independent S t a t e |
|
| Algiers
4 | Andorra
| P r i n c i p a l i t y o f Andorra
| Independent S t a t e |
|
| Andorra l a V e l l a
5 | Angola
| R e p u b l i c o f Angola
| Independent S t a t e |
|
| Luanda
6 | Antigua and Barbuda |
| Independent S t a t e |
|
| S a i n t John s
7 | Argentina
| Argentine Republic
| Independent S t a t e |
|
| Buenos A i r e s
8 | Armenia
| R e p u b l i c o f Armenia
| Independent S t a t e |
|
| Yerevan
9 | Australia
| Commonwealth o f A u s t r a l i a
| Independent S t a t e |
|
| Canberra
10 | A u s t r i a
| Republic of Austria
| Independent S t a t e |
|
| Vienna
( 1 0 rows )

182

9.10. Pgaudit
FDW
Multicorn FDW LDAP .
LDAP FDW LDAP . FDW
, .
FDW
Multicorn FDW. .

PostgreSQL 9.3+
PostgreSQL 9.1 9.2 FDW
, 9.3 FDW
. Multicorn 1.0.0.

Multicorn PostgreSQL, FDW Python.

9.10 Pgaudit
: Open Source
: github.com/2ndQuadrant/pgaudit
Pgaudit PostgreSQL,
PostgreSQL
CSV c , , , ( ) . DDL, DML (
SELECT) . PostgreSQL
9.3 .
PostgreSQL :
9.51 Pgaudit
Line 1

shared_preload_libraries = pgaudit

p g a u d i t . l o g = read , w r i t e , u s e r

183

9.11. Ltree
9.52 Pgaudit
Line 1

# CREATE EXTENSION p g a u d i t ;

pgaudit:
9.53 Pgaudit
Line 1

5
-

LOG:
[ AUDIT] , 2 0 1 4 - 0 4 - 3 0 1 7 : 1 3 : 5 5 . 2 0 2 8 5 4 + 0 9 , auditdb , ianb ,
ianb , DEFINITION ,CREATE TABLE,TABLE, p u b l i c . x ,CREATE TABLE
p u b l i c . x ( a pg_catalog . i n t 4
, b pg_catalog . i n t 4
)
WITH ( o i d s=OFF)
LOG:
[ AUDIT] , 2 0 1 4 - 0 4 - 3 0 1 7 : 1 4 : 0 6 . 5 4 8 9 2 3 + 0 9 , auditdb , ianb ,
ianb ,WRITE, INSERT ,TABLE, p u b l i c . x , INSERT INTO x VALUES
(1 ,1) ;
LOG:
[ AUDIT] , 2 0 1 4 - 0 4 - 3 0 1 7 : 1 4 : 2 1 . 2 2 1 8 7 9 + 0 9 , auditdb , ianb ,
ianb ,READ, SELECT,TABLE, p u b l i c . x , SELECT * FROM x ;
LOG:
[ AUDIT] , 2 0 1 4 - 0 4 - 3 0 1 7 : 1 5 : 2 5 . 6 2 0 2 1 3 + 0 9 , auditdb , ianb ,
ianb ,READ, SELECT,VIEW, p u b l i c . v_x , SELECT * from v_x ;
LOG:
[ AUDIT] , 2 0 1 4 - 0 4 - 3 0 1 7 : 1 5 : 2 5 . 6 2 0 2 6 2 + 0 9 , auditdb , ianb ,
ianb ,READ, SELECT,TABLE, p u b l i c . x , SELECT * from v_x ;
LOG:
[ AUDIT] , 2 0 1 4 - 0 4 - 3 0 1 7 : 1 6 : 0 0 . 8 4 9 8 6 8 + 0 9 , auditdb , ianb ,
ianb ,WRITE,UPDATE,TABLE, p u b l i c . x ,UPDATE x SET a=a +1;
LOG:
[ AUDIT] , 2 0 1 4 - 0 4 - 3 0 1 7 : 1 6 : 1 8 . 2 9 1 4 5 2 + 0 9 , auditdb , ianb ,
ianb ,ADMIN,VACUUM, , ,VACUUM x ;
LOG:
[ AUDIT] , 2 0 1 4 - 0 4 - 3 0 1 7 : 1 8 : 0 1 . 0 8 2 9 1 + 0 9 , auditdb , ianb , ianb
, DEFINITION ,CREATE FUNCTION,FUNCTION, p u b l i c . func_x ( ) ,
CREATE FUNCTION p u b l i c . func_x ( ) RETURNS pg_catalog . i n t 4
LANGUAGE s q l VOLATILE CALLED ON NULL INPUT SECURITY
INVOKER COST 1 0 0 . 0 0 0 0 0 0
AS $dprs_$SELECT a FROM x LIMIT
1 ; $dprs_$

README.

9.11 Ltree
: Open Source
Ltree , ,
.

Ltree?
Materialized Path (
, );
, CTE
(Common Table Expressions) (
);
184

9.11. Ltree
;
(!!!).

:
9.54 Ltree
Line 1

# CREATE EXTENSION l t r e e ;

, .
9.55 Ltree
Line 1
5
10
15
-

CREATE TABLE comments ( u s e r _ i d i n t e g e r , d e s c r i p t i o n t e x t ,


path l t r e e ) ;
INSERT INTO comments ( user_id , d e s c r i p t i o n , path ) VALUES (
1 , md5( random ( ) : : t e x t ) , 0001 ) ;
INSERT INTO comments ( user_id , d e s c r i p t i o n , path ) VALUES (
2 , md5( random ( ) : : t e x t ) , 0 0 0 1 . 0 0 0 1 . 0 0 0 1 ) ;
INSERT INTO comments ( user_id , d e s c r i p t i o n , path ) VALUES (
2 , md5( random ( ) : : t e x t ) , 0 0 0 1 . 0 0 0 1 . 0 0 0 1 . 0 0 0 1 ) ;
INSERT INTO comments ( user_id , d e s c r i p t i o n , path ) VALUES (
1 , md5( random ( ) : : t e x t ) , 0 0 0 1 . 0 0 0 1 . 0 0 0 1 . 0 0 0 2 ) ;
INSERT INTO comments ( user_id , d e s c r i p t i o n , path ) VALUES (
5 , md5( random ( ) : : t e x t ) , 0 0 0 1 . 0 0 0 1 . 0 0 0 1 . 0 0 0 3 ) ;
INSERT INTO comments ( user_id , d e s c r i p t i o n , path ) VALUES (
6 , md5( random ( ) : : t e x t ) , 0 0 0 1 . 0 0 0 2 ) ;
INSERT INTO comments ( user_id , d e s c r i p t i o n , path ) VALUES (
6 , md5( random ( ) : : t e x t ) , 0 0 0 1 . 0 0 0 2 . 0 0 0 1 ) ;
INSERT INTO comments ( user_id , d e s c r i p t i o n , path ) VALUES (
6 , md5( random ( ) : : t e x t ) , 0 0 0 1 . 0 0 0 3 ) ;
INSERT INTO comments ( user_id , d e s c r i p t i o n , path ) VALUES (
8 , md5( random ( ) : : t e x t ) , 0 0 0 1 . 0 0 0 3 . 0 0 0 1 ) ;
INSERT INTO comments ( user_id , d e s c r i p t i o n , path ) VALUES (
9 , md5( random ( ) : : t e x t ) , 0 0 0 1 . 0 0 0 3 . 0 0 0 2 ) ;
INSERT INTO comments ( user_id , d e s c r i p t i o n , path ) VALUES (
1 1 , md5( random ( ) : : t e x t ) , 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 1 ) ;
INSERT INTO comments ( user_id , d e s c r i p t i o n , path ) VALUES (
2 , md5( random ( ) : : t e x t ) , 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 ) ;
INSERT INTO comments ( user_id , d e s c r i p t i o n , path ) VALUES (
5 , md5( random ( ) : : t e x t ) , 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 3 ) ;
INSERT INTO comments ( user_id , d e s c r i p t i o n , path ) VALUES (
7 , md5( random ( ) : : t e x t ) , 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 1 ) ;
INSERT INTO comments ( user_id , d e s c r i p t i o n , path ) VALUES (
2 0 , md5( random ( ) : : t e x t ) , 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 2 ) ;
INSERT INTO comments ( user_id , d e s c r i p t i o n , path ) VALUES (
3 1 , md5( random ( ) : : t e x t ) , 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 3 ) ;

185

9.11. Ltree
INSERT
22 ,
- INSERT
34 ,
20 INSERT
22 ,
-

INTO comments ( user_id , d e s c r i p t i o n , path ) VALUES (


md5( random ( ) : : t e x t ) , 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 4 ) ;
INTO comments ( user_id , d e s c r i p t i o n , path ) VALUES (
md5( random ( ) : : t e x t ) , 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 5 ) ;
INTO comments ( user_id , d e s c r i p t i o n , path ) VALUES (
md5( random ( ) : : t e x t ) , 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 6 ) ;

:
9.56 Ltree
# CREATE INDEX path_gist_comments_idx ON comments USING GIST
( path ) ;
- # CREATE INDEX path_comments_idx ON comments USING b t r e e (
path ) ;

Line 1

comments path,
(
4 ).
,
0001.0003:
9.57 Ltree
Line 1
5
10
15
-

# SELECT user_id , path FROM comments WHERE path <@


0001.0003 ;
use r _ i d |
path
- - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - - - - - - 6 | 0001.0003
8 | 0001.0003.0001
9 | 0001.0003.0002
11 | 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 1
2 | 0001.0003.0002.0002
5 | 0001.0003.0002.0003
7 | 0001.0003.0002.0002.0001
20 | 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 2
31 | 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 3
22 | 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 4
34 | 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 5
22 | 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 6
( 1 2 rows )

:
9.58 Ltree
# SET e n a b l e _ s e q s c a n=f a l s e ;
- SET
- # EXPLAIN ANALYZE SELECT user_id , path FROM comments WHERE
path <@ 0 0 0 1 . 0 0 0 3 ;

Line 1

QUERY PLAN

186

9.11. Ltree
5

-------------------------------------------------------------------------

Index Scan u s i n g path_gist_comments_idx on comments ( c o s t


= 0 . 0 0 . . 8 . 2 9 rows=2 width =38) ( a c t u a l time = 0 . 0 2 3 . . 0 . 0 3 4
rows=12 l o o p s =1)
Index Cond : ( path <@ 0 0 0 1 . 0 0 0 3 : : l t r e e )
T o t a l runtime : 0 . 0 7 6 ms
( 3 rows )

Line 1
5
10
15
-

9.59 Ltree
# SELECT user_id , path FROM comments WHERE path ~
0001.0003.* ;
user _ i d |
path
- - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - - - - - - 6 | 0001.0003
8 | 0001.0003.0001
9 | 0001.0003.0002
11 | 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 1
2 | 0001.0003.0002.0002
5 | 0001.0003.0002.0003
7 | 0001.0003.0002.0002.0001
20 | 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 2
31 | 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 3
22 | 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 4
34 | 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 5
22 | 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 6
( 1 2 rows )

Line 1
5
10
-

9.60 Ltree
# INSERT INTO comments ( user_id , d e s c r i p t i o n , path ) VALUES (
9 , md5( random ( ) : : t e x t ) , 0 0 0 1 . 0 0 0 3 . 0 0 0 1 . 0 0 0 1 ) ;
# INSERT INTO comments ( user_id , d e s c r i p t i o n , path ) VALUES (
9 , md5( random ( ) : : t e x t ) , 0 0 0 1 . 0 0 0 3 . 0 0 0 1 . 0 0 0 2 ) ;
# INSERT INTO comments ( user_id , d e s c r i p t i o n , path ) VALUES (
9 , md5( random ( ) : : t e x t ) , 0 0 0 1 . 0 0 0 3 . 0 0 0 1 . 0 0 0 3 ) ;
# SELECT user_id , path FROM comments WHERE path ~
0001.0003.* ;
user _ i d |
path
- - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - - - - - - 6 | 0001.0003
8 | 0001.0003.0001
9 | 0001.0003.0002
11 | 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 1
2 | 0001.0003.0002.0002

187

9.11. Ltree
15
20
25
30
35
40
-

5 | 0001.0003.0002.0003
7 | 0001.0003.0002.0002.0001
20 | 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 2
31 | 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 3
22 | 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 4
34 | 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 5
22 | 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 6
9 | 0001.0003.0001.0001
9 | 0001.0003.0001.0002
9 | 0001.0003.0001.0003
( 1 5 rows )
# SELECT user_id , path FROM comments WHERE path ~
0 0 0 1 . 0 0 0 3 . * ORDER by path ;
use r _ i d |
path
- - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - - - - - - 6 | 0001.0003
8 | 0001.0003.0001
9 | 0001.0003.0001.0001
9 | 0001.0003.0001.0002
9 | 0001.0003.0001.0003
9 | 0001.0003.0002
11 | 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 1
2 | 0001.0003.0002.0002
7 | 0001.0003.0002.0002.0001
20 | 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 2
31 | 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 3
22 | 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 4
34 | 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 5
22 | 0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 6
5 | 0001.0003.0002.0003
( 1 5 rows )

. ( | ):
9.61 Ltree
Line 1
5
10
-

# SELECT user_id , path FROM comments WHERE path ~


0 0 0 1 . * { 1 , 2 } . 0 0 0 1 | 0 0 0 2 . * ORDER by path ;
use r _ i d |
path
- - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - - - - - - 2 | 0001.0001.0001
2 | 0001.0001.0001.0001
1 | 0001.0001.0001.0002
5 | 0001.0001.0001.0003
6 | 0001.0002.0001
8 | 0001.0003.0001
9 | 0001.0003.0001.0001
9 | 0001.0003.0001.0002

188

9.12. PostPic
15
20
-

9 |
9 |
11 |
2 |
7 |
20 |
31 |
22 |
34 |
22 |
5 |
( 1 9 rows )

0001.0003.0001.0003
0001.0003.0002
0001.0003.0002.0001
0001.0003.0002.0002
0001.0003.0002.0002.0001
0001.0003.0002.0002.0002
0001.0003.0002.0002.0003
0001.0003.0002.0002.0004
0001.0003.0002.0002.0005
0001.0003.0002.0002.0006
0001.0003.0002.0003

, 0001.0003:

Line 1
5
-

9.62 Ltree
# SELECT user_id , path FROM comments WHERE path ~
0 0 0 1 . 0 0 0 3 . * { 1 } ORDER by path ;
use r _ i d |
path
- - - - - - - - -+ - - - - - - - - - - - - - - - 8 | 0001.0003.0001
9 | 0001.0003.0002
( 2 rows )

0001.0003.0002.0002.0005:

Line 1
5

9.63 Ltree
# SELECT user_id , path FROM comments WHERE path = subpath (
0 0 0 1 . 0 0 0 3 . 0 0 0 2 . 0 0 0 2 . 0 0 0 5 , 0 , - 1 ) ORDER by path ;
use r _ i d |
path
- - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - 2 | 0001.0003.0002.0002
( 1 row )

Ltree ,
Materialized Path PostgreSQL.

9.12 PostPic
: Open Source
: github.com/drotiro/postpic
PostPic PostgreSQL, , PostGIS . image,
( ,
189

9.13. Fuzzystrmatch
, ..) (, , ).

9.13 Fuzzystrmatch
: Open Source
Fuzzystrmatch
. soundex
. difference soundex ,
. soundex
, : 0 , 4 ( ,
similarity):
9.64 soundex
Line 1
5
-

# CREATE EXTENSION f u z z y s t r m a t c h ;
CREATE EXTENSION
# SELECT soundex ( h e l l o world ! ) ;
soundex
-- ------H464
( 1 row )

10
-

# SELECT soundex ( Anne ) , soundex ( Ann ) , d i f f e r e n c e ( Anne ,


Ann ) ;
soundex | soundex | d i f f e r e n c e
- - - - - - - - -+ - - - - - - - - -+ - - - - - - - - - - - A500
| A500
|
4
( 1 row )

15
-

# SELECT soundex ( Anne ) , soundex ( Andrew ) , d i f f e r e n c e (


Anne , Andrew ) ;
soundex | soundex | d i f f e r e n c e
- - - - - - - - -+ - - - - - - - - -+ - - - - - - - - - - - A500
| A536
|
2
( 1 row )

20
25

# SELECT soundex ( Anne ) , soundex ( Margaret ) , d i f f e r e n c e (


Anne , Margaret ) ;
soundex | soundex | d i f f e r e n c e
- - - - - - - - -+ - - - - - - - - -+ - - - - - - - - - - - A500
| M626
|
0
( 1 row )

# CREATE TABLE s (nm t e x t ) ;

190

9.13. Fuzzystrmatch
30
35
-

CREATE TABLE
# INSERT INTO s VALUES ( john ) , ( j o a n ) , ( wobbly ) , (
jack ) ;
INSERT 0 4
# SELECT * FROM s WHERE soundex (nm) = soundex ( john ) ;
nm
-- ---john
joan
( 2 rows )

40
-

# SELECT * FROM s WHERE d i f f e r e n c e ( s . nm, john ) > 2 ;


nm
-- ---john
joan
jack
( 3 rows )

levenshtein
. levenshtein_less_equal levenshtein :
9.65 levenshtein
Line 1
5

# SELECT l e v e n s h t e i n ( GUMBO , GAMBOL ) ;


levenshtein
-- ----------2
( 1 row )

# SELECT l e v e n s h t e i n ( GUMBO , GAMBOL , 2 , 1 , 1 ) ;


levenshtein
- -- ----------10
3
- ( 1 row )
-

15
-

# SELECT l e v e n s h t e i n _ l e s s _ e q u a l ( e x t e n s i v e , e x h a u s t i v e ,
2) ;
levenshtein_less_equal
-- ---------------------3
( 1 row )

20
-

t e s t=# SELECT l e v e n s h t e i n _ l e s s _ e q u a l ( e x t e n s i v e ,
exhaustive , 4) ;
levenshtein_less_equal
-- ---------------------4

191

9.14. Tsearch2
-

( 1 row )

metaphone, soundex,
: , ,
. metaphone . dmetaphone
:
9.66 metaphone
Line 1
5
10

# SELECT metaphone ( GUMBO , 4 ) ;


metaphone
-- --------KM
( 1 row )
# SELECT dmetaphone ( p o s t g r e s q l ) ;
dmetaphone
-- ---------PSTK
( 1 row )

# SELECT dmetaphone_alt ( p o s t g r e s q l ) ;
dmetaphone_alt
- -- -------------15
PSTK
- ( 1 row )
-

9.14 Tsearch2
: Open Source
Tsearch2 .
PostgreSQL 8.3.

9.15 OpenFTS
: Open Source
: openfts.sourceforge.net
OpenFTS (Open Source Full Text Search engine)
PostgreSQL , . , .

192

9.16. PL/Proxy

9.16 PL/Proxy
: Open Source
: pgfoundry.org/projects/plproxy
PL/Proxy - . 5.2 .

9.17 Texcaller
: Open Source
: www.profv.de/texcaller
Texcaller TeX, . C,
, , TeX.
TeX NULL,
. , , NOTICEs.

9.18 Pgmemcache
: Open Source
: pgfoundry.org/projects/pgmemcache
Pgmemcache PostgreSQL API libmemcached
memcached.
PostgreSQL , ,
memcached. 8.2 Pgmemcache .

9.19 Prefix
: Open Source
: pgfoundry.org/projects/prefix
Prefix ( prefix @> text). Prefix , / .

9.20 Dblink
: Open Source

193

9.21.
Dblink , SQL, .

9.21
PostgreSQL . PostgreSQL , , ,
.

194

10
PostgreSQL

,
, ,


-

, .

10.1
.
, ,
, - .
PostgreSQL . !
, ,
, PostgreSQL .
, , (DELETE, DROP),
( ).
PostgreSQL:
SQL ;
;
;
.

195

10.2. SQL

10.2 SQL
SQL.

, . PostgreSQL
pg_dump.
pg_dump:
10.1 pg_dump
Line 1

$ pg_dump dbname > o u t f i l e

:
10.2
Line 1

$ p s q l dbname < i n f i l e

dbname . ,
, ( , ). ,
, :
10.3
Line 1

$ p s q l - - s e t ON_ERROR_STOP=on dbname < i n f i l e

, :
10.4
Line 1

$ pg_dump - h h o s t 1 dbname | p s q l - h h o s t 2 dbname

ANALYZE,
.
, , ,
? PostgreSQL pg_dumpall. pg_dumpall
PostgreSQL:
10.5 PostgreSQL
Line 1

$ pg_dumpall > o u t f i l e

:
10.6 PostgreSQL
Line 1

$ psql - f i n f i l e postgres

196

10.2. SQL

SQL
,
pg_dump. , pg_dump . Unix,
. :
.
, GZIP:
10.7 PostgreSQL
Line 1

$ pg_dump dbname | g z i p > f i l e n a m e . gz

:
10.8 PostgreSQL
Line 1

$ g u n z i p - c f i l e n a m e . gz | p s q l dbname

10.9 PostgreSQL
Line 1

c a t f i l e n a m e . gz | g u n z i p | p s q l dbname

split.
split , . , 1 :
10.10 PostgreSQL
Line 1

$ pg_dump dbname | s p l i t - b 1m - f i l e n a m e

:
10.11 PostgreSQL
Line 1

$ c a t f i l e n a m e * | p s q l dbname

pg_dump
PostgreSQL Zlib,
.
GZIP, :
10.12 PostgreSQL
Line 1

$ pg_dump - Fc dbname > f i l e n a m e

psql ,
pg_restore:
197

10.3.
10.13 PostgreSQL
Line 1

$ p g _ r e s t o r e - d dbname f i l e n a m e

, split
.

10.3
, PostgreSQL
. :
10.14 PostgreSQL
Line 1

$ t a r - c f backup . t a r / u s r / l o c a l / p g s q l / data

, , , , SQL :
PostgreSQL , ,
(PostgreSQL , ). ,
PostgreSQL;

;
, (snapshot)
( PostgreSQL). PostgreSQL
. , , , ,
. PostgreSQL , ,
WAL. ,
( WAL ). ,
PostgreSQL ,
(!!!). ,
.
rsync.
rsync PostgreSQL (PostgreSQL
). PostgreSQL
rsync. rsync , ,
198

10.4.
, .
.

10.4
PostgreSQL (Write Ahead
Log, WAL) pg_xlog , . .
PostgreSQL: ,
. , :
WAL . ,
, WAL .
, , :
.

( , );
;
WAL
, ,

PostgreSQL ( ).
, . ,
WAL .

.
WAL
pg_xlog. postgresql.conf:
10.15
Line 1

archive_mode = on # e n a b l e a r c h i v i n g

199

10.5.
-

archive_command = cp - v %p / data / p g s q l / a r c h i v e s/% f


a r c h i v e _ t i m e o u t = 300 # t i m e o u t t o c l o s e b u f f e r s

( )
. rsync.
cron , ,
:
10.16 WAL
$ r s y n c - avz - - d e l e t e prod1 : / data / p g s q l / a r c h i v e s / \
- / data / p g s q l / a r c h i v e s / > / dev / n u l l

Line 1

, pg_xlog
PostgreSQL ( ). PostgreSQL recovery.conf
:
10.17 recovery.conf
Line 1

restore_command = cp / data / p g s q l / a r c h i v e s/% f "%p"

PostgreSQL
, (,
, ).
www.postgresql.org/docs/current/static/continuous-archiving.html.

10.5

.
, .
, ..
( , ), ,
. ,

( AWS S3). ,
.

WAL-E
: github.com/wal-e/wal-e

200

10.5.
WAL-E PostgreSQL
WAL-logs Amazon S3 Windows Azure ( 0.7) pg_start_backup pg_stop_backup. Python Heroku, .

WAL-E : lzop, psql, pv ( mbuffer), python 2.6+ python (gevent >=


0.13, boto >= 2.0, azure).
daemontools. Ubuntu
:
10.18 WAL-E
Line 1
-

# PostgreSQL
$ a p t i t u d e i n s t a l l g i t - c o r e python - dev python - s e t u p t o o l s
python - p i p b u i l d - e s s e n t i a l l i b e v e n t - dev l z o p pv
daemontools daemontools - run

WAL-E:
10.19 WAL-E
Line 1

$ p i p i n s t a l l h t t p s : / / g i t h u b . com/ wal - e / wal - e / a r c h i v e / v0 . 7 a1 .


t a r . gz

WAL-E.

, WAL-E AWS S3,
Access Key ID Secret Access Key (
Amazon AWS).
S3:
10.20 S3
AWS_SECRET_ACCESS_KEY= . . . wal - e
\
- k AWS_ACCESS_KEY_ID
\
- - s3 - p r e f i x=s 3 : / / some - bucket / d i r e c t o r y / o r / whatever \
backup - push / var / l i b / p o s t g r e s q l / 9 . 2 / main

Line 1
-

s3- prefix URL, S3 (bucket)


, .
WAL- S3:
10.21 WAL- S3
AWS_SECRET_ACCESS_KEY= . . . wal - e
\
- k AWS_ACCESS_KEY_ID
\
- - s3 - p r e f i x=s 3 : / / some - bucket / d i r e c t o r y / o r / whatever \

Line 1
-

201

10.5.
wal - push / var / l i b / p o s t g r e s q l / 9 . 2 / main/ pg_xlog /
WAL_SEGMENT_LONG_HEX


envdir ( daemontools). envdir
:
10.22 WAL-E envdir
$
- $
- $
- $

mkdir - p / e t c / wal - e . d/ env


echo " s e c r e t - key " > / e t c / wal - e . d/ env /AWS_SECRET_ACCESS_KEY
echo " a c c e s s - key " > / e t c / wal - e . d/ env /AWS_ACCESS_KEY_ID
echo s 3 : / / some - bucket / d i r e c t o r y / o r / whatever > / e t c / wal - e
. d/ env /WALE_S3_PREFIX
5 $ chown -R r o o t : p o s t g r e s / e t c / wal - e . d

Line 1


WAL-E :
10.23 WAL-E envdir
$ e n v d i r / e t c / wal - e . d/ env wal - e backup - push . . .
- $ e n v d i r / e t c / wal - e . d/ env wal - e wal - push . . .

Line 1

PostgreSQL WAL- S3 c WAL-E. postgresql.conf:


10.24 PostgreSQL
w a l _ l e v e l = hot_standby # a r c h i v e , PostgreSQL < 9 . 0
archive_mode = on
- archive_command = e n v d i r / e t c / wal - e . d/ env / u s r / l o c a l / b i n /
wal - e wal - push %p
- a r c h i v e _ t i m e o u t = 60

Line 1
-

WAL-E ( which
wal-e), PostgreSQL . PostgreSQL. - :
10.25 PostgreSQL
Line 1
5

2 0 1 2 - 1 1 - 0 7 1 4 : 5 2 : 1 9 UTC LOG: d a t a b a s e system was s h u t down


a t 2 0 1 2 - 1 1 - 0 7 1 4 : 5 1 : 4 0 UTC
2 0 1 2 - 1 1 - 0 7 1 4 : 5 2 : 1 9 UTC LOG: d a t a b a s e system i s ready t o
accept connections
2 0 1 2 - 1 1 - 0 7 1 4 : 5 2 : 1 9 UTC LOG: autovacuum l a u n c h e r s t a r t e d
2 0 1 2 - 1 1 - 0 7 T14 : 5 2 : 1 9 . 7 8 4 + 0 0 p i d =7653 wal_e . worker . s3_worker
INFO
MSG: b e g i n a r c h i v i n g a f i l e
DETAIL : Uploading " pg_xlog /000000010000000000000001 "
t o " s 3 : / / c l e v e r d b - pg - backups /pg/ wal_005
/000000010000000000000001. l z o " .
2 0 1 2 - 1 1 - 0 7 1 4 : 5 2 : 1 9 UTC LOG: i n c o m p l e t e s t a r t u p p a c k e t

202

10.5.
-

10

15
-

2 0 1 2 - 1 1 - 0 7 T14 : 5 2 : 2 8 . 2 3 4 + 0 0 p i d =7653 wal_e . worker . s3_worker


INFO
MSG: completed a r c h i v i n g t o a f i l e
DETAIL : A r c h i v i n g t o " s 3 : / / c l e v e r d b - pg - backups /pg/
wal_005 / 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 . l z o " c o m p l e t e a t 2 1 5 8 3 . 3
KiB/ s .
2 0 1 2 - 1 1 - 0 7 T14 : 5 2 : 2 8 . 3 4 1 + 0 0 p i d =7697 wal_e . worker . s3_worker
INFO
MSG: b e g i n a r c h i v i n g a f i l e
DETAIL : Uploading " pg_xlog
/ 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 . 0 0 0 0 0 0 2 0 . backup " t o " s 3 : / /
c l e v e r d b - pg - backups /pg/ wal_005
/ 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 . 0 0 0 0 0 0 2 0 . backup . l z o " .
2 0 1 2 - 1 1 - 0 7 T14 : 5 2 : 3 4 . 0 2 7 + 0 0 p i d =7697 wal_e . worker . s3_worker
INFO
MSG: completed a r c h i v i n g t o a f i l e
DETAIL : A r c h i v i n g t o " s 3 : / / c l e v e r d b - pg - backups /pg/
wal_005 / 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 . 0 0 0 0 0 0 2 0 . backup . l z o "
c o m p l e t e a t 00KiB/ s .
2 0 1 2 - 1 1 - 0 7 T14 : 5 2 : 3 4 . 1 8 7 + 0 0 p i d =7711 wal_e . worker . s3_worker
INFO
MSG: b e g i n a r c h i v i n g a f i l e
DETAIL : Uploading " pg_xlog /000000010000000000000002 "
t o " s 3 : / / c l e v e r d b - pg - backups /pg/ wal_005
/000000010000000000000002. l z o " .
2 0 1 2 - 1 1 - 0 7 T14 : 5 2 : 4 0 . 2 3 2 + 0 0 p i d =7711 wal_e . worker . s3_worker
INFO
MSG: completed a r c h i v i n g t o a f i l e
DETAIL : A r c h i v i n g t o " s 3 : / / c l e v e r d b - pg - backups /pg/
wal_005 / 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 . l z o " c o m p l e t e a t 2 4 6 6 . 6 7
KiB/ s .

,
.
, :
10.26 S3
Line 1
-

5
-

$ e n v d i r / e t c / wal - e . d/ env wal - e backup - push / var / l i b /


p o s t g r e s q l / 9 . 2 / main
2 0 1 2 - 1 1 - 0 7 T14 : 4 9 : 2 6 . 1 7 4 + 0 0 p i d =7493 wal_e . o p e r a t o r .
s 3 _ o p e r a t o r INFO
MSG: s t a r t upload p o s t g r e s v e r s i o n
metadata
DETAIL : Uploading t o s 3 : / / c l e v e r d b - pg - backups /pg/
basebackups_005 / base_000000010000000000000006_00000032 /
extended_version . txt .
2 0 1 2 - 1 1 - 0 7 T14 : 4 9 : 3 2 . 7 8 3 + 0 0 p i d =7493 wal_e . o p e r a t o r .
s 3 _ o p e r a t o r INFO
MSG: p o s t g r e s v e r s i o n metadata
upload c o m p l e t e
2 0 1 2 - 1 1 - 0 7 T14 : 4 9 : 3 2 . 8 5 9 + 0 0 p i d =7493 wal_e . worker . s3_worker
INFO
MSG: b e g i n n i n g volume c o m p r e s s i o n
DETAIL : B u i l d i n g volume 0 .
...

203

10.5.
HINT : Check t h a t your archive_command i s e x e c u t i n g p r o p e r l y
. pg_stop_backup can be c a n c e l e d s a f e l y , but t h e
d a t a b a s e backup w i l l not be u s a b l e w i t h o u t a l l t h e WAL
segments .
- NOTICE :
pg_stop_backup complete , a l l r e q u i r e d WAL segments
have been a r c h i v e d
-

. 10.1: S3

. 10.2: S3
(,
crontab). 10.1-10.3 S3.
lzop. gzip,
( 25/ 5% ).
pv ( cluster -read-rate- limit
, ).
.
backup-fetch :
10.27 S3
Line 1

$ sudo - u p o s t g r e s bash - c " e n v d i r / e t c / wal - e . d/ env wal - e


- - s3 - p r e f i x=s 3 : / / some - bucket / d i r e c t o r y / o r / whatever backup
- f e t c h / var / l i b / p o s t g r e s q l / 9 . 2 / main LATEST"

204

10.5.

. 10.3: WAL- S3
LATEST
(PostgreSQL ).
:
10.28
Line 1

$ sudo - u p o s t g r e s bash - c " e n v d i r / e t c / wal - e . d/ env wal - e


- - s3 - p r e f i x=s 3 : / / some - bucket / d i r e c t o r y / o r / whatever backup
- f e t c h / var / l i b / p o s t g r e s q l / 9 . 2 / main
base_LONGWALNUMBER_POSITION_NUMBER"

backup
-list :
10.29
$ e n v d i r / e t c / wal - e . d/ env wal - e backup - l i s t
- name
last_modified
expanded_size_bytes
wal_segment_backup_start
wal_segment_offset_backup_start wal_segment_backup_stop
wal_segment_offset_backup_stop
- base_000000010000000000000008_00000032
2 0 1 2 - 1 1 - 0 7 T14
:00:07.000Z
000000010000000000000008
00000032
- base_00000001000000000000000C_00000032
2 0 1 2 - 1 1 - 0 8 T15
:00:08.000Z
00000001000000000000000C
00000032

Line 1


WAL- (
). recovery.conf:
205

10.5.
10.30 recovery.conf
Line 1

restore_command = e n v d i r / e t c / wal - e . d/ env / u s r / l o c a l / b i n /


wal - e wal - f e t c h "%f " "%p"

PostgreSQL.
.
( )
delete :
10.31
Line 1
5
-

#
base_00000004000002DF000000A6_03626144
$ e n v d i r / e t c / wal - e . d/ env wal - e d e l e t e - - c o n f i r m b e f o r e
base_00000004000002DF000000A6_03626144
#
$ e n v d i r / e t c / wal - e . d/ env wal - e d e l e t e - - c o n f i r m e v e r y t h i n g
# 20
$ e n v d i r / e t c / wal - e . d/ env wal - e d e l e t e - - c o n f i r m r e t a i n 20

-- confirm ,
, (dry run).

WAL-E PostgreSQL
Amazon S3.

Barman
: www.pgbarman.org
Barman, WAL-E, PostgreSQL .
Barman , , PostgreSQL
.

PostgreSQL (
) pghost.
( brhost
).
SSH ( , ).
authorized_keys .
10.32 SSH
Line 1

# PostgreSQL ( p g h o s t )

206

10.5.
$ s s h barman@brhost
# ( b r h o s t )
- $ ssh postgres@pghost
-

barman. barman
python : python 2.6+, rsync >= 3.0.4
python (argh, psycopg2, python-dateutil < 2.0 ( python 3.0
), distribute). Ubuntu
:
10.33 barman
Line 1

$ a p t i t u d e i n s t a l l python - dev python - argh python - psycopg2


python - d a t e u t i l r s y n c python - s e t u p t o o l s

barman:
10.34 barman
$
- $
- $
- $

Line 1

t a r - x z f barman - 1 . 3 . 2 . t a r . gz
cd barman - 1 . 3 . 2 /
. / s e t u p . py b u i l d
sudo . / s e t u p . py i n s t a l l

PostgreSQL Community APT :


10.35 barman
Line 1

$ apt - g e t i n s t a l l barman

PostgreSQL. , barman
,
PostgreSQL:
10.36 postgresql.conf
Line 1

listen_adress = *
10.37 pg_hba.conf

Line 1

host

all

all

b r h o s t /32

trust

PostgreSQL.
PostgreSQL:
10.38
Line 1
5

$ p s q l - c SELECT v e r s i o n ( ) -U p o s t g r e s - h p g h o s t
version
--------------------------------------------------------------------------PostgreSQL 9 . 3 . 1 on x86_64 - unknown - l i n u x - gnu , c o m p i l e d by
g c c ( Ubuntu/ L i n a r o 4 . 7 . 2 - 2 ubuntu1 ) 4 . 7 . 2 , 64 - b i t
( 1 row )

207

10.5.

:
10.39
Line 1
-

$ sudo mkdir - p / s r v /barman


$ sudo chown barman : barman / s r v /barman

barman /etc/barman.conf:
10.40 barman.conf
[ barman ]
- ; Main d i r e c t o r y
- barman_home = / s r v /barman

Line 1

5
-

; Log l o c a t i o n
l o g _ f i l e = / var / l o g /barman/barman . l o g

; D e f a u l t c o m p r e s s i o n l e v e l : p o s s i b l e v a l u e s a r e None (
d e f a u l t ) , bzip2 , g z i p o r custom
compression = gzip

10
-

; main PostgreSQL S e r v e r c o n f i g u r a t i o n
[ main ]
; Human r e a d a b l e d e s c r i p t i o n
d e s c r i p t i o n = "Main PostgreSQL Database "

15

; SSH o p t i o n s
- ssh_command = s s h p o s t g r e s @ p g h o s t
20

; PostgreSQL c o n n e c t i o n s t r i n g
c o n n i n f o = h o s t=p g h o s t u s e r=p o s t g r e s

main ( barman PostgreSQL )


PostgreSQL . :
10.41 barman
Line 1
5
10
-

$ barman show - s e r v e r main


S e r v e r main :
active : true
d e s c r i p t i o n : Main PostgreSQL Database
ssh_command : s s h p o s t g r e s @ p g h o s t
c o n n i n f o : h o s t=p g h o s t u s e r=p o s t g r e s
b a c k u p _ d i r e c t o r y : / s r v /barman/main
b a s e b a c k u p s _ d i r e c t o r y : / s r v /barman/main/ b a s e
w a l s _ d i r e c t o r y : / s r v /barman/main/ w a l s
i n c o m i n g _ w a l s _ d i r e c t o r y : / s r v /barman/main/ incoming
l o c k _ f i l e : / s r v /barman/main/main . l o c k

208

10.5.
compression : gzip
c u s t o m _ c o m p r e s s i o n _ f i l t e r : None
c u s t o m _ d e c o m p r e s s i o n _ f i l t e r : None
r e t e n t i o n _ p o l i c y : None
w a l _ r e t e n t i o n _ p o l i c y : None
pre_backup_script : None
post_backup_script : None
c u r r e n t _ x l o g : None
last_shipped_wal : None
archive_command : None
server_txt_version : 9 . 3 . 1
d a t a _ d i r e c t o r y : / var / l i b / p o s t g r e s q l / 9 . 3 / main
archive_mode : o f f
c o n f i g _ f i l e : / e t c / p o s t g r e s q l / 9 . 3 / main/ p o s t g r e s q l .

15
20
25

conf
-

h b a _ f i l e : / e t c / p o s t g r e s q l / 9 . 3 / main/pg_hba . c o n f
i d e n t _ f i l e : / e t c / p o s t g r e s q l / 9 . 3 / main/ pg_ident . c o n f

30
35
-

# barman check main


S e r v e r main :
s s h : OK
PostgreSQL : OK
archive_mode : FAILED ( p l e a s e s e t i t t o on )
archive_command : FAILED ( p l e a s e s e t i t a c c o r d i n g l y
t o documentation )
d i r e c t o r i e s : OK
c o m p r e s s i o n s e t t i n g s : OK

, PostgreSQL .
PostgreSQL :
10.42 PostgreSQL
w a l _ l e v e l = hot_standby # a r c h i v e PostgreSQL < 9 . 0
archive_mode = on
- archive_command = r s y n c - a %p barman@brhost :
INCOMING_WALS_DIRECTORY/% f

Line 1
-

INCOMING_WALS_DIRECTORY
WAL-. barman show-server main
( 10.41, /srv/barman/main/incoming). PostgreSQL.
:
10.43
$ barman check main
S e r v e r main :
s s h : OK
PostgreSQL : OK

Line 1
-

209

10.5.
5
-

archive_mode : OK
archive_command : OK
d i r e c t o r i e s : OK
c o m p r e s s i o n s e t t i n g s : OK

. , barman.conf .

:
10.44
$ barman l i s t - s e r v e r
- main - Main PostgreSQL Database

Line 1

PostgreSQL ( ):
10.45
Line 1
5
-

$ barman backup main


S t a r t i n g backup f o r s e r v e r main i n / s r v /barman/main/ b a s e
/20121109 T090806
Backup s t a r t a t x l o g l o c a t i o n : 0/3000020
(000000010000000000000003 , 00000020)
Copying f i l e s .
Copy done .
Asking PostgreSQL s e r v e r t o f i n a l i z e t h e backup .
Backup end a t x l o g l o c a t i o n : 0/30000D8
( 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 , 000000D8)
Backup completed

( cron).
:
10.46
$ barman l i s t - backup main
- main 20121110 T091608 - F r i Nov 10 0 9 : 2 0 : 5 8 2012 - S i z e : 1 . 0
GiB - WAL S i z e : 4 4 6 . 0 KiB
- main 20121109 T090806 - F r i Nov
9 0 9 : 0 8 : 1 0 2012 - S i z e : 2 3 . 0
MiB - WAL S i z e : 4 7 7 . 0 MiB

Line 1

:
10.47
$ barman show - backup main 20121110 T091608
- Backup 20121109 T091608 :
S e r v e r Name
: main
Status :
: DONE

Line 1

210

10.5.
5
-

PostgreSQL V e r s i o n : 90201
PGDATA d i r e c t o r y
: / var / l i b / p o s t g r e s q l / 9 . 3 / main

10
15
-

Base backup i n f o r m a t i o n :
Disk u s a g e
: 1 . 0 GiB
Timeline
: 1
Begin WAL
: 00000001000000000000008C
End WAL
: 000000010000000000000092
WAL number
: 7
Begin time
: 2012 -11 -10 0 9 : 1 6 : 0 8 . 8 5 6 8 8 4
End time
: 2012 -11 -10 0 9 : 2 0 : 5 8 . 4 7 8 5 3 1
Begin O f f s e t
: 32
End O f f s e t
: 3576096
Begin XLOG
: 0/8 C000020
End XLOG
: 0/92369120

20
-

WAL i n f o r m a t i o n :
No o f f i l e s
Disk u s a g e
Last a v a i l a b l e

: 1
: 4 4 6 . 0 KiB
: 000000010000000000000093

25
-

Catalog information :
P r e v i o u s Backup : 20121109 T090806
Next Backup
: - ( t h i s i s t h e l a t e s t b a s e backup )

WAL-,
cron:
10.48 WAL-
Line 1
5
-

$ barman c r o n
P r o c e s s i n g x l o g segments f o r main
000000010000000000000001
000000010000000000000002
000000010000000000000003
0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 . 0 0 0 0 0 0 2 0 . backup
000000010000000000000004
000000010000000000000005
000000010000000000000006

cron. , WAL- (
- ). Barman WAL- gzip, bzip2 ( custom_compression_filter
custom_decompression_filter ).
network_compression
( ). bandwidth_limit (

211

10.5.
0, ) tablespace_bandwidth_limit .
recover:
Line 1
5
-

10.49
$ barman r e c o v e r - - remote - ssh - command " s s h p o s t g r e s @ p g h o s t "
main 20121109 T090806 / var / l i b / p o s t g r e s q l / 9 . 3 / main
S t a r t i n g remote r e s t o r e f o r s e r v e r main u s i n g backup
20121109 T090806
D e s t i n a t i o n d i r e c t o r y : / var / l i b / p o s t g r e s q l / 9 . 3 / main
Copying t h e b a s e backup .
Copying r e q u i r e d wal segments .
The archive_command was s e t t o f a l s e t o p r e v e n t data
losses .

Your PostgreSQL s e r v e r has been s u c c e s s f u l l y p r e p a r e d f o r


recovery !

10
-

P l e a s e r e v i e w network and a r c h i v e r e l a t e d s e t t i n g s i n t h e
PostgreSQL
c o n f i g u r a t i o n f i l e b e f o r e s t a r t i n g the j u s t recovered
instance .

WARNING: B e f o r e s t a r t i n g up t h e r e c o v e r e d PostgreSQL s e r v e r ,
- p l e a s e review a l s o the s e t t i n g s of the f o l l o w i n g
configuration
15 o p t i o n s a s they might i n t e r f e r e with your c u r r e n t r e c o v e r y
attempt :
-

20

d a t a _ d i r e c t o r y = / var / l i b / p o s t g r e s q l / 9 . 3 / main
# u s e data i n a n o t h e r d i r e c t o r y
e x t e r n a l _ p i d _ f i l e = / var / run / p o s t g r e s q l / 9 . 3 - main . pid
# w r i t e an e x t r a PID f i l e
h b a _ f i l e = / e t c / p o s t g r e s q l / 9 . 3 / main/pg_hba . conf
#
host - based a u t h e n t i c a t i o n f i l e
i d e n t _ f i l e = / e t c / p o s t g r e s q l / 9 . 3 / main/ pg_ident . conf
# ident configuration f i l e

Barman SSH ( remote-ssh-command). barman


PITR:
target -time ( ) target -xid (id ).

Barman
PostgreSQL . , PostgreSQL .
212

10.6.

10.6
, , , . ,
PostgreSQL (, ).

213

11

PostgreSQL
,
(),

.
-
,
, .

11.1
,
-
. -
( , ).
, SQL , , PostgreSQL ,
. , . :
?
( ..),
, , .
, ,
.

214

11.2.


, - - , . ,
:
;
.
,
, ( Twitter Facebook
). , .

11.2
, , .
, .. ,
,
.


PgPool-II v.3 + PostgreSQL v.9 Streaming Replication , . :

;
;
(failover);
;
(
pg_current_xlog_location pg_last_xlog_receive_location);
pgpool-II;
.

PgPool-II v.3 + PostgreSQL Slony/Londiste/Bucardo


,
Slony/Londiste/Bucardo. :
(failover);
;

;
pgpool-II;
Postgresql 9 .
Postgres-XC 5.3 Postgres-XC .
215

11.3.

11.3
,
( Google Analytics).
( ).



.
PgQ , PostgreSQL. Skype. Londiste (
4.5 Londiste). :

PostgreSQL;
,
;
PgQ , ;
(batches);
API SQL ;
.
RabbitMQ.
RabbitMQ , (Message
Oriented Middleware) AMQP (Advanced Message
Queuing Protocol). RabbitMQ Mozilla Public License.
RabbitMQ Open Telecom Platform, Erlang.
Postgres-XC 5.3 Postgres-XC .

11.4
PostgreSQL.
,
. ,
/ .

216

12

(Performance Snippets)



.
. -
.
, ,
.
(House M.D.),
1 1

12.1
PostgreSQL,
, ( ).
, PostgreSQL.
, PostgreSQL , .

12.2

(,
).

217

12.2.
snippets/biggest_relations.sql
Line 1
5
-

SELECT nspname | | . | | relname AS " r e l a t i o n " ,


p g _ s i z e _ p r e t t y ( p g _ r e l a t i o n _ s i z e (C . o i d ) ) AS " s i z e "
FROM p g _ c l a s s C
LEFT JOIN pg_namespace N ON (N. o i d = C . r e l n a m e s p a c e )
WHERE nspname NOT IN ( pg_catalog , information_schema )
ORDER BY p g _ r e l a t i o n _ s i z e (C. o i d ) DESC
LIMIT 2 0 ;

:
12.1 .
Line 1
5
-

relation
|
size
- - - - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - public . accounts
| 326 MB
p u b l i c . accounts_pkey
| 44 MB
public . history
| 592 kB
public . tellers_pkey
| 16 kB
p u b l i c . branches_pkey
| 16 kB
public . t e l l e r s
| 16 kB
public . branches
| 8192 b y t e s


.
snippets/biggest_tables.sql
Line 1
5
-

SELECT nspname | | . | | relname AS " r e l a t i o n " ,


p g _ s i z e _ p r e t t y ( p g _ t o t a l _ r e l a t i o n _ s i z e (C . o i d ) ) AS "
total_size "
FROM p g _ c l a s s C
LEFT JOIN pg_namespace N ON (N. o i d = C . r e l n a m e s p a c e )
WHERE nspname NOT IN ( pg_catalog , information_schema )
AND C. r e l k i n d <> i
AND nspname ! ~ ^pg_toast
ORDER BY p g _ t o t a l _ r e l a t i o n _ s i z e (C. o i d ) DESC
LIMIT 2 0 ;

:
12.2 .
Line 1
5

relation
| total_size
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - public . actions
| 4249 MB
p u b l i c . p r o d u c t _ h i s t o r y _ r e c o r d s | 197 MB
p u b l i c . product_updates
| 52 MB

218

12.2.
-

p u b l i c . import_products
public . products
public . v i s i t s

| 34 MB
| 29 MB
| 25 MB

count

. , count.
snippets/count_estimate.sql
Line 1
5
-

CREATE FUNCTION c ou n t _ es t i ma t e ( query t e x t ) RETURNS i n t e g e r


AS $$
DECLARE
rec
record ;
rows i n t e g e r ;
BEGIN
FOR r e c IN EXECUTE EXPLAIN | | query LOOP
rows := s u b s t r i n g ( r e c . "QUERY PLAN" FROM rows = ( [ [ :
digit :]]+) ) ;
EXIT WHEN rows IS NOT NULL;
END LOOP;

10

RETURN rows ;
END;
- $$ LANGUAGE p l p g s q l VOLATILE STRICT ;
-

:
12.3 count.
CREATE TABLE f o o ( r d o u b l e p r e c i s i o n ) ;
INSERT INTO f o o SELECT random ( ) FROM g e n e r a t e _ s e r i e s ( 1 ,
1000) ;
- ANALYZE f o o ;

Line 1
-

# SELECT count ( * ) FROM f o o WHERE r < 0 . 1 ;


count
- -- ----92
- ( 1 row )

5
-

10
15

# SELECT c o u nt _ e st i m a te ( SELECT * FROM f o o WHERE r < 0 . 1 ) ;


coun t _ es t i ma t e
-- -------------94
( 1 row )

219

12.2.


( DEFAULT).
snippets/default_value.sql
Line 1
5
-

CREATE FUNCTION r e t _ d e f ( t e x t , t e x t , t e x t ) RETURNS t e x t AS $$


SELECT
COLUMNS. column_default : : t e x t
FROM
information_schema .COLUMNS
WHERE table_name = $2
AND table_schema = $1
AND column_name = $3
$$ LANGUAGE s q l IMUTABLE;

:
12.4 .
Line 1

# SELECT r e t _ d e f ( schema , t a b l e , column ) ;

SELECT r e t _ d e f ( p u b l i c , i m a g e _ f i l e s , i d ) ;
ret_def
5 -- --------------------------------------nextval ( image_files_id_seq : : r e g c l a s s )
- ( 1 row )
-

SELECT r e t _ d e f ( p u b l i c , schema_migrations , v e r s i o n ) ;
10
ret_def
- -- -------

( 1 row )


( ).
snippets/random_from_range.sql
CREATE OR REPLACE FUNCTION random ( numeric , numeric )
RETURNS numeric AS
- $$
SELECT ( $1 + ( $2 - $1 ) * random ( ) ) : : numeric ;
5 $$ LANGUAGE s q l VOLATILE ;

Line 1
-

:
12.5 .

220

12.2.
SELECT random ( 1 , 1 0 ) : : i n t , random ( 1 , 1 0 ) ;
random |
random
- - - - - - - - -+ - - - - - - - - - - - - - - - - - 6 | 5.11675184825435
5 ( 1 row )

Line 1
-

SELECT random ( 1 , 1 0 ) : : i n t , random ( 1 , 1 0 ) ;


random |
random
- - - - - - - - -+ - - - - - - - - - - - - - - - - - 10
7 | 1.37060070643201
- ( 1 row )
-


, . , , , . IBM
1960 .
,
.
SQL. ,
.
snippets/luhn_algorithm.sql
Line 1
5
10
-

CREATE OR REPLACE FUNCTION l u h n _ v e r i f y ( i n t 8 ) RETURNS BOOLEAN


AS $$
- - Take t h e sum o f t h e
- - doubled d i g i t s and t h e even - numbered undoubled d i g i t s ,
and s e e i f
- - t h e sum i s e v e n l y d i v i s i b l e by z e r o .
SELECT
- - Doubled d i g i t s might i n t u r n be two d i g i t s . In
that case ,
- - we must add each d i g i t i n d i v i d u a l l y r a t h e r than
adding t h e
- - doubled d i g i t v a l u e t o t h e sum . I e i f t h e
o r i g i n a l d i g i t was
- - 6 t h e doubled r e s u l t was 1 2 and we must add
1+2 t o t h e
- - sum r a t h e r than 1 2 .
MOD(SUM( d o u b l e d _ d i g i t / INT8 10 + d o u b l e d _ d i g i t %
INT8 10 ) , 1 0 ) = 0
FROM

221

12.2.
15
20
25

- - Double odd - numbered d i g i t s ( c o u n t i n g l e f t with


- - l e a s t s i g n i f i c a n t a s z e r o ) . I f t h e doubled d i g i t s end up
- - having v a l u e s
- - > 10 ( i e they r e two d i g i t s ) , add t h e i r d i g i t s t o g e t h e r .
(SELECT
- - E x t r a c t d i g i t n c o u n t i n g l e f t from l e a s t
significant
- - as zero
MOD( ( $1 : : i n t 8 / (10^ n ) : : i n t 8 ) , 1 0 : : i n t 8 )
- - Double odd - numbered d i g i t s
* (MOD( n , 2 ) + 1 )
AS d o u b l e d _ d i g i t
FROM g e n e r a t e _ s e r i e s ( 0 , CEIL (LOG( $1 ) ) : : INTEGER 1 ) AS n
) AS d o u b l e d _ d i g i t s ;

$$ LANGUAGE SQL
IMMUTABLE
- STRICT ;
-

30
35
40
45
-

COMMENT ON FUNCTION l u h n _ v e r i f y ( i n t 8 ) IS Return t r u e i f f


the l a s t d i g i t of the
i n p u t i s a c o r r e c t check d i g i t f o r t h e r e s t o f t h e i n p u t
a c c o r d i n g t o Luhn s
algorithm . ;
CREATE OR REPLACE FUNCTION l u h n _ g e n e r a t e _ c h e c k d i g i t ( i n t 8 )
RETURNS i n t 8 AS $$
SELECT
- - Add t h e d i g i t s , d o u b l i n g even - numbered d i g i t s (
counting l e f t
- - with l e a s t - s i g n i f i c a n t a s z e r o ) . S u b t r a c t t h e
remainder of
- - d i v i d i n g t h e sum by 10 from 1 0 , and t a k e t h e
remainder
- - o f d i v i d i n g t h a t by 10 i n t u r n .
( ( INT8 10 - SUM( d o u b l e d _ d i g i t / INT8 10 +
d o u b l e d _ d i g i t % INT8 10 ) %
INT8 10 ) % INT8 10 ) : : INT8
FROM (SELECT
- - E x t r a c t d i g i t n c o u n t i n g l e f t from l e a s t
significant\
- - as zero
MOD( ( $1 : : i n t 8 / (10^ n ) : : i n t 8 ) , 1 0 : : i n t 8 )
- - d o u b l e even - numbered d i g i t s
* ( 2 - MOD( n , 2 ) )
AS d o u b l e d _ d i g i t
FROM g e n e r a t e _ s e r i e s ( 0 , CEIL (LOG( $1 ) ) : : INTEGER - 1 )

222

12.2.

50

AS n
) AS d o u b l e d _ d i g i t s ;

$$ LANGUAGE SQL
IMMUTABLE
- STRICT ;
-

55
60
-

COMMENT ON FUNCTION l u h n _ g e n e r a t e _ c h e c k d i g i t ( i n t 8 ) IS For


the input
value , g e n e r a t e a check d i g i t a c c o r d i n g t o Luhn s a l g o r i t h m
;
CREATE OR REPLACE FUNCTION l u h n _ g e n e r a t e ( i n t 8 ) RETURNS i n t 8
AS $$
SELECT 10 * $1 + l u h n _ g e n e r a t e _ c h e c k d i g i t ( $1 ) ;
$$ LANGUAGE SQL
IMMUTABLE
STRICT ;

65
70
-

COMMENT ON FUNCTION l u h n _ g e n e r a t e ( i n t 8 ) IS Append a check


d i g i t generated
a c c o r d i n g t o Luhn s a l g o r i t h m t o t h e i n p u t v a l u e . The i n p u t
v a l u e must be no
g r e a t e r than ( maxbigint / 1 0 ) . ;
CREATE OR REPLACE FUNCTION l u h n _ s t r i p ( i n t 8 ) RETURNS i n t 8 AS
$$
SELECT $1 / 1 0 ;
$$ LANGUAGE SQL
IMMUTABLE
STRICT ;

COMMENT ON FUNCTION l u h n _ s t r i p ( i n t 8 ) IS S t r i p t h e l e a s t
s i g n i f i c a n t d i g i t from
- t h e i n p u t v a l u e . I n t e n d e d f o r u s e when s t r i p p i n g t h e check
d i g i t from a number
75 i n c l u d i n g a Luhn s a l g o r i t h m check d i g i t . ;
-

:
12.6 .
Line 1
5

S e l e c t luhn_verify (49927398716) ;
luhn_verify
-- ----------t
( 1 row )

S e l e c t luhn_verify (49927398714) ;
luhn_verify
-- -----------

223

12.2.
10
-

f
( 1 row )


IN. , . :
: (2,6,4,10,25,7,9). .. 2 2 2 6 6 4 4
snippets/order_like_in.sql
SELECT f o o . * FROM f o o
- JOIN (SELECT i d . val , row_number ( ) o v e r ( ) FROM (VALUES( 3 ) , ( 2 )
, ( 6 ) , ( 1 ) , ( 4 ) ) AS
- i d ( v a l ) ) AS i d
- ON ( f o o . c a t a l o g _ i d = i d . v a l ) ORDER BY row_number ;

Line 1

VALUES(3),(2),(6),(1),(4)
foo ,
foo.catalog_id , ( foo.
catalog_id IN (3,2,6,1,4) )

Quine
, (. quine) ( ),
.
snippets/quine.sql
Line 1
-

s e l e c t a | | from ( s e l e c t | | q u o t e _ l i t e r a l ( a ) | | b | | ,
| | q u o t e _ l i t e r a l ( b ) | | : : t e x t a s b ) a s q u i n e from
( s e l e c t s e l e c t a | | from ( s e l e c t | | q u o t e _ l i t e r a l ( a )
| | b | | , | | q u o t e _ l i t e r a l (b) | | : : text as b) as
quine : : text as a , : : text as a : : text as b) as quine ;

LIKE
web .
LIKE some\%, some , . , ( )
.
LIKE bla%
text_pattern_ops ( varchar_pattern_ops varchar).
224

12.2.
snippets/speed_like.sql
Line 1
5
10
-

p r e f i x _ t e s t=# c r e a t e t a b l e t a g s (
p r e f i x _ t e s t (# t a g
t e x t primary key ,
p r e f i x _ t e s t (# name
t e x t not n u l l ,
p r e f i x _ t e s t (# shortname t e x t ,
p r e f i x _ t e s t (# s t a t u s
char d e f a u l t S ,
p r e f i x _ t e s t (#
p r e f i x _ t e s t (# check ( s t a t u s i n ( S , R ) )
p r e f i x _ t e s t (# ) ;
NOTICE : CREATE TABLE / PRIMARY KEY w i l l c r e a t e i m p l i c i t
i n d e x " tags_pkey " f o r t a b l e " t a g s "
CREATE TABLE
p r e f i x _ t e s t=# CREATE INDEX i_tag ON t a g s USING b t r e e ( l o w e r (
t a g ) text_pattern_ops ) ;
CREATE INDEX

15
20
-

p r e f i x _ t e s t=# c r e a t e t a b l e i n v a l i d _ t a g s (
p r e f i x _ t e s t (# t a g
t e x t primary key ,
p r e f i x _ t e s t (# name
t e x t not n u l l ,
p r e f i x _ t e s t (# shortname t e x t ,
p r e f i x _ t e s t (# s t a t u s
char d e f a u l t S ,
p r e f i x _ t e s t (#
p r e f i x _ t e s t (# check ( s t a t u s i n ( S , R ) )
p r e f i x _ t e s t (# ) ;
NOTICE : CREATE TABLE / PRIMARY KEY w i l l c r e a t e i m p l i c i t
index " invalid_tags_pkey " f o r t a b l e " invalid_tags "
CREATE TABLE

25
30
-

p r e f i x _ t e s t=# s e l e c t count ( * ) from t a g s ;


count
-- ----11966
( 1 row )

35
-

p r e f i x _ t e s t=# s e l e c t count ( * ) from i n v a l i d _ t a g s ;


count
-- ----11966
( 1 row )

# EXPLAIN ANALYZE s e l e c t * from i n v a l i d _ t a g s where l o w e r ( t a g


) LIKE l o w e r ( 0146% ) ;
40
QUERY PLAN
- -------------------------------------------------------------------------

225

12.2.

45

Seq Scan on i n v a l i d _ t a g s ( c o s t = 0 . 0 0 . . 2 6 5 . 4 9 rows=60 width


=26) ( a c t u a l time = 0 . 3 5 9 . . 2 0 . 6 9 5 rows=1 l o o p s =1)
F i l t e r : ( l o w e r ( t a g ) ~~ 0146% : : t e x t )
T o t a l runtime : 2 0 . 8 0 3 ms
( 3 rows )

# EXPLAIN ANALYZE s e l e c t * from i n v a l i d _ t a g s where l o w e r ( t a g


) LIKE l o w e r ( 0146% ) ;
QUERY PLAN
- -------------------------------------------------------------------------

50
-

Seq Scan on i n v a l i d _ t a g s ( c o s t = 0 . 0 0 . . 2 6 5 . 4 9 rows=60 width


=26) ( a c t u a l time = 0 . 5 4 9 . . 1 9 . 5 0 3 rows=1 l o o p s =1)
F i l t e r : ( l o w e r ( t a g ) ~~ 0146% : : t e x t )
T o t a l runtime : 1 9 . 5 5 0 ms
( 3 rows )

# EXPLAIN ANALYZE s e l e c t * from t a g s where l o w e r ( t a g ) LIKE


l o w e r ( 0146% ) ;
QUERY
PLAN
- -------------------------------------------------------------------------

55

60
-

Bitmap Heap Scan on t a g s ( c o s t = 5 . 4 9 . . 9 7 . 7 5 rows =121 width


=26) ( a c t u a l time = 0 . 0 5 4 . . 0 . 0 5 7 rows=1 l o o p s =1)
F i l t e r : ( l o w e r ( t a g ) ~~ 0146% : : t e x t )
-> Bitmap Index Scan on i_tag ( c o s t = 0 . 0 0 . . 5 . 4 6 rows =120
width =0) ( a c t u a l time = 0 . 0 3 2 . . 0 . 0 3 2 rows=1 l o o p s =1)
Index Cond : ( ( l o w e r ( t a g ) ~>=~ 0146 : : t e x t ) AND (
l o w e r ( t a g ) ~<~ 0147 : : t e x t ) )
T o t a l runtime : 0 . 1 1 9 ms
( 5 rows )

# EXPLAIN ANALYZE s e l e c t * from t a g s where l o w e r ( t a g ) LIKE


l o w e r ( 0146% ) ;
QUERY
PLAN
- -------------------------------------------------------------------------

65

70

Bitmap Heap Scan on t a g s ( c o s t = 5 . 4 9 . . 9 7 . 7 5 rows =121 width


=26) ( a c t u a l time = 0 . 0 2 5 . . 0 . 0 2 5 rows=1 l o o p s =1)
F i l t e r : ( l o w e r ( t a g ) ~~ 0146% : : t e x t )
-> Bitmap Index Scan on i_tag ( c o s t = 0 . 0 0 . . 5 . 4 6 rows =120

226

12.2.

width =0) ( a c t u a l time = 0 . 0 1 6 . . 0 . 0 1 6 rows=1 l o o p s =1)


Index Cond : ( ( l o w e r ( t a g ) ~>=~ 0146 : : t e x t ) AND (
l o w e r ( t a g ) ~<~ 0147 : : t e x t ) )
T o t a l runtime : 0 . 0 5 0 ms
( 5 rows )


, ( , ).
snippets/duplicate_indexes.sql
Line 1
5
-

SELECT p g _ s i z e _ p r e t t y ( sum ( p g _ r e l a t i o n _ s i z e ( i d x ) ) : : b i g i n t ) AS
size ,
( array_agg ( i d x ) ) [ 1 ] AS idx1 , ( array_agg ( i d x ) ) [ 2 ] AS
idx2 ,
( array_agg ( i d x ) ) [ 3 ] AS idx3 , ( array_agg ( i d x ) ) [ 4 ] AS
idx4
FROM (
SELECT i n d e x r e l i d : : r e g c l a s s AS idx , ( i n d r e l i d : : t e x t | | E
\n | | i n d c l a s s : : t e x t | | E \n | | i n d k e y : : t e x t | | E \n | |
coalesce ( indexprs : :
t e x t , ) | | E \n | | c o a l e s c e ( i n d p r e d : : t e x t , ) ) AS KEY
FROM pg_index ) sub
GROUP BY KEY HAVING count ( * )>1
ORDER BY sum ( p g _ r e l a t i o n _ s i z e ( i d x ) ) DESC;


snippets/indexes_statustic.sql
Line 1
5
10
-

SELECT
t . tablename ,
indexname ,
c . r e l t u p l e s AS num_rows ,
p g _ s i z e _ p r e t t y ( p g _ r e l a t i o n _ s i z e ( quote_ident ( t . tablename )
: : t e x t ) ) AS t a b l e _ s i z e ,
p g _ s i z e _ p r e t t y ( p g _ r e l a t i o n _ s i z e ( quote_ident ( i n d e x r e l n a m e
) : : t e x t ) ) AS i n d e x _ s i z e ,
CASE WHEN x . i s _ u n i q u e = 1 THEN Y
ELSE N
END AS UNIQUE,
idx_scan AS number_of_scans ,
idx_tup_read AS t u p l e s _ r e a d ,
idx_tup_fetch AS t u p l e s _ f e t c h e d
FROM p g _ t a b l e s t

227

12.2.
15
20
-

25
-

LEFT OUTER JOIN p g _ c l a s s c ON t . tablename=c . relname


LEFT OUTER JOIN
(SELECT i n d r e l i d ,
max(CAST( i n d i s u n i q u e AS i n t e g e r ) ) AS i s _ u n i q u e
FROM pg_index
GROUP BY i n d r e l i d ) x
ON c . o i d = x . i n d r e l i d
LEFT OUTER JOIN
( SELECT c . relname AS ctablename , i p g . relname AS
indexname , x . i n d n a t t s AS number_of_columns , idx_scan ,
idx_tup_read , idx_tup_fetch , i n d e x r e l n a m e FROM pg_index x
JOIN p g _ c l a s s c ON c . o i d = x . i n d r e l i d
JOIN p g _ c l a s s i p g ON i p g . o i d = x . i n d e x r e l i d
JOIN p g _ s t a t _ a l l _ i n d e x e s p s a i ON x . i n d e x r e l i d =
psai . indexrelid )
AS f o o
ON t . tablename = f o o . ctablename
WHERE t . schemaname= p u b l i c
ORDER BY 1 , 2 ;

(bloat)
, bloat () :
snippets/bloating.sql
Line 1
5
10
15
-

WITH c o n s t a n t s AS (
SELECT c u r r e n t _ s e t t i n g ( b l o c k _ s i z e ) : : numeric AS bs , 23 AS
hdr , 4 AS ma
) , b l o a t _ i n f o AS (
SELECT
ma, bs , schemaname , tablename ,
( datawidth +(hdr+ma- ( c a s e when hdr%ma=0 THEN ma ELSE hdr%
ma END) ) ) : : numeric AS datahdr ,
( maxfracsum * ( n u l l h d r+ma- ( c a s e when n u l l h d r%ma=0 THEN ma
ELSE n u l l h d r%ma END) ) ) AS n u l l h d r 2
FROM (
SELECT
schemaname , tablename , hdr , ma, bs ,
SUM( ( 1 - n u l l _ f r a c ) * avg_width ) AS datawidth ,
MAX( n u l l _ f r a c ) AS maxfracsum ,
hdr+(
SELECT 1+count ( * ) /8
FROM pg_ st ats s 2
WHERE n u l l _ f r a c <>0 AND s 2 . schemaname = s . schemaname
AND s 2 . tablename = s . tablename
) AS n u l l h d r
FROM pg_ st ats s , c o n s t a n t s

228

12.2.
20
25
-

30
-

35
-

40
45
50
-

GROUP BY 1 , 2 , 3 , 4 , 5
) AS f o o
) , t a b l e _ b l o a t AS (
SELECT
schemaname , tablename , c c . r e l p a g e s , bs ,
CEIL ( ( c c . r e l t u p l e s * ( ( datahdr+ma(CASE WHEN datahdr%ma=0 THEN ma ELSE datahdr%ma END) )+
n u l l h d r 2 +4) ) / ( bs - 2 0 : : f l o a t ) ) AS o t t a
FROM b l o a t _ i n f o
JOIN p g _ c l a s s c c ON c c . relname = b l o a t _ i n f o . tablename
JOIN pg_namespace nn ON c c . r e l n a m e s p a c e = nn . o i d AND nn .
nspname = b l o a t _ i n f o . schemaname AND nn . nspname <>
information_schema
) , i n d e x _ b l o a t AS (
SELECT
schemaname , tablename , bs ,
COALESCE( c2 . relname , ? ) AS iname , COALESCE( c2 . r e l t u p l e s
, 0 ) AS i t u p l e s , COALESCE( c2 . r e l p a g e s , 0 ) AS i p a g e s ,
COALESCE( CEIL ( ( c2 . r e l t u p l e s * ( datahdr - 1 2 ) ) / ( bs - 2 0 : : f l o a t )
) , 0 ) AS i o t t a - - v e r y rough approximation , assumes a l l
cols
FROM b l o a t _ i n f o
JOIN p g _ c l a s s c c ON c c . relname = b l o a t _ i n f o . tablename
JOIN pg_namespace nn ON c c . r e l n a m e s p a c e = nn . o i d AND nn .
nspname = b l o a t _ i n f o . schemaname AND nn . nspname <>
information_schema
JOIN pg_index i ON i n d r e l i d = c c . o i d
JOIN p g _ c l a s s c2 ON c2 . o i d = i . i n d e x r e l i d
)
SELECT
type , schemaname , object_name , b l o a t , p g _ s i z e _ p r e t t y (
raw_waste ) a s waste
FROM
(SELECT
t a b l e a s type ,
schemaname ,
tablename a s object_name ,
ROUND(CASE WHEN o t t a=0 THEN 0 . 0 ELSE t a b l e _ b l o a t . r e l p a g e s /
o t t a : : numeric END, 1 ) AS b l o a t ,
CASE WHEN r e l p a g e s < o t t a THEN 0 ELSE ( bs * ( t a b l e _ b l o a t .
r e l p a g e s - o t t a ) : : b i g i n t ) : : b i g i n t END AS raw_waste
FROM
table_bloat
UNION
SELECT
i n d e x a s type ,
schemaname ,

229

12.2.
55
60

tablename | | : : | | iname a s object_name ,


ROUND(CASE WHEN i o t t a =0 OR i p a g e s =0 THEN 0 . 0 ELSE i p a g e s /
i o t t a : : numeric END, 1 ) AS b l o a t ,
CASE WHEN i p a g e s < i o t t a THEN 0 ELSE ( bs * ( i p a g e s - i o t t a ) )
: : b i g i n t END AS raw_waste
FROM
i n d e x _ b l o a t ) bloat_summary
ORDER BY raw_waste DESC, b l o a t DESC

230


[1]

(Sad
Spirit)
borz_off@cs.msu.su
PostgreSQL:

http://www.phpclub.ru/detail/store/pdf/postgresql-performance.pdf
[2] Eugene Kuzin eugene@kuzin.net PostgreSQL
Slony-I http://www.kuzin.net/work/slonikiprivet.html
[3] Sergey Konoplev gray.ru@gmail.com Londiste
http://gray-hemp.blogspot.com/2010/04/londiste.html
[4] Dmitry
Stasyuk

pgpool-II
http://undenied.ru/2009/03/04/uchebnoe-rukovodstvo-po-pgpool-ii/
[5]

dmitry.chirkin@gmail.com

PostgreSQL

PL/Proxy
http://habrahabr.ru/blogs/postgresql/45475/
[6] wordpress@insight-it.ru Hadoop http://www.insightit.ru/masshtabiruemost/hadoop/
[7] Padraig
OSullivan
Up
and
Running
with
HadoopDB
http://posulliv.github.com/2010/05/10/hadoopdb-mysql.html
[8] PostgreSQL:
Skype http://postgresmen.ru/articles/view/25
[9] Streaming Replication. http://wiki.postgresql.org/wiki/Streaming_Replication
[10] Den Golotyuk , , - ? http://highload.com.ua/index.php/2009/05/06/-/
[11] Postgres-XC

A
PostgreSQL
Clustering
Solution
http://www.linuxforu.com/2012/01/postgres-xc-database-clusteringsolution/
[12] PostgreSQL BDR http://habrahabr.ru/post/227959/

231