Работа с PostgreSQL. Настройка и масштабирование

PostgreSQL:
. . aka leopard
Creative Commons Attribution-Noncommercial 2.5
20102012
2
2.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . . .
. . . . . . .
2.2 . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . .
2.3 . . . . . . . . . . . . . . . . . .
. . . . . .
CLUSTER . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4 . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . .

(1), 2 . . . . . . . . . . . . . . . . . . . .
Web , 2
. . . . . . . . . . . . . . . . . . . . . . . .
Web , 8
. . . . . . . . . . . . . . . . . . . . . . . .
2.5 : pgtune .
2.6 . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . .
1
.
.
.
.
.
.
.
.
.
.
.
.
.
7
7
8
8
9
10
10
14
16
18
18
19
19
20
. 20
. 20
. 20
.
.
.
.
.
.
.
.
21
21
22
22
23
26
26
28

2.7
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3
3.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3 . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
constraint_exclusion
3.4 . . . . . . . . . . . . . . . . . . . . . . . . . . .
4
4.1 . . . . . . . . . . . . . . . . . . . . .
4.2 Streaming Replication ( )
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
4.3 Slony-I . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
. . . . . . . . . . .
4.4 Londiste . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
. . . . . . . . . . .
4.5 Bucardo . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
4.6 RubyRep . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . .
4.7 . . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
33
33
34
35
35
37
39
39
41
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
42
42
44
44
45
45
52
53
53
53
53
59
61
64
64
64
65
69
71
72
72
72
72
75
75
75
76
76
78
79
5
81
5.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
2

5.2
5.3
5.4
5.5
PL/Proxy . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
? . . . . . . . . . . . . . . .
Postgres-XC . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . .
(HA) . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
HadoopDB . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
6 PgPool-II
6.1 . . . . . . . . . . . . . . . . . . . . .
6.2 ! . . . . . . . . . . . . . . . . .
pgpool-II . . . . . . . . . . . . . . .
. . . . . . . . . . . . . .
PCP . . . . . . . . . . . . .
. . . . . . . . .
/ pgpool-II . . . . . . . . . .
6.3 . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . . .
6.4 . . . . . . .
. . . . . . .
SystemDB . . . . . . . . . . . . . .
. . .
. . . . . . . . .
. . . . . . .
6.5 Master-slave . . . . . . . . . . . . . . .
Streaming Replication ( )
6.6 . . . . . . . . . . . . .
6.7 . . . . . . . . . . . . . . . . . . . .
7
7.1 .
7.2 PgBouncer .
7.3 PgPool-II vs
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
82
83
84
87
88
88
90
90
91
97
98
98
98
103
115
116
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
117
117
118
118
119
120
121
121
122
122
123
124
124
125
128
129
130
131
131
132
133
135
136
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
PgBouncer . . . . . . . . . . . . . . . . . . . . . . 137

8 PostgreSQL
8.1 . . . . . . .
8.2 Pgmemcache . . . . . .
. . . . . . .
. . . . . . .
. . . . . . .
. . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
139
139
140
140
141
142
146
9
9.1 . . . . . . . . . . . . . . . .
9.2 PostGIS . . . . . . . . . . . . . . . . .
9.3 pgSphere . . . . . . . . . . . . . . . . .
9.4 HStore . . . . . . . . . . . . . . . . . .
. . . . . . . .
. . . . . . . . . . . . . . .
9.5 PLV8 . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
9.6 Smlar . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
. . . . . . . . . . .
Smlar . . . . . . . . . . . . . . . . . .
:
. . . . . . . . . . . . . . . . . .
9.7 PostPic . . . . . . . . . . . . . . . . .
9.8 Fuzzystrmatch . . . . . . . . . . . . . .
9.9 Tsearch2 . . . . . . . . . . . . . . . . .
9.10 OpenFTS . . . . . . . . . . . . . . . .
9.11 PL/Proxy . . . . . . . . . . . . . . . .
9.12 Texcaller . . . . . . . . . . . . . . . . .
9.13 Pgmemcache . . . . . . . . . . . . . . .
9.14 Prefix . . . . . . . . . . . . . . . . . .
9.15 Dblink . . . . . . . . . . . . . . . . . .
9.16 Ltree . . . . . . . . . . . . . . . . . . .
9.17 . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
147
147
147
147
148
148
149
150
150
152
155
155
155
156
157
159
162
162
163
165
165
165
166
166
166
166
167
167
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
. . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
168
168
169
170
171
172
172
173
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
10 PostgreSQL
10.1 . . . . . . . . . . . . . . . . .
10.2 SQL . . . . . . . . . . . . . . . . .
SQL . . . .
10.3 . . .
10.4 .
. . . . . . . . . . . . . . . . .
10.5
4

WAL-E . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
Barman . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
10.6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
11 PostgreSQL
11.1 . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . .
11.2 . . . . . . . . .
. . . . . . . . . . . . . .
11.3 . . . . . . . . .
. . . . . . . . . . . . . .
11.4 . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
12 (Performance Snippets)
12.1 . . . . . . . . . . . . . . . . . . . . . . . .
12.2 . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . . .
count . . . . . . . . . . . . . . . . . . . .
.
. . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . .

, . . . . .
LIKE . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . .
. . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
187
187
188
188
188
189
189
189
.
.
.
.
.
.
.
.
.
.
.
.
.
190
190
190
190
191
192
193
193
194
197
197
197
200
200
202
,

,
.
PostgreSQL.
PostgreSQL,
. ,

.
2

,
.

2.1
, ,
. ,
.

;
.
,
.
, ,
?
, ?
,
.
PostgreSQL.
,
PostgreSQL , ,
FAQ.
postgresql-performance,
.
,
,
. :
- .
7
2.1.

PostgreSQL ,

.
.
: ,
PostgreSQL,
.

( ) ,
. PostgreSQL,
, .
, , ..
PostgreSQL :
;
, ;
;
,
(, ).

PostgreSQL,
, .

.
7.1 ,
.
7.2 :
VACUUM, ;
ANALYZE,
,
;
.
7.4
( IN/NOT IN).
8.0 ,
, CHECKPOINT VACUUM .
8.1
, MIN()
MAX(), pg_autovacuum (),
.
8.2 SQL ,
.
8
2.1.
8.3 , SQL/XML
,
.
8.4 ,
, ,
EXISTS/NOT EXISTS .
9.0 , VACUUM/VACUUM FULL , .
9.1 ,
( ,
), ,

, MIN/MAX.
9.2 ,
, JSON , ,
25%, COPY.
,
8.4.

, ,
,
. ,
,
.
,
,
, ? ,
,

. ,
,
. , ,
:
.
(
).
(, ,
).

.
9
2.2.
(.
3.4).
2.2
,
.
postgresql.conf
.

: shared_buffers
PostgreSQL
. ,
,
, .
,
.
,
, .
, ,
.

,
, .

: , PostgreSQL,
PostgreSQL ,
.

,
PostgreSQL , ,
. ,
,
, .
, shared_buffers,
, ,
,
.
8
2 . ,
, , ,
,
. , ,
10
2.2.
.
, .
:
4 (512)
256512 : 1632
(20484096)
14 : 64256
(819232768)

.
ipcs
(, free vmstat).
1,2 2 ,
. ,
, . ,

. PostgreSQL
, : www.postgresql.org
,
:
Laptop, Celeron processor, 384 RAM, 25 : 12
Athlon server, 1 RAM,
10 : 200
Quad PIII server, 4 RAM, 40 , 150 ,
: 1
Quad Xeon server, 8 RAM, 200 , 300 ,
: 2
: work_mem
sort_mem, ,
,
, .
, work_mem (
).
:
( , ,
, shared_buffers)

, .

, .
, .
11
2.2.
work_mem postgresql.conf.
1 . 1024.
24%
. -
work_mem, , ,
512 2048 . ,

work_mem 500 . ,
, ,
, . , 1
4 32128 MB.
VACUUM: maintenance_work_mem
PostgreSQL 7.x vacuum_mem.
, VACUUM, ANALYZE,
CREATE INDEX, .
,
, .
50 75%
, , 32
256 . , work_mem.
. ,
14 128512 MB.
Free Space Map: VACUUM FULL
(
PostgreSQL) :
, ,
, , ( );
( UPDATE
DELETE) 1 ( ).
, PostgreSQL VACUUM (
3.1.1).
7.2 VACUUM .
7.2, VACUUM
, SELECT,
INSERT, UPDATE DELETE .
VACUUM FULL.
1
12
2.2.

, , ,
, .
:
max_fsm_relations
,
.
VACUUM. max_fsm_relations

( ).
max_fsm_pages
,
,
.
, ,

VACUUM. ,
VACUUM VERBOSE ANALYZE
, . max_fsm_pages
,
.

FSM, VACUUM
, VACUUM FULL,
.
! 8.4 fsm ,
Free Space Map , .

temp_buffers
, .
16 .
max_prepared_transactions
(PREPARE TRANSACTION). 5.
vacuum_cost_delay
,
, ,
I/O VACUUM, .
,
vacuum_cost_delay 0.
13
2.2.
50 200 .
vacuum_cost_page_hit vacuum_cost_page_limit.
VACUUM, .
(Jan Wieck) ,
delay 200, page_hit 6 100
VACUUM 80%,
.
max_stack_depth
,
, .
, , .
24 MB.
max_files_per_process
,
.

, Too many
open files.

PostgreSQL :
( )
,
,
.

:
. ,
:
( checkpoint_segments,
3) ,
( checkpoint_timeout, ,
300).
,
, .
: checkpoint_segments
,
1 .
- .
1
.
checkpoint_warning ( ):
, .
14
2.2.

(checkpoint_segments).
( 16 ) .
,
,

.

12 256 ,
(warning) ,
, . ,
, (checkpoint_segments * 2 + 1) * 16 ,
, . ,
32, 1
.
,
,
.
fsync

off fsync.
,
. : ,
, !
,

.
.

commit_delay ( , 0 ) commit_siblings
(5 )

.
commit_siblings ,
commit_delay.

,
, .
,
.
wal_sync_method
15
2.2.
,
. fsync=off, .
:
open_datasync open()
O_DSYNC
fdatasync fdatasync() commit
fsync_writethrough fsync() commit,

fsync fsync() commit
open_sync open()
O_SYNC
.
, .
full_page_writes
off, fsync=off.
,
on, PostgreSQL

. ,
, .
,
.

,
.
full_page_writes
,
(
checkpoint_interval).
wal_buffers
SHARED MEMORY
1 . 256512 ,
. ,
14 2561024 .

. 3 ,
:
default_statistics_target
, ANALYZE (. 3.1.2).
1
16
2.2.
,
, .

ALTER TABLE . . . SET STATISTICS.
effective_cache_size
PostgreSQL
,
1 .
1,5 , shared_buffers
32 , effective_cache_size 800 .
700 , PostgreSQL ,

merge joins.
effective_cache_size 200 ,
,
.
effective_cache_size
2/3 ;
RAM
, .
random_page_cost
,
.

3.0, 2.5 2.0.

,
.
.
, ,
(sequential scans)
(index scans), .
,
, ,
.
.
random_page_cost 2.0; , random_page_cost ,
.
1
,

17
2.3.

PostgreSQL
,
.
, ,
.
, true/false:
track_counts .

, autovacuum .
,
( autovacuum).
track_functions
.
track_activities
.
. ,

,
, .
, ,
.
,
: ,
(
autovacuum ).
2.3
,
.
, , ,
.
PostgreSQL
, ,
.
,
,
.

, ,
, noatime1 .
1
18
2.3.

, .
(
, , :
),
.
, ,
,
,
.
:
(!).
pg_clog pg_xlog,
, .
.
.
,
, ,
,
, , .
CLUSTER
CLUSTER table [ USING index ]
,
.
,
.
,
.
:

, ,
. , CLUSTER
ACCESS EXCLUSIVE ,
( )
. PostgreSQL
,
CLUSTER .
19
2.4.
2.4

.
PostgreSQL
.
RAM ;
shared_buffers = 1/8 RAM ( 1/4);

work_mem 1/20 RAM;
maintenance_work_mem 1/4 RAM;
max_fsm_relations * 1.5;
max_fsm_pages max_fsm_relations * 2000;
fsync = true;
wal_sync_method = fdatasync;
commit_delay = 10 100 ;
commit_siblings = 5 10;
effective_cache_size = 0.9 cached,
free;
random_page_cost = 2 cpu, 4 ;
cpu_tuple_cost = 0.001 cpu, 0.01 ;
cpu_index_tuple_cost = 0.0005 cpu, 0.005
;
autovacuum = on;
autovacuum_vacuum_threshold = 1800;
autovacuum_analyze_threshold = 900;

(1), 2
maintenance_work_mem = 128MB
effective_cache_size = 512MB
work_mem = 640kB
wal_buffers = 1536kB
shared_buffers = 128MB
max_connections = 500
Web , 2
maintenance_work_mem = 128MB;
checkpoint_completion_target = 0.7
effective_cache_size = 1536MB
20
2.5. : pgtune
work_mem = 4MB
wal_buffers = 4MB
checkpoint_segments = 8
shared_buffers = 512MB
Web , 8
maintenance_work_mem = 512MB
checkpoint_completion_target = 0.7
effective_cache_size = 6GB
work_mem = 16MB
wal_buffers = 4MB
checkpoint_segments = 8
shared_buffers = 2GB
2.5
: pgtune
PostgreSQL Gregory Smith
pgtune1
.
Linux
. , .
:
2.1 Pgtune
pgtune - i $PGDATA/ p o s t g r e s q l . c o n f \
- - o $PGDATA/ p o s t g r e s q l . c o n f . pgtune
Line 1
- i , -- input-config postgresql.conf, -o,

-- output-config postgresql.conf.
.
-M, --memory ,
. , pgtune
.
-T, -- type . : DW, OLTP, Web,
Mixed, Desktop.
-c, -- connections .
,
.
1
http://pgtune.projects.postgresql.org/
21
2.6.
, pgtune
PostgreSQL.
, ,
,
.
2.6

:
1. ,
. :
a) .
.
b) , .
2. .
3. .
4. .

,
.
( cron)
.
ANALYZE

.
.
VACUUM ANALYZE.
, ,
,
ANALYZE.
.
REINDEX
REINDEX
. :
;
.
22
2.6.
. , ,
. PostgreSQL
,
. ,
.
- ,
REINDEX.
: REINDEX, VACUUM FULL,
, ,
.

,
.
,
, , .
. , ,
:
, ,
. , ,
.
.
,
.
,
, , ,
, .
EXPLAIN [ANALYZE]
EXPLAIN [] , PostgreSQL
. EXPLAIN ANALYZE []
1 ,
.
, .
:
(seq scan).

(nested loop).
1
EXPLAIN ANALYZE DELETE . . .
23
2.6.
EXPLAIN ANALYZE:
?
,
.
,
. , ,
, - ,
,
. 80%
,
.
EXPLAIN ANALYZE ,
. ,
SET enable_seqscan=false;
,
, ,
.
postgresql.conf! ,
!

.

:
pg_stat_user_tables

,
,
, ,
.
pg_stat_user_indexes
,
, ,
(
, ,
).
pg_statio_user_tables
,
, ,
(. 2.1.1),
24
2.6.
, ,
TOAST.
, :
(

).
.
, , , ,
PRIMARY KEY UNIQUE.
.
,
,
.
PostgreSQL

/ , ,
. , , foo foo_name,
foo_name = ,
.
CREATE INDEX foo_name_first_idx
ON foo ((lower(substr(foo_name, 1, 1))));

SELECT * FROM foo
WHERE lower(substr(foo_name, 1, 1)) = ;
.
(partial indexes)
WHERE. , ,
scheta uplocheno boolean. ,
uplocheno = false , uplocheno = true,
.
CREATE INDEX scheta_neuplocheno ON scheta (id)
WHERE NOT uplocheno;

SELECT * FROM scheta WHERE NOT uplocheno AND ...;
, ,
WHERE, .
25
2.6.

PostrgeSQL
, PostgreSQL
, .

, , 1
. ,
,
;
.
,
: ,
.

,
,
. ,

, .
SELECT count(*) FROM < >
count() :
, ,

.

(
!)
, ,
,
.

2.2 SQL
Line 1
SELECT count ( * ) FROM f o o ;
foo,
.
, , .
:
1
RULE PostgreSQL SQL, ,

,
26
2.6.
1. , 1 ,
,
ANALYZE:
2.3 SQL
Line 1
SELECT r e l t u p l e s FROM p g _ c l a s s WHERE relname = f o o ;
2. ,
, ,
.
,
. ,

.
3. ,

(cron).
DISTINCT
DISTINCT .
GROUP BY DISTINCT. GROUP BY
, ,
DISTINCT ( 8.4 ).
2.4 DISTINCT
Line 1
5
p o s t g r e s=# s e l e c t count ( * ) from ( s e l e c t d i s t i n c t i from g )

a;
count
-- ----19125
( 1 row )
Time : 5 8 0 , 5 5 3 ms
10
-
p o s t g r e s=# s e l e c t count ( * ) from ( s e l e c t d i s t i n c t i from g )

a;
count
-- ----19125
( 1 row )
15
-
Time : 3 6 , 2 8 1 ms
1
10000 ,
50000 !
27
2.6.
2.5 GROUP BY
Line 1
5
p o s t g r e s=# s e l e c t count ( * ) from ( s e l e c t i from g group by

i) a;
count
-- ----19125
( 1 row )
Time : 2 6 , 5 6 2 ms
10
-
p o s t g r e s=# s e l e c t count ( * ) from ( s e l e c t i from g group by

i) a;
count
-- ----19125
( 1 row )
15
-
Time : 2 5 , 2 7 0 ms

pgFouine
pgFouine1 log- PostgreSQL,
log- PostgreSQL. pgFouine
,
. pgFouine PHP
-
,

GNU General Public License. ,
log- .
pgFouine PostgreSQL
log-:
syslog
2.6 pgFouine
Line 1
-
log_destination = syslog
redirect_stderr = off
silent_mode = on
, n :
1
http://pgfouine.projects.pgfoundry.org/
28
2.6.
2.7 pgFouine
log_min_duration_statement = n
- log_duration = o f f
- l o g_ s t a te m e n t = none
Line 1
log_min_duration_statement 0. ,
-1.
pgFouine .
HTML- :
2.8 pgFouine
Line 1
p g f o u i n e . php - f i l e your / l o g / f i l e . l o g > your - r e p o r t . html
10
:
2.9 pgFouine
Line 1
p g f o u i n e . php - f i l e your / l o g / f i l e . l o g - top 10 - format t e x t
,
,
pgfouine.projects.pgfoundry.org.
pgBadger
pgBadger1 , pgFouine,
Perl. ,
(
pgFouine 24.02.2010, pgBadger
12.10.2012). pgBadger :
2.10 pgBadger
t a r x z f pgbadger - 2 . x . t a r . gz
cd pgbadger - 2 . x/
- p e r l M a k e f i l e . PL
- make && sudo make i n s t a l l
Line 1
-
pgFouine PostgreSQL :
2.11 PostgreSQL
l o g g i n g _ c o l l e c t o r = on
log_min_messages = debug1
- log_min_error_statement = debug1
- log_min_duration_statement = 0
5 l o g _ l i n e _ p r e f i x = %t [%p ] : [% l - 1 ] u s e r=%u , db=%d
Line 1
-
http://dalibo.github.com/pgbadger/
29
2.6.
10
l o g _ c h e c k p o i n t s = on
l o g _ c o n n e c t i o n s = on
l o g _ d i s c o n n e c t i o n s = on
l o g _ l o c k _ w a i t s = on
log_temp_files = 0
PostgreSQL pgBadger:
2.12 pgBadger
Line 1
5
-
$ . / pgbadger
~/ p g s q l / master / pg_log / p o s t g r e s q l
[========================>] Parsed
10485768 ( 1 0 0 . 0 0 % )
[========================>] Parsed
10485828 ( 1 0 0 . 0 0 % )
[========================>] Parsed
10485851 ( 1 0 0 . 0 0 % )
[========================>] Parsed
10485848 ( 1 0 0 . 0 0 % )
[========================>] Parsed
10485839 ( 1 0 0 . 0 0 % )
[========================>] Parsed
(100.00%)
- 2 0 1 2 - 0 8 - 3 0 _132*
10485768 b y t e s o f
10485828 b y t e s o f
10485851 b y t e s o f
10485848 b y t e s o f
10485839 b y t e s o f
982536 b y t e s o f 982536
HTML ,
PostgreSQL.
dalibo.github.com/pgbadger.
pg_stat_statements
pg_stat_statements
.
, PostgreSQL,
pgFouine pgBadger. :
2.13 pg_stat_statements postgresql.conf
Line 1
-
s h a r e d _ p r e l o a d _ l i b r a r i e s = pg_stat_statements
c u s t o m _ v a r i a b l e _ c l a s s e s = pg_stat_statements #
PostgreSQL 9 . 1
pg_stat_statements . max = 10000

pg_stat_statements . t r a c k = a l l
PostgreSQL .
pg_stat_statements:
1. pg_stat_statements.max (integer) sql
, (
)
30
2.6.
2. pg_stat_statements.track (enum) SQL
.
: top (
/), all ( , )
none ( ).
3. pg_stat_statements.save (boolean)
PostgreSQL.
.
:
2.14 pg_stat_statements
Line 1
# CREATE EXTENSION pg_stat_statements ;
:
2.15 pg_stat_statements
Line 1
5
10
15
20
-
# SELECT query , c a l l s , t o t a l _ t i m e , rows , 1 0 0 . 0 *

shared_blks_hit /
n u l l i f ( s h a r e d _ b l k s _ h i t + shared_blks_read ,
0 ) AS h i t _ p e r c e n t
FROM pg_stat_statements ORDER BY t o t a l _ t i m e DESC
LIMIT 1 0 ;
- [ RECORD 1
] -- --------------------------------------------------------------------query
| SELECT query , c a l l s , t o t a l _ t i m e , rows , ? *
shared_blks_hit /
|
n u l l i f ( shared_blks_hit +
shared_blks_read , ? ) AS h i t _ p e r c e n t
|
FROM pg_stat_statements ORDER BY
t o t a l _ t i m e DESC LIMIT ? ;
calls
| 3
total_time | 0.994
rows
| 7
hit_percent | 100.0000000000000000
- [ RECORD 2
] -- --------------------------------------------------------------------query
| insert into x ( i ) select generate_series (? ,?) ;
calls
| 2
total_time | 0.591
rows
| 110
hit_percent | 100.0000000000000000
- [ RECORD 3
] -- --------------------------------------------------------------------query
| s e l e c t * from x where i = ? ;
calls
| 2
total_time | 0.157
rows
| 6
31
2.7.
25
-
hit_percent | 100.0000000000000000
- [ RECORD 4
] -- --------------------------------------------------------------------query
| SELECT p g _ s t a t _ s t a t e m e n t s _ r e s e t ( ) ;
calls
| 1
total_time | 0.102
rows
| 1
hit_percent |
pg_stat_statements_reset:
2.16
# SELECT p g _ s t a t _ s t a t e m e n t s _ r e s e t ( ) ;
- [ RECORD 1 ] - - - - - - - - - - - - + - pg_stat_statements_reset |
Line 1
5
10
-
# SELECT query , c a l l s , t o t a l _ t i m e , rows , 1 0 0 . 0 *

shared_blks_hit /
n u l l i f ( s h a r e d _ b l k s _ h i t + shared_blks_read ,
0 ) AS h i t _ p e r c e n t
FROM pg_stat_statements ORDER BY t o t a l _ t i m e DESC
LIMIT 1 0 ;
- [ RECORD 1 ] - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - query
| SELECT p g _ s t a t _ s t a t e m e n t s _ r e s e t ( ) ;
calls
| 1
total_time | 0.175
rows
| 1
hit_percent |
, PostgreSQL
9.2 contrib SQL . 9.1 SQL
, select * from table where id =
3 select * from table where id = 21 ,
.
2.7
, PostgreSQL .

,
.
.
32
- ,

.
, ,

.

3.1
(partitioning, )
(, ) .
, .
(

).
(.. ,
).
( 5. . . 10 ) 40. . . 50%
, ( ),
, .
,
( )
, (
). , ,
.
SELECT * FROM articles ORDER BY id DESC LIMIT 10
,
.
, :
33
3.2.
(, ,
)
.

(DROP TABLE ,
DELETE).

.
3.2
PostgreSQL
:
(range)

, ,
. , .
(list)
.
,
:
,
. .
,
.
,
.
,
. ,
. :
3.1
Line 1
-
CHECK ( o u t l e t I D BETWEEN 100 AND 200 )

CHECK ( o u t l e t I D BETWEEN 200 AND 300 )
,
200.
(
), .
,

.
34
3.3.
, constraint_exclusion
postgresql.conf.
,
.
3.3
. ,
,
.
. , , ()
().
, 3 .
, . ,
, , ,
(
). .
.
, :
3.2
Line 1
5
-
CREATE TABLE my_logs (

id
SERIAL PRIMARY KEY,
user_id
INT NOT NULL,
logdate
TIMESTAMP NOT NULL,
data
TEXT,
some_state
INT
);
,
.
.
my_logs,
. ():
3.3
Line 1
5
-
CREATE TABLE my_logs2010m10

CHECK ( l o g d a t e >= DATE
2010 -11 -01 )
) INHERITS ( my_logs ) ;
2010 -12 -01 )
(
2 0 1 0 - 1 0 - 0 1 AND l o g d a t e < DATE
(
2 0 1 0 - 1 1 - 0 1 AND l o g d a t e < DATE
35
3.3.
10
-

2011 -01 -01 )
2010 -02 -01 )
(
2 0 1 0 - 1 2 - 0 1 AND l o g d a t e < DATE
(
2 0 1 1 - 0 1 - 0 1 AND l o g d a t e < DATE
my_logs2010m10,
my_logs2010m11 ..,
( ). CHECK
, (
, !).
logdate,
:
3.4
CREATE INDEX my_logs2010m10_logdate
( logdate ) ;
- CREATE INDEX my_logs2010m11_logdate
( logdate ) ;
( logdate ) ;
( logdate ) ;
Line 1
ON my_logs2010m10
ON my_logs2010m11
ON my_logs2010m12
ON my_logs2011m01
,
.
3.5
Line 1
5
10
15
-
CREATE OR REPLACE FUNCTION m y _ l o g s _ i n s e r t _ t r i g g e r ( )

RETURNS TRIGGER AS $$
BEGIN
IF ( NEW. l o g d a t e >= DATE 2 0 1 0 - 1 0 - 0 1 AND
NEW. l o g d a t e < DATE 2 0 1 0 - 1 1 - 0 1 ) THEN
INSERT INTO my_logs2010m10 VALUES (NEW. * ) ;
ELSIF ( NEW. l o g d a t e >= DATE 2 0 1 0 - 1 1 - 0 1 AND
ELSE
36
3.3.
20
-
RAISE EXCEPTION Date out o f r a n g e .

my_logs_insert_trigger ( ) function ! ;
END IF ;
RETURN NULL;
END;
$$
LANGUAGE p l p g s q l ;
Fix t h e
: logdate,
.
.
:
3.6
CREATE TRIGGER i n s e r t _ m y _ l o g s _ t r i g g e r
BEFORE INSERT ON my_logs
FOR EACH ROW EXECUTE PROCEDURE m y _ l o g s _ i n s e r t _ t r i g g e r ( ) ;
Line 1
-
my_logs:
3.7
INSERT INTO my_logs ( user_id , l o g d a t e , data , some_state )
VALUES( 1 , 2 0 1 0 - 1 0 - 3 0 , 3 0 . 1 0 . 2 0 1 0 data , 1 ) ;
- INSERT INTO my_logs ( user_id , l o g d a t e , data , some_state )
VALUES( 2 , 2 0 1 0 - 1 1 - 1 0 , 1 0 . 1 1 . 2 0 1 0 data2 , 1 ) ;
- INSERT INTO my_logs ( user_id , l o g d a t e , data , some_state )
VALUES( 1 , 2 0 1 0 - 1 2 - 1 5 , 1 5 . 1 2 . 2 0 1 0 data3 , 1 ) ;
Line 1
:
3.8
Line 1
-
p a r t i t i o n i n g _ t e s t=# SELECT * FROM ONLY my_logs ;

i d | u s e r _ i d | l o g d a t e | data | some_state
- - - -+ - - - - - - - - -+ - - - - - - - - -+ - - - - - -+ - - - - - - - - - - - ( 0 rows )
.
:
3.9
Line 1
-
p a r t i t i o n i n g _ t e s t=# SELECT * FROM my_logs ;

id | user_id |
logdate
|
data
|
some_state
- - - -+ - - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - -
37
3.3.
-
1 |
1 | 2 0 1 0 - 1 0 - 3 0 0 0 : 0 0 : 0 0 | 3 0 . 1 0 . 2 0 1 0 data
1
5
2 |
3 |
2 | 2 0 1 0 - 1 1 - 1 0 0 0 : 0 0 : 0 0 | 1 0 . 1 1 . 2 0 1 0 data2 |
1
1 | 2 0 1 0 - 1 2 - 1 5 0 0 : 0 0 : 0 0 | 1 5 . 1 2 . 2 0 1 0 data3 |
1
( 3 rows )
.
:
3.10
Line 1
5
p a r t i t i o n i n g _ t e s t=# S e l e c t * from my_logs2010m10 ;

id | user_id |
logdate
|
data
|
some_state
- - - -+ - - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - 1 |
1 | 2 0 1 0 - 1 0 - 3 0 0 0 : 0 0 : 0 0 | 3 0 . 1 0 . 2 0 1 0 data |
1
( 1 row )
10
-
p a r t i t i o n i n g _ t e s t=# S e l e c t * from my_logs2010m11 ;

id | user_id |
logdate
|
data
|
some_state
- - - -+ - - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - 2 |
2 | 2 0 1 0 - 1 1 - 1 0 0 0 : 0 0 : 0 0 | 1 0 . 1 1 . 2 0 1 0 data2 |
1
( 1 row )
! .
my_logs :
3.11
Line 1
5
p a r t i t i o n i n g _ t e s t=# SELECT * FROM my_logs WHERE u s e r _ i d = 2 ;

id | user_id |
logdate
|
data
|
some_state
- - - -+ - - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - 2 |
2 | 2 0 1 0 - 1 1 - 1 0 0 0 : 0 0 : 0 0 | 1 0 . 1 1 . 2 0 1 0 data2 |
1
( 1 row )
10
-
p a r t i t i o n i n g _ t e s t=# SELECT * FROM my_logs WHERE data LIKE

%0.1% ;
id | user_id |
logdate
|
data
|
some_state
- - - -+ - - - - - - - - -+ - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - 1 |
1 | 2 0 1 0 - 1 0 - 3 0 0 0 : 0 0 : 0 0 | 3 0 . 1 0 . 2 0 1 0 data |
1
2 |
2 | 2 0 1 0 - 1 1 - 1 0 0 0 : 0 0 : 0 0 | 1 0 . 1 1 . 2 0 1 0 data2 |
1
38
3.3.
-
( 2 rows )

.

. ,
2008 , 10 .
:
3.12
Line 1
DROP TABLE my_logs2008m10 ;
DROP TABLE ,
DELETE.
, ,
, ,
:
3.13
Line 1
ALTER TABLE my_logs2008m10 NO INHERIT my_logs ;
,
.
constraint_exclusion
constraint_exclusion ,
.
, :
3.14 constraint_exclusion OFF
Line 1
-
p a r t i t i o n i n g _ t e s t=# SET c o n s t r a i n t _ e x c l u s i o n = o f f ;
p a r t i t i o n i n g _ t e s t=# EXPLAIN SELECT * FROM my_logs WHERE
logdate > 2010 -12 -01 ;
5
10
QUERY PLAN
-- ------------------------------------------------------------------------R e s u l t ( c o s t = 6 . 8 1 . . 1 0 4 . 6 6 rows =1650 width =52)
-> Append ( c o s t = 6 . 8 1 . . 1 0 4 . 6 6 rows =1650 width =52)
-> Bitmap Heap Scan on my_logs ( c o s t = 6 . 8 1 . . 2 0 . 9 3
rows =330 width =52)
Recheck Cond : ( l o g d a t e > 2 0 1 0 - 1 2 - 0 1
0 0 : 0 0 : 0 0 : : timestamp w i t h o u t time zone )
-> Bitmap Index Scan on my_logs_logdate
( c o s t = 0 . 0 0 . . 6 . 7 3 rows =330 width =0)
39
3.3.
-
15
-
20
-
25
-
Index Cond : ( l o g d a t e > 2 0 1 0 - 1 2 - 0 1

-> Bitmap Heap Scan on my_logs2010m10 my_logs
( c o s t = 6 . 8 1 . . 2 0 . 9 3 rows =330 width =52)
-> Bitmap Index Scan on
my_logs2010m10_logdate ( c o s t = 0 . 0 0 . . 6 . 7 3 rows =330
width =0)
( c o s t = 6 . 8 1 . . 2 0 . 9 3 rows =330 width =52)
width =0)
( c o s t = 6 . 8 1 . . 2 0 . 9 3 rows =330 width =52)
width =0)
( c o s t = 6 . 8 1 . . 2 0 . 9 3 rows =330 width =52)
width =0)
( 2 2 rows )
EXPLAIN,
, ,
logdate > 2010-12-01 ,
, .
constraint_exclusion:
3.15 constraint_exclusion ON
Line 1
p a r t i t i o n i n g _ t e s t=# SET c o n s t r a i n t _ e x c l u s i o n = on ;
40
3.4.
5
10
-
15
-
SET
p a r t i t i o n i n g _ t e s t=# EXPLAIN SELECT * FROM my_logs WHERE
logdate > 2010 -12 -01 ;
QUERY PLAN
-- ------------------------------------------------------------------------R e s u l t ( c o s t = 6 . 8 1 . . 4 1 . 8 7 rows =660 width =52)
-> Append ( c o s t = 6 . 8 1 . . 4 1 . 8 7 rows =660 width =52)
-> Bitmap Heap Scan on my_logs ( c o s t = 6 . 8 1 . . 2 0 . 9 3
-> Bitmap Index Scan on my_logs_logdate
( c o s t = 0 . 0 0 . . 6 . 7 3 rows =330 width =0)
( c o s t = 6 . 8 1 . . 2 0 . 9 3 rows =330 width =52)
width =0)
( 1 0 rows )
, ,
, .
constraint_exclusion , ,
CHECK ,
, .
8.4 PostgreSQL constraint_exclusion on,
off partition.
( )
constraint_exclusion on, off, partition,
CHECK .
3.4

.
,
. ,
, (
) 50%
.
41
,
.
,
.

4.1
(.
replication)
(,
). ,

. , ,
.
.
, ,

. ,
.

(, ).

,
,
- ( , ,
).

,
. ,
, ,
(
,
).

42
4.1.

,
.
,
,

. ,
(
). ,
, , , ,
.
(, ,
).
,
, (
) .
, ,
.
, ,

X, B ,
, Y X.
Y, Y (, -
), B Y
, , ,
. ,
Y , X .

. ,
, ,
(
) .

: ,

. , ,
.

, .

,
.
,
,
,
.
,
43
4.2. Streaming Replication ( )

.
PostgreSQL ,
, .
(, ).
:
Slony-I Master-Slave ,
(cascading) (failover).
Slony-I
PostgreSQL INSERT/
DELETE/UPDATE
.
PGCluster Multi-Master .
, .
pgpool-I/II PostgreSQL (
II ). :
( ,
stand-by );
online-;
pooling ;
;

SELECT-
postgresql-;

.
Bucardo ,
Multi-Master Master-Slave ,
.
Londiste Master-Slave .
Skytools1 . , Slony-I.
Mammoth Replicator Multi-Master .
Postgres-R Multi-Master .
RubyRep Ruby, Multi-Master
, PostgreSQL MySQL.
, , ,
PostgreSQL.
4.2 Streaming Replication ( )
(Streaming Replication, SR)

wall xlog
1
http://pgfoundry.org/projects/skytools/
44

.
PostgreSQL 9 ( !).
, , ,

, PostgreSQL.
:
PostgreSQL

-
:

PostgreSQL
(
, ,
,
, Hot Standby)
PostgreSQL 9 .
9.0.1 . ,
, Linux.
masterdb(192.168.0.10)
slavedb(192.168.0.20).

ssh. postgres . ,
:
4.1 userssh
$sudo groupadd u s e r s s h
$sudo u s e r a d d -m - g u s e r s s h - d /home/ u s e r s s h - s / b i n / bash \
- - c " user ssh allow " userssh
Line 1
-
(
postgres):
45

4.2 postgres
Line 1
su p o s t g r e s
RSA-
:
4.3 RSA-
Line 1
5
-
p o s t g r e s @ l o c a l h o s t ~ $ ssh - keygen - t r s a -P ""

G e n e r a t i n g p u b l i c / p r i v a t e r s a key p a i r .
Enter f i l e i n which t o s a v e t h e key
( / var / l i b / p o s t g r e s q l / . s s h / i d _ r s a ) :
Created d i r e c t o r y / var / l i b / p o s t g r e s q l / . s s h .
Your i d e n t i f i c a t i o n has been saved i n
/ var / l i b / p o s t g r e s q l / . s s h / i d _ r s a .
Your p u b l i c key has been saved i n
/ var / l i b / p o s t g r e s q l / . s s h / i d _ r s a . pub .
The key f i n g e r p r i n t i s :
1 6 : 0 8 : 2 7 : 9 7 : 2 1 : 3 9 : b5 : 7 b : 8 6 : e1 : 4 6 : 9 7 : b f : 1 2 : 3 d : 7 6
postgres@localhost
:
4.4
Line 1
c a t $HOME/ . s s h / i d _ r s a . pub >> $HOME/ . s s h / a u t h o r i z e d _ k e y s
,
:
4.5 ssh
Line 1
ssh l o c a l h o s t
sshd:
4.6 sshd
Line 1
/ e t c / i n i t . d/ ss hd s t a r t
$HOME/.ssh
slavedb.
ssh.
pg_hba.conf ,
(trust) ( replication):
4.7 pg_hba.conf
Line 1
host
replication
all
192.168.0.20/32
trust
192.168.0.10/32
trust
4.8 pg_hba.conf
Line 1
host
replication
all
postgresql .
46

masterdb. postgresql.conf
:
4.9
# To e n a b l e read - o n l y q u e r i e s on a standby s e r v e r ,
w a l _ l e v e l must be s e t t o
- # " hot_standby " . But you can c h o o s e " a r c h i v e " i f you n e v e r
connect to the
- # s e r v e r i n standby mode .
- w a l _ l e v e l = hot_standby
Line 1
# S e t t h e maximum number o f c o n c u r r e n t c o n n e c t i o n s from t h e

standby s e r v e r s .
- max_wal_senders = 5
-
10
15
-
# To p r e v e n t t h e primary s e r v e r from removing t h e WAL

segments r e q u i r e d f o r
# t h e standby s e r v e r b e f o r e s h i p p i n g them , s e t t h e minimum
number o f segments
# r e t a i n e d i n t h e pg_xlog d i r e c t o r y . At l e a s t
wal_keep_segments s h o u l d be
# l a r g e r than t h e number o f segments g e n e r a t e d between t h e
beginning of
# o n l i n e - backup and t h e s t a r t u p o f s t r e a m i n g r e p l i c a t i o n .
I f you e n a b l e WAL
# a r c h i v i n g t o an a r c h i v e d i r e c t o r y a c c e s s i b l e from t h e
standby , t h i s may
# not be n e c e s s a r y .
wal_keep_segments = 32
20
-
# Enable WAL a r c h i v i n g on t h e primary t o an a r c h i v e

d i r e c t o r y a c c e s s i b l e from
# t h e standby . I f wal_keep_segments i s a h i g h enough number
t o r e t a i n t h e WAL
# segments r e q u i r e d f o r t h e standby s e r v e r , t h i s may not be
necessary .
archive_mode
= on
archive_command = cp %p / path_to / a r c h i v e/% f
:
wal_level = hot_standby WAL
archive, ,
( archive,
).
max_wal_senders = 5 .
47

wal_keep_segments = 32 c
WAL pg_xlog .
archive_mode = on WAL
archive_command .
/path_to/archive/.
. 9.1
synchronous_standby_names,
. application_name,
recovery.conf:
4.10 recovery.conf
restore_command = cp /tmp/% f %p
# e . g . cp
/mnt/ s e r v e r / a r c h i v e d i r /% f %p
- standby_mode = on
- primary_conninfo = h o s t=l o c a l h o s t p o r t =59121
u s e r=r e p l i c a t i o n password=r e p l i c a t i o n
application_name=n e w c l u s t e r
# e.g.
h o s t=l o c a l h o s t p o r t =5432
- t r i g g e r _ f i l e = /tmp/ t r i g _ f _ n e w c l u s t e r
Line 1
PostgreSQL .
slavedb.

slavedb
masterdb. .
masterdb . :
4.11
Line 1
p s q l - c "SELECT pg_start_backup ( l a b e l , t r u e ) "
.
:
4.12
Line 1
-
r s y n c -C - a - - d e l e t e - e s s h - - e x c l u d e p o s t g r e s q l . c o n f
- - exclude postmaster . pid \
- - e x c l u d e p o s t m a s t e r . o p t s - - e x c l u d e pg_log - - e x c l u d e
pg_xlog \
- - e x c l u d e r e c o v e r y . c o n f master_db_datadir /
s l a v e d b _ h o s t : slave_db_datadir /
master_db_datadir postgresql
masterdb
48

slave_db_datadir postgresql slavedb
slavedb_host slavedb( - 192.168.1.20)
,
. :
4.13
Line 1
p s q l - c "SELECT pg_stop_backup ( ) "
postgresql.conf,
( ).
:
4.14
Line 1
hot_standby = on
! wal_level = archive,
(hot_standby = off).
slavedb PostgreSQL
recovery.conf :
4.15 recovery.conf
# S p e c i f i e s whether t o s t a r t t h e s e r v e r a s a standby . In
streaming r e p l i c a t i o n ,
- # t h i s parameter must t o be s e t t o on .
- standby_mode
= on
Line 1
# S p e c i f i e s a c o n n e c t i o n s t r i n g which i s used f o r t h e
standby s e r v e r t o c o n n e c t
- # with t h e primary .
- primary_conninfo
= h o s t = 1 9 2 . 1 6 8 . 0 . 1 0 p o r t =5432
u s e r=p o s t g r e s
# S p e c i f i e s a t r i g g e r f i l e whose p r e s e n c e s h o u l d c a u s e
streaming r e p l i c a t i o n to
10 # end ( i . e . , f a i l o v e r ) .
- t r i g g e r _ f i l e = / path_to / t r i g g e r
-
15
-
# S p e c i f i e s a command t o l o a d a r c h i v e segments from t h e WAL

archive . I f
# wal_keep_segments i s a h i g h enough number t o r e t a i n t h e
WAL segments
# r e q u i r e d f o r t h e standby s e r v e r , t h i s may not be
n e c e s s a r y . But
# a l a r g e workload can c a u s e segments t o be r e c y c l e d b e f o r e
t h e standby
# i s f u l l y s y n c h r o n i z e d , r e q u i r i n g you t o s t a r t a g a i n from
a new b a s e backup .
49

-
restore_command = s c p masterdb_host : / path_to / a r c h i v e/% f

"%p"
standby_mode=on
primary_conninfo
trigger_file -,
.
restore_command ,
WAL .
scp masterdb
(masterdb_host - masterdb).
PostgreSQL slavedb.

:
4.16
Line 1
5
$ p s q l - c "SELECT p g _ c u r r e n t _ x l o g _ l o c a t i o n ( ) "
- h192 . 1 6 8 . 0 . 1 0 ( masterdb )
pg_current_xlog_location
-- -----------------------0/2000000
( 1 row )
10
-
$ psql - c " s e l e c t pg_last_xlog_receive_location () "

- h192 . 1 6 8 . 0 . 2 0 ( s l a v e d b )
pg_last_xlog_receive_location
-- ----------------------------0/2000000
( 1 row )
15
-
$ psql - c " s e l e c t pg_last_xlog_replay_location () "

- h192 . 1 6 8 . 0 . 2 0 ( s l a v e d b )
pg_last_xlog_replay_location
-- ---------------------------0/2000000
( 1 row )
9.1 view
. master slaves:
4.17
Line 1
# SELECT * from p g _ s t a t _ r e p l i c a t i o n ;
50

-
procpid | usesysid |
usename
| application_name |
client_addr | client_hostname | client_port |
backend_start
|
state
| sent_location |
write_location | flush_location | replay_location |
sync_priority | sync_state
- - - - - - - - -+ - - - - - - - - - -+ - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - -+ - - - 17135 |
16671 | r e p l i c a t i o n | n e w c l u s t e r
|
127.0.0.1
|
|
43745 | 2 0 1 1 - 0 5 - 2 2
1 8 : 1 3 : 0 4 . 1 9 2 8 3 + 0 2 | s t r e a m i n g | 1/30008750
|
1/30008750
| 1/30008750
| 1/30008750
|
1 | sync
9.1 view pg_stat_database_conflicts,

:
4.18
Line 1
-
# SELECT * from p g _ s t a t _ d a t a b a s e _ c o n f l i c t s ;
d a t i d | datname | c o n f l _ t a b l e s p a c e | c o n f l _ l o c k |
confl_snapshot | confl_bufferpin | confl_deadlock
5
-
- - - - - - -+ - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - -+ - - - - - - - - - - - - - - - -+ - - 1 | template1 |
0 |
0 |
0 |
0 |
0
11979 | t e m p l a t e 0 |
0 |
0 |
0 |
0 |
0
11987 | p o s t g r e s |
0 |
0 |
0 |
0 |
0
16384 | marc
|
0 |
0 |
1 |
0 |
0
ps:
4.19
Line 1
-
[ masterdb ] $ ps - e f | g r e p s e n d e r
p o s t g r e s 6879 6831 0 1 0 : 3 1 ?
00:00:00 postgres :
wal s e n d e r p r o c e s s p o s t g r e s 1 2 7 . 0 . 0 . 1 ( 4 4 6 6 3 ) s t r e a m i n g
0/2000000
[ s l a v e d b ] $ ps - e f | g r e p r e c e i v e r
p o s t g r e s 6878 6872 1 1 0 : 3 1 ?
00:00:01 postgres :
wal r e c e i v e r p r o c e s s
s t r e a m i n g 0/2000000
. :
4.20
Line 1
$ p s q l test_db
51

5
-
test_db=# c r e a t e t a b l e t e s t 3 ( i d
key , name v a r c h a r ( 2 0 ) ) ;
NOTICE : CREATE TABLE / PRIMARY
i n d e x " test3_pkey " f o r t a b l e
CREATE TABLE
test_db=# i n s e r t i n t o t e s t 3 ( id ,
INSERT 0 1
test_db=#
i n t not n u l l primary
KEY w i l l c r e a t e i m p l i c i t
" test3 "
name ) v a l u e s ( 1 , t e s t 1 ) ;
:
4.21
Line 1
5
-
$ p s q l test_db
test_db=# s e l e c t * from t e s t 3 ;
i d | name
- - - -+ - - - - - - 1 | test1
( 1 row )
,
.

- (trigger_file) ,
.

- (trigger_file) . 9.1
pg_xlog_replay_pause() pg_xlog_replay_resume()
.

.
,
.

PostgreSQL .
52
4.3. Slony-I

, ,
. PostgreSQL
.
4.3 Slony-I
Slony ,
PostgreSQL .
Slony Postgre INSERT/
DELETE/UPDATE .
Slony
: slony
slonik. slonik-,
slon .
, slon , .
slonik-e
slonik stdin.

slonik-a ,
, ,
slonik syntax error,
. . .
Ubuntu :
4.22
Line 1
sudo a p t i t u d e i n s t a l l s l o n y 1 - b i n

customers
(
,
).

: customers
master_host: customers_master.com
slave_host_1: customers_slave.com
cluster name ( ): customers_rep
53
4.3. Slony-I
master-
Postgres,
Slony. , ,
slony.
4.23 master-
Line 1
-
pgsql@customers_master$ c r e a t e u s e r - a - d s l o n y
pgsql@customers_master$ p s q l - d t e m p l a t e 1 - c " a l t e r \
u s e r s l o n y with password slony_user_password ; "

slony, slon.
, ( slon)
.
slave-
,
Internet ( ),
PostgreSQL
-, . , :
4.24 slave-
Line 1
-
anyuser@customers_slave$ p s q l - d c u s t o m e r s \
- h customers_master . com -U s l o n y
- ( ,
). - ,
firewall-a, pg_hba.conf, $PGDATA.
slave- PostgreSQL.
, Postgres up and ready,
- ,
(
postmaster):
4.25 slave-
Line 1
5
-
p g s q l @ c u s t o m e r s _ s l a v e $ rm - r f $PGDATA
p g s q l @ c u s t o m e r s _ s l a v e $ mkdir $PGDATA
p g s q l @ c u s t o m e r s _ s l a v e $ i n i t d b -E UTF8 -D $PGDATA
pgsql@customers_slave$ c r e a t e u s e r - a -d slony
pgsql@customers_slave$ psql -d template1 - c " a l t e r \
u s e r s l o n y with password slony_user_password ; "
postmaster.
!

. !
54
4.3. Slony-I
4.26 slave-
Line 1
-
p g s q l @ c u s t o m e r s _ s l a v e $ c r e a t e u s e r - a - d customers_owner
pgsql@customers_slave$ psql -d template1 - c " a l t e r \
u s e r customers_owner with password
customers_owner_password ; "
customers_master,
-h customers_slave,
slave.
slave, master, Slony.
plpgsql slave
slony.

(slony_user_password). :
4.27 plpgsql slave
Line 1
-
slony@customers_master$ c r e a t e d b -O customers_owner \
- h c u s t o m e r s _ s l a v e . com c u s t o m e r s
slony@customers_master$ c r e a t e l a n g - d c u s t o m e r s \
- h c u s t o m e r s _ s l a v e . com p l p g s q l
! , replication set
primary key. -
, primary
key ALTER TABLE ADD PRIMARY KEY.
primary key ,
serial (ALTER TABLE ADD COLUMN),
. table add
key slonik-a.
. slave:
4.28 plpgsql slave
Line 1
-
slony@customers_master$ pg_dump - s c u s t o m e r s | \
p s q l -U s l o n y - h c u s t o m e r s _ s l a v e . com c u s t o m e r s
pg_dump -s .
pg_dump -s customers , psql
-U slony -h customers_slave.com customers
(slony_user_pass). : -
Slony ( make install),
sl_*, . , :

( 5)
:-) slave
:
55
4.3. Slony-I
4.29 plpgsql slave
Line 1
5
-
s l o n i k <<EOF
c l u s t e r name = c u s t o m e r s _ s l a v e ;
node Y admin c o n n i n f o = dbname=c u s t o m e r s
h o s t=customers_master . com
p o r t =5432 u s e r=s l o n y password=slony_user_pass ;
u n i n s t a l l node ( i d = Y) ;
echo okay ;
EOF
Y . . : , cluster
name - , T1 (default).
uninstall.
( ),
uninstall ( master slave).

PgSQL
, - ,
.
- :
4.30
Line 1
#!/ b i n / sh
CLUSTER=customers_rep
5
-
DBNAME1=c u s t o m e r s
HOST1=customers_master . com
- HOST2=c u s t o m e r s _ s l a v e . com
-
10
PORT1=5432
- PORT2=5432
-
SLONY_USER=s l o n y
15
20
-
s l o n i k <<EOF
c l u s t e r name = $CLUSTER ;
node 1 admin c o n n i n f o = dbname=$DBNAME1 h o s t=$HOST1
p o r t=$PORT1
u s e r=s l o n y password=slony_user_password ;
p o r t=$PORT2 u s e r=s l o n y password=slony_user_password ;
56
4.3. Slony-I
-
i n i t c l u s t e r ( i d = 1 , comment = Customers DB
replication cluster ) ;
25
echo C r e a t e s e t ;
c r e a t e s e t ( i d = 1 , o r i g i n = 1 , comment = Customers
- DB r e p l i c a t i o n s e t ) ;
30
echo Adding t a b l e s t o t h e s u b s c r i p t i o n s e t ;
echo Adding t a b l e p u b l i c . c u s t o m e r s _ s a l e s . . . ;
- s e t add t a b l e ( s e t i d = 1 , o r i g i n = 1 , i d = 4 , f u l l
qualified
- name = p u b l i c . c u s t o m e r s _ s a l e s , comment = Table
public . customers_sales ) ;
35 echo done ;
-
40
-
echo Adding t a b l e p u b l i c . customers_something . . . ;

s e t add t a b l e ( s e t i d = 1 , o r i g i n = 1 , i d = 5 , f u l l
qualified
name = p u b l i c . customers_something ,
comment = Table p u b l i c . customers_something ) ;
echo done ;
45
50
echo done adding ;

s t o r e node ( i d = 2 , comment = Node 2 , $HOST2 ) ;
echo s t o r e d node ;
s t o r e path ( s e r v e r = 1 , c l i e n t = 2 , c o n n i n f o =
dbname=$DBNAME1 h o s t=$HOST1
p o r t=$PORT1 u s e r=s l o n y password=slony_user_password ) ;
echo s t o r e d path ;
dbname=$DBNAME2 h o s t=$HOST2
p o r t=$PORT2 u s e r=s l o n y password=slony_user_password ) ;
store l i s t e n ( origin = 1 , provider = 1 , receiver = 2 ) ;

- EOF
-
, ,
. : ,
, id
, primary key.
: replication set .
set.
:
. unsubscribe subscribe .
57
4.3. Slony-I
slave- replication set
:
4.31 slave- replication set
Line 1
#!/ b i n / sh
- DBNAME2=c u s t o m e r s
5
-
- HOST2=c u s t o m e r s _ s l a v e . com
-
10
PORT1=5432
- PORT2=5432
-
15
20
-
s l o n i k <<EOF
echo s u b s c r i b i n g ;
s u b s c r i b e s e t ( id = 1 , provider = 1 , r e c e i v e r = 2 , forward
= no ) ;
25
-
EOF

, .
4.32
Line 1
-
slony@customers_master$ s l o n customers_rep \
"dbname=c u s t o m e r s u s e r=s l o n y "
4.33
Line 1
-
s l o n y @ c u s t o m e r s _ s l a v e $ s l o n customers_rep \
58
4.3. Slony-I
.
COPY, slave DB
.
slave-
10- . slon
, .

4.3.1 4.3.2.
id = 3. customers_slave3.com,
- PgSQL.
( 4.3.2) :
4.34
Line 1
5
-
s l o n i k <<EOF
c l u s t e r name = c u s t o m e r s _ s l a v e ;
node 3 admin c o n n i n f o = dbname=c u s t o m e r s
h o s t=c u s t o m e r s _ s l a v e 3 . com
p o r t =5432 u s e r=s l o n y password=slony_user_pass ;
u n i n s t a l l node ( i d = 3 ) ;
echo okay ;
EOF
, ,
.
.

:
4.35
Line 1
#!/ b i n / sh
5
-
- HOST3=c u s t o m e r s _ s l a v e 3 . com
-
10
-
PORT1=5432
PORT2=5432
15
59
4.3. Slony-I
20
-
s l o n i k <<EOF
p o r t=$PORT1 u s e r=s l o n y password=slony_user_pass ;
node 3 admin c o n n i n f o = dbname=$DBNAME3
h o s t=$HOST3 p o r t=$PORT2 u s e r=s l o n y
password=slony_user_pass ;
echo done adding ;
25
30
-
s t o r e node ( i d = 3 , comment = Node 3 , $HOST3 ) ;

echo s o r e d node ;
dbname=$DBNAME1
password=slony_user_pass ) ;
echo s t o r e d path ;
dbname=$DBNAME3
password=slony_user_pass ) ;
35
echo a g a i n ;
EOF
id 3, 2 .
3 replication set:
4.36
Line 1
#!/ b i n / sh
5
-
- HOST3=c u s t o m e r s _ s l a v e 3 . com
-
10
PORT1=5432
- PORT2=5432
-
15
-
s l o n i k <<EOF
60
4.3. Slony-I
- p o r t=$PORT1 u s e r=s l o n y password=slony_user_pass ;
20 node 3 admin c o n n i n f o = dbname=$DBNAME3 h o s t=$HOST3
- p o r t=$PORT2 u s e r=s l o n y password=slony_user_pass ;
-
echo s u b s c r i b i n g ;
s u b s c r i b e s e t ( id = 1 , provider = 1 , r e c e i v e r = 3 , forward
= no ) ;
25
-
EOF
slon , .
slon .
4.37
Line 1
-
s l o n y @ c u s t o m e r s _ s l a v e 3 $ s l o n customers_rep \

,
: ,
:
4.38
Line 1
5
10
15
-
%s l o n customers_rep "dbname=c u s t o m e r s u s e r=s l o n y _ u s e r "

CONFIG main : s l o n v e r s i o n 1 . 0 . 5 s t a r t i n g up
CONFIG main : l o c a l node i d = 3
CONFIG main : l o a d i n g c u r r e n t c l u s t e r c o n f i g u r a t i o n
CONFIG s t o r e N o d e : no_id=1 no_comment= CustomersDB
replication cluster
CONFIG s t o r e N o d e : no_id=2 no_comment= Node 2 ,
node2 . example . com
CONFIG s t o r e N o d e : no_id=4 no_comment= Node 4 ,
node4 . example . com
CONFIG s t o r e P a t h : pa_server=1 p a _ c l i e n t=3
pa_conninfo="dbname=c u s t o m e r s
h o s t=mainhost . com p o r t =5432 u s e r=s l o n y _ u s e r
password=slony_user_pass " pa_connretry=10
CONFIG s t o r e L i s t e n : l i _ o r i g i n =1 l i _ r e c e i v e r =3
l i _ p r o v i d e r =1
CONFIG s t o r e S e t : s e t _ i d=1 s e t _ o r i g i n =1
set_comment= CustomersDB r e p l i c a t i o n s e t
61
4.3. Slony-I
20
25
30
35
40
45
50
-
WARN remoteWorker_wakeup : node 1 - no worker t h r e a d

CONFIG s t o r e S u b s c r i b e : sub_set=1 sub_provider=1
sub_forward= f
CONFIG e n a b l e S u b s c r i p t i o n : sub_set=1
CONFIG main : c o n f i g u r a t i o n c o m p l e t e - s t a r t i n g t h r e a d s
CONFIG enableNode : no_id=1
ERROR remoteWorkerThread_1 : " b e g i n t r a n s a c t i o n ; s e t
transaction isolation level
s e r i a l i z a b l e ; l o c k t a b l e " _customers_rep " . s l _ c o n f i g _ l o c k ;
select
" _customers_rep " . e n a b l e S u b s c r i p t i o n ( 1 , 1 , 4 ) ;
n o t i f y " _customers_rep_Event " ; n o t i f y
" _customers_rep_Confirm " ;
i n s e r t i n t o " _customers_rep " . s l _ e v e n t ( e v _ o r i g i n , ev_seqno ,
ev_timestamp , ev_minxid , ev_maxxid , ev_xip ,
ev_type , ev_data1 , ev_data2 , ev_data3 , ev_data4 ) v a l u e s
( 1 , 219440 ,
2005 -05 -05 18:52:42.708351 , 52501283 , 52501292 ,
5 2 5 0 1 2 8 3 , ENABLE_SUBSCRIPTION ,
1 , 1 , 4 , f ) ; i n s e r t i n t o " _customers_rep " .
s l _ c o n f i r m ( c o n _ o r i g i n , con_received ,
con_seqno , con_timestamp ) v a l u e s ( 1 , 3 , 2 1 9 4 4 0 ,
CURRENT_TIMESTAMP) ; commit t r a n s a c t i o n ; "
PGRES_FATAL_ERROR ERROR: i n s e r t o r update on t a b l e
" s l _ s u b s c r i b e " v i o l a t e s f o r e i g n key
c o n s t r a i n t " s l _ s u b s c r i b e - sl_path - r e f "
DETAIL : Key ( sub_provider , s u b _ r e c e i v e r ) =(1 ,4)
i s not p r e s e n t i n t a b l e " sl_path " .
INFO remoteListenThread_1 : d i s c o n n e c t i n g from
dbname=c u s t o m e r s h o s t=mainhost . com
p o r t =5432 u s e r=s l o n y _ u s e r password=slony_user_pass
%
_< >.sl_path;,
_customers_rep.sl_path
. , id 4, (1,4)
sl_path .
, Slony.
.
,
( ,
(1,4)):
4.39
62
4.3. Slony-I
Line 1
-
slon y _ u s e r @ m a s t e r h o s t $ p s q l - d c u s t o m e r s - h
_every_one_of_slaves -U s l o n y
c u s t o m e r s=# i n s e r t i n t o _customers_rep . sl_path
v a l u e s ( 1 , 4 , dbname=c u s t o m e r s h o s t=mainhost . com
p o r t =5432 u s e r=s l o n y _ u s e r
password=slony_user_password , 10 ) ;
,
.
_< >,
_customers_rep.

master-, SELECT-
. pg_stat_activity :
4.40
Line 1
5
10
-
s e l e c t e v _ o r i g i n , ev_seqno , ev_timestamp , ev_minxid ,

ev_maxxid , ev_xip ,
ev_type , ev_data1 , ev_data2 , ev_data3 , ev_data4 , ev_data5 ,
ev_data6 ,
ev_data7 , ev_data8 from " _customers_rep " . s l _ e v e n t e where
( e . e v _ o r i g i n = 2 and e . ev_seqno > 336996 ) o r
( e . e v _ o r i g i n = 10 and e . ev_seqno > 1692318 )
o r d e r by e . e v _ o r i g i n , e . ev_seqno ;
_customers_rep ,
.
sl_event - ,
.
:
4.41
Line 1
-
d e l e t e from _customers_rep . s l _ e v e n t where

ev_timestamp<NOW( ) - 1 DAY : : i n t e r v a l ;
.
_customers_rep.sl_log_*
, -
63
4.4. Londiste
, _customers_rep.sl_log_1
.
4.4 Londiste
Londiste ,
python.
:
.
-
, Slony-I. Londiste
PgQ (
,

,
PostgreSQL). :
,
(failover)
(switchover) ( 3 )
, Linux,
Ubuntu Server. ,
( Windows) ,
PostgreSQL Windows, ,
.
Londiste Skytools,
. , Debian Ubuntu skytools
:
4.42
Line 1
sudo a p t i t u d e i n s t a l l s k y t o o l s

.
2.1.11. , :
64
4.4. Londiste
4.43
Line 1
5
10
15
-
$wget
h t t p : / / pgfoundry . o r g / f r s / download . php /2561/ s k y t o o l s - 2 . 1 . 1 1 . t a r . gz
$ t a r z x v f s k y t o o l s - 2 . 1 . 1 1 . t a r . gz
$cd s k y t o o l s - 2 . 1 . 1 1 /
# deb
$sudo a p t i t u d e i n s t a l l b u i l d - e s s e n t i a l a u t o c o n f \
automake a u t o t o o l s - dev dh - make \
debhelper devscripts fakeroot x u t i l s l i n t i a n pbuilder \
python - dev yada
# p o s t g r e s q l 8 . 4 . x
$sudo a p t i t u d e i n s t a l l p o s t g r e s q l - s e r v e r - dev - 8 . 4
# python - psycopg L o n d i s t e
$sudo a p t i t u d e i n s t a l l python - psycopg2
# deb
# p o s t g r e s q l 8 . 4 . x ( 8 . 3 . x "make deb83 " )
$sudo make deb84
$cd . . /
# s k y t o o l s
$dpkg - i s k y t o o l s - modules - 8 . 4 _2 . 1 . 1 1 _i386 . deb
s k y t o o l s _ 2 . 1 . 1 1 _i386 . deb
Skytools
4.44
./ configure
make
- make i n s t a l l
Line 1
-
,
4.45
$ l o n d i s t e . py -V
- Skytools version 2.1.11
- $pgqadm . py -V
- Skytools version 2.1.11
Line 1
,
.
:
host1 ;
host2 ;
65
4.4. Londiste
ticker-
Londiste ticker ,
. , ,
, .

ticker- (
/etc/skytools/db1-ticker.ini):
4.46 ticker-
[ pgqadm ]
- #
- job_name = db1 - t i c k e r
Line 1
5
-
#
db = dbname=P h o s t=h o s t 1
#
# ( . . )
10 maint_delay = 600
-
#
# ( )
- loop_delay = 0 . 1
-
15
# l o g p i d
l o g f i l e = / var / l o g /%(job_name ) s . l o g
- p i d f i l e = / var / p i d /%(job_name ) s . p i d
-
(SQL)
ticker .
pgqadm.py :
4.47 ticker-
Line 1
-
pgqadm . py / e t c / s k y t o o l s /db1 - t i c k e r . i n i i n s t a l l
pgqadm . py / e t c / s k y t o o l s /db1 - t i c k e r . i n i t i c k e r - d
, (/var/log/skytools/db1-tickers.log)
. (
).
ticker,
:
4.48 ticker-
Line 1
pgqadm . py / e t c / s k y t o o l s /db1 - t i c k e r . i n i t i c k e r - s
ticker:
4.49 ticker-
Line 1
pgqadm . py / e t c / s k y t o o l s /db1 - t i c k e r . i n i t i c k e r - k
66
4.4. Londiste

Londiste .
slave
, .

( /etc/skytools/db1-londiste.ini):
4.50
[ londiste ]
#
- job_name = db1 - l o n d i s t e
Line 1
-
#
- provider_db = dbname=db1 p o r t =5432 h o s t=h o s t 1
- #
- s u b s c r i b e r _ d b = dbname=db1 h o s t=h o s t 2
10
15
#
# SQL - , . .
# .
# ! ,
# pgq_queue_name = db1 - l o n d i s t e - queue
# l o g p i d
l o g f i l e = / var / l o g /%(job_name ) s . l o g
- p i d f i l e = / var / run/%(job_name ) s . p i d
-
20
#
- l o g _ s i z e = 5242880
- log_count = 3
-
Londiste
SQL
.
:
4.51 Londiste
Line 1
l o n d i s t e . py / e t c / s k y t o o l s /db1 - l o n d i s t e . i n i p r o v i d e r i n s t a l l
67
4.4. Londiste
4.52 Londiste
Line 1
l o n d i s t e . py / e t c / s k y t o o l s /db1 - l o n d i s t e . i n i s u b s c r i b e r
install
.
Londiste
:
4.53
Line 1
l o n d i s t e . py / e t c / s k y t o o l s /db1 - l o n d i s t e . i n i r e p l a y - d
, , ..
,
.
(/var/log/db1-londistes.log).

:
4.54
Line 1
l o n d i s t e . py / e t c / s k y t o o l s /db1 - l o n d i s t e . i n i p r o v i d e r add
-- all
:
4.55
Line 1
l o n d i s t e . py / e t c / s k y t o o l s /db1 - l o n d i s t e . i n i s u b s c r i b e r add
-- all
- all,
,
, .
(sequence)
. :
4.56
Line 1
l o n d i s t e . py / e t c / s k y t o o l s /db1 - l o n d i s t e . i n i p r o v i d e r add - s e q
-- all
:
4.57
Line 1
l o n d i s t e . py / e t c / s k y t o o l s /db1 - l o n d i s t e . i n i s u b s c r i b e r
add - s e q - - a l l
68
4.4. Londiste

all.
, .
Londiste
bulk copy , ( COPY)
,
.
:
4.58
Line 1
l e s s / var / l o g /db1 - l o n d i s t e . l o g
, .

, "ok".
4.59
Line 1
l o n d i s t e . py / e t c / s k y t o o l s /db1 - l o n d i s t e . i n i s u b s c r i b e r t a b l e s
5
10
Table S t a t e
public . table1
public . table2
public . table3
public . table4
public . table5
public . table6
...
ok
ok
in - copy
-

( ):
4.60
Line 1
5
-
(
while [ $ (
python l o n d i s t e . py / e t c / s k y t o o l s /db1 - l o n d i s t e . i n i
subscriber tables |
t a i l - n+2 | awk { p r i n t $2 } | g r e p - v ok | wc - l ) - ne 0 ] ;
do s l e e p 6 0 ; done ; echo | m a i l - s R e p l i c a t i o n done EOM
user@domain . com
) &

:
69
4.4. Londiste
4.61
Line 1
l o n d i s t e . py p r o v i d e r t a b l e s | x a r g s l o n d i s t e . py 

s u b s c r i b e r add

.
4.62
Line 1
-
SELECT queue_name , consumer_name , l a g , l a s t _ s e e n

FROM pgq . get_consumer_info ( ) ;
lag ,
last_seen .
, 60 .

Londiste
, . PGQ,
, API:
4.63
Line 1
SELECT pgq . u n r e g i s t e r _ c o n s u m e r ( queue_name ,

consumer_name ) ;
pgqadm.py:
4.64
Line 1
pgqadm . py < t i c k e r . i n i > u n r e g i s t e r queue_name consumer_name

:
1.
2.
3.
4.

BEGIN;

SELECT
londiste.provider_refresh_trigger(queue_name,
tablename);
5. COMMIT;
70
4.4. Londiste

1. BEGIN;
2.
3. SELECT
londiste.provider_refresh_trigger(queue_name,
tablename);
4. COMMIT;
5. lag, londiste
6.
, ,
.

Londiste lag
, ,
ticker. UPDATE
DELETE ,
. . .
,
pgq.subscription sub_last_tick sub_next_tick.
4.65
Line 1
5
-
SELECT count ( * )
FROM pgq . event_1 ,
(SELECT t i c k _ s n a p s h o t
FROM pgq . t i c k
WHERE t i c k _ i d BETWEEN 5715138 AND 5715139
) as t ( snapshots )
WHERE t x i d _ v i s i b l e _ i n _ s n a p s h o t ( ev_txid , s n a p s h o t s ) ;
, 5 400 .
.
Londiste, .
Londiste .
INI ticker- :
4.66
Line 1
pgq_lazy_fetch = 500
Londiste 500
. .
71
4.5. Bucardo
4.5 Bucardo
Bucardo master-master master-slave

PostgreSQL, Perl.
,
.
Ubuntu Server.
DBIx::Safe Perl .
4.67
Line 1
sudo a p t i t u d e i n s t a l l l i b d b i x - s a f e - p e r l
1 :
4.68
t a r x v f z DBIx - Safe - 1 . 2 . 5 . t a r . gz
- cd DBIx - Safe - 1 . 2 . 5
- p e r l M a k e f i l e . PL
- make && make t e s t && sudo make i n s t a l l
Line 1
Bucardo. 2 :
4.69
Line 1
5
t a r x v f z Bucardo - 4 . 4 . 0 . t a r . gz
cd Bucardo - 4 . 4 . 0
p e r l M a k e f i l e . PL
make
sudo make i n s t a l l
Bucardo pl/perlu
PostgreSQL.
4.70
Line 1
sudo a p t i t u d e i n s t a l l p o s t g r e s q l - p l p e r l - 8 . 4
Bucardo
:
1
2
http://search.cpan.org/CPAN/authors/id/T/TU/TURNSTEP/
http://bucardo.org/wiki/Bucardo#Obtaining_Bucardo
72
4.5. Bucardo
4.71 Bucardo
Line 1
bu ca r d o _ c t l i n s t a l l
Bucardo PostgreSQL,
:
4.72 Bucardo
This w i l l i n s t a l l t h e bucardo d a t a b a s e i n t o an e x i s t i n g
Postgres c l u s t e r .
- P o s t g r e s must have been c o m p i l e d with P e r l support ,
- and you must c o n n e c t a s a s u p e r u s e r
Line 1
5
-
We w i l l c r e a t e a new s u p e r u s e r named bucardo ,

and make i t t h e owner o f a new d a t a b a s e named bucardo
10
-
Current c o n n e c t i o n s e t t i n g s :
1 . Host :
<none>
2 . Port :
5432
3 . User :
postgres
4 . Database :
postgres
5 . PID d i r e c t o r y : / var / run / bucardo
,
Bucardo bucardo bucardo.
Unix socket,
pg_hda.conf.

,
Bucardo. master_db slave_db.
:
4.73
Line 1
-
bu ca r d o _ c t l add db master_db name=master

bu ca r d o _ c t l add a l l t a b l e s herd=a l l _ t a b l e s
bu ca r d o _ c t l add a l l s e q u e n c e s herd=a l l _ t a b l e s
master (
, master_db slave_db
Bucardo ).
,
all_tables.
slave_db:
4.74
Line 1
bu ca r d o _ c t l add db slave_db name=r e p l i c a p o r t =6543

h o s t=s l a v e _ h o s t
73
4.5. Bucardo
replica Bucardo.

. (master-slave):
4.75
Line 1
bu ca r d o _ c t l add sync d e l t a type=p u s h d e l t a s o u r c e=a l l _ t a b l e s

t a r g e t d b=r e p l i c a
Bucardo PostgreSQL.
:
type
. 3 :
Fullcopy. .
Pushdelta. Master-slave .
Swap.
Master-master .

Bucardo
.
goat
( )
standard_conflict (
):
* source source
(master_db ).
* target target
(slave_db ).
* skip .
.
* random ,
.
* latest ,
.
* abort .
source
.
targetdb
, .
master-master:
4.76
Line 1
bu ca r d o _ c t l add sync d e l t a type=swap s o u r c e=a l l _ t a b l e s

t a r g e t d b=r e p l i c a
74
4.6. RubyRep
/
:
4.77
Line 1
bu ca r d o _ c t l s t a r t
:
4.78
Line 1
bu ca r d o _ c t l s t o p

:
4.79
Line 1
bu ca r d o _ c t l show a l l

4.80
Line 1
bu ca r d o _ c t l s e t name=v a l u e
:
4.81
Line 1
bu ca r d o _ c t l s e t s y s l o g _ f a c i l i t y=LOG_LOCAL3

4.82
Line 1
bu ca r d o _ c t l r e l o a d _ c o n f i g
bucardo.org/wiki/Bucardo_ctl
4.6 RubyRep
RubyRep
, ruby. :
. master-master,
master-slave , PostgreSQL MySQL.
:
75
4.6. RubyRep

:
MySQL

RubyRep : Ruby
JRuby. JRuby
.
JRuby
Java ( 1.6).
1. JRuby rubyrep c Rubyforge.
2.
3.
Ruby
1. Ruby, Rubygems.
2. .
Mysql:
4.83
Line 1
sudo gem i n s t a l l mysql
PostgreSQL:
4.84
Line 1
sudo gem i n s t a l l p o s t g r e s
3. rubyrep:
4.85
Line 1
sudo gem i n s t a l l rubyrep

:
4.86
Line 1
rubyrep g e n e r a t e myrubyrep . c o n f
76
4.6. RubyRep
generate
myrubyrep.conf:
4.87
Line 1
5
-
RR : : I n i t i a l i z e r : : run do | c o n f i g |
config . l e f t = {
: a d a p t e r => p o s t g r e s q l , # o r mysql
: d a t a b a s e => SCOTT ,
: username => s c o t t ,
: password => t i g e r ,
: host
=> 1 7 2 . 1 6 . 1 . 1
}
10
15
-
config . right
: a d a p t e r =>
: d a t a b a s e =>
: username =>
: password =>
: host
=>
}
= {
postgresql ,
SCOTT ,
scott ,
tiger ,
172.16.1.2
c o n f i g . i n c l u d e _ t a b l e s dept
- c o n f i g . i n c l u d e _ t a b l e s /^ e / # r e g e x p matches a l l t a b l e s
s t a r t i n g with e
20 # c o n f i g . i n c l u d e _ t a b l e s / . / # r e g e x p matches a l l t a b l e s
- end
-
. left
right. config.include_tables
( RegEx).

:
4.88
Line 1
rubyrep s c a n - c myrubyrep . c o n f
:
4.89
Line 1
-
dept 100% . . . . . . . . . . . . . . . . . . . . . . . . .
emp 100% . . . . . . . . . . . . . . . . . . . . . . . . .
0
1
dept , emp
.
77
4.6. RubyRep

:
4.90
Line 1
rubyrep sync - c myrubyrep . c o n f

:
4.91
Line 1
rubyrep sync - c myrubyrep . c o n f dept /^ e /

.

.
:
4.92
Line 1
rubyrep r e p l i c a t e - c myrubyrep . c o n f
(
) .

, .
, rubyrep.
, .
:
4.93
Line 1
rubyrep u n i n s t a l l - c myrubyrep . c o n f

rubyrep Ruby :
4.94
Line 1
5
-
$rubyrep r e p l i c a t e - c myrubyrep . c o n f
V e r i f y i n g RubyRep t a b l e s
Checking f o r and removing rubyrep t r i g g e r s from
unconfigured tables
V e r i f y i n g rubyrep t r i g g e r s o f c o n f i g u r e d t a b l e s
Starting replication
E x c e p t i o n caught : Thread#j o i n : d e a d l o c k 0 xb76ee1ac - mutual
j o i n (0 xb758cfac )
78
4.7.
Ruby. :
1. rubyrep JRuby ( )
2. rubyrep :
4.95
Line 1
-- -
/ L i b r a r y /Ruby/Gems / 1 . 8 / gems/ rubyrep - 1 . 1 . 2 / l i b / rubyrep /

r e p l i c a t i o n _ r u n n e r . rb 2 0 1 0 - 0 7 - 1 6 1 5 : 1 7 : 1 6 . 0 0 0 0 0 0 0 0 0
-0400
- +++ . / r e p l i c a t i o n _ r u n n e r . rb 2 0 1 0 - 0 7 - 1 6
1 7 : 3 8 : 0 3 . 0 0 0 0 0 0 0 0 0 -0400
- @@ - 2 , 6 +2 ,12 @@
-
5
10
-
require optparse
r e q u i r e thread
+r e q u i r e monitor
+
+c l a s s Monitor
+ a l i a s l o c k mon_enter
+ a l i a s u n l o c k mon_exit
+end
15
20
25
-
module RR
# This c l a s s implements t h e f u n c t i o n a l i t y o f t h e
r e p l i c a t e command .
@@ - 9 4 , 7 +100 ,7 @@
# I n i t i a l i z e s t h e w a i t e r t h r e a d used f o r
r e p l i c a t i o n pauses
# and p r o c e s s i n g
# t h e p r o c e s s TERM s i g n a l .
def init_waiter
- @termination_mutex = Mutex . new
+ @termination_mutex = Monitor . new
@termination_mutex . l o c k
@waiter_thread | | = Thread . new
{ @termination_mutex . l o c k ;
s e l f . termination_requested = true }
%w(TERM INT) . each do | s i g n a l |
4.7
,
PostgreSQL.
,
, ..
79
4.7.
PostgreSQL.
.

, 9.0 PostgreSQL. Slony-I
,
,
, (failover)
(switchover). Londiste
, . Bucardo
master-master, master-slave ,
,
(failover) (switchover). RubyRep,
master-master , ,

(
).
80
,
.

5.1
.

.
.
, .
.
,
(, 90%
). , (
!) ,
().
,
, , ..
.
,
, .
..
. ? .
,
. ,
10 , 10 -
.. .. - ,
,
.
, ID
(user_id)
81
5.2. PL/Proxy
. ..
:
,
user_id

:
-
=. , ,
.
,
.
=
()
. , ,
( ,
).

.
,
.. .
, ,
,
(, ,
).
. , ,
, ID , 100 .

.
PostgreSQL
:
Postgres-XC
PL/Proxy
HadoopDB (Shared-nothing clustering)
Greenplum Database
5.2 PL/Proxy
PL/Proxy -
.
,
, ,
82
5.2. PL/Proxy
(,
, , -
).
PL/Proxy ?
.

, ,
26 . ,
-, :
, , -
.

.
PL/Proxy
OLTP . failover-
, -,
.
:
autocommit-

SELECT;

-
-;

-, PgBouncer
( , )
-
1. PL/Proxy1 .
2. PL/Proxy make make install.
PL/Proxy .
Ubuntu Server PostgreSQL
8.4:
5.1
Line 1
sudo a p t i t u d e i n s t a l l p o s t g r e s q l - 8 . 4 - p l p r o x y
1
http://pgfoundry.org/projects/plproxy
83
5.2. PL/Proxy
3 PostgreSQL. 2
node1 node2, ,
proxy. pl/proxy
.
plproxytest, users. !
node1 node2.
.
plproxytest( ):
5.2
CREATE DATABASE p l p r o x y t e s t
WITH OWNER = p o s t g r e s
ENCODING = UTF8 ;
Line 1
-
users:
5.3
Line 1
5
-
CREATE TABLE p u b l i c . u s e r s
(
username c h a r a c t e r v a r y i n g ( 2 5 5 ) ,
email character varying (255)
)
WITH (OIDS=FALSE) ;
ALTER TABLE p u b l i c . u s e r s OWNER TO p o s t g r e s ;
users:
5.4
Line 1
5
-
CREATE OR REPLACE FUNCTION p u b l i c . i n s e r t _ u s e r ( i_username

text ,
i_emailaddress
text )
RETURNS i n t e g e r AS
$BODY$
INSERT INTO p u b l i c . u s e r s ( username , e m a i l ) VALUES ( $1 , $2 ) ;
SELECT 1 ;
$BODY$
LANGUAGE s q l VOLATILE ;
ALTER FUNCTION p u b l i c . i n s e r t _ u s e r ( t e x t , t e x t ) OWNER TO
postgres ;
. proxy.
, (proxy)
:
5.5
Line 1
CREATE DATABASE p l p r o x y t e s t
84
5.2. PL/Proxy
-
WITH OWNER = p o s t g r e s
ENCODING = UTF8 ;

pl/proxy:
5.6
Line 1
5
10
CREATE OR REPLACE FUNCTION p u b l i c . p l p r o x y _ c a l l _ h a n d l e r ( )

RETURNS l a n g u a g e _ h a n d l e r AS
$ l i b d i r / plproxy , plproxy_call_handler
LANGUAGE c VOLATILE
COST 1 ;
ALTER FUNCTION p u b l i c . p l p r o x y _ c a l l _ h a n d l e r ( )
OWNER TO p o s t g r e s ;
- - language
CREATE LANGUAGE p l p r o x y HANDLER p l p r o x y _ c a l l _ h a n d l e r ;
CREATE LANGUAGE p l p g s q l ;
, ,
3 , pl/proxy
. .
key-value:
5.7
Line 1
5
10
15
-
CREATE OR REPLACE FUNCTION p u b l i c . g e t _ c l u s t e r _ c o n f i g

( IN cluster_name t e x t ,
OUT " key " t e x t , OUT v a l t e x t )
RETURNS SETOF r e c o r d AS
$BODY$
BEGIN
- - l e t s u s e same c o n f i g f o r a l l c l u s t e r s
key := c o n n e c t i o n _ l i f e t i m e ;
v a l := 3 0 * 6 0 ; - - 30m
RETURN NEXT;
RETURN;
END;
$BODY$
LANGUAGE p l p g s q l VOLATILE
COST 100
ROWS 1 0 0 0 ;
ALTER FUNCTION p u b l i c . g e t _ c l u s t e r _ c o n f i g ( t e x t )
, .
DSN :
5.8
CREATE OR REPLACE FUNCTION
- p u b l i c . g e t _ c l u s t e r _ p a r t i t i o n s ( cluster_name t e x t )
Line 1
85
5.2. PL/Proxy
5
10
15
-
RETURNS SETOF t e x t AS
$BODY$
BEGIN
IF cluster_name = u s e r c l u s t e r THEN
RETURN NEXT dbname=p l p r o x y t e s t h o s t=node1
u s e r=p o s t g r e s ;
RETURN NEXT dbname=p l p r o x y t e s t h o s t=node2
u s e r=p o s t g r e s ;
RETURN;
END IF ;
RAISE EXCEPTION Unknown c l u s t e r ;
END;
$BODY$
COST 100
ROWS 1 0 0 0 ;
ALTER FUNCTION p u b l i c . g e t _ c l u s t e r _ p a r t i t i o n s ( t e x t )
:
5.9
Line 1
5
10
15

p u b l i c . g e t _ c l u s t e r _ v e r s i o n ( cluster_name t e x t )
$BODY$
BEGIN
IF cluster_name = u s e r c l u s t e r THEN
RETURN 1 ;
END IF ;
RAISE EXCEPTION Unknown c l u s t e r ;
END;
$BODY$
COST 1 0 0 ;
ALTER FUNCTION p u b l i c . g e t _ c l u s t e r _ v e r s i o n ( t e x t )
,
:
5.10
Line 1
5
-

p u b l i c . i n s e r t _ u s e r ( i_username t e x t , i _ e m a i l a d d r e s s t e x t )
$BODY$
CLUSTER u s e r c l u s t e r ;
RUN ON h a s h t e x t ( i_username ) ;
86
5.2. PL/Proxy
$BODY$
LANGUAGE p l p r o x y VOLATILE
COST 1 0 0 ;
10 ALTER FUNCTION p u b l i c . i n s e r t _ u s e r ( t e x t , t e x t )
- OWNER TO p o s t g r e s ;
-
. proxy :
5.11
SELECT i n s e r t _ u s e r ( Sven , sven@somewhere . com ) ;
- SELECT i n s e r t _ u s e r ( Marko , marko@somewhere . com ) ;
- SELECT i n s e r t _ u s e r ( S t e v e , steve@somewhere . com ) ;
Line 1
.
:
5.12
Line 1
5
10
-

p u b l i c . get_user_email ( i_username t e x t )
RETURNS SETOF t e x t AS
$BODY$
CLUSTER u s e r c l u s t e r ;
RUN ON h a s h t e x t ( i_username ) ;
SELECT e m a i l FROM p u b l i c . u s e r s
WHERE username = i_username ;
$BODY$
LANGUAGE p l p r o x y VOLATILE
COST 100
ROWS 1 0 0 0 ;
ALTER FUNCTION p u b l i c . get_user_email ( t e x t )
:
5.13
Line 1
s e l e c t p l p r o x y . get_user_email ( S t e v e ) ;
,
, users .
?
pl/proxy
. ,
. 16 .
- .
?
Highload++ 2008,

87
5.3. Postgres-XC
Skype,
opensource.
,
,
.
,
,
get_cluster_partitions.
5.3 Postgres-XC
Postgres-XC - ,

. Postgres-XC
,
:
,
GTM
(Global Transaction Manager).
,
Postgres-XC 64%, .. 10

6.4 , (
).

PostgreSQL,

,
( PostgreSQL API).
(GTM),
, ,
(
, ).
Postgres-XC MultiMaster,
.
, -

( ).
. 5.1 Postgres-XC
:
88
5.3. Postgres-XC
. 5.1: Postgres-XC
1. (GTM)
Postgres-XC,

(
).
, .
,
.
2. (coordinators)
().

( ). ,
(datanodes)
SQL PostgreSQL .
, ,
.
, ,
..
3. (datanodes)
89
5.3. Postgres-XC
.
Datanodes
.
Postgres-XC
. Ubuntu 12.10 postgres-xc :
5.14 Postgres-XC
Line 1
sudo apt - g e t i n s t a l l p o s t g r e s - xc p o s t g r e s - xc - c l i e n t
p o s t g r e s - xc - c o n t r i b p o s t g r e s - xc - s e r v e r - dev

Postgres-XC :
. 5.2:
1. (distributed tables, . 5.2):

(hash, round-robin, modulo).

.
.
.
2. (replicated tables, . 5.3):
()
.
( )
90
5.3. Postgres-XC
. 5.3:
.
,
,
.

.

Postgres-XC
PostgreSQL.
( 5432).
.
5.15
CREATE TABLE
- users_with_hash ( i d SERIAL , type INT , . . . )
- DISTRIBUTE by HASH( i d ) TO NODE dn1 , dn2 ;
Line 1
CREATE TABLE
users_with_modulo ( i d SERIAL , type INT , . . . )
- DISTRIBUTE by MODULO( i d ) TO NODE dn1 , dn2 ;
5
-
CREATE TABLE
10 u s e r s _ w i t h _ r r o b i n ( i d SERIAL , type INT , . . . )
- DISTRIBUTE by ROUND ROBIN TO NODE dn1 , dn2 ;
-
5.15 3 :
1. users_with_hash
( id)
. 15 :
91
5.3. Postgres-XC
5.16
#
- $ psql
- # SELECT id , type from users_with_hash ORDER BY i d ;
id
| type
5 - - - - - - -+ - - - - - - 1 |
946
2 |
153
3 |
484
4 |
422
10
5 |
785
6 |
906
7 |
973
8 |
699
9 |
434
15
10 |
986
11 |
135
12 | 1012
13 |
395
14 |
667
20
15 |
324
Line 1
#
- $ p s q l - p15432
25
i d | type
- - - - - - -+ - - - - - - 1 |
946
2 |
153
5 |
785
30
6 |
906
8 |
699
9 |
434
12 | 1012
13 |
395
35
15 |
324
-
#
- $ p s q l - p15433
40
id
| type
- - - - - - - -+ - - - - - - 3 |
484
4 |
422
7 |
973
45
10 |
986
11 |
135
-
92
5.3. Postgres-XC
-
14 |
667
2. users_with_modulo
( id)
. 15 :
5.17
#
$ psql
- # SELECT id , type from users_with_modulo ORDER BY i d ;
id
| type
5 - - - - - - -+ - - - - - - 1 |
883
2 |
719
3 |
29
4 |
638
10
5 |
363
6 |
946
7 |
440
8 |
331
9 |
884
15
10 |
199
11 |
78
12 |
791
13 |
345
14 |
476
20
15 |
860
Line 1
-
#
$ p s q l - p15432
25
id
| type
- - - - - - - -+ - - - - - - 2 |
719
4 |
638
6 |
946
30
8 |
331
10 |
199
12 |
791
14 |
476
-
#
$ p s q l - p15433
i d | type
- - - - - - -+ - - - - - - 40
1 |
883
3 |
29
35
-
93
5.3. Postgres-XC
45
-
5
7
9
11
13
15
|
|
|
|
|
|
363
440
884
78
345
860
3.
users_with_rrobin
(round-robin) .
15 :
5.18
#
$ psql
- # SELECT id , type from u s e r s _ w i t h _ r r o b i n ORDER BY i d ;
id
| type
5 - - - - - - -+ - - - - - - 1 |
890
2 |
198
3 |
815
4 |
446
10
5 |
61
6 |
337
7 |
948
8 |
446
9 |
796
15
10 |
422
11 |
242
12 |
795
13 |
314
14 |
240
20
15 |
733
Line 1
-
#
$ p s q l - p15432
- # SELECT id , type from u s e r s _ w i t h _ r r o b i n ORDER BY i d ;
25
id
| type
- - - - - - - -+ - - - - - - 2 |
198
4 |
446
6 |
337
30
8 |
446
10 |
422
12 |
795
14 |
240
-
35
-
#
$ p s q l - p15433
94
5.3. Postgres-XC
40
45
-
# SELECT id , type from u s e r s _ w i t h _ r r o b i n ORDER BY i d ;

i d | type
- - - - - -+ - - - - - - 1 |
890
3 |
815
5 |
61
7 |
948
9 |
796
11 |
242
13 |
314
15 |
733
:
5.19
CREATE TABLE
u s e r s _ r e p l i c a t e d ( i d SERIAL , type INT , . . . )
- DISTRIBUTE by REPLICATION TO NODE dn1 , dn2 ;
Line 1
-
:
5.20
Line 1
5
10
15
-
# SELECT id , type from u s e r s _ r e p l i c a t e d

id
| type
- - - - - - -+ - - - - - - 1 |
75
2 |
262
3 |
458
4 |
779
5 |
357
6 |
51
7 |
249
8 |
444
9 |
890
10 |
810
11 |
809
12 |
166
13 |
605
14 |
401
15 |
58
ORDER BY i d ;
.
:
5.21
# EXPLAIN VERBOSE SELECT * from users_with_modulo ORDER BY
id ;
QUERY PLAN
Line 1
95
5.3. Postgres-XC
5
10
-
-- ------------------------------------------------------------------------S o r t ( c o s t = 4 9 . 8 3 . . 5 2 . 3 3 rows =1000 width =8)

Output : id , type
S o r t Key : users_with_modulo . i d
-> R e s u l t ( c o s t = 0 . 0 0 . . 0 . 0 0 rows =1000 width =8)
Output : id , type
-> Data Node Scan on users_with_modulo
( c o s t = 0 . 0 0 . . 0 . 0 0 rows =1000 width =8)
Output : id , type
Node/ s : dn1 , dn2
Remote query : SELECT id , type FROM ONLY
users_with_modulo WHERE t r u e
( 9 rows )
5.21
, .
:
5.22
Line 1
# EXPLAIN VERBOSE SELECT sum ( i d ) from users_with_modulo

GROUP BY type ;
5
10
-
QUERY PLAN
-- ------------------------------------------------------------------------HashAggregate ( c o s t = 5 . 0 0 . . 5 . 0 1 rows=1 width =8)
Output : pg_catalog . sum ( ( sum ( users_with_modulo . i d ) ) ) ,
users_with_modulo . type
-> M a t e r i a l i z e ( c o s t = 0 . 0 0 . . 0 . 0 0 rows=0 width =0)
Output : ( sum ( users_with_modulo . i d ) ) ,
-> Data Node Scan on "__REMOTE_GROUP_QUERY__"
( c o s t = 0 . 0 0 . . 0 . 0 0 rows =1000 width =8)
Output : sum ( users_with_modulo . i d ) ,
Node/ s : dn1 , dn2
Remote query : SELECT sum ( group_1 . i d ) ,
group_1 . type FROM (SELECT id , type FROM ONLY
users_with_modulo WHERE t r u e ) group_1 GROUP BY 2
( 8 rows )
JOIN , JOIN

. JOIN

( 5.23).
5.23
96
5.3. Postgres-XC
Line 1
5
10
15
# EXPLAIN VERBOSE SELECT * from users_with_modulo ,

users_with_hash WHERE users_with_modulo . i d =
users_with_hash . i d ;
QUERY PLAN
-- ------------------------------------------------------------------------Nested Loop ( c o s t = 0 . 0 0 . . 0 . 0 1 rows=1 width =16)
Output : users_with_modulo . id , users_with_modulo . type ,
users_with_hash . id , users_with_hash . type
J o i n F i l t e r : ( users_with_modulo . i d = users_with_hash . i d )
-> Data Node Scan on users_with_modulo
( c o s t = 0 . 0 0 . . 0 . 0 0 rows =1000 width =8)
Output : users_with_modulo . id ,
Node/ s : dn1 , dn2
users_with_modulo WHERE t r u e
-> Data Node Scan on users_with_hash ( c o s t = 0 . 0 0 . . 0 . 0 0
Output : users_with_hash . id , users_with_hash . type
Node/ s : dn1 , dn2
users_with_hash WHERE t r u e
( 1 1 rows )
:
5.24
Line 1
5
-
# EXPLAIN VERBOSE SELECT * from u s e r s _ r e p l i c a t e d ;

QUERY PLAN
-- ------------------------------------------------------------------------Data Node Scan on "__REMOTE_FQS_QUERY__" ( c o s t = 0 . 0 0 . . 0 . 0 0
rows=0 width =0)
Output : u s e r s _ r e p l i c a t e d . id , u s e r s _ r e p l i c a t e d . type
Node/ s : dn1
Remote query : SELECT id , type FROM u s e r s _ r e p l i c a t e d
( 4 rows )

, ( ).
(HA)
Postgres-XC .
CAP1 .

,
1
http://en.wikipedia.org/wiki/CAP_theorem
97
5.4. HadoopDB
.
Postgres-XC
PostgreSQL (streaming)
hot-standby .

,
. GTM
, GTM-standby
.
.
1. Postgres-XC PostgreSQL 9.1 (9.2 )

2.
( )
3. UNIQUE
4. foreign keys

5. ( )
6. INSERT . . . RETURNING ( )
7.
( )
Postgres-XC
PostgreSQL. ,
(
) ,
PostgreSQL.
5.4 HadoopDB
Hadoop ,
.

, :
:
Hadoop
,
;
:
, .
;
98
5.4. HadoopDB
:
,
;
:
, .

;
:
, Java,
,
JVM.
HDFS

Hadoop Distributed File System.
,
:
,
, ;

;

;
;
:
;
:
,
;
HDFS
Namenode
.
.

, .
.
Datanode

.

. Rack,
, , .
99
5.4. HadoopDB
. 5.4: HDFS

,
. .
HDFS
:
.

( , ;
64 mb) , Datanode.

,
. ,
- ,
/trash

.
, .
- ,
Datanode Namenode
.

Namenode ,
-
100
5.4. HadoopDB
.
,

.

, TCP/IP.
Namenode ClientProtocol,
DatanodeProtocol,
Remote Procedure Call (RPC).
,
DFSShell,
DFSAdmin, , -.
API : Java
API, C pipeline, WebDAV .
MapReduce
, Hadoop framework
,
. Job ()
, , :
Map

( -)
-.
.
Reduce
map
.

,
.
, ,
( ).
JobTracker,
TaskTracker.
JobTracker
,
, .
, framework,
map reduce,
.
JobTracker
.

, , ,
.
101
5.4. HadoopDB
HBase
Hadoop ,
.
Google
BigTable.
HBase
.
, .
HQL ( Hadoop
Query Language),
SQL.
.
HBase
,
--,
.
,
, ,
.
- ,
. ,
.
HQL , SQL,

help;,
. SELECT, INSERT, UPDATE, DROP ,
.
HBase Shell,
HBase API
: Java, Jython, REST Thrift.
HadoopDB
HadoopDB Yale Brown

, MapReduce,
. MapReduce
,
,
. SQL,
MapReduce,
. MapReduce
,
.
102
5.4. HadoopDB

Ubuntu Server .
Hadoop
, Hadoop,
,
:
ssh
,
[hadoop]:
5.25
$sudo groupadd hadoop
- $sudo u s e r a d d -m - g hadoop - d /home/ hadoop - s / b i n / bash
\
- - c "Hadoop s o f t w a r e owner " hadoop
Line 1
:
5.26 hadoop
Line 1
su hadoop
RSA-
:
5.27 RSA-
Line 1
5
-
h a d o o p @ l o c a l h o s t ~ $ ssh - keygen - t r s a -P ""

G e n e r a t i n g p u b l i c / p r i v a t e r s a key p a i r .
Enter f i l e i n which t o s a v e t h e key
( / home/ hadoop / . s s h / i d _ r s a ) :
Your i d e n t i f i c a t i o n has been saved i n
/home/ hadoop / . s s h / i d _ r s a .
Your p u b l i c key has been saved i n
/home/ hadoop / . s s h / i d _ r s a . pub .
The key f i n g e r p r i n t i s :
7b : 5 c : c f : 7 9 : 6 b : 9 3 : d6 : d6 : 8 d : 4 1 : e3 : a6 : 9 d : 0 4 : f 9 : 8 5
hadoop@localhost
:
5.28
Line 1
c a t $HOME/ . s s h / i d _ r s a . pub >> $HOME/ . s s h / a u t h o r i z e d _ k e y s
,
:
103
5.4. HadoopDB
5.29 ssh
Line 1
ssh l o c a l h o s t
sshd:
5.30 sshd
Line 1
/ e t c / i n i t . d/ ss hd s t a r t
JVM
1.5.0 .
5.31 JVM
Line 1
sudo a p t i t u d e i n s t a l l openjdk - 6 - j d k
Hadoop:
5.32 Hadoop
Line 1
5
-
cd / opt
sudo wget h t t p : / /www. g t l i b . g a t e c h . edu /pub/ apache / hadoop
/ c o r e / hadoop - 0 . 2 0 . 2 / hadoop - 0 . 2 0 . 2 . t a r . gz
sudo t a r z x v f hadoop - 0 . 2 0 . 2 . t a r . gz
sudo l n - s / opt / hadoop - 0 . 2 0 . 2 / opt / hadoop
sudo chown -R hadoop : hadoop / opt / hadoop / opt / hadoop - 0 . 2 0 . 2
sudo mkdir - p / opt / hadoop - data /tmp - b a s e
sudo chown -R hadoop : hadoop / opt / hadoop - data /
/opt/hadoop/conf/hadoop-env.sh
:
5.33
Line 1
5
-
export
export
export
export
export
export
JAVA_HOME=/u s r / l i b /jvm/ java - 6 - openjdk

HADOOP_HOME=/opt / hadoop
HADOOP_CONF=$HADOOP_HOME/ c o n f
HADOOP_PATH=$HADOOP_HOME/ b i n
HIVE_HOME=/opt / h i v e
HIVE_PATH=$HIVE_HOME/ b i n
e x p o r t PATH=$HIVE_PATH:$HADOOP_PATH:$PATH
/opt/hadoop/conf/hadoop-site.xml:
5.34 hadoop
Line 1
5
<c o n f i g u r a t i o n>

<name>hadoop . tmp . d i r</name>
<v a l u e>/ opt / hadoop - data /tmp - b a s e</ v a l u e>
<d e s c r i p t i o n>A b a s e f o r o t h e r temporary
d i r e c t o r i e s</ d e s c r i p t i o n>
104
5.4. HadoopDB
-

10
-

<name> f s . d e f a u l t . name</name>
<v a l u e>l o c a l h o s t : 5 4 3 1 1</ v a l u e>
<d e s c r i p t i o n>
The name o f t h e d e f a u l t f i l e system .
</ d e s c r i p t i o n>

15
20
-

<name>hadoopdb . c o n f i g . f i l e </name>
<v a l u e>HadoopDB . xml</ v a l u e>
<d e s c r i p t i o n>The name o f t h e HadoopDB
c l u s t e r c o n f i g u r a t i o n f i l e </ d e s c r i p t i o n>

</ c o n f i g u r a t i o n>
/opt/hadoop/conf/mapred-site.xml:
5.35 mapreduce
Line 1
5
10

<name>mapred . j o b . t r a c k e r</name>
<v a l u e>l o c a l h o s t : 5 4 3 1 0</ v a l u e>
The h o s t and p o r t t h a t t h e
MapReduce j o b t r a c k e r r u n s a t .

/opt/hadoop/conf/hdfs-site.xml:
5.36 hdfs
Line 1
5
-

<name>d f s . r e p l i c a t i o n</name>
<v a l u e>1</ v a l u e>
Default block r e p l i c a t i o n .

Namenode:
5.37 Namenode
Line 1
$ hadoop namenode - format
105
5.4. HadoopDB
5
10
15
20
25
-
10/05/07 1 4 : 2 4 : 1 2 INFO namenode . NameNode : STARTUP_MSG:

/* ***********************************************************
STARTUP_MSG: S t a r t i n g NameNode
STARTUP_MSG:
h o s t = hadoop1 / 1 2 7 . 0 . 1 . 1
STARTUP_MSG:
a r g s = [ - format ]
STARTUP_MSG:
version = 0.20.2
STARTUP_MSG:
b u i l d = h t t p s : / / svn . apache . o r g / r e p o s
/ a s f / hadoop /common/ b r a n c h e s / branch - 0 . 2 0 - r
9 1 1 7 0 7 ; c o m p i l e d by c h r i s d o on F r i Feb 19 0 8 : 0 7 : 3 4 UTC
2010
*********************************************************** */
10/05/07 1 4 : 2 4 : 1 2 INFO namenode . FSNamesystem :
fsOwner=hadoop , hadoop
s u p e r g r o u p=s u p e r g r o u p
i s P e r m i s s i o n E n a b l e d=t r u e
10/05/07 1 4 : 2 4 : 1 2 INFO common . S t o r a g e :
Image f i l e o f s i z e 96 saved i n 0 s e c o n d s .
10/05/07 1 4 : 2 4 : 1 2 INFO common . S t o r a g e :
S t o r a g e d i r e c t o r y / opt / hadoop - data /tmp - b a s e / d f s /name has
been
s u c c e s s f u l l y formatted .
10/05/07 1 4 : 2 4 : 1 2 INFO namenode . NameNode :
SHUTDOWN_MSG:
/* ***********************************************************
SHUTDOWN_MSG: S h u t t i n g down NameNode a t hadoop1 / 1 2 7 . 0 . 1 . 1
*********************************************************** */
. Hadoop:
5.38 Hadoop
Line 1
5
10
-
$ s t a r t - a l l . sh
s t a r t i n g namenode , l o g g i n g t o / opt / hadoop / b i n / . .
/ l o g s / hadoop - hadoop - namenode - hadoop1 . out
l o c a l h o s t : s t a r t i n g datanode , l o g g i n g t o
/ opt / hadoop / b i n / . . / l o g s / hadoop - hadoop - datanode - hadoop1 . out
l o c a l h o s t : s t a r t i n g secondarynamenode , l o g g i n g t o
/ opt / hadoop / b i n / . . / l o g s / hadoop - hadoop - secondarynamenode - hadoop1 . out
s t a r t i n g jobtracker , l o g g i n g to
/ opt / hadoop / b i n / . . / l o g s / hadoop - hadoop - j o b t r a c k e r - hadoop1 . out
l o c a l h o s t : s t a r t i n g tasktracker , l o g g i n g to
/ opt / hadoop / b i n / . . / l o g s / hadoop - hadoop - t a s k t r a c k e r - hadoop1 . out
Hadoop stop-all.sh.
106
5.4. HadoopDB
HadoopDB Hive
HaddopDB1
$HADOOP_HOME/lib:
hadoopdb.jar
5.39 HadoopDB
Line 1
$cp hadoopdb . j a r $HADOOP_HOME/ l i b
PostgreSQL JDBC . 2
$HADOOP_HOME/lib.
Hive HadoopDB SQL .
HDFS Hive:
5.40 HadoopDB
hadoop
hadoop
- hadoop
- hadoop
Line 1
-
fs
fs
fs
fs
- mkdir
- mkdir
- chmod
- chmod
/tmp
/ u s e r / h i v e / warehouse
g+w /tmp
g+w / u s e r / h i v e / warehouse
HadoopDB SMS_dist. :
5.41 HadoopDB
t a r z x v f SMS_dist . t a r . gz
- sudo mv d i s t / opt / h i v e
- sudo chown -R hadoop : hadoop h i v e
Line 1
Hadoop,
Hive :
5.42 HadoopDB
$ hive
- Hive h i s t o r y f i l e =/tmp/ hadoop /
- hive_job_log_hadoop_201005081717_1990651345 . t x t
- hive >
Line 1
5
-
hive > q u i t ;
. :
5.43
Line 1
-
svn co h t t p : / / g r a f f i t i . c s . brown . edu / svn / benchmarks /

cd benchmarks / datagen / t e r a g e n
benchmarks/datagen/teragen/teragen.pl :
1
2
http://sourceforge.net/projects/hadoopdb/files/
http://jdbc.postgresql.org/download.html
107
5.4. HadoopDB
5.44
Line 1
-
use s t r i c t ;
use warnings ;
my $CUR_HOSTNAME = hostname - s ;
5 chomp ($CUR_HOSTNAME) ;
10
-
my
my
my
my
my
my
my
$NUM_OF_RECORDS_1TB
= 10000000000;
$NUM_OF_RECORDS_535MB = 1 0 0 ;
$BASE_OUTPUT_DIR
= "/ data " ;
$PATTERN_STRING
= "XYZ" ;
$PATTERN_FREQUENCY = 1 0 8 2 9 9 ;
$TERAGEN_JAR
= " teragen . jar " ;
$HADOOP_COMMAND
= $ENV{ HADOOP_HOME } . "/ b i n / hadoop " ;
15
20
-
my %f i l e s = ( " 535MB" => 1 ,

);
system ( "$HADOOP_COMMAND f s - rmr $BASE_OUTPUT_DIR" ) ;
f o r e a c h my $ t a r g e t ( k e y s %f i l e s ) {
my $output_dir = $BASE_OUTPUT_DIR. " / S o r t G r e p $ t a r g e t " ;
my $num_of_maps = $ f i l e s { $ t a r g e t } ;
my $num_of_records = ( $ t a r g e t eq " 535MB" ?
$NUM_OF_RECORDS_535MB : $NUM_OF_RECORDS_1TB) ;
p r i n t " G e n e r a t i n g $num_of_maps f i l e s i n $output_dir \ n" ;
25
30
35
-
##
## EXEC: hadoop j a r t e r a g e n . j a r 10000000000
## / data / SortGrep / XYZ 108299 100
##
my @args = ( $num_of_records ,
$output_dir ,
$PATTERN_STRING,
$PATTERN_FREQUENCY,
$num_of_maps ) ;
my $cmd = "$HADOOP_COMMAND j a r $TERAGEN_JAR " . j o i n ( " " ,
@args ) ;
p r i n t "$cmd\n" ;
system ( $cmd ) == 0 | | d i e ( "ERROR: $ ! " ) ;
} # FOR
exit (0) ;
Perl ,
HDFS.
, HDFS.

. , ,
HDFS, :
108
5.4. HadoopDB
5.45
Line 1
5
-
$hadoop f s - g e t / data /SortGrep535MB/ part - 0 0 0 0 0 my_file

$psql
p s q l > CREATE DATABASE g r e p 0 ;
p s q l > USE g r e p 0 ;
p s q l > CREATE TABLE g r e p (
->
key1 c h a r a c t e r v a r y i n g ( 2 5 5 ) ,
->
f i e l d character varying (255)
-> ) ;
COPY g r e p FROM my_file WITH DELIMITER | ;
HadoopDB. HadoopDB
Catalog.properties. :
5.46
Line 1
5
10
15
20
25
30
-
#P r o p e r t i e s f o r C a t a l o g G e n e r a t i o n
##################################
n o d e s _ f i l e=machines . t x t
r e l a t i o n s _ u n c h u n k e d=grep , E n t i r e R a n k i n g s
r e l a t i o n s _ c h u n k e d=Rankings , U s e r V i s i t s
c a t a l o g _ f i l e=HadoopDB . xml
##
#DB Connection Parameters
##
p o r t =5432
username=p o s t g r e s
password=password
d r i v e r=com . p o s t g r e s q l . D r i v e r
u r l _ p r e f i x=j d b c \ : p o s t g r e s q l \ : / /
##
#Chunking p r o p e r t i e s
##
chunks_per_node=0
unchunked_db_prefix=g r e p
chunked_db_prefix=cdb
##
#R e p l i c a t i o n P r o p e r t i e s
##
dump_script_prefix=/r o o t /dump_
r e p l i c a t i o n _ s c r i p t _ p r e f i x =/r o o t / l o a d _ r e p l i c a _
dump_file_u_prefix=/mnt/dump_udb
dump_file_c_prefix=/mnt/dump_cdb
##
#C l u s t e r Connection
##
ssh_key=i d _ r s a
109
5.4. HadoopDB
machines.txt localhost (
). HadoopDB HDFS:
5.47
j a v a - cp $HADOOP_HOME/ l i b / hadoopdb . j a r \
> edu . y a l e . c s . hadoopdb . c a t a l o g . S i m p l e C a t a l o g G e n e r a t o r \
- > Catalog . p r o p e r t i e s
- hadoop d f s - put HadoopDB . xml HadoopDB . xml
Line 1
-
:
5.48
Line 1
j a v a - cp hadoopdb . j a r
edu . y a l e . c s . hadoopdb . c a t a l o g . SimpleRandomReplicationFactorTwo
Catalog . p r o p e r t i e s
HadoopDB.xml,
.
HDFS:
5.49
Line 1
-
hadoop d f s - rmr HadoopDB . xml

hadoop d f s - put HadoopDB . xml HadoopDB . xml
true
hadoopdb.config.replication
HADOOP_HOME/conf/hadoop-site.xml.
HadoopDB.
, HDFS:
5.50
j a v a - cp $CLASSPATH : hadoopdb . j a r \
> edu . y a l e . c s . hadoopdb . benchmark . GrepTaskDB \
- > - p a t t e r n %wo% - output p a d r a i g - hadoop . c o n f i g . f i l e
HadoopDB . xml
Line 1
-
:
5.51
Line 1
5
$ j a v a - cp $CLASSPATH : hadoopdb . j a r
edu . y a l e . c s . hadoopdb . benchmark . GrepTaskDB \
> - p a t t e r n %wo% - output p a d r a i g - hadoop . c o n f i g . f i l e
HadoopDB . xml
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 8 edu . y a l e . c s . hadoopdb . e x e c . DBJobBase
initConf
INFO : SELECT key1 , f i e l d FROM g r e p WHERE f i e l d LIKE
%%wo%% ;
14.08.2010 19:08:48
o r g . apache . hadoop . m e t r i c s . jvm . JvmMetrics i n i t
110
5.4. HadoopDB
10
-
15
-
20
-
25
30
-
INFO : I n i t i a l i z i n g JVM M e t r i c s with processName=JobTracker ,

s e s s i o n I d=
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 8 o r g . apache . hadoop . mapred . J o b C l i e n t
configureCommandLineOptions
WARNING: Use G e n e r i c O p t i o n s P a r s e r f o r p a r s i n g t h e arguments .
A p p l i c a t i o n s s h o u l d implement Tool f o r t h e same .
monitorAndPrintJob
INFO : Running j o b : job_local_0001
14.08.2010 19:08:48
edu . y a l e . c s . hadoopdb . c o n n e c t o r . AbstractDBRecordReader
getConnection
INFO : Data l o c a l i t y f a i l e d f o r l e o - p g s q l
14.08.2010 19:08:48
getConnection
INFO : Task from l e o - p g s q l i s c o n n e c t i n g t o chunk 0 on h o s t
l o c a l h o s t with
db u r l j d b c : p o s t g r e s q l : / / l o c a l h o s t : 5 4 3 4 / g r e p 0
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 8 o r g . apache . hadoop . mapred . MapTask
runOldMapper
INFO : numReduceTasks : 0
14.08.2010 19:08:48
close
INFO : DB t i m e s (ms) : c o n n e c t i o n = 1 0 4 , query e x e c u t i o n =
2 0 , row r e t r i e v a l = 79
14.08.2010 19:08:48
close
INFO : Rows r e t r i e v e d = 3
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 8 o r g . apache . hadoop . mapred . Task done
INFO : Task : attempt_local_0001_m_000000_0 i s done . And i s i n
t h e p r o c e s s o f commiting
14.08.2010 19:08:48
o r g . apache . hadoop . mapred . LocalJobRunner$Job s t a t u s U p d a t e
INFO :
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 8 o r g . apache . hadoop . mapred . Task commit
INFO : Task attempt_local_0001_m_000000_0 i s a l l o w e d t o
commit now
14.08.2010 19:08:48
o r g . apache . hadoop . mapred . FileOutputCommitter commitTask
INFO : Saved output o f t a s k attempt_local_0001_m_000000_0
t o f i l e : / home/ l e o / p a d r a i g
14.08.2010 19:08:48
INFO :
111
5.4. HadoopDB
35
40
45
50
55
-
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 8 o r g . apache . hadoop . mapred . Task sendDone

INFO : Task attempt_local_0001_m_000000_0 done .
monitorAndPrintJob
INFO : map 100% r e d u c e 0%
monitorAndPrintJob
INFO : Job c o m p l e t e : job_local_0001
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 9 o r g . apache . hadoop . mapred . Counters l o g
INFO : Counters : 6
INFO :
FileSystemCounters
INFO :
FILE_BYTES_READ=141370
INFO :
FILE_BYTES_WRITTEN=153336
INFO :
Map- Reduce Framework
INFO :
Map i n p u t r e c o r d s =3
INFO :
S p i l l e d Records=0
INFO :
Map i n p u t b y t e s=3
INFO :
Map output r e c o r d s =3
1 4 . 0 8 . 2 0 1 0 1 9 : 0 8 : 4 9 edu . y a l e . c s . hadoopdb . e x e c . DBJobBase run
INFO :
JOB TIME : 1828 ms .
HDFS, padraig:
5.52
$ cd p a d r a i g
$ c a t part - 0 0 0 0 0
- some data
Line 1
-
PostgreSQL:
5.53
Line 1
5
-
p s q l > s e l e c t * from g r e p where f i e l d l i k e %wo% ;

+- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - - -+
| key1
| field
|
+- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - - - - - - -+
some data
1 rows i n s e t ( 0 . 0 0 s e c )
112
5.4. HadoopDB
10
psql>
. .
. PostgreSQL:
5.54
Line 1
-
p s q l > INSERT i n t o g r e p ( key1 ,

Maybe ) ;
Maybewqe ) ;
Maybewqesad ) ;
string ! ) ;
f i e l d ) VALUES( I am l i v e ! ,
f i e l d ) VALUES( : ) , May c o o l
HadoopDB:
5.55
Line 1
5
-
10
-
$ j a v a - cp $CLASSPATH : hadoopdb . j a r
edu . y a l e . c s . hadoopdb . benchmark . GrepTaskDB - p a t t e r n %May%
- output p a d r a i g - hadoopdb . c o n f i g . f i l e
/ opt / hadoop / c o n f /HadoopDB . xml
padraig
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 5 edu . y a l e . c s . hadoopdb . e x e c . DBJobBase
initConf
INFO : SELECT key1 , f i e l d FROM g r e p WHERE f i e l d LIKE
%%May%% ;
01.11.2010 23:14:46
o r g . apache . hadoop . m e t r i c s . jvm . JvmMetrics i n i t
INFO : I n i t i a l i z i n g JVM M e t r i c s with processName=JobTracker ,
s e s s i o n I d=
configureCommandLineOptions
WARNING: Use G e n e r i c O p t i o n s P a r s e r f o r p a r s i n g t h e
arguments . A p p l i c a t i o n s s h o u l d implement Tool f o r t h e
same .
monitorAndPrintJob
INFO : Running j o b : job_local_0001
01.11.2010 23:14:46
getConnection
INFO : Data l o c a l i t y f a i l e d f o r l e o - p g s q l
01.11.2010 23:14:46
getConnection
113
5.4. HadoopDB
-
15
-
20
25
30
35
40
-
INFO : Task from l e o - p g s q l i s c o n n e c t i n g t o chunk 0 on h o s t

l o c a l h o s t with db u r l
jdbc : p o s t g r e s q l :// l o c a l h o s t :5434/ grep0
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 o r g . apache . hadoop . mapred . MapTask
runOldMapper
INFO : numReduceTasks : 0
01.11.2010 23:14:47
close
INFO : DB t i m e s (ms) : c o n n e c t i o n = 1 8 1 , query e x e c u t i o n =
2 2 , row r e t r i e v a l = 96
01.11.2010 23:14:47
close
INFO : Rows r e t r i e v e d = 4
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 o r g . apache . hadoop . mapred . Task done
INFO : Task : attempt_local_0001_m_000000_0 i s done . And i s i n
t h e p r o c e s s o f commiting
01.11.2010 23:14:47
INFO :
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 o r g . apache . hadoop . mapred . Task commit
INFO : Task attempt_local_0001_m_000000_0 i s a l l o w e d t o
commit now
01.11.2010 23:14:47
o r g . apache . hadoop . mapred . FileOutputCommitter commitTask
INFO : Saved output o f t a s k attempt_local_0001_m_000000_0
t o f i l e : / home/ hadoop / p a d r a i g
01.11.2010 23:14:47
INFO :
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 o r g . apache . hadoop . mapred . Task sendDone
INFO : Task attempt_local_0001_m_000000_0 done .
monitorAndPrintJob
INFO : map 100% r e d u c e 0%
monitorAndPrintJob
INFO : Job c o m p l e t e : job_local_0001
INFO : Counters : 6
INFO :
FileSystemCounters
INFO :
FILE_BYTES_READ=141345
INFO :
FILE_BYTES_WRITTEN=153291
114
5.4. HadoopDB
45
50
55
-

INFO :
Map- Reduce Framework
INFO :
Map i n p u t r e c o r d s =4
INFO :
S p i l l e d Records=0
INFO :
Map i n p u t b y t e s=4
INFO :
Map output r e c o r d s =4
0 1 . 1 1 . 2 0 1 0 2 3 : 1 4 : 4 7 edu . y a l e . c s . hadoopdb . e x e c . DBJobBase run
INFO :
JOB TIME : 2332 ms .
May.
. :
5.56
Line 1
5
-
$ cd p a d r a i g
$ c a t part - 0 0 0 0 0
I am l i v e !
Maybe
I am l i v e !
Maybewqe
I am l i v e !
Maybewqesad
:)
May c o o l s t r i n g !
PostgreSQL
. HadoopDB
PostgreSQL,
PostgreSQL, shared-nothing .
HadoopDB
hadoopdb.sourceforge.net.
, Hive,
HadoopDB. ,
c Hadoop.
Hadoop HaddopDB.
HadoopDB Hadoop.
,

.
HadoopDB
,

, Hadoop.
HadoopDB, ,
, ,
115
5.5.
PostgreSQL , , PostgreSQL
. , Hadoop Hive
.
HadoopDB
Hadoop ,
,
,
, MapReduce. HadoopDB
Hadoop
( ) HadoopDB

.
5.5
.
PostgreSQL ,
, .
, ,
.
116
6
PgPool-II

.

6.1
pgpool-II , PostgreSQL
PostgreSQL. :

pgpool-II PostgreSQL

(.. , ,
).
.

pgpool-II PostgreSQL.

2
,
.

,
SELECT .
pgpool-II
PostgreSQL
SELECT ,
.

PostgreSQL.

.
117
6.2. !

PostgreSQL,
.

, ,
. pgpool-II
,

.

,

.
.
pgpool-II PostgreSQL
. ,
() pgpool-II PostgreSQL,
() pgpool-II .
pgpool-II , ,
, ,
pgpool-II .
pgpool.projects.pgfoundry.org.
6.2 !

pgpool-II
.
pgpool-II
pgpool-II .
,
,
.
6.1 pgpool-II
./ configure
make
- make i n s t a l l
Line 1
-
configure
.
118
6.2. !
configure ,
, , . pgpool-II
/usr/local.
make , make install
.
.
: pgpool-II libpq
PostgreSQL 7.4 (3 ).
configure ,
libpq 3 .
6.2 pgpool-II
Line 1
c o n f i g u r e : e r r o r : l i b p q i s not i n s t a l l e d o r l i b p q i s o l d
3 , -
, libpq, ,
configure.
configure libpq
/usr/local/pgsql. PostgreSQL
/usr/local/pgsql with-pgsql
with-pgsql-includedir with-pgsql-libdir
configure.
Linux pgpool-II
. Ubuntu Linux, , :
6.3 pgpool-II
Line 1
sudo a p t i t u d e i n s t a l l p g p o o l 2

pgpool-II pgpool.conf.
: = .
pgpool-II pgpool.conf.sample.
pgpool.conf,
.
6.4
Line 1
cp / u s r / l o c a l / e t c / p g p o o l . c o n f . sample
/ usr / l o c a l / etc / pgpool . conf
pgpool-II localhost 9999.

,
listen_addresses *.
6.5
Line 1
-
listen_addresses = localhost
p o r t = 9999
119
6.2. !
.
Ubuntu Linux /etc/pgpool.conf.
PCP
pgpool-II
, pgpool-II ..
. PCP,
.

PostgreSQL.
pcp.conf.
, (:). .
md5.
6.6 PCP
Line 1
postgres : e8a48653851e28c69d0506508fb27fc5
pgpool-II
pcp.conf.sample. pcp.conf
.
6.7 PCP
Line 1
$ cp / u s r / l o c a l / e t c / pcp . c o n f . sample / u s r / l o c a l / e t c / pcp . c o n f
Ubuntu Linux /etc/pcp.conf.

md5
pg_md5,
pgpool-II. pg_md5
md5 .
, postgres
pg_md5 md5 .
6.8 PCP
$ / u s r / b i n /pg_md5 p o s t g r e s
- e8a48653851e28c69d0506508fb27fc5
Line 1
PCP , pgpool.conf
pcp_port.

pcp_port 9898 .
6.9 PCP
Line 1
pcp_port = 9898
120
6.2. !

PostgreSQL
pgpool-II. pgpool-II
.
,
. ,

pgpool-II.

pgpool-II 5432, 5433,
5434 . pgpool-II
pgpool.conf .
6.10
Line 1
5
-
backend_hostname0 = l o c a l h o s t
backend_port0 = 5432
backend_weight0 = 1
backend_weight1 = 1
backend_weight2 = 1
backend_hostname, backend_port, backend_weight

,
.

0 (.. 0, 1, 2).
backend_weight 1,
SELECT .
/ pgpool-II
pgpool-II .
6.11
Line 1
pgpool
, ,
pgpool .
pgpool, -n
pgpool. pgpool-II -
.
6.12
Line 1
pgpool -n &
121
6.3.
,
,
, .
6.13
Line 1
p g p o o l - n - d > /tmp/ p g p o o l . l o g 2>&1 &
-d .

/tmp/pgpool.log.
, ,
. , , cronolog.
6.14
Line 1
-
p g p o o l - n 2>&1 | / u s r / s b i n / c r o n o l o g
- - h a r d l i n k =/var / l o g / p g s q l / p g p o o l . l o g
/ var / l o g / p g s q l/%Y-%m-%d - p g p o o l . l o g &
pgpool-II, .
6.15
Line 1
pgpool stop
- , pgpool-II
.
pgpool-II ,
.
6.16
Line 1
p g p o o l -m f a s t s t o p
6.3

.
,
6.2. !,
.

pgbench.

true replication_mode pgpool.conf.
122
6.3.
6.17
Line 1
replication_mode = true
replication_mode true, pgpool-II

.
load_balance_mode true, pgpool-II
SELECT .
6.18
Line 1
load_balance_mode = t r u e
replication_mode
load_balance_mode.

, pgpool.conf, pgpool-II
.

/ pgpool-II.
pgpool.conf pgpool-II,
.
,
. bench_replication.
. createdb
pgpool-II .
6.19
Line 1
c r e a t e d b - p 9999 b e n c h _ r e p l i c a t i o n
pgbench -i.
-i

.
6.20
Line 1
pgbench - i - p 9999 b e n c h _ r e p l i c a t i o n

, pgbench -i.
,
.

branches
1
tellers
10
accounts
100000
history
0

shell.
123
6.4.
branches, tellers, accounts history
(5432, 5433, 5434).
6.21
f o r p o r t i n 5432 5433 5 4 3 4 ; do
>
echo $ p o r t
- >
f o r table_name i n b r a n c h e s t e l l e r s a c c o u n t s h i s t o r y ;
do
- >
echo $table_name
5 >
p s q l - c "SELECT count ( * ) FROM $table_name " - p \
- >
$port bench_replication
- >
done
- > done
Line 1
-
6.4

.
( partitioning . ).
,
.
pgpool-II
,
(System Database) ( SystemDB).
SystemDB
,

. SystemDB
dblink.
,
6.2. !,
.
pgbench.

parallel_mode true
pgpool.conf.
6.22
Line 1
parallel_mode = true
parallel_mode true
.
pgpool-II
124
6.4.
SystemDB
.
SystemDB dblink
pgpool-II. ,
listen_addresses pgpool-II .
6.23
Line 1
listen_addresses = *
:
,
, ,
. , -

, bench_replication, 6.3.
.
6.24
replication_mode = true
- load_balance_mode = f a l s e
Line 1
6.25
replication_mode = f a l s e
- load_balance_mode = t r u e
Line 1
parallel_mode
load_balance_mode true, listen_addresses *,
replication_mode false.
SystemDB
,
. , dblink
,
. dist_def . ,
,
pgpool-II
.
SystemDB 5432.
SystemDB
6.26 SystemDB
system_db_hostname = l o c a l h o s t
- system_db_port = 5432
- system_db_dbname = p g p o o l
- system_db_schema = p g p o o l _ c a t a l o g
Line 1
125
6.4.
5
-
system_db_user = p g p o o l
system_db_password =
,
pgpool.conf.
pgpool pgpool
pgpool.
6.27 SystemDB
Line 1
-
c r e a t e u s e r - p 5432 p g p o o l
c r e a t e d b - p 5432 -O p g p o o l p g p o o l
dblink
dblink pgpool. dblink
contrib
PostgreSQL.
dblink
.
6.28 dblink
USE_PGXS=1 make -C c o n t r i b / d b l i n k
- USE_PGXS=1 make -C c o n t r i b / d b l i n k i n s t a l l
Line 1
dblink
dblink pgpool. PostgreSQL
/usr/local/pgsql, dblink.sql ( )
/usr/local/pgsql/share/contrib.
dblink.
6.29 dblink
Line 1
p s q l - f / u s r / l o c a l / p g s q l / s h a r e / c o n t r i b / d b l i n k . s q l - p 5432
pgpool
dist_def
dist_def,
.
pgpool-II , system_db.sql
/usr/local/share/system_db.sql (

/usr/local). system_db.sql
, dist_def.
dist_def.
6.30 dist_def
126
6.4.
Line 1
$ p s q l - f / u s r / l o c a l / s h a r e / system_db . s q l - p 5432 -U p g p o o l
pgpool
system_db.sql,
dist_def,
pgpool_catalog.
system_db_schema ,
, system_db.sql.
dist_def .
.
6.31 dist_def
Line 1
5
10
-
CREATE TABLE p g p o o l _ c a t a l o g . d i s t _ d e f (
dbname t e x t , - -
schema_name t e x t , - -
table_name t e x t , - -
col_name t e x t NOT NULL CHECK ( col_name = ANY
( col_list ) ) ,
- -
c o l _ l i s t t e x t [ ] NOT NULL, - -
t y p e _ l i s t t e x t [ ] NOT NULL, - -
d i s t _ d e f _ f u n c t e x t NOT NULL,
- -
PRIMARY KEY ( dbname , schema_name , table_name )
);
, dist_def, .
(col_name, dist_def_func)
- (dbname, schema_name, table_name,
col_list, type_list)

.
col_name. dist_def_func
, col_name
,
, .
- .

, -,
.
replicate_def
,
SQL, dist_def
, ,
,
127
6.4.
replicate_def. replicate_def
system_db.sql dist_def.
replicate_def .
6.32 replicate_def
Line 1
5
-
CREATE TABLE p g p o o l _ c a t a l o g . r e p l i c a t e _ d e f (
dbname t e x t , - -
schema_name t e x t , - -
table_name t e x t , - -
c o l _ l i s t t e x t [ ] NOT NULL, - -
t y p e _ l i s t t e x t [ ] NOT NULL, - -
PRIMARY KEY ( dbname , schema_name , table_name )
);

, pgbench, .
pgbench -i -s 3 (..
3).
bench_parallel.
sample pgpool-II
dist_def_pgbench.sql.
pgbench.
pgpool-II.
6.33
Line 1
p s q l - f sample / dist_def_pgbench . s q l - p 5432 p g p o o l
dist_def_pgbench.sql.
dist_def_pgbench.sql
dist_def. accounts.
- aid.
6.34
Line 1
5
10
INSERT INTO p g p o o l _ c a t a l o g . d i s t _ d e f VALUES (

bench_parallel ,
public ,
accounts ,
aid ,
ARRAY[ a i d , b i d , a b a l a n c e , f i l l e r ] ,
ARRAY[ i n t e g e r , i n t e g e r , i n t e g e r ,
character (84) ] ,
pgpool_catalog . dist_def_accounts
);
128
6.4.

accounts. ,
.
SQL (, PL/pgSQL, PL/Tcl,
..).
accounts
3, aid 1 300000.

.
SQL- .
6.35
Line 1
5
-

p g p o o l _ c a t a l o g . d i s t _ d e f _ b r a n c h e s ( anyelement )
RETURNS i n t e g e r AS $$
SELECT CASE WHEN $1 > 0 AND $1 <= 1 THEN 0
WHEN $1 > 1 AND $1 <= 2 THEN 1
ELSE 2
END;
$$ LANGUAGE s q l ;

.
pgbench
branches tellers.
, accounts
, branches tellers.
6.36
Line 1
5
-
INSERT INTO p g p o o l _ c a t a l o g . r e p l i c a t e _ d e f VALUES (

bench_parallel ,
public ,
branches ,
ARRAY[ b i d , b b a l a n c e , f i l l e r ] ,
ARRAY[ i n t e g e r , i n t e g e r , c h a r a c t e r ( 8 8 ) ]
);
10
15
INSERT INTO p g p o o l _ c a t a l o g . r e p l i c a t e _ d e f VALUES (

bench_parallel ,
public ,
tellers ,
ARRAY[ t i d , b i d , t b a l a n c e , f i l l e r ] ,
ARRAY[ i n t e g e r , i n t e g e r , i n t e g e r , c h a r a c t e r ( 8 4 ) ]
);
129
6.4.
Replicate_def_pgbench.sql
sample. psql ,
, , .
6.37
Line 1
p s q l - f sample / r e p l i c a t e _ d e f _ p g b e n c h . s q l - p 5432 p g p o o l

, pgpool.conf, pgpool-II
.

/ pgpool-II.
pgpool.conf pgpool-II,
.
, .
bench_parallel.
. createdb pgpool-II
.
6.38
Line 1
c r e a t e d b - p 9999 b e n c h _ p a r a l l e l
pgbench -i -s 3.
-i
.
-s .
6.39
Line 1
pgbench - i - s 3 - p 9999 b e n c h _ p a r a l l e l

.

SELECT pgpool-II
.
bench_parallel .

branches
3
tellers
30
accounts
300000
history
0

pgpool-II shell.

accounts 5432, 5433, 5434 9999.
6.40
130
6.5. Master-slave
f o r p o r t i n 5432 5433 5434 i 9 9 9 9 ; do
>
echo $ p o r t
- >
p s q l - c "SELECT min ( a i d ) , max( a i d ) FROM a c c o u n t s " \
- >
-p $port bench_parallel
5 > done
Line 1
-
6.5 Master-slave
pgpool-II
( Slony-I, Londiste).
.
master_slave_mode
load_balance_mode true. pgpool-II
INSERT/UPDATE/DELETE Master DB (1 ),
SELECT , .
, DDL DML
. SELECT ,
/*NO LOAD BALANCE*/
SELECT.
Master/Slave replication_mode false,
master_slave_mode true.
master-slave ,
,
pgpool-II. PostgreSQL,
pgpool-II (
), (
). , ,
( recovery.conf,
trigger_file), PostgreSQL
. :
6.41 PostgreSQL
Line 1
5
-
#! / b i n / sh
# F a i l o v e r command f o r s t r e m i n g r e p l i c a t i o n .
# This s c r i p t assumes t h a t DB node 0 i s primary , and 1 i s
standby .
#
# I f standby g o e s down , d o e s n o t h i n g . I f primary g o e s down ,
create a
# t r i g g e r f i l e s o t h a t standby t a k e o v e r primary node .
#
# Arguments : $1 : f a i l e d node i d . $2 : new master hostname .
$3 : path t o
131
6.6.
-
# trigger f i l e .
10
f a i l e d _ n o d e=$1
- new_master=$2
- t r i g g e r _ f i l e=$3
-
# Do n o t h i n g i f standby g o e s down .
i f [ $ f a i l e d _ n o d e = 1 ] ; then
exit 0;
- fi
15
-
20
-
# Create t r i g g e r f i l e .
/ u s r / b i n / s s h -T $new_master / b i n / touch $ t r i g g e r _ f i l e
exit 0;
: ,
.
failover_stream.sh pgpool.conf :
6.42
Line 1
failover_command = / p a th _ t o_ s c r ip t / f a i l o v e r _ s t r e a m . sh %d
%H /tmp/ t r i g g e r _ f i l e
/tmp/trigger_file , recovery.conf.
, ,
.
6.6
pgpool-II, ,
pgpool.
. ,
.
.
pgpool,
.
:
CHECKPOINT;
;
, ;
CHECKPOINT;
;
postmaster ( pgpool_remote_start);
;
132
6.6.

:
backend_data_directory
PostgreSQL .
recovery_user
PostgreSQL.
recovery_password
PostgreSQL.
recovery_1st_stage_command

.

- .
,
recovery_1st_stage_command = some_script, pgpool-II
$PGDATA/some_script.
, pgpool-II

recovery_1st_stage.
recovery_2nd_stage_command

.

- .
,
recovery_2st_stage_command = some_script, pgpool-II
$PGDATA/some_script.
, pgpool-II

recovery_2st_stage. , pgpool-II ,
.
master-slave ,
PostgreSQL.
,
.
PostgreSQL pgpool-II (
).
:
recovery_user. postgres.
6.43 recovery_user
Line 1
recovery_user = postgres
133
6.6.

recovery_password
.
recovery_user
6.44 recovery_password
Line 1
recovery_password = some_password
recovery_1st_stage_command.
basebackup.sh
($PGDATA), .
:
6.45 basebackup.sh
Line 1
5
-
#! / b i n / sh
# Recovery s c r i p t f o r s t r e a m i n g r e p l i c a t i o n .
# This s c r i p t assumes t h a t DB node 0 i s primary , and 1
i s standby .
#
d a t a d i r=$1
d e s t h o s t=$2
d e s t d i r=$3
p s q l - c "SELECT pg_start_backup ( Streaming

Replication , true )" postgres
10
-
r s y n c -C - a - - d e l e t e - e s s h - - e x c l u d e p o s t g r e s q l . c o n f
- - exclude postmaster . pid \
- - e x c l u d e p o s t m a s t e r . o p t s - - e x c l u d e pg_log - - e x c l u d e
pg_xlog \
- - exclude recovery . conf $datadir / $desthost : $destdir /
15
s s h -T l o c a l h o s t mv $ d e s t d i r / r e c o v e r y . done
$destdir / recovery . conf
p s q l - c "SELECT pg_stop_backup ( ) " p o s t g r e s
,
rsync .
SSH , recovery_user
.

:
6.46 recovery_1st_stage_command
Line 1
recovery_1st_stage_command = basebackup . sh
recovery_2nd_stage_command .
,
134
6.7.
,
basebackup.sh, WAL .
C SQL
.
6.47 C SQL
$
$
- $
- $
Line 1
-
cd pgpool - I I - x . x . x/ s q l / pgpool - r e c o v e r y
make
make i n s t a l l
p s q l - f pgpool - r e c o v e r y . s q l t e m p l a t e 1
. pcp_recovery_node
.
6.7
PgPool-II ,
PostgreSQL.
135
7

,
.
.
?

7.1
(
) ,
,
PostgreSQL.
Windows, .
-,
.
, :
PgBouncer
Pgpool
7.2 PgBouncer
PostgreSQL Skype.
.
Session Pooling. .
;

.
Transaction Pooling.
. PgBouncer ,
, .
136
7.3. PgPool-II vs PgBouncer

Statement Pooling. .
.
,
.
(prepared statements) .
PgBouncer :
( 2 );
;
.
:
7.1 PgBouncer
Line 1
pgbouncer [ - d ] [ - R ] [ - v ] [ - u u s e r ] <pgbouncer . i n i >
:
7.2 PgBouncer
Line 1
5
10
[ databases ]
t e m p l a t e 1 = h o s t = 1 2 7 . 0 . 0 . 1 p o r t =5432 dbname=t e m p l a t e 1
[ pgbouncer ]
l i s t e n _ p o r t = 6543
listen_addr = 127.0.0.1
auth_type = md5
auth_file = u s e r l i s t . txt
l o g f i l e = pgbouncer . l o g
p i d f i l e = pgbouncer . p i d
admin_users = someuser
userlist.txt
:someuser same_password_as_in_server
pgbouncer:
7.3 PgBouncer
Line 1
p s q l - h 1 2 7 . 0 . 0 . 1 - p 6543 pgbouncer

SHOW.
7.3 PgPool-II vs PgBouncer

. PgBouncer
, PgPool-II. ,
PgPool-II (
), PgBouncer.
137
7.3. PgPool-II vs PgBouncer

PgBouncer , PgPool-II
PgBouncer
PgBouncer (
)
PgBouncer PgPool-II .
138
8
PostgreSQL
- ,
- .

8.1
,
,
.
SELECT
PostgreSQL.

, , ,
.
SQL ,
, , . PostgreSQL
. ? -,
. ?
(MVCC MultiVersion
Concurrency Control)
,
, ,

.
, ,
.
, ,
. -,
, , ,
.
( ),

.
139
8.2. Pgmemcache
, ,
.
PostgreSQL:
Pgmemcache ( memcached)
Pgpool-II (query cache)
8.2 Pgmemcache
Memcached1 ,

-.

.

-.
.
,

memcached .
Pgmemcache2 PostgreSQL API libmemcached
memcached.

PostgreSQL , ,
memcached.
,
.
2.0.4
pgmemcache3 . Pgmemcache
PostgreSQL 8.4 ( 9.0 ),
Ubuntu Server 10.10. Pgmemcache ,
PostgreSQL PGXS ( ,
Linux PGXS). memcached
libmemcached 0.38.
,
Pgmemcache:

:
1
http://memcached.org/
http://pgfoundry.org/projects/pgmemcache/
3
http://pgfoundry.org/frs/download.php/2672/pgmemcache_2.0.4.tar.bz2
2
140
8.2. Pgmemcache
8.1
Line 1
-
$ make
$ sudo make i n s t a l l
deb ( Debian, Ubuntu)

, Debian Ubuntu,
deb
PostgreSQL:
8.2 deb
$ sudo apt - g e t i n s t a l l libmemcached - dev
p o s t g r e s q l - s e r v e r - dev - 8 . 4 l i b p q - dev d e v s c r i p t s yada
f l e x bison
- $ make deb84
- # deb
- $ sudo dpkg - i . . / p o s t g r e s q l - pgmemcache - 8 . 4 * . deb
Line 1
2.0.4 yada deb

:
8.3 deb
Cannot r e c o g n i z e s o u r c e name i n d e b i a n / c h a n g e l o g a t
/ u s r / b i n / yada l i n e 1 4 5 , <CHANGELOG> l i n e 1 .
- make : *** [ deb84 ] 9
Line 1

debian/changelog , :
8.4 deb
$PostgreSQL : pgmemcache/ d e b i a n / c h a n g e l o g , v 1 . 2
2010/05/05 1 9 : 5 6 : 5 0 ormod Exp $ < - - - -
- pgmemcache ( 2 . 0 . 4 ) u n s t a b l e ; urgency=low
Line 1
* v2 . 0 . 4
, deb
.
Pgmemcache
( Pgmemcache)
:
8.5
Line 1
-
% p s q l [ mydbname ] [ p g u s e r ]
[ mydbname]=# BEGIN ;
141
8.2. Pgmemcache
[ mydbname]=# \ i
/ u s r / l o c a l / p o s t g r e s q l / s h a r e / c o n t r i b /pgmemcache . s q l
- # Debian : \ i
/ u s r / s h a r e / p o s t g r e s q l / 8 . 4 / c o n t r i b /pgmemcache . s q l
5 [ mydbname]=# COMMIT;
-
memcached
memcache_server_add . .
memcached
PostgreSQL. ,
postgresql.conf :
pgmemcache shared_preload_libraries (
pgmemcache PostgreSQL)
pgmemcache custom_variable_classes (
pgmemcache)
pgmemcache.default_servers, host:port
(port - ) . :
8.6 default_servers
Line 1
pgmemcache . d e f a u l t _ s e r v e r s = 1 2 7 . 0 . 0 . 1 ,
1 9 2 . 1 6 8 . 0 . 2 0 : 1 1 2 1 1 # memcached
pgmemcache
pgmemcache.default_behavior.
libmemcached. :
8.7 pgmemcache
Line 1
pgmemcache . d e f a u l t _ b e h a v i o r= BINARY_PROTOCOL: 1
PostgreSQL
memcached.
pgmemcache,
memcached :
.
memcached :
8.8 memcache_stats
pgmemcache=# SELECT memcache_stats ( ) ;
memcache_stats
- -- -------------------------
Line 1
5
-
Server : 1 2 7 . 0 . 0 . 1 (11211)
p i d : 1116
142
8.2. Pgmemcache
Table 8.1: pgmemcache
10
-
memcache_server_add(hostname:port::TEXT)
memcache_server_add(hostname::TEXT)
memcached
. ,
11211.
memcache_add(key::TEXT, value::TEXT,
expire::TIMESTAMPTZ)
memcache_add(key::TEXT, value::TEXT,
expire::INTERVAL)
memcache_add(key::TEXT, value::TEXT)
newval = memcache_decr(key::TEXT,
decrement::INT4)
newval = memcache_decr(key::TEXT)

,
( ).
.
memcache_delete(key::TEXT,
hold_timer::INTERVAL)
memcache_delete(key::TEXT)
.

,

.
memcache_flush_all()
memcached
.
value = memcache_get(key::TEXT)
. NULL,
,
.
memcache_get_multi(keys::TEXT[])
memcache_get_multi(keys::BYTEA[])
.

=.
newval = memcache_incr(key::TEXT,
increment::INT4)
newval = memcache_incr(key::TEXT)

,
( ).
.
memcache_replace(key::TEXT, value::TEXT,
memcache_replace(key::TEXT, value::TEXT,
expire::INTERVAL)
memcache_replace(key::TEXT, value::TEXT)
memcache_set(key::TEXT, value::TEXT,
memcache_set(key::TEXT, value::TEXT,
expire::INTERVAL)
memcache_set(key::TEXT, value::TEXT)
.

.
stats = memcache_stats()
memcached.
uptime : 70
time : 1289598098
version : 1.4.5
p o i n t e r _ s i z e : 32
rusage_user : 0.0
rusage_system : 0 . 2 4 0 0 1
cur r _ it e m s : 0
total_items : 0
143
8.2. Pgmemcache
15
20
25
-
bytes : 0
curr_connections : 5
total_connections : 7
connection_structures : 6
cmd_get : 0
cmd_set : 0
get_hits : 0
get_misses : 0
evictions : 0
bytes_read : 20
b y t e s _ w r i t t e n : 782
limit_maxbytes : 67108864
threads : 4
( 1 row )
memcached :
8.9
pgmemcache=# SELECT memcache_add ( some_key , t e s t _ v a l u e ) ;
memcache_add
- -- -----------t
5 ( 1 row )
Line 1
-
pgmemcache=# SELECT memcache_get ( some_key ) ;

memcache_get
- -- -----------10
test_value
- ( 1 row )
-
memcached (
):
8.10
pgmemcache=# SELECT memcache_add ( some_seq , 10 ) ;
memcache_add
- -- -----------t
5 ( 1 row )
Line 1
-
pgmemcache=# SELECT memcache_incr ( some_seq ) ;

memcache_incr
- -- ------------10
11
- ( 1 row )
-
pgmemcache=# SELECT memcache_incr ( some_seq ) ;
144
8.2. Pgmemcache
15
-
memcache_incr
-- ------------12
( 1 row )
20
-
pgmemcache=# SELECT memcache_incr ( some_seq , 1 0 ) ;

memcache_incr
-- ------------22
( 1 row )
pgmemcache=# SELECT memcache_decr ( some_seq ) ;

memcache_decr
- -- ------------21
- ( 1 row )
25
-
30
35
pgmemcache=# SELECT memcache_decr ( some_seq ) ;

memcache_decr
-- ------------20
( 1 row )
40
-
pgmemcache=# SELECT memcache_decr ( some_seq , 6 ) ;

memcache_decr
-- ------------14
( 1 row )
pgmemcache , ,
.
,
memcached ( ),
, .
:
8.11
Line 1
5
-
CREATE OR REPLACE FUNCTION auth_passwd_upd ( ) RETURNS

TRIGGER AS $$
BEGIN
IF OLD. passwd != NEW. passwd THEN
PERFORM memcache_set ( user_id_ | |
NEW. u s e r _ i d | | _password , NEW. passwd ) ;
END IF ;
RETURN NEW;
END;
$$ LANGUAGE p l p g s q l ;
145
8.2. Pgmemcache
:
8.12
Line 1
CREATE TRIGGER auth_passwd_upd_trg AFTER UPDATE ON passwd

FOR EACH ROW EXECUTE PROCEDURE auth_passwd_upd ( ) ;
(!!!)
.
:
8.13
Line 1
5
-
CREATE OR REPLACE FUNCTION auth_passwd_upd ( ) RETURNS

TRIGGER AS $$
BEGIN
IF OLD. passwd != NEW. passwd THEN
PERFORM memcache_delete ( user_id_ | |
NEW. u s e r _ i d | | _password ) ;
END IF ;
RETURN NEW;
END; $$ LANGUAGE p l p g s q l ;
:
8.14
Line 1
CREATE TRIGGER auth_passwd_del_trg AFTER DELETE ON passwd

FOR EACH ROW EXECUTE PROCEDURE auth_passwd_upd ( ) ;
, memcached
( ) .
PostgreSQL Pgmemcache
memcached ,
.
memcached,
SQL ,
PostgreSQL.
146
9.1
PostgreSQL
.
.
9.2 PostGIS
: Open Source
: www.postgis.org
PostGIS
PostgreSQL. PostGIS PostgreSQL

(), , ESRI SDE
Oracle. PostGIS OpenGIS
SQL .
9.3 pgSphere
: Open Source
: pgsphere.projects.postgresql.org
pgSphere PostgreSQL ,
.
( PostGIS)
.
147
9.4. HStore
9.4 HStore
: Open Source
HStore ,
/ PostgreSQL (
). ,
, ,
- .
.

:
9.1 hstore
Line 1
# CREATE EXTENSION h s t o r e ;
:
9.2 hstore
Line 1
5
# SELECT a=>1,a=>2 : : h s t o r e ;
hstore
-- -------"a"=>"1 "
( 1 row )
9.2 hstore .
:
9.3 hstore
Line 1
5
10
15
-
CREATE TABLE p r o d u c t s (
i d s e r i a l PRIMARY KEY,
name varchar ,
attributes hstore
);
INSERT INTO p r o d u c t s ( name , a t t r i b u t e s )
VALUES (
Geek Love : A Novel ,
author
=> " K a t h e r i n e Dunn" ,
pa g e s
=> 3 6 8 ,
c a t e g o r y => f i c t i o n
),
(
L e i c a M9 ,
m a n u f a c t u r e r => L e i c a ,
type
=> camera ,
megapixels
=> 1 8 ,
sensor
=> " f u l l - frame 35mm"
148
9.4. HStore
20
25
-
),
( MacBook Air 11 ,
m a n u f a c t u r e r => Apple ,
type
=> computer ,
ram
=> 4GB,
storage
=> 256GB,
processor
=> " 1 . 8 ghz I n t e l i 7 d u e l c o r e " ,
weight
=> 2 . 3 8 l b s
);
:
9.4
Line 1
5
# SELECT name , a t t r i b u t e s -> p a g e s a s page FROM p r o d u c t s

WHERE a t t r i b u t e s ? p a g e s ;
name
| page
- - - - - - - - - - - - - - - - - - - -+ - - - - - Geek Love : A Novel | 368
( 1 row )
:
9.5
Line 1
5
# SELECT name , a t t r i b u t e s -> m a n u f a c t u r e r a s m a n u f a c t u r e r

FROM p r o d u c t s WHERE a t t r i b u t e s -> type = computer ;
name
| manufacturer
- - - - - - - - - - - - - - - -+ - - - - - - - - - - - - - MacBook Air 11 | Apple
( 1 row )
:
9.6
CREATE INDEX p r o d u c t s _ h s t o r e _ i n d e x ON p r o d u c t s USING GIST
( attributes ) ;
- CREATE INDEX p r o d u c t s _ h s t o r e _ i n d e x ON p r o d u c t s USING GIN
( attributes ) ;
Line 1
c :
9.7
Line 1
-
CREATE INDEX product_manufacturer

ON p r o d u c t s ( ( p r o d u c t s . a t t r i b u t e s -> m a n u f a c t u r e r ) ) ;
HStore
PostgreSQL.
149
9.5. PLV8
9.5 PLV8
: Open Source
: code.google.com/p/plv8js
PLV8 , PostgreSQL
V8 JavaScript.

PostgreSQL JavaScript ,
SQL.

V81 JavaScript
.
.
plpgsql:
9.8 plpgsql
Line 1
5
-

p s q l f i b ( n i n t ) RETURNS i n t AS $$
BEGIN
IF n < 2 THEN
RETURN n ;
END IF ;
RETURN p s q l f i b ( n - 1 ) + p s q l f i b ( n - 2 ) ;
END;
$$ LANGUAGE p l p g s q l IMMUTABLE STRICT ;
:
9.9 plpgsql
Line 1
5
10
-
SELECT n , p s q l f i b ( n ) FROM g e n e r a t e _ s e r i e s ( 0 , 3 0 , 5 ) a s n ;
n | psqlfib
- - - -+ - - - - - - - - 0 |
0
5 |
5
10 |
55
15 |
610
20 |
6765
25 |
75025
30 | 832040
( 7 rows )
Time : 1 6 0 0 3 , 2 5 7 ms
, PLV8:
1
http://en.wikipedia.org/wiki/V8_(JavaScript_engine)
150
9.5. PLV8
9.10 plv8
- f i b ( n i n t ) RETURNS i n t a s $$
Line 1
5
-
function fib (n) {

r e t u r n n<2 ? n : f i b ( n - 1 ) + f i b ( n - 2 )
}
return f i b (n)
$$ LANGUAGE p l v 8 IMMUTABLE STRICT ;
:
9.11 plv8
Line 1
5
10
-
SELECT n , f i b ( n ) FROM g e n e r a t e _ s e r i e s ( 0 , 3 0 , 5 ) a s n ;
n |
fib
- - - -+ - - - - - - - 0 |
0
5 |
5
10 |
55
15 |
610
20 |
6765
25 | 75025
30 | 832040
( 7 rows )
Time : 5 9 , 2 5 4 ms
PLV8 270 (16003.257/59.254)

plpgsql. PLV8
:
9.12 plv8
Line 1
5
10

f i b 1 ( n i n t ) RETURNS i n t a s $$
var memo = { 0 : 0 , 1 : 1 } ;
function fib (n) {
i f ( ! ( n i n memo) )
memo [ n ] = f i b ( n - 1 ) + f i b ( n - 2 )
r e t u r n memo [ n ]
}
return f i b (n) ;
:
9.13 plv8
Line 1
SELECT n , f i b 1 ( n ) FROM g e n e r a t e _ s e r i e s ( 0 , 3 0 , 5 ) a s n ;
151
9.5. PLV8
5
10
-
n |
fib1
- - - -+ - - - - - - - 0 |
0
5 |
5
10 |
55
15 |
610
20 |
6765
25 | 75025
30 | 832040
( 7 rows )
Time : 0 , 7 6 6 ms
PLV8 20892 (16003.257/0.766)

, plpgsql.
PLV8
PostgreSQL .

hstore,
:

( )

JSON (MongoDB, CouchDB, Couchbase ..).
PostgreSQL 9.2 JSON.
PostgreSQL 9.1 PLV8 DOMAIN:
9.14 JSON
Line 1
5
-

valid_json ( json text )
RETURNS BOOLEAN AS $$
try {
JSON . p a r s e ( j s o n ) ; r e t u r n t r u e ;
} catch ( e ) {
return f a l s e ;
}
10
-
CREATE DOMAIN j s o n AS TEXT

CHECK( v a l i d _ j s o n (VALUE) ) ;
valid_json JSON .
:
152
9.5. PLV8
9.15 JSON
Line 1
5
10
-
$ INSERT INTO members

VALUES( not good j s o n ) ;
ERROR: v a l u e f o r domain j s o n
v i o l a t e s check c o n s t r a i n t " json_check "
$ INSERT INTO members
VALUES( {" good " : " j s o n " , " i s " : t r u e } ) ;
INSERT 0 1
$ s e l e c t * from members ;
profile
-- ---------------------------{" good " : " j s o n " , " i s " : t r u e }
( 1 row )
JSON PLV8
. :
9.16 JSON
Line 1
5
-
$ CREATE TABLE members ( i d SERIAL , p r o f i l e j s o n ) ;

$ SELECT count ( * ) FROM members ;
count
-- ------1000000
( 1 row )
Time : 2 0 1 . 1 0 9 ms
profile JSON:
9.17 JSON
Line 1
5
10
15
-
+
"name" : " L i t z y S a t t e r f i e l d " ,
+
" age " : 2 4 ,
+
"siblings": 2,
+
" faculty " : false ,
+
" numbers " : [
+
{
+
" type " :
" work " ,
+
"number" : " 6 8 4 . 5 7 3 . 3 7 8 3 x368 "+
},
+
{
+
" type " :
"home" ,
+
"number" : " 6 2 5 . 1 1 2 . 6 0 8 1 "
+
}
+
]
+
JSON (
):
153
9.5. PLV8
9.18 JSON
Line 1
5
CREATE OR REPLACE FUNCTION get_numeric ( json_raw j s o n , key

text )
RETURNS numeric AS $$
var o = JSON . p a r s e ( json_raw ) ;
r e t u r n o [ key ] ;

age, siblings :
9.19 JSON
$ SELECT * FROM members WHERE get_numeric ( p r o f i l e , age ) =
36;
- Time : 9 3 4 0 . 1 4 2 ms
- $ SELECT * FROM members WHERE get_numeric ( p r o f i l e ,
siblings ) = 1;
- Time : 1 4 3 2 0 . 0 3 2 ms
Line 1
, .
, :
9.20
CREATE INDEX member_age ON members ( get_numeric ( p r o f i l e ,
age ) ) ;
- CREATE INDEX member_siblings ON members
( get_numeric ( p r o f i l e , s i b l i n g s ) ) ;
Line 1
JSON :
9.21 JSON
Line 1
5
-
$ SELECT * FROM members WHERE get_numeric ( p r o f i l e , age ) =

36;
Time : 5 7 . 4 2 9 ms
$ SELECT * FROM members WHERE get_numeric ( p r o f i l e ,
siblings ) = 1;
Time : 6 5 . 1 3 6 ms
$ SELECT count ( * ) from members where get_numeric ( p r o f i l e ,
age ) = 26 and get_numeric ( p r o f i l e , s i b l i n g s ) = 1 ;
Time : 1 0 6 . 4 9 2 ms

PostgreSQL.
PLV8 JavaScript
PostgreSQL. Mustache1 :
9.22 Mustache
1
http://mustache.github.com/
154
9.6. Smlar
Line 1
5
CREATE OR REPLACE FUNCTION mustache ( t e m p l a t e t e x t , view

json )
RETURNS t e x t a s $$
// . . . 400 l i n e s o f mustache . . . . j s
r e t u r n Mustache . r e n d e r ( template , JSON . p a r s e ( view ) )
9.23
Line 1
5
-
$ SELECT mustache (
h e l l o {{# t h i n g s } } { { . } } {{/ t h i n g s } } : )
{{#data }}{{ key }}{{/ data }} ,
{" t h i n g s " : [ " world " , " from " , " p o s t g r e s q l " ] , " data " :
{" key " : "and me"}}
);
mustache
-- ------------------------------------h e l l o world from p o s t g r e s q l : ) and me
( 1 row )
10
Time : 0 . 8 3 7 ms
PLV8.
Mustache PostgreSQL .
PLV8 PostgreSQL
V8 JavaScript, JavaScript
, JSON
.
9.6 Smlar
: Open Source
: sigaev.ru

( ),
- ( ), (
, ) ..
PostgreSQL
. ,
, .
.
, ,
155
9.6. Smlar
- , , ..
,
, ( 1 , n-grams2 ).
.
?

.
, :
,

-

:
(, ) =
( + )
(9.1)
(9.2)

(, ) =
:

: * log

3 :
(, ) =
(9.3)
:
: * log

:

: ,
1
http://en.wikipedia.org/wiki/Fingerprint
http://en.wikipedia.org/wiki/N-gram
3
http://en.wikipedia.org/wiki/Law_of_cosines
2
156
9.6. Smlar
.
-
.
TF/IDF1
:
< ,< , = *
(, ) =
2
2
<
< *
(9.4)
:
= log (
+ 1)
(9.5)
:
= *
(9.6)
! smlar ,
( ) . ,
TF/IDF
, .
Smlar
.
PostgreSQL smlar,
(
)
GIST GIN.
(PostgreSQL ):
9.24 smlar
g i t c l o n e g i t : / / s i g a e v . ru / s m l a r
- cd s m l a r
- USE_PGXS=1 make && make i n s t a l l
Line 1
PostgreSQL 9.2 ,
PostgreSQL 9.1
. smlar_guc.c 214 :
9.25 9.1
Line 1
s e t _ c o n f i g _ o p t i o n ( " s m l ar . t h r e s h o l d " , buf , PGC_USERSET,

PGC_S_SESSION ,GUC_ACTION_SET, t r u e , 0 ) ;
( ):
1
http://en.wikipedia.org/wiki/Tf*idf
157
9.6. Smlar
9.26 9.1
Line 1
s e t _ c o n f i g _ o p t i o n ( " s m l ar . t h r e s h o l d " , buf , PGC_USERSET,

PGC_S_SESSION ,GUC_ACTION_SET, t r u e ) ;
:
9.27 smlar
$ psql
psql ( 9 . 2 . 1 )
- Type " h e l p " f o r h e l p .
Line 1
-
t e s t=# CREATE EXTENSION s m l a r ;

- CREATE EXTENSION
5
10
-
t e s t=# SELECT s m l a r ( { 1 , 4 , 6 } : : i n t [ ] ,
sm l a r
-- -------0.666667
( 1 row )
{5 ,4 ,6} : : int [ ] ) ;
t e s t=# SELECT s m l a r ( { 1 , 4 , 6 } : : i n t [ ] ,
N. i / s q r t (N. a * N. b ) ) ;
sm l a r
-- -------0.666667
( 1 row )
{5 ,4 ,6} : : int [ ] ,
15
-
,
. , :
float4 smlar(anyarray, anyarray)
. .
float4 smlar(anyarray, anyarray, bool useIntersect)
.
:
9.28
Line 1
CREATE TYPE type_name AS ( element_name anytype ,

weight_name f l o a t 4 ) ;
useIntersect

float4 smlar( anyarray a, anyarray b, text formula )
,
. :
N.i
()
158
9.6. Smlar
N.a
N.b
anyarray % anyarray ,
, .
PostgreSQL:
9.29 Smlar
Line 1
-
custom_variable_classes = smlar
s m l a r . t h r e s h o l d = 0 . 8 # 0 1

smlar:
9.30 Smlar
Line 1
-
custom_variable_classes = smlar
s m l a r . t h r e s h o l d = 0 . 8 # 0 1
s m l a r . type = c o s i n e #
: c o s i n e , t f i d f , o v e r l a p
s m l a r . s t a t t a b l e = s t a t #
t f i d f
README .
GiST GIN %.
:
.
,
( , ,
). , ,
. ,
?
.
. , ,
, .
:
(
), 15X15 (. 9.1).
(
0.299 * + 0.587 * +
0.114 * ).
, .

(. 9.2).
159
9.6. Smlar
. 9.1:
(
+ ).
.
. 9.2:
, ,
:
160
9.6. Smlar
9.31
Line 1
5
-
CREATE TABLE images (

i d s e r i a l PRIMARY KEY,
name v a r c h a r ( 5 0 ) ,
img_path v a r c h a r ( 2 5 0 ) ,
image_array i n t e g e r [ ]
);
GIN GIST :
9.32 GIN GIST
CREATE INDEX image_array_gin ON images USING
GIN( image_array _int4_sml_ops ) ;
- CREATE INDEX image_array_gist ON images USING
GIST ( image_array _int4_sml_ops ) ;
Line 1
:
9.33
Line 1
5
t e s t=# SELECT count ( * ) from images ;

count
-- ------1000000
( 1 row )
t e s t=# EXPLAIN ANALYZE SELECT count ( * ) FROM images WHERE

images . image_array %
{1010259 ,1011253 ,... ,2423253 ,2424252} : : int [ ] ;
10
-
Bitmap Heap Scan on images ( c o s t = 2 8 6 . 6 4 . . 3 9 6 9 . 4 5 rows =986

width =4) ( a c t u a l time = 5 0 4 . 3 1 2 . . 2 0 4 7 . 5 3 3 rows =200000
l o o p s =1)
Recheck Cond : ( image_array %
{1010259 ,1011253 ,... ,2423253 ,2424252} : : integer [ ] )
-> Bitmap Index Scan on image_array_gist
( c o s t = 0 . 0 0 . . 2 8 6 . 3 9 rows =986 width =0) ( a c t u a l
time = 4 4 6 . 1 0 9 . . 4 4 6 . 1 0 9 rows =200000 l o o p s =1)
Index Cond : ( image_array %
{1010259 ,1011253 ,... ,2423253 ,2424252} : : integer [ ] )
T o t a l runtime : 2 1 5 2 . 4 1 1 ms
( 5 rows )
1010259,...,2424252::int[] ,
. smlar.threshold
% (
).

:
161
9.7. PostPic
9.34
Line 1
t e s t=# EXPLAIN ANALYZE SELECT s m l a r ( images . image_array ,

{ 1 0 1 0 2 5 9 , . . . , 2 4 2 4 2 5 2 } : : i n t [ ] ) a s s i m i l a r i t y FROM
images WHERE images . image_array % { 1 0 1 0 2 5 9 , 1 0 1 1 2 5 3 ,
. . . , 2 4 2 3 2 5 3 , 2 4 2 4 2 5 2 } : : i n t [ ] ORDER BY s i m i l a r i t y DESC;
5
-
10
-
S o r t ( c o s t = 4 0 2 0 . 9 4 . . 4 0 2 3 . 4 1 rows =986 width =924) ( a c t u a l

time = 2 8 8 8 . 4 7 2 . . 2 9 0 1 . 9 7 7 rows =200000 l o o p s =1)
S o r t Key : ( s m l a r ( image_array ,
{... ,2424252} : : integer [ ] ) )
S o r t Method : q u i c k s o r t Memory : 15520kB
-> Bitmap Heap Scan on images ( c o s t = 2 8 6 . 6 4 . . 3 9 7 1 . 9 1
rows =986 width =924) ( a c t u a l time = 4 7 4 . 4 3 6 . . 2 7 2 9 . 6 3 8
rows =200000 l o o p s =1)
Recheck Cond : ( image_array %
{... ,2424252} : : integer [ ] )
-> Bitmap Index Scan on image_array_gist
( c o s t = 0 . 0 0 . . 2 8 6 . 3 9 rows =986 width =0) ( a c t u a l
time = 4 2 1 . 1 4 0 . . 4 2 1 . 1 4 0 rows =200000 l o o p s =1)
Index Cond : ( image_array %
{... ,2424252} : : integer [ ] )
T o t a l runtime : 2 9 1 2 . 2 0 7 ms
( 8 rows )
1 . P.S.
PostgreSQL ,
, (
).
Smlar ,
, , , , ,
, , .
9.7 PostPic
: Open Source
: github.com/drotiro/postpic
PostPic PostgreSQL,
, PostGIS
. image,
(,
, ..) (, ,
).
162
9.8. Fuzzystrmatch
9.8 Fuzzystrmatch
: Open Source
Fuzzystrmatch
. soundex

. difference soundex
, . soundex
,
: 0 , 4 ( ,
similarity):
9.35 soundex
Line 1
5
-
# CREATE EXTENSION f u z z y s t r m a t c h ;
CREATE EXTENSION
# SELECT soundex ( h e l l o world ! ) ;
soundex
-- ------H464
( 1 row )
10
-
# SELECT soundex ( Anne ) , soundex ( Ann ) ,

d i f f e r e n c e ( Anne , Ann ) ;
soundex | soundex | d i f f e r e n c e
- - - - - - - - -+ - - - - - - - - -+ - - - - - - - - - - - A500
| A500
|
4
( 1 row )
15
-
# SELECT soundex ( Anne ) , soundex ( Andrew ) ,

d i f f e r e n c e ( Anne , Andrew ) ;
- - - - - - - - -+ - - - - - - - - -+ - - - - - - - - - - - A500
| A536
|
2
( 1 row )
20
25
# SELECT soundex ( Anne ) , soundex ( Margaret ) ,

d i f f e r e n c e ( Anne , Margaret ) ;
- - - - - - - - -+ - - - - - - - - -+ - - - - - - - - - - - A500
| M626
|
0
( 1 row )
# CREATE TABLE s (nm t e x t ) ;

CREATE TABLE
- # INSERT INTO s VALUES ( john ) , ( j o a n ) , ( wobbly ) ,
( jack ) ;
30 INSERT 0 4
-
163
9.8. Fuzzystrmatch
35
-
# SELECT * FROM s WHERE soundex (nm) = soundex ( john ) ;

nm
-- ---john
joan
( 2 rows )
40
-
# SELECT * FROM s WHERE d i f f e r e n c e ( s . nm, john ) > 2 ;

nm
-- ---john
joan
jack
( 3 rows )
levenshtein 1
. levenshtein_less_equal levenshtein
:
9.36 levenshtein
Line 1
5
# SELECT l e v e n s h t e i n ( GUMBO , GAMBOL ) ;

levenshtein
-- ----------2
( 1 row )
# SELECT l e v e n s h t e i n ( GUMBO , GAMBOL , 2 , 1 , 1 ) ;

levenshtein
- -- ----------10
3
- ( 1 row )
-
15
-
# SELECT l e v e n s h t e i n _ l e s s _ e q u a l ( e x t e n s i v e , e x h a u s t i v e ,
2) ;
levenshtein_less_equal
-- ---------------------3
( 1 row )
20
-
t e s t=# SELECT l e v e n s h t e i n _ l e s s _ e q u a l ( e x t e n s i v e ,
exhaustive , 4) ;
levenshtein_less_equal
-- ---------------------4
( 1 row )
1
http://en.wikipedia.org/wiki/Levenshtein_distance
164
9.9. Tsearch2
metaphone, soundex,
: , ,
.
metaphone . dmetaphone
:
9.37 metaphone
Line 1
5
10
# SELECT metaphone ( GUMBO , 4 ) ;

metaphone
-- --------KM
( 1 row )
# SELECT dmetaphone ( p o s t g r e s q l ) ;
dmetaphone
-- ---------PSTK
( 1 row )
# SELECT dmetaphone_alt ( p o s t g r e s q l ) ;
dmetaphone_alt
- -- -------------15
PSTK
- ( 1 row )
-
9.9 Tsearch2
: Open Source
Tsearch2 .
PostgreSQL 8.3.
9.10 OpenFTS
: Open Source
: openfts.sourceforge.net
OpenFTS (Open Source Full Text Search engine)
PostgreSQL ,

.
,
.
9.11 PL/Proxy
: Open Source
165
9.12. Texcaller
: pgfoundry.org/projects/plproxy
PL/Proxy -
.
5.2 .
9.12 Texcaller
: Open Source
: www.profv.de/texcaller
Texcaller TeX,
. C,
, , TeX.
TeX NULL,
. , ,
NOTICEs.
9.13 Pgmemcache
: Open Source
: pgfoundry.org/projects/pgmemcache
Pgmemcache PostgreSQL API libmemcached
memcached.

PostgreSQL , ,
memcached. 8.2 .
9.14 Prefix
: Open Source
: pgfoundry.org/projects/prefix
Prefix (prefix @> text). Prefix
,
/
.
9.15 Dblink
: Open Source
Dblink ,
SQL,
.
166
9.16. Ltree
9.16 Ltree
: Open Source
Ltree ,
,
. Materialized Path (
, ).
9.17
PostgreSQL
. PostgreSQL
, , ,
.
167
10
PostgreSQL

,
, ,

-

, .

10.1
.
, ,
, - .
PostgreSQL .
! , ,
, PostgreSQL
.
, ,
(DELETE, DROP),
( ).

PostgreSQL:
SQL ;
;
;
.
168
10.2. SQL
10.2 SQL
SQL.

, . PostgreSQL
pg_dump.
pg_dump:
10.1 pg_dump
Line 1
pg_dump dbname > o u t f i l e
:
10.2
Line 1
p s q l dbname < i n f i l e
dbname
. ,
, (
, ).
,
, :
10.3
Line 1
p s q l - - s e t ON_ERROR_STOP=on dbname o u t f i l e
10.6 PostgreSQL
Line 1
psql - f i n f i l e postgres
169
10.2. SQL
SQL

,
pg_dump.
, pg_dump
.
Unix, .
:
.
, GZIP:
10.7 PostgreSQL
Line 1
pg_dump dbname | g z i p > f i l e n a m e . gz
:
10.8 PostgreSQL
Line 1
g u n z i p - c f i l e n a m e . gz | p s q l dbname
10.9 PostgreSQL
Line 1
c a t f i l e n a m e . gz | g u n z i p | p s q l dbname
split.
split
,
. , 1 :
10.10 PostgreSQL
Line 1
pg_dump dbname | s p l i t - b 1m - f i l e n a m e
:
10.11 PostgreSQL
Line 1
c a t f i l e n a m e * | p s q l dbname
pg_dump
PostgreSQL Zlib,
.
GZIP,
:
10.12 PostgreSQL
Line 1
pg_dump - Fc dbname > f i l e n a m e
170
10.3.
psql ,
pg_restore:
10.13 PostgreSQL
Line 1
p g _ r e s t o r e - d dbname f i l e n a m e
, split
.
10.3

, PostgreSQL
. :
10.14 PostgreSQL
Line 1
t a r - c f backup . t a r / u s r / l o c a l / p g s q l / data
,

, , , SQL :
PostgreSQL , ,
(PostgreSQL
, ).
,
PostgreSQL.

.
, (snapshot)
( PostgreSQL). PostgreSQL
. , , ,
,
.

PostgreSQL , ,
WAL.
, ( WAL
). , PostgreSQL
,

(!!!).
,
.
rsync.
rsync PostgreSQL (PostgreSQL
171
10.4.
). PostgreSQL
rsync. rsync ,
,
,
.
.
10.4
PostgreSQL (Write Ahead
Log, WAL) pg_xlog ,
.
.
PostgreSQL: ,
.
,
:
WAL .
,
,
WAL .
,
, :
.

( ,
).

.
WAL
, ,
PostgreSQL
( ).
,
. ,
WAL .
.
WAL
pg_xlog. postgresql.conf:
172
10.5.
10.15
archive_mode = on # e n a b l e a r c h i v i n g
- archive_command = cp - v %p / data / p g s q l / a r c h i v e s/% f
- a r c h i v e _ t i m e o u t = 300 # t i m e o u t t o c l o s e b u f f e r s
Line 1
( )
. rsync.
cron , ,
.
10.16 WAL
r s y n c - avz - - d e l e t e prod1 : / data / p g s q l / a r c h i v e s / \
- / data / p g s q l / a r c h i v e s / > / dev / n u l l
Line 1
, pg_xlog
PostgreSQL ( ).
PostgreSQL recovery.conf
:
10.17 recovery.conf
Line 1
restore_command = cp / data / p g s q l / a r c h i v e s/% f "%p"
PostgreSQL
, (,
,
).
http://www.postgresql.org/docs/9.0/static/continuous-archiving.html.
10.5

.
, (
,
). ,
( S31 ).

,

.
1
http://aws.amazon.com/s3/
173
10.5.
WAL-E
: github.com/heroku/WAL-E
WAL-E PostgreSQL
WAL-logs S3 pg_start_backup
pg_stop_backup. Python
Heroku, .
WAL-E : lzop, psql, mbuffer, python 2.6+

python (gevent >= 0.13, boto >= 2.0).
daemontools.
Ubuntu :
10.18 WAL-E
Line 1
-
# PostgreSQL
a p t i t u d e i n s t a l l g i t - c o r e python - dev python - s e t u p t o o l s
b u i l d - e s s e n t i a l l i b e v e n t - dev l z o p m b u f f e r daemontools
daemontools - run
WAL-E:
10.19 WAL-E
g i t c l o n e g i t : / / g i t h u b . com/ heroku /WAL-E . g i t
cd WAL-E
- python s e t u p . py b u i l d
- sudo python s e t u p . py i n s t a l l
Line 1
-
WAL-E.

, WAL-E AWS S3,
Access Key ID Secret Access Key (
Amazon AWS).
S3:
10.20 S3
AWS_SECRET_ACCESS_KEY= . . . wal - e
\
- k AWS_ACCESS_KEY_ID
\
- - s3 - p r e f i x=s 3 : / / some - bucket / d i r e c t o r y / o r / whatever \
backup - push / var / l i b / p o s t g r e s q l / 9 . 2 / main
Line 1
-
s3-prefix , S3 (bucket)
, .
WAL- S3:
174
10.5.
10.21 WAL- S3
AWS_SECRET_ACCESS_KEY= . . . wal - e
\
- k AWS_ACCESS_KEY_ID
\
- - s3 - p r e f i x=s 3 : / / some - bucket / d i r e c t o r y / o r / whatever \
wal - push
/ var / l i b / p o s t g r e s q l / 9 . 2 / main/ pg_xlog /WAL_SEGMENT_LONG_HEX
Line 1
-

envdir ( daemontools). envdir
:
10.22 WAL-E envdir
$
$
- $
- $
mkdir - p / e t c / wal - e . d/ env

echo " s e c r e t - key " > / e t c / wal - e . d/ env /AWS_SECRET_ACCESS_KEY
echo " a c c e s s - key " > / e t c / wal - e . d/ env /AWS_ACCESS_KEY_ID
echo s 3 : / / some - bucket / d i r e c t o r y / o r / whatever >
/ e t c / wal - e . d/ env /WALE_S3_PREFIX
5 $ chown -R r o o t : p o s t g r e s / e t c / wal - e . d
Line 1
-

WAL-E
:
10.23 WAL-E envdir
Line 1
-
$ e n v d i r / e t c / wal - e . d/ env wal - e backup - push . . .

$ e n v d i r / e t c / wal - e . d/ env wal - e wal - push . . .
PostgreSQL WAL- S3 c
WAL-E. postgresql.conf:
10.24 PostgreSQL
w a l _ l e v e l = hot_standby # a r c h i v e , PostgreSQL < 9 . 0
- archive_mode = on
- archive_command = e n v d i r / e t c / wal - e . d/ env
/ u s r / l o c a l / b i n / wal - e wal - push %p
- a r c h i v e _ t i m e o u t = 60
Line 1
WAL-E ( which
wal-e), PostgreSQL .
PostgreSQL. :
10.25 PostgreSQL
Line 1
-
2 0 1 2 - 1 1 - 0 7 1 4 : 5 2 : 1 9 UTC LOG: d a t a b a s e system was s h u t down

a t 2 0 1 2 - 1 1 - 0 7 1 4 : 5 1 : 4 0 UTC
2 0 1 2 - 1 1 - 0 7 1 4 : 5 2 : 1 9 UTC LOG: d a t a b a s e system i s ready t o
accept connections
2 0 1 2 - 1 1 - 0 7 1 4 : 5 2 : 1 9 UTC LOG: autovacuum l a u n c h e r s t a r t e d
2 0 1 2 - 1 1 - 0 7 T14 : 5 2 : 1 9 . 7 8 4 + 0 0 p i d =7653 wal_e . worker . s3_worker
INFO
MSG: b e g i n a r c h i v i n g a f i l e
175
10.5.
5
10
15
-
DETAIL : Uploading
" pg_xlog /000000010000000000000001 " t o
" s 3 : / / c l e v e r d b - pg - backups /pg/ wal_005 / 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 . l z o " .
2 0 1 2 - 1 1 - 0 7 1 4 : 5 2 : 1 9 UTC LOG: i n c o m p l e t e s t a r t u p p a c k e t
INFO
MSG: completed a r c h i v i n g t o a f i l e
DETAIL : A r c h i v i n g t o
" s 3 : / / c l e v e r d b - pg - backups /pg/ wal_005 / 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 . l z o "
c o m p l e t e a t 2 1 5 8 3 . 3 KiB/ s .
INFO
DETAIL : Uploading
" pg_xlog / 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 . 0 0 0 0 0 0 2 0 . backup " t o
" s 3 : / / c l e v e r d b - pg - backups /pg/ wal_005 / 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 . 0 0 0 0 0 0 2 0 . b
INFO
" s 3 : / / c l e v e r d b - pg - backups /pg/ wal_005 / 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 . 0 0 0 0 0 0 2 0 . b
c o m p l e t e a t 00KiB/ s .
INFO
DETAIL : Uploading
" pg_xlog /000000010000000000000002 " t o
" s 3 : / / c l e v e r d b - pg - backups /pg/ wal_005 / 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 . l z o " .
INFO
" s 3 : / / c l e v e r d b - pg - backups /pg/ wal_005 / 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 . l z o "
c o m p l e t e a t 2 4 6 6 . 6 7 KiB/ s .
,
.
,
:
10.26 S3
Line 1
-
$ e n v d i r / e t c / wal - e . d/ env wal - e backup - push

/ var / l i b / p o s t g r e s q l / 9 . 2 / main
2 0 1 2 - 1 1 - 0 7 T14 : 4 9 : 2 6 . 1 7 4 + 0 0 p i d =7493
wal_e . o p e r a t o r . s 3 _ o p e r a t o r INFO
MSG: s t a r t upload
p o s t g r e s v e r s i o n metadata
DETAIL : Uploading t o
s 3 : / / c l e v e r d b - pg - backups /pg/ basebackups_005 / base_000000010000000000000006
2 0 1 2 - 1 1 - 0 7 T14 : 4 9 : 3 2 . 7 8 3 + 0 0 p i d =7493
wal_e . o p e r a t o r . s 3 _ o p e r a t o r INFO
MSG: p o s t g r e s
v e r s i o n metadata upload c o m p l e t e
INFO
MSG: b e g i n n i n g volume c o m p r e s s i o n
176
10.5.
DETAIL : B u i l d i n g volume 0 .
...
- HINT :
Check t h a t your archive_command i s e x e c u t i n g
p r o p e r l y . pg_stop_backup can be c a n c e l e d s a f e l y , but
t h e d a t a b a s e backup w i l l not be u s a b l e w i t h o u t a l l t h e
WAL segments .
- NOTICE :
pg_stop_backup complete , a l l r e q u i r e d WAL segments
have been a r c h i v e d
-
. 10.1: S3
. 10.2: S3
(,
crontab). 10.1-10.3 S3.
lzop1 .

gzip, ( 25/
5% ).
mbuffer ( cluster-read-rate-limit
, ).
.
backup-fetch :
10.27 S3
1
http://en.wikipedia.org/wiki/Lzop
177
10.5.
. 10.3: WAL- S3
Line 1
$ sudo - u p o s t g r e s bash - c " e n v d i r / e t c / wal - e . d/ env wal - e

- - s3 - p r e f i x=s 3 : / / some - bucket / d i r e c t o r y / o r / whatever
backup - f e t c h / var / l i b / p o s t g r e s q l / 9 . 2 / main LATEST"
LATEST
(PostgreSQL ).
:
10.28
Line 1
$ sudo - u p o s t g r e s bash - c " e n v d i r / e t c / wal - e . d/ env wal - e

- - s3 - p r e f i x=s 3 : / / some - bucket / d i r e c t o r y / o r / whatever
backup - f e t c h / var / l i b / p o s t g r e s q l / 9 . 2 / main
base_LONGWALNUMBER_POSITION_NUMBER"

backup-list:
10.29
$ e n v d i r / e t c / wal - e . d/ env wal - e backup - l i s t
name
last_modified
expanded_size_bytes
wal_segment_backup_start
wal_segment_offset_backup_start wal_segment_backup_stop
wal_segment_offset_backup_stop
- base_000000010000000000000008_00000032
2 0 1 2 - 1 1 - 0 7 T14 : 0 0 : 0 7 . 0 0 0 Z
000000010000000000000008
00000032
- base_00000001000000000000000C_00000032
2 0 1 2 - 1 1 - 0 8 T15 : 0 0 : 0 8 . 0 0 0 Z
00000001000000000000000C
00000032
Line 1
-
178
10.5.

WAL- (
). recovery.conf:
10.30 recovery.conf
Line 1
restore_command = e n v d i r / e t c / wal - e . d/ env

/ u s r / l o c a l / b i n / wal - e wal - f e t c h "% f " "%p"
PostgreSQL.
.
( )
delete:
10.31
#
base_00000004000002DF000000A6_03626144
- $ e n v d i r / e t c / wal - e . d/ env wal - e d e l e t e - - c o n f i r m b e f o r e
base_00000004000002DF000000A6_03626144
- #
- $ e n v d i r / e t c / wal - e . d/ env wal - e d e l e t e - - c o n f i r m e v e r y t h i n g
Line 1
confirm ,
, (dry
run).
WAL-E PostgreSQL
Amazon S3.
Barman
: www.pgbarman.org
Barman, WAL-E,
PostgreSQL
. Barman ,
,
PostgreSQL .

PostgreSQL (
) pghost.
(
brhost).
SSH ( , ).
authorized_keys .
179
10.5.
10.32 SSH
# PostgreSQL ( p g h o s t )
- s s h barman@brhost
- # ( b r h o s t )
- ssh postgres@pghost
Line 1
barman. barman
python : python 2.6+, rsync >= 3.0.4
python (argh, psycopg2, python-dateutil < 2.0 ( python 3.0
), distribute). Ubuntu
:
10.33 barman
Line 1
a p t i t u d e i n s t a l l python - dev python - argh python - psycopg2

python - d a t e u t i l r s y n c python - s e t u p t o o l s
barman:
10.34 barman
#
- #
- #
- #
Line 1
t a r - x z f barman - 1 . 1 . 1 . t a r . gz
cd barman - 1 . 1 . 1 /
python s e t u p . py b u i l d
sudo python s e t u p . py i n s t a l l
PostgreSQL. , barman
,
PostgreSQL:
10.35 postgresql.conf
Line 1
listen_adress = *
10.36 pg_hba.conf
Line 1
host
all
all
b r h o s t /32
trust
PostgreSQL.
PostgreSQL:
10.37
Line 1
5
$ p s q l - c SELECT v e r s i o n ( ) -U p o s t g r e s - h p g h o s t
version
-- ------------------------------------------------------------------------PostgreSQL 9 . 2 . 1 on x86_64 - unknown - l i n u x - gnu , c o m p i l e d by
g c c ( Ubuntu/ L i n a r o 4 . 7 . 2 - 2 ubuntu1 ) 4 . 7 . 2 , 64 - b i t
( 1 row )
180
10.5.
10.38
# sudo mkdir - p / s r v /barman
- # sudo chown barman : barman / s r v /barman
Line 1
barman /etc/barman.conf:
10.39 barman.conf
[ barman ]
; Main d i r e c t o r y
- barman_home = / s r v /barman
Line 1
5
-
; Log l o c a t i o n
l o g _ f i l e = / var / l o g /barman/barman . l o g
; D e f a u l t c o m p r e s s i o n l e v e l : p o s s i b l e v a l u e s a r e None
( d e f a u l t ) , bzip2 , g z i p o r custom
compression = gzip
10
-
; main PostgreSQL S e r v e r c o n f i g u r a t i o n
[ main ]
; Human r e a d a b l e d e s c r i p t i o n
d e s c r i p t i o n = "Main PostgreSQL Database "
15
; SSH o p t i o n s
- ssh_command = s s h p o s t g r e s @ p g h o s t
20
; PostgreSQL c o n n e c t i o n s t r i n g
c o n n i n f o = h o s t=p g h o s t u s e r=p o s t g r e s
main ( barman PostgreSQL )

PostgreSQL .
:
10.40 barman
Line 1
5
10
-
# barman show - s e r v e r main

S e r v e r main :
active : true
d e s c r i p t i o n : Main PostgreSQL Database
ssh_command : s s h p o s t g r e s @ p g h o s t
c o n n i n f o : h o s t=p g h o s t u s e r=p o s t g r e s
b a c k u p _ d i r e c t o r y : / s r v /barman/main
b a s e b a c k u p s _ d i r e c t o r y : / s r v /barman/main/ b a s e
w a l s _ d i r e c t o r y : / s r v /barman/main/ w a l s
i n c o m i n g _ w a l s _ d i r e c t o r y : / s r v /barman/main/ incoming
l o c k _ f i l e : / s r v /barman/main/main . l o c k
compression : gzip
c u s t o m _ c o m p r e s s i o n _ f i l t e r : None
c u s t o m _ d e c o m p r e s s i o n _ f i l t e r : None
181
10.5.
15
20
25
-
r e t e n t i o n _ p o l i c y : None
w a l _ r e t e n t i o n _ p o l i c y : None
pre_backup_script : None
post_backup_script : None
c u r r e n t _ x l o g : None
last_shipped_wal : None
archive_command : None
server_txt_version : 9 . 2 . 1
d a t a _ d i r e c t o r y : / var / l i b / p o s t g r e s q l / 9 . 2 / main
archive_mode : o f f
config_file :
/ e t c / p o s t g r e s q l / 9 . 2 / main/ p o s t g r e s q l . c o n f
h b a _ f i l e : / e t c / p o s t g r e s q l / 9 . 2 / main/pg_hba . c o n f
i d e n t _ f i l e : / e t c / p o s t g r e s q l / 9 . 2 / main/ pg_ident . c o n f
30
35
-
# barman check main

S e r v e r main :
s s h : OK
PostgreSQL : OK
archive_mode : FAILED ( p l e a s e s e t i t t o on )
archive_command : FAILED ( p l e a s e s e t i t a c c o r d i n g l y
t o documentation )
d i r e c t o r i e s : OK
c o m p r e s s i o n s e t t i n g s : OK
, PostgreSQL .
PostgreSQL :
10.41 PostgreSQL
w a l _ l e v e l = hot_standby # a r c h i v e PostgreSQL < 9 . 0
archive_mode = on
- archive_command = r s y n c - a %p
barman@brhost :INCOMING_WALS_DIRECTORY/% f
Line 1
-
INCOMING_WALS_DIRECTORY
WAL-.

barman show-server main( 10.40,
/srv/barman/main/incoming).

PostgreSQL. :
10.42
Line 1
5
-
# barman check main

S e r v e r main :
s s h : OK
PostgreSQL : OK
archive_mode : OK
archive_command : OK
d i r e c t o r i e s : OK
182
10.5.
c o m p r e s s i o n s e t t i n g s : OK
.
, barman.conf .
:
10.43
Line 1
-
# barman l i s t - s e r v e r
main - Main PostgreSQL Database
PostgreSQL (
):
10.44
Line 1
5
-
# barman backup main

S t a r t i n g backup f o r s e r v e r main i n
/ s r v /barman/main/ b a s e /20121109 T090806
Backup s t a r t a t x l o g l o c a t i o n : 0/3000020
(000000010000000000000003 , 00000020)
Copying f i l e s .
Copy done .
Asking PostgreSQL s e r v e r t o f i n a l i z e t h e backup .
Backup end a t x l o g l o c a t i o n : 0/30000D8
( 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 , 000000D8)
Backup completed
( cron).
:
10.45
# barman l i s t - backup main
main 20121110 T091608 - F r i Nov 10 0 9 : 2 0 : 5 8 2012 - S i z e : 1 . 0
GiB - WAL S i z e : 4 4 6 . 0 KiB
- main 20121109 T090806 - F r i Nov
9 0 9 : 0 8 : 1 0 2012 - S i z e :
2 3 . 0 MiB - WAL S i z e : 4 7 7 . 0 MiB
Line 1
-
:
10.46
Line 1
5
-
# barman show - backup main 20121110 T091608

Backup 20121109 T091608 :
S e r v e r Name
: main
Status :
: DONE
PostgreSQL V e r s i o n : 90201
PGDATA d i r e c t o r y
: / var / l i b / p o s t g r e s q l / 9 . 2 / main
183
10.5.
10
15
-
Base backup i n f o r m a t i o n :
Disk u s a g e
: 1 . 0 GiB
Timeline
: 1
Begin WAL
: 00000001000000000000008C
End WAL
: 000000010000000000000092
WAL number
: 7
Begin time
: 2012 -11 -10 0 9 : 1 6 : 0 8 . 8 5 6 8 8 4
End time
: 2012 -11 -10 0 9 : 2 0 : 5 8 . 4 7 8 5 3 1
Begin O f f s e t
: 32
End O f f s e t
: 3576096
Begin XLOG
: 0/8 C000020
End XLOG
: 0/92369120
20
-
WAL i n f o r m a t i o n :
No o f f i l e s
Disk u s a g e
Last a v a i l a b l e
: 1
: 4 4 6 . 0 KiB
: 000000010000000000000093
25
-
Catalog information :
P r e v i o u s Backup : 20121109 T090806
Next Backup
: - ( t h i s i s t h e l a t e s t b a s e backup )
WAL-,
cron:
10.47 WAL-
Line 1
5
-
# barman c r o n
P r o c e s s i n g x l o g segments f o r main
000000010000000000000001
000000010000000000000002
000000010000000000000003
0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 . 0 0 0 0 0 0 2 0 . backup
000000010000000000000004
000000010000000000000005
000000010000000000000006
cron.
, WAL- (
- ). Barman
WAL- gzip, bzip2 (
custom_compression_filter
custom_decompression_filter ).
recover:
10.48
Line 1
# barman r e c o v e r - - remote - ssh - command " s s h p o s t g r e s @ p g h o s t "

main 20121109 T090806 / var / l i b / p o s t g r e s q l / 9 . 2 / main
184
10.5.
5
-
S t a r t i n g remote r e s t o r e f o r s e r v e r main u s i n g backup

20121109 T090806
D e s t i n a t i o n d i r e c t o r y : / var / l i b / p o s t g r e s q l / 9 . 2 / main
Copying t h e b a s e backup .
Copying r e q u i r e d wal segments .
The archive_command was s e t t o f a l s e t o p r e v e n t data
losses .
Your PostgreSQL s e r v e r has been s u c c e s s f u l l y p r e p a r e d f o r

recovery !
10
-
P l e a s e r e v i e w network and a r c h i v e r e l a t e d s e t t i n g s i n t h e
PostgreSQL
c o n f i g u r a t i o n f i l e b e f o r e s t a r t i n g the j u s t recovered
instance .
WARNING: B e f o r e s t a r t i n g up t h e r e c o v e r e d PostgreSQL s e r v e r ,
p l e a s e review a l s o the s e t t i n g s of the f o l l o w i n g
configuration
15 o p t i o n s a s they might i n t e r f e r e with your c u r r e n t r e c o v e r y
attempt :
-
20
d a t a _ d i r e c t o r y = / var / l i b / p o s t g r e s q l / 9 . 1 / main
# u s e data i n a n o t h e r d i r e c t o r y
external_pid_file =
/ var / run / p o s t g r e s q l / 9 . 1 - main . p i d
# write
an e x t r a PID f i l e
h b a _ f i l e = / e t c / p o s t g r e s q l / 9 . 1 / main/pg_hba . c o n f
#
host - based a u t h e n t i c a t i o n f i l e
ident_file =
/ e t c / p o s t g r e s q l / 9 . 1 / main/ pg_ident . c o n f
# ident
configuration f i l e
Barman
SSH ( remote-ssh-command).
barman PITR1 :
target-time ( ) target-xid
(id ).
Barman
PostgreSQL . ,
PostgreSQL .
1
http://en.wikipedia.org/wiki/Point-in-time_recovery
185
10.6.
10.6
, ,
, .
,
PostgreSQL (,
).
186
11

PostgreSQL
,
(),

.
-
,
, .

11.1
,

. -
( ,
).

, SQL ,
, PostgreSQL ,
.
, .
: ?

( ..),
, ,
. , ,
.
187
11.2.

, - - ,
. ,
:
;
;
,
, ( Twitter Facebook
).
, .
11.2
,
, .
, ..
,
, .

PgPool-II v.3 + PostgreSQL v.9 Streaming Replication
,
. :

(failover)

(
pg_current_xlog_location pg_last_xlog_receive_location).
pgpool-II

PgPool-II v.3 + PostgreSQL Slony
, Slony. :
(failover)

.
pgpool-II
Postgresql 9
Postgres-XC 5.3 .
188
11.3.
11.3
,
( Google Analytics).
(
).

.
PgQ , PostgreSQL.
Skype.
Londiste
( 4.4). :

PostgreSQL
,

PgQ ,
,
(batches)
API SQL

RabbitMQ.
RabbitMQ ,
(Message Oriented
Middleware) AMQP (Advanced Message Queuing
Protocol).
RabbitMQ Mozilla Public License.
RabbitMQ Open Telecom Platform,

Erlang.
Postgres-XC 5.3 .
11.4

PostgreSQL.
, .
,
/
.
189
12

(Performance Snippets)

.
. -
.
, ,
.
(House M.D.),
1 1
12.1
PostgreSQL,
,
( ).
,
PostgreSQL.
,
PostgreSQL ,
.
12.2

(
).
190
12.2.
snippets/biggest_relations.sql
Line 1
5
-
SELECT nspname | | . | | relname AS " r e l a t i o n " ,

p g _ s i z e _ p r e t t y ( p g _ r e l a t i o n _ s i z e (C . o i d ) ) AS " s i z e "
FROM p g _ c l a s s C
LEFT JOIN pg_namespace N ON (N. o i d = C . r e l n a m e s p a c e )
WHERE nspname NOT IN ( pg_catalog , information_schema )
ORDER BY p g _ r e l a t i o n _ s i z e (C. o i d ) DESC
LIMIT 2 0 ;
:
12.1 .
Line 1
5
-
relation
|
size
- - - - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - public . accounts
| 326 MB
p u b l i c . accounts_pkey
| 44 MB
public . history
| 592 kB
public . tellers_pkey
| 16 kB
p u b l i c . branches_pkey
| 16 kB
public . t e l l e r s
| 16 kB
public . branches
| 8192 b y t e s

.
snippets/biggest_tables.sql
Line 1
5
-
SELECT nspname | | . | | relname AS " r e l a t i o n " ,

p g _ s i z e _ p r e t t y ( p g _ t o t a l _ r e l a t i o n _ s i z e (C . o i d ) ) AS
" total_size "
FROM p g _ c l a s s C
LEFT JOIN pg_namespace N ON (N. o i d = C . r e l n a m e s p a c e )
WHERE nspname NOT IN ( pg_catalog , information_schema )
AND C. r e l k i n d <> i
AND nspname ! ~ ^pg_toast
ORDER BY p g _ t o t a l _ r e l a t i o n _ s i z e (C. o i d ) DESC
LIMIT 2 0 ;
:
12.2 .
Line 1
5
relation
| total_size
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -+ - - - - - - - - - - - public . actions
| 4249 MB
p u b l i c . p r o d u c t _ h i s t o r y _ r e c o r d s | 197 MB
p u b l i c . product_updates
| 52 MB
191
12.2.
-
p u b l i c . import_products
public . products
public . v i s i t s
| 34 MB
| 29 MB
| 25 MB
count

. ,
count.
snippets/count_estimate.sql
Line 1
5
10
CREATE LANGUAGE p l p g s q l ;
CREATE FUNCTION c ou n t _ es t i ma t e ( query t e x t ) RETURNS i n t e g e r
AS $$
DECLARE
rec
record ;
rows i n t e g e r ;
BEGIN
FOR r e c IN EXECUTE EXPLAIN | | query LOOP
rows := s u b s t r i n g ( r e c . "QUERY PLAN" FROM
rows = ( [ [ : d i g i t : ] ] + ) ) ;
EXIT WHEN rows IS NOT NULL;
END LOOP;
RETURN rows ;
END;
- $$ LANGUAGE p l p g s q l VOLATILE STRICT ;
-
:
12.3 count.
CREATE TABLE f o o ( r d o u b l e p r e c i s i o n ) ;
- INSERT INTO f o o SELECT random ( ) FROM g e n e r a t e _ s e r i e s ( 1 ,
1000) ;
- ANALYZE f o o ;
Line 1
5
-
# SELECT count ( * ) FROM f o o WHERE r < 0 . 1 ;

count
-- ----92
( 1 row )
10
# SELECT c o u nt _ e st i m a te ( SELECT * FROM f o o WHERE r < 0 . 1 ) ;

coun t _ es t i ma t e
- -- -------------94
15 ( 1 row )
-
192
12.2.

( DEFAULT).
snippets/default_value.sql
Line 1
5
-
CREATE OR REPLACE FUNCTION r e t _ d e f ( t e x t , t e x t , t e x t ) RETURNS

t e x t AS $$
SELECT
COLUMNS. column_default : : t e x t
FROM
information_schema .COLUMNS
WHERE table_name = $2
AND table_schema = $1
AND column_name = $3
$$ LANGUAGE s q l IMMUTABLE;
:
12.4 .
Line 1
# SELECT r e t _ d e f ( schema , t a b l e , column ) ;
5
-
SELECT r e t _ d e f ( p u b l i c , i m a g e _ f i l e s , i d ) ;
ret_def
-- --------------------------------------nextval ( image_files_id_seq : : r e g c l a s s )
( 1 row )
SELECT r e t _ d e f ( p u b l i c , schema_migrations , v e r s i o n ) ;
ret_def
- -- -------
10
-
( 1 row )

(random)
( ).
snippets/random_from_range.sql
CREATE OR REPLACE FUNCTION random ( numeric , numeric )
RETURNS numeric AS
- $$
SELECT ( $1 + ( $2 - $1 ) * random ( ) ) : : numeric ;
5 $$ LANGUAGE s q l VOLATILE ;
Line 1
-
193
12.2.
12.5 .
Line 1
5
SELECT random ( 1 , 1 0 ) : : i n t , random ( 1 , 1 0 ) ;

random |
random
- - - - - - - -+ - - - - - - - - - - - - - - - - - 6 | 5.11675184825435
( 1 row )
10
-
SELECT random ( 1 , 1 0 ) : : i n t , random ( 1 , 1 0 ) ;

random |
random
- - - - - - - -+ - - - - - - - - - - - - - - - - - 7 | 1.37060070643201
( 1 row )

1
, .
, ,
, .
IBM
1960 .

,
.
SQL. ,
.
snippets/luhn_algorithm.sql
Line 1
5
10
CREATE OR REPLACE FUNCTION l u h n _ v e r i f y ( i n t 8 ) RETURNS

BOOLEAN AS $$
- - Take t h e sum o f t h e
- - doubled d i g i t s and t h e even - numbered undoubled d i g i t s ,
and s e e i f
- - t h e sum i s e v e n l y d i v i s i b l e by z e r o .
SELECT
- - Doubled d i g i t s might i n t u r n be two d i g i t s . In
that case ,
- - we must add each d i g i t i n d i v i d u a l l y r a t h e r than
adding t h e
- - doubled d i g i t v a l u e t o t h e sum . I e i f t h e
o r i g i n a l d i g i t was
- - 6 t h e doubled r e s u l t was 1 2 and we must add
1+2 t o t h e
- - sum r a t h e r than 1 2 .
1
http://en.wikipedia.org/wiki/Luhn_algorithm
194
12.2.
15
20
25
MOD(SUM( d o u b l e d _ d i g i t / INT8 10 + d o u b l e d _ d i g i t
% INT8 10 ) , 1 0 ) = 0
FROM
- - Double odd - numbered d i g i t s ( c o u n t i n g l e f t with
- - l e a s t s i g n i f i c a n t a s z e r o ) . I f t h e doubled d i g i t s end up
- - having v a l u e s
- - > 10 ( i e they r e two d i g i t s ) , add t h e i r d i g i t s t o g e t h e r .
(SELECT
- - E x t r a c t d i g i t n c o u n t i n g l e f t from l e a s t
significant
- - as zero
MOD( ( $1 : : i n t 8 / (10^ n ) : : i n t 8 ) , 1 0 : : i n t 8 )
- - Double odd - numbered d i g i t s
* (MOD( n , 2 ) + 1 )
AS d o u b l e d _ d i g i t
FROM g e n e r a t e _ s e r i e s ( 0 , CEIL (LOG( $1 ) ) : : INTEGER 1 ) AS n
) AS d o u b l e d _ d i g i t s ;
$$ LANGUAGE SQL
IMMUTABLE
- STRICT ;
-
30
35
40
45
-
COMMENT ON FUNCTION l u h n _ v e r i f y ( i n t 8 ) IS Return t r u e i f f

the l a s t d i g i t of the
i n p u t i s a c o r r e c t check d i g i t f o r t h e r e s t o f t h e i n p u t
a c c o r d i n g t o Luhn s
algorithm . ;
CREATE OR REPLACE FUNCTION l u h n _ g e n e r a t e _ c h e c k d i g i t ( i n t 8 )
RETURNS i n t 8 AS $$
SELECT
- - Add t h e d i g i t s , d o u b l i n g even - numbered d i g i t s
( counting l e f t
- - with l e a s t - s i g n i f i c a n t a s z e r o ) . S u b t r a c t t h e
remainder of
- - d i v i d i n g t h e sum by 10 from 1 0 , and t a k e t h e
remainder
- - o f d i v i d i n g t h a t by 10 i n t u r n .
( ( INT8 10 - SUM( d o u b l e d _ d i g i t / INT8 10 +
d o u b l e d _ d i g i t % INT8 10 ) %
INT8 10 ) % INT8 10 ) : : INT8
FROM (SELECT
- - E x t r a c t d i g i t n c o u n t i n g l e f t from l e a s t
significant\
- - as zero
MOD( ( $1 : : i n t 8 / (10^ n ) : : i n t 8 ) , 1 0 : : i n t 8 )
- - d o u b l e even - numbered d i g i t s
195
12.2.
50
* ( 2 - MOD( n , 2 ) )
AS d o u b l e d _ d i g i t
FROM g e n e r a t e _ s e r i e s ( 0 , CEIL (LOG( $1 ) ) : : INTEGER 1 ) AS n
) AS d o u b l e d _ d i g i t s ;
$$ LANGUAGE SQL
IMMUTABLE
- STRICT ;
-
55
60
-
COMMENT ON FUNCTION l u h n _ g e n e r a t e _ c h e c k d i g i t ( i n t 8 ) IS For

the input
value , g e n e r a t e a check d i g i t a c c o r d i n g t o Luhn s
algorithm ;
CREATE OR REPLACE FUNCTION l u h n _ g e n e r a t e ( i n t 8 ) RETURNS i n t 8
AS $$
SELECT 10 * $1 + l u h n _ g e n e r a t e _ c h e c k d i g i t ( $1 ) ;
$$ LANGUAGE SQL
IMMUTABLE
STRICT ;
65
70
-
COMMENT ON FUNCTION l u h n _ g e n e r a t e ( i n t 8 ) IS Append a check

d i g i t generated
a c c o r d i n g t o Luhn s a l g o r i t h m t o t h e i n p u t v a l u e . The
i n p u t v a l u e must be no
g r e a t e r than ( maxbigint / 1 0 ) . ;
CREATE OR REPLACE FUNCTION l u h n _ s t r i p ( i n t 8 ) RETURNS i n t 8 AS
$$
SELECT $1 / 1 0 ;
$$ LANGUAGE SQL
IMMUTABLE
STRICT ;
COMMENT ON FUNCTION l u h n _ s t r i p ( i n t 8 ) IS S t r i p t h e l e a s t
s i g n i f i c a n t d i g i t from
- t h e i n p u t v a l u e . I n t e n d e d f o r u s e when s t r i p p i n g t h e check
d i g i t from a number
75 i n c l u d i n g a Luhn s a l g o r i t h m check d i g i t . ;
-
:
12.6 .
Line 1
5
S e l e c t luhn_verify (49927398716) ;
luhn_verify
-- ----------t
( 1 row )
196
12.2.
10
-
S e l e c t luhn_verify (49927398714) ;
luhn_verify
-- ----------f
( 1 row )

IN.
, .
:
: (2,6,4,10,25,7,9)
.. 2 2 2 6 6 4 4
snippets/order_like_in.sql
SELECT f o o . * FROM f o o
- JOIN (SELECT i d . val , row_number ( ) o v e r ( ) FROM
(VALUES( 3 ) , ( 2 ) , ( 6 ) , ( 1 ) , ( 4 ) ) AS
- i d ( v a l ) ) AS i d
- ON ( f o o . c a t a l o g _ i d = i d . v a l ) ORDER BY row_number ;
Line 1
VALUES(3),(2),(6),(1),(4)
foo ,
foo.catalog_id (
foo.catalog_id IN(3,2,6,1,4))
,
, (. quine) (
),
.
snippets/quine.sql
Line 1
-
s e l e c t a | | from ( s e l e c t | | q u o t e _ l i t e r a l ( a ) | | b | | ,
| | q u o t e _ l i t e r a l ( b ) | | : : t e x t a s b ) a s q u i n e from
( s e l e c t s e l e c t a | | from ( s e l e c t | | q u o t e _ l i t e r a l ( a )
| | b | | , | | q u o t e _ l i t e r a l (b) | | : : text as b) as
quine : : text as a , : : text as a : : text as b) as quine ;
LIKE
web .
LIKE some%, some ,
197
12.2.
. , ( )
.
LIKE bla%
text_pattern_ops ( varchar_pattern_ops varchar).
snippets/speed_like.sql
Line 1
5
10
-
p r e f i x _ t e s t=# c r e a t e t a b l e t a g s (
p r e f i x _ t e s t (# t a g
t e x t primary key ,
p r e f i x _ t e s t (# name
t e x t not n u l l ,
p r e f i x _ t e s t (# shortname t e x t ,
p r e f i x _ t e s t (# s t a t u s
char d e f a u l t S ,
p r e f i x _ t e s t (#
p r e f i x _ t e s t (# check ( s t a t u s i n ( S , R ) )
p r e f i x _ t e s t (# ) ;
NOTICE : CREATE TABLE / PRIMARY KEY w i l l c r e a t e i m p l i c i t
i n d e x " tags_pkey " f o r t a b l e " t a g s "
CREATE TABLE
p r e f i x _ t e s t=# CREATE INDEX i_tag ON t a g s USING
b t r e e ( l o w e r ( t a g ) text_pattern_ops ) ;
CREATE INDEX
15
20
-
p r e f i x _ t e s t=# c r e a t e t a b l e i n v a l i d _ t a g s (
p r e f i x _ t e s t (# t a g
t e x t primary key ,
p r e f i x _ t e s t (# name
t e x t not n u l l ,
p r e f i x _ t e s t (# shortname t e x t ,
p r e f i x _ t e s t (# s t a t u s
char d e f a u l t S ,
p r e f i x _ t e s t (#
p r e f i x _ t e s t (# check ( s t a t u s i n ( S , R ) )
p r e f i x _ t e s t (# ) ;
NOTICE : CREATE TABLE / PRIMARY KEY w i l l c r e a t e i m p l i c i t
index " invalid_tags_pkey " f o r t a b l e " invalid_tags "
CREATE TABLE
25
30
-
p r e f i x _ t e s t=# s e l e c t count ( * ) from t a g s ;

count
-- ----11966
( 1 row )
35
-
p r e f i x _ t e s t=# s e l e c t count ( * ) from i n v a l i d _ t a g s ;

count
-- ----11966
( 1 row )
198
12.2.
40
45
# EXPLAIN ANALYZE s e l e c t * from i n v a l i d _ t a g s where

l o w e r ( t a g ) LIKE l o w e r ( 0146% ) ;
QUERY PLAN
-- ------------------------------------------------------------------------Seq Scan on i n v a l i d _ t a g s ( c o s t = 0 . 0 0 . . 2 6 5 . 4 9 rows=60
width =26) ( a c t u a l time = 0 . 3 5 9 . . 2 0 . 6 9 5 rows=1 l o o p s =1)
F i l t e r : ( l o w e r ( t a g ) ~~ 0146% : : t e x t )
T o t a l runtime : 2 0 . 8 0 3 ms
( 3 rows )
50
-
# EXPLAIN ANALYZE s e l e c t * from i n v a l i d _ t a g s where

l o w e r ( t a g ) LIKE l o w e r ( 0146% ) ;
QUERY PLAN
-- ------------------------------------------------------------------------Seq Scan on i n v a l i d _ t a g s ( c o s t = 0 . 0 0 . . 2 6 5 . 4 9 rows=60
width =26) ( a c t u a l time = 0 . 5 4 9 . . 1 9 . 5 0 3 rows=1 l o o p s =1)
T o t a l runtime : 1 9 . 5 5 0 ms
( 3 rows )
55
60
-
# EXPLAIN ANALYZE s e l e c t * from t a g s where l o w e r ( t a g ) LIKE

l o w e r ( 0146% ) ;
QUERY
PLAN
-- ------------------------------------------------------------------------Bitmap Heap Scan on t a g s ( c o s t = 5 . 4 9 . . 9 7 . 7 5 rows =121
width =26) ( a c t u a l time = 0 . 0 5 4 . . 0 . 0 5 7 rows=1 l o o p s =1)
-> Bitmap Index Scan on i_tag ( c o s t = 0 . 0 0 . . 5 . 4 6 rows =120
Index Cond : ( ( l o w e r ( t a g ) ~>=~ 0146 : : t e x t ) AND
( l o w e r ( t a g ) ~<~ 0147 : : t e x t ) )
T o t a l runtime : 0 . 1 1 9 ms
( 5 rows )
65
70
-
# EXPLAIN ANALYZE s e l e c t * from t a g s where l o w e r ( t a g ) LIKE

l o w e r ( 0146% ) ;
QUERY
PLAN
-- ------------------------------------------------------------------------Bitmap Heap Scan on t a g s ( c o s t = 5 . 4 9 . . 9 7 . 7 5 rows =121
-> Bitmap Index Scan on i_tag ( c o s t = 0 . 0 0 . . 5 . 4 6 rows =120
Index Cond : ( ( l o w e r ( t a g ) ~>=~ 0146 : : t e x t ) AND
( l o w e r ( t a g ) ~<~ 0147 : : t e x t ) )
199
12.2.
-
T o t a l runtime : 0 . 0 5 0 ms
( 5 rows )

,
( , ).
snippets/duplicate_indexes.sql
Line 1
5
SELECT p g _ s i z e _ p r e t t y ( sum ( p g _ r e l a t i o n _ s i z e ( i d x ) ) : : b i g i n t )
AS s i z e ,
( array_agg ( i d x ) ) [ 1 ] AS idx1 , ( array_agg ( i d x ) ) [ 2 ] AS
idx2 ,
( array_agg ( i d x ) ) [ 3 ] AS idx3 , ( array_agg ( i d x ) ) [ 4 ] AS
idx4
FROM (
SELECT i n d e x r e l i d : : r e g c l a s s AS idx , ( i n d r e l i d : : t e x t
| | E \n | | i n d c l a s s : : t e x t | | E \n | | i n d k e y : : t e x t | | E \n | |
c o a l e s c e ( i n d e x p r s : : t e x t , ) | | E \n | |
c o a l e s c e ( i n d p r e d : : t e x t , ) ) AS KEY
FROM pg_index ) sub
- GROUP BY KEY HAVING count ( * )>1
- ORDER BY sum ( p g _ r e l a t i o n _ s i z e ( i d x ) ) DESC;

snippets/indexes_statustic.sql
SELECT
t . tablename ,
indexname ,
c . r e l t u p l e s AS num_rows ,
Line 1
-
p g _ s i z e _ p r e t t y ( p g _ r e l a t i o n _ s i z e ( quote_ident ( t . tablename ) : : t e x t ) )
AS t a b l e _ s i z e ,
-
10
-
p g _ s i z e _ p r e t t y ( p g _ r e l a t i o n _ s i z e ( quote_ident ( i n d e x r e l n a m e ) : : t e x t ) )
AS i n d e x _ s i z e ,
CASE WHEN x . i s _ u n i q u e = 1 THEN Y
ELSE N
END AS UNIQUE,
idx_scan AS number_of_scans ,
idx_tup_read AS t u p l e s _ r e a d ,
idx_tup_fetch AS t u p l e s _ f e t c h e d
FROM p g _ t a b l e s t
200
12.2.
15
20
-
25
-
LEFT OUTER JOIN p g _ c l a s s c ON t . tablename=c . relname

LEFT OUTER JOIN
(SELECT i n d r e l i d ,
max(CAST( i n d i s u n i q u e AS i n t e g e r ) ) AS i s _ u n i q u e
FROM pg_index
GROUP BY i n d r e l i d ) x
ON c . o i d = x . i n d r e l i d
LEFT OUTER JOIN
( SELECT c . relname AS ctablename , i p g . relname AS
indexname , x . i n d n a t t s AS number_of_columns , idx_scan ,
idx_tup_read , idx_tup_fetch , i n d e x r e l n a m e FROM pg_index x
JOIN p g _ c l a s s c ON c . o i d = x . i n d r e l i d
JOIN p g _ c l a s s i p g ON i p g . o i d = x . i n d e x r e l i d
JOIN p g _ s t a t _ a l l _ i n d e x e s p s a i ON x . i n d e x r e l i d =
psai . indexrelid )
AS f o o
ON t . tablename = f o o . ctablename
WHERE t . schemaname= p u b l i c
ORDER BY 1 , 2 ;
201

[1]
(Sad
Spirit)
borz_off@cs.msu.su
PostgreSQL:
http://www.phpclub.ru/detail/store/pdf/postgresql-performance.pdf
[2] Eugene
Kuzin
eugene@kuzin.net
PostgreSQL
Slony-I
http://www.kuzin.net/work/sloniki-privet.html
[3] Sergey Konoplev gray.ru@gmail.com Londiste
http://gray-hemp.blogspot.com/2010/04/londiste.html
[4] Dmitry
Stasyuk
pgpool-II
http://undenied.ru/2009/03/04/uchebnoe-rukovodstvo-po-pgpool-ii/
[5]
dmitry.chirkin@gmail.com
PostgreSQL
PL/Proxy
http://habrahabr.ru/blogs/postgresql/45475/
[6]
wordpress@insight-it.ru
http://www.insight-it.ru/masshtabiruemost/hadoop/
Hadoop
[7] Padraig
OSullivan
Up
and
Running
with
HadoopDB
http://posulliv.github.com/2010/05/10/hadoopdb-mysql.html
[8] PostgreSQL:
Skype http://postgresmen.ru/articles/view/25
[9] Streaming Replication. http://wiki.postgresql.org/wiki/Streaming_Replication
[10] Den Golotyuk , , -

? http://highload.com.ua/index.php/2009/05/06/--
[11] Postgres-XC
A
PostgreSQL
Clustering
Solution
http://www.linuxforu.com/2012/01/postgres-xc-database-clustering-solution/
202

Работа с PostgreSQL. Настройка и масштабирование

Загружено:

Сведения о документе

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Работа с PostgreSQL. Настройка и масштабирование

Загружено:

Авторское право:

Доступные форматы

PostgreSQL:

shared_buffers = 1/8 RAM ( 1/4);

- i , -- input-config postgresql.conf, -o,

EXPLAIN ANALYZE DELETE . . .

SELECT count ( * ) FROM f o o ;

RULE PostgreSQL SQL, ,

SELECT r e l t u p l e s FROM p g _ c l a s s WHERE relname = f o o ;

p o s t g r e s=# s e l e c t count ( * ) from ( s e l e c t d i s t i n c t i from g )

p o s t g r e s=# s e l e c t count ( * ) from ( s e l e c t d i s t i n c t i from g )

p o s t g r e s=# s e l e c t count ( * ) from ( s e l e c t i from g group by

p o s t g r e s=# s e l e c t count ( * ) from ( s e l e c t i from g group by

p g f o u i n e . php - f i l e your / l o g / f i l e . l o g > your - r e p o r t . html

p g f o u i n e . php - f i l e your / l o g / f i l e . l o g - top 10 - format t e x t

pg_stat_statements . max = 10000

# CREATE EXTENSION pg_stat_statements ;

# SELECT query , c a l l s , t o t a l _ t i m e , rows , 1 0 0 . 0 *

# SELECT query , c a l l s , t o t a l _ t i m e , rows , 1 0 0 . 0 *

CHECK ( o u t l e t I D BETWEEN 100 AND 200 )

CREATE TABLE my_logs (

CREATE TABLE my_logs2010m10

CREATE TABLE my_logs2010m12

CREATE OR REPLACE FUNCTION m y _ l o g s _ i n s e r t _ t r i g g e r ( )

RAISE EXCEPTION Date out o f r a n g e .

p a r t i t i o n i n g _ t e s t=# SELECT * FROM ONLY my_logs ;

p a r t i t i o n i n g _ t e s t=# SELECT * FROM my_logs ;

p a r t i t i o n i n g _ t e s t=# S e l e c t * from my_logs2010m10 ;

p a r t i t i o n i n g _ t e s t=# S e l e c t * from my_logs2010m11 ;

p a r t i t i o n i n g _ t e s t=# SELECT * FROM my_logs WHERE u s e r _ i d = 2 ;

p a r t i t i o n i n g _ t e s t=# SELECT * FROM my_logs WHERE data LIKE

DROP TABLE my_logs2008m10 ;

ALTER TABLE my_logs2008m10 NO INHERIT my_logs ;

Index Cond : ( l o g d a t e > 2 0 1 0 - 1 2 - 0 1

4.2. Streaming Replication ( )

4.2 Streaming Replication ( )

(Streaming Replication, SR)

4.2. Streaming Replication ( )

4.2. Streaming Replication ( )

p o s t g r e s @ l o c a l h o s t ~ $ ssh - keygen - t r s a -P ""

c a t $HOME/ . s s h / i d _ r s a . pub >> $HOME/ . s s h / a u t h o r i z e d _ k e y s

4.2. Streaming Replication ( )

# S e t t h e maximum number o f c o n c u r r e n t c o n n e c t i o n s from t h e

# To p r e v e n t t h e primary s e r v e r from removing t h e WAL

# Enable WAL a r c h i v i n g on t h e primary t o an a r c h i v e

4.2. Streaming Replication ( )

p s q l - c "SELECT pg_start_backup ( l a b e l , t r u e ) "

4.2. Streaming Replication ( )

p s q l - c "SELECT pg_stop_backup ( ) "

# S p e c i f i e s a command t o l o a d a r c h i v e segments from t h e WAL

4.2. Streaming Replication ( )

restore_command = s c p masterdb_host : / path_to / a r c h i v e/% f

$ psql - c " s e l e c t pg_last_xlog_receive_location () "

$ psql - c " s e l e c t pg_last_xlog_replay_location () "

4.2. Streaming Replication ( )

9.1 view pg_stat_database_conflicts,

4.2. Streaming Replication ( )

echo Adding t a b l e p u b l i c . customers_something . . . ;

echo done adding ;

store l i s t e n ( origin = 1 , provider = 1 , receiver = 2 ) ;

echo done adding ;

s t o r e node ( i d = 3 , comment = Node 3 , $HOST3 ) ;

%s l o n customers_rep "dbname=c u s t o m e r s u s e r=s l o n y _ u s e r "

WARN remoteWorker_wakeup : node 1 - no worker t h r e a d

s e l e c t e v _ o r i g i n , ev_seqno , ev_timestamp , ev_minxid ,

d e l e t e from _customers_rep . s l _ e v e n t where