Skip to content

Commit 2ae23dd

Browse files
digoal zhoudigoal zhou
digoal zhou
authored and
digoal zhou
committed
new doc
1 parent 5cf7567 commit 2ae23dd

File tree

6 files changed

+262
-0
lines changed

6 files changed

+262
-0
lines changed

‎202201/20220119_01.md‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
## PostgreSQL 15 preview - pg_basebackup 增强, 支持扩展COPY协议
2+
3+
### 作者
4+
digoal
5+
6+
### 日期
7+
2022-01-19
8+
9+
### 标签
10+
PostgreSQL , pg_basebackup
11+
12+
----
13+
14+
## 背景
15+
pg_basebackup 扩展协议, 将来将支持更好的进度展示、压缩格式、等.
16+
17+
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=cc333f32336f5146b75190f57ef587dff225f565
18+
19+
```
20+
Modify pg_basebackup to use a new COPY subprotocol for base backups.
21+
author Robert Haas <[email protected]>
22+
Tue, 18 Jan 2022 18:47:26 +0000 (13:47 -0500)
23+
committer Robert Haas <[email protected]>
24+
Tue, 18 Jan 2022 18:47:49 +0000 (13:47 -0500)
25+
commit cc333f32336f5146b75190f57ef587dff225f565
26+
tree 8bcc3bf1b50cf4cbfe597a519b37e7aa8fcc5bde tree
27+
parent 3414099c338bf619c00dd82d96f29a9668a3bf07 commit | diff
28+
Modify pg_basebackup to use a new COPY subprotocol for base backups.
29+
30+
In the new approach, all files across all tablespaces are sent in a
31+
single COPY OUT operation. The CopyData messages are no longer raw
32+
archive content; rather, each message is prefixed with a type byte
33+
that describes its purpose, e.g. 'n' signifies the start of a new
34+
archive and 'd' signifies archive or manifest data. This protocol
35+
is significantly more extensible than the old approach, since we can
36+
later create more message types, though not without concern for
37+
backward compatibility.
38+
39+
The new protocol sends a few things to the client that the old one
40+
did not. First, it sends the name of each archive explicitly, instead
41+
of letting the client compute it. This is intended to make it easier
42+
to write future patches that might send archives in a format other
43+
that tar (e.g. cpio, pax, tar.gz). Second, it sends explicit progress
44+
messages rather than allowing the client to assume that progress is
45+
defined by the number of bytes received. This will help with future
46+
features where the server compresses the data, or sends it someplace
47+
directly rather than transmitting it to the client.
48+
49+
The old protocol is still supported for compatibility with previous
50+
releases. The new protocol is selected by means of a new
51+
TARGET option to the BASE_BACKUP command. Currently, the
52+
only supported target is 'client'. Support for additional
53+
targets will be added in a later commit.
54+
55+
Patch by me. The patch set of which this is a part has had review
56+
and/or testing from Jeevan Ladhe, Tushar Ahuja, Suraj Kharage,
57+
Dipesh Pandit, and Mark Dilger.
58+
59+
Discussion: http://postgr.es/m/CA+TgmoaYZbz0=Yk797aOJwkGJC-LK3iXn+wzzMx7KdwNpZhS5g@mail.gmail.com
60+
```
61+
62+
https://www.postgresql.org/docs/devel/protocol-replication.html
63+
64+
```
65+
new archive (B)
66+
Byte1('n')
67+
Identifes the messaage as indicating the start of a new archive.
68+
69+
String
70+
The file name for this archive.
71+
72+
String
73+
For the main data directory, an empty string. For other tablespaces, the full path to the directory from which this archive was created.
74+
75+
manifest (B)
76+
Byte1('m')
77+
Identifes the message as indicating the start of the backup manifest.
78+
79+
archive or manifest data (B)
80+
Byte1('d')
81+
Identifes the message as containing archive or manifest data.
82+
83+
Byten
84+
Data bytes.
85+
86+
progress report (B)
87+
Byte1('p')
88+
Identifes the message as a progress report.
89+
90+
Int64
91+
The number of bytes from the current tablespace for which processing has been completed.
92+
```
93+
94+
95+
96+
#### [期望 PostgreSQL 增加什么功能?](https://github.com/digoal/blog/issues/76 "269ac3d1c492e938c0191101c7238216")
97+
98+
99+
#### [PolarDB for PostgreSQL云原生分布式开源数据库](https://github.com/ApsaraDB/PolarDB-for-PostgreSQL "57258f76c37864c6e6d23383d05714ea")
100+
101+
102+
#### [PostgreSQL 解决方案集合](https://yq.aliyun.com/topic/118 "40cff096e9ed7122c512b35d8561d9c8")
103+
104+
105+
#### [德哥 / digoal's github - 公益是一辈子的事.](https://github.com/digoal/blog/blob/master/README.md "22709685feb7cab07d30f30387f0a9ae")
106+
107+
108+
![digoal's wechat](../pic/digoal_weixin.jpg "f7ad92eeba24523fd47a6e1a0e691b59")
109+

‎202201/20220119_02.md‎

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
## 一起学PolarDB - 第19期 - 为什么做检查点会导致性能抖动?
2+
3+
### 作者
4+
digoal
5+
6+
### 日期
7+
2022-01-19
8+
9+
### 标签
10+
PostgreSQL , PolarDB
11+
12+
----
13+
14+
## 背景
15+
懂PostgreSQL, 学PolarDB不难, 就好像有九阳神功护体, 可以快速融会贯通.
16+
对于DBA只要学会PolarDB精髓即可.
17+
对于开发者来说不需要学习, 使用PolarDB和PostgreSQL一样.
18+
19+
#### 为什么做检查点会导致性能抖动?
20+
21+
社区版本:
22+
- 创建检查点
23+
- 标记当前LSN为LSN1 , page在这之后第一次被修改时都要写full page (如果开启了full page write)
24+
- 写FULL PAGE会增加WAL的写入压力
25+
- 扫描shared buffer, 标记LSN < LSN1的脏页 FLAG, 表示这些页需要fsync到磁盘
26+
- 遍历buffer, 有点耗费资源
27+
- 开始对已标记的page进行持久化动作 (根据checkpointer调度参数控制刷脏速度)
28+
- 刷脏, 非常耗费资源, 特别是写IO
29+
- 写控制文件, 标记检查点逻辑开始与结束位置
30+
31+
优化:
32+
1、拉长检查点周期:
33+
- 设置较大的max_wal_size, 设置触发检查点的最大wal日志数据量, max_wal_size越大, 两次检查点的时间跨度越大.
34+
- 推荐设置 `max_wal_size = 3*shared_buffers`
35+
- 设置较大的checkpoint_completion_target, 一次检查点预计什么时候做完, 例如上一次检查点距离这次检查点之间生成了10GB WAL, checkpoint_completion_target = 0.9 表示这次检查点计划在`10*0.9`也就是生成9GB的WAL的过程中做完. checkpoint_completion_target越大, 检查点刷脏的力度越稀疏. checkpoint_completion_target越小, 检查点刷脏的力度越密集.
36+
37+
弊端, 以上方法虽然可以减轻检查点抖动, 但是也有一定的负面影响:
38+
- 检查点很长, 如果数据库崩溃很可能最近一次检查点还没做完, 那么数据库得从上一次检查点的逻辑开始位点开始应用WAL, 需要应用很多WAL文件.
39+
- 例如检查点跨度周期是16GB, 配置为0.9, 有90%的可能奔溃时检查点还没有做完, 所以要应用16GB以上的WAL日志. 如果崩溃时刚好做到80%, 则需要应用`16+0.8*16GB` wal日志.
40+
41+
2、调小shared buffer
42+
弊端: 虽然检查点带来的性能抖动影响变小了, 但是影响数据库整体的读写性能, 内存没有用起来.
43+
44+
PolarDB:
45+
https://github.com/ApsaraDB/PolarDB-for-PostgreSQL/blob/POLARDB_11_STABLE/docs/zh/architecture/buffer-management.md
46+
47+
1、PolarDB采用DIO
48+
49+
2、PolarDB增加了一个list存储每个buffer page的oldest LSN, 按LSN从小到大顺序刷buffer, 整个shared buffer内最小的oldest LSN就是一致性位点(临界点, 崩溃恢复时从这个位置开始获取WAL进行恢复), 因为这个位点之前的buffer page都持久化了.
50+
PolarDB的检查点已经不需要刷脏, 而是更新oldest LSN这个逻辑位点即可, 所以PolarDB做检查点不影响性能.
51+
![pic](20220119_02_pic_001.png)
52+
53+
3、PolarDB通过并行bgwriter来提高刷脏速度. 目前PG社区版本bgwriter为单进程模型.
54+
![pic](20220119_02_pic_002.png)
55+
56+
本期问题1:
57+
PG社区版本做检查点时引起性能抖动的主要原因有哪些?
58+
- a. full page write
59+
- b. 遍历shared buffer标记需要刷的脏页
60+
- c. 写控制文件
61+
- d. 将已标记的脏页刷到持久化存储
62+
63+
答案:
64+
- abd
65+
66+
解释:
67+
- 参考本文内容
68+
69+
本期问题2:
70+
PG社区版本可以如何优化检查点的稳定性?
71+
- a. 设置较大的max_wal_size, 拉长检查点的周期
72+
- b. 调大checkpoint_completion_target, 从而设置检查点的刷脏平滑度
73+
- c. 设置较小的shared buffer
74+
- d. 设置较大的shared buffer
75+
76+
答案:
77+
- abc
78+
79+
解释:
80+
- 参考本文内容
81+
82+
本期问题3:
83+
PG社区版本拉长检查点会带来什么负面影响?
84+
- a. 导致数据读取性能变差
85+
- b. 崩溃恢复时可能需要恢复非常多的WAL文件, 导致崩溃恢复变慢, 影响可用性.
86+
- c. 导致数据写入性能变差
87+
- d. 导致产生更多的WAL日志
88+
89+
答案:
90+
- b
91+
92+
解释:
93+
- 参考本文内容
94+
95+
本期问题4:
96+
PolarDB 检查点要做哪些动作?
97+
- a. 将脏页刷到持久化存储
98+
- b. 将shared buffer中的oldest LSN作为检查点的一致性位点(临界点)
99+
- c. 冻结数据库的写入操作
100+
- d. 结束所有的进行中事务
101+
102+
答案:
103+
- b
104+
105+
解释:
106+
- 参考本文内容
107+
108+
本期问题5:
109+
PolarDB 一致性位点是通过什么机制来实现的?
110+
- a. 新增一个list结构按LSN从小到大顺序存储shared buffer脏页每个page的oldest LSN, bgwriter按oldest LSN从小到大的顺序刷脏. shared buffer中最小的oldest LSN就是数据库一致性位点(临界点)
111+
- b. 通过checkpoint 刷脏, 将此次刷脏开始时的LSN作为一致性位点.
112+
- c. 通过将last LSN设置为一致性LSN来保证一致性位点.
113+
- d. 将已标记的脏页刷到持久化存储
114+
115+
答案:
116+
- a
117+
118+
解释:
119+
- 参考本文内容
120+
121+
本期问题6:
122+
PolarDB 通过什么机制来提高bgwriter刷脏速度?
123+
- a. 并行 bgwriter
124+
- b. 批量刷脏页
125+
- c. 提高bgwriter的刷脏频率
126+
- d. 将bgwriter的刷脏任务分配给checkpointer和background process一起来执行
127+
128+
答案:
129+
- a
130+
131+
解释:
132+
- 参考本文内容
133+
134+
135+
136+
#### [期望 PostgreSQL 增加什么功能?](https://github.com/digoal/blog/issues/76 "269ac3d1c492e938c0191101c7238216")
137+
138+
139+
#### [PolarDB for PostgreSQL云原生分布式开源数据库](https://github.com/ApsaraDB/PolarDB-for-PostgreSQL "57258f76c37864c6e6d23383d05714ea")
140+
141+
142+
#### [PostgreSQL 解决方案集合](https://yq.aliyun.com/topic/118 "40cff096e9ed7122c512b35d8561d9c8")
143+
144+
145+
#### [德哥 / digoal's github - 公益是一辈子的事.](https://github.com/digoal/blog/blob/master/README.md "22709685feb7cab07d30f30387f0a9ae")
146+
147+
148+
![digoal's wechat](../pic/digoal_weixin.jpg "f7ad92eeba24523fd47a6e1a0e691b59")
149+

‎202201/20220119_02_pic_001.png‎

43.8 KB
Loading

‎202201/20220119_02_pic_002.png‎

86.6 KB
Loading

‎202201/readme.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22

33
### 文章列表
44
----
5+
##### 20220119_02.md [《一起学PolarDB - 第19期 - 为什么做检查点会导致性能抖动?》](20220119_02.md)
6+
##### 20220119_01.md [《PostgreSQL 15 preview - pg_basebackup 增强, 支持扩展COPY协议》](20220119_01.md)
57
##### 20220118_05.md [《一起学PolarDB - 第18期 - 为什么创建索引慢?》](20220118_05.md)
68
##### 20220118_04.md [《开源项目的活跃度、用户、开发者贡献度 计算方法》](20220118_04.md)
79
##### 20220118_03.md [《一个网页的 "流量能力、引流能力、权威性" 评判维度》](20220118_03.md)

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,8 @@ digoal's|PostgreSQL|文章|归类
8787

8888
### 所有文档如下
8989
----
90+
##### 202201/20220119_02.md [《一起学PolarDB - 第19期 - 为什么做检查点会导致性能抖动?》](202201/20220119_02.md)
91+
##### 202201/20220119_01.md [《PostgreSQL 15 preview - pg_basebackup 增强, 支持扩展COPY协议》](202201/20220119_01.md)
9092
##### 202201/20220118_05.md [《一起学PolarDB - 第18期 - 为什么创建索引慢?》](202201/20220118_05.md)
9193
##### 202201/20220118_04.md [《开源项目的活跃度、用户、开发者贡献度 计算方法》](202201/20220118_04.md)
9294
##### 202201/20220118_03.md [《一个网页的 "流量能力、引流能力、权威性" 评判维度》](202201/20220118_03.md)

0 commit comments

Comments
 (0)