Polardb故障处理分享

随着国产数据库Polardb的推广及应用，数据库故障开始增多，今天给大家分享一篇Polardb故障应急处理思路。10月下旬，应用反馈某模块登陆异常，查看数据库有大量进程active状态，持续时间超过30分钟，通过kill进程并重启应用无法恢复，通过与业务协商、尝试通过重启数据库实例恢复。

查看日志文件postgresql-10-26_024511.log，日志中存在以下日志：

ERROR: dsa_area could not attach to segment

涉及进程pid= 51789 53315 53316，其中53316和53315进程都退出了，日志如下：

  53315              2020-10-26 03:07:11 UTC XX000                      ERROR:  dsa_area could not attach to segment

 53316            2020-10-26 03:07:11 UTC XX000                      ERROR:  dsa_area could not attach to segment

......

 53314            2020-10-26 03:07:11 UTC XX000                      FATAL:  cannot unpin a segment that is not pinned

   54             2020-10-26 03:07:11 UTC 00000                       LOG: background worker "parallel worker" (PID53316) exited withexit code 1

   54             2020-10-26 03:07:11 UTC 00000                       LOG: background worker "parallel worker" (PID53315) exitedwith exit code 1

而pid=51789的进程没有正常退出，很有可能是hang，并且影响到其他的进程。

……

51789 192.168.7.36(41423)  ywxt  ywxt  2020-10-26 03:07:07 UTC  00000  13 0  0  0    LOG:  execute S_1: COMMIT

 51789 192.168.7.36(41423)  ywxt  ywxt  2020-10-26 03:07:09 UTC  00000  43 0  0  0    LOG:  statement: BEGIN

 51789 192.168.7.36(41423)  ywxt  ywxt  2020-10-26 03:07:09 UTC  00000  59 1  0  40    LOG:  execute :  SELECTordersum,onestatus,twostatus,threestatus,fourstatus,archive FROM(……………… )

 51789 192.168.7.36(41423)  ywxt  ywxt  2020-10-26 03:07:09 UTC  00000  22 0  0  0    LOG:  execute S_1: COMMIT

 51789 192.168.7.36(41423)  ywxt  ywxt  2020-10-26 03:07:11 UTC  00000  35 0  0  0    LOG:  statement: BEGIN

 51789 192.168.7.36(41423)  ywxt  ywxt  2020-10-26 03:07:11 UTC  00000 69701  1  0  170881    LOG:  execute : SELECT count(1)FROM (SELECT ………………) XTABLE

 51789 192.168.7.36(41423)  ywxt  ywxt  2020-10-26 03:07:11 UTC  XX000           ERROR:  dsa_area could not attach to segment

 51789 192.168.7.36(41423)  ywxt  ywxt  2020-10-26 03:07:11 UTC  XX000           STATEMENT:  SELECT * FROM (SELECT RET.*, ROWNUM AS FSDPRN FROM(………………)WHERE FSDPRN BETWEEN 1 AND 10

 51789 192.168.7.36(41423)  ywxt  ywxt  2020-10-26 03:07:11 UTC  XX000           LOG:  statement: SELECT * FROM (SELECT RET.*, ROWNUM AS FSDPRNFROM (………………) WHERE FSDPRN BETWEEN 1 AND 10

 params:$1 = d6d9c702-8aa9-4e9b-8cc2-4d756b1adaa2, $2 = 202009, $3 =HLW, $4 = 15255864202, $5 =1a251789-9cc9-4e77-84cd-6b651da3be5a, $6 = 2020-09-14 11:06:18

 params:$1 = 40f12349-97cc-4945-94b1-e7fda75cc984, $2 = 202009, $3 =HLW, $4 = 18326858272, $5 =eaceb00f-9f78-4c6f-8cee-517898f0b200, $6 = 2020-09-15 15:16:40

查过文档发现是属于11.3之前已知的问题：

• Fix race conditions in management of dynamic shared memory (Thomas Munro)

These could lead to “dsa_area couldnot attach to segment” or “cannot unpin a segment thatis not pinned” errors.

if you just happen to hit PostgreSQL bugsuch as this one that was fixed in v11.3:

Once this problem occurs, your databasewill appear to be running, and can possibly be used to service somequeries... but most queries will hang, and eventually yourapplication servers will timeout causing application downtime.

通过重启实例恢复，由于重启过程中，实例无法正常停止，导致管控重启流程超时失败，通过手工修复拉起实例。手动流程如下：

• 将长时间运行的流程直接设置为中断

cat/root/.kube/config |grep client-certificate-data|awk {print$2}|base64 -d >/root/.kube/admin.crt

cat/root/.kube/config |grep client-key-data|awk {print $2}|base64 -d>/root/.kube/admin.key

KUBEAPISERVER=127.0.0.1

PPASCLUSTERNAME=polar-xxxxx

curl-k --cert /root/.kube/admin.crt --key /root/.kube/admin.keyhttps://$KUBEAPISERVER:6443/apis/ppas.polardb.aliyun.com/v1beta1/namespaces/default/ppasclusters/$PPASCLUSTERNAME| python -m json.tool  > polar-single.yaml

vimpolar-single.yaml

找到clusterStatus:xxxxxx，改为clusterStatus:Interrupt，保存

curl-X PUT -H "Content-Type: application/json" -k --cert/root/.kube/admin.crt --key /root/.kube/admin.keyhttps://$KUBEAPISERVER:6443/apis/ppas.polardb.aliyun.com/v1beta1/namespaces/default/ppasclusters/$PPASCLUSTERNAME/status--data @polar-single.yaml

• 控制台查看集群列表，看是否进入到中断状态。

• 清理ins_lock文件

kubectlget ppascluster polar-xxxx -o yaml

找到status- masterCluster - rwHostIns - insId，这个值是实例ID。

在三个计算节点找以下路径：/data/polardb_ppas/{insId}/data，和直接进入容器查找文件效果一样。

找到ins_lock文件，删除。

• 将中断的集群进行重建

kubectledit ppascluster polar-xxxx

在annotations下面增加下面两项：

polarbox.interrupt.recover:"T"

polarbox.interrupt.previous.status:"RebuildingRW"

• 刷新控制台，查看集群状态进入重建中，等待重建完成

1、此问题是并行查询过程中触发，短期内通过关闭并行查询来规避，目前生产已经关闭并行。

2、Polardb应急手工拉起流程的熟悉。

3、相关工具配套的场景沉淀。