Abnormal State of MySQL Replication Group
When some members fail, the number of remaining members is less than quorum, and the MySQL replication group is compromised.
For example, run the following commands to delete two members:
kubectl delete pod mysql-set-1-0 mysql-set-2-0
The two members "mysql-set-1-0" and "mysql-set-2-0" are deleted.
Since they are in Kubernetes stateful set, they will be recreated by Kubernetes. Before the two members are recreated, the original MySQL replication group is compromised and cannot recover without manual intervention.
The followings are example outputs from each member after the two members are recreated:
ubectl exec mysql-set-0-0 -- mysql -uroot -pchangeme -e "SELECT * FROM performance_schema.replication_group_members;" CHANNEL_NAME MEMBER_ID MEMBER_HOST MEMBER_PORT MEMBER_STATE group_replication_applier 070f57ea-e271-11ea-bca7-ce95fa60e6f7 mysql-set-2-0.mysql-svc-2.default.svc.cluster.local 3306 UNREACHABLE group_replication_applier 836c7441-e270-11ea-b6fa-4ea77323e96f mysql-set-0-0.mysql-svc-0.default.svc.cluster.local 3306 ONLINE group_replication_applier c7ae2e67-e270-11ea-ba2b-ea4a486569d2 mysql-set-1-0.mysql-svc-1.default.svc.cluster.local 3306 UNREACHABLE
kubectl exec mysql-set-1-0 -- mysql -uroot -pchangeme -e "SELECT * FROM performance_schema.replication_group_members;" CHANNEL_NAME MEMBER_ID MEMBER_HOST MEMBER_PORT MEMBER_STATE group_replication_applier c7ae2e67-e270-11ea-ba2b-ea4a486569d2 mysql-set-1-0.mysql-svc-1.default.svc.cluster.local 3306 ONLINE
kubectl exec mysql-set-2-0 -- mysql -uroot -pchangeme -e "SELECT * FROM performance_schema.replication_group_members;" CHANNEL_NAME MEMBER_ID MEMBER_HOST MEMBER_PORT MEMBER_STATE group_replication_applier 070f57ea-e271-11ea-bca7-ce95fa60e6f7 mysql-set-2-0.mysql-svc-2.default.svc.cluster.local 3306 ONLINE
Copyright © 2022. Cloud Software Group, Inc. All Rights Reserved.