Abnormal State of MySQL Replication Group

When some members fail, the number of remaining members is less than quorum, and the MySQL replication group is compromised.

For example, run the following commands to delete two members:
kubectl delete pod mysql-set-1-0 mysql-set-2-0

The two members "mysql-set-1-0" and "mysql-set-2-0" are deleted.

Since they are in Kubernetes stateful set, they will be recreated by Kubernetes. Before the two members are recreated, the original MySQL replication group is compromised and cannot recover without manual intervention.

The followings are example outputs from each member after the two members are recreated:
ubectl exec mysql-set-0-0 -- mysql -uroot -pchangeme -e "SELECT * FROM performance_schema.replication_group_members;"
 
 
CHANNEL_NAME    MEMBER_ID   MEMBER_HOST MEMBER_PORT MEMBER_STATE
group_replication_applier   070f57ea-e271-11ea-bca7-ce95fa60e6f7    mysql-set-2-0.mysql-svc-2.default.svc.cluster.local 3306    UNREACHABLE
group_replication_applier   836c7441-e270-11ea-b6fa-4ea77323e96f    mysql-set-0-0.mysql-svc-0.default.svc.cluster.local 3306    ONLINE
group_replication_applier   c7ae2e67-e270-11ea-ba2b-ea4a486569d2    mysql-set-1-0.mysql-svc-1.default.svc.cluster.local 3306    UNREACHABLE
kubectl exec mysql-set-1-0 -- mysql -uroot -pchangeme -e "SELECT * FROM performance_schema.replication_group_members;"
 
 
CHANNEL_NAME    MEMBER_ID   MEMBER_HOST MEMBER_PORT MEMBER_STATE
group_replication_applier   c7ae2e67-e270-11ea-ba2b-ea4a486569d2    mysql-set-1-0.mysql-svc-1.default.svc.cluster.local 3306    ONLINE
kubectl exec mysql-set-2-0 -- mysql -uroot -pchangeme -e "SELECT * FROM performance_schema.replication_group_members;"
 
 
CHANNEL_NAME    MEMBER_ID   MEMBER_HOST MEMBER_PORT MEMBER_STATE
group_replication_applier   070f57ea-e271-11ea-bca7-ce95fa60e6f7    mysql-set-2-0.mysql-svc-2.default.svc.cluster.local 3306    ONLINE