Changing nodes name of an existing cluster cause dbs _replicator _users _ global_changes in state "this database failed to load"

[mescaler0]

Hello everyone,
in a test environment i tried to perform a rename 2 out of 3 nodes of an existing couchdb cluster ver 2.3.1, here's the details:

Starting configuration of the cluster

• node1 (couchdb@xxx.xxx.xxx.xxx)
• node2 (xouxhdb@yyy.yyy.yyy.yyy)

both the server were configured as cluster but using ips instead of FQDN, so the further steps were A) add a third node to the existing configuration and B) replace both the pre-existing nodes with the FQDN instead of ip,

Ending configuration of the cluster

• node1 (couchdb@node1.local)
• node2 (xouxhdb@node2.local)
• node3 (xouxhdb@node3.local)

I was able to add the third node without particular issues, but when i tried to rename the node1 from vm.args couchdb@xxx.xxx.xxx.xxx --> couchdb@hostname.local everything seems to work fine, but the moment i stop the couchdb daemon on node1 here's what happen to node2 and node3

https://ibb.co/kGjqZK1

On Fauxton you can clearly see that the first 3 local database are in "failed to load state" which is odd considering that stopping node1 should have 0 impact on the other 2 active nodes.

I don't think this is the correct behaviour, i also check if the metadata on all the 3 servers are ok and it seems so:

On node1 and is the same on all 3 nodes:
curl -X PUT http://user:pwd@127.0.0.1:5986/_dbs/_users -d '{
"_id": "_users",
"_rev": "4-5bd38e58d6a0dfbf08279a7cfcdacf81",
"by_node": {
"couchdb@node1.local": [
"00000000-1fffffff",
"20000000-3fffffff",
"40000000-5fffffff",
"60000000-7fffffff",
"80000000-9fffffff",
"a0000000-bfffffff",
"c0000000-dfffffff",
"e0000000-ffffffff"
],
"couchdb@node2.local": [
"00000000-1fffffff",
"20000000-3fffffff",
"40000000-5fffffff",
"60000000-7fffffff",
"80000000-9fffffff",
"a0000000-bfffffff",
"c0000000-dfffffff",
"e0000000-ffffffff"
],
"couchdb@node3.local": [
"00000000-1fffffff",
"20000000-3fffffff",
"40000000-5fffffff",
"60000000-7fffffff",
"80000000-9fffffff",
"a0000000-bfffffff",
"c0000000-dfffffff",
"e0000000-ffffffff"
]
},
"by_range": {
"00000000-1fffffff": [
"couchdb@node1.local",
"couchdb@node2.local",
"couchdb@node3.local"
],
"20000000-3fffffff": [
"couchdb@node1.local",
"couchdb@node2.local",
"couchdb@node3.local"
],
......
......
},
"changelog": [
[
"add",
"00000000-1fffffff",
"couchdb@node1.local"
],
[
"add",
"00000000-1fffffff",
"couchdb@node2.local"
],
[
"add",
"00000000-1fffffff",
"couchdb@node3.local"
],
.....
.....
],
"shard_suffix": [
46,
49,
53,
53,
51,
51,
56,
56,
51,
50,
49
]
}'

• CouchDB version used: 2.3.1
• Browser name and version: Chrome 84
• Operating system and version: Ubuntu 18.04

[wohali]

What is the output of `curl http://user:pwd@127.0.0.1/_membership' on all 3 nodes?

[kripper]

@mescaler0 never confirmed if it was a connection problem...

[mescaler0]

@kripper never confirmed because it was never a connection\network related issue. I contacted the Couchdb developers on the Slack channels and the developers informed me that it's better destroy and create a new cluster than add members and so i did.