Cluster Running with 2 nodes and having difficulty to add 3rd node in the cluster

[DipindasThayyil]

As per the below url ,noticed cluster is running with 2 node and we tried with adding 3rd nodes into the cluster via " curl -X PUT "http://xxx.xxx.xxx.xxx/_node/_local/_nodes/node2@yyy.yyy.yyy.yyy" -d {} " however it not working.

Moreover , we have noticed difference in " adminPassword,cookieAuthSecret,erlangCookie" for the 3rd node.Therefore we have manaually updated these values and after that all the nodes were restarted and 3rd node were joined into cluster .

However after some time,again noticed cluster membership is in degraded state . Any suggestion from here ?

version: 3.3.3

https://score-db.cec.net/_membership

{
"all_nodes": [
"couchdb@couchdb-couchdb-0.couchdb-couchdb.apache-couchdb.svc.cluster.local",
"couchdb@couchdb-couchdb-1.couchdb-couchdb.apache-couchdb.svc.cluster.local"
],
"cluster_nodes": [
"couchdb@couchdb-couchdb-0.couchdb-couchdb.apache-couchdb.svc.cluster.local",
"couchdb@couchdb-couchdb-1.couchdb-couchdb.apache-couchdb.svc.cluster.local",
"couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local"
]
}

https://score-db.cec.net/_cluster_setup

{
"state": "cluster_finished"
}

Logs :

[notice] 2024-12-16T06:59:18.773373Z couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local <0.520.0> -------- mem3_reshard finished reloading jobs
[info] 2024-12-16T06:59:18.796240Z couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local <0.527.0> -------- Apache CouchDB has started. Time to relax.

[info] 2024-12-16T06:59:18.796317Z couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local <0.527.0> -------- Apache CouchDB has started on http://any:5984/
[notice] 2024-12-16T06:59:18.863063Z couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local <0.619.0> -------- All system databases exist.
[warning] 2024-12-16T06:59:18.866103Z couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local <0.617.0> -------- 6 shards in cluster with only 1 copy on nodes that are currently up
[warning] 2024-12-16T06:59:18.868596Z couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local <0.617.0> -------- 6 shards in cluster with only 1 copy on nodes not in maintenance mode
[error] 2024-12-16T06:59:19.760792Z couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local emulator -------- Error in process <0.688.0> on node 'couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local' with exit value:
{{rexi_DOWN,{'couchdb@couchdb-couchdb-0.couchdb-couchdb.apache-couchdb.svc.cluster.local',noconnect}},[{mem3_rpc,rexi_call,3,[{file,"src/mem3_rpc.erl"},{line,384}]},{mem3_seeds,'-start_replication/1-fun-0-',1,[{file,"src/mem3_seeds.erl"},{line,107}]}]}

[error] 2024-12-16T06:59:19.760855Z couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local emulator -------- Error in process <0.688.0> on node 'couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local' with exit value:
{{rexi_DOWN,{'couchdb@couchdb-couchdb-0.couchdb-couchdb.apache-couchdb.svc.cluster.local',noconnect}},[{mem3_rpc,rexi_call,3,[{file,"src/mem3_rpc.erl"},{line,384}]},{mem3_seeds,'-start_replication/1-fun-0-',1,[{file,"src/mem3_seeds.erl"},{line,107}]}]}

[error] 2024-12-16T06:59:20.761868Z couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local emulator -------- Error in process <0.692.0> on node 'couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local' with exit value:
{{rexi_DOWN,{'couchdb@couchdb-couchdb-1.couchdb-couchdb.apache-couchdb.svc.cluster.local',noconnect}},[{mem3_rpc,rexi_call,3,[{file,"src/mem3_rpc.erl"},{line,384}]},{mem3_seeds,'-start_replication/1-fun-0-',1,[{file,"src/mem3_seeds.erl"},{line,107}]}]}