Couchdb sometimes timeout and slow.

[kathy1121]

Couchdb sometimes timeout and very slow, after the period it returns normal.

Description

We observed that sometimes when the application check the couchdb, for example:
acurl -w "%{time_starttransfer}\n" 'https://couchdb-services.***net:5984/prod_cnprod_iot_db/_design/srchEngine/_search/equipmentMasterDocV4Search?q=EquipmentNumberKENformatted:"30013508"&EquipmentLocationCompanyCode="KCC"'
The searching result hung and lots of error appears:
[error] 2020-12-21T07:51:23.348710Z couchdb@couchdb-couchdb-1.couchdb-couchdb.couchdb.svc.clusterb.local <0.25195.30> 3420357146 fabric_worker_timeout update_docs,'couchdb@couchdb-couchdb-0.couchdb-couchdb.couchdb.svc.clusterb.local',<<"shards/00000000-7fffffff/prod_cnprod_iot_db.1575029777">>

During this period, searching this view is slow, but if we curl other doc in this db for test, the response time is normal(generally 0.2s)

But later it may also show the result again and the search time return to normal.
Please help investigate why the couchdb met this issue?

Steps to Reproduce

It appears randomly but frequently.

Expected Behaviour

Find out the root cause and why leads the issue.

Your Environment

• CouchDB version used: v. 3.1.0
• Browser name and version: Not applicable
• Operating system and version: cluster in Linux 3.10.0-957.27.2.el7.x86_64

Additional Context

[kathy1121]

Hi team ,we also found this:
In the latency period, here are lots of [info]logs like this each second:
[info] 2020-12-21T05:56:42.280439Z couchdb@couchdb-couchdb-0.couchdb-couchdb.couchdb.svc.clusterb.local <0.5366.382> -------- OS Process #Port<0.626317> Log :: function raised exception (new TypeError("doc.value.devices is undefined", "", 2)) with doc._id 44106***
With hundreds of [info], then it will appear the error:
[error] 2020-12-21T06:00:04.109270Z couchdb@couchdb-couchdb-0.couchdb-couchdb.couchdb.svc.clusterb.local <0.31741.388> 6b398ec149 fabric_worker_timeout update_docs,'couchdb@couchdb-couchdb-2.couchdb-couchdb.couchdb.svc.clusterb.local',<<"shards/00000000-7fffffff/prod_cnprod_iot_db.1575029777">>
Can you please also advice on this? Thanks.