OS Process Error on View Execution, CouchDB 3.1.1, Ubuntu 20.04

[notequal1]

Description

Hi Everyone. I have an interesting issue that I would say is not a mission critical issue, but is enough of an annoyance that I'd like to see if there is a way to resolve it. After I create/update a view on a modestly sized database (~200 MB, ~550,000 documents), and I execute the view, I receive the following error after several seconds:

{"error":"os_process_error","reason":"{exit_status,1}","ref":3722911048}

Interestingly, I can watch the the index for the view being built through the active tasks in Fauxton, but it still fails after several seconds. If continue to manually re-execute the view, the same thing continues to occur until the index is completely built. So, in the end, I do get a working index, but I have to continually execute the view to get it!

Here is where I think it gets even more head-scratchingly interesting: This problem occurs on a CouchDB 3.1.1 instance on Ubuntu 20.04, but it does not occur on CouchDB 3.1.1 instance on Ubuntu 18.04. I set up the same test database on Ubuntu 18.04 (not replicated from the problem database), used the same view, and the index built as expected with no error being returned (except for the timeout, which is expected). The CouchDB instance on 18.04 was automatically updated from 2.3.1 to 3.0 -> 3.1 -> 3.1.1 using the standard apt-get mechanism. The CouchDB instance on 20.04 was installed on a fresh OS install as version 3.1.1 from the beginning. In both cases I have not done any modifications to the default configuration files, with the exception of adding security certificates and changing the admin password.

Here is the output of the CouchDB log.

[info] 2021-06-09T04:15:44.376911Z couchdb@127.0.0.1 <0.6608.1814> -------- Starting index update for db: shards/00000000-7fffffff/test.1616963133 idx: _design/rpts-prt
[info] 2021-06-09T04:15:44.392423Z couchdb@127.0.0.1 <0.6608.1814> -------- Index update finished for db: shards/00000000-7fffffff/test.1616963133 idx: _design/rpts-prt
[notice] 2021-06-09T15:57:28.554997Z couchdb@127.0.0.1 <0.2721.1841> dcac6b7359 my_server.my_org.com:6984 my_remote_ip my_couch_user GET /test/_design/rpts-prt/_view/rpts-prt?descending=true&reduce=false&skip=0&limit=101 200 ok 56
[notice] 2021-06-09T16:43:53.523438Z couchdb@127.0.0.1 <0.32134.1842> 48a2bac4ae my_server.my_org.com:6984 my_remote_ip my_couch_user GET /test/_design/rpts-prt/_view/rpts-prt?descending=true&reduce=false&skip=0&limit=101 200 ok 9
[notice] 2021-06-09T16:44:09.288677Z couchdb@127.0.0.1 <0.32134.1842> 37c1c8778b my_server.my_org.com:6984 my_remote_ip my_couch_user PUT /test/_design/rpts-prt 201 ok 83
[info] 2021-06-09T16:44:09.346994Z couchdb@127.0.0.1 <0.24246.1842> -------- Starting index update for db: shards/00000000-7fffffff/test.1616963133 idx: _design/rpts-prt
[info] 2021-06-09T16:44:09.405678Z couchdb@127.0.0.1 <0.1988.1843> -------- Starting index update for db: shards/80000000-ffffffff/test.1616963133 idx: _design/rpts-prt
[notice] 2021-06-09T16:44:34.453822Z couchdb@127.0.0.1 <0.12722.1842> 8ef8f3138a my_server.my_org.com:6984 my_remote_ip my_couch_user GET /test/_design/rpts-prt/_view/rpts-prt?descending=true&skip=0&limit=101&reduce=false 500 ok 24997
[info] 2021-06-09T16:45:09.355555Z couchdb@127.0.0.1 <0.24246.1842> -------- Starting index update for db: shards/00000000-7fffffff/test.1616963133 idx: _design/rpts-prt
[info] 2021-06-09T16:46:35.794787Z couchdb@127.0.0.1 <0.24246.1842> -------- Starting index update for db: shards/00000000-7fffffff/test.1616963133 idx: _design/rpts-prt
[info] 2021-06-09T16:46:35.796524Z couchdb@127.0.0.1 <0.8870.1843> -------- Starting index update for db: shards/80000000-ffffffff/test.1616963133 idx: _design/rpts-prt
[notice] 2021-06-09T16:47:00.565525Z couchdb@127.0.0.1 <0.26500.1842> fa1ec82354 my_server.my_org.com:6984 my_remote_ip my_couch_user GET /test/_design/rpts-prt/_view/rpts-prt 500 ok 24773

Here is the output snippit from the syslog:

Jun  9 16:44:34 my_server couchdb[1661763]: out of memory
Jun  9 16:44:34 my_server couchdb[1661762]: out of memory
Jun  9 16:45:33 my_server couchdb[1661764]: out of memory
Jun  9 16:47:00 my_server couchdb[1661016]: out of memory
Jun  9 16:47:00 my_server couchdb[1661017]: out of memory

Here is the view in question:

function (doc) {

    // View: rpts-prt
    //
    // Emit every report entry in the document with any non-numeric
    // associated metadata included.  The key will consist of the
    // platform ID, the report type, and a six element date array which
    // includes the time in UTC.

    // Split the document ID into the platform ID and date/time
    // components.
    var _id_splt = doc._id.split("__");
    var p = _id_splt[1];
    var t = _id_splt[0];

    // Use this flag to help determine if the document is valid.
    var is_valid = true;

    // Make sure there are only two components of the document ID.
    if (_id_splt.length != 2) {

        is_valid = false;
    }

    // Determine if the platform ID component does not equal the
    // "GBL" or "GRP" value, representing a global or group document,
    // respectively.  Data should not be saved using either of those
    // values, anyway.
    if (p == "GBL" || p == "GRP") {

        is_valid = false;
    }

    // Determine that the date component in the "_id" value represents
    // an actual date.
    var date = null;

    if (t.length == 14) {

        date = new Date(
            t.slice(0, 4) + "-" +
            t.slice(4, 6) + "-" +
            t.slice(6, 8) + "T" +
            t.slice(8, 10) + ":" +
            t.slice(10, 12) + ":" +
            t.slice(12, 14) + "Z");
    }

    // Is the new date string a valid data?
    if (date.toString() == "Invalid Date") {

        is_valid = false;
    }

    // Loop over the data reports in the document, if the document is
    // valid.
    if (is_valid === true) {

        for (var rpt in doc["data"]) {

            // Define the key array.
            var key = [
                p,
                rpt,
                t.slice(0, 4),
                t.slice(4, 6),
                t.slice(6, 8),
                t.slice(8, 10),
                t.slice(10, 12),
                t.slice(12, 14)];

            // Emit the key with the report entry.
            emit(key, doc["data"][rpt]);
        }
    }
}

Here is a typical document in the database:

{
  "_id": "20210609190000__xyz321",
  "_rev": "2-123ad97e06286a7b1003f67f70520ab0",
  "data": {
    "fm": {
      "ta2m": 29.28,
      "rh2m": 6.812,
      "ws3mx": 6.352,
      "ws3ma": 3.518,
      "ws3m": 2.916,
      "wd3ma": 118.5,
      "wd3ms": 19.79,
      "wd3m": 114.9,
      "rain": 0,
      "srad": 1.059,
      "insl": 0.3181167,
      "tsl10cm": 24.93,
      "vbxt": 13.14,
      "tpnl": 31.23
    },
    "hr": {
      "ta2mx": {
        "value": 29.52,
        "date": "20210609182540"
      },
      "ta2mn": {
        "value": 28.7,
        "date": "20210609180550"
      },
      "ta2ma": 29.04,
      "rh2mx": {
        "value": 11.72,
        "date": "20210609182240"
      },
      "rh2mn": {
        "value": 5.177,
        "date": "20210609181530"
      },
      "rh2ma": 8.09,
      "ws3ma": 2.741,
      "wd3ma": 94.9,
      "wd3ms": 24.69,
      "sradx": {
        "value": 1.063,
        "date": "20210609185520"
      },
      "sradn": {
        "value": 1.033,
        "date": "20210609180010"
      },
      "srada": 1.05
    }
  }
}

Steps to Reproduce

Fresh install of Ubuntu 20.04 with fresh install of CouchDB 3.1.1. Create new database. Populate with > ~100,000 documents, Create the view. Execute the view. I've deleted and created multiple databases to test this and it displays the same behavior.

Expected Behaviour

The index begins to be built after making the first request to the view. I receive a timeout error, but the view index continues to be built.

Your Environment

• CouchDB version used: 3.1.1
• Browser name and version: curl, Firefox 88
• Operating system and version: Ubuntu 20.04

Additional Context

Once the database is in production, we expect it to grow to several GB quite quickly. Our views will mostly be stable, so this shouldn't be a deal breaker in the future. However, if we do have to modify a view, it would be quite annoying to have to set up a cron job to make a request to the modified view every minute just so we can get a valid index built.

Praise

I'm a sole developer at my organization and was using a MySQL database to store our mission critical data for several years. The MySQL database was never replicated on another server (the data was backed up, though) because I was so busy being the only tech guy (programming, field work, and sys admin) that I kept pushing that off. I reached a point where the limitations of the MySQL database was causing me to spend too much time on application development to get around those issues. I began to seriously look at NoSQL database, and, after experimenting with MongoDB, I decided that I liked the way CouchDB did things (also the license). I can now say that I love CouchDB. I had most of our data transitioned to CouchDB in two days with no-fuss replication to another server at the same time. The production databases have been reliably serving data for several years now. It wasn't until we had a hard drive failure on one of our servers that we started fresh with Ubuntu 20.04 and discovered this problem. However, it's not enough to convince me even a little bit to leave the CouchDB way of doing things. Hat tip to the developers for producing a great database system. :)

[wohali]

Hey there, thanks for the nice words.

CouchDB 3.1.1 on Ubuntu 20.04 uses a more modern JavaScript library (libmozjs-68-0) than it did on 18.04 (CouchDB supplied 1.8.5). This likely translates to more RAM usage.

Further analysis: Your for loop is at least not tail-recursive, but it will be increasing the size of the object that must be constructed for each document's emit. This isn't ideal. In fact, if your loop is emitting more than 10 times per document, you are likely running into a known limitation of JavaScript views that is an architectural limitation, related to how data is passed between the external couchjs process and the Erlang couchdb. This won't be changed anytime soon.

If you have any way to restructure your data so a loop over the document isn't required, I'd start there. This is especially true if you can do something with Mango, which will never be subject to these kinds of architectural limitations. (It's also generally 2x-10x faster than JS views.) Briefly glancing at your doc, this would entail some preformatting and denormalization of your data, including splitting your larger docs up into smaller units.

If document modification is out of the question, the second option is to increase the amout of RAM on your machines. CouchDB hasn't been tuned for low CPU/RAM usage. quite the contrary. It's been tuned for big servers, and has no special functionality to avoid running out of RAM. If you're experiencing oom_killer under Linux (you'd see this in dmesg output), the standard recommendation is to double your RAM until the problem stops happening.

As to the view building question, 3.1.1 comes with an auto view builder that is configurable, that you wouldn't have had under 2.x. You can read more about it here: https://docs.couchdb.org/en/stable/config/indexbuilds.html#background-indexing .

I'm suspicious of your Couch log since the crashes you're showing in your syslog aren't reflected there. This is probably oom_killer at work, especially if you've removed the CouchDB startup noise from the logs in your paste.

[wohali]

Set log level to debug and see if you can get the reason for the exit status of 1 from couchjs.

[notequal1]

I changed the log level to debug in the default.ini file. Unfortunately, I'm not seeing any debug output in the couchdb.log file. I still only see the error shown above in my previous post. Does couchjs write a separate log?

ETA

So according to documentation, the query server only logs at the info level. So we wouldn't see any debug info from the 'couchjs' server. Also, if the couchjs process is crashing, it would most likely not be logging that, right?

Executing my test view again on my desktop machine, I see the following info level logs:

[info] 2021-06-22T17:39:20.485501Z couchdb@127.0.0.1 <0.16994.2> -------- Starting index update for db: shards/80000000-ffffffff/test.1623428417 idx: _design/test-1
[info] 2021-06-22T17:39:20.485500Z couchdb@127.0.0.1 <0.16995.2> -------- Starting index update for db: shards/00000000-7fffffff/test.1623428417 idx: _design/test-1
[info] 2021-06-22T17:39:31.111803Z couchdb@127.0.0.1 <0.17139.2> -------- Compaction started for db: shards/00000000-7fffffff/test.1623428417 idx: _design/test-1
[info] 2021-06-22T17:39:39.987164Z couchdb@127.0.0.1 <0.237.0> -------- couch_proc_manager <0.17009.2> died normal
[info] 2021-06-22T17:39:40.046963Z couchdb@127.0.0.1 <0.237.0> -------- couch_proc_manager <0.17011.2> died normal
[info] 2021-06-22T17:39:41.705333Z couchdb@127.0.0.1 <0.17139.2> -------- Compaction restarting for db: shards/00000000-7fffffff/test.1623428417 idx: _design/test-1
[info] 2021-06-22T17:39:41.705364Z couchdb@127.0.0.1 <0.17139.2> -------- Compaction started for db: shards/00000000-7fffffff/test.1623428417 idx: _design/test-1
[info] 2021-06-22T17:39:54.068473Z couchdb@127.0.0.1 <0.237.0> -------- couch_proc_manager <0.17306.2> died normal
[info] 2021-06-22T17:40:06.141976Z couchdb@127.0.0.1 <0.237.0> -------- couch_proc_manager <0.17434.2> died normal
[info] 2021-06-22T17:40:18.430332Z couchdb@127.0.0.1 <0.237.0> -------- couch_proc_manager <0.17562.2> died normal
[info] 2021-06-22T17:40:30.740359Z couchdb@127.0.0.1 <0.237.0> -------- couch_proc_manager <0.17708.2> died normal
[info] 2021-06-22T17:40:42.884658Z couchdb@127.0.0.1 <0.237.0> -------- couch_proc_manager <0.17842.2> died normal
[info] 2021-06-22T17:40:46.810338Z couchdb@127.0.0.1 <0.17139.2> -------- Compaction finished for db: shards/00000000-7fffffff/test.1623428417 idx: _design/test-1
[info] 2021-06-22T17:40:51.822923Z couchdb@127.0.0.1 <0.18072.2> -------- Compaction started for db: shards/00000000-7fffffff/test.1623428417 idx: _design/test-1
[info] 2021-06-22T17:41:05.241834Z couchdb@127.0.0.1 <0.18072.2> -------- Compaction finished for db: shards/00000000-7fffffff/test.1623428417 idx: _design/test-1

And then, once again, the error entries:

[error] 2021-06-22T17:39:39.987146Z couchdb@127.0.0.1 <0.17006.2> -------- OS Process Error <0.17009.2> :: {os_process_error,{exit_status,1}}
[error] 2021-06-22T17:39:39.987338Z couchdb@127.0.0.1 <0.16986.2> d89889dab5 rexi_server: from: couchdb@127.0.0.1(<0.16731.2>) mfa: fabric_rpc:map_view/5 throw:{os_process_error,{exit_status,1}} [{couch_mrview_util,get_view_index_state,5,[{file,"src/couch_mrview_util.erl"},{line,138}]},{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,82}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,262}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,138}]}]
[error] 2021-06-22T17:39:39.987481Z couchdb@127.0.0.1 <0.16731.2> d89889dab5 req_err(3722911048) os_process_error : {exit_status,1}
[error] 2021-06-22T17:39:39.988829Z couchdb@127.0.0.1 emulator -------- Error in process <0.17006.2> on node 'couchdb@127.0.0.1' with exit value:
{{nocatch,{os_process_error,{exit_status,1}}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,68}]},{couch_mrview_updater,'-map_docs/2-fun-0-',3,[{file,"src/couch_mrview_updater.erl"},{line,165}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{couch_mrview_updater,map_docs,2,[{file,"src/couch_mrview_updater.erl"},{line,172}]}]}
[error] 2021-06-22T17:39:40.046964Z couchdb@127.0.0.1 <0.17004.2> -------- OS Process Error <0.17011.2> :: {os_process_error,{exit_status,1}}
[error] 2021-06-22T17:39:40.047112Z couchdb@127.0.0.1 emulator -------- Error in process <0.17004.2> on node 'couchdb@127.0.0.1' with exit value:
{{nocatch,{os_process_error,{exit_status,1}}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,68}]},{couch_mrview_updater,'-map_docs/2-fun-0-',3,[{file,"src/couch_mrview_updater.erl"},{line,165}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{couch_mrview_updater,map_docs,2,[{file,"src/couch_mrview_updater.erl"},{line,172}]}]}
[error] 2021-06-22T17:39:40.047126Z couchdb@127.0.0.1 <0.16987.2> d89889dab5 rexi_server: from: couchdb@127.0.0.1(<0.16731.2>) mfa: fabric_rpc:map_view/5 throw:{os_process_error,{exit_status,1}} [{couch_mrview_util,get_view_index_state,5,[{file,"src/couch_mrview_util.erl"},{line,138}]},{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,82}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,262}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,138}]}]
[error] 2021-06-22T17:39:54.068334Z couchdb@127.0.0.1 <0.17303.2> -------- OS Process Error <0.17306.2> :: {os_process_error,{exit_status,1}}
[error] 2021-06-22T17:39:54.068481Z couchdb@127.0.0.1 emulator -------- Error in process <0.17303.2> on node 'couchdb@127.0.0.1' with exit value:
{{nocatch,{os_process_error,{exit_status,1}}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,68}]},{couch_mrview_updater,'-map_docs/2-fun-0-',3,[{file,"src/couch_mrview_updater.erl"},{line,165}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{couch_mrview_updater,map_docs,2,[{file,"src/couch_mrview_updater.erl"},{line,172}]}]}
[warning] 2021-06-22T17:39:54.068701Z couchdb@127.0.0.1 <0.17139.2> -------- Error during recompaction: {nocatch,{os_process_error,{exit_status,1}}} at couch_os_process:prompt/2(line:59) <= couch_query_servers:map_doc_raw/2(line:68) <= couch_mrview_updater:'-map_docs/2-fun-0-'/3(line:165) <= lists:foldl/3(line:1263) <= couch_mrview_updater:map_docs/2(line:172)
[error] 2021-06-22T17:40:06.141976Z couchdb@127.0.0.1 <0.17431.2> -------- OS Process Error <0.17434.2> :: {os_process_error,{exit_status,1}}
[warning] 2021-06-22T17:40:06.142126Z couchdb@127.0.0.1 <0.17139.2> -------- Error during recompaction: {nocatch,{os_process_error,{exit_status,1}}} at couch_os_process:prompt/2(line:59) <= couch_query_servers:map_doc_raw/2(line:68) <= couch_mrview_updater:'-map_docs/2-fun-0-'/3(line:165) <= lists:foldl/3(line:1263) <= couch_mrview_updater:map_docs/2(line:172)
[error] 2021-06-22T17:40:06.142137Z couchdb@127.0.0.1 emulator -------- Error in process <0.17431.2> on node 'couchdb@127.0.0.1' with exit value:
{{nocatch,{os_process_error,{exit_status,1}}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,68}]},{couch_mrview_updater,'-map_docs/2-fun-0-',3,[{file,"src/couch_mrview_updater.erl"},{line,165}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{couch_mrview_updater,map_docs,2,[{file,"src/couch_mrview_updater.erl"},{line,172}]}]}
[error] 2021-06-22T17:40:18.430331Z couchdb@127.0.0.1 <0.17559.2> -------- OS Process Error <0.17562.2> :: {os_process_error,{exit_status,1}}
[error] 2021-06-22T17:40:18.430474Z couchdb@127.0.0.1 emulator -------- Error in process <0.17559.2> on node 'couchdb@127.0.0.1' with exit value:
{{nocatch,{os_process_error,{exit_status,1}}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,68}]},{couch_mrview_updater,'-map_docs/2-fun-0-',3,[{file,"src/couch_mrview_updater.erl"},{line,165}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{couch_mrview_updater,map_docs,2,[{file,"src/couch_mrview_updater.erl"},{line,172}]}]}
[warning] 2021-06-22T17:40:18.430521Z couchdb@127.0.0.1 <0.17139.2> -------- Error during recompaction: {nocatch,{os_process_error,{exit_status,1}}} at couch_os_process:prompt/2(line:59) <= couch_query_servers:map_doc_raw/2(line:68) <= couch_mrview_updater:'-map_docs/2-fun-0-'/3(line:165) <= lists:foldl/3(line:1263) <= couch_mrview_updater:map_docs/2(line:172)
[error] 2021-06-22T17:40:30.740356Z couchdb@127.0.0.1 <0.17703.2> -------- OS Process Error <0.17708.2> :: {os_process_error,{exit_status,1}}
[error] 2021-06-22T17:40:30.740483Z couchdb@127.0.0.1 emulator -------- Error in process <0.17703.2> on node 'couchdb@127.0.0.1' with exit value:
{{nocatch,{os_process_error,{exit_status,1}}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,68}]},{couch_mrview_updater,'-map_docs/2-fun-0-',3,[{file,"src/couch_mrview_updater.erl"},{line,165}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{couch_mrview_updater,map_docs,2,[{file,"src/couch_mrview_updater.erl"},{line,172}]}]}
[warning] 2021-06-22T17:40:30.740506Z couchdb@127.0.0.1 <0.17139.2> -------- Error during recompaction: {nocatch,{os_process_error,{exit_status,1}}} at couch_os_process:prompt/2(line:59) <= couch_query_servers:map_doc_raw/2(line:68) <= couch_mrview_updater:'-map_docs/2-fun-0-'/3(line:165) <= lists:foldl/3(line:1263) <= couch_mrview_updater:map_docs/2(line:172)
[error] 2021-06-22T17:40:42.884621Z couchdb@127.0.0.1 <0.17839.2> -------- OS Process Error <0.17842.2> :: {os_process_error,{exit_status,1}}
[error] 2021-06-22T17:40:42.884794Z couchdb@127.0.0.1 emulator -------- Error in process <0.17839.2> on node 'couchdb@127.0.0.1' with exit value:
{{nocatch,{os_process_error,{exit_status,1}}},[{couch_os_process,prompt,2,[{file,"src/couch_os_process.erl"},{line,59}]},{couch_query_servers,map_doc_raw,2,[{file,"src/couch_query_servers.erl"},{line,68}]},{couch_mrview_updater,'-map_docs/2-fun-0-',3,[{file,"src/couch_mrview_updater.erl"},{line,165}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{couch_mrview_updater,map_docs,2,[{file,"src/couch_mrview_updater.erl"},{line,172}]}]}
[warning] 2021-06-22T17:40:42.884843Z couchdb@127.0.0.1 <0.17139.2> -------- Error during recompaction: {nocatch,{os_process_error,{exit_status,1}}} at couch_os_process:prompt/2(line:59) <= couch_query_servers:map_doc_raw/2(line:68) <= couch_mrview_updater:'-map_docs/2-fun-0-'/3(line:165) <= lists:foldl/3(line:1263) <= couch_mrview_updater:map_docs/2(line:172)

And again, the syslog on my desktop machine. I assume that the syslog is logging in local time, as the times seem to line up when adjusting my local time to UTC.

Jun 22 11:39:39 my_desktop couchdb[8028]: out of memory
Jun 22 11:39:40 my_desktop couchdb[8029]: out of memory
Jun 22 11:39:54 my_desktop couchdb[8047]: out of memory
Jun 22 11:40:06 my_desktop couchdb[8064]: out of memory
Jun 22 11:40:18 my_desktop couchdb[8080]: out of memory
Jun 22 11:40:30 my_desktop couchdb[8091]: out of memory
Jun 22 11:40:42 my_desktop couchdb[8108]: out of memory

Looking at the configuration documentation, it suggests that I increase the memory allocated to the couchjs server by defining an environmental variable with a memory parameter, if I see out of memory errors. So, I added a file called couchdb.sh to my '/etc/profile.d/' directory containing the following statement:

export COUCHDB_QUERY_SERVER_JAVASCRIPT="/opt/couchdb/bin/couchjs -S 536870912 /opt/couchdb/share/server/main.js"

While I can us the printenv command to see that the variable is set under my user, how do I know CouchDB is picking up that variable and using it to execute the query server?

I feel like I am a broken record at this point. What other options are available to help diagnose this issue?

[notequal1]

Based on the edit to my latest post, are there other strategies to help find the root cause of this issue? Is my attempt to increase the memory to the couchjs query server correct?

[wohali]

debug level will log everything being sent to and from the couchjs process:

couchdb/src/couch_js/src/couch_js_os_process.erl

Lines 105 to 124 in 15cbb55

	?LOG_DEBUG(#{
	what => writeline,
	in => os_process,
	port => OsProc#os_proc.port,
	data => JsonData
	}),
	couch_log:debug(
	"OS Process ~p Input :: ~s",
	[OsProc#os_proc.port, JsonData]
	),
	true = writeline(OsProc, JsonData).
	readjson(OsProc) when is_record(OsProc, os_proc) ->
	Line = iolist_to_binary(readline(OsProc)),
	?LOG_DEBUG(#{
	what => readline,
	in => os_process,
	port => OsProc#os_proc.port,
	data => Line
	}),

you need to check your settings and make sure it actually picked up that log level.

The -S option only changes the couchjs stack size and may not fix your root cause, but worth a try. I don't know if your profile.d approach works based on how you're starting couchdb, sorry.

I have very limited time to help with CouchDB things right now so this is the best I can offer. You may want to try joining the CouchDB Slack or IRC channels, as described at https://couchdb.apache.org/ for more real-time help from another developer.

[notequal1]

Can't believe it's been 20 days since I last looked at this. Thanks for your time, and when I have the chance, I'll look into Slack.