Error with "Implicit Operators"

[hamiltonsalazar]

Description

I'm putting on practice the CouchDB documentation and I read about "Implicit Operators", but COUCHD ignore my firts condiction when i send two condictions and it should use the implicit $and operartor

Current Behavior

I use the next query:

{     
    "selector" : {        
        "age": {
            "$lte": 30
        },
        "age": {
            "$gte": 25
        }
    },    
    "fields": [   "name",   "age",    "_id"     ], 
    "sort": [
        {
            "age": "desc"
        }   
    ]
}

But COUCHDB is ignoring my firts condition:

{"docs":[
{"name":"Erick","age":70,"_id":"9578"},
{"name":"Lucas","age":70,"_id":"9269"},
{"name":"Valeria","age":70,"_id":"9016"},
{"name":"Pablo","age":70,"_id":"8963"},
{"name":"Lucas","age":70,"_id":"8592"},
{"name":"Olivia","age":70,"_id":"8557"},
{"name":"Paula","age":70,"_id":"7961"},
{"name":"Adrián","age":70,"_id":"7653"},
{"name":"Isla","age":70,"_id":"7615"},
{"name":"Javier","age":70,"_id":"6625"},
{"name":"Daniel","age":70,"_id":"640"},
{"name":"Alice","age":70,"_id":"6318"},
{"name":"Chloe","age":70,"_id":"6118"},
{"name":"Benjamín","age":70,"_id":"5713"},
{"name":"Adeline","age":70,"_id":"5674"},
{"name":"David","age":70,"_id":"5572"},
{"name":"Nicolás","age":70,"_id":"5289"},
{"name":"Thiago","age":70,"_id":"5232"},
{"name":"Julián","age":70,"_id":"5137"},
{"name":"Dulce","age":70,"_id":"4738"},
{"name":"Valeria","age":70,"_id":"4699"},
{"name":"Sofía","age":70,"_id":"4389"},
{"name":"Matías","age":70,"_id":"4323"},
{"name":"Evelyn","age":70,"_id":"4248"},
{"name":"Carlos","age":70,"_id":"4192"}
],
"bookmark": "g2wAAAACaAJkAA5zdGFydGtleV9kb2NpZG0AAAAENDE5MmgCZAAIc3RhcnRrZXlrAAFGag"}

And changing the condictions:

{     
    "selector" : {    
        "age": {
            "$gte": 25
        },
        "age": {
            "$lte": 30
        }
    },    
    "fields": [   "name",   "age",    "_id"     ], 
    "sort": [
        {
            "age": "asc"
        }   
    ]
}

CouchDB also ignored my firts condiction:

{"docs":[
{"name":"Amalia","age":18,"_id":"5642"},
{"name":"Evelyn","age":18,"_id":"6528"},
{"name":"Eleanor","age":18,"_id":"7476"},
{"name":"Dulce","age":18,"_id":"8990"},
{"name":"Valentín","age":18,"_id":"9755"},
{"name":"Valeria","age":19,"_id":"1016"},
{"name":"Jorge","age":19,"_id":"1164"},
{"name":"Evelyn","age":19,"_id":"1172"},
{"name":"Erick","age":19,"_id":"1246"},
{"name":"Cora","age":19,"_id":"1385"},
{"name":"Pablo","age":19,"_id":"146"},
{"name":"Samantha","age":19,"_id":"1633"},
{"name":"Marcos","age":19,"_id":"1645"},
{"name":"Regina","age":19,"_id":"1758"},
{"name":"Charlotte","age":19,"_id":"1797"},
{"name":"Marcos","age":19,"_id":"21"},
{"name":"Álvaro","age":19,"_id":"2271"},
{"name":"Eleanor","age":19,"_id":"2283"},
{"name":"Thiago","age":19,"_id":"230"},
{"name":"Adrián","age":19,"_id":"2401"},
{"name":"Samantha","age":19,"_id":"2681"},
{"name":"Ava","age":19,"_id":"2863"},
{"name":"Julia","age":19,"_id":"287"},
{"name":"Daniela","age":19,"_id":"2912"},
{"name":"Pablo","age":19,"_id":"296"}
],
"bookmark": "g2wAAAACaAJkAA5zdGFydGtleV9kb2NpZG0AAAADMjk2aAJkAAhzdGFydGtleWsAARNq"}

Expected Behaviour

COUCHD should response with:

{"docs":[
{"name":"Cora","age":25,"_id":"104"},
{"name":"Pablo","age":25,"_id":"11"},
{"name":"Daniela","age":25,"_id":"1192"},
{"name":"Sim├│n","age":25,"_id":"1305"},
{"name":"Amalia","age":25,"_id":"1358"},
{"name":"Benjamín","age":25,"_id":"1488"},
{"name":"Cora","age":25,"_id":"1599"},
{"name":"Pablo","age":25,"_id":"1722"},
{"name":"Benjamín","age":25,"_id":"1781"},
{"name":"Mario","age":25,"_id":"1798"},
{"name":"Elizabeth","age":25,"_id":"2"},
{"name":"Daniel","age":25,"_id":"2060"},
{"name":"Adrián","age":25,"_id":"209"},
{"name":"Chloe","age":25,"_id":"2173"},
{"name":"Elizabeth","age":25,"_id":"2211"},
{"name":"Álvaro","age":25,"_id":"2237"},
{"name":"Samantha","age":25,"_id":"2281"},
{"name":"Marcos","age":25,"_id":"2334"},
{"name":"Iván","age":25,"_id":"2409"},
{"name":"Benjamín","age":25,"_id":"2523"},
{"name":"Matías","age":25,"_id":"2772"},
{"name":"Eleanor","age":25,"_id":"2781"},
{"name":"Valentín","age":25,"_id":"2926"},
{"name":"Paula","age":25,"_id":"2927"},
{"name":"Mario","age":25,"_id":"2958"}
],
"bookmark": "g2wAAAACaAJkAA5zdGFydGtleV9kb2NpZG0AAAAEMjk1OGgCZAAIc3RhcnRrZXlrAAEZag"}

But I only can get this with Explicit Operators:

{     
    "selector" : {    
        "$and": [
            {
                "age": {        
                    "$gte": 25    
                }
            },    
            {
                "age": {        
                    "$lte": 30    
                }
            } 
        ]
    },    
    "fields": [   "name",   "age",    "_id"     ], 
    "sort": [
        {
            "age": "asc"
        }   
    ]
}

Your Environment

• CouchDB version used: 3.1.0
• Operating system and version: Windows 10 Pro 1909

[wohali]

Hi @hamiltonsalazar ,

That's because what you're specifying isn't really valid JSON. In any JSON object in CouchDB, each key must be unique.

Consider the following CouchDB document:

{
  "_id": "myname",
  "name": "Joan",
  "name": "Fred"
}

We can't store each name, so we pick the last one and use that. "Last key wins."

In your query:

    "selector" : {        
        "age": {
            "$lte": 30
        },
        "age": {
            "$gte": 25
        }
    },    
    ...

you have two keys under selector that are the same: age (i.e., selector.age). CouchDB picks the latter, which is why age > 25 wins.

Implicit operators work fine when each provided key is different, so that the actual JSON that you're entering is valid JSON.

[wohali]

@jaydoane Arguably this is the right thing to do, per RFC7493.

I don't know if we can actually detect this as a 400 due to where jiffy sits in the web stack, but if you think you can file a PR to fix it (at least for GET /{db}/_index, I don't see why it wouldn't be accepted.

Some links for reference:

https://lists.apache.org/thread.html/b728fb00d670747e312f25228ad1d962e93dfd8b4277de9af1160332%40%3Cdev.couchdb.apache.org%3E

apache/couchdb-jiffy@323568e
davisp/jiffy#162

[hamiltonsalazar]

@wohali this is confusing. For example on the CouchDB documetacion for topics with "The $not operator" and "The $nor operator" in "1.3.6.1.7. Combination Operators" it shows the below query:

    "year": {
        "$gte": 1900
    },
    "year": {
        "$lte": 1903
    },

What is the objective with that?

[wohali]

Hrm, I wouldn't expect that to work at all. Does it?

@garrensmith any comment here?

[hamiltonsalazar]

No really, it doesn't. I don't understand why is it there.

[wohali]

OK, looks like a documentation bug then. Thanks.

Once there's confirmation from @garrensmith we'll get the docs updated. Thanks for bringing this to our attention!

[garrensmith]

I would say its a documentation bug. If you run explain on your query what does it return back? If it doesn't automatically add in the $and then its a documentation bug.

[hamiltonsalazar]

@garrensmith The responses to this query are at the top

[jaydoane]

It appears that the dedupe_keys option causes jiffy to complain when set:

(node1@127.0.0.1)377> jiffy:decode(<<"{\"foo\":\"bar\", \"foo\":\"baz\"}">>, [return_maps]).
#{<<"foo">> => <<"baz">>}
(node1@127.0.0.1)378> jiffy:decode(<<"{\"foo\":\"bar\", \"foo\":\"baz\"}">>, [dedup_keys, return_maps]).
** exception error: bad argument
     in function  jiffy:nif_decode_init/2
        called as jiffy:nif_decode_init(<<"{\"foo\":\"bar\", \"foo\":\"baz\"}">>,
                                        [dedup_keys,return_maps])
     in call from jiffy:decode/2 (src/jiffy.erl, line 69)

And on master at least, it seems that the _find endpoint obtains the request body in maybe_set_partition, which makes a call to the ?JSON_DECODE macro which in turn ultimately calls couch_util:json_decode, which calls jiffy:decode using the dedupe_keys option here, so in principle it seems like it should already be preventing this from occurring.

So the reported behavior seems a bit puzzling on the surface. I'll try to repro once I have more time.

[garrensmith]

The query should rather be:

{     
    "selector" : {        
        "age": {
            "$lte": 30,
            "$gte": 25
        }
    },    
    "fields": [   "name",   "age",    "_id"     ], 
    "sort": [
        {
            "age": "desc"
        }   
    ]
}

I think there is also a documentation bug we can't have two keys in the selector with the same name