Allow schema less fields

[awb99]

It would be great if schema-less fields could be added to datahike.

Or provide an example how to add say an index field that then can be loaded from a store in konserve.

Without this feature I have to do first the query and then later on join data via some kind of index field. This is easy to do.

But what what is not so easy is that if I have say 5 different type of unstructured document storages that I then have to create 5 different konserve storages.

Above is related to queries.

There is alos the update workflow. If I can not stote unstructured data then I have to make a hook on each update that first extracts the schema compliant part and then later saves the schema uncomplicated part to a different storage.

This extra work makes my head spin.

Perhaps it really can be dealt with a simple 100 line clojure file. Perhaps interceptor patterns such as used in pedestal would make sense for this topic.

Thanks for your great work!

[kordano]

As I understand you correctly, you would like to have something like a blob or any value type? What kind of data do you have in the konserve store? For sure we can add something like the any type but the question would be on how to query that data. So how would you for instance compare and order that? We could maybe do that with hashes.

[awb99]

My usecase is pretty simple: I interact with 3 apis to pull data. So when I get the data from the apis, I just store everything I get. I use konserve for this. Each api I work with has a unique id for each entity, so I just store it in konserve say as [:xero/invoice 17] or [:aftership/tracking 333]. Then I start to add gradually the fields that I am interested in into datahike. I guess that a lot of developers
build apps like that. I also use it for somethign else: Since datahike does not support schema-on-write, I store all the data I have
in this konserve store, and once I have added a new field to my datahike schema, I simply run the import-conversion functions on
all items I have in the konserve store, and then I have a new datahike database from scratch. It is not so easy to rebuild a database
from api calls, because once data gets a little bigger, there are api throttel limits. A blob or a edn structure would be enough for me
to store with konserve. It took me a week to get running with my dual datahike + konserve solution. And now I actually like it.
Its clear that all datahike fields are searchable and useable in queries, and everything that is not in datahike is not. But I am worried
that my solution will get "out of sync" at some stage. This happens for example when my convert-to-datahike-tx fails, in which case
I wont have an index of the value I have stored in konserve.