Deserializing json into a general struct

[klemenpeganc]

hi! Is it possible to somehow read in json without providing the struct+bindings (perhaps reading it in as some kind of an unordered_map)? I am migrating from nlohmann and this is the only feature that I need that feels like it is missing. If you look at this example, you can probably see what I mean: https://github.com/nlohmann/json#json-as-first-class-data-type . Is this perhaps possible with glaze or do I need to create a struct+bindings for everything?

Thanks for the help!

[stephenberry]

std::variant support exists, but it currently doesn't deduce the type from the JSON. This is intentional, for performance reasons and to encourage working directly with C++ types.

If you know nothing about the JSON and purely want to query what it contains, then libraries like nlohmann and simdjson are probably better suited. But, most of the time you do know something about what you are trying to load. Glaze is intended to know something about the input, even if that means some values are optional or variants.

Generic writing is easily supported as this example shows:

std::map<std::string, std::variant<std::string, int, bool>> request; // or std::unordered_map

request["username"] = "paulo";
request["password"] = "123456";
request["remember"] = true;

auto str = glz::write_json(request);

expect(str == R"({"password":"123456","remember":true,"username":"paulo"})");

Could you give a small example of what you are trying to achieve with generic reading?

We are looking at automatic type deduction for std::variant of basic types (e.g. strings, numbers). However, this will not support arbitrarily nested objects.

The fact is that once you write code to query a specific item, you are expressing that you know the expected structure. But, we do realize it can be painful to define all these potential structures and sometimes a generic library like nlohmann is just easier to work with (albeit with a loss in performance).

[klemenpeganc]

Indeed I know exactly what I am receiving! I have two use cases for this:

1. I receive a json message, which can be deserialized into one of three structs (structA, structB, structC). I already provide bindings for these structs. Currently, I search in that std::string json message if I can find specific keywords which make it possible to know if it is A, B or C and only then i call glz::read_json(). Is there a better way to do that perhaps?

2. I have a json file with manually written partial-structs:

{
    "partialStructsA": [
        {
            "number": "1",
            "name": "one"
        },
        {
            "number": "2",
            "name": "two"
        }
    ],
    "partialStructsB": [
        {
            "number": "1",
            "value": "oneone",
            "name": "one"
        },
        {
            "number": "2",
            "value": "twotwo",
            "name": "one"
        }
    ]
}

Based on this data, I create the whole object. Is there a way to read this partial data?

EDIT: Solved 2. with:

std::unordered_map<std::string, std::vector<std::unordered_map<std::string, std::string>>> json;
glz::read_file_json(json,jsonFilePath);

[stephenberry]

Great, thanks for the clarification:

Case 1: When you have multiple types with similar data, Glaze supports variant type deduction for custom objects.
See the Variant Handling wiki link for more details. This lets you have different types with the same data and it will properly read into the correct type in a variant.
However, if you're getting the object data from a third party source, then you won't have the "type" information. In that case your method of searching for a specific key and then calling the appropriate glz::read_json is probably your best bet.

Case 2: Your solution is fine, but if you wanted your "number" to actually be a number and not a std::string, then this would require variant type deduction for basic types, which I had mentioned we were considering adding. So, I'll keep this example in mind.

[stephenberry]

I added an issue concerning basic variant type deduction, as I think its worth supporting.

[mwalcott3]

Yeah once you have type deduction for variants when they only have one of each basic types you could do something like this

   // Unfortunately recursive std::variants are not a thing so we cant do
   // using json_val= std::variant<double, std::string, bool, std::vector<json_val>, std::unordered_map<std::string, json_val>>;
   // But we can put it in a struct/class and refer to the struct
   struct generic_json
   {
      using array_t = std::vector<generic_json>;
      using object_t = std::unordered_map<std::string, generic_json>;
      using val_t = std::variant<double, std::string, bool, array_t, object_t>;
      val_t data{};
      
      //Hack to get it treat data as the value in the read/write code
      constexpr operator bool() const { return true; }
      val_t* operator->() { return &data; }
      val_t& operator*() { return data; }
   };

And that should be handled automatically. Getting null to work is a bit more annoying without making changes to read (About two lines of code if we are okay with it being type specific).

[mwalcott3]

Glaze has typically focused on strongly typed data, But I was thinking about this recently and with some modifications to how variant support works we should be able to handle completely generic json. I support this and don't think it'll be that hard to add.

[mwalcott3]

@klemenpeganc, Just added a very basic generic json type glz::json_t (glaze/json/json_t.hpp) in #132

glz::json_t json{};
std::string buffer = R"([5,"Hello World",{"pi":3.14}])";
glz::read_json(json, buffer);
assert(json[0].get<double>() == 5.0);
assert(json[1].get<std::string>() == "Hello World");
assert(json[2]["pi"].get<double>() == 3.14);

You can use it by itself or nested within one of your own meta objects or in a container. I would recommend using typed JSON like variants if possible but this offers a way to handle things when thats not possible.

As pointed out by @stephenberry your example use case of having 3 structs (structA, structB, structC) is probably better handled by using std::variant as detailed here https://github.com/stephenberry/glaze/wiki/Variant-Handling#objects. Right now this depends on a type key when you have multiple object types but perhaps we might add a way for uses to inject their own methodology to deduce types but I can't think of a clean way to do that.

Edit:
Modified initialization for the generic glz::json_t type to more closely match nlohmann's api in #133
Allows for this:

glz::json_t json = {
   {"pi", 3.141},
   {"happy", true},
   {"name", "Niels"},
   {"nothing", nullptr},
   {"answer", {{"everything", 42.0}}},
   {"list", {1.0, 0.0, 2.0}},
   {"object", {
      {"currency", "USD"},
      {"value", 42.99}
   }}
};

Or this:

glz::json_t json;
json["pi"] = 3.14;
json["happy"] = true;
json["name"] = "Niels";
json["nothing"] = nullptr;
json["answer"]["everything"] = 42.0;
json["list"] = { 1.0, 0.0, 2.0 };
json["object"] = { {"currency", "USD"}, {"value", 42.99} };