Ignore `specificData` field on serialization #508

pjfanning · 2024-08-05T10:53:16Z

Fixes 1 of the 21 broken tests when you run with Avro 1.11.3.

The issues below are when running with 2.18 branch but avro 1.11.3. The one test that this PR fixes is:

[ERROR]   SerializeGeneratedTest.testWriteGeneratedEvent:18 » JsonMapping No field named 'specificData' (through reference chain: com.fasterxml.jackson.dataformat.avro.gen.Event35["specificData"])

[ERROR]   AvroAliasTest.testAliasedEnumForwardsCompatible:113->checkSchemaIsCompatible:144 Expected schema to be compatible but was not. Reason:
Data encoded using writer schema:
{
  "type" : "enum",
  "name" : "Size",
  "namespace" : "com.fasterxml.jackson.dataformat.avro.AvroTestBase",
  "symbols" : [ "SMALL", "LARGE" ]
}
will or may fail to decode using reader schema:
{
  "type" : "enum",
  "name" : "NewSize",
  "namespace" : "com.fasterxml.jackson.dataformat.avro.interop.annotations.AvroAliasTest",
  "symbols" : [ "SMALL", "LARGE" ],
  "aliases" : [ "com.fasterxml.jackson.dataformat.avro.AvroTestBase$.Size" ]
}

[ERROR]   AvroAliasTest.testAliasedEnumForwardsCompatible:113->checkSchemaIsCompatible:144 Expected schema to be compatible but was not. Reason:
Data encoded using writer schema:
{
  "type" : "enum",
  "name" : "Size",
  "namespace" : "com.fasterxml.jackson.dataformat.avro.AvroTestBase",
  "symbols" : [ "SMALL", "LARGE" ]
}
will or may fail to decode using reader schema:
{
  "type" : "enum",
  "name" : "NewSize",
  "namespace" : "com.fasterxml.jackson.dataformat.avro.interop.annotations.AvroAliasTest",
  "symbols" : [ "SMALL", "LARGE" ],
  "aliases" : [ "com.fasterxml.jackson.dataformat.avro.AvroTestBase$.Size" ]
}

[ERROR]   AvroAliasTest.testAliasedEnumForwardsCompatible:113->checkSchemaIsCompatible:144 Expected schema to be compatible but was not. Reason:
Data encoded using writer schema:
{
  "type" : "enum",
  "name" : "Size",
  "namespace" : "com.fasterxml.jackson.dataformat.avro.AvroTestBase",
  "symbols" : [ "SMALL", "LARGE" ]
}
will or may fail to decode using reader schema:
{
  "type" : "enum",
  "name" : "NewSize",
  "namespace" : "com.fasterxml.jackson.dataformat.avro.interop.annotations.AvroAliasTest",
  "symbols" : [ "SMALL", "LARGE" ],
  "aliases" : [ "com.fasterxml.jackson.dataformat.avro.AvroTestBase$.Size" ]
}

[ERROR]   AvroAliasTest.testAliasedEnumForwardsCompatible:113->checkSchemaIsCompatible:144 Expected schema to be compatible but was not. Reason:
Data encoded using writer schema:
{
  "type" : "enum",
  "name" : "Size",
  "namespace" : "com.fasterxml.jackson.dataformat.avro.AvroTestBase",
  "symbols" : [ "SMALL", "LARGE" ]
}
will or may fail to decode using reader schema:
{
  "type" : "enum",
  "name" : "NewSize",
  "namespace" : "com.fasterxml.jackson.dataformat.avro.interop.annotations.AvroAliasTest",
  "symbols" : [ "SMALL", "LARGE" ],
  "aliases" : [ "com.fasterxml.jackson.dataformat.avro.AvroTestBase$.Size" ]
}

[ERROR]   AvroAliasTest.testAliasedRecordForwardsCompatible:69->checkSchemaIsCompatible:144 Expected schema to be compatible but was not. Reason:
Data encoded using writer schema:
{
  "type" : "record",
  "name" : "Employee",
  "namespace" : "com.fasterxml.jackson.dataformat.avro.AvroTestBase",
  "fields" : [ {
    "name" : "age",
    "type" : "int"
  }, {
    "name" : "boss",
    "type" : "Employee"
  }, {
    "name" : "emails",
    "type" : {
      "type" : "array",
      "items" : "string",
      "java-class" : "[Ljava.lang.String;"
    }
  }, {
    "name" : "name",
    "type" : "string"
  } ]
}
will or may fail to decode using reader schema:
{
  "type" : "record",
  "name" : "NewEmployee",
  "namespace" : "com.fasterxml.jackson.dataformat.avro.interop.annotations.AvroAliasTest",
  "fields" : [ {
    "name" : "age",
    "type" : "int"
  }, {
    "name" : "boss",
    "type" : "NewEmployee"
  }, {
    "name" : "emails",
    "type" : {
      "type" : "array",
      "items" : "string",
      "java-class" : "[Ljava.lang.String;"
    }
  }, {
    "name" : "name",
    "type" : "string"
  } ],
  "aliases" : [ "com.fasterxml.jackson.dataformat.avro.AvroTestBase$.Employee" ]
}

[ERROR]   AvroAliasTest.testAliasedRecordForwardsCompatible:69->checkSchemaIsCompatible:144 Expected schema to be compatible but was not. Reason:
Data encoded using writer schema:
{
  "type" : "record",
  "name" : "Employee",
  "namespace" : "com.fasterxml.jackson.dataformat.avro.AvroTestBase",
  "fields" : [ {
    "name" : "age",
    "type" : "int"
  }, {
    "name" : "boss",
    "type" : "Employee"
  }, {
    "name" : "emails",
    "type" : {
      "type" : "array",
      "items" : "string",
      "java-class" : "[Ljava.lang.String;"
    }
  }, {
    "name" : "name",
    "type" : "string"
  } ]
}
will or may fail to decode using reader schema:
{
  "type" : "record",
  "name" : "NewEmployee",
  "namespace" : "com.fasterxml.jackson.dataformat.avro.interop.annotations.AvroAliasTest",
  "fields" : [ {
    "name" : "age",
    "type" : "int"
  }, {
    "name" : "boss",
    "type" : "NewEmployee"
  }, {
    "name" : "emails",
    "type" : {
      "type" : "array",
      "items" : "string",
      "java-class" : "[Ljava.lang.String;"
    }
  }, {
    "name" : "name",
    "type" : "string"
  } ],
  "aliases" : [ "com.fasterxml.jackson.dataformat.avro.AvroTestBase$.Employee" ]
}

[ERROR]   AvroAliasTest.testAliasedRecordForwardsCompatible:69->checkSchemaIsCompatible:144 Expected schema to be compatible but was not. Reason:
Data encoded using writer schema:
{
  "type" : "record",
  "name" : "Employee",
  "namespace" : "com.fasterxml.jackson.dataformat.avro.AvroTestBase",
  "fields" : [ {
    "name" : "age",
    "type" : "int"
  }, {
    "name" : "boss",
    "type" : "Employee"
  }, {
    "name" : "emails",
    "type" : {
      "type" : "array",
      "items" : "string",
      "java-class" : "[Ljava.lang.String;"
    }
  }, {
    "name" : "name",
    "type" : "string"
  } ]
}
will or may fail to decode using reader schema:
{
  "type" : "record",
  "name" : "NewEmployee",
  "namespace" : "com.fasterxml.jackson.dataformat.avro.interop.annotations.AvroAliasTest",
  "fields" : [ {
    "name" : "age",
    "type" : "int"
  }, {
    "name" : "boss",
    "type" : "NewEmployee"
  }, {
    "name" : "emails",
    "type" : {
      "type" : "array",
      "items" : "string",
      "java-class" : "[Ljava.lang.String;"
    }
  }, {
    "name" : "name",
    "type" : "string"
  } ],
  "aliases" : [ "com.fasterxml.jackson.dataformat.avro.AvroTestBase$.Employee" ]
}

[ERROR]   AvroAliasTest.testAliasedRecordForwardsCompatible:69->checkSchemaIsCompatible:144 Expected schema to be compatible but was not. Reason:
Data encoded using writer schema:
{
  "type" : "record",
  "name" : "Employee",
  "namespace" : "com.fasterxml.jackson.dataformat.avro.AvroTestBase",
  "fields" : [ {
    "name" : "age",
    "type" : "int"
  }, {
    "name" : "boss",
    "type" : "Employee"
  }, {
    "name" : "emails",
    "type" : {
      "type" : "array",
      "items" : "string",
      "java-class" : "[Ljava.lang.String;"
    }
  }, {
    "name" : "name",
    "type" : "string"
  } ]
}
will or may fail to decode using reader schema:
{
  "type" : "record",
  "name" : "NewEmployee",
  "namespace" : "com.fasterxml.jackson.dataformat.avro.interop.annotations.AvroAliasTest",
  "fields" : [ {
    "name" : "age",
    "type" : "int"
  }, {
    "name" : "boss",
    "type" : "NewEmployee"
  }, {
    "name" : "emails",
    "type" : {
      "type" : "array",
      "items" : "string",
      "java-class" : "[Ljava.lang.String;"
    }
  }, {
    "name" : "name",
    "type" : "string"
  } ],
  "aliases" : [ "com.fasterxml.jackson.dataformat.avro.AvroTestBase$.Employee" ]
}

[ERROR] Errors: 
[ERROR]   SerializeGeneratedTest.testWriteGeneratedEvent:18 » JsonMapping No field named 'specificData' (through reference chain: com.fasterxml.jackson.dataformat.avro.gen.Event35["specificData"])
[ERROR]   UnionTest.testInterfaceUnionWithCat:125->InteropTestBase.roundTrip:131->InteropTestBase.roundTrip:151 » UnresolvedUnion Not in union ["null",{"type":"record","name":"Cat","namespace":"com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$","fields":[{"name":"color","type":["null","string"]}]},{"type":"record","name":"Dog","namespace":"com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$","fields":[{"name":"size","type":{"type":"int","java-class":"java.lang.Integer"}}]}]: com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$Cat@6ed16657 (field=animal)
[ERROR]   UnionTest.testInterfaceUnionWithCat:125->InteropTestBase.roundTrip:131->InteropTestBase.roundTrip:151 » UnresolvedUnion Not in union ["null",{"type":"record","name":"Cat","namespace":"com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$","fields":[{"name":"color","type":["null","string"]}]},{"type":"record","name":"Dog","namespace":"com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$","fields":[{"name":"size","type":{"type":"int","java-class":"java.lang.Integer"}}]}]: com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$Cat@7cea0110 (field=animal)
[ERROR]   UnionTest.testInterfaceUnionWithDog:134->InteropTestBase.roundTrip:131->InteropTestBase.roundTrip:151 » UnresolvedUnion Not in union ["null",{"type":"record","name":"Cat","namespace":"com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$","fields":[{"name":"color","type":["null","string"]}]},{"type":"record","name":"Dog","namespace":"com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$","fields":[{"name":"size","type":{"type":"int","java-class":"java.lang.Integer"}}]}]: com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$Dog@1bc49bc5 (field=animal)
[ERROR]   UnionTest.testInterfaceUnionWithDog:134->InteropTestBase.roundTrip:131->InteropTestBase.roundTrip:151 » UnresolvedUnion Not in union ["null",{"type":"record","name":"Cat","namespace":"com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$","fields":[{"name":"color","type":["null","string"]}]},{"type":"record","name":"Dog","namespace":"com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$","fields":[{"name":"size","type":{"type":"int","java-class":"java.lang.Integer"}}]}]: com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$Dog@38aafb53 (field=animal)
[ERROR]   UnionTest.testListWithInterfaceUnion:155->InteropTestBase.roundTrip:131->InteropTestBase.roundTrip:151 » UnresolvedUnion Not in union ["null",{"type":"array","items":[{"type":"record","name":"Cat","namespace":"com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$","fields":[{"name":"color","type":["null","string"]}]},{"type":"record","name":"Dog","namespace":"com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$","fields":[{"name":"size","type":{"type":"int","java-class":"java.lang.Integer"}}]}]}]: [com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$Cat@44e93c1f, com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$Dog@42373389, com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$Dog@9b21bd3, com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$Cat@a62c7cd] (field=pets)
[ERROR]   UnionTest.testListWithInterfaceUnion:155->InteropTestBase.roundTrip:131->InteropTestBase.roundTrip:151 » UnresolvedUnion Not in union ["null",{"type":"array","items":[{"type":"record","name":"Cat","namespace":"com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$","fields":[{"name":"color","type":["null","string"]}]},{"type":"record","name":"Dog","namespace":"com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$","fields":[{"name":"size","type":{"type":"int","java-class":"java.lang.Integer"}}]}]}]: [com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$Cat@11a8042c, com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$Dog@6a4ccef7, com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$Dog@69391e08, com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$Cat@35eb4a3b] (field=pets)
[ERROR]   UnionTest.testMapWithInterfaceUnion:166->InteropTestBase.roundTrip:131->InteropTestBase.roundTrip:151 » UnresolvedUnion Not in union ["null",{"type":"array","items":[{"type":"record","name":"Cat","namespace":"com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$","fields":[{"name":"color","type":["null","string"]}]},{"type":"record","name":"Dog","namespace":"com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$","fields":[{"name":"size","type":{"type":"int","java-class":"java.lang.Integer"}}]}]}]: [com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$Cat@5c080ef3, com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$Dog@188cbcde, com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$Dog@4ee6291f, com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$Cat@2b03d52f] (field=pets)
[ERROR]   UnionTest.testMapWithInterfaceUnion:166->InteropTestBase.roundTrip:131->InteropTestBase.roundTrip:151 » UnresolvedUnion Not in union ["null",{"type":"array","items":[{"type":"record","name":"Cat","namespace":"com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$","fields":[{"name":"color","type":["null","string"]}]},{"type":"record","name":"Dog","namespace":"com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$","fields":[{"name":"size","type":{"type":"int","java-class":"java.lang.Integer"}}]}]}]: [com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$Cat@459b187a, com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$Dog@6b4283c4, com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$Dog@d0865a3, com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$Cat@636bbbbb] (field=pets)
[ERROR]   UnionTest.testRootUnionWithAnimal:116->InteropTestBase.roundTrip:151 » UnresolvedUnion Not in union [{"type":"record","name":"Cat","namespace":"com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$","fields":[{"name":"color","type":["null","string"]}]},{"type":"record","name":"Dog","namespace":"com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$","fields":[{"name":"size","type":{"type":"int","java-class":"java.lang.Integer"}}]}]: com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$Cat@65c33b92
[ERROR]   UnionTest.testRootUnionWithAnimal:116->InteropTestBase.roundTrip:151 » UnresolvedUnion Not in union [{"type":"record","name":"Cat","namespace":"com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$","fields":[{"name":"color","type":["null","string"]}]},{"type":"record","name":"Dog","namespace":"com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$","fields":[{"name":"size","type":{"type":"int","java-class":"java.lang.Integer"}}]}]: com.fasterxml.jackson.dataformat.avro.interop.annotations.UnionTest$Cat@49ec6a9f
[ERROR]   RecordWithComplexTest.testRecordWithOptionalEnumField:87->InteropTestBase.roundTrip:151 » UnresolvedUnion Not in union ["null",{"type":"enum","name":"DummyEnum","namespace":"com.fasterxml.jackson.dataformat.avro.interop.InteropTestBase$","symbols":["NORTH","SOUTH","EAST","WEST"]}]: SOUTH (field=optionalEnum)
[ERROR]   RecordWithComplexTest.testRecordWithOptionalEnumField:87->InteropTestBase.roundTrip:151 » UnresolvedUnion Not in union ["null",{"type":"enum","name":"DummyEnum","namespace":"com.fasterxml.jackson.dataformat.avro.interop.InteropTestBase$","symbols":["NORTH","SOUTH","EAST","WEST"]}]: SOUTH (field=optionalEnum)
[INFO] 
[ERROR] Tests run: 1448, Failures: 8, Errors: 13, Skipped: 50

cowtowncoder · 2024-08-11T23:50:58Z

Thank you @pjfanning -- I think this makes sense (tho didn't have time to dig deep to verify) and should help in small part to possibly offer a way to upgrade to later Avro apache codecs. At least for Jackson 3.0.

ignore specificData field

591f5dd

pjfanning mentioned this pull request Aug 5, 2024

Incompatibility with Avro >=1.9.0 (upgrade to Avro 1.11.3) #167

Closed

bgalek approved these changes Aug 5, 2024

View reviewed changes

cowtowncoder approved these changes Aug 11, 2024

View reviewed changes

cowtowncoder merged commit e1138de into FasterXML:2.18 Aug 11, 2024
4 checks passed

cowtowncoder changed the title ~~ignore specificData field~~ Ignore specificData field Aug 11, 2024

cowtowncoder added the avro label Aug 11, 2024

cowtowncoder changed the title ~~Ignore specificData field~~ Ignore specificData field on serialization Aug 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ignore `specificData` field on serialization #508

Ignore `specificData` field on serialization #508

pjfanning commented Aug 5, 2024

cowtowncoder commented Aug 11, 2024

Ignore specificData field on serialization #508

Ignore specificData field on serialization #508

Conversation

pjfanning commented Aug 5, 2024

cowtowncoder commented Aug 11, 2024

Ignore `specificData` field on serialization #508

Ignore `specificData` field on serialization #508