Jackson not invoking Custom deserializer when set on a field as well as for custom annotations

[ipsi-apant]

I have added all details in this question in stack overflow.

https://stackoverflow.com/questions/77122208/jackson-custom-annotation-for-field-level-deserialization

In my case the incoming json is less jackson friendly. It has nested objects, plus the field names are different. Also, information of one nested object needs to be set on multiple object values of a Person class. (OrderRequest in the below example json).

Tech stack

• Java 17
• Springboot 3.1.3
• WebFlux

Is there a way to achieve this ? The fact that it needs to work with Flux and Mono (webflux).
After this attempt, I have already wrote multiple custom deserializers. Jackson fails for invoke custom deserializer.

ProductDeserializer and AddressDeserializer never invoked in unit test.

Input json sample

{
  "request": {
    "requestModel": {
      "age": 30,
      "ownerName": "test",
      "streetName": "some street",
      "cityName": "some city",
      "country": "some country",
      "zip": "123456",
      "type": "online",
      "referenceNo": "36f15755-b75a-4825-9771-4f6cb01bdafb",
      "requestorIp": "0.0.0.0",
      "productName": "product101",
      "category": "books",
      "amount": 25.99,
      "code": 32,
      "transactionId": "123d195b6a50e44",
      "channel": "api",
      "txnTimeStamp": 1694654251067
    }
  },
  "baggage": {
    "distributedTimeout": {
      "expiryTime": {
        "seconds": 1694654278,
        "nanos": 125000000
      }
    }
  }
}

Java classes

@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
@JsonIgnoreProperties(ignoreUnknown = true)
public class OrderRequest {

    // this works with unpackBaggage() and jsonproperty 'baggage'
    private String baggageExpiryTime;

    // value present in request.requestmodel
    private Address address;

    // value present in request.requestmodel
    private Product product;
}


@Value
@Builder
@With
@AllArgsConstructor
@JsonIgnoreProperties(ignoreUnknown = true)
@JsonDeserialize(using = AddressDeserializer.class)
public class Address {

// field names in input json are different
  String city;
  String street;
  String country;
  String postalCode;
}


@Value
@Builder
@With
@AllArgsConstructor
@JsonIgnoreProperties(ignoreUnknown = true)
@JsonDeserialize(using = ProductDeserializer.class)
public class Product {

// field names in input json are different
  String productCode;
  String name;
  String category;
}

[JooHyukKim]

Unfortunately usage via 3rd party frameworks (such as Spring, WebFlux, etc..) is not the best shape to treate issues in Jackson, since they have their own configuration settings, mechanisms and overrides --which happen to be the cause in most cases. They usaully wrap Jackson with something their own.

So could you provide tests, classes, configurations in plain Jackson? That would be

• ObjectMapper configuration how you configured it
• Classes to de/serialize
• Expected input/output JSON and the actual

Thanks!

[ipsi-apant]

Hi @JooHyukKim, thank you for your time for reviewing this.

ah, I forgot to mention the attempt was without springboot ObjectMapper (my bad), but expectation it needs to work with bodyToMono() (I can work on this one when below example works).
The classes and unit tests I have written in stackoverflow here are pure Jackson. A new ObjectMapper instance created in unit tests, then set the introspector before readValue(). I am copying the deserializer and introspector here for a reference.

Also, tried without lombok assuming it could have not passed the annotations. The effect is the same, it never reached to deserialize().

Expected input data

{
  "request": {
    "requestModel": {
      "age": 30,
      "ownerName": "test",
      "streetName": "some street",
      "cityName": "some city",
      "country": "some country",
      "zip": "123456",
      "type": "online",
      "referenceNo": "36f15755-b75a-4825-9771-4f6cb01bdafb",
      "requestorIp": "0.0.0.0",
      "productName": "product101",
      "category": "books",
      "amount": 25.99,
      "code": 32,
      "transactionId": "123d195b6a50e44",
      "channel": "api",
      "txnTimeStamp": 1694654251067
    }
  },
  "baggage": {
    "distributedTimeout": {
      "expiryTime": {
        "seconds": 1694654278,
        "nanos": 125000000
      }
    }
  }
}

Custom annotation

// custom annotation

@Target({ElementType.FIELD})
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface JsonPointerProperty {

    String value();
}

POJO classes

Expected output

// main POJO for incoming json
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
@JsonIgnoreProperties(ignoreUnknown = true)
public class OrderRequest {

    // this works with unpackBaggage() and jsonproperty 'baggage'
    private String baggageExpiryTime;

    // value present in request.requestmodel
    // @JsonProperty("request")
    private Address address;

    // value present in request.requestmodel
    // @JsonProperty("request")
    private Product product;

    private void unpackBaggage(JsonNode baggage) {
        
        final JsonNode expiryTime = baggage.at(JsonPointer.compile("/distributedTimeout/expiryTime"));
        this.baggageExpiryTime = Instant
        .ofEpochSecond(
            expiryTime.get("seconds").asLong(),
            expiryTime.get("nanos").asLong()
        );

    }
}

// nested object
@Value
@Builder
@With
@AllArgsConstructor
@JsonIgnoreProperties(ignoreUnknown = true)
@JsonDeserialize(using = AddressDeserializer.class)
public class Address {

// field names in input json are different
  @JsonPointerProperty("/request/requestModel/streetName")
  String street;
  
  @JsonPointerProperty("/request/requestModel/cityName")
  String city;
  
  @JsonPointerProperty("/request/requestModel/country")
  String country;

  @JsonPointerProperty("/request/requestModel/zip")
  String postalCode;
}

// nested object
@Value
@Builder
@With
@AllArgsConstructor
@JsonIgnoreProperties(ignoreUnknown = true)
@JsonDeserialize(using = ProductDeserializer.class)
public class Product {

// field names in input json are different
  @JsonPointerProperty("/request/requestModel/code")
  String productCode;

  @JsonPointerProperty("/request/requestModel/productName")
  String name;

  @JsonPointerProperty("/request/requestModel/category")
  String category;
}

Annotation introspector

// introspector 

public class JsonPointerPropertyIntrospector
        extends NopAnnotationIntrospector {

    @Override
    public Object findDeserializer(Annotated am) {

        final JsonPointerProperty jsonPointerProperty = am.getAnnotation(JsonPointerProperty.class);
        if (jsonPointerProperty != null) {
            return new JsonPointerPropertyDeserializer(
                    jsonPointerProperty.value(),
                    am.getType().getRawClass()
            );
        }

        return super.findDeserializer(am);
    }
}

Customer Deserializer - not reaching here

Another thought, if could get the Class<?> where @JsonDeserialize() is declared, then I could use reflection to read fields and their custom annotations for processing. I didn't find any method in JsonParser and DeserializationContext which could give declared class.

// custom deserializer
public class JsonPointerPropertyDeserializer
        extends JsonDeserializer<Object> {

    private final String jsonPath;
    private final Class<?> clazz;

    public JsonPointerPropertyDeserializer() {
        this(null, null);
    }

    public JsonPointerPropertyDeserializer(String jsonPath,
                                           Class<?> clazz) {
        this.jsonPath = jsonPath;
        this.clazz = clazz;
    }

    @Override
    public Object deserialize(JsonParser parser, DeserializationContext ctx)
            throws IOException, JacksonException {

        // not reaching here
        JsonNode jsonNode = parser.readValueAs(JsonNode.class);
        JsonNode at = jsonNode.at(jsonPath);

        String value = ctx.readTreeAsValue(jsonNode, String.class);

        System.out.println(value);

        return value;
    }
}

Unit test

// Unit test - not running with @SpringBootTest

class PersonTest {
    String json;

    @BeforeEach
    void setup() throws IOException {
        json = "{ 'request': { 'requestModel': { 'age': 30, 'ownerName': 'test', 'streetName': 'some street', 'cityName': 'some city', 'country': 'some country', 'zip': '123456', 'type': 'online', 'referenceNo': '36f15755-b75a-4825-9771-4f6cb01bdafb', 'requestorIp': '0.0.0.0', 'productName': 'product101', 'category': 'books', 'amount': 25.99, 'code': 32, 'transactionId': '123d195b6a50e44', 'channel': 'api', 'txnTimeStamp': 1694654251067 }}, 'baggage': { 'distributedTimeout': { 'expiryTime': { 'seconds': 1694654278, 'nanos': 125000000 }} }}"
                .replace(
                "'",
                "\""
        );
    }

    @Test
    void testPerson() throws JsonProcessingException {
        ObjectMapper mapper = new ObjectMapper();
        mapper.setAnnotationIntrospector(new JsonPointerPropertyIntrospector());

        final Person person = mapper.readValue(json, Person.class);

        assertEquals("some street", person.getAddress().getStreet(), "Street mismatch");
    }
}

[cowtowncoder]

Ok this is not going to work the way you envision it: incoming structure must match to some degree wrt outer OrderRequest class and address and product properties; they need to match JSON properties of same names to be called. How else would Jackson know when to invoke them?
But since your JSON does not seem to have compatible structure, you need to do something else.
You could bind request to a deserializer, for example, and that deserializer would get called.
As things are, there is nothing connecting POJO fields to incoming JSON.

[ipsi-apant]

Hi @cowtowncoder,
I tried binding a customer deserializer to OrderRequest. Unfortunately, @JsonProperty("request") I could use only once within the same POJO. It works for address but unable to apply it for product.

Curious to know, when there is a field level @JsonDeserializer(), can it be force to execute ? A boolean flag maybe, default false for backward compatibility. Or an enum constant, DeserializationStrategy having FIELD_NAME, ALWAYS (this is for force execute).
Currently, it only execute if the incoming json has matching field (address or product in this example). Force execute can give an opportunity to execute ser-de and within custom ser-de I can use JsonPointer to map the fields.

Let me know your thought on this.

[cowtowncoder]

No, that's not how serialization/deserialization works. Instead, incoming JSON (etc) input has to match a POJO property (field, setter) and only then JsonDeserializer is fetched and used. Process starts with content to match, not with POJO properties to find.
Beyond that, all deserializers operate on input/token stream, accessible via JsonParser: there is no buffering except as instruction by deserializer. It is not possible to apply more than one deserializer on same content; delegation is possible and is done for POJO fields of course (so sub-trees are handled by deserializers delegate to by "parent" deserializer).

So if you need to process content where structure of incoming JSON does not match POJO structure, you will need custom deserialization at highest level where structures diverge.

[steinarb]

I am trying to parse the JSON structures of the ActivityStreams vocabulary into a hierachy of sealed interfaces (with the leaf nodes being records).

The plan is to make something that parses ActivityPub responses into Java objects that can be handled with Java 21 pattern matching (with the hope that this will make the logic handling the responses as clear as possible). We'll see if I get there how that works out.

I'm working my way through the examples of the ActivityStreams vocabulary and I'm currently stuck at example 13.

Some (many, in fact) fields in the ActivityStreams JSON may be either the object inlined, or they may be a link to where the JSON for the object can be downloaded from.

E.g. a property may be either this

  "actor": {
    "type": "Person",
    "name": "Sally"
  },

or it may be a link to the object

  "actor": {
    "type": "Link",
    "href": "https://mymastodon.com/people/sally"
  },

So far, so good. I'm able to parse this.

But then I ran into the fact that Link is often represented as just the URL as a string, i.e. the above will be just:

  "actor": "https://mymastodon.com/people/sally",

The first thing I tried was custom deserializers. But @JsonDeserialize(using=MyDeserializer.class) doesn't work on record fields and custom deserializing the entire class felt like too much work.

But then I thought I had found a solution by using @JsonDeserialize(converter=StringToLinkConverter.class) .

I.e. I could make a converter like this

public class StringToLinkConverter extends StdConverter<String, Link> {
    @Override
    public Link convert(String value) {
        return Link.with().href(value).build();
    }
}

and use it like this (in a sealed inheritance hierarchy leaf node):

public record Invite(
    @JsonGetter("@context") Object context,
    ActivityStreamObjectType type,
    String id,
    String name,
    String summary,
    @JsonDeserialize(converter = StringToLinkConverter.class)
    LinkOrObject actor,
    LinkOrObject target,
    ActivityStreamObject object
) implements Offer
{
}

This worked fine for translating strings into Link objects, so I thought I had solved it... until I ran into an "actor" field that wasn't a string but an object.

Then I got:

 com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type `java.lang.String` from Object value (token `JsonToken.START_OBJECT`)

I.e. now only string is accepted.

So now I'm wondering what to do...?

[cowtowncoder]

  But @JsonDeserialize(using=MyDeserializer.class) doesn't work on record fields a

It definitely should -- as of 2.18.0 at least. If not, it'd be good if you could file an issue with minimized reproduction? (trying to trim down details to minimum).

[steinarb]

I'm currently on 2.17.2.

I will try bumping jackson to 2.18.0 and if @JsonDeserialize(using=MyDeserializer.class) still doesn't work on record fields I'll create a minimized test case and create an issue.

[steinarb]

Field annotation of @JsonDeserialize(using=LinkDeserializerHandlingStringUrl.class) still not picked up with jackson 2.18.0.

Or at least: a breakpoint in LinkDeserializerHandlingStringUrl.deserialize() is never called

Before I create a test case, just checking that I'm not doing something wrong.

This is the deserializer (just a skeleton so far to see if I can make jackson pick it up and try to use it):

public class LinkDeserializerHandlingStringUrl extends StdDeserializer<LinkOrObject> {

    public LinkDeserializerHandlingStringUrl(Class<?> vc) {
        super(vc);
    }

    protected LinkDeserializerHandlingStringUrl() {
        this(null);
    }

    private static final long serialVersionUID = 1L;

    @Override
    public LinkOrObject deserialize(JsonParser p, DeserializationContext ctxt) throws IOException, JacksonException {
        return null;
    }

}

(Note: I have no idea what's the right thing to do with constructor here? I don't know what the correct base class to use is? and I'm not sure of what the generic type should be?)

This is the record type with the annotation:

public record Add(
    @JsonGetter("@context") Object context,
    ActivityStreamObjectType type,
    String id,
    String name,
    String summary,
    Link url,
    LinkOrObject actor,
    LinkOrObject target,
    LinkOrObject origin,
    @JsonDeserialize(using = LinkDeserializerHandlingStringUrl.class)
    LinkOrObject object
) implements Activity
{
}

This is the file I'm trying to parse:

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "summary": "Sally added an object",
  "type": "Add",
  "actor": {
    "type": "Person",
    "name": "Sally"
  },
  "object": "http://example.org/abc"
}

This is the error message I'm getting (fails on object because it doesn't find the type property)

com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Could not resolve subtype of [simple type, class no.priv.bang.ratatoskr.asvocabulary.LinkOrObject]: missing type id property 'type' (for POJO property 'object')
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 9, column: 13] (through reference chain: no.priv.bang.ratatoskr.asvocabulary.Add["object"])
	at com.fasterxml.jackson.databind.exc.InvalidTypeIdException.from(InvalidTypeIdException.java:43)
	at com.fasterxml.jackson.databind.DeserializationContext.missingTypeIdException(DeserializationContext.java:2051)
	at com.fasterxml.jackson.databind.DeserializationContext.handleMissingTypeId(DeserializationContext.java:1622)
	at com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._handleMissingTypeId(TypeDeserializerBase.java:309)
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedUsingDefaultImpl(AsPropertyTypeDeserializer.java:211)
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:122)
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromAny(AsPropertyTypeDeserializer.java:240)
	at com.fasterxml.jackson.databind.deser.std.StdDeserializer.deserializeWithType(StdDeserializer.java:172)
	at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:540)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:578)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:447)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1497)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:348)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:220)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187)
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:170)
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:136)
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:263)
	at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:74)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:342)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4917)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3897)
	at no.priv.bang.ratatoskr.asvocabulary.ParseTest.testParseExample012(ParseTest.java:256)

This is the code that does the parsing (the bottom line in the above stack trace):

        LinkOrObject object = mapper.readValue(refactorExample("example_012.json"), LinkOrObject.class);

This is the LinkOrObject top level interface:

@JsonTypeInfo(use=JsonTypeInfo.Id.NAME, include=JsonTypeInfo.As.EXTERNAL_PROPERTY, property = "type" )
@JsonSubTypes({
    @Type(value = ActivityStreamObjectRecord.class, name = ActivityStreamObjectType.Names.OBJECT),
    @Type(value = ActivityRecord.class, name = ActivityStreamObjectType.Names.ACTIVITY),
    @Type(value = AcceptRecord.class, name = ActivityStreamObjectType.Names.ACCEPT),
    @Type(value = TentativeAccept.class, name = ActivityStreamObjectType.Names.TENTATIVE_ACCEPT),
    @Type(value = Add.class, name = ActivityStreamObjectType.Names.ADD),
    @Type(value = Invite.class, name = ActivityStreamObjectType.Names.INVITE),
    @Type(value = Travel.class, name = ActivityStreamObjectType.Names.TRAVEL),
    @Type(value = CollectionRecord.class, name = ActivityStreamObjectType.Names.COLLECTION),
    @Type(value = OrderedCollectionRecord.class, name = ActivityStreamObjectType.Names.ORDERED_COLLECTION),
    @Type(value = CollectionPage.class, name = ActivityStreamObjectType.Names.COLLECTION_PAGE),
    @Type(value = OrderedCollectionPage.class, name = ActivityStreamObjectType.Names.ORDERED_COLLECTION_PAGE),
    @Type(value = Person.class, name = ActivityStreamObjectType.Names.PERSON),
    @Type(value = Group.class, name = ActivityStreamObjectType.Names.GROUP),
    @Type(value = Image.class, name = ActivityStreamObjectType.Names.IMAGE),
    @Type(value = Note.class, name = ActivityStreamObjectType.Names.NOTE),
    @Type(value = Place.class, name = ActivityStreamObjectType.Names.PLACE),
    @Type(value = Event.class, name = ActivityStreamObjectType.Names.EVENT),
    @Type(value = Link.class, name = ActivityStreamObjectType.Names.LINK)
})
public sealed interface LinkOrObject permits Link, ActivityStreamObject {

    public ActivityStreamObjectType type();

}

(and basically what I'm looking for is a way to parse a string property, where LinkOrObject is expected, into a Link record (with Link.href holding the string value))

[steinarb]

Note: I am waiting for a confirmation that what I have tried to do above looks legit before trying to create a test case.

Alternatively I can just split out the asvocabulary project as a singleton maven project (keeping all of the types in the hierarchy and all of the test parse files). That would be simpler to do than creating a minimal test case (the project as it is just contains a hierarchy of interfaces terminated in records and a JUnit 5 test parsing (currently) 13 example files).

Please let me know what's preferred?

[JooHyukKim]

Minimal reproduction would be great, thanks!
This is so we can add to (most probably) jackson-databind repo.
Hopefully minimal and Jackson only so that we can just copy paste into databind test suite..

Also, wrt interfaces, I don't think we need all of them especially reproduction input JSON is just one?

[steinarb]

>>>> "Kim, Joo Hyuk" ***@***.***>: Minimal reproduction would be great, thanks!

Before I start the work on creating a testcase: what I have done so far looks correct? Ie. I haven't done anything that looks stupid or wrong in the definition of the deserializer? (the class used in the constructor? The choice of base class?) And the placement and usage of the annotation looks correct?

This is so we can add to (most probably) jackson-databind repo. Hopefully minimal and Jackson only so that we can just copy paste into [databind test suite.](https://github.com/FasterXML/jackson-databind). Also, wrt interfaces, I don't think we need all of them especially reproduction input JSON is just one?

No, there is 3 actually. The top level object is a Transaction of type Add (Transaction is a subtype IntransitiveAction, which is a subtype of ActivityStreamsObject, which in turn is a subtype of LinkOrObject). Then there is a Person, that is a subtype of type Actor, that is a subtype of ActivityStreamsObject which is a subtype of LinkOrObject. And finally there is the object which is a string that should be turned into a Link. And the object is the one that turned into a problem when parsing example 13, because there, instead of a string that should be turned into a Link, there is an inlined object of type Image (which is a subtype of Document which is a subtype of ActivityStreamsObject which is a subtype the top level interface LinkOrObject). https://www.w3.org/TR/activitystreams-vocabulary/#dfn-add (I kinda wonder if the people writing the spec actually tried parsing this stuff...? One thing I have flipped around is the inheritance order of Action and IntransitiveAction because there they operated with removing members in the derived class, and that... Java can't do... oh well!)

[cowtowncoder]

At this point, the minimal reproduction really should focus on that one Record field with annotated custom deserializer; deserializer itself returning whatever constants test can verify and not try to do anything else.
So minimizing types used (just one impl etc, whatever interface types needed).

Failing test would go under src/test-jdk17/java/com/fasterxml/jackson/databind/tofix/ for pr, ideally.

So, assuming a simple reproduction is doable, we'd need a pr on jackson-databind to show it.
Let's solve one problem at a time.

[steinarb]

Here is a somewhat minimized test case https://github.cottps://m/steinarb/record-field-deserializer-repro

The troublesome record, is: https://github.com/steinarb/record-field-deserializer-repro/blob/master/src/main/java/no/priv/bang/ratatoskr/asvocabulary/Add.java#L31

Run all tests in the project in an IDE or just run with

mvn clean install

[cowtowncoder]

Hmmh. I must have bit different definition of minimized -- I was hoping for a single class, stand-alone unit test. :)

But I will have a look -- thank you for providing the test case!

[cowtowncoder]

Ok; only 1 test fails (with 2.18.1-SNAPSHOT; build from 2.18 branch), and that for missing type id.
It does not indicate specific problem on missing deserializer.

But this is way too big for me to digest; I do not really have time to spend to fully understanding everything here.
What is needed is specific, targeted, minimal test case.

But FWTW: one thing that seems wrong in LinkOrObject definition is use of

include=JsonTypeInfo.As.EXTERNAL_PROPERTY

in class definition: it should only be used on properties, not classes.
I am not sure why you use that choice: either As.PROPERTY or As.EXISTING_PROPERTY would make more sense.

[cowtowncoder]

Another note: while LinkDeserializerHandlingStringUrl may return null for testing, it actually MUST also consume content: so it should call

    p.skipChildren();

to make sure all content matching expected value is consumed.

This does not matter if the current content is a single scalar value; but if value was Object or Array (i.e. multi-token value), leaving content unread would lead to JsonParser state being incorrect and further parsing failing.
I don't think it matters here (adding that won't make test pass) but thought worth mentioning.

[steinarb]

>>>> Tatu Saloranta ***@***.***>: Ok; only 1 test fails (with 2.18.1-SNAPSHOT; build from `2.18` branch), and that for missing type id. It does not indicate specific problem on missing deserializer.

No, but that error message comes even when this annotation isn't present ``` @JsonDeserialize(using = LinkDeserializerHandlingStringUrl.class) ``` The error message comes because a string is encountered https://github.com/steinarb/record-field-deserializer-repro/blob/master/src/test/resources/json/activitystreams-vocabulary/example_012.json#L9 where a LinkOrObject is expected https://github.com/steinarb/record-field-deserializer-repro/blob/master/src/main/java/no/priv/bang/ratatoskr/asvocabulary/LinkOrObject.java#L43 I managed to make this object field parse by annotating the field with ``` @JsonDeserialize(converter = StringToLinkConverter.class) ``` But then an object field that contained a LinkOrObject subtype failed: https://github.com/steinarb/record-field-deserializer-repro/blob/master/src/test/resources/json/activitystreams-vocabulary/example_013.json#L9 (the test for this, you can currently see, is green) Apparently ***@***.***(converter = StringToLinkConverter.class)` made the field be "string" only, when what I hoped for was "parse LinkOrObject normally, but if this is a string, create a Link object and use the string value as Link.href".

But this is way too big for me to digest; I do not really have time to spend to fully understanding everything here. What is needed is specific, targeted, minimal test case. But FWTW: one thing that seems wrong in `LinkOrObject` definition is use of include=JsonTypeInfo.As.EXTERNAL_PROPERTY in class definition: it should only be used on properties, not classes. I am not sure why you use that choice: either `As.PROPERTY` or `As.EXISTING_PROPERTY` would make more sense.

What was trying for here, was to use the LinkOrObject.type field to determine the actual type to instantiate. I used the example I found here and adapted to my type hierarchy https://www.baeldung.com/java-jackson-polymorphic-deserialization#polymorphic-deserialization-using-jsontypeinfo-with-jsonsubtypes

[cowtowncoder]

Ah! Look closely at https://www.baeldung.com/java-jackson-polymorphic-deserialization#polymorphic-deserialization-using-jsontypeinfo-with-jsonsubtypes and you notice:

@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, include = JsonTypeInfo.As.EXISTING_PROPERTY, 
 property = "type", visible = true)

so EXISTING_PROPERTY and not EXTERNAL_PROPERTY.

Problem is that EXTERNAL_PROPERTY must be used on polymorphic field (or setter/getter) because of... reasoning has to do with internal implementation but essentially handling must be enclosing class, and this cannot be done when annotation is on value class.

[steinarb]

Thanks! I blame IDEs and their auto expansion... 😄

Corrected and pushed on both branches (and checked if test still ran red. It did).

[steinarb]

>>>> Tatu Saloranta ***@***.***>: Another note: while `LinkDeserializerHandlingStringUrl` may return `null` for testing, it actually MUST also consume content: so it should call p.skipChildren(); to make sure all content matching expected value is consumed. This does not matter if the current content is a single scalar value; but if value was Object or Array (i.e. multi-token value), leaving content unread would lead to `JsonParser` state being incorrect and further parsing failing. I don't think it matters here (adding that won't make test pass) but thought worth mentioning.

Ok, LinkDeserializerHandlingStringUrl.deserialize() now calls JsonParser.skipChildren().

[steinarb]

>>>> Tatu Saloranta ***@***.***>: Hmmh. I must have bit different definition of minimized -- I was hoping for a single class, stand-alone unit test. :)

I don't have an easy way to get there, but the minimized-example branch has cut the inheritance hierarchy down to the code that actually participates in the test https://github.com/steinarb/record-field-deserializer-repro/tree/minimized-example But maybe I should have kept example_013 and the code needed for that, since that is where ***@***.***(converter = StringToLinkConverter.class)` broke...?

[JooHyukKim]

Okay, can we pause the discussion here @steinarb ? and think of better way?
Let me organize things bit more, for this discussion getting to long :/ (just to get tests right)

I filed PR including test verifying that your statement (below) is not valid.

The first thing I tried was custom deserializers. But @JsonDeserialize(using=MyDeserializer.class) doesn't work on record fields and custom deserializing the entire class felt like too much work.

This leaves us to sealed class and interface hirearchy that you declared @steinarb, which you can personally isolate failing case further into smaller piece. We don't even need that many to start with.

So please investigate further on your side (try googling) and comeback at different discussion.

Thank you!

[cowtowncoder]

Verifying that custom deserialize annotation works in isolation (the PR) is great. So what remains is isolating what is not working. So +1 to @JooHyukKim

[steinarb]

@JooHyukKim On this branch I've minimized the example to one interface derived by two record types, and the interface is a regular interface and not a sealed hierarchy.
https://github.com/steinarb/record-field-deserializer-repro/tree/minimized-example

The custom deserializer still isn't called.

[cowtowncoder]

Notes on reproduction: the problem is due to value being deserialized being polymorphic: method to call is not plain deserialize() but JsonDeserialize.deserializeWithType().
This because type LinkOrObject is polymorphic (has @JsonTypeInfo)... and is recursive definition too.

Btw, this is probably not related to Records, fwtw, but for any polymorphic types.

But I am not sure I really understand the intent of this design: object model looks quite strange, wich LinkOrObject implementations referring to other LinkOrObjects.

So, adding method:

    @Override
    public Object deserializeWithType(JsonParser p, DeserializationContext ctxt,
            TypeDeserializer typeDeserializer) throws IOException, JacksonException;

is the first step and shows custom deserializer is being used.

This method needs to call one of typeDeser.deserializeFromXxx() method:

• deserializeTypedFromObject() if representation of the value is JSON Object
• deserializeTypedFromArray() if representation of the value is JSON Array
• deserializeTypedFromScalar() if representation of the value is scalar (single-token, not array or object)
• deserializeTypedFromAny() if representation can be any JSON type

In this case, assuming value (with type id) is always JSON Object, we'd use:

    @Override
    public Object deserializeWithType(JsonParser p, DeserializationContext ctxt,
            TypeDeserializer typeDeserializer) throws IOException, JacksonException
    {
        return typeDeserializer.deserializeTypedFromObject(p, ctxt);
    }

This will then lead to another problem, however:

com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Could not resolve subtype of [simple type, class no.priv.bang.ratatoskr.asvocabulary.LinkOrObject]: missing type id property 'type' (for POJO property 'object')
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 4, column: 13] (through reference chain: no.priv.bang.ratatoskr.asvocabulary.Add["object"])
	at 
...

which is because we have

{
  "summary": "Sally added an object",
  "type": "Add",
  "object": "http://example.org/abc"
}

where Object has String value. But wait! "object" is:

    @JsonDeserialize(using = LinkDeserializerHandlingStringUrl.class)
    LinkOrObject object

and NOT JSON Object at all. Rats.

So... yeah it's a humongous mess. And I think you may be right that As.EXTERNAL_PROPERTY would be needed. It just cannot be a Class attribute.

But at the end of the day I don't know how to make this work: there is something wrong with the Object model. The problem is not, however, that custom deserializer wasn't getting called -- it is.
The problem is rather with inclusion of Type Id for polymorphic typing and type hierarchy. Those are defined in a way that does not and cannot work.

[cowtowncoder]

@steinarb Wait! Why did you attach your question on different discussion?!?!? What the...

Why did you not START A NEW DISCUSSION?!?!?!!

Now we have two unrelated threads -- just because you thought original might be somehow related does not mean you should hi-jack it.

This is ridiculous, closing this Discussion: feel free to open new one, will not continue on this mixed-up monster.