Rewrite Bean Property Introspection logic in Jackson 2.x #4515

cowtowncoder · 2024-05-05T00:46:03Z

Describe your Issue

We need to rewrite the Bean Property introspection (see #3719 for background); and while this is ambitious undertaking, I think it should be done in Jackson 2.x timeframe, not 3.x. That way code remains potentially mergeable from 2.x to 3.x.

The main drivers for rewrite are to solve problems that are difficult if not impossible to solve with the current mechanism:

Merging of annotations between Field/Method accessors and Creator (constructor/factory method) arguments
Handling of Record types and in particular alternate (non-canonical) constructors
Support for canonical constructors for non-Java JVM languages (Kotlin, Scala)

This issue is an umbrella ticket as refactoring will likely be spread across multiple PRs; they can all refer to this issue,

cowtowncoder · 2024-05-14T03:44:21Z

Started reading the code and clearly more of code wrt Creator detection needs to move from BeanDeserializerFactory/BasicDeserializerFactory (later part of processing) into POJOPropertiesCollector (earlier part of processing).

But not sure whether to try to move abstractions from Factory part, or rewrite; and whether to try move things incrementally or not.

Started removing deprecated methods from code paths, to help isolate actual in-use code.

JooHyukKim · 2024-05-14T07:34:08Z

But not sure whether to try to move abstractions from Factory part, or rewrite; and whether to try move things incrementally or not.

If we agree that we have enough test coverage, I guess.... rewrite, move big-step-incrementally?

cowtowncoder · 2024-05-14T15:40:20Z

@JooHyukKim Yes, I think test coverage is sufficient to allow refactoring. I am struggling at finding the steps tho. Especially in a way that would allow merging 2.18 -> master.
So it's not yet about mechanics but even conceptually which things to move or rewrite/re-implement.

And one problem part is that the latest BeanDeserializerFactory code for processing Creators relies on [Basic]BeanDescription, which would not be available during POJOPropertiesCollector (since BeanDescription is mostly used as wrapper for actual POJOPropertiesCollector.

cowtowncoder · 2024-05-14T23:06:33Z

Ok one big challenge that is coming up now, looking through functionality in BasicDeserializerFactory is the coupling to deserialization side in general (DeserializationConfig and DeserializationContext), and also dependency on context (DeserializationContext). This because POJOPropertiesCollector only operates on generic MapperConfig (base config of DeserializationConfig and SerializationConfig), and does not need or use context.
Dependencies are not trivial to remove; but they are also not exactly essential. But just means it is not easy to start by "lift-and-shift"ing (that is, by basically moving code from one place to the other), some rewriting required.

In the meantime I am making small incremental changes, including removal of deprecated code in affected areas.

JooHyukKim · 2024-05-15T02:01:52Z

Right. Also from my experience, many times it was necessary to both modify existing and add additional "internal" structure to make the code base in better shape. More of "tidy up just before making changes".

Just out of curiosity, are we going to introduce POJOPropertiesCollector, RecordPropertiesCollector and such for new version? This question is just from my imagination 😆. I am asking this because, though not necessary, if there's any plan in written format that I can refer to, so I can help.

cowtowncoder · 2024-05-15T02:19:09Z

Just out of curiosity, are we going to introduce POJOPropertiesCollector, RecordPropertiesCollector and such for new version?

Not planned yet. I am thinking it should be possible to make things work without fully separate implementation. But of course if it becomes necessary that could be done.

At this first I would really want to make existing logic work same as now, but get Creator properties discovered around time others are... so move it out of BasicDeserializerFactory into either POJOPropertiesCollector or where BasicBeanDescription is resolved.

cowtowncoder · 2024-05-15T03:39:57Z

Hmmmh. Despite incremental refactoring, I don't think functional from BasicDeserializerFactory can moved as-is into POJOPropertiesCollector -- it does "too much", including instantiation of deserializers and so on.
So what we need is a subset. Need to re-think things a bit.

...

Ok. So, in POJOPropertiesCollector I think handling needs to incorporate selection of actual Creator method and NOT just individual properties. And that information will then be used by BasicDeserializerFactory.

cowtowncoder · 2024-05-18T22:07:08Z

Good progress:

All JDK-8 tests pass with re-factored POJOPropertiesCollector
Only 4 Record tests fail
Still have duplication with BasicDeserializerFactory: will leave resolving that to second PR

Hoping to solve remaining Record tests, to get first PR merged into 2.18, then master.

First part of refactoring towards #4515.

cowtowncoder · 2024-05-21T00:22:54Z

Forgot to add an update: #4532 was merged 2 days ago, and it handles front-end half of changes:

POJOPropertiesCollector fully resolves explicit Properties-based creators, linking properties
Code in BasicDeserializerFactory still handles rest of introspection, including linking from (1) to Creators. This is to be removed as the next step.

I also started to look into the second half (reducing/removing code from BasicDeserializerFactory); it is somewhat complicated too, but can hopefully be tackled incrementally as well.

cowtowncoder · 2024-05-22T03:37:06Z

Ok some more progress: now BasicDeserializerFactory actually uses primary Properties-based Creator discovered by POJOPropertiesCollector. Additionally POJOPropertiesCollector keeps Creator-properties from that primary one better in-sync with getter/setter/field accessor info (in fact making failing test for #4119 pass -- but will not yet marked as fixed until getting further with refactoring).

But quite a bit remains to rewrite in BasicDeserializerFactory still.

…ods (#4549)

cowtowncoder · 2024-05-30T03:14:04Z

@JooHyukKim one quick note: while fixing issues with jackson-databind/3.0, I backported them into 2.x -- basically these are uncovered in 3.0 since it included parameter names module (unlike 2.x where it is separate). So there's a slight chance that something wrt Kotlin could be fixed by these changes.

JooHyukKim · 2024-05-30T23:32:26Z

Kotlin module still failing. So I filed a PR moving the test to failing FasterXML/jackson-module-kotlin#802.

StefanBratanov · 2024-06-13T13:59:58Z

Hi, just to confirm after release 2.18 we would no longer need the following workaround that I had to use in one of my projects - https://stackoverflow.com/questions/68394911/why-record-class-cant-be-properly-deserialized-with-jackson/68998917#68998917 ?

JooHyukKim · 2024-06-13T14:26:59Z

@StefanBratanov Probably. You are free to test ur case against Jackson 2.18 👍🏼.

In case it doesn't, let us know

yihtserns · 2024-06-13T14:42:48Z

@StefanBratanov I think your use case is already supported since 2.14 (see #2992 that was referenced in the StackOverflow answer).

…legating JsonCreator The refactor to bean property introspection in FasterXML#4515 caused existing no-arg delegatring constructors to fail deserialization. Example from this branch where we test RC releases: palantir/conjure-java#2349 specifically this test: https://github.com/palantir/conjure-java/blob/19d7f54eb0001c49b5d8a4bb897b9ad4cb5a28de/conjure-java-core/src/test/java/com/palantir/conjure/java/types/NoFieldBeanTests.java#L36-L39 Using this type: https://github.com/palantir/conjure-java/blob/19d7f54eb0001c49b5d8a4bb897b9ad4cb5a28de/conjure-java-core/src/integrationInput/java/com/palantir/product/EmptyObjectExample.java ``` InvalidDefinitionException: Invalid type definition for type `com.palantir.product.EmptyObjectExample`: No argument left as delegating for Creator [method com.palantir.product.EmptyObjectExample#of()]: exactly one required at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 1] at app//com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:62) at app//com.fasterxml.jackson.databind.DeserializationContext.reportBadTypeDefinition(DeserializationContext.java:1866) at app//com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._addExplicitDelegatingCreator(BasicDeserializerFactory.java:489) at app//com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._addExplicitDelegatingCreators(BasicDeserializerFactory.java:365) at app//com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._constructDefaultValueInstantiator(BasicDeserializerFactory.java:270) at app//com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findValueInstantiator(BasicDeserializerFactory.java:219) at app//com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.buildBeanDeserializer(BeanDeserializerFactory.java:262) at app//com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:151) at app//com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:471) at app//com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:415) at app//com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:317) at app//com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:284) at app//com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:174) at app//com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:669) at app//com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:5048) at app//com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4918) at app//com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3860) at app//com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3828) at app//com.palantir.conjure.java.types.NoFieldBeanTests.testDeserializeUsesSingleton(NoFieldBeanTests.java:38) ``` It's not entirely clear that we are correctly using the `DELEGATING` type here, perhaps the `PROPERTIES` type would be a better fit, however neither necessarily document support for a no-arg constructor. In general we prefer `DELEGATING` when possible to avoid ambiguity with potential property sources -- we want the ability to fail deserialization in this case when any properties are included in the incoming object. In 2.17.2, this branch allowed the code to functiona as we desired: https://github.com/FasterXML/jackson-databind/blob/091d968751ef00150d22a788fe7f45b7cdcb337a/src/main/java/com/fasterxml/jackson/databind/deser/BasicDeserializerFactory.java#L658-L662 Is there any chance we could allow the previous DELEGATING behavior to continue to handle the no-arg case? If we'd prefer to move away from `DELEGATING` for that case, is there any chance we could emit a deprecation warning for some time? I've included a potential patch which recovers the previous behavior.

cowtowncoder added to-evaluate Issue that has been received but not yet evaluated 2.18 and removed to-evaluate Issue that has been received but not yet evaluated labels May 5, 2024

cowtowncoder mentioned this issue May 5, 2024

Handling of annotations assigned to constructor parameters of the record class #4513

Open

cowtowncoder mentioned this issue May 14, 2024

Remove almost all deprecated methods from (Basic)BeanDescription #4527

Merged

cowtowncoder mentioned this issue May 14, 2024

Add ConstructorDetector configurability to SerializationConfig too #4530

Closed

cowtowncoder added a commit that referenced this issue May 15, 2024

More refactoring for #4515

3694240

cowtowncoder mentioned this issue May 17, 2024

Refactor Bean property introspection #4532

Merged

cowtowncoder added a commit that referenced this issue May 18, 2024

Merge minor pieces of #4515 work, test improvements etc

6c782df

cowtowncoder mentioned this issue May 18, 2024

@JsonProperty not serializing field names properly on @JsonCreator in Record #4452

Closed

1 task

cowtowncoder added a commit that referenced this issue May 18, 2024

Refactor Bean property introspection in POJOPropertiesCollector (#4532)

e56bc0b

First part of refactoring towards #4515.

cowtowncoder changed the title ~~Rewrite Bean Property Introspection logic in Jackson 2.x (ideally for 2.18)~~ Rewrite Bean Property Introspection logic in Jackson 2.x May 18, 2024

This was referenced May 19, 2024

Fix #4452 by introspecting Creators for Record serialization too #4537

Merged

constructorDetector seems to invalidate defaultSetterInfo for nullability #3241

Closed

Constructor autodetection does not take naming strategy into account #3846

Closed

cowtowncoder added a commit that referenced this issue May 20, 2024

More work towards #4515; intermediate clean up

5c1058c

cowtowncoder added a commit that referenced this issue May 20, 2024

More work towards #4515; intermediate clean up (#4538)

c2be01a

cowtowncoder added a commit that referenced this issue May 21, 2024

More work on #4515: start refactoring BasicDeserializerFactory

812b034

cowtowncoder added a commit that referenced this issue May 22, 2024

More work on #4515: start refactoring BasicDeserializerFactory (#4539)

bc7667d

cowtowncoder mentioned this issue May 23, 2024

@JsonProperty ignored when serialising Kotlin @JvmRecord in 2.16.X FasterXML/jackson-module-kotlin#773

Open

3 tasks

cowtowncoder added a commit that referenced this issue May 30, 2024

One more tweak post #4515 to remove non-visible implicit factory methods

c211002

cowtowncoder added a commit that referenced this issue May 30, 2024

One more tweak post #4515 to remove non-visible implicit factory meth…

f91ec27

…ods (#4549)

cowtowncoder added a commit that referenced this issue May 30, 2024

One more (last?) fix backported from 3.0 for #4515

201f45a

cowtowncoder added a commit that referenced this issue May 30, 2024

One more (last?) fix backported from 3.0 for #4515 (#4550)

8798504

cowtowncoder added a commit that referenced this issue May 30, 2024

Merge #4515 from 2.18 to master (#4547)

246fd12

cowtowncoder added a commit that referenced this issue May 30, 2024

Mark #4119 as fixed (by #4515)

3f55434

This was referenced May 30, 2024

Exception when deserialization uses a record with a constructor property with access=READ_ONLY #4119

Closed

Problem with NamingStrategy, creator methods with implicit names #806

Closed

JooHyukKim mentioned this issue May 30, 2024

Now failing test in 2.18 after Property Introspection Rewrite FasterXML/jackson-module-kotlin#801

Closed

3 tasks

This was referenced May 31, 2024

Unexpected deserialization behavior with @JsonCreator, @JsonProperty and javac -parameters #4545

Closed

Allow @JsonAnySetter on Creator constructors #4366

Closed

cowtowncoder added a commit that referenced this issue May 31, 2024

Yet more post-merge clean up for #4515

1fda8f8

cowtowncoder mentioned this issue Jun 1, 2024

BeanDeserializerModifier::updateBuilder shall work for immutable beans too (#4356) #4357

Closed

cowtowncoder mentioned this issue Jun 16, 2024

Provide extension point for detecting "primary" Constructor for Kotlin (and similar) data classes #4584

Closed

carterkozak mentioned this issue Sep 4, 2024

Should allow deserializing with no-arg @JsonCreator(mode = DELEGATING) (regression) #4688

Closed

1 task

carterkozak mentioned this issue Sep 4, 2024

Fix #4688 2.18.0-rc1 Regression deserializing with no-arg delegating JsonCreator #4689

Merged

yihtserns mentioned this issue Sep 24, 2024

Default ordering for record with @JsonProperties fields mixed with fields without annotations #4712

Closed

1 task

mariofusco mentioned this issue Oct 2, 2024

Bump com.fasterxml.jackson:jackson-bom from 2.17.2 to 2.18.0 quarkusio/quarkus#43637

Closed

JooHyukKim mentioned this issue Oct 20, 2024

Deserialization with Builder, External type id, @JsonCreator not yet implemented #4742

Open

1 task

JooHyukKim mentioned this issue Nov 11, 2024

Polymorphic class with @JsonUnwrapped throwing "Unrecognized field" after upgrade to 2.18 #4792

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rewrite Bean Property Introspection logic in Jackson 2.x #4515

Rewrite Bean Property Introspection logic in Jackson 2.x #4515

cowtowncoder commented May 5, 2024

cowtowncoder commented May 14, 2024

JooHyukKim commented May 14, 2024

cowtowncoder commented May 14, 2024 •

edited

Loading

cowtowncoder commented May 14, 2024

JooHyukKim commented May 15, 2024

cowtowncoder commented May 15, 2024

cowtowncoder commented May 15, 2024 •

edited

Loading

cowtowncoder commented May 18, 2024

cowtowncoder commented May 21, 2024

cowtowncoder commented May 22, 2024

cowtowncoder commented May 30, 2024 •

edited

Loading

JooHyukKim commented May 30, 2024

StefanBratanov commented Jun 13, 2024

JooHyukKim commented Jun 13, 2024

yihtserns commented Jun 13, 2024 •

edited

Loading

Rewrite Bean Property Introspection logic in Jackson 2.x #4515

Rewrite Bean Property Introspection logic in Jackson 2.x #4515

Comments

cowtowncoder commented May 5, 2024

Describe your Issue

cowtowncoder commented May 14, 2024

JooHyukKim commented May 14, 2024

cowtowncoder commented May 14, 2024 • edited Loading

cowtowncoder commented May 14, 2024

JooHyukKim commented May 15, 2024

cowtowncoder commented May 15, 2024

cowtowncoder commented May 15, 2024 • edited Loading

cowtowncoder commented May 18, 2024

cowtowncoder commented May 21, 2024

cowtowncoder commented May 22, 2024

cowtowncoder commented May 30, 2024 • edited Loading

JooHyukKim commented May 30, 2024

StefanBratanov commented Jun 13, 2024

JooHyukKim commented Jun 13, 2024

yihtserns commented Jun 13, 2024 • edited Loading

cowtowncoder commented May 14, 2024 •

edited

Loading

cowtowncoder commented May 15, 2024 •

edited

Loading

cowtowncoder commented May 30, 2024 •

edited

Loading

yihtserns commented Jun 13, 2024 •

edited

Loading