Add configuration item to control how connector deals with Null items

[Strabox]

Description

Nullpointer exception in the connector when writing data into Cosmos from an Avro (with confluent Schema Registry) topic.

Expected Behavior

A clear error of what is happening when writing the data.

Additional Context

I was debugging a little bit more, and problem resumes to the fact that we have an Avro Schema with a field which is a struct/record in terms of type, if this struct field is populated with null instead of the struct/record than the NullPointerException is raised.

This is supposed to happen? To me having nulls in struct/record fields is a common practice and I faced it a lot when processing CDC events from CDC tools like IBM WebShphere replicator and Oracle's Golden Gate.

Thanks!

[ryancrawcour]

what is the outcome you expect when using nulls in a struct?
are you expecting the field with a null value to show up in cosmos db with a null? or would you expect to not see the field at all in cosmos db?

[Strabox]

Hi @ryancrawcour ,

I was expecting a null to be present in the document in the Cosmos DB which is the natural 1-1 mapping between the Avro Record and the Json Document.

But a boolean flag in the connector that allow to change the way null are processing e.g. hideNullFields could be a nice way to cover more people's scenarios.

Thanks

[ryancrawcour]

Thanks @Strabox. I think adding a flag in config would be a nice future feature. For now, I say we should go with your solution and leave this open as a feature request for future.

[ryancrawcour]

changed from bug to feature request and edited title to track feature request to put null behaviour under a config flag.