Common: write out Avro format rather than TSV

[chuwy]

No description provided.

[chuwy]

Migrated from snowplow/snowplow#2035 (comments are auto-generated)

[julianbei]

Hi @chuwy thanks for all the wonderful work here.

I like this proposal.

You might also consider to make use of the CNCF CloudEvents spec https://cloudevents.io/.
Because you are using json schemas all over the place already, defining the output format as config would be cool.

(can be generated from a json schema in many cases):

• https://toolslick.com/generation/metadata/avro-schema-from-json
• https://transform.tools/json-schema-to-protobuf

This would make the pipeline more composable and fit into the overarching architecture of the users. For us we have potentially Go, Python and Javascript/Typescript services consuming enich.good. TSV is not helpful with that and thrift is the same issue.

Preferable formats:

• avro
• protobuf
• json (easy to debug)
• tsv

[chuwy]

Hi @julianbei,

Honestly it's the first time I hear of CloudEvents, but at first glance the spec looks very exciting. Do you want to create a separate ticket maybe to not lost it, I'm keen to learn more about it at some point.

[julianbei]

Sure. I will. -> #304
I have another exciting experiment I want to show you. If you have time. No Idea how to reach out since the chats are migrated to a forum. But the forum isnt the right place. Can you ping me somehow?

https://twitter.com/jbinsen <- ping me