Body

The current capabilities of StreamPipes' Connect module are limited in two key areas:

1. Complex JSON Structures: Applying necessary data transformations to nested or complex JSON structures is currently tedious and overly complicated for end-users.
2. Rigid Adapter Schema: Connecting new data sources with unique schemas often requires implementing a new adapter

Proposed Solution

Implement a Function Transformation Step into Connect. This feature is intended to fully resolve both issues by providing a flexible mechanism for defining complex parsing and schema mapping logic.

Goal & Scope

To validate the architectural approach required for supporting complex JSON structures, a prototype of the Function Transformation Step should be implemented before migrating existing adapters to the new data model.

StreamPipes Committer

I acknowledge that I am a maintainer/committer of the Apache StreamPipes project.