Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How can I get the connector to submit the follow curl? #170

Open
alberttwong opened this issue Jun 16, 2023 · 0 comments
Open

How can I get the connector to submit the follow curl? #170

alberttwong opened this issue Jun 16, 2023 · 0 comments
Labels
question Further information is requested

Comments

@alberttwong
Copy link
Contributor

There are not enough examples of how to get the http connector to work. Best example I could find is https://redpanda.com/blog/unify-log-data-parseable-redpanda

Here's what I need to submit

curl -v --location-trusted -u <username>:<password> -H "strict_mode: true" \
    -H "format: json" -H "jsonpaths: [\"$.name\", \"$.code\"]" \
    -H "columns: city,tmp_id, id = tmp_id * 100" \
    -T example2.json -XPUT \
    http://someserver:<443>/api/test_db/table2/_stream_load

I need multiple headers to be submitted but the following won't work since http.headers.additional is a String.

{
  "name": "http-sink-test",
  "config": {
    "connector.class": "io.aiven.kafka.connect.http.HttpSinkConnector",
    "topics.regex": "m1",
    "http.authorization.type": "oauth2",
    "oauth2.client.id": "username",
    "oauth2.client.secret": "password",
    "http.headers.additional": [
      {
        "format": "json"
      },
      {
        "jsonpaths": "[\"$.userid\",\"$.time\",\"$._time\",\"$.remote_user\",\"$.ip\",\"$.request\",\"$.status\",\"$.bytes\",\"$.referrer\",\"$.agent\"]"
      }
    ],
    "http.url": "http://https://6dihxzedg.cloud-app.celerdata.com/api/demo/visits/_stream_load",
    "batching.enabled": true,
    "batch.max.size": 2,
    "tasks.max": "1",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter"
  }
}

Any advice?

@jeqo jeqo added the question Further information is requested label Jul 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants