Skip to content

feat(source-twilio): Add configurable slice step duration (default: 1 month) #72494

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tolik0 wants to merge 4 commits into
base: master
Choose a base branch
from
+31 −14
Open

feat(source-twilio): Add configurable slice step duration (default: 1 month) #72494

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
323d322
Change the default slice duration from one year to one month
tolik0 Jan 30, 2026
1b9bb8e
fix(source-twilio): Correct PR number in changelog entry
devin-ai-integration[bot] Jan 30, 2026
245d04c
fix(source-twilio): Update test to reflect 1-month slice duration
devin-ai-integration[bot] Jan 30, 2026
a6bbe83
feat(source-twilio): Add configurable slice step duration
devin-ai-integration[bot] Feb 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 18 additions & 2 deletions airbyte-integrations/connectors/source-twilio/manifest.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ definitions:
- "%Y-%m-%d"
- "%Y-%m-%dT%H:%M:%S.%f%z"
cursor_granularity: P1D
step: P1Y
step: "{{ config.get('slice_step_duration', 'P1M') }}"
lookback_window: "PT{{ config.get('lookback_window', 0) }}M"
start_datetime:
type: MinMaxDatetime
Expand Down Expand Up @@ -221,7 +221,7 @@ definitions:
cursor_field: date_generated
datetime_format: "%Y-%m-%dT%H:%M:%SZ"
cursor_granularity: PT1S
step: P1Y
step: "{{ config.get('slice_step_duration', 'P1M') }}"
lookback_window: "PT{{ config.get('lookback_window', 0) }}M"
start_datetime:
type: MinMaxDatetime
Expand Down Expand Up @@ -880,6 +880,22 @@ spec:
examples: [1, 2, 3]
order: 5
description: The number of worker threads to use for the sync.
slice_step_duration:
type: string
title: Slice Step Duration
description: The time window size for each data slice when syncing incremental streams. Smaller windows may help avoid timeouts for accounts with large data volumes.
default: P1M
enum:
- P1D
- P1W
- P1M
- P1Y
enum_labels:
P1D: 1 Day
P1W: 1 Week
P1M: 1 Month
P1Y: 1 Year
order: 6

schemas:
accounts:
Expand Down
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-twilio/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ data:
connectorSubtype: api
connectorType: source
definitionId: b9dc6155-672e-42ea-b10d-9f1f1fb95ab1
dockerImageTag: 0.17.4
dockerImageTag: 0.17.5
dockerRepository: airbyte/source-twilio
documentationUrl: https://docs.airbyte.com/integrations/sources/twilio
githubIssueLabel: source-twilio
Expand Down
4 changes: 2 additions & 2 deletions airbyte-integrations/connectors/source-twilio/unit_tests/test_streams.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,12 +163,12 @@ def test_calls_includes_date_window_params(self, requests_mock):
"states": [
{
"partition": {"subresource_uri": "/2010-04-01/Accounts/AC123/Recordings.json"},
"cursor": {"date_created": "2021-11-13 00:00:00Z"},
"cursor": {"date_created": "2022-10-13 00:00:00Z"},
}
]
},
[
("2021-11-13 00:00:00Z", "2022-11-12 23:59:59Z"),
("2022-10-13 00:00:00Z", "2022-11-12 23:59:59Z"),
("2022-11-13 00:00:00Z", "2022-11-16 12:03:11Z"),
],
),
Expand Down
19 changes: 10 additions & 9 deletions docs/integrations/sources/twilio.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,12 +99,13 @@ For more information, see [the Twilio docs for rate limitations](https://support
<summary>Expand to review</summary>

| Version | Date | Pull Request | Subject |
|:------------|:-----------| :------------------------------------------------------- |:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 0.17.4 | 2026-01-22 | [72260](https://github.com/airbytehq/airbyte/pull/72260) | Update CDK version from 7.0.1 to 7.6.5 |
| 0.17.3 | 2025-11-06 | [68680](https://github.com/airbytehq/airbyte/pull/68680) | Handle 404 errors gracefully for date ranges with no data |
| 0.17.2 | 2025-10-22 | [68591](https://github.com/airbytehq/airbyte/pull/68591) | Add `suggestedStreams` |
| 0.17.1 | 2025-09-15 | [66090](https://github.com/airbytehq/airbyte/pull/66090) | Update to CDK v7 |
| 0.17.0 | 2025-09-05 | [65955](https://github.com/airbytehq/airbyte/pull/65955) | Promoting release candidate 0.17.0-rc.2 to a main version. |
|:------------|:-----------|:---------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 0.17.5 | 2026-01-30 | [72494](https://github.com/airbytehq/airbyte/pull/72494) | Add configurable slice step duration (default: 1 month) |
| 0.17.4 | 2026-01-22 | [72260](https://github.com/airbytehq/airbyte/pull/72260) | Update CDK version from 7.0.1 to 7.6.5 |
| 0.17.3 | 2025-11-06 | [68680](https://github.com/airbytehq/airbyte/pull/68680) | Handle 404 errors gracefully for date ranges with no data |
| 0.17.2 | 2025-10-22 | [68591](https://github.com/airbytehq/airbyte/pull/68591) | Add `suggestedStreams` |
| 0.17.1 | 2025-09-15 | [66090](https://github.com/airbytehq/airbyte/pull/66090) | Update to CDK v7 |
| 0.17.0 | 2025-09-05 | [65955](https://github.com/airbytehq/airbyte/pull/65955) | Promoting release candidate 0.17.0-rc.2 to a main version. |
| 0.17.0-rc.2 | 2025-09-04 | [65936](https://github.com/airbytehq/airbyte/pull/65936) | Fix lookback window |
| 0.17.0-rc.1 | 2025-08-20 | [65072](https://github.com/airbytehq/airbyte/pull/65072) | Migrate to manifest-only |
| 0.16.0 | 2025-08-28 | [65593](https://github.com/airbytehq/airbyte/pull/65593) | Promoting release candidate 0.16.0-rc.1 to a main version. |
Expand Down Expand Up @@ -160,8 +161,8 @@ For more information, see [the Twilio docs for rate limitations](https://support
| 0.1.5 | 2022-06-22 | [13896](https://github.com/airbytehq/airbyte/pull/13896) | Add lookback window parameters to fetch messages with a rolling window and catch status updates |
| 0.1.4 | 2022-04-22 | [12157](https://github.com/airbytehq/airbyte/pull/12157) | Use Retry-After header for backoff |
| 0.1.3 | 2022-04-20 | [12183](https://github.com/airbytehq/airbyte/pull/12183) | Add new subresource on the call stream + declare a valid primary key for conference_participants stream |
| 0.1.2 | 2021-12-23 | [9092](https://github.com/airbytehq/airbyte/pull/9092) | Correct specification doc URL |
| 0.1.1 | 2021-10-18 | [7034](https://github.com/airbytehq/airbyte/pull/7034) | Update schemas and transform data types according to the API schema |
| 0.1.0 | 2021-07-02 | [4070](https://github.com/airbytehq/airbyte/pull/4070) | Native Twilio connector implemented |
| 0.1.2 | 2021-12-23 | [9092](https://github.com/airbytehq/airbyte/pull/9092) | Correct specification doc URL |
| 0.1.1 | 2021-10-18 | [7034](https://github.com/airbytehq/airbyte/pull/7034) | Update schemas and transform data types according to the API schema |
| 0.1.0 | 2021-07-02 | [4070](https://github.com/airbytehq/airbyte/pull/4070) | Native Twilio connector implemented |

</details>
Loading