[connector-builder] Substream Only Fetches Records from the First Page of Parent Stream

[avirajsingh7]

Topic

Connector-Builder-Ui

Relevant information

We're encountering an issue with source connectors configured through the UI where substreams are only making API calls to records fetched from the first page of their parent streams. This means that even if the parent stream has a significant number of records (e.g., 23k in this case), the substream will only process parent_id from first_page record of parent_stream (e.g., 546 records).

I have verified the records on DB and last record of substream is from last record of parent stream fetched from first api call.

Here's manifest.yaml file, I have configured this to get record for each api_call in substream(For debugging purpose).

[marcosmarxm]

@airbytehq/dev-marketplace-contributions can someone take a look in this issue?

[Stockotaco]

I have the same issue. My substream runs for every item in the parent stream but only the first page of the parent stream.

[natikgadzhi]

Mentioned on Community Slack: https://airbytehq-team.slack.com/archives/C021JANJ6TY/p1720192514494299

We're looking, but no ETA yet.

[natikgadzhi]

@Stockotaco @avirajsingh7 I have a hunch — does it matter if the substream OR the parent stream are full refresh or incremental? If incremental is possible and you switch to incremental, does the problem go away?

[Stockotaco]

In my case both parent and sub-streams are full refresh syncs, not incremental.

Incremental won't work in my case.

[avirajsingh7]

@natikgadzhi both are full refresh, #40573 is also a same issue, looks like bug

[natikgadzhi]

Oh, I think we have a fix for this here: #40671

[natikgadzhi]

@ChristoGrab, can you please test Builder locally with CDK with this patch applied, and confirm this works for Builder as well? If yes, also review and approve @brianjlai's pull request, and let's make sure to ship it on Monday.