[connector-builder] Child Stream only returns records from the first page of the parent stream when using pagination

[argylepayments]

Topic

Parent Stream Pagination

Relevant information

I have setup a no code connector that has a stream that is using a parent stream. The child stream only returns the first page of records. It does not go through all the pages of the parent stream.

[zoloogg]

Same. Example configuration:

Parent pagination

type: DefaultPaginator
page_token_option:
  type: RequestOption
  inject_into: request_parameter
  field_name: start
pagination_strategy:
  type: CursorPagination
  cursor_value: '{{ response.get("next", {}) }}'
  stop_condition: '{{ not response.get("next", {}) }}'

Child config

- type: SubstreamPartitionRouter
  parent_stream_configs:
    - type: ParentStreamConfig
      parent_key: ID
      request_option:
        type: RequestOption
        field_name: id
        inject_into: request_parameter
      partition_field: parent_id
      stream:
        $ref: '#/definitions/streams/deals'

[natikgadzhi]

@zoloogg we're tracking this in #40734 — closing this, but we're going to wrap it up and ship later this week!