Is there any reason why my child streams might fail to run e Meltano #singer-tap-development

Is there any reason why my child streams might fai...

Andy Carter

04/14/2023, 1:01 PM

Is there any reason why my child streams might fail to run, either with

meltano run

or when testing in VSCode? 👇 EDIT: Found it, difference between

$.items[*]

and

$.items.[*]

in json path. Have to look at the json path evaluator with "Output Paths" turned on

Andy Carter

04/14/2023, 1:02 PM

Copy code

class CampaignsStream(MailchimpStream):
    """Define custom stream."""

    name = "campaigns"
    path = "/campaigns"
    response_key = "campaigns"
    primary_keys = ["id"]
    replication_method = 'FULL_TABLE'

    def get_child_context(self, record: dict, context: dict | None) -> dict | None:
        return {
            'campaign_id': record['campaign_id']
        }

class ReportsEmailActivity(MailchimpStream):

    name = 'reports_email_activity'
    path = '/reports/{campaign_id}/email_activity'
    parent_stream_type = CampaignsStream
    response_key = 'emails'
    primary_keys = [
        'campaign_id',
        'action',
        'email_id',
        'timestamp',
    ]
    ignore_parent_replication_key = True

I can only see Campaigns running and paginating correctly, but ReportsEmailActivity isn't running, it's like the stream simply doesn't exist.

Andy Carter

04/14/2023, 1:02 PM

And in the parent Tap:

Copy code

def discover_streams(self) -> list[streams.MailchimpStream]:
        """Return a list of discovered streams.

        Returns:
            A list of discovered streams.
        """
        return [
            streams.CampaignsStream(self),
            streams.ReportsEmailActivity(self),
            streams.ListsStream(self),
            streams.ListsMembersStream(self),
        ]

Andy Carter

04/14/2023, 1:12 PM

Both the 'top' level streams run fine, but neither of the childs run at all.

Andy Carter

04/14/2023, 1:16 PM

Mystery solved 🙂

Andy Carter

04/14/2023, 1:18 PM

Should have got alarm bells ringing when no output to jsonl.

edgar_ramirez_mondragon

04/14/2023, 1:55 PM

@Andy Carter perhaps the SDK should try to log a warning if the jsonpath expression didn't match anything. Would you mind creating an issue?

visch

04/14/2023, 3:33 PM

We just stumbled on a similarish bug with json path. I think we did

$.item.*

instead of

$.item[*]

and it return 0. A feature like this would help

Andy Carter

04/14/2023, 3:37 PM

Would this cause issues when the call is correct but retrieves no records? Some of the APIs I am working with have basic pagination, so the only way to test if the pagination

has_more

is to keep incrementing page number until get an empty page, and then return

bool(len(response.json()))

. You don't know you're at the end of the records until the first page that returns an empty list.

edgar_ramirez_mondragon

04/14/2023, 5:32 PM

That’s a good call out. I think we need only log an event at info/warning level when the jsonpath match is empty, if possible.

Open in Slack

Previous Next