How to stop meltano run after hitting today's date ? When I extract data from shopify, meltano run turns in cycles, after hitting today's date it returns back to 'EXTRACT_START_DATE' and continues to run in cycles.

@dima_petukhov What do you mean by "running in cycles"? 1. Meltano itself only does batch processes, so there's no way to run taps forever (say for streaming data), so by default nothing should run "in cycles". 2. Taps like the tap-shopify should put out their API calls, possibly multiple ones based on your configuration e.g. extract start date, wait for the data and that's it. There shouldn't be any looping behavior. So I'm a bit lost how "running in cycles" looks like, could you explain that? What happens on the meltano run side, what happens on the target side?

So we have start_date = '2018-01-01' After hitting '2023-03-29' (today) extractor(shopify singer ) went back to 2018-01-01, what's why I'm asking is this a normal behavior? I'm using target-postgres(datamill)

meltano run tap-shopify target-postgres

@dima_petukhov I'm still not sure what you mean with "it went back to XYZ". Do you mean the ingestion simply continues? Or does it start at the same date on the next extraction?

after hitting today date, data ingestion continues, the next date after today will be the start date, in my case 2018-01-01

That sounds weird. If the meltano tap-shopify target-postgres command never stops, then there's a bug in the tap (https://github.com/singer-io/tap-shopify), I couldn't tell that from a quick look, but then you should file a bug there or switch over to one of the other variants. Have you tried another variant?

I've tried mataika, it's still in development, has bugs. I've switched to singer variant. The Airbyte variant I didn't tried because it runs inside it's own docker image for now I prefer not to complexify my setup with docker-in-docker .

Well it sounded like matatika was working on fixing the bugs they had. Maybe you try that one again? Btw. with tap-shopify target-jsonl the same thing happens right? (Just want to make sure the problem is indeed the tap, not something else).

If I remember well, it had same problem, because I've had multiple duplicates in jsonl output

@Reuben (Matatika) I'm curious, how is the status of your shopify tap? I find it hard to believe that we don't have a working tap for shopify anywhere. Do you have any suggestions? Or do you know someone who is successfully using tap-shopify?

We test our

tap-shopify

with

target-postgres

automatically every day, although (from a previous post):

since we don't use the platform in any real capacity there is only so much data we can mock - as in this case, clearly some scenarios are not full tested

Without access to "real" data, it's hard to know how robust the tap truly is, especially when there are many streams that may or may not be applicable for a Store, or are contingent on the user having a Shopify Plus account.

@dima_petukhov since it's really hard to debug your case from afar, I would suggest you try out the tap-shopify from matatika again. And if you stumble over an actual bug, file a bug report. I'm sure @Reuben (Matatika) et. al. are quick to fix any bugs, once you supply them with a specific issue.

So the latest issue was related to a

400 Bad Request

from the Shopify API (amongst

200 OK

responses for almost identical requests in the same stream - InventoryLevelsStream), but we reduced the priority on diagnosing this issue as @dima_petukhov was going to switch to another solution. We would be happy to pick this up again.

@Sven Balnojan I've solved it, thanks to my boss that used ChatGPT 😄 in the doc it was not presiced that I could use

bookmark_properties

with this tap

2023-03-30T14:32:44.996108Z [info     ] Incremental state has been updated at 2023-03-30 14:32:44.995956.
2023-03-30T14:32:45.141931Z [error    ] Extractor failed
2023-03-30T14:32:45.142590Z [error    ] Block run completed.           block_type=ExtractLoadBlocks err=RunnerError('Extractor failed') exit_codes={<PluginType.EXTRACTORS: 'extractors'>: 1} set_number=0 success=False
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Run invocation could not be completed as block failed: Extractor failed

sharing the config, no more cycles, finally !

extractors:
    - name: tap-shopify
      variant: singer-io
      pip_url: tap-shopify
      config:
        bookmark_properties:
          orders: updated_at
          customers: updated_at

@dima_petukhov sadly not at all, I did read something about it but then disregarded it. If you are able to provide a few more details, then maybe @pat_nadolny can include a note into https://hub.meltano.com/extractors/tap-shopify such that other people are able to use the tap in the same way you do.

@Sven Balnojan Which kind of details could I provide for you ? I could just resume that reference for all singer spec based programs, readme must include link to https://hub.meltano.com/singer/spec/ additionnaly, by my humble opinion, the use case in this thread could be added in the documentation

@dima_petukhov but if I understand you correctly, this tap is basically not usable in specific (your) situations if you don't know about that option, right? That's not the case for any other tap I know. So in that case the details would be something like: "If you have configuration XYZ for this tap, then you also need to set up ...."

@Sven Balnojan "I you have configured mandatory fields TAP_SHOPIFY_SHOP, TAP_SHOPIFY_START_DATE and TAP_SHOPIFY_API_KEY then you also need to set up the "stop condition"(optional) for data extraction by setting

bookmark_properties

for each table you sync. In this case the sync will fetch everything and will stop at the today date

config:
        bookmark_properties:
          orders: updated_at
          customers: updated_at

Something like that 🧐

Ok, but the other properties are mandatory, aren't they? Sounds like you always need to set a stop condition if you set a start_date?

Without

bookmark_properties

the tap will work without stopping, meaning that when it hits today's date, it will return to the starting point of TAP_SHOPIFY_START_DATE and continue to collect data in cycles

Just wanted to emphasize again, that's not how taps should work 😉 Like never. So good to know. seems like something is going wrong here but if we can fix it with a simple note on top, then that's very helpful!