Hey everyone! Didn't see this channel before - i think my question suits here better than to #C01TCRBBJD7 I’m developing a data warehouse for jira ->vertica db. This is my first ELT project ever. As i’m here on this slack one could assume I’m using meltano for that :). With tap-jira i can only set the start_date in the config file. I understand the regular daily loading...but how am i supposed to do the first extract say from 2018 until today? There's a lot of data and the probability that smth doesn't crash during that extract is pretty much 0. i've seen that some taps have also configurable end_date parameter which allows one to extract in controllable chunks. How would one add this configurable parameter to tap-jira, or is there some workaround for that problem? All the help is very appreciated guys!

tap-jira (and quite a lot of other taps) implement full backfilling by default, so will just retrieve data iteratively up until the timestamp in the state reaches today/ yesterday/ whatever is available

Also check out https://docs.meltano.com/guide/integration#incremental-replication-state

In the case of target-vertica (not on the hub?) something like

batch_size_rows

being set to less than the default of 100000 would cause more frequent loads

Thank you guys! I've tried that vertica target and it doesn't work. Doesn't install - I don't think there's any quick fix for that, or am i mistaken?

can't help there I'm afraid, haven't used it personally