https://linen.dev logo
Join Slack
Powered by
Hi folks! I have been manually running a google-an...
# troubleshooting
c
Hi folks! I have been manually running a google-analytics pipelines with a Redshift loader in single day increments with a start_date of yesterday and an end_date of today. Google shows that I can set the start_date to 
yesterday
and end_date to 
today
in order to create a daily scheduled run. However when I try to set a schedule I receive this error. Is there any methods to set a start and end date for a pipeline with a fluctuating 
yesterday
start, and 
today
end? Thank you in advance!
@aaronsteers Related to this issue.... Is there currently a method that will define an incremental start_date (of yesterday) for an extractors?
e
@connor_flynn I think the error's coming from Meltano's parsing of 
date_iso8601
settings here: https://gitlab.com/meltano/meltano/-/blob/15dc7e987b63edab73a3ff077a6430baee122888/src/meltano/core/utils/__init__.py#L265-282 One workaround might be to override the setting 
kind
to make it a simple string
a
This has come up before so I've added a generic answer here: https://stackoverflow.com/a/71709771/4298208
@connor_flynn - This is a slightly different direction but wanted to provide this as another option, which should work for any tap. In the terminal:
Copy code
N_DAYS=1
TAP_MYSOURCE_START_DATE=$(date +%Y-%m-%d -d "$N_DAYS day ago")
echo "Using dynamic start date of today-$N_DAYS: $TAP_MYSOURCE_START_DATE"
meltano elt tap-mysource target-mydest
(There's a mac OS version also at the StackOverflow link.)
c
Thank you for the suggestions @aaronsteers & @edgar_ramirez_mondragon! I will test these out!
Open in Slack
PreviousNext
Powered by