https://linen.dev logo
#plugins-general
Title
# plugins-general
w

wyatt

06/10/2021, 7:24 PM
I am testing out the tap-spreadsheets-anywhere tap on a S3 bucket. It looks like I can pass a catalog json file for specification (which includes a s3 bucket and paths for the files I want to ingest). My bucket contains data under a directory structure {year}/{month}/{day}/{hour}. Ideally, I want to pass a dynamic variable to the search path on an hourly execution of the pipeline so that only the files listed under the matching criteria are selected. Is it possible to override search_path as a command line arg when calling
meltano elt
? Then, I could avoid re-writing a catalog json file every hour and instead pass in the relevant execution time as a variable!