Hello all, is there a way to "reset" the `select` ...
# getting-started
t
Hello all, is there a way to "reset" the
select
-ed entities everytime
meltano select
is run? I have a web application where I am using a
tap-google-sheets
extractor and would like to provide clients the option to
select
different entities corresponding to their sheet name of choice in the spreadsheet they have provided. For instance: • Client A may have chosen "Sheet1" from Spreadsheet A on my front end to import into my application. Then I would run
meltano select tap-google-sheets Sheet1
and then
meltano elt ...
to extract the relevant data into my database of choice. • However, a different Client B may choose "Sheet2" from Spreadsheet B on my front end to import instead. This would run
meltano select tap-google-sheets Sheet2
but with the previous "Sheet1.*" entity still in my
meltano.yml
. Hence, the new entity "Sheet2.*" would be merely appended. If Client B also has another sheet named "Sheet1" in their own spreadsheet, this may cause undesired data imports for them. Overall, I am currently running a single
meltano.yml
configuration in a single project directory, and injecting each client's refresh token and chosen spreadsheet as environment variables at runtime when invoking
meltano elt
. I realize I could use
meltano select tap-google-sheets --rm
but that would be rather manual, and it would be difficult to keep track of the patterns all my clients have added so far. Is creating a new configuration for each client the best way to go about this? Using plugin inheritance, probably. Perhaps I am overthinking this, but any alternatives would be greatly appreciated!
p
Hey @tedmund_ho! You can inject the selection criteria using environment variables as well https://meltano.com/docs/plugins.html#select-extra. That would allow you to store specific selection criteria in your application for specific clients and still share the same
tap-google-sheets
meltano config. Is that what youre looking for?
t
Hello @pat_nadolny, thank you! Can't believe I missed that. It works perfectly for what I have now.