tedmund_ho
10/21/2021, 12:25 PMselect -ed entities everytime meltano select is run? I have a web application where I am using a tap-google-sheets extractor and would like to provide clients the option to select different entities corresponding to their sheet name of choice in the spreadsheet they have provided.
For instance:
• Client A may have chosen "Sheet1" from Spreadsheet A on my front end to import into my application. Then I would run meltano select tap-google-sheets Sheet1 and then meltano elt ... to extract the relevant data into my database of choice.
• However, a different Client B may choose "Sheet2" from Spreadsheet B on my front end to import instead. This would run meltano select tap-google-sheets Sheet2 but with the previous "Sheet1.*" entity still in my meltano.yml. Hence, the new entity "Sheet2.*" would be merely appended. If Client B also has another sheet named "Sheet1" in their own spreadsheet, this may cause undesired data imports for them.
Overall, I am currently running a single meltano.yml configuration in a single project directory, and injecting each client's refresh token and chosen spreadsheet as environment variables at runtime when invoking meltano elt.
I realize I could use meltano select tap-google-sheets --rm but that would be rather manual, and it would be difficult to keep track of the patterns all my clients have added so far. Is creating a new configuration for each client the best way to go about this? Using plugin inheritance, probably. Perhaps I am overthinking this, but any alternatives would be greatly appreciated!pat_nadolny
10/21/2021, 1:02 PMtap-google-sheets meltano config. Is that what youre looking for?tedmund_ho
10/22/2021, 12:46 AM