I'm wondering if it makes sense to have multiple m...
# troubleshootingj
jon_court
05/18/2021, 10:21 PMI'm wondering if it makes sense to have multiple meltano projects (project === git repo) - one per-pipeline (or one per group of related pipelines) to help with the ungainly meltano.yml problem... deployment would be containerising each project individually and deploying to kube in our case... comments?
jon_court
05/18/2021, 10:23 PMi believe regardless of the ungainly meltano.yml problem that this would be tidier and more 'domain oriented' - i.e. it nicely packages up related pipelines isolated from un-related.
jon_court
05/18/2021, 10:30 PMthe UI bit may be a problem - if we use it.
a
aaronsteers
05/19/2021, 12:15 AMCertainly you can have separate projects, either in the same or separate repos. For anything that is not tightly connected or synchronized, I think it makes sense to keep things in their own repos. If the problem is just having one large meltano.yml, you can easily put multiple ‘projects’ into the same repo, just within distinct subfolders.
aaronsteers
05/19/2021, 12:17 AMAnd fyi, we also have this issue in our backlog, which (once delivered) would guard against yaml file sprawl. https://gitlab.com/meltano/meltano/-/issues/2456. Merge requests welcome. 😄
j
jon_court
05/19/2021, 2:54 AMthanks @aaronsteers.
c
chris_kings-lynne
05/19/2021, 3:40 AMI just put everything in one repo, but have a docker image built from it that can be commanded to run any elt, hydrating the necessary env vars using chamber/parameter store
chris_kings-lynne
05/19/2021, 6:13 AMI’ll publish the solution properly at some point but here’s a gist of some of the work https://gist.github.com/chriskl/495ff766f9dfa8e8f6fcd00fae6e58f3
v
visch
05/19/2021, 11:55 AMnice!
2 Views