Can Meltano only work on a single node, can it wor...
# troubleshootingh
hello_dev
07/06/2022, 5:41 AMCan Meltano only work on a single node, can it work with multiple nodes?
hello_dev
07/06/2022, 10:38 AM@Reuben (Matatika) can help me? 🥺
r
Reuben (Matatika)
07/06/2022, 11:13 AMNot sure what you mean, sorry. Can you explain more?
h
hello_dev
07/06/2022, 1:42 PMHi , @Reuben (Matatika) When some tables have a large amount of data, I want multiple nodes to work faster, just like airbyte workers
https://docs.airbyte.com/understanding-airbyte/jobs#message-passing
hello_dev
07/06/2022, 1:43 PMEnglish is not my native language, sorry for the poor expression
r
Reuben (Matatika)
07/06/2022, 1:51 PMNo worries! 😅 I don't think Meltano supports that kind of processing yet, although maybe someone from the team has more to say about this than I do.
Reuben (Matatika)
07/06/2022, 1:53 PMThere is an #C01QS0RV78D session today at 16:00 UTC if you want a more direct answer, but looks like that might be a bit late for you.
h
hello_dev
07/06/2022, 1:54 PMthanks for your reply,I currently cannot understand spoken EnglishðŸ˜
r
Reuben (Matatika)
07/06/2022, 1:58 PMAh I see, well just keep an eye on this thread then. I'm sure someone can answer your question better than me! 😄
p
pat_nadolny
07/06/2022, 2:22 PM@hello_dev check out https://meltano.com/blog/deploying-meltano-for-meltano/ the way we've approached multiple nodes in our Meltano deployment is to run a kubernetes pod for each pipeline sync. You can also do this with Airflow workers where you scale up the amount of workers to consume your queue of jobs. Is that what you were looking for?