Hi, two quick questions: 1) can meltano schedule be less than 1 minute for every sync? 2) is it possible to make a pipeline with few extractors into one loader?

Hi @omri_cohen 1. Meltano itself only has basic schedule management functionality. Essentially you can define the schedule, but it's another components job to run the schedule (e.g. Airflow). When you create these schedules meltano accepts a cron syntax with minutes, hours, days, months, years (https://meltano.com/docs/command-line-interface.html#examples-5) There is nothing stopping you running meltano many times, in parallel. You do have to consider your approach to any state you need to keep track of (don't use a --job-id and no central db for example https://meltano.com/docs/integration.html#incremental-replication-state). I'd love to know what your thinking about doing, but frankly it sounds like a stretch to use meltano if this is a streaming use case. 2. This is not the idea. In meltano a pipeline (an elt / run in meltano lingo) is one source to one target, but you can have many pipelines all running at once to a single loader, to different schemas or the same schema if the stream is incremental.

@omri_cohen we’re actually considering the latter use case! @aaronsteers has been leading the charge on that one as it’s not uncommon to have a single target (i.e. warehouse such as Snowflake, BigQuery, etc.). @aaronsteers do we have an issue top of mind to best track that? I know it was part of the proposed environments spec originally but I’m not sure if we spun out a separate issue