Hi, does meltano have a system to deal with data "on-demand"? Some of our integrations (e.g google analytics, facebook ads) require that our users can send custom queries to the source systems and retrieve data in real-time. That is, it is not possible to fetch the data to our data warehouse beforehand. Is this use case supported?

Technically you can call meltano whenever you want so "yes" but you have to get meltano to run when your users want it to. Which means the work is on you here. I'm looking into this for a few reasons myself, not for dashboarding though. My thoughts are use DBT. My target model creates a dag so theoretically I should be able to do something like

dbt run -m +targetmodel

, in DBT the source models that this model depends on I'm sure I can query some how and refer to them. If they are a meltano source, then trigger the source to run. After that run the transform (or trigger the transform via Meltano). Profit!

Hi @jens_dahl_mollerhoj! I'm not completely sure about facebook ads, but you can definitely set the reports definitions for google analytics dynamically with Meltano using environment variables

TAP_GOOGLE_ANALYTICS_REPORTS='[{"name": "my-report", "dimensions": [...]}]' meltano elt tap-google-analytics target-whatever

Docs in the Hub: https://hub.meltano.com/extractors/google-analytics#reports. Does that work for your use case?

@edgar_ramirez_mondragon & @visch thanks for your replies! I'm not exactly sure this works for my use case. Since the end user is literally requesting the data and then sitting there, waiting for a response, it is critical that the data is fetched and shown as quickly as possible. That is, we should only fetch the data for this specific user and parameters, we can't wait for an entire etl pipeline to run :-)