Hello Everyone 👋 I am trying to use meltano for data ingestion into snowflake data lake from various rdbms source (orcl, msssql, Postgres ) , I am glad that I found all these db extractors available already I was able to install metlano in a docker container and do basic full table sync into snowflake , planning to incorporate meltano into our dbt+airflow project if it works for all various batch data ingestion patterns we currently have Here are some patterns, i might need some help on 1. Can we use custom sql query as source instead of a table, may be I can create a view in source db and use that as source but wanted to confirm if there is a way to use sql query as source , we need custom sql since we call few functions which is hard to re design in snowflake 2. can we do a

pre sql or post sql

?, we do have some huge tables where we don’t have a primary key , so to minimize the el job run time can extract last 3 months of data (based on creation date since we know older data won’t change) from source and delete the data with same extraction logic in target table prior to starting the target load , is there a way to achieve this ? (Transformer ??) 3. also appreciate if someone can help me on overriding the table name in target , let’s say I have a table cust in source and I want it to be loaded into target as cust_source 4. For incremental extraction (using a timestamp column last_update_date ) can we override the value we pass in to timestamp column? Ex:- can we override the delta logic to go few hours back from last run time , something like

where last_update_date>= add_hours(-6,LastRunTime)

Appreciate if you can point me to docs or share any example Yml files

I'll let others speak on the rest, but for 3 you can alias a stream with stream maps.

Hi @choudary kukkapalli for 2 you'd probably be able to use dbt pre / post hooks https://docs.getdbt.com/reference/resource-configs/pre-hook-post-hook