connor_lough
09/01/2022, 2:51 PMdouwe_maan
09/01/2022, 2:54 PMaaronsteers
09/01/2022, 4:16 PMaaronsteers
09/01/2022, 4:16 PMaaronsteers
09/01/2022, 4:17 PMaaronsteers
09/01/2022, 4:21 PMaaronsteers
09/01/2022, 4:23 PMaaronsteers
09/01/2022, 4:23 PMkelly_mondor
09/01/2022, 4:24 PMhassan_syyid
09/01/2022, 5:04 PMhassan_syyid
09/01/2022, 5:04 PMaaronsteers
09/01/2022, 5:08 PMconnor_lough
09/01/2022, 6:11 PMalexander_butler
09/01/2022, 7:10 PMhassan_syyid
09/01/2022, 7:10 PMconnor_lough
09/06/2022, 3:42 PMalexander_butler
09/06/2022, 8:10 PMrun
function and have some niceties available like a connector to DWH + GCS storage + Feature Flag client
3. Each integration is its own .py
file with a main
function exposed as a poetry script (executable)
a. main
just instantiates the relevant class and executes the run
method of the class, __enter__
and __exit__
are conveniences for clean up
4. Meltano use pip_url: -e modules/reverse_etl
to install the sub dir
5. Currently I use inherits_from
to expose each function (which poetry creates as an executable) as its own utility. It could be valid to just use commands
and the "executable" config option exposed in meltano to keep them as a single utility
6. In production, just run it as needed. so meltano run tap-salesforce target-bigquery py_001_opportunity_enrichment
as an example pipeline
reasons are a bit long to get into but mostly a business-specific solution for reverse etl made more sense for us than a generic solution and had lower risk for critical operational systems