proud-pillow-5593503/18/2021, 9:56 PM
directory, what changes will be reflected in my
? I can't seem to find the documentation on this. The ones I've found are showing what to do if I am importing a dbt package, but not sure how to set a custom transform to a particular tap. Thank you!
flat-bear-8154603/19/2021, 1:10 AM
so you just configure whatever you want in the transform folder
proud-pillow-5593503/19/2021, 1:24 PM
file. However, all of the examples I am looking at load in dbt packages and I don't see how to utilize a custom transform I have defined in the
. Anybody been through this before?
Transform <transform_name> is not known to Meltano
flat-bear-8154603/22/2021, 2:16 PM
proud-pillow-5593503/22/2021, 2:24 PM
tap that also runs custom dbt transform code?
miniature-answer-2649503/22/2021, 2:27 PM
salmon-salesclerk-7770903/22/2021, 2:32 PM
proud-pillow-5593503/22/2021, 2:55 PM
salmon-salesclerk-7770903/22/2021, 2:58 PM
proud-pillow-5593503/22/2021, 3:04 PM
salmon-salesclerk-7770903/22/2021, 3:36 PM
proud-pillow-5593503/22/2021, 4:57 PM
I know I need to add a transform plugin for the
Your Meltano project does not contain a transform plugin for this extractor. Only proceed with running transformations as part of your pipeline if you've added these manually.
tap but it looks like I can only add custom plugins for extractors and loaders: https://meltano.com/docs/plugin-management.html#custom-plugins
salmon-salesclerk-7770903/22/2021, 5:26 PM
proud-pillow-5593503/22/2021, 5:31 PM
file would have to be configured:
# etc plugins: extractors: - name: tap-spreadsheets-anywhere pip_url: git+<https://github.com/ets/tap-spreadsheets-anywhere.git> config: tables: - path: <gs://bucket-name> name: example pattern: blob-path/folder/.* start_date: '2017-05-01T00:00:00Z' delimiter: '|' key_properties: - id format: csv # etc transforms: - name: tap-spreadsheets-anywhere # other settings here...
directory. If I do this, how would I configure my
plugin section in my
? 2. Add a section in my
that downloads the dbt package via a reference to the GitHub repo.
salmon-salesclerk-7770903/22/2021, 5:40 PM
and then you could run the elt with
# etc plugins: extractors: - name: tap-spreadsheets-anywhere pip_url: git+<https://github.com/ets/tap-spreadsheets-anywhere.git> model_name: my_dbt_models # this is where you can put whatever dbt models you want config: tables: - path: <gs://bucket-name> name: example pattern: blob-path/folder/.* start_date: '2017-05-01T00:00:00Z' delimiter: '|' key_properties: - id format: csv # etc transformers: - name: dbt # other settings here... config: models: $MELTANO_EXTRACT__MODEL_NAME
proud-pillow-5593503/22/2021, 5:42 PM
plugin section pretty sure
section and a
is where something like
is defined and configured, but
is where dbt packages would be defined and configured
salmon-salesclerk-7770903/22/2021, 5:45 PM
then that would override what’s passed to meltano elt
proud-pillow-5593503/22/2021, 5:46 PM
plugin section, but I don't really understand how to add a custom dbt package to that section. Does that make sense?
salmon-salesclerk-7770903/22/2021, 5:50 PM
ripe-musician-5993303/22/2021, 5:52 PM
proud-pillow-5593503/22/2021, 5:53 PM