Hi, I'm trying to alias all my streams with a pref...
# troubleshooting
g
Hi, I'm trying to alias all my streams with a prefix using this target config:
- name: target-postgres
variant: meltanolabs
pip_url: meltanolabs-target-postgres
config:
host: ${INGESTION_DB_HOST}
port: ${INGESTION_DB_PORT}
database: ${INGESTION_DB_NAME}
user: ${INGESTION_DB_USER}
password: ${INGESTION_DB_PASSWORD}
ssl_mode: ${INGESTION_DB_SSL_MODE}
default_target_schema: ${INGESTION_DB_SCHEMA}
_`# https://sdk.meltano.com/en/latest/stream_maps.html`_
stream_maps:
"*":
__alias__: config['tables_prefix'] + __stream_name__
stream_map_config:
tables_prefix: ${TABLES_PREFIX}
But after running the job, the Python evaluation is not executed and I end up with a table
config['tables_prefix'] + __stream_name__
1
am I missing something?
forgot to mention that I'm using meltano v3.5.4
e
__stream_name__
is an upcoming feature of the Singer SDK, which doesn't have a final release yet: https://sdk.meltano.com/en/latest/stream_maps.html#built-in-alias-variable-names Once it's released, we generally update target-postgres fairly quickly. I'm also planning on doing pre-releases of target-postgres that in turn use pre-release versions of singer-sdk, but that's a bit down the line.
g
My bad, just realized that this page was related to the sdk and not to the engine: https://sdk.meltano.com/en/latest/stream_maps.html#aliasing-a-stream-using-alias ... thanks for the clarifications @Edgar Ramírez (Arch.dev). P.S: • Any ETA for this update? • Any workaround I can use in the meantime?
e
Milestone 0.42.0 is due by the 11th of November and there's one open item in it so I'm not sure it can be completed earlier than that. What I can try to do is open a PR to bump to a 0.42.0 pre-release and make a pre-release of target-postgres from there. Then you can update the plugin's
pip_url
in
meltano.yml
to get the bleeding edge.
Ok, I've published an alpha release of target-postgres. You can try it out with
Copy code
pip_url: meltanolabs-target-postgres==0.1.0a1
g
Works great using something like: stream_maps: "*": __alias__: "'${TABLES_PREFIX}' + __stream_name__" Thanks a lot @Edgar Ramírez (Arch.dev). For the record I was unable to use stream maps config like: stream_map_config: tables_prefix: ${TABLES_PREFIX} Using the "config['tables_prefix']" syntax
🙌 1