Morning all, i'm trying to create a simple setup t...
# troubleshooting
r
Morning all, i'm trying to create a simple setup that will copy data from a MySQL instance to a Postgres instance. I've got it mostly working, but there's mappings I need to do on the way through (in my case, weird date issues in the MySQL that Postgres doesn't like). I came across an exponential problem with naming and came to the conclusion that there must be a better way:
Copy code
version: 1
default_environment: dev
project_id: demo-project

environments:
- name: dev
- name: staging
- name: prod

plugins:
  extractors:
  - name: tap-mysql
    variant: transferwise
    pip_url: git+<https://github.com/transferwise/pipelinewise.git#subdirectory=singer-connectors/tap-mysql>
    config:
      <db config with database=testdb>
    select:
    - testdb-TABLE_A.*
    - testdb-TABLE_B.*

  loaders:
  - name: target-postgres
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/target-postgres.git>
    config:
      <db config>

  mappers:
  - name: meltano-map-transformer
    variant: meltano
    pip_url: meltano-map-transform
    mappings:
    - name: demo-mapping
      config:
        stream_maps:
          testdb-TABLE_A:
            __alias__: table_a
          stagingdb-TABLE_A:
            __alias__: table_a
          proddb-TABLE_A:
            __alias__: table_a
          testdb-TABLE_B:
            __alias__: table_b
          stagingdb-TABLE_B:
            __alias__: table_b
          proddb-TABLE_B:
            __alias__: table_b
Is there any way to "rename" or "alias" the select so that I don't have to have the database name everywhere?
r
Yep:
Copy code
config:
        stream_maps:
          "*-TABLE_A":
            __alias__: table_a
          "*-TABLE_B":
            __alias__: table_b
or if you wanna get really fancy:
Copy code
config:
        stream_maps:
          "*":
            __alias__: "__stream_name__.rsplit('-', 1)[-1].lower()"  # testdb-TABLE_A -> table_a
https://sdk.meltano.com/en/latest/stream_maps.html#aliasing-two-or-more-streams
👌 1
✅ 1