https://linen.dev logo
Join SlackCommunities
Powered by
hi all, I am using `tap-postgres` (meltanolabs) to...
# singer-taps
a
hi all, I am using 
tap-postgres
(meltanolabs) to extract data to load into BigQuery, the destination tables are coming through as 
schema_table
(ex. 
public_table1
). Is there a way to rename the destination table?
✅ 1
r
a
thanks, I'll take a look
@Reuben (Matatika) is that a config for the tap or the target?
r
Either if they are both built with the SDK: > 1. For taps, the SCHEMA and RECORD messages will automatically be transformed, duplicated, filtered, or aliased, as per the 
stream_maps
config settings after all other tap-specific logic is executed. > • Because this process happens automatically after all other tap logic is executed, the tap developer does not have to write any custom handling logic. > • The tap development process is fully insulated from this ‘out-of-box’ functionality. > 2. Similarly for targets, the received streams are processed by the 
stream_maps
config setting prior to any Sink processing functions. > • This means that the target developer can assume that all streams and records are transformed, aliased, filtered, etc. before any custom target code is executed. https://sdk.meltano.com/en/latest/stream_maps.html#a-feature-for-all-singer-users-enabled-by-the-sdk
a
great I'll give it a shot. Thanks @Reuben (Matatika)
👍 1
@Reuben (Matatika) running into some issues
Copy code
- name: tap-postgres
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-postgres.git>
  - name: tap-postgres-test
    inherit_from: tap-postgres
    config:
      default_replication_method: INCREMENTAL
      filter_schemas:
        - public
      sqlalchemy_url: ${URI}
    select:
      - public-blc_offer.*
      - public-blc_product.*
    metadata:
      public-blc_offer:
        replication-method: INCREMENTAL
        replication-key: date_updated
        table-key-properties: [offer_id]
      public-blc_product:
        replication-method: INCREMENTAL
        replication-key: date_updated
        table-key-properties: [product_id]
    stream_maps:
      public-blc_offer:
        __alias__: blc_offer
      public-blc_product:
        __alias__: blc_product
Added the 
stream_maps
to the tap but destination table names are unaffected (still 
public_blc_offer
and 
public_blc_product
)
r
I believe 
stream_maps
goes under 
config
section.
a
that did the trick! Thanks again @Reuben (Matatika). Rock star ⭐
np 1
🙌 1
2 Views
Open in Slack
PreviousNext
Powered by