Hi Team, I am trying to create a pipeline from ta...
# best-practices
s
Hi Team, I am trying to create a pipeline from tap-snowflake to target-salesforce. Below is the snippet of my root meltano.yml file for both tap-snowflake and target-salesforce.
Copy code
- name: tap-snowflake
    namespace: tap_snowflake
    pip_url: ./connectors/tap-snowflake
    executable: tap-snowflake
    capabilities:
    - state
    - catalog
    - discover
    - about
    - stream-maps
    settings:
    - name: account
      kind: string
      value: aigoiop-hq79023
      description: The Snowflake account identifier.
    - name: user
      kind: string
      value: udey
      description: The Snowflake username.
    - name: password
      kind: string
      value: ***********
      description: The Snowflake password.
      sensitive: true
    - name: database
      kind: string
      value: PARTHAN_DB
      description: The Snowflake database name.
    - name: warehouse
      kind: string
      value: COMPUTE_WH
      description: The Snowflake warehouse name.
    select:
    - public-account.* 
-------------------------------------------------------------------
  - name: target-salesforce
    namespace: target_salesforce
    pip_url: ./connectors/target-salesforce
    executable: target-salesforce
    capabilities:
    - about
    - stream-maps
    - schema-flattening
    config:
      username: udey@vcs.sandbox
      password: ********
      security_token: nanLbbN3lexEw70gK7tLrzP4s
      api_type: sandbox
      #sobject: account
      action: insert
      stream_maps:
        public-employees:   
          target: Account     
          #key_properties: []   
          mappings:
          - source: name      
            target: Name
I am getting an error as below for this stream_maps.
Copy code
2025-04-02T12:36:00.006657Z [info     ]        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ cmd_type=elb consumer=True job_name=dev:tap-postgres-to-target-salesforce name=target-salesforce producer=False run_id=a2dbeee1-5078-41a8-a7d9-aba40dd70d46 stdio=stderr string_id=target-salesforce
2025-04-02T12:36:00.008151Z [warning  ] Received state is invalid, incremental state has not been updated
2025-04-02T12:36:00.098874Z [info     ] Incremental state has been updated at 2025-04-02 12:36:00.098808+00:00.
2025-04-02T12:36:00.100101Z [info     ] TypeError: unhashable type: 'list' cmd_type=elb consumer=True job_name=dev:tap-postgres-to-target-salesforce name=target-salesforce producer=False run_id=a2dbeee1-5078-41a8-a7d9-aba40dd70d46 stdio=stderr string_id=target-salesforce
2025-04-02T12:36:00.220899Z [error    ] Loader failed
2025-04-02T12:36:00.221982Z [error    ] Block run completed.           block_type=ExtractLoadBlocks err=RunnerError('Loader failed') exit_codes={<PluginType.LOADERS: 'loaders'>: 1} set_number=0 success=False
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Run invocation could not be completed as block failed: Loader failed
(meltanoEnv) PS C:\Siba_\Work\POC_ConnectorFactory\Gerrit\Connector_Factory_Development\meltano-backend>
Seems there is some issue with the stream_maps ! Can anyone please guide
1
r
Copy code
stream_maps:
        public-employees:   
          target: Account     
          #key_properties: []   
          mappings:
          - source: name      
            target: Name
This is not a valid
stream_maps
config. Specifically,
mappings
- what are you trying to do here?
s
I am trying to load the table(PUBLIC.EMPLOYEES) on snowflake to an object "Account" on Salesforce. If we dont do the mapping, the loader is trying to find the public-employees object on Salesforce.
r
You are probably conflating
mappings
from a mapper plugin (likely
meltano-map-transformer
) vs using
stream_maps
directly on an SDK tap/target.
You want to create the
public-employees
stream as an
Account
table in Snowflake?
Copy code
stream_maps:
        public-employees:
            __alias__: Account
https://sdk.meltano.com/en/latest/stream_maps.html#aliasing-a-stream-using-alias
s
yeah I want to copy the records from public-employees from Snowflake (Schema: Public & Table: Employees) to Salesforce Object called Account) The issue is Snowflake has column name as "name" but Salesforce has column name as "Name". That's why mapping is necessary.
r
Try
Copy code
stream_maps:
        public-employees:
            __alias__: Account
           Name: name
           name: __NULL__
https://meltano.slack.com/archives/C069CPBCVPV/p1743762477375819?thread_ts=1743761997.595339&amp;cid=C069CPBCVPV
s
Seems working.. Thanks @Reuben (Matatika).. Sorry I missed the last thread message.
👍 1
r
All good 🙂
🙌 1