Hi, I have set up a clean new meltano project to t...
# plugins-general
c
Hi, I have set up a clean new meltano project to try out the the new target-redshift. I note that it uses the pipelinewise connector that previous worked fine for me as a custom exgractor. I’m tapping SF. When I run it I get the target
account
table, with the 13,000 rows in it I need, but…with every single value NULL except for the
id
primary key from SF. What’s going on?
Config is like this:
Copy code
plugins:
  extractors:
  - name: tap-salesforce
    variant: meltano
    pip_url: git+<https://gitlab.com/meltano/tap-salesforce.git>
    config:
      api_type: BULK
      client_id: xxx
      start_date: '2021-04-01T00:00:00Z'
    select:
      - Account
  loaders:
  - name: target-redshift
    variant: transferwise
    pip_url: pipelinewise-target-redshift
    config:
      host: xxx
      port: 5439
      user: xxx
      password: null
      dbname: xxx
      s3_bucket: xxx
      s3_key_prefix: xxx
      compression: gzip
      slices: 8
      default_target_schema_select_permission: ro_group
      add_metadata_columns: true
v
I actually have this same exact thing happening with my tap-oracle. I'm going to dive in later and I"ll let you know what I find on my end
haha we did the exact same thing!
Copy code
select:
      - Account
You need to do
Copy code
select:
      - Account.*
It selects the defaulted included items for that stream which is normally just the key_properties. I just forget to include the * to include all of the attributes!
Easiest way to find it is to go look at your catalog in .meltano/run/tap*/tap.properties
c
lol
re-running now…
v
And now that I'm thinking more clearly, the easiest way to find this is to use meltano
meltano select --list tap-name
which would have shown you all the items that were deselected