Linen
Join Slack
Hi all. I'm using pipelinewise-tap-postgres and target-redshift. One of my tables has a TEXT field i...
# troubleshooting
d
Hi all. I'm using pipelinewise-tap-postgres and target-redshift. One of my tables has a TEXT field in PG, and the max length is about 30k characters. The column in Redshift has been created as varchar(10000). Is there a way I can tell meltano or the target to use a wider column? I've ended up with some values truncated.
e
If you're using pipelinewise-target-redshift, there seems to be support for varchar(65535) if the incoming string field has 
maxLength
greater than 10,000. You can force the schema of your TEXT field with Meltano's schema extra:
Copy code
extractors:
- name: tap-postgres
  schema:
    my_schema-my_table:
      my_text_column:
        type: ["string", "null"]
        maxLength: 30000
d
Thank you. I'll give that a shot.
I can't seem to get this to work. Every time the column ends up as varchar(10000)
Copy code
- name: tap-postgres
    variant: transferwise
    pip_url: pipelinewise-tap-postgres==2.1.0
  - name: tap-hamilton
    inherit_from: tap-postgres
    load_schema: hamilton_raw_20220422
    select:
    - public-organisation.*
    - public-organisation_member.*
    - public-phone_call.*
    - public-platform.*
    - public-platform_member.*
    - public-subdomain.*
    - public-tag_definition.*
    - public-tag_value.*
    - public-team.*
    - public-team_member.*
    - public-team_setting.*
    - public-queued_call_cleaned.*
    - public-user.*
    schema:
      public-team_setting:
        # target-redshift defaults to a max length of 10k, but team-settings can be much longer
        value:
          type: [ "string", "null" ]
          maxLength: 65535
I can see this in my logs:
Copy code
Property `schema.properties.value` was not found in the schema of stream `public-team_setting`
I switched from pipelinewise-tap-postgres to the meltanolabs variant and it worked fine.
👍 1
3 Views
Open in Slack
PreviousNext
Powered by

Meltano

Meltano community

Powered by