Hi all. I'm using pipelinewise-tap-postgres and ta...
# troubleshootingd
dylan_just
03/05/2024, 4:22 AMHi all. I'm using pipelinewise-tap-postgres and target-redshift. One of my tables has a TEXT field in PG, and the max length is about 30k characters. The column in Redshift has been created as varchar(10000). Is there a way I can tell meltano or the target to use a wider column?
I've ended up with some values truncated.
e
Edgar Ramírez (Arch.dev)
03/05/2024, 6:34 PMIf you're using pipelinewise-target-redshift, there seems to be support for varchar(65535) if the incoming string field has
maxLength Copy code
extractors:
- name: tap-postgres
schema:
my_schema-my_table:
my_text_column:
type: ["string", "null"]
maxLength: 30000Edgar Ramírez (Arch.dev)
03/05/2024, 6:34 PMd
dylan_just
03/06/2024, 1:19 AMThank you. I'll give that a shot.
dylan_just
03/06/2024, 7:04 AMI can't seem to get this to work. Every time the column ends up as varchar(10000)
dylan_just
03/06/2024, 7:04 AM Copy code
- name: tap-postgres
variant: transferwise
pip_url: pipelinewise-tap-postgres==2.1.0
- name: tap-hamilton
inherit_from: tap-postgres
load_schema: hamilton_raw_20220422
select:
- public-organisation.*
- public-organisation_member.*
- public-phone_call.*
- public-platform.*
- public-platform_member.*
- public-subdomain.*
- public-tag_definition.*
- public-tag_value.*
- public-team.*
- public-team_member.*
- public-team_setting.*
- public-queued_call_cleaned.*
- public-user.*
schema:
public-team_setting:
# target-redshift defaults to a max length of 10k, but team-settings can be much longer
value:
type: [ "string", "null" ]
maxLength: 65535dylan_just
03/06/2024, 7:28 AMI can see this in my logs:
Copy code
Property `schema.properties.value` was not found in the schema of stream `public-team_setting`dylan_just
03/06/2024, 12:58 PMI switched from pipelinewise-tap-postgres to the meltanolabs variant and it worked fine.
👍 1
3 Views