https://linen.dev logo
Join Slack
Powered by
This message contains interactive elements.
# troubleshooting
c
This message contains interactive elements.
meltano-error-log.txtmeltano.yml
t
@visch have you seen this before?
v
I haven't seen this with this tap/target combo, I like the test though. Two issues it seems to dive into 1. Are decimals just not working? 
numeric
should probably be 
numbers
with 
multipleOf
set and max/mins. I'm not sure that's happenign today with 
tap-postgres
if it's not should the tap transform the data to a string since that's currently what seems to be getting chosen as the type? 2. Does the target handle 
numbers
correct? I think right now if the target sees a 
multipleOf
in the schema it uses varchar for the field, and then the target "should" convert to a string and push that data along but it's clear that isn't happening here @carlos_bernal_carvajal thank you for the issue submission. To get a solution out asap for you there's two solutions I can think of one is you could override the schema on the tap side for your field using https://docs.meltano.com/concepts/plugins#schema-extra for your field I think you could override the schema to just be 
number
and I think it would work but you may have to experiment with it. Really this should work out of the box and it looks like a missing test on both the tap and the target here but I"ll have to dive to be sure that's what's going on
If all else fails @carlos_bernal_carvajal and my override doesn't work you could make a view in your source DB, convert that column to a varchar and then select from there (not ideal by any means but it would work if you really need that data) Another evem quick solution would be to just not select the offending field in 
public-ChainEntityCollectionStage
@taylor here's a similar issue https://github.com/MeltanoLabs/tap-postgres/issues/150 Looks like I stumbled on it 3 weeks ago myself, I do have a good replication path.
c
Thank you @visch. I’ll give it a try, and let you know. If it helps, as FYI, the source and target Postgres databases have different versions. Source Postgres version: 13.7 Target Postgres version: 15.3
@visch What does 
some_stream_id
refer to? I’m not sure I understand what I’m supposed to set there. (sorry if it’s a dumb question)
v
@carlos_bernal_carvajal in your case it would be 
public-ChainEntityCollectionStage
Would it be ok to just exclude that field @carlos_bernal_carvajal ?
c
Not ideal, but if it’s temporary, I think we exclude it
v
I guess start with the 
schema
override first, if that fails we can exclude it!
c
Will do 👍
The 
schema
override stoped the error for blocking the 
meltano run
, however on the target, the 
price
column got stored as 
varchar
I was expecting that it would be stored as number as well
This is the override I used, on the tap-postgres extractor:
Copy code
schema:
  public-ChainEntityCollectionStage:
    price:
      type: number
v
@carlos_bernal_carvajal awesome I'm glad it's running now for you. You would expect the same type to come across we have an issue in for that now!
c
Got it, it’s good enough for now. Thanks!
Two more questions though.
1. Another 
price
column needed the same treatment, but when I added it to the same schema extra, it didn’t work, I kept getting the same 
is not of type 'string'
error. Maybe I’m doing it wrong? This is the override I used was:
Copy code
schema:
  public-ChainEntityCollectionStage:
    price:
      type: number
  public-Collection:
    price:
      type: number
2. Is there a way to increase the “performance” of the replication? On a 2 vCPU and 8 Gb Ram EC2 instance (t3.large), replication 10 tables took Airbyte 8 minutes, but Meltano took more that 30 minutes. Is there any configuration I can modify to improve speed?
Open in Slack
PreviousNext
Powered by