https://linen.dev logo
Join Slack
Powered by
Hello, Peeps! Having hard time fixing this error. ...
# troubleshooting
m
Hello, Peeps! Having hard time fixing this error. I am running 'tap-mysql' 'target-s3' pipeline with default variants and am getting:
Copy code
jsonschema.exceptions.ValidationError: '"{\\"uuid\\":\\"b0f01821-5ced-4ad1-a430-9c3765c1e777\\",\\"email\\":\\"<mailto:xyz@email.com|xyz@email.com>\"}"' is not of type 'null', 'object'
here is the later part of the error
Copy code
023-11-02T16:59:44.788635Z [info     ] Failed validating 'type' in schema['properties']['data']: cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3
2023-11-02T16:59:44.788760Z [info     ]     {'inclusion': 'available', 'type': ['null', 'object']} cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3
2023-11-02T16:59:44.788873Z [info     ]                                cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3
2023-11-02T16:59:44.789003Z [info     ] On instance['data']:           cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3
2023-11-02T16:59:44.789120Z [info     ]     '"{\\"uuid\\":\\"b0f01821-5ced-4ad1-a430-9c3765c1e777\\",\\"email\\":\\"<mailto:sammy.deveille+saml@springpod.com|sammy.deveille+saml@springpod.com>\\"}"' cmd_type=elb consumer=True name=target-s3 producer=False stdio=stderr string_id=target-s3
2023-11-02T16:59:44.872938Z [error    ] Loader failed
2023-11-02T16:59:44.873102Z [error    ] Block run completed.           block_type=ExtractLoadBlocks err=RunnerError('Loader failed') exit_codes={<PluginType.LOADERS: 'loaders'>: 1} set_number=0 success=False
Read through similar threads here, and tried them, but haven't been able to resolve it. Does anyone knows the solution?
the variant is 'transferwise' if that helps
e
Can you try using schema to override the JSON schema type for the 
data
field?
Copy code
extractors:
- name: tap-mysql
  schema:
    some_stream_id:
      data:
        type: ["string", "null"]
m
Hi, @edgar_ramirez_mondragon I tried the schema override method. Not sure, if I did it right though. Below are my plugin configs:
Copy code
- name: test-tap-mysql
    inherit_from: tap-mysql
    schema:
      students-partner_students:
        data:
          type: ['string', 'null']
    config:
      database: students
      engine: mysql
      filter_dbs: students
      host: <removed for privacy>
      port: 3306
      session_sqls:
      - SET @@session.time_zone='+0:00'
      - SET @@session.wait_timeout=28800
      - SET @@session.net_read_timeout=3600
      - SET @@session.innodb_lock_wait_timeout=3600
      ssl: true
      user: root

    select:
    - students-partner_students.data
    - students-partner_students.uuid
    metadata:
      students-partner_students:
        replication-method: FULL_TABLE
I am still getting the same error. Read about chaecking the 'catalog' or 'properties' json file for tap from your previous responses. Here is what i am getting for executing: meltano invoke test-tap-mysql --discover > properties.json
Copy code
meltano invoke test-tap-mysql --discover > properties.json
2023-11-07T00:54:36.922036Z [info     ] Environment 'dev' is active
time=2023-11-06 16:54:37 name=tap_mysql level=INFO message=Attempting SSL connection
time=2023-11-06 16:54:39 name=tap_mysql level=INFO message=Server Parameters: version: 8.0.23, wait_timeout: 28800, innodb_lock_wait_timeout: 3600, max_allowed_packet: 67108864, interactive_timeout: 28800
time=2023-11-06 16:54:40 name=tap_mysql level=INFO message=Server SSL Parameters(blank means SSL is not active): [ssl_version: TLSv1.2], [ssl_cipher: ECDHE-RSA-AES256-GCM-SHA384]
and this when I execute 'meltano invoke test-tap-mysql --discover > catalog.json'
Copy code
meltano invoke test-tap-mysql --discover > catalog.json
2023-11-07T00:51:33.315695Z [info     ] Environment 'dev' is active
time=2023-11-06 16:51:34 name=tap_mysql level=INFO message=Attempting SSL connection
time=2023-11-06 16:51:36 name=tap_mysql level=INFO message=Server Parameters: version: 8.0.23, wait_timeout: 28800, innodb_lock_wait_timeout: 3600, max_allowed_packet: 67108864, interactive_timeout: 28800
time=2023-11-06 16:51:36 name=tap_mysql level=INFO message=Server SSL Parameters(blank means SSL is not active): [ssl_version: TLSv1.2], [ssl_cipher: ECDHE-RSA-AES256-GCM-SHA384]
but, still getting stuck at the json validation error.
e
Hmm, one way to inspect the schema is to run 
meltano invoke test-tap-mysql > singer.jsonl
and look for the 
SCHEMA
message. You should see 
"data": {"type": ["string", "null"]}
if everything's working properly. Might be worth manually removing 
.meltano/run/test-tap-mysql/tap.properties.json
too.
Open in Slack
PreviousNext
Powered by