GM , I am having issues in relication from mysql...
# troubleshootinga
ajay_kulkarni
03/01/2023, 5:45 PMGM , I am having issues in relication from mysql to Snowflake . Could someone help please? Thanks
t
thomas_briggs
03/01/2023, 6:42 PM@ajay_kulkarni what problems are you having?
a
ajay_kulkarni
03/01/2023, 6:43 PMHi Thomas, I am trying to sync tables from tap-mysql to snowflake
ajay_kulkarni
03/01/2023, 6:43 PMfull sync
ajay_kulkarni
03/01/2023, 6:44 PMI do not see anything replicated to snowflake schema.
ajay_kulkarni
03/01/2023, 6:44 PMHere is the screen log:
ajay_kulkarni
03/01/2023, 6:44 PM2023-03-01T175842.192932Z [info ] Environment 'prod' is active
2023-03-01T175846.437187Z [info ] time=2023-03-01 175845 name=tap_mysql level=INFO message=Server Parameters: version: 8.0.28, wait_timeout: 28800, innodb_lock_wait_timeout: 3600, max_allowed_packet: 67108864, interactive_timeout: 28800 cmd_type=elb consumer=False name=tap-mysql producer=True stdio=stderr string_id=tap-mysql
2023-03-01T175846.437700Z [info ] time=2023-03-01 175845 name=tap_mysql level=INFO message=Server SSL Parameters(blank means SSL is not active): [ssl_version: ], [ssl_cipher: ] cmd_type=elb consumer=False name=tap-mysql producer=True stdio=stderr string_id=tap-mysql
2023-03-01T175847.192776Z [info ] time=2023-03-01 175847 name=target_snowflake level=INFO message=Getting catalog objects from table cache... cmd_type=elb consumer=True name=mysql-sbizprod-snowflake producer=False stdio=stderr string_id=mysql-sbizprod-snowflake
2023-03-01T175849.220494Z [info ] time=2023-03-01 175849 name=target_snowflake level=WARNING message=No columns discovered in the schema "DATA_LAKE.SMARTBIZ_PRODUCTION_TEST" cmd_type=elb consumer=True name=mysql-sbizprod-snowflake producer=False stdio=stderr string_id=mysql-sbizprod-snowflake
2023-03-01T175849.221041Z [info ] time=2023-03-01 175849 name=target_snowflake level=INFO message=Emitting state {"currently_syncing": null} cmd_type=elb consumer=True name=mysql-sbizprod-snowflake producer=False stdio=stderr string_id=mysql-sbizprod-snowflake
2023-03-01T175849.236986Z [info ] Incremental state has been updated at 2023-03-01 175849.236876.
2023-03-01T175849.371571Z [info ] Block run completed. block_type=ExtractLoadBlocks err=None set_number=0 success=True
t
thomas_briggs
03/01/2023, 6:45 PMI think your "select" rules under the mysql tap are probably missing or wrong. What do they look like?
a
ajay_kulkarni
03/01/2023, 6:46 PMHere is meltano.yml, tap-mysql
ajay_kulkarni
03/01/2023, 6:46 PM- name: tap-mysql
variant: transferwise
pip_url: pipelinewise-tap-mysql
config:
host: sbizdbslave.clblw9gih8y7.us-west-2.rds.amazonaws.com
user: meltano
database: smartbiz_production
port: 3306
filter_dbs: smartbiz_production
select:
- smartbiz_production.loans.*
- smartbiz_production.users.*
- smartbiz_production.admins.*
metadata:
smartbiz_production.loans:
replication-method: FULL_TABLE
smartbiz_production.users:
replication-method: FULL_TABLE
smartbiz_production.admins:
replication-method: FULL_TABLE
t
thomas_briggs
03/01/2023, 6:47 PMIs smartbiz_production the schema name?
a
ajay_kulkarni
03/01/2023, 6:48 PMyes .. It is database and schema in mysql
t
thomas_briggs
03/01/2023, 6:48 PMThe streams produced by the tap will be named <schema>-<table> e.g. smartbiz_production-loans
thomas_briggs
03/01/2023, 6:49 PMSo you'll need to change the select rules to match that pattern
a
ajay_kulkarni
03/01/2023, 6:52 PMlooks like it is proceeding further .. with schema-tablename change
ajay_kulkarni
03/01/2023, 6:55 PMNow it is throwing error
ajay_kulkarni
03/01/2023, 6:55 PMtime=2023-03-01 185317 name=target_snowflake level=ERROR message=Error while executing MERGE query for table "SMARTBIZ_PRODUCTION_TEST."LOANS"" in stream "smartbiz_production-loans" cmd_type=elb consumer=True name=mysql-sbizprod-snowflake producer=False stdio=stderr string_id=mysql-sbizprod-snowflake
ajay_kulkarni
03/01/2023, 6:58 PMFile "/home/meltano/meltano-project/.meltano/loaders/target-snowflake/venv/lib/python3.9/site-packages/snowflake/connector/errors.py", line 210, in default_errorhandler cmd_type=elb consumer=True name=mysql-sbizprod-snowflake producer=False stdio=stderr string_id=mysql-sbizprod-snowflake
2023-03-01T185317.069524Z [info ] raise error_class( cmd_type=elb consumer=True name=mysql-sbizprod-snowflake producer=False stdio=stderr string_id=mysql-sbizprod-snowflake
2023-03-01T185317.069643Z [info ] snowflake.connector.errors.ProgrammingError: 002043 (02000): SQL compilation error: cmd_type=elb consumer=True name=mysql-sbizprod-snowflake producer=False stdio=stderr string_id=mysql-sbizprod-snowflake
2023-03-01T185317.069766Z [info ] Object does not exist, or operation cannot be performed. cmd_type=elb consumer=True name=mysql-sbizprod-snowflake producer=False stdio=stderr string_id=mysql-sbizprod-snowflake
2023-03-01T185317.474520Z [error ]
t
thomas_briggs
03/01/2023, 6:58 PMI don't know how much help I can be with that... I know almost nothing about Snowflake. Have you been through all the setup steps described here? When I tested with SF I remember having to jump through a bunch of hoops on their side to get everything to work.
a
ajay_kulkarni
03/01/2023, 7:31 PMseeing this error
ajay_kulkarni
03/01/2023, 7:31 PMmessage=Error while executing MERGE query for table