Hi I am using tap mysql and have set `add metadata columns t Meltano #troubleshooting

Hi, I am using tap-mysql and have set: `add_metada...

Haaris Isaac Hussain

01/13/2025, 2:01 PM

Hi, I am using tap-mysql and have set:

add_metadata_columns: true

However I have noticed between variants that the metadata columns (i.e. prefixed _sdc) are not consistent. For the pipelinewise variant these columns are populated And for the MeltanoLabs variant these columns are not populated Anyone experienced this before?

Edgar Ramírez (Arch.dev)

01/15/2025, 4:09 AM

Probably because the MeltanoLabs variant is on an older sdk version. I can review a PR to bump it.

✅ 1

Haaris Isaac Hussain

01/16/2025, 10:56 AM

@Edgar Ramírez (Arch.dev) that would be great can you do that please as that's the only blocker for meltano at the moment

Haaris Isaac Hussain

01/16/2025, 10:56 AM

or could you tell me what the change is?

Haaris Isaac Hussain

01/20/2025, 10:55 AM

@Edgar Ramírez (Arch.dev)

Edgar Ramírez (Arch.dev)

01/20/2025, 11:59 PM

We need to bump the singer-sdk dependency in https://github.com/MeltanoLabs/tap-mysql. I can happily review a PR.

Haaris Isaac Hussain

01/21/2025, 2:07 PM

Is this where it needs to be updated? and just put it to the latest version? https://github.com/MeltanoLabs/tap-mysql/blob/0d6fcddce37c0f85fee1f32d451b251802579abf/poetry.lock#L1471

Edgar Ramírez (Arch.dev)

01/21/2025, 2:58 PM

Rather in

pyproject.toml

and then refresh the lock file.

Haaris Isaac Hussain

01/21/2025, 7:05 PM

@Edgar Ramírez (Arch.dev) I've raised a PR let me know if it requires any further changes

Haaris Isaac Hussain

01/29/2025, 9:23 AM

@Edgar Ramírez (Arch.dev) Thanks for merging my PR. I'm encountering an error with the

tap-mysql

extractor in Meltano. The error occurs during the catalog discovery phase and seems to be related to the

sqlalchemy

library. Here are the details: Error Message:

Copy code

AttributeError: module 'sqlalchemy.engine.reflection' has no attribute 'ObjectKind'

Full Traceback:

Copy code

<path_to_tap_mysql_executable> --config <path_to_config_json> --discover returned 1 with stderr:
2025-01-29 08:51:11,309 | INFO     | sqlconnector         | No is_vitess configuration provided, dynamically checking if we are using a Vitess instance.
2025-01-29 08:51:12,274 | INFO     | sqlconnector         | Instance is not a Vitess instance, using standard configuration.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "<path_to_tap_mysql_executable>\\__main__.py", line 7, in <module>
  File "<path_to_click_core>\\core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<path_to_click_core>\\core.py", line 1081, in main
    with self.make_context(prog_name, args, **extra) as ctx:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<path_to_click_core>\\core.py", line 949, in make_context
    self.parse_args(ctx, args)
  File "<path_to_click_core>\\core.py", line 1417, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<path_to_click_core>\\core.py", line 2403, in handle_parse_result
    value = self.process_value(ctx, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<path_to_click_core>\\core.py", line 2365, in process_value
    value = self.callback(ctx, self, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<path_to_singer_sdk>\\tap_base.py", line 554, in cb_discover
    tap.run_discovery()
  File "<path_to_singer_sdk>\\tap_base.py", line 309, in run_discovery
    catalog_text = self.catalog_json_text
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "<path_to_singer_sdk>\\tap_base.py", line 329, in catalog_json_text
    return dump_json(self.catalog_dict, indent=2)
                     ^^^^^^^^^^^^^^^^^
  File "<path_to_tap_mysql>\\tap.py", line 322, in catalog_dict
    result["streams"].extend(self.connector.discover_catalog_entries())
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<path_to_singer_sdk>\\connectors\\sql.py", line 998, in discover_catalog_entries
    (reflection.ObjectKind.TABLE, False),
     ^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'sqlalchemy.engine.reflection' has no attribute 'ObjectKind'

Haaris Isaac Hussain

02/07/2025, 3:24 PM

Hi @Edgar Ramírez (Arch.dev) We've merged this PR but I've noticed that SQLAlchemy <2 doesn't work with the latest Singer-sdk

Edgar Ramírez (Arch.dev)

02/07/2025, 4:25 PM

Yeah, we need to bump tap-mysql to use SQLAlchemy 2

Haaris Isaac Hussain

02/07/2025, 5:03 PM

@Edgar Ramírez (Arch.dev) I've been doing some testing locally. When bumping the SQLAlchemy version we need to change all instances of SQLs in conn.execute to be wrapped in text(). After changing these I'm getting this error now:

Copy code

File "C:\Users\haaris.hussain\OneDrive - Vistair Systems Ltd\Documents\tapmysqlcurrent\tap-mysql\.meltano\extractors\tap-mysql\venv\Lib\site-packages\simplejson\encoder.py", line 734, in _iterencode
    o = _default(o)
        ^^^^^^^^^^^
  File "C:\Users\haaris.hussain\OneDrive - Vistair Systems Ltd\Documents\tapmysqlcurrent\tap-mysql\.meltano\extractors\tap-mysql\venv\Lib\site-packages\simplejson\encoder.py", line 274, in default
    raise TypeError('Object of type %s is not JSON serializable' %
TypeError: Object of type FullyQualifiedName is not JSON serializableAfter reading through my script what can you suggest?

Edgar Ramírez (Arch.dev)

02/07/2025, 7:14 PM

Yeah, some chores are needed to make it work

Haaris Isaac Hussain

02/17/2025, 3:51 PM

@Edgar Ramírez (Arch.dev) What changes need to be made? Can we expect that to be done?

Haaris Isaac Hussain

02/17/2025, 5:36 PM

@Edgar Ramírez (Arch.dev) I think have found the problem, its the batch config we have implemented. After removing the audit fields are being populated

👍 1

10 Views

Open in Slack

Previous Next