Hi folks, I am trying to create a new tap-redshift (SDK variant). I am facing a small issue 😞 I am trying to add a column mapping for

super

data type in redshift to

json

, but I am not able to find any flexible type in typing.py. The closest I see is CustomType, but I think it requires the schema as an input.

def __init__(self, jsonschema_type_dict: dict) -> None:

Is there a way to have a flexible schema to incorporate this ?

I havent dug into the sql tap side of things enough to help with the type mapping but maybe @edgar_ramirez_mondragon @ken_payne @visch would be able to help. I would refer you to https://github.com/MeltanoLabs/tap-snowflake/pull/15 for reducing the amount of tables that are discovered. It adds the tables config option that limits discovery to a list of tables https://github.com/MeltanoLabs/tap-snowflake/blob/37fea116dce1f81544e194cbb1195df8ef2cee7d/tap_snowflake/tap.py#L56

Thanks alot @pat_nadolny, let me check this out :) Also, it looks like there is no way to add a composite primary key for tables? I am trying to add an option to manually specify a list of PKs for table.

hi @silverbullet1 !

is not able to load selected streams

Are you encountering an exception, or is it that that tap seems to be skipping those streams. If it’s the former, sharing the traceback might help us investigate what’s going on.

Hey Edgar, I am not getting any exception. Its just that, it starts scanning everything by default. It also doesn’t read my type mapping changes. These are the logs:

Did not recognize type 'super' of column 'account_info'
  for column_def in inspected.get_columns(table_name, schema=schema_name):
/Users/xx/Library/Caches/pypoetry/virtualenvs/tap-redshift-sI6JsuHw-py3.11/lib/python3.11/site-packages/singer_sdk/connectors/sql.py:436: SAWarning: Did not recognize type 'super' of column 'attributes'
  for column_def in inspected.get_columns(table_name, schema=schema_name):
/Users/xx/Library/Caches/pypoetry/virtualenvs/tap-redshift-sI6JsuHw-py3.11/lib/python3.11/site-packages/singer_sdk/connectors/sql.py:436: SAWarning: Did not recognize type 'super' of column 'artifacts'
..
..

I suspect that there is a problem with function overriding here and here because its complaining about super data type not being mapped, even though I have mapped it to

ObjectType()

. Maybe the problem with table selection is also related.

I took a quick peak, for it to be easy to help you really need to give a reproducible example, give the input, the output, and show the full exception. Too many things missing here for me to be able to help / give you an answer

That looks like a sqlalchemy warning so I wonder if you have to add the

sqlalchemy-redshift

package