https://linen.dev logo
Join Slack
Powered by
Hi, seeming some issues with some of my target-sno...
# singer-targets
m
Hi, seeming some issues with some of my target-snowflake(s). anybody having an issue with timestamp columns i.e. 
TIMESTAMP
type instead of 
TIMESTAMP_NTZ
? anybody else. just popped recently... thinking it's the new tag possibly?
e
It's possible 0.13.0 introduced this via f25852e7e53f0edf86ea6b2d92522f5ae4f9e2d0, so it might worth confirming 0.12.0 doesn't have the same problem
m
so are we going to 
TIMESTAMP
from 
TIMESTAMP_NTZ
for this? just curious. seems to be tied to sql-alchemy dependency from the logs
fyi, i did downgrade and it passed
e
sql-alchemy dependency from the logs
can you share a link to that?
fyi, i did downgrade and it passed
do you mean the error persisted after downgrading?
m
1. log-error
Copy code
NotImplementedError: Default TypeEngine.as_generic() heuristic method was unsuccessful for target_snowflake.snowflake_types.TIMESTAMP_NTZ. A custom as_generic() method must be implemented for this type class.
2. yes, went to 0.12.0 and it doesn't have this issue
e
Ok, so both
Copy code
TIMESTAMP_NTZ().as_generic()
sct.TIMESTAMP_NTZ().as_generic()
fail also in v0.12.0 so I don't think it's related to snowflake-sqlalchemy updates. Which makes me think it's the upstream Singer SDK changes.
Ok I think this should do it for that type: https://github.com/MeltanoLabs/target-snowflake/pull/304
m
interesting... i dunhno what to say, i added that tag and the error subsided
c
I'm experiencing this issue as well however using the plugin version from that PR I still experience the same errors
Copy code
event='ERROR:sqlconnector:Error adapting column type for \'"DEV_WAREHOUSE"."RAW_HALO"."SQLFILTER"._SDC_DELETED_AT\', \'TIMESTAMP_NTZ\' to \'DATETIME\' (new sql type)'
Copy code
event='NotImplementedError: Default TypeEngine.as_generic() heuristic method was unsuccessful for target_snowflake.snowflake_types.TIMESTAMP_NTZ. A custom as_generic() method must be implemented for this type class.'
Over the 12 or so jobs I have it seems to be the _SDC_DELETED_AT which I believe is added by the target so it seems other date fields may be fine? I'm going to try reverting to 0.12.0
I seem to be having the same issue on 0.12.0 as well 🤔
If it helps my source loader is pipelinewise-tap-mssql using LOG_BASED replication so the _SDC_DELETED_AT is not really used as the tap adds it's own _LSN_DELETED_AT
e
using the plugin version from that PR
Did you reference the branch in your 
pip_url
?
m
are you importing singer directly? seems like you'll have to keep downgrading till you get the right config that works like it did before?
c
Yeah I tried ref directly, I'm new to python so taking a bit longer than I'd like to find my way in certain areas. I confirmed locally that .meltano folder restores the new version when I run meltano install. I'm running my deployed workload in a container on ECS so not overly easy to observe. Ref I used have been pip_url: git+https://github.com/MeltanoLabs/target-snowflake.git@edgarrmondragon/fix/SQLConnector.jsonschema_to_sql pip_url: meltanolabs-target-snowflake==0.13.0 pip_url: meltanolabs-target-snowflake==0.12.0 I did also update meltano last week as well, what I'm struggling with is working out what versions my production workload is on as that is still running fine but my dev pipelines are all broken and I can't see anything being changed other than me rebuilding the container which I assume has pulled in latest versions of either meltano of one of it plugins
I don't believe I'm importing singer directly
m
you doin't have a pyproject.toml or requriements.txt that specifies the meltano dependency on your build?
c
No pyproject.toml and my requirements.txt just has boto3 in it, I am using the meltano/meltano:latest image from docker hub. Is it possible this isn't purely target related and something that is a combination of 3.6.0 meltano and targets? I see there were changes to Activate Version in recent releases and I seem to only be having issues with _SDC_DELETED_AT but hard to tell cause can't see any other date fields in error messages
Still having trouble resolving this within my Meltano project, if I clone updated branch and run locally using poetry I have no issues, however running within my Meltano project still errors even when invoking manually and injecting the input.json file that was used under poetry. I did note that Meltano was updating my plugin on invoke back to the meltano labs default due to the package lock file not updating the pip url from the meltano.yml file when running an update of the lock file. However manually setting the pip url resolved the auto install, I was then able to see if no longer upgrading on invoke however the problem persists. I guess this must be something to do with my environment and having burnt close to a day on it I guess I'll wait for the fix to be merged and hopefully it addresses my issue then.
e
I'll be working on it in the coming days
version 
0.14.1
at least pins snowflake-sqlalchemy to version known to work. Let me know if that fixes the problem for you!
28 Views
Open in Slack
PreviousNext
Powered by