https://linen.dev logo
Join Slack
Powered by
:waves: Getting an error that I'm not sure what to...
# troubleshooting
b
waves Getting an error that I'm not sure what to do with when trying to run a custom tap and loading to snowflake. We recently migrated from using edcast-target-snowflake (which seems to be 💀) to pipelinewise-target-snowflake. We can successfully run the tap and the loader separately by using 
meltano invoke
, but when we put them together with 
meltano elt
the job fails. We are using an older version of the Meltano SDK, 0.3.18, not sure if that matters. Full log in thread 👉
Copy code
2023-01-25T21:15:26.511195Z [error    ] Loading failed                 code=-9 job_id=jaris_snowflake message=time=2023-01-25 21:14:34 name=target_snowflake level=INFO message=Adding column: ALTER TABLE raw_jaris."MERCHANTS" ADD COLUMN "SHOPS" variant name=meltano run_id=a68d986c-b6e7-42a5-9a97-d319844b4bda
2023-01-25T21:15:26.511773Z [info     ] ELT could not be completed: Loader failed cmd_type=elt job_id=jaris_snowflake name=meltano run_id=a68d986c-b6e7-42a5-9a97-d319844b4bda stdio=stderr
ELT could not be completed: Loader failed
2023-01-25T21:15:26.576513Z [error    ] Future exception was never retrieved
future: <Future finished exception=BrokenPipeError()> 
BrokenPipeError
meltano_log
meltano_log
p
@ben_cohen are those columns in snowflake now? It seems like it succeeds in creating them but I wonder if the user doesn't have proper permissions to alter those tables 🤔
How are you testing it with invoke?
b
Copy code
meltano invoke tap-jaris > singer.jsonl
cat singer.jsonl | meltano invoke taret-snowflake
p
hmmm thats so strange
b
if it were an access issue in snowflake that would fail, right?
p
yeah it would
b
meltano is running on k8s, maybe my pod is just too small and its running out of memory???
but that error is not super helpful
p
yeah if your data shows up in snowflake with https://meltano.slack.com/archives/C01TCRBBJD7/p1674682978875289?thread_ts=1674682083.707649&amp;cid=C01TCRBBJD7 then its unlikely that its related to the tap or the target having bugs. Are you able to run locally or with a smaller data set?
b
yeah let me try...
yeah im able to get that to work using a smaller data set
i bumped the ram on the k8s pod and chef kiss -- thanks for the moral support and guidance
Open in Slack
PreviousNext
Powered by