https://linen.dev logo
Join Slack
Powered by
Hi everyone; quick comment about Meltano's variant...
# plugins-general
j
Hi everyone; quick comment about Meltano's variant of 
target-snowflake
. This is kind of a follow up to https://meltano.slack.com/archives/C01TCRBBJD7/p1689179609824739 . While I no longer have problems E+L into Snowflake on my Windows environment, I have noticed that the loader is leaving behind the 
.json.gz
files after execution while working in Windows. I will note that I haven't seen this behaviour in the container which I created in docker-compose that runs Meltano in a linux container (Debian), as it gets rid of the files afterwards, proof attached of screenshots. It's not that big of a deal right now since I'm still prototyping and assumption is whenever this gets to prod I won't even have this issue because I'm using a Linux container. But thought I'd let here know anyways.
To clarify a bit, I did not expect those `.json.gz`in the 2nd screenshot to be cleaned up in the first place as Meltano is not working with those files. I was using that as proof for myself that Meltano was cleaning up the files it was using at the time while in a linux environment, but keeping them behind in Windows
Again, relatively minor issue 😛
p
@joshua_janicas yes the intended behavior is that those files are cleaned up after theyve been uploaded to snowflake. I wonder if its having another path issue when its trying to clean those files up. Would you mind creating an issue in the target repo to track this as a bug?
Oh and one quick check - you're not overriding this default 
clean_up_batch_files
config, correct? https://hub.meltano.com/loaders/target-snowflake#clean_up_batch_files-setting. I assume not but have to ask
j
ya thats definitely not a thing i have in my solution 😮
and I will! currently away f rom my work computer but i'll take care of it in the morning
Open in Slack
PreviousNext
Powered by