Hello, I have a question, where does meltano store...
# plugins-general
h
Hello, I have a question, where does meltano store source data before loading it into destination ? I guess since it says
Writing batch with 2223 records
it stores them somewhere but i couln't find out where. I guess it's documented somewhere so sorry I couldn't find it
v
the target handles that in memory, technically there's a few buffers the data goes through before it gets to STDIN and the target
h
So if I understand, it's handled in the target, so if I use a postgre or snowflake target it will be handeled there and not in the container where I deploy meltano ?
v
Those really aren't mutually exclusive, but it depends on what you're doing and how your container is setup. Normally the answer is the container that has meltano would run something like
meltano elt tap-abc target-abc
Both the tap and target exist in the container
h
Yeah exactly i'm trying to understand where the data is and in what format (i'm guessing Json) while I run
meltano elt
command. I couldn't find documentation on it
v
Could you be more specefic with your ask? Like what commands are you currently running? Are you asking what the singer messages are?
meltano invoke tap-abc > out
look at the messages that come from your tap here
h
Sorry if i'm not clear. I can see that the output is json of the source data , i'm running the command
meltano elt tap-google-analytics target-postgres
(I also run meltano invoke) what i'm trying to find is if this json data is added directly to the destination (i'm guessing not) and if not where is it temporarly stored before being pushed to the destination (in memory of the container or in a temporary table or in a file...)
v
We're getting somewhere! the data listed in
out
above with invoke is what's sent to STDIN of the target. The target decides what to batch up and when and how to send the data to postgres. Almost always this is done in memory
h
Oh yes i did know that it's what's sent to the STDIN target, but i did not know that it's the target that decides how to manage it. I'll try to find out more by looking at the postgres target implementation. Thank you !