Hello, I have a question, where does meltano store source data before loading it into destination ? I guess since it says

Writing batch with 2223 records

it stores them somewhere but i couln't find out where. I guess it's documented somewhere so sorry I couldn't find it

the target handles that in memory, technically there's a few buffers the data goes through before it gets to STDIN and the target

So if I understand, it's handled in the target, so if I use a postgre or snowflake target it will be handeled there and not in the container where I deploy meltano ?

Those really aren't mutually exclusive, but it depends on what you're doing and how your container is setup. Normally the answer is the container that has meltano would run something like

meltano elt tap-abc target-abc

Both the tap and target exist in the container

Yeah exactly i'm trying to understand where the data is and in what format (i'm guessing Json) while I run

meltano elt

command. I couldn't find documentation on it

Could you be more specefic with your ask? Like what commands are you currently running? Are you asking what the singer messages are?

meltano invoke tap-abc > out

look at the messages that come from your tap here

Sorry if i'm not clear. I can see that the output is json of the source data , i'm running the command

meltano elt tap-google-analytics target-postgres

(I also run meltano invoke) what i'm trying to find is if this json data is added directly to the destination (i'm guessing not) and if not where is it temporarly stored before being pushed to the destination (in memory of the container or in a temporary table or in a file...)

We're getting somewhere! the data listed in

out

above with invoke is what's sent to STDIN of the target. The target decides what to batch up and when and how to send the data to postgres. Almost always this is done in memory

Oh yes i did know that it's what's sent to the STDIN target, but i did not know that it's the target that decides how to manage it. I'll try to find out more by looking at the postgres target implementation. Thank you !