are there any docs on how data is passed from sing...
# docker
b
are there any docs on how data is passed from singer tap to target? im wondering if this is a legit idea… in a single k8s pod executed by airflow: • tap runs in container1 with a mounted config file • tap writes to a fifo file (shared volume between container 1 & 2) • target reads from fifo file • target runs in container2 with a mounted config file the result would be more generic, leaner, reusable images (only 1 image per tap/target, rather than 1 phat image for a whole meltano project)
a
Hi, @ben_marengo. I think you might have difficulty with the synchronous read and write from the same file, and at least as of today I don’t think there’s a smooth way to orchestrate this. In theory though, what you describe could probably work.
The recommended way is that the data is piped between tap and target using STDOUT->STDIN. This allows streaming to work and the tap and target can each handle pagination and batching as makes sense for each. Have you seen the Singer Spec docs on meltano.com? @taylor just added these recently as a simplified version of the full spec. Singer Spec | Meltano