Hi, what is the status of MeltanoLab's

target-csv

? It's not listed in MeltanoHub. And what's the status of

target-parquet

? I'm unable to use the default (estrategiahq).

Good call out on target-csv! We need to get that added to the Hub - @pat_nadolny can you add that to your list?

Thanks Murphy. For now I've abandoned using the parquet target.

@peter_pezon the meltanolabs target-csv is now on the hub https://hub.meltano.com/loaders/target-csv--meltanolabs/ you can install it using

meltano add loader target-csv --variant meltanolabs

Hey, what does it mean that

target-csv

supports

stream_maps

? does that mean i can set a stream map on the target config, instead of tap config, or using a mapper in between tap and target?

https://sdk.meltano.com/en/latest/stream_maps.html it means you can define the stream map within the target config and it applies to all data coming in - that’s only if the target was built on the SDK though. For those that aren’t built on the SDK you can insert a mapper in between the tap and target using

meltano run

Related to target-parquet, I also wasnt able to get it working 😞 . I get

AttributeError: Can't pickle local object 'persist_messages.<locals>.consumer'

when I try to run it. Although I know @jacob_matson uses it in his MDS in a box https://github.com/matsonj/nba-monte-carlo/blob/0e19c0974af1c0a6decb781cae62ea9f4473e243/meltano.yml#L50 so its possible its a system related issue 🤔

Yes been using the target-parquet successfully for a while. Wonder if it’s related to python version or ARM / m1

yeah interesting, I tried running inside the same python:3.9 docker image you use and it worked for me. So its definitely a system related issue.

docker run -it -v $(pwd):/projects -w /projects python:3.9 /bin/bash

the original dependency was on superset, it didn't work on 3.10 or higher so had to do 3.9. never tested it on higher versions.

Following up - what is outcome here? Are we saying that Mac OS users should use target-parquet only in a docker image? Would like to run this locally without having to build an image first - as building a docker image is pretty slow. Over 5 minutes to build, >1 GB image size. (Separately, also seeking to reduce image size.)

I created https://github.com/estrategiahq/target-parquet/issues/10 to track the issue but I havent had time to look into it and I dont have much multi processing experience. I'm imagining that its not a huge refactor to fix it, I assume the target should be updated to use a more platform compatible method 🤷

Got it.