Can you pass data into a tap? I am developing a tap for Appcues and in order to pull users you must provide a

user_id

with this endpoint (can't export a list of user). I have all the user_ids as they are the external ids from our company. But I am wondering if you can stream data from a target->tap->target? Is this also possible in Meltano?

Could you add the list of user ids into the tap

config

? This is a pretty common pattern, a bit like how

tap-spreadsheets-anywhere

handles multiple source definitions.

Hmmm, that would be possible. But I feel like adding 200k users to the

config

would be bit "janky". Not the cleanest solution but it might be worth it. Thanks Andy!

Where does the list of 200k user_ids live right now? Similar to Andy's solution some taps require a path input to another file like reports for tap-ga4 or csv_file_definition for tap-csv. Its not exactly the same but you could have a required config in the tap that is a path to a CSV or JSON file with the giant list of user IDs in it. To go even further you could then do

meltano run tap-postgres target-csv tap-appcues target-xyz

where target-csv writes the input file that tap-appcues uses, so its effectively dynamic. Just brainstorming though 🤔

Oh thats a good thought Pat. Yes we could do

meltano run tap-postgres target-csv tap-appcues target-xyz

. Thats a fair idea and is pretty dynamic. Initially I was just curious if we can pass records from target to tap like we do with tap to target but I think Andy and yours suggestions look pretty good when laid out. Thank you, I'll look into implementing something of the sort but am open to all suggestions.

Yeah this is an interesting question - this issue is somewhat relevant https://github.com/meltano/sdk/issues/319. I wonder if you could build a tap that also supports target features like this. Its not impossible, the SDK has a tap_base and a target_base so theoretically you could implement the target functions to call the tap when it receives a batch of data. Sounds like a fun thing to experiment with

That sounds interesting. If I didn't have a tight deadline I'd implement now, but I think I'll start with reading from a

csv

and then look into that. I like the idea of tap/target hybrids and could honestly use them in many different places in our current pipeline.

@bubba I ended up working on something similar to your use case where I needed to call an API for every input record. If youre interested check out https://github.com/MeltanoLabs/map-gpt-embeddings/pull/7. I basically used the SDK tap functionality in a mapper so I could effectively go

tap-x tap-y target-z

but in reality its

tap-x mapper-y target-z

where the mapper makes a request for every input record from tap-x.

@pat_nadolny Very cool, I think I could implement this in the future! Thank you for sharing with me 😊 The concept of merging a tap and mapper has crossed my mind in the past, so it's definitely helpful to see an example! Thanks again!