I was thinking on writing a guide on built a tap u...
# singer-tap-development
a
I was thinking on writing a guide on built a tap under 30 mins with tips. If I do it; any tips that folks would like to share.
t
Who’s the audience? Should we assume they’re having to install Python / poetry? Or more advanced?
Exciting either way!
p
@albert_m Awesome, I'm excited to check it out! Not an SDK related tip but I think it would be helpful include a quick thing on getting a VS code debugger going so users can step through their code and solve issues faster. Heres a thread around that topic
t
I made an MR to add some docs to the SDK based on some trouble I ran into https://gitlab.com/meltano/sdk/-/merge_requests/198 @pat_nadolny feel free to add anything to that MR or another
a
@pat_nadolny that is a great idea. @taylor I was thinking something basic for everyone. But you gave me an idea it can be a series from basic concepts to advance concept.
s
@albert_m i had started a tap guide but then abandoned it like i do most blog posts. one thing i was going to talk about was making multiple "passes" at the tap. something like: • rd 1: get a simple stream syncing at all (first auth, first schemas, full tap config) • rd 2: add pagination to stream 1 • rd 3: add more simple streams • rd 4: add replication keys for any incremental streams • rd 5: refactor to consolidate any multi-stream logic etc
but i thought the "do one thing, run a full sync, git commit" pattern might be helpful for beginners
a
@stephen_bailey, if you don't mind me using your structure. Of course, I will give you credit. 😀
s
i give it to you
a
Thank you.
m
@albert_m that would be super useful. One thing that I got stuck on was how one should decide where to build in features in either
client.py
streams.py
tap.py
I think this is worth addressing as the code samples have zero usage of
client.py
https://sdk.meltano.com/en/latest/code_samples.html
Another thing I wasn't sure about was the need to add API metadata to the schema in a Stream. Initially I had included all of the metadata in the Stream because I assumed I needed to add it to use it. Later on I stripped all the metadata out as the SDK handled all of that quite well. The only thing I then realised is that for incremental, I needed to add a field in the
parse_response()
, which then needed a corresponding field in
schema =