https://linen.dev logo
#singer-tap-development
Title
# singer-tap-development
m

matt_arderne

10/16/2021, 3:53 PM
Hey all, trying my hand at the hackernews API over the weekend. Bit of a noob when it comes to building production apps in python as opposed to analysis. https://github.com/mattarderne/tap-hackernews How should I iterate and develop? After making a cookiecutter, adding some obvious bits and then running it (it worked!), I'm wondering what the most effective workflowfor doing the more subtle refinements would look like? Currently I have just hardcoded a single HN endpoint into the Streams module but I need to do a few things: 1. I need to check the max_id from one endpoint and use that as an input for another endpoint 2. the URL needs a
.json
appended, ie the path needs to look something like
path = "/item/{item_id}.json"
Do I just keep running the whole package ie
poetry run tap-hackernews --config config.json
blindly until the outputs make sense? Is there a way to run single modules? How do I add and check logic as I go. I've been trying to setup poetry with VSCode debugger so that I can analyse each step, but I'm getting stuck with the debugger not actually executing the tap (it doesn't log the process nor show any outputs, just loads it)