Hi everyone. I am trying to build a custom tap to ...
# singer-tap-development
Hi everyone. I am trying to build a custom tap to get astronomy data from REST api(sunrise-sunset.org/api). Currently the api serves the data based on a particular date with few other parameters. I have a requirement to load the data since 1st Jan 2020. I am currently using meltano sdk to build a custom tap. Attached is the code snippet (streams.py) I am using to stream the data. Currently I will get only one record for a particular day. How can I change the meltano stream such that I can load the data over an interval. Ex: from 1st Jan 2020 to 20 Oct 2022. i.e, How to make multiple rest api calls for each day as the api doesn't have any pagination or date interval support?? Detailed requirement: We want to build a dataset for Sunrise and Sunset timing for Pune( lat=18.5204 long=73.8567). The tap should perform the following function: 1. Historical Load : load historical data since 1 Jan 2020 2. Incremental Load : Append today’s data in existing target 3. Transform data : Transform the timestamp from UTC to IST