Hi everyone! Need advice:
So I'm working on a tap that extracts data from API, this API have a specific:
• API can be queried with granularity of one day, i.e. I can ask for events starting from 2023-02-24 but not from specific time
What I need to do:
• Express in stream schema, that stream is incremental
• Set incrementality to synthetic date field
• Emit bookmark manually when all the records for a specific date are emitted
• Iterate by date in REST API calls
Questions:
• Is RESTStream a good base for such tap?
• How do I provide information about date? Do I ingest sythetic field like
event_date
to each event and rely on built-in functionality?
• How do I ensure that bookmark is emitted at the end of the batch?