Hey folks, diving deep into the Meltano SDK to bui...
# singer-tap-developmenth
hawkar_mahmod
03/21/2023, 5:22 PMHey folks, diving deep into the Meltano SDK to build a REST based Tap. If I have an API that is parameterised by a path parameter (e.g. app_id) is this a good candidate for using stream partitioning? It's not quite clear to me where it would be useful.
hawkar_mahmod
03/21/2023, 5:23 PMTo put a bit more flesh on that - the app ID is a value you set to get data for a specific ID from this API. At first I thought I'd just have it as a config that I set at run time.
e
edgar_ramirez_mondragon
03/21/2023, 5:25 PMHi @hawkar_mahmod!
Yeah, using partitions would allow users of your tap to configure one or more app ids and the state for each app would be safely partitioned
h
hawkar_mahmod
03/21/2023, 5:27 PMThanks @edgar_ramirez_mondragon! That makes sense. Is the idea that a partition is a implementation detail of the data source but we'd like to combine the data downstream. Is that the reason for their existence?
e
edgar_ramirez_mondragon
03/21/2023, 8:40 PMExactly, the loader is not aware of the partitions and it sees a single stream