#8080 feature: Partial ("Preview") Syncs
Issue created by
anden-akkio
### Feature scope
API
### Description
At
Akkio, we have a use case where we want to fetch a limited subset of user-imported datasets. We do this so that we can show users a small preview of what their end data might look like after what's essentially an internal transformation process we run, and we want them to be able to potentially make changes and make sure that transformation process looks good before they commit to the full data sync.
After a brief discussion with
@DouweM, this feature doesn't yet seem to exist on Meltano, but is something that would potentially fall under Meltano's purview. The current state of Meltano essentially forces you to do the full data sync before you can actually show it to the user, resulting in subpar experience if the end output isn't really what they want. While split-second queries aren't what Meltano is built for, nor are what I'm really proposing here, allowing for a preview window of sorts still seems like a positive addition cohesive with the rest of the tool to me.
Dropping some other comments that were made, for full context:
One option would be to add a limit config on the Tap SDK to only load a certain number of records, and another to add a mode to meltano elt to stop after the first state message is output by the target. (Targets flush records and write state messages at different batch sizes (often configurable), but until the first state message comes we can’t assume any of the records have made it to the destination, so we couldn’t base this on number-of-records on the target or meltano elt side, that’d need to be controlled by the target.)
meltano/meltano