Hi, what do you think minimal (optimal) sync interval of Meltano could be? Let's say I use

tap-mysql

(CDC approach) with

target-s3

and want to continuously stream the data changes into S3. I know that Meltano is designed for batch / micro batch approach. How do you use CDC + Meltano?

That's a good question. Based on my experience, my initial response would be something like every 5 minutes, as long as you don't have a more sophisticated Meltano orchestration setup. Mostly because with the minimal sync frequency, you want to make the runtime < sync interval, and considering: • Warm-up of containers • Warm-up of Meltano • The data run time My feeling would be that with 5 minutes, you're very much on the safe side while still being pretty minimal.