Meltano

Dinking around with VizViewer anyone seen this? Guessing I'm just missing a large number of traces (maxed out the trace size or something) :thread:

<https://viztracer.readthedocs.io/en/latest/basic_usage.html#circular-buffer-size> found it

<@U06BX9QQ4A3> Looks like there's a ton of room for improvement here, but this target isn't using the latest sdk so there could be a bunch of optimizations I'm missing.

*Validate is slow with the basic json schema package, maybe we could speed that up (40% improvement disabling validate)
*Sequentially batching and parsing could be sped up significantly by parsing messages in their own threads. (N threads faster?)
*Compiling instead of interpreting? I wonder how much faster that would be :shrug:

I'm going to take my 40% easy gain and move on for now but there seems like a lot of room to improve

Oh, there's definitely room for improvement though I wonder if performance issues on targets that may accept bulk uploads are better handled by BATCH message types

Batch types would still run into this type of issue though as you still have to parse each record to get it into the file to upload