I’m using pipelinewise tap-mysql and adswerve target-bigquery, and I’ve found a pretty serious bug in each. Has there been any thinking/work done on Singer spec conformance testing?

tap-mysql

turns out to not properly implement FULL_TABLE sync, and

target-bigquery

ignores

STATE

messages that do a

singer.statediff.Remove

state operation. (The combination together causes a catastrophic state corruption that breaks the state and requires manual state patching to sync new rows on a per-table basis). I wonder if there is a generic test harness that we can come up with to demonstrate conformance? Or perhaps the

singer

/

meltano

SDKs can offer some stub test suites for taps/targets that drive some common test data? For example if we had some target conformance tests simulating

tap-csv => your-target

(leaving the target’s tests to define what is the correct end-state in the loaded dataset) then some of the edge-cases could start to be fleshed out.