question regarding targets that set undesired / incorrect types - we're currently using

RealData/target-bigquery

which hardcodes all

number

to

FLOAT

instead of

NUMERIC

(which is critical for applications relying on accurate values). I don't see a way to override the target configuration in order to fix things, so I'm wondering what the best practices are and where the responsibility for changes / fixes lies in cases like this - should we use a different target (which invariably would end up with us forking the target repo and having to maintain it separately)? should meltano be responsible for managing config changes like this instead? is there something else I'm missing?

@daren_makuck I suggest filing an issue and submitting a pull request on

RealSelf/target-bigquery

to either change the type for

number

or make it configurable. There's not much Meltano can do to affect what happens -inside the target: https://github.com/RealSelf/target-bigquery/blob/a30ec3198081b97d820b33d10d8743e63b00ddea/target_bigquery.py#L93 Note though that the https://github.com/adswerve/target-bigquery fork is much more actively maintained than RealSelf's original, so I'd suggest using and contributing to that

haha yeah I have an open issue there already with the same note 😃 - just curious as to whether or not there are other considerations to give when troubleshooting / developing - is this the kind of thing we'll always have to fix at the target level? does any of this make sense to be implemented in meltano or elsewhere?

Translating the Singer schema and record messages output by a tap into a appropriate representation for a specific destination is exactly the responsibility of the target 🙂 Part of the power of Meltano/Singer is that each connector is an individual project, with each tap compatible with each target, and Meltano compatible with all taps and targets, without Meltano needing to know anything about any specific source or destination. We are building a Singer SDK to make it easier to build taps and targets that behave correctly and consistently, but the source or destination specific logic is never going to be in Meltano itself

oh good spot, thanks!

Yeah, Meltano can have some influence on what makes it into the target, but it's up to the target to interpret that for the destination in question

ok cool, well thanks for the help, really appreciate you taking the time here 😃

Happy to help!

look forward to seeing how the SDK progresses, too!