Has anyone thought about / implemented Schemas for...
# singer-target-development
v
Has anyone thought about / implemented Schemas for Targets? Use case: SaaS Style targets, where the "sinks" are very schema specific. Right now I just hard fail if the data comes in and doesn't follow the desired schema, but following the Singer spec for Schemas would give me the same result I think
a
I like it! So, basically the target's JSON schema for one or more named target datasets/types would serve a validator and "contract" for incoming data?
v
exactly
a
Wow - yeah, I like it a lot! I imagine many things could be easily gated that way, like accepting "customerId" and not "CUSTOMERID" or "customer_id".
v
exactly, right now I"m hacky hacky, could implement a version of this with some basic json schema stuff
Also I didn't realize assert can be removed during compile time if you're running with a certain optimization level with python. eek ha
"compile time" ha
b
I have been having a wonderful time using pydantic for this sort of thing, recently. You declare a model as a class with python type annotations, and can add whatever custom validators you want. Lets me work in python using
snake_case
fields and then output as
camelCase
to pass to 3rd party apis pretty easy, as well. I think they have a generator utility that can build the models out of jsonschema, as well, but I haven't tried that out yet.
Bit late on the response, tho, haha
v
Popping back here to say splitting targets into seperate sinks just likes taps do for "streams" would be very nice and the pattern seems great. My use cases is users, and I'm just now adding groups.