Just thinking out loud here, but I’m wondering if ...
# plugins-generala
andrew_stewart
08/30/2020, 3:01 AMJust thinking out loud here, but I’m wondering if it’d be feasible to create a conversion tool to generate singer specs from OpenAPI/Swagger specs. (possibly even utilizing/abusing the OpenAPI codegen).
Could be a nice way to streamline API singer specs.
a
al_whatmough
08/31/2020, 9:06 AMThat’s a cool idea!
al_whatmough
08/31/2020, 9:07 AMThis isn’t the same thing but there's some useful stuff in https://github.com/singer-io/singer-tools
al_whatmough
08/31/2020, 9:09 AMSpecifically the singer-infer-schema tool to generate a schema based on the output of the tap. I’ve also started trying out https://github.com/anelendata/tap-rest-api which has an —infer_schema option
a
andrew_stewart
08/31/2020, 6:00 PMOh right, these are great.
andrew_stewart
08/31/2020, 6:01 PMIt looks like it wouldn’t be all that much work to use one of these as a starting point to read an OpenAPI spec
andrew_stewart
08/31/2020, 6:02 PMMaybe I’ll give that a shot as a weekend project.
n
nevin_morgan
08/31/2020, 6:44 PMI was actually talking to @douwe_maan about that last week. I can't figure out if it would be better to have a tap that reads the openapi spec, or a java powered codegen to have the swagger tools build a well written tap off of a spec
a
andrew_stewart
08/31/2020, 6:54 PMYep, I was thinking the same thing.
n
nevin_morgan
08/31/2020, 6:58 PMI finally made an issue to start collecting thoughts from the community on this possibility here https://gitlab.com/meltano/meltano/-/issues/2333
a
andrew_stewart
08/31/2020, 7:01 PMI can see pros and cons to both.
• an openapi spec reader should stay up to date with a changing API spec, but that might also not be a good thing if radical changes break your ingest.
• a codegen approach is useful to create more purpose built taps/targets, but then you have the issue of the taps/targets going stale if the spec changes. (of course with a proper REST API, that shouldn’t be a problem)
n
nevin_morgan
08/31/2020, 7:05 PMA spec reader tap could work well though if the user is supplying the spec each run. But does doing that coincide with the singer spec? I don't remember if it dictates tap inputs or not
d
douwe_maan
09/01/2020, 3:28 PM@nevin_morgan Are you suggesting passing the spec over stdin? Why not just use a key in
config.jsonn
nevin_morgan
09/01/2020, 3:29 PMnot over stdin, but yeah a config.json key would work if a generic tap that understands the spec is built out
nevin_morgan
09/01/2020, 3:29 PMI don't really know how that sort of tap would handle authentication env vars