Looking for a something like a `tap-jsonl`… I have...
# plugins-general
r
Looking for a something like a `tap-jsonl`… I have this massive API client written in JS. I’d rather not rewrite it in Python, so was thinking of sticking it together with an exchange format like jsonl. Except
tap-jsonl
doesn’t exist.
tap-csv
does. But anyone have any other recommendations for data formats for deeply nested data (CSV not a great fit)?
Orrr are my options either writing a custom
tap-jsonl
or reimplementing the whole client as a
tap-custom-api
p
e
Thanks for sharing that Peter! I think tap-singer-jsonl is not a good fit, though. It expects the JSONL file to contain valid Singer messages.
are my options either writing a custom
tap-jsonl
or reimplementing the whole client as a
tap-custom-api
FWIW implementing a custom tap-jsonl should be fairly easy, and would have the advantage of letting you decide how to select files and where to pull them from.
m
r
@Edgar Ramírez (Arch.dev) good point, though I feel directory structures could be abstracted away into a unified interface, I’m happy to move files around a bit if it saves writing/maintaining a plugin. @Matt Menzenski didn’t realise it had JSONL support! Diving in now. I also just came across https://github.com/MeltanoLabs/tap-universal-file
👀 1
Idea for hub.meltano.com: allow tags for extractors etc that are searchable. There are apparently at least two taps that support jsonl that didn’t come up in search
🙌 1
👍 1
r
FWIW I get some results when I type in "jsonl":
💡 1
but not if I search for "tap-jsonl":
e
right, search does include the plugin's keywords!
👍 1
r
Then I’m simply blind as a bat : ) I only saw the Singer JSONL one
e
I was curious why tap-universal-file hadn't yet been added to the Hub: https://github.com/meltano/hub/issues/1445
👀 2
r
Not to hijack this thread, but I had a similar dilemma with
tap-msaccess
(also using
fsspec
under the hood): https://github.com/meltano/hub/pull/1663