Hello, I'm using https://github.com/MeltanoLabs/tap-universal-file tap to get the data from S3. Is there any way to override schema definition in tap? Currently It auto detect based on first record from the file.

What do you want to achieve with overriding? Change datatype of a column? Rename a column?

I want to hardcode schema/catalog in tap. I know It's possible in tap config but Is there any way to reference catalog file in tap config?

What type of file are you pulling? For Delimited Files (CSVs etc) there's https://github.com/MeltanoLabs/tap-universal-file/tree/main#:~:text=delimited_override_headers The

select

itself works to allow you to select / deselect fields in the tap. Which gets to @Andy Carter’s point, what's your purpose of overriding the schema? Maybe it's a feature that is / isn't supported by the tap maybe it is but it all depends on the purpose of what you're doing You can definitely override the schema here https://docs.meltano.com/concepts/plugins#schema-extra

Yep, Thanks. We can also provide a schema json path here in tap, https://github.com/meltano/sdk/blob/main/singer_sdk/streams/core.py#L106

Nice so

delimited_override_headers

worked for you here?

I''m fetching jsonl files not the csv

Did you inherit from the universal file to setup multiple taps for each of the different files? Mostly curious now about your use case as we're trying to understand how to design this tap still #C05CNUF699B 😄 , we went with a single definition for files instead of offering a list of files and I'm still not 100% sold on it

Tap https://github.com/MeltanoLabs/tap-universal-file from MeltanoLabs already supports multiple file types

I know we made it 😅

No slightly different schema

Got it so

jsonl_sampling_strategy

could support

all

which would sample the whole file using something like https://github.com/wolverdude/GenSON and auto fill the json Then for

jsonl_type_coercion_strategy

would fit into this as well. Hmmm The hack we have right now is to use

jsonl_type_coercion_strategy

and then use

envelope

or

any

depending on what you're after. envelope will put everything in a record object you can expand in your target. I don't love it. Sounds like for you, it'd be much better if we sampled everything with genson and provided a schema. Or if you could provide your own schema mappings for the file

I already made the change in my forked one to provide schema manually. I can open a PR If you want.

@chintan_patel yes please! That would be great 😄