Hi, is it possible to specify a setting

kind

when adding a custom plugin with

meltano add --custom {type} {name}

? I’ve forked and modified

tap-csv

to be able to pull a CSV file from a URL, but I haven’t been able to find a way of specifying the setting

kind

when adding the plugin. For

tap-csv

, the setting

files

is an array of objects as defined in

discovery.yml

here, but not being able to provide its

kind

through the prompt means that Meltano treats the setting as a string by default - this leads to type errors being thrown in the tap. Is there any way around this which doesn’t involve manually editing

meltano.yml

?

@Reuben (Matatika) No, there's not currently a way to specify a kind (or any other setting properties) through the

add --custom

prompt. Can you explain why editing

meltano.yml

is not an option in your case?

Thanks for the quick reply! We are running various Meltano commands inside a Kubernetes pod from a series of shell scripts. This means that in the case where we need to run

meltano elt

with

tap-csv

, the CSV file to be loaded has to exist in the pod in the first place (hence the modification). With the change, the idea is to use the modified

tap-csv

and the

TAP_CSV_FILES

environment variable to resolve the

files

setting in order to load CSVs by URL and run

meltano elt

successfully. Hopefully that makes sense!

All right, makes sense, you're building the project on the fly using the CLI instead of having it pre-built in a repo. Right now I think your best bet it is to create a script that reads the

meltano.yml

file, adds

kind: array

where you need it, and then writes it back to

meltano.yml

. Another approach would be to not create a custom tap-csv, but to add the regular one and modify

meltano.yml

(programmatically) to update the

pip_url

to point at your fork, so that you inherit the settings definition including

kind: array

from the original plugin.

@Reuben (Matatika) just in case you've missed it,

tap-spreadsheets-anywhere

can pull CSVs from a variety of locations (local file, url, s3...)

@laurent Thanks for the heads up - did not know that tap existed… That solves my problem entirely! @douwe_maan I will file an issue for specifying

kind

for settings through

meltano add --custom

. Thanks for the help!