Is there a way to put the set of tables for tap-csv/tap-spreadsheets-anywhere into a separate file? I have 1000+ csvs which is loaded into redshift and my meltano.yml isn't looking pretty. My config looks like this:

plugins:
  extractors:
  - name: tap-vendor-a
    namespace: vendor-a
    pip_url: git+<https://github.com/ets/tap-spreadsheets-anywhere.git>
    executable: tap-spreadsheets-anywhere
    capabilities:
    - catalog
    - discover
    - state
    config:
      tables:
      - path: <s3://some_path>
        name: staging_table_a
        pattern: path_a/table_a_*
        start_date: '2020-10-01T00:00:00Z'
        key_properties:
        - key
        format: csv
        universal_newlines: false
        prefer_number_vs_integer: true
      - path: <s3://some_path>
        name: staging_table_b
        pattern: path_b/table_b_*
        start_date: '2020-10-01T00:00:00Z'
        key_properties:
        - id
        - updated_at
        format: csv
        universal_newlines: false
        prefer_number_vs_integer: true
        ...

would love to have something like:

plugins:
  extractors:
  - name: tap-vendor-a
    namespace: vendor-a
    pip_url: git+<https://github.com/ets/tap-spreadsheets-anywhere.git>
    executable: tap-spreadsheets-anywhere
    capabilities:
    - catalog
    - discover
    - state
    config:
      tables: PATH_TO_SOME_FILE

@hisham_osman

tap-csv

supports a

csv_files_definition

setting that you could use: https://meltano.com/plugins/extractors/csv.html#csv-files-definition, but

tap-spreadsheets-anywhere

doesn't currently have an equivalent. I expect that @eric_simmerman would be open to a contribution to https://github.com/ets/tap-spreadsheets-anywhere that would add that, though! There's an open Meltano issue to let project plugin definitions (and their configurations) be housed in individual YML files (https://gitlab.com/meltano/meltano/-/issues/2206#note_388496082), which would move this config to

extractors/tap-spreadsheets-anywhere.yml

and make

meltano.yml

less heavy, which I think would address your issue. We could also explore supporting

config: path/to/config.{yml,json}

so that you can define all of your config (including

tables

) in a separate file, or a way to defer to a file for the contents of a specific array or object setting (like

tables

) if a string is provided instead of a proper JSON array/object, like

tables: path/to/tables.{yml,json}

. If we wanted to support loading a value from a file for arbitrary settings (not just those holding arrays or objects), we would need a way to indicate that the path is not supposed to be interpreted as a string value, but rather as a path to a file holding the value, using some special syntax that wouldn't be confused for or used in regular configuration. Something like

"@tables": path/to/tables.yml

, which would evaluate to

tables: <contents of path/to/tables.yml>

in the actual config passed to the plugin.

@douwe_maan I'd absolutely love to contribute this. I'll create an issue then we can discuss from there

@hisham_osman That'd be awesome, thanks!