I ve just seen a reference to the ` from file` option Is thi Meltano #troubleshooting

I've just seen a reference to the `--from-file` op...

ian_lewis

11/16/2023, 1:34 PM

I've just seen a reference to the

--from-file

option. Is this still in use anywhere? It looks very useful but not well documented. https://docs.meltano.com/reference/command-line-interface/#how-to-use-read-setting-value-from-a-file Edit: This appears recent. https://github.com/meltano/meltano/blob/dd62d102928535ad23308753de79e7a7e4a79402/src/meltano/cli/config.py#L360

edgar_ramirez_mondragon

11/16/2023, 10:34 PM

Yeah, it hasn't been shipped yet!

It looks very useful but not well documented.

PRs welcome if you think something's not clear and a short example or explanation might help! I'm planning to ship a prerelease next week, but in the meantime you could install the edge with

Copy code

pipx install 'git+<https://github.com/meltano/meltano.git@main>' --suffix='@latest'

and use it as

Copy code

meltano@latest run ...

ian_lewis

11/17/2023, 4:50 PM

I'll see what I can do on the PR front. Thanks for the installation tip, I'll give that a go.

ian_lewis

11/20/2023, 3:34 PM

Hi @edgar_ramirez_mondragon I have now tried the

--from-file

function and it looks very promising. I created a yaml file containing the following:

Copy code

- name: table1
    selected: true
    format: json
    path: <s3://some-bucket>
    pattern: some-prefix-here/table1/.*.singer.gz
    key_properties: [id]
    start_date: '2015-01-01T00:00:00Z'
  - name: table2
    selected: true
    format: json
    path: <s3://some-bucket>
    pattern: some-prefix-here/table2/.*.singer.gz
    key_properties: [ id ]
    start_date: '2015-01-01T00:00:00Z'

I run the command:

$ meltano@latest --environment=data-prod config tap-spreadsheets-anywhere set tables --from-file tables.yml

Produces the following output:

Copy code

Extractor 'tap-spreadsheets-anywhere' setting 'tables' was set in the active environment in `meltano.yml`: "- name: table1\n    selected: true\n    format: json\n    path: <s3://some-bucket>\n    pattern: some-prefix-here/table1/.*.singer.gz\n    key_properties: [id]\n    start_date: '2015-01-01T00:00:00Z'\n  - name: table2\n    selected: true\n    format: json\n    path: <s3://some-bucket>\n    pattern: some-prefix-here/table2/.*.singer.gz\n    key_properties: [ id ]\n    start_date: '2015-01-01T00:00:00Z'\n

Which is rendered as follows - clearly not usable:

Copy code

- name: tap-spreadsheets-anywhere
        config:
          tables: "- name: table1\n    selected: true\n    format: json\n    path:\
            \ <s3://some-bucket>\n    pattern: some-prefix-here/table1/.*.singer.gz\n\
            \    key_properties: [id]\n    start_date: '2015-01-01T00:00:00Z'\n  -\
            \ name: table2\n    selected: true\n    format: json\n    path: <s3://some-bucket>\n\
            \    pattern: some-prefix-here/table2/.*.singer.gz\n    key_properties:\
            \ [ id ]\n    start_date: '2015-01-01T00:00:00Z'\n

Is there something about the formatting of the source file that should be taken into account here?

edgar_ramirez_mondragon

11/21/2023, 7:31 PM

Thanks for trying it out @ian_lewis! Can you try converting the YAML array to JSON and passing that? That is,

$ meltano@latest --environment=data-prod config tap-spreadsheets-anywhere set tables --from-file tables.json

where

tables.json

contains

Copy code

[
  {
    "name": "table1",
    "selected": true,
    "format": "json",
    "path": "<s3://some-bucket>",
    "pattern": "some-prefix-here/table1/.*.singer.gz",
    "key_properties": [
      "id"
    ],
    "start_date": "2015-01-01T00:00:00Z"
  },
  {
    "name": "table2",
    "selected": true,
    "format": "json",
    "path": "<s3://some-bucket>",
    "pattern": "some-prefix-here/table2/.*.singer.gz",
    "key_properties": [
      "id"
    ],
    "start_date": "2015-01-01T00:00:00Z"
  }
]

ian_lewis

11/22/2023, 8:35 AM

I'll give that a try and report back. Thanks @edgar_ramirez_mondragon

ian_lewis

11/22/2023, 10:06 AM

Hi @edgar_ramirez_mondragon I tried and the data is rendered as a string.

edgar_ramirez_mondragon

11/22/2023, 3:03 PM

Hmm, I can confirm that worked for me so I think your plugin definition may be missing the fact that

tables

if of

kind: array

. You might wanna run

meltano lock --update --all

ian_lewis

11/22/2023, 3:03 PM

Ah, right. I'll do that and try again.

ian_lewis

11/22/2023, 3:03 PM

Thanks again!!

ian_lewis

11/22/2023, 3:22 PM

Unfortunately the file is still being read as a string which is then deposited into the tables definition unchanged. Looks like the

.lock

file is correct

Copy code

"settings": [
    {
      "name": "tables",
      "kind": "array",
      "label": "Tables",
      "description": "An array holding json objects .... or see the [repo README](<https://github.com/ets/tap-spreadsheets-anywhere>) for more details.\n"
    }
  ]

edgar_ramirez_mondragon

11/22/2023, 4:15 PM

That's weird. This is the project I'm using to test if you wanna take a look: https://github.com/edgarrmondragon/meltano-s3-csv-example/blob/main/meltano.yml

ian_lewis

11/22/2023, 4:17 PM

Thanks @edgar_ramirez_mondragon I'll make some time to take a look.

edgar_ramirez_mondragon

11/23/2023, 6:05 PM

FWIW I published a prerelease (

v3.2.0b1

) and here's a link to a codespace where I confirmed the following commands work as expected:

Copy code

meltano config tap-spreadsheets-anywhere unset tables
meltano config tap-spreadsheets-anywhere set tables --from-file tables.json

Open in Slack

Previous Next