just tried to ```meltano add --custom loader targ...
# plugins-general
e
just tried to
Copy code
meltano add --custom loader target-parquet
Picked some sane defaults (I only customized the namespace) .. but now it just sits here. anyone experienced this ?
full log
Copy code
meltano add --custom loader target-parquet
Adding new custom loader with name 'target-parquet'...

Specify the plugin's namespace, which will serve as the:
- identifier to find related/compatible plugins
- default target database dialect (`dialect` extra),
  for use by transformers that connect with the database

Hit Return to accept the default: plugin name with underscores instead of dashes

(namespace) [target_parquet]: dump_stonks_raw

Specify the plugin's `pip install` argument, for example:
- PyPI package name:
        target-parquet
- Git repository URL:
        git+<https://gitlab.com/meltano/target-parquet.git>
- local directory, in editable/development mode:
        -e extract/target-parquet
- 'n' if using a local executable (nothing to install)

Default: plugin name as PyPI package name

(pip_url) [target-parquet]: 

Specify the plugin's executable name

Default: name derived from `pip_url`

(executable) [target-parquet]: 

Specify the target's supported settings (`config.json` keys)

Multiple setting names (keys) can be separated using commas.

A setting kind can be specified alongside the name (key) by using the `:` delimiter,
e.g. `port:integer` to set the kind `integer` for the name `port`

Supported setting kinds:
string | integer | boolean | date_iso8601 | email | password | oauth | options | file | array | object | hidden

- Credentials and other sensitive setting types should use the password kind.
- If not specified, setting kind defaults to string.
- Nested properties can be represented using the `.` separator, e.g. `auth.username` for `{ "auth": { "username": value } }`.
- To find out what settings a target supports, reference its documentation.

Default: no settings

(settings) [[]]: 
Added loader 'target-parquet' to your Meltano project

2022-03-05T18:04:53.049721Z [info     ] Emitter initialized with endpoint <http://sp.meltano.com/i>
Installing loader 'target-parquet'...
tried to ignore and power through by setting up a pipeline to tap postgres.. but met with a ton of these errors or warnings.. and then failure
Copy code
03-05T18:37:40.179269Z [warning  ] Python-dotenv could not parse statement starting at line 1
2022-03-05T18:37:40.179602Z [warning  ] Python-dotenv could not parse statement starting at line 1
2022-03-05T18:37:40.179881Z [warning  ] Python-dotenv could not parse statement starting at line 1
2022-03-05T18:37:40.180270Z [warning  ] Python-dotenv could not parse statement starting at line 1
2022-03-05T18:37:40.180748Z [warning  ] Python-dotenv could not parse statement starting at line 1
2022-03-05T18:37:40.180978Z [warning  ] Python-dotenv could not parse statement starting at line 1
2022-03-05T18:37:40.182940Z [warning  ] No state was found, complete import.
2022-03-05T18:37:40.183213Z [warning  ] Python-dotenv could not parse statement starting at line 1
2022-03-05T18:37:40.261995Z [info     ] ELT could not be completed: Cannot start extractor: Catalog discovery failed: command ['/meltano-stonks-ib/.meltano/extractors/tap-postgres/venv/bin/tap-postgres', '--config', '/meltano-stonks-ib/.meltano/run/elt/postgres-to-parquet/3977c95d-6ac7-48a7-ba24-92ceb1e29169/tap.ed6a0b5b-7ef8-4ffd-869a-cbe3a826b0f8.config.json', '--discover'] returned 1. cmd_type=elt job_id=postgres-to-parquet name=meltano run_id=3977c95d-6ac7-48a7-ba24-92ceb1e29169 stdio=stderr
2022-03-05T18:37:40.262250Z [info     ] For more detailed log messages re-run the command using 'meltano --log-level=debug ...' CLI flag. cmd_type=elt job_id=postgres-to-parquet name=meltano run_id=3977c95d-6ac7-48a7-ba24-92ceb1e29169 stdio=stderr
2022-03-05T18:37:40.262396Z [info     ] Note that you can also check the generated log file at '/meltano-stonks-ib/.meltano/logs/elt/postgres-to-parquet/3977c95d-6ac7-48a7-ba24-92ceb1e29169/elt.log'. cmd_type=elt job_id=postgres-to-parquet name=meltano run_id=3977c95d-6ac7-48a7-ba24-92ceb1e29169 stdio=stderr
2022-03-05T18:37:40.262526Z [info     ] For more information on debugging and logging: <https://docs.meltano.com/reference/command-line-interface#debugging> cmd_type=elt job_id=postgres-to-parquet name=meltano run_id=3977c95d-6ac7-48a7-ba24-92ceb1e29169 stdio=stderr
ill check these log files..
never tapped postgresql before
I think it's stuck because the installer is hanging on
target-parquet
.. Ive tripple checked my setup and now it seems the last error says the loader is not installed.. and when I go to install it I just get it hanging again..
Copy code
pip install git+<https://github.com/estrategiahq/target-parquet.git>
seemed to reveal more of the issue
I'll post on their GitHub when I can..
after much digging.. it turns out numpy is not yet building on python 3.10 .. and so this target fails
hmm strange, I am seeing this on 3.8.10 too though
Can anyone get the target-partquet to install ? I now have gotten the pip based installation to work.. but everytime I try to the the
target-parquet
alias… nothing finishes and I am led to believe it’s failed but.. unsure where to check or fix it?
I am next going to try overloading the pip_url with the git+https
im on python 3.8.10 , kubuntu 22.04 LTS
okay I think I have traced the issue.. it appears the default namespace points to GITLAB. not the GITHUB end point.. and when I manually try to install that.. i get greeted with a username prompt… guessing the repo is accidently not set to public or ..?
when I tried visting the meltano based project URL, 404
t
Where did the Meltano link come from? We don’t have a
target-parquet
in that namespace. https://hub.meltano.com/targets/parquet
e
@taylor this is the URL popping out when I attempt to perform the meltano install of the target-parquet.. according to the instructions on the hub https://hub.meltano.com/targets/parquet Under the “usage with meltano” section .. at first you see a github link in the HTML.. that URL is valid.. but when you actually read the output of the meltano add command.. it points to this 404'd url, and the command just hangs and hangs
Copy code
meltano add --custom loader target-parquet
t
oh wow! I see what you’re saying. yeah that’s a big problem. I’ll make an issue right now.
e
thanks Taylor.. sorry I didn't start it myself but I thought I might be .. not getting something others knew
t
no worries! Thanks for brining it up as a UX concern 😄