$ meltano add orchestrator airflow
Orchestrator 'airflow' variant 'original' is not known to Meltano. Variants: ['apache (default)']

Doc source : https://docs.meltano.com/getting-started#schedule-pipelines-to-run-regularly

extract from my projects

meltano.yml

orchestrators:
  - name: airflow
    pip_url: apache-airflow==2.1.0 --constraint <https://raw.githubusercontent.com/apache/airflow/constraints-2.1.0/constraints-3.7.txt>

perhaps cleanest thing is to just start with a fresh project and diff the two?

i'll keep googling if there's like a "Upgrade from 1.x to 2.x release notes"

https://docs.meltano.com/guide/v2-migration ah hah! okay I am cracking this open then....

hmmmm , I seem to be stuck due to permissions? this is all occurring in a venv already so.. unsure what more I need to do

$ meltano add transformer dbt-postgres
Added transformer 'dbt-postgres' to your Meltano project
Variant:        dbt-labs (default)
Repository:     <https://github.com/dbt-labs/dbt-core>
Documentation:  <https://docs.meltano.com/guide/transformation>

Adding required file bundle 'files-dbt-postgres' to your Meltano project...
Variant:        meltano (default)
Repository:     <https://github.com/meltano/files-dbt-postgres>
Documentation:  <https://hub.meltano.com/files/files-dbt-postgres>

Installing transformer 'dbt-postgres'...
Installing file bundle 'files-dbt-postgres'...
Transformer 'dbt-postgres' could not be installed: could not create the virtualenv for 'transformers/dbt-postgres'

File bundle 'files-dbt-postgres' could not be installed: could not create the virtualenv for 'files/files-dbt-postgres'

Installed 0/2 plugins
Failed to install plugin(s)

I thought a

chown -R ....

would sort it but, apparently not

Or sometimes when I don't want to even second guess it I just

rm -rf .meltano

then reinstall

is my python too new, let me go check .. I am on 3.10

ahh good point on it being the taps/targets perhaps

my guess is.. DBT is out.. now there are dbt variants so that is failing for that reason.. airflow likely a similar story how'd you turn on debug? I did

meltano install --debug --verbose

but both those options are not valid

feels cleaner if I just start a blank project and reinstall piece-wise

I gotta say, looking over the documentation site.. it feels there is this missing gap between installation.. and getting a project started.. almost all I can find is guides for once you're up and running with all the plugins you want installed.. I will need to re-google to find the "my first tap-target" flows I used in the past

Citation : https://docs.meltano.com/guide/installation - I get to install and then... that's it.. would be good to nudge me next to getting things going further.. maybe I am blind here. a separate page works to start me off : https://meltano.com/documentation/getting-started/ ah and also seeing https://docs.meltano.com/getting-started#create-your-meltano-project

so I took a chance with a blank project .. and getting similar errors

(settings) [[]]: key:value
'value' is not a valid SettingKind
(settings) [[]]: key:string
Added extractor 'tap-ibkr' to your Meltano project

Installing extractor 'tap-ibkr'...
Extractor 'tap-ibkr' could not be installed: could not create the virtualenv for 'extractors/tap-ibkr'

Failed to install plugin(s)

I am going to take a guess that perhaps pipx went from a suggestion to a requirement?

im gonna google the troubleshooting page

I guess I had this problem a while back https://github.com/meltano/meltano/issues/2862 my prior workaround was to use venvs... but now that doesn't even work

I guess I have to give up and use pipx (or at least try that and report back)

ahhhh , I think I have found the problem

python3 -m venv .venv
The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

    apt install python3.10-venv

You may need to use sudo with that command.  After installing the python3-venv
package, recreate your virtual environment.

need to update my docs with this

i was using venvs built via pycharm and .. hence not seeing the core error

# okay, python virtualenv installed but now when I

meltano install

it hangs on the post

Installed transfer dbt

.. I am going to wipe everything and attempt to upgrade first, THEN install

dang no luck (I upgraded and that went okay, but it still hangs on install in the same place).. okay this is a DBT thing based on the migration docs that I need to solve ..

I think it was compounded I should say by the venv thing Hope it helps!

well... despite all the

meltano upgrade files

and

meltano upgrade

.. and

meltano remove transformer dbt

after getting the new postgres specific variant... I still hang on the very end of

meltano install

... I'll let it run but.. i would guess somethings a bit off with my migration and it might be best to just.. re-install a project from scratch

Is dbt-postgres even on there 🤔

yes, I had added it

one line I see in the migration.. Removing a file bundle does not remove any files from your

transform/

directory. Manually remove

transform/profiles.yml

is that path correct? I do not have

transform/profiles.yml

.. but i DO have

transform/profile/profiles.yml

I am sure it's just some legacy stuff, or maybe my custom tap

thanks Alex

I've filed a issue on the github, I think the issue is actually of all things airflow ( well .. not airflow but airflow+py3.10) https://github.com/meltano/meltano/issues/6536

okay I went back to my OG project.. modified the path to apache airflow and... we're installed!

$ meltano install
Installing 7 plugins...
Installing extractor 'tap-ibkr-tickers'...
Skipped installing extractor 'tap-ibkr-tickers'...
Installing extractor 'tap-ibkr-news'...
Skipped installing extractor 'tap-ibkr-news'...
Installing loader 'target-postgres'...
Installing loader 'target-jsonl-blob'...
Skipped installing loader 'target-jsonl-blob'...
Installing orchestrator 'airflow'...
Installing transformer 'dbt-postgres'...
Installing file bundle 'airflow'...
Installed file bundle 'airflow'
Run `meltano upgrade files` to update your project's 'airflow' files.
Installed loader 'target-postgres'
Installed transformer 'dbt-postgres'
Installed orchestrator 'airflow'
Installed 4/7 plugins
Skipped installing 3/7 plugins

Thank you @visch and @alexander_butler again

I should have said more explicitly I was trying to fix one issue, not the whole thing ha. I'll put that in my head for next time!