Hi team, I am currently testing Meltano as an option for our EL(T) - from Mongo into Snowflake. I was able to test it out locally and things looked fine. Next steps were to add meltano to our k8s cluster using helm and argocd. I am using this Repository of Helm Charts for use with Meltano from: https://github.com/meltano/helm-meltano for the same. What I have done so far: •

cd <path-to-meltano-folder-on-our-k8s-infra-repo>

•

helm repo add meltano <https://meltano.gitlab.io/infra/helm-meltano/meltano-ui>

•

helm search repo meltano-ui

◦ This returns: ▪︎

meltano/meltano-ui   0.3.0   1.96.0   Meltano UI webservice

• So, I do a

helm pull meltano/meltano-ui

followed by

tar -xvzf meltano-ui-0.3.0.tgz

(and delete the tar file) • After this I add the meltano app to argocd and finally run

kubectl apply -f meltano.yml

After pushing the changes, I see my pod in argocd but with an error that says:

`meltano ui` must be run inside a Meltano project.
Use `meltano init <project_directory>` to create one.

So, I went into the deployment.yaml file and changed:

command: ["meltano", "ui"]

to

command: ["/bin/sh","-c"]
args:
  - meltano init mongo-snowflake-sync;
    cd mongo-snowflake-sync;
    echo pwd;
    pwd;
    find .;

and also updated the mount path to:

mountPath: "/project/mongo-snowflake-sync/.env"

because “pwd” said the my current directory was

/project/mongo-snowflake-sync/

Also, under the values.yaml file, I updated the database URI like so:

database_uri: "sqlite:///project/mongo-snowflake-sync/.meltano/meltano.db"

Now, I see this error:

Could not connect to the database after 3 attempts. Max retries exceeded.
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.
(sqlite3.OperationalError) unable to open database file
(Background on this error at: <https://sqlalche.me/e/14/e3q8>)

OR

(sqlite3.OperationalError) unable to open database file
(Background on this error at: <https://sqlalche.me/e/14/e3q8>)

I tried multiple different options for the database URI and mount path with slightly varying results. For instance, I tried adding a “find .” to the commands to check all the files that were created in the meltano init step and while sometimes I could see the meltano.db most other times I did not(this might be a k8s thing though because I cannot reproduce that scenario where the meltano.db was getting listed in the find command 🤷). However, in all cases, I did still get the same error as above i.e. Could not connect to the database after 3 attempts. Max retries exceeded. I feel pretty much stuck at this point and out of ideas. Would appreciate any help/suggestions! TIA 🙏

Not certain about everything, but I know that version

1.96.0

is old, and I know the database connect issue was fixed after that version, so if you could upgrade meltano maybe you'd be off to the races?

CC @ken_payne

@visch That is interesting to know but I am not sure how to add a version to this repo. When I run:

helm search repo meltano-ui

(or meltano), I only get this one version:

NAME                    CHART VERSION   APP VERSION     DESCRIPTION

meltano/meltano-ui      0.3.0           1.96.0          Meltano UI webservice

Seems like this version needs to be updated: https://github.com/meltano/helm-meltano/blob/704ad4bb0adbd1c2b3f34f1e215c96571133cdf5/meltano-ui/Chart.yaml#L24 ?

Possibly, but the helm chart I would imagine* should add a DB (that's not sqllite) to the project so it could be something else too @Will Da Silva (Arch)’s tag is right here

Thanks for checking this out @abhinav_prakash 🙏 Both the Meltano image to use and the database URI are configurable, via the normal

values.yaml

route for Helm. Here are the defaults, which is where the SQLite URI is coming from. In the squared project (our internal deployment of Mletano) we substitute the docker `repository`/`tag` for custom values with our project installed in it and the

database_uri

for a running RDS Postgres instance (different for prod/staging) managed by Terraform. You can do the same with your project 🙂

@ken_payne thanks for the response thankyou Couldn't get to test it out as yet, driving for most of today. But I'll try to implement what you suggested and reach out in case I get stuck.

@abhinav_prakash The double quotes around

_ID

seem fine: https://docs.snowflake.com/en/sql-reference/identifiers-syntax.html#label-delimited-identifier. But you’re saying

CREATE TABLE IF NOT EXISTS MELTANO.TABLENAME ("_ID" varchar)

fails with that same “invalid identifier” error?

Yea that is what I was wondering and found the github link where you mentioned about explicitly specifying each schema in the meltano.yml file. I am about to test that now.