Second I attempted to use the target postgres but I am unabl Meltano #announcements

Second: I attempted to use the target-postgres, bu...

adam_roderick

07/09/2019, 2:11 PM

Second: I attempted to use the target-postgres, but I am unable to configure the parameters. When I attempt to update host, port, etc from the UI, the logs show messages like

Setting port is currently set via $PG_PORT.

So I set the environment variables and ran

meltano ui

again, but the desired values do not show up in the UI. Am I missing something?

adam_roderick

07/09/2019, 2:32 PM

I confirmed the target-csv settings are saved through the UI. But target-sqlite setting (db name) is not. The log shows the message

Setting database is currently set via $SQLITE_DATABASE

adam_roderick

07/09/2019, 2:36 PM

I changed the value of the "database" property from "meltano" to something else in the file

.meltano/loaders/target-sqlite/target.config.json

. Ran the UI again, but the "database" configuration value still shows "meltano" instead of the new value.

adam_roderick

07/09/2019, 2:37 PM

Ran

meltano elt tap-carbon-intensity target-sqlite

, but received the same error as before. So it appears that the config file is not utilized

gert_jan

07/10/2019, 7:05 AM

Hi Adam I had the same problem, the settings from the UI are not saved into the .env. It seems they are using a different storage for the UI. However when I changed the settings for postgres in the .env I could use the DB correctly. So I guess you can move forward with your project. I'm sure they will fix this in the next release!

yannis_roussos

07/10/2019, 10:10 AM

This seems to be a bug on the Meltano UI @derek_knox @micael_bergeron

yannis_roussos

07/10/2019, 10:16 AM

@adam_roderick when you say that you

received the same error as before

, do you mean the

$SQLITE_DATABASE

one?

yannis_roussos

07/10/2019, 10:23 AM

As far as I can see, there are 2 different issues that @adam_roderick is facing: + The Sqlite DB we are using is locked --> Depending on whether it is locked from the UI or from a 3rd process (airflow) that could turn Meltano UI and even Meltano in general unusable for Adam. + The settings are not properly set in Meltano UI --> Until this one is fixed, you should either wait or use the cli:

Copy code

meltano init my-test-project
cd my-test-project
source .env
meltano add extractor tap-carbon-intensity
meltano add loader target-sqlite

meltano elt tap-carbon-intensity target-sqlite

Where the .env is the default .env built by Meltano when you start a new project:

Copy code

export FLASK_ENV=development
  export SQLITE_DATABASE=meltano
  export PG_PASSWORD=warehouse
  export PG_USERNAME=warehouse
  export PG_ADDRESS=localhost
  export PG_SCHEMA=meltano
  export PG_PORT=5502
  export PG_DATABASE=warehouse

yannis_roussos

07/10/2019, 10:26 AM

The

source .env

part makes sure that

$SQLITE_DATABASE

is loaded to the environment before you run

meltano elt

and you should not get an error about

$SQLITE_DATABASE

. You can also change it in the .env to whatever you want. Also, if @micael_bergeron or @derek_knox have not made any change during the past days that I missed, running:

Copy code

source .env
meltano ui

Should make sure that all the aforementioned env vars are available to Meltano UI. (they should be auto-loaded but maybe there is some other issue with Windows)

derek_knox

07/10/2019, 12:10 PM

Thank you @adam_roderick and @gert_jan for sharing your issues.

Hi Adam I had the same problem, the settings from the UI are not saved into the .env. It seems they are using a different storage for the UI.

There is an order of precedence, so you are indeed correct @gert_jan. This landed a week or two ago and there was a related docs update https://www.meltano.com/docs/tutorial.html#plugin-setting. Does this help add any clarity?

adam_roderick

07/10/2019, 12:26 PM

Thanks for all the info, I will try updating settings through the UI. @yannis_roussos yes, by "same error as before" I am referring to the locked sqlite db file and resulting errro.

adam_roderick

07/10/2019, 12:27 PM

Correct, the first issue I encountered was the sqlite db file locking. I am running on Ubuntu via the Windows Linux Subsystem

adam_roderick

07/10/2019, 12:27 PM

The second issue was the saving via the UI. Where would be best for me to record this as a bug?

micael_bergeron

07/10/2019, 5:54 PM

@adam_roderick you can workaround the database locking using the SQLite WAL, see https://www.sqlite.org/wal.html. To enable it, use

sqlite3 <dbname> "PRAGMA journal_mode=WAL"

micael_bergeron

07/10/2019, 5:55 PM

I'll open an issue on target-sqlite to enable this per default.

micael_bergeron

07/10/2019, 6:17 PM

Here's the MR that will fix it.

micael_bergeron

07/10/2019, 6:17 PM

https://gitlab.com/meltano/target-sqlite/merge_requests/4

micael_bergeron

07/10/2019, 6:19 PM

Once that's merged, I suggest you do a

meltano install

to update

target-sqlite

. The next

elt

run should enable the WAL.

adam_roderick

07/11/2019, 1:51 PM

I see that the merge happened. I ran

sqlite3 meltano.db "PRAGMA journal_mode=WAL"

as well as ran

meltano install

adam_roderick

07/11/2019, 1:52 PM

Not sure how to confirm.

meltano --version

reports 0.30.0

adam_roderick

07/11/2019, 1:52 PM

But the issue persists

adam_roderick

07/11/2019, 1:52 PM

Anything I can do to troubleshoot? I can pass info here

micael_bergeron

07/11/2019, 6:59 PM

Okay, you are still getting database locks?

10 Views

Open in Slack

Previous Next