https://meltano.com/ logo
Channels
Powered by
#cli
Title
# cli
a

agreeable-church-13271

04/07/2019, 3:12 AM
I tried adding the Singer postgres tap but got back a message that it isn't supported. I read the issue about [Singer Taps/Targets] (https://gitlab.com/meltano/meltano/issues/94) but wasn't clear about what to expect in terms of today's release.
e

echoing-needle-59281

04/09/2019, 3:18 PM
Hi Rob, Meltano has a list of curated taps/targets. If you want to use a plugin outside that list, you can use the 
meltano add --custom
command, as documented here: http://meltano.com/docs/plugins.html#add-the-plugin-to-your-meltano-project-custom
Feel free to ping me if you run into any problems
Something along the lines of
Copy code
$ meltano add --custom tap-postgres
> pip_url:  git+<https://github.com/singer-io/tap-postgres.git@v0.0.51>`
> executable: tap-postgres
You'll most probably have to add configuration options in the 
meltano.yml
manually though.
a

agreeable-church-13271

04/10/2019, 1:20 PM
Thanks! I'll give that a shot. Is there any means to save state for a tap? I'd like to do incremental updates so that's why I ask.
e

echoing-needle-59281

04/10/2019, 1:27 PM
Per default, Meltano store state in a database (SQLite per default) in a Job log. Each elt run can be named to be treated as an incremental run.
Using either 
meltano elt --job_id
or 
MELTANO_JOB_ID
env variable.
If your target outputs a state, it'll be re-hydrated in the next ELT run automatically as long as the job has the same job_id.
a

agreeable-church-13271

04/10/2019, 1:29 PM
I see. If I wanted to run the Docker instance of Meltano can I connect it to an external DB?
e

echoing-needle-59281

04/10/2019, 1:35 PM
Sure, the Meltano job log can be stored in a Postgres database too.
It is configurable using env variables:
Copy code
MELTANO_BACKEND=sqlite|postgres`
# these are the Postgres specific config
PG_ADDRESS=
PG_USERNAME=
PG_PASSWORD=
PG_DATABASE=
PG_PORT=
i

important-island-7384

04/10/2019, 3:44 PM
Thanks for the details. When using the tap how does one specify the config and catalog parameters?
e

echoing-needle-59281

04/10/2019, 3:50 PM
For the config section you must edit 
meltano.yml
and add whatever keys you need under the 
config:
section of a plugin: these will get injected into the tap when it runs.
i

important-island-7384

04/10/2019, 3:51 PM
Is there an example or docs you can point me to?
e

echoing-needle-59281

04/10/2019, 3:51 PM
For the catalog, it depends - Meltano has the 
meltano select
command to help you manipulate the catalog output via 
--discover
so you can use that to control what you want to be selected. You may find documentation about that here https://meltano.com/docs/meltano-cli.html#meltano-select
i

important-island-7384

04/10/2019, 3:54 PM
OK, thanks. Very helpful info. I appreciate your help.
e

echoing-needle-59281

04/10/2019, 3:55 PM
Hey I really appreciate you taking time to look at our stuff 😄
Now I have an itch to get 
tap-postgres
running.
i

important-island-7384

04/10/2019, 3:57 PM
Sure thing! I know the airflow support is recent but is there any info on how to configure it? Specifically, if I'm running Meltano in Docker can I persist airflow config to something like a database?
I'm not sure if the airflow integration is even available in the current Docker image so perhaps this is premature.
e

echoing-needle-59281

04/10/2019, 4:00 PM
Well airflow's database is independant of Meltano's (might change in the future, but for now it's isolated) So I would think you could have meltano run airflow scheduler and having Meltano images in docker running the ELT. Sadly right now the only dag we ship is a BashExecutor.
And yes, Airflow integration is <1 week out 😄
@important-island-7384 got it running, you can clone this and follow the readme, I hope that helps you. https://gitlab.com/mbergeron/meltano-tap-postgres
i

important-island-7384

04/10/2019, 4:32 PM
Nice! Much appreciated.
a

agreeable-church-13271

04/11/2019, 11:48 PM
OK, getting around to trying this out. I start by running 
docker-compose up warehouse_db
and 
meltano install
goes fine. However, 
meltano select tap-postgres --list
fails.
This seems notable: 
CRITICAL 0 tables were discovered across the entire cluster
I haven't modified any of the config files.
e

echoing-needle-59281

04/12/2019, 12:54 AM
You'll have to update your 
meltano.yml
to set the postgres instance you want to pull from.
a

agreeable-church-13271

04/12/2019, 2:50 AM
So replace the values here with my source db?
Copy code
plugins:
  extractors:
  - config:
      dbname: warehouse
      host: localhost
      password: warehouse
      port: 5502
      user: warehouse
      default_replication_method: FULL_TABLE
    executable: tap-postgres
    name: tap-postgres
    pip_url: git+<https://github.com/singer-io/tap-postgres.git@v0.0.51>
If so, changing the config params didn't change the output. However, do I need to run anything else once this change is made and prior to 
meltano select tap-postgres --list
?
Just for fun, I ran 
meltano init
again. It failed but it updated the file in 
.meltano/.../tap.config.json
which allowed the select command to succeed.
I wanted to use my target-postgres and wanted to use my own db but couldn't figure out how to generate the it so I edited the 
target.config.json
by hand.
However, not sure how to call 
meltano elt ...
with invoke.
e

echoing-needle-59281

04/12/2019, 12:28 PM
meltano invoke tap-postgres --properties .meltano/run/tap-postgres/tap.properties.json | meltano invoke target-sqlite
I'm using 
target-sqlite
here but you could use any other target in 
meltano discover loaders
Thing is, that will effectively extract the data, but you'll not have the 
elt
job log with the state management sadly.
tap-postgres
should be updated to support 
--catalog
instead of 
--properties
(deprecated)
It's not as seemless as I'd it to be, but that is why this tap is not included in the 
meltano discover
.
🙌 1
a

agreeable-church-13271

04/12/2019, 1:43 PM
Regarding the properties vs. catalog issue, I think this is a legacy thing with Singer so it might make sense to effectively use both. That is, call the tap with both 
--catalog
and 
--properties
. I've seen this done with another tool to deal with this very issue.
However, I'll open a ticket with the tap-postgres project to switch to properties.
Thanks for your pointer on doing the full elt cycle. I'd like to do transform as well as part of the process. Is it possible to include that as well?
e

echoing-needle-59281

04/12/2019, 2:02 PM
Regarding the properties vs. catalog issue, I think this is a legacy thing with Singer so it might make sense to effectively use both. That is, call the tap with both 
--catalog
and 
--properties
. I've seen this done with another tool to deal with this very issue.
IIRC I got a tap crashing when I sent 
--catalog
and it wasn't supported. I'll retry that.
a

agreeable-church-13271

04/12/2019, 2:07 PM
OK, thanks. This works for me when I run it from the command line: 
tap-postgres --config tap-postgres-config.json --catalog local-properties.json --properties local-properties.json
FWIW
e

echoing-needle-59281

04/12/2019, 2:09 PM
Thanks for your pointer on doing the full elt cycle. I'd like to do transform as well as part of the process. Is it possible to include that as well?
You'll have to add custom dbt transforms. I suggest you run 
meltano add transformer dbt
first to setup your meltano dbt project inside 
transform/
The main problem we are facing with tap is that they are not created equal, so it might be another tap failing on that. I could add it as an extra config but I try to not have to do that. We'll see.
i

important-island-7384

04/12/2019, 4:05 PM
That makes sense.
On the point about the transform step, I meant that normally you can indicate the transform should be run as part of the 
etl
command. However, in the case of the approach you mentioned (
meltano invoke tap-postgres --properties .meltano/run/tap-postgres/tap.properties.json | meltano invoke target-sqlite
), can I specify it should be performed?
e

echoing-needle-59281

04/12/2019, 5:26 PM
I would run 
meltano elt tap-postgres target-sqlite --transform only
i

important-island-7384

04/12/2019, 7:50 PM
👍
a

agreeable-church-13271

04/18/2019, 7:26 PM
Hi, I'm using tap-postgres and have the LOG_BASED replication enabled. I've set a MELTANO_JOB_ID value and see that it found state from the last run. However, when I look at the database column payload, it looks like the state is just 
{"singer_state": {}}
and the tap behaves as if there's no state.
I can open a ticket on this but wanted to confirm I'm not missing something first.
e

echoing-needle-59281

04/18/2019, 7:35 PM
@important-island-7384 please open a ticket, which target are you using?
a

agreeable-church-13271

04/18/2019, 7:37 PM
target-postgres
I'll open a ticket.
e

echoing-needle-59281

04/18/2019, 7:37 PM
Thanks!
a

agreeable-church-13271

04/18/2019, 7:38 PM
Sure thing. BTW, I created a PR against the meltano target-postgres repo.
e

echoing-needle-59281

04/18/2019, 7:38 PM
Good will check it out.
a

agreeable-church-13271

04/18/2019, 7:38 PM
I ran into an issue with validation with some data we have.
This addresses it.
e

echoing-needle-59281

04/18/2019, 7:46 PM
I responded.
a

agreeable-church-13271

04/18/2019, 7:57 PM
Thanks!
👍 1
Powered by