https://linen.dev logo
Join Slack
Powered by
Hello everyone, sorry for the question. I am chec...
# announcements
c
Hello everyone, sorry for the question. I am checking the following link https://gitlab.com/meltano/meltano/-/blob/master/docs/src/tutorials/postgres-with-postgres.md But they don't explain how to do it "INCREMENTALLY", does anyone have an example?
d
@cristian Don't be sorry, asking questions is what this channel is for! Have you tried setting the 
default_replication_method
setting to 
INCREMENTAL
instead of 
FULL_TABLE
? I expect it to work correctly as long as you specify a 
--job_id
option on your 
meltano elt
call (see https://meltano.com/docs/integration.html#pipeline-state).
c
yes, I have changed "FULL_TABLE" to "INCREMENTAL"
message has been deleted
but when creating the connection it always gives me that error.
message has been deleted
d
@cristian Can you try running the tap using the CLI, so that we get the full output and any error messages?
Copy code
meltano elt tap-postgres target-postgres --job_id=pg-to-pg
In general, the CLI is recommended over the UI, especially while you're still getting started
c
But I have not configured the username and password of tap-postgres, as it gives me an error when doing the test. Should I run a command to save the postgres credentials?
d
You can either use 
meltano config tap-postgres set <setting> <value>
or directly add env vars to 
.env
as suggested under https://gitlab.com/meltano/meltano/-/blob/master/docs/src/tutorials/postgres-with-postgres.md#adding-a-custom-extractor, if you scroll down to 
.env
c
ahhh I understand, I will create the .env in the root of the project. Maybe the error is from the UI.
d
The error is probably real, but at least the CLI will actually tell you what's wrong 🙂
I'm off for the day, hasta mañana!
c
Hi, sorry for bothering asking the same thing. I'm running 
meltano elt tap-postgres target-postgres --job_id test_meltano
but it gives me the following error.
message has been deleted
I have made several changes in meltano.yml but with "INCREMENTAL" it fails, apparently it only works with "FULL_TABLE"? my file is now like this.
message has been deleted
d
@cristian That error indicates that in this line: https://github.com/singer-io/tap-postgres/blob/master/tap_postgres/sync_strategies/incremental.py#L53, 
md_map.get(('properties', replication_key))
is 
None
, meaning that no metadata exists for the configured 
replication_key
in the stream in question. A few lines up in the output, it says that the stream 
cars_dev-public-brands
has replication key 
None
, which is coming from https://github.com/singer-io/tap-postgres/blob/c8e5fc5a9619df59df769abc96fd28a88a17a250/tap_postgres/__init__.py#L458. Obviously, a 
None
replication key won't work, and this explains the error you're seeing. It looks like 
default_replication_key
is not actually used, and that a 
replication-key
metadata should really be set for each stream. You can set it for all streams in one by changing 
zone_class
under 
metadata
to 
'*'
, so that it will match all streams and set the metadata for all of them. Can you give that a try?
c
I can't understand what is happening, the table "cars_dev-public-brands" is not selected. The only table I have selected in the metadata is "zone_class"
message has been deleted
d
@cristian By default, all streams/tables are selected, unless you select only specific ones using 
meltano select
: https://meltano.com/docs/integration.html#selecting-entities-and-attributes-for-extraction
c
Thank you very much for your answer and sorry to answer so far, I was incapacitated by strong pain.
one last question and I don't bother anymore :). Is there any way to generalize the connection to the db? For example, for postgres it was necessary for me to create the variable 
"TAP_PG_DATABASE"
, I imagine that for postgres or oracle it will be necessary for me to create something like 
"TAP_MY_DATABASE"
. In my case I am doing a wizard for the company where I work, the idea is that only with the graphical interface they configure the source db and dentin to be synchronized by singer taps, they program a date and time to do this. But I see that it will be difficult since it is necessary to modify the 
meltano.yml
by hand or I do not know if there is any convention to standardize?
d
@cristian The specific environment variable to use will depend on the tap/target you're using, but they typically look like with 
TAP_<DB>_<SETTING>
. You can use 
meltano config <plugin> list
to list all settings and the env vars you can use. As for a graphical interface, did you see that Meltano has a UI as well with a configuration interface? https://meltano.com/docs/ui.html#extractor-configuration 🙂
7 Views
Open in Slack
PreviousNext
Powered by