https://linen.dev logo
Join Slack
Powered by
Good morning, Meltano community :sunglasses: my pi...
# plugins-general
r
Good morning, Meltano community 😎 my pipeline using 
target-postgres
is letting me know that it succeeded but when I check the database, I do not see any data. I'm running my target Postgres database in a Docker container as well as my Meltano pipeline, so I'm wondering if it's some networking issue. However, it's weird that it is telling me it succeeded. Has anyone seen this before?
t
I haven’t directly used 
target-postgres
but my first guess would be a permissions issue on the database. Are you logging in with the user used to load the data or a separate use? Also, any logs you can share? running with 
meltano --log-level=debug
may be useful too
r
Copy code
tap-spreadsheets-anywhere       | INFO Wrote 22 records for stream "customer_discount".
target-postgres                 | time=2021-03-17 13:16:26 name=target_postgres level=INFO message=Loading 22 rows into 'excel_source_files_meltano."customer_discount"'
target-postgres                 | time=2021-03-17 13:16:26 name=target_postgres level=INFO message=Loading into excel_source_files_meltano."customer_discount": {"inserts": 0, "updates": 22, "size_bytes": 4014}
target-postgres (out)           | {"customer_discount": {"modified_since": "2021-03-17T12:56:37.050000+00:00"}}
I'm logging in with the superuser. This is just a test instance on my local Docker setup to test out pipelines before we push out to our GCP environment
Copy code
version: "3.8"
services:
  bw-meltano-ui:
    build: .
    command: ui
    depends_on: 
      - bw-meltano-db
      - bw-meltano-target-db
    env_file: 
      - ./config/meltano-ui.env.dev
    # environment:
      # - MELTANO_PROJECT_READONLY: 1
    ports:
      - target: 5000
        published: 5000
    networks:
      - bw-meltano-net
    volumes:
      - ./meltano.yml:/project/meltano.yml
      - ./extract/:/project/extract/
      - ./load/:/project/load/
      - ./model/:/project/model/
      - ./orchestrate/:/project/orchestrate/
      - ./transform/:/project/transform/
  bw-meltano-target-db:
    image: "postgres"
    env_file: 
      - ./config/meltano-target-db.env.dev
    networks:
      bw-meltano-net:
    ports:
      - target: 5432
        published: 5433
  bw-meltano-db:
    image: "postgres"
    env_file: 
      - ./config/meltano-db.env.dev
    networks:
      bw-meltano-net:
    ports:
      - target: 5432
        published: 5432

networks:
  bw-meltano-net:
Here is our 
docker-compose.yml
file. So basically I'm trying to create an environment that we can test out our pipelines by just running 
docker-compose up
It gets a little fuzzy because meltano can connect to the Meltano DB running in a container, but it doesn't seem to be writing to the postgres target DB.
t
does 
meltano config target-postgres
return what you expect?
I may have to call in backup on this one 😅
r
I'm wondering if it is when I am trying to access the target DB on my host machine (i.e. some networking misunderstanding on my end)
d
@ricky_renner So target-postgres is set up to point at that 
bw-meltano-target-db
? How are you accessing it from your host machine?
r
@douwe_maan it's binded to port 5433, so I access it just through 
localhost:5433
d
Ah, I see. And in Meltano, you're setting the target-postgres host to 
bw-meltano-target-db
and port to 
5432
? (Since internal networking should use the regular port, not the published one)
r
Yep it gets weird lol @douwe_maan
d
@ricky_renner If the target isn't complaining, the data is successfully getting loaded, so my hunch is that you're looking in the wrong DB schema, or that you're somehow not connecting with the right database...
r
That's what I'm thinking too... but I don't see anything in the DB I am connecting to over the 5433 port on my local machine via pgAdmin. So I'm not sure what it would be writing to and what would be on that port if not the same database
@douwe_maan just following up on this... any ideas? Any chance you're free for screenshare tomorrow?
Just following up on this! Hopefully somebody would be able to help out
d
@ricky_renner Sorry, I've been off Slack most of last week. Did you figure this one out? cc @aaronsteers
r
No worries @douwe_maan . I was not able to figure this out either. Really strange because I can connect to the DB that I am spinning up in a container via pgAdmin, and the pipeline with 
target-postgres
is running successfully, but when it finishes, there is still nothing in the database. I'm assuming I'm just targeting the wrong postgres database, but I'll have to go back and confirm that
d
Very weird. It doesn't sounds like a Meltano or target-postgres specific issue though, so I suggest debugging the Docker side of things some more!
a
Agreed. @ricky_renner - I’m have some time this afternoon if we want to do another quick screenshare. I agree with Douwe though that it sounds like something to do with the postgres connection string or an issue in docker. Any change the meltano sys-db and target-db creds or endpoints are getting swapped somewhere?
r
Yeah that's what I figured. I took a break from this to get our dbt pipeline up and running, so will have to transition back
Open in Slack
PreviousNext
Powered by