https://linen.dev logo
Join Slack
Powered by
Hi, I'm trying to use the tap-mongodb with Meltano...
# troubleshooting
n
Hi, I'm trying to use the tap-mongodb with Meltano but don't seem to be able to get past first-base. Platform: Windows 11, Python 3.8, Pycharm with venv configured. I have installed Meltano using pip install meltano followed by meltano add extractor tap-mongodb. I have configured a local MongoDB and added a user with readWriteAnyDatabase permissions and I can successfully connect with that user via Studio 3T and read all the databases so I know that side is working. I have created a vanilla Meltano project using Python 3.8 in PyCharm Pro and installed tap-mongodb. I have set up the simplest meltano.yml file as follows:
Copy code
version: 1
default_environment: dev
project_id: e5ad94c4-fc61-47ae-96b0-9ce655d945df
plugins:
  extractors:
  - name: tap-mongodb
    variant: singer-io
    pip_url: tap-mongodb
    config:
      host: localhost/admin
      port: 27017
      user: "data-reader"
      database: myDatabase
      password: "abc123"
environments:
- name: dev
When I try and run any Meltano commands against the tap, I just get time out errors. E.g. If I run 
meltano invoke tap-mongodb --discover
I get:
Copy code
raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: localhost/admin:27017: [Errno 11001] getaddrinfo failed
Also if I run 
meltano ui
I get the following error and the terminal hangs:
Copy code
File "C:\Users\nigel\PycharmProjects\MeltanoPOC\venv\lib\site-packages\gunicorn\util.py", line 8, in <module>
    import fcntl
I've spent half a day on this now and am beginning to wonder whether I have fundamentally misunderstood something or if it just doesn't work?
p
The 
localhost/admin:27017
part looks suspicious to me. Are you sure the host of 
localhost/admin
is correct? Having a forward slash inside a hostname doesn’t seem correct to me. Are you able to connect to mongodb via a different method? If so, what does your config look like for that?
b
Sorry to say but 
meltano ui
currently does not work on Windows. Here is a link to the issue about it https://gitlab.com/meltano/meltano/-/issues/3383
n
Thanks @BuzzCutNorman at least that problem is solved (known).
@ptd that's pretty standard with MongoDB - it's the auth database, without it I get an immediate 
CRITICAL Authentication failed.
error.
v
@nigel_ainscoe 
CRITICAL Authentication failed.
sounds better than 
pymongo.errors.ServerSelectionTimeoutError:
auth failure seems like creds / access
n
creds are the same in Studio 3T and Meltano
v
Have you tried anything else around the credentials side?
n
Such as?
I've tried a couple of different user accounts with different creds from admin to simple reader
You might have something though, just trying connecting from the command line with those creds and getting the same errors typod the password at command line. Worked fine when I fixed that
v
Getting somewhere! I don't have experiance with mongo so I'm not super helpful but have you followed these steps https://github.com/singer-io/tap-mongodb/blob/master/spikes/local_mongo_setup.md ?
n
Yep. And 
mongo --host localhost -u data-reader -p abc123
from the command line connects fine.
v
n
TBC I can connect from command line using 
mongo --host localhost --port 27017 --authenticationDatabase admin --username data-reader --password abc123
v
Copy code
database: myDatabase
Doesn't match admin? 🤷
n
JFC - Nowhere in the docs does it even suggest that the database is the auth database - does it....? 👀
Awesome I now have:
Copy code
{
  "streams": [
    {
      "table_name": "myCollection",
      "stream": "myCollection",
      "metadata": [
        {
          "breadcrumb": [],
          "metadata": {
            "table-key-properties": [
              "_id"
            ],
            "database-name": "myDatabase",
            "row-count": 1,
            "is-view": false,
            "valid-replication-keys": [
              "_id"
            ]
          }
        }
      ],

Loads more good database stuff.....

      "schema": {
        "type": "object"
      }
    }
  ]
}
Thanks for the help 🙂
Doh: • 
Name: database
• 
Environment variable: TAP_MONGODB_DATABASE
This is the database used for authentication, not the database used for extraction.
v
Hindsight is 20/20 glad you came to ask for help and we got you running. Welcome 😄
19 Views
Open in Slack
PreviousNext
Powered by