Hallo, I have a question about the correct use of ...
# getting-started
m
Hallo, I have a question about the correct use of meltano tap extractor. I have an MSSQL Server as Source which contains multiple databases. Is it possible with one tap-mssql extractor variant buzzcutnorman to extract multiple databases at once? Or what is the recommended approach in this case? Thanks for answers.
b
I know that you can if you have read rights to multiple databases on a mssql server you can query a table in another database from the database you are connected. I didn't write anything into tap-mssql to accommodate this workflow and to be honest wasn't planning to. Have I tried this via tap-mssql no. Could it possibly work as a side effect of how the Singer-SDK and SQLAlchemy are written maybe? I am curious and will give it a try and let you know what I find.
🙌 1
I played around with this nothing exhaustive but enough to run into walls and road blocks. 💥 😵. I added some tables located in another database to tap-mssql select. When you ask the tap-mssql to show what is selected
meltano select tap-mssql --list
it doesn't show the other database tables but does give an error that the tables are not found in the catalog. Looking at the catalog it only shows tables located in the database given in the tap-mssql config. Since they are not present they can not be selected to extract. Even if I invoke the tap via
meltano invoke tap-mssql
I don't see data come over from the other database tables. Ok so how to deal with this. There are two options 1. Plugin inheritance and call
meltano run
for each inherited tap linked to a specific database (this is what I do) 2. Create views to the tables in the other databases and select the views. (This one is dependent on how much control you have of the mssql server) These are only two suggestion, there might be more ways to accomplish what you want. If anyone can think of other please add them. cc @Edgar Ramírez (Arch.dev), @wesseljt, @Chris Jones Here an example of the config I tried:
Copy code
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-mssql
    variant: buzzcutnorman
    pip_url: git+<https://github.com/BuzzCutNorman/tap-mssql.git>
    config:
      dialect: mssql
      driver_type: pyodbc
      host: [your sql server here]
      port: 1433
      database: testdata
      sqlalchemy_url_query:
        driver: ODBC Driver 18 for SQL Server
        TrustServerCertificate: yes
        MultiSubnetFailover: yes
      sqlalchemy_eng_params:
        fast_executemany: 'true'
    select:
    - 'dbo-tags.*'
    - 'AdventureWorks2019\\.dbo-ErrorLog.*'
    - 'AdventureWorks2019-dbo-ErrorLog.*'
Here is the results that I got from running this config on a test server.
Copy code
PS C:\development\projects\my-test-stuff> meltano select tap-mssql --list
2024-05-02T16:58:34.625856Z [info     ] The default environment 'dev' will be ignored for `meltano select`. To configure a specific environment, please use the option `--environment=<environment name>`.
2024-05-02T16:58:36.094588Z [warning  ] Stream `AdventureWorks2019\.dbo-ErrorLog` was not found in the catalog
2024-05-02T16:58:36.094588Z [warning  ] Stream `AdventureWorks2019-dbo-ErrorLog` was not found in the catalog
Legend:
        selected
        excluded
        automatic

Enabled patterns:
        dbo-tags.*
        AdventureWorks2019\\.dbo-ErrorLog.*
        AdventureWorks2019-dbo-ErrorLog.*

Selected attributes:
        [selected ] dbo-tags.Count
        [selected ] dbo-tags.ExcerptPostId
        [automatic] dbo-tags.Id
        [selected ] dbo-tags.TagName
        [selected ] dbo-tags.WikiPostId
👍 2
w
interesting - haven't run into this but interested in what you find
e
Yeah, inheritance is the way to go if the tap doesn't support extracting from multiple databases at once.
b
> if the tap doesn't support extracting from multiple databases at once. Is that a thing? pulling form multiple databases in one run. I am trying to just imagine what that would take. Unfortunately my imagination isn't the best 😀
m
Thank you for your answers and I will try what @BuzzCutNorman posted.
174 Views