Michal Ďurik
05/02/2024, 2:44 PMBuzzCutNorman
05/02/2024, 3:31 PMBuzzCutNorman
05/02/2024, 5:19 PMmeltano select tap-mssql --list
it doesn't show the other database tables but does give an error that the tables are not found in the catalog. Looking at the catalog it only shows tables located in the database given in the tap-mssql config. Since they are not present they can not be selected to extract. Even if I invoke the tap via meltano invoke tap-mssql
I don't see data come over from the other database tables.
Ok so how to deal with this. There are two options
1. Plugin inheritance and call meltano run
for each inherited tap linked to a specific database (this is what I do)
2. Create views to the tables in the other databases and select the views. (This one is dependent on how much control you have of the mssql server)
These are only two suggestion, there might be more ways to accomplish what you want. If anyone can think of other please add them.
cc @Edgar Ramírez (Arch.dev), @wesseljt, @Chris Jones
Here an example of the config I tried:
environments:
- name: dev
- name: staging
- name: prod
plugins:
extractors:
- name: tap-mssql
variant: buzzcutnorman
pip_url: git+<https://github.com/BuzzCutNorman/tap-mssql.git>
config:
dialect: mssql
driver_type: pyodbc
host: [your sql server here]
port: 1433
database: testdata
sqlalchemy_url_query:
driver: ODBC Driver 18 for SQL Server
TrustServerCertificate: yes
MultiSubnetFailover: yes
sqlalchemy_eng_params:
fast_executemany: 'true'
select:
- 'dbo-tags.*'
- 'AdventureWorks2019\\.dbo-ErrorLog.*'
- 'AdventureWorks2019-dbo-ErrorLog.*'
Here is the results that I got from running this config on a test server.
PS C:\development\projects\my-test-stuff> meltano select tap-mssql --list
2024-05-02T16:58:34.625856Z [info ] The default environment 'dev' will be ignored for `meltano select`. To configure a specific environment, please use the option `--environment=<environment name>`.
2024-05-02T16:58:36.094588Z [warning ] Stream `AdventureWorks2019\.dbo-ErrorLog` was not found in the catalog
2024-05-02T16:58:36.094588Z [warning ] Stream `AdventureWorks2019-dbo-ErrorLog` was not found in the catalog
Legend:
selected
excluded
automatic
Enabled patterns:
dbo-tags.*
AdventureWorks2019\\.dbo-ErrorLog.*
AdventureWorks2019-dbo-ErrorLog.*
Selected attributes:
[selected ] dbo-tags.Count
[selected ] dbo-tags.ExcerptPostId
[automatic] dbo-tags.Id
[selected ] dbo-tags.TagName
[selected ] dbo-tags.WikiPostId
wesseljt
05/02/2024, 7:20 PMEdgar Ramírez (Arch.dev)
05/02/2024, 7:26 PMBuzzCutNorman
05/02/2024, 7:27 PMMichal Ďurik
05/03/2024, 6:09 AM