https://linen.dev logo
Join Slack
Powered by
If I want to configure multiple databases for tap-...
# singer-taps
d
If I want to configure multiple databases for tap-mssql, do I have to create multiple meltano projects, or is there a way to specify a different name for the extractor? It didn't seem to like it when I changed the value of 
- name
✅ 1
v
d
Hmm, it does sound like it. I tried it just now and ran 
meltano config tap-mssql--test list
but it still says the plugin is not known to meltano.
Copy code
plugins:
  extractors:
  - name: tap-mssql--test
    inherit_from: tap-mssql
  - name: tap-mssql
    variant: wintersrd
    pip_url: tap-mssql
v
Can you paste the entire error message
d
Plugin 'tap-mssql--test' is not known to Meltano
2024-08-20T131515.505714Z [info ] The default environment 'dev' will be ignored for 
meltano config
. To configure a specific environment, please use the option 
--environment=<environment name>
. Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to join our friendly Slack community.
I guess there's that, but it's just generic
v
which meltano version are you on? When I just tested your code here's the error I get
Copy code
Extractor 'tap-mssql' is not known to Meltano. Try running `meltano lock --update --all` to ensure your plugins are up to date.
If I follow the steps we're good
d
meltano, version 3.5.0
v
can you share your whole meltano.yml? The error message for me was the lock file, I ran that and it all works
d
Let me try the lock step
I didn't get that in my error message, which version are you on?
v
Copy code
$ meltano --version
meltano, version 3.5.0
can you share your meltano.yml ?
d
One min, sanitizing it
➕ 1
Copy code
version: 1
default_environment: dev
project_id: 5eb30a8f-fdfa-4025-9543-597f53c7503e
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-mssql
    variant: wintersrd
    pip_url: tap-mssql
    config:
      cursor_array_size: 10000
      use_date_datatype: true
      use_singer_decimal: true
    select:
    - dim-Test.*
    - '!*.*%*' # Percent sign is not valid in column name
    metadata:
      dim-*:
        replication-method: FULL_TABLE
  - name: tap-mssql--test
    inherit_from: tap-mssql
  loaders:
    - name: target-snowflake
      variant: meltanolabs
      pip_url: meltanolabs-target-snowflake
      config:
        batch_size_rows: 10000
        load_method: upsert
        validate_records: true
👀 1
v
You changed your meltano.yml file so I had to restart this 😕
Copy code
$ meltano init daniel_luo
$ cd daniel_luo
$ cat meltano.yml # matches yours
$ meltano config tap-mssql--test list
2024-08-20T13:27:32.510057Z [info     ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
Extractor 'tap-mssql' is not known to Meltano. Try running `meltano lock --update --all` to ensure your plugins are up to date.
$ meltano lock --update --all
Locking 3 plugin(s)...
Locked definition for extractor tap-mssql
Extractor tap-mssql--test is an inherited plugin
Locked definition for loader target-snowflake
$ meltano config tap-mssql--test list
2024-08-20T13:29:09.716843Z [info     ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
characterset [env: TAP_MSSQL__TEST_CHARACTERSET] current value: None (default)
        Characterset: The characterset for the database / source system. The default is utf8, however older databases might use a charactersets like cp1252 for the encoding.
If you have errors with a UnicodeDecodeError: 'utf-8' codec can't decode byte .... then a solution is to examine the characterset of the source database / system
and make an appropriate substitution for utf8 like cp1252.

cursor_array_size [env: TAP_MSSQL__TEST_CURSOR_ARRAY_SIZE] (default: 1) current value: 10000 (inherited from 'tap-mssql')
        Cursor Array Size: This allows fetching records in batches speeding up network traffic.
Recommend a setting of `10000` records.

database [env: TAP_MSSQL__TEST_DATABASE] current value: None (default)
        Database: Your MSSQL database name.
filter_dbs [env: TAP_MSSQL__TEST_FILTER_DBS] current value: None (default)
        Filter Databases: A comma separated list of databases to filter for.
host [env: TAP_MSSQL__TEST_HOST] current value: None (default)
        Host: Your MSSQL server host.
password [env: TAP_MSSQL__TEST_PASSWORD] current value: None (default)
        Password: Your MSSQL password.
port [env: TAP_MSSQL__TEST_PORT] current value: None (default)
        Port: The MSSQL port, defaulted to 1433.
tds_version [env: TAP_MSSQL__TEST_TDS_VERSION] current value: None (default)
        TDS Version: The TDS version of the connection. Defaults to 7.3, connecting to older
databases (ex MS SQL 2005) requires 7.0 or earlier. Please see the pymssql or FreeTSD documentation for more details.
Also note, while this is a confusing topic version 7.3 is more current than 8.0.

use_date_datatype [env: TAP_MSSQL__TEST_USE_DATE_DATATYPE] (default: False) current value: True (inherited from 'tap-mssql')
        Use Date Datatype: To emit a date datatype as a date without a time component or time without an UTC offset.
If this boolean config item is not set, the default behaviour is false i.e. emit date datatypes as a datetime.
It is recommended to set this on if you have time datatypes and are having issues uploading into into a target database.

use_singer_decimal [env: TAP_MSSQL__TEST_USE_SINGER_DECIMAL] current value: True (inherited from 'tap-mssql')
        Use Singer Decimal: The ability to output decimal, float, and integer data as strings to avoid loss of precision.

user [env: TAP_MSSQL__TEST_USER] current value: None (default)
        User: Your MSSQL username.

Custom extras, plugin-specific options handled by Meltano:
_metadata.dim-*.replication-method [env: TAP_MSSQL__TEST__METADATA_DIM___REPLICATION_METHOD] current value: 'FULL_TABLE' (inherited from 'tap-mssql')

To learn more about extractor 'tap-mssql--test' and its settings, visit <https://hub.meltano.com/extractors/tap-mssql--wintersrd>
It works and it displayed the error message about locks as I thought it should!
d
Sorry, I just had to change the name to match previous error message. That's strange, I guess let me try to reinitialize it..
➕ 1
v
fyi you could also do 
rm -rf .meltano/ plugins/
Really shouldn't matter here though
d
Argh, I see what happened. I was trying to figure some environment specific issue between Windows and Linux, so I was in my WSL terminal, but the project I was editing was my Windows one. Sorry about that!
np 1
v
np, glad we figured it out
d
Thanks for your help!
v
My workflow on windows is windows terminal, I open a tab for WSL. Then cd to my coding directory, and run 
code .
to open that directory in vscode on Windows
Might help you here 🤷
d
Yeah, that's what I usually do as well. It's just that I had started this yesterday and forgot that what I had open was the Windows one.
I guess I shouldn't have assumed and just open another
58 Views
Open in Slack
PreviousNext
Powered by