Hello, I'm starting using meltano. I setup a simpl...
# getting-started
g
Hello, I'm starting using meltano. I setup a simple copy of a table from MSSQL and PGSQL, this is my meltano.yml:
Copy code
default_environment: dev
project_id: 019af9e5-8081-74b6-9332-7a4a1333b3b7
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-mssql
    variant: buzzcutnorman
    pip_url: git+<https://github.com/BuzzCutNorman/tap-mssql.git>
    config:
      database: SBOFPM
      host: 10.48.149.4
      user: ro_sap
    select:
    - 'dbo-OCRD.*'
  loaders:
  - name: target-postgres
    variant: meltanolabs
    pip_url: meltanolabs-target-postgres
    config:
      database: meltano_db
      host: postgres
      user: meltano_user
  utilities:
  - name: dbt-postgres
    variant: dbt-labs
    pip_url: dbt-core dbt-postgres meltano-dbt-ext~=0.3.0
jobs:
- name: copia
  tasks:
  - tap-mssql target-postgres
When I run
meltano run copia
I get errorr like:
Copy code
╭──────────────────────────────────────────────────────────────── Error details ─────────────────────────────────────────────────────────────────╮
│                                                                                                                                                │
│ /project/.meltano/loaders/target-postgres/venv/lib/python3.10/site-packages/singer_sdk/sinks/core.py:125 in validate                           │
│                                                                                                                                                │
│   125 │   │   │   raise InvalidRecord(e.message, record) from e                                                                                │
│                                                                                                                                                │
│ /project/.meltano/loaders/target-postgres/venv/lib/python3.10/site-packages/singer_sdk/sinks/core.py:561 in _validate_and_parse                │
│                                                                                                                                                │
│   561 │   │   │   │   self._validator.validate(record)                                                                                         │
│                                                                                                                                                │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
InvalidRecord: Record Message Validation Error: '\\\\SWSAP02\\Documenti_SAP\\Allegati\\ORDINE 6 aafsigned.pdf' is too long
What am I doing wrong ?
v
meltano.yml , entire stack trace would be helpful, and the schema and record here would all help To do that
meltano invoke tap-mssql > out
and then figure out the minimal replicable thing to get
cat out | meltano invoke target-postgres
to fail and then you can narrow it down I'd be if you want a quick win here without understaing it deeply, you could probably turn off record validation in target-postgres and it'd work Not 100% sure as that's a very truncated trace
g
Hello Derek, thank you for your reply. Attached is the full log of the transaction. Hope to be helpful.
meltano.log
v
Yes very helpful
Copy code
ValidationError: '001460000199999' is too long

Failed validating 'maxLength' in schema['properties']['AliasName']:
    {'maxLength': 8, 'type': ['string', 'null']}

On instance['AliasName']:
    '001460000199999'
One of the fields in the MSSQL database thinks that it has a max length of 8, but the data is longer. Maybe you have a cached catalog? You could run
meltano install --clean
or
rm -rf .meltano
and try again. That'd be my guess
1
Two other ways to handle cached catalogs 1. https://docs.meltano.com/concepts/plugins/#use_cached_catalog-extra 2.
meltano run --refresh-catalog copia
Also if you still can't get it we need your 1. meltano.yml 2. Docker build file
Note for exceptions to scroll up a bit, there's a lot of noise unforunately, it's gotten better but it's still there 😕