Hi Team, I am using target-snowflake --meltano variant. While executing

metano elt tap-mongodb target-snowflake --job_id=mongo-to-snowflake

I get the following error

CRITICAL Not supported schema by target-snowflake: cmd_type=loader job_id=2022-05-03T123721--tap-mongodb--target-snowflake name=target-snowflake run_id=15fbfdb0-36d0-440d-a88a-231e547a7408 stdio=stderr
2022-05-03T12:37:40.497702Z [info     ] CRITICAL  {"type": "SCHEMA", "stream": "submissions", "schema": {"type": "object"}, "key_properties": ["_id"]} cmd_type=loader job_id=2022-05-03T123721--tap-mongodb--target-snowflake name=target-snowflake run_id=15fbfdb0-36d0-440d-a88a-231e547a7408 stdio=stderr
2022-05-03T12:37:40.498653Z [info     ] CRITICAL                       cmd_type=loader job_id=2022-05-03T123721--tap-mongodb--target-snowflake name=target-snowflake run_id=15fbfdb0-36d0-440d-a88a-231e547a7408 stdio=stderr
2022-05-03T12:37:40.499027Z [info     ] CRITICAL It should at least have one top level property in schema. cmd_type=loader job_id=2022-05-03T123721--tap-mongodb--target-snowflake name=target-snowflake run_id=15fbfdb0-36d0-440d-a88a-231e547a7408 stdio=stderr
2022-05-03T12:37:40.581085Z [error    ] Loading failed                 code=1 job_id=2022-05-03T123721--tap-mongodb--target-snowflake message=CRITICAL It should at least have one top level property in schema. name=meltano run_id=15fbfdb0-36d0-440d-a88a-231e547a7408

Could you share your

meltano.yml

?

@visch

version: 1
default_environment: dev
project_id: 0df72b50-ba8a-4169-884b-bafd34832785
plugins:
  extractors:
  - name: tap-mongodb
    variant: singer-io
    pip_url: tap-mongodb
    config:
      host: localhost
      port: 27017
      user: ashish
      database: admin
      replica_set: atlas-f6gbhi-shard-0
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
    config:
      do_timestamp_file: true
  - name: target-snowflake
    variant: meltano
    pip_url: git+<https://gitlab.com/meltano/target-snowflake.git>
    config:
      account: opu40841.west-europe.azure
      dbname: ANALYSIS
      database: ANALYSIS
      user: ASHISH_DEV
      username: ASHISH_DEV
      warehouse: DEV_TEST_WH
      file_format: ANALYSIS.MELTANO_ASHISH.FILE_FORMAT_CSV
      role: DBT_DEVELOPER
      default_target_schema: MELTANO_ASHISH
      schema: MELTANO_ASHISH
      add_metadata_columns: false
      logging_level: DEBUG
      batch_size_rows: 1000000
      primary_key_required: true
      snowflake_account: ou40841.west-europe.azure
      batch_size: 10000
schedules:
- name: mongodb-to-jsonl
  extractor: tap-mongodb
  loader: target-jsonl
  transform: skip
  interval: '@once'
  start_date: 2022-04-30 09:55:11.751603
environments:
- name: dev
  config:
    plugins:
      extractors:
      - name: tap-mongodb
        config:
          ssl: true
          filter_schemas: public
          default_replication_method: LOG_BASED
        metadata:
          '*':
            replication-method: LOG_BASED
        select:
        - koalas-applications-dev-hometrackbrokervaluations
        - koalas-applications-dev-submissions
      loaders:
      - name: target-jsonl
        config:
          destination_path: origination_elt/output
      - name: target-snowflake
        config:
          account: ou40841.west-europe.azure
          dbname: ANALYSIS
          user: ASHISH_DEV
          warehouse: DEV_TEST_WH
          file_format: ANALYSIS.MELTANO_ASHISH.FILE_FORMAT_CSV
          role: DBT_DEVELOPER
          default_target_schema: MELTANO_ASHISH
          add_metadata_columns: false
          logging_level: DEBUG
          batch_size_rows: 1000000
          primary_key_required: true
          snowflake_account: ou40841.west-europe.azure
          batch_size: 10000
- name: staging
- name: prod

@ashish_kumar have you read https://docs.meltano.com/reference/command-line-interface#select specefically the list part to see what's being selected right now from Mongo?

i tried adding this but I am not getting anything under selected coloumn meltano select tap-mongodb --list

2022-05-03T14:09:45.459630Z [info     ] Environment 'dev' is active
Legend:
        selected
        excluded
        automatic

Enabled patterns:
        koalas-applications-dev-hometrackbrokervaluations.*
        koalas-applications-dev-submissions.*

Selected attributes:

We're getting somewhere though right? 😄

yeah

without the .* what do you get. then we just need to figure that out

Actually I am getting confused here. I have a database in mongodb koalas-applications-dev and collection submissions. So how can i select all data from submissions

Read through https://docs.meltano.com/reference/command-line-interface#select first 🙂

now I am getting another error

2022-05-03T14:23:49.228581Z [info     ] snowflake.connector.errors.ProgrammingError: 000904 (42000): SQL compilation error: error line 1 at position 69 cmd_type=loader job_id=2022-05-03T142240--tap-mongodb--target-snowflake name=target-snowflake run_id=1ec8e924-d1fe-4d68-9b0c-5ad8874ad931 stdio=stderr

2022-05-03T14:23:49.228777Z [info     ] invalid identifier '_ID'       cmd_type=loader job_id=2022-05-03T142240--tap-mongodb--target-snowflake name=target-snowflake run_id=1ec8e924-d1fe-4d68-9b0c-5ad8874ad931 stdio=stderr

2022-05-03T14:23:49.335809Z [error    ] [Errno 32] Broken pipe

Traceback (most recent call last):

🤷 sounds like a different issue? Not clear to me but good luck

do we have to define schema for target?

no

I tried another version of snowflake variant --meltano it give another error

you have the same issue as before

it works fine I am getting expected output in jsonl