Hello all I m trying to understand if meltano can be used to Meltano #getting-started

Hello all, I’m trying to understand if meltano can...

giorgio_baldelli

08/04/2023, 4:15 PM

Hello all, I’m trying to understand if meltano can be used to load data into a non-aws s3 bucket (minio). I’m running minio locally and have an minio s3 bucket at localhost:9000. I’ve found the target-s3, but have had no luck so far. Has anyone been able to achieve something similar?

giorgio_baldelli

08/04/2023, 4:15 PM

Copy code

version: 1
default_environment: dev
project_id: 39b83dca-20cb-49a0-b7fe-8c610f621e19
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-sftp
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-sftp.git>
    config:
      host: dummy_host
      port: 22
      username: ubuntu
      private_key_file: ~/.ssh/dummy_key
      tables:
      - table_name: dummy_customer
        search_prefix: /dummy_path/
        search_pattern: dummy_file*
        delimiter: ','
        encoding: utf-8
        sanitize_header: true
      start_date: 2020-01-01
    select:
    - '*.*'
  loaders:
  - name: target-s3
    variant: crowemi
    pip_url: git+<https://github.com/crowemi/target-s3.git>
    config:
      cloud_provider:
        aws:
          aws_bucket: test
          aws_endpoint_override: localhost:9000

edgar_ramirez_mondragon

08/04/2023, 4:36 PM

Hi @giorgio_baldelli! That config looks ok at first glance. What problems are you running into?

giorgio_baldelli

08/04/2023, 5:26 PM

@edgar_ramirez_mondragon: I’m seeing the following error:

Copy code

2023-08-04T17:18:48.855465Z [error    ] Loading failed                 code=1 message=AttributeError: 'NoneType' object has no attribute 'get' name=meltano run_id=ecb527fd-0a6f-4420-a184-fabb89fccd9d state_id=2023-08-04T171838--tap-sftp--target-s3

I tested a dummy loader to understand if the data is being extracted from the sftp host in the first place and that works well. No luck with

target-s3

however.

user

08/04/2023, 5:33 PM

Ok, so it seems that

target-s3

may be misconfigured. Can you inspect the contents of the output from

meltano config target-s3

to see if there's anything that looks off?

giorgio_baldelli

08/04/2023, 6:10 PM

@edgar_ramirez_mondragon: the issue is fixed. Thank you for the help! I inspected the logs in

.meltano

and found a bunch of errors hinting to missing settings. I got it to work with a custom endpoint. The config:

Copy code

{
  "append_date_to_filename": true,
  "append_date_to_filename_grain": "day",
  "append_date_to_prefix": true,
  "append_date_to_prefix_grain": "day",
  "cloud_provider": {
    "aws": {
      "aws_access_key_id": "dummy_key",
      "aws_bucket": "dummy_bucket",
      "aws_endpoint_override": "<https://localhost:9000>",
      "aws_region": "us-east-1",
      "aws_secret_access_key": "dummy_secret"
    },
    "cloud_provider_type": "aws"
  },
  "format": {
    "format_type": "parquet"
  },
  "prefix": "dummy/path"
}

user

08/04/2023, 6:11 PM

Awesome! It's probably worth logging an issue in the tap's repo so users get better error messages in the future

Open in Slack

Previous Next