Hi what is the correct way to set the elt buffersize in melt Meltano #singer-target-development

Hi, what is the correct way to set the elt buffers...

Nir Diwakar (Nir)

10/02/2024, 2:47 PM

Hi, what is the correct way to set the elt buffersize in meltano.yml? Is it this:

Copy code

loaders:
  - name: target-elasticsearch
    namespace: target_elasticsearch
    pip_url: -e ./targets/target-elasticsearch
    executable: target-elasticsearch
    config:
      url: ${ES_URL}
      scheme: https
      slack_channel: ${SLACK_CHANNEL}
      slack_token: ${SLACK_TOKEN}
      dagit_url: ${DAGIT_BASE_URL}
      environment: ${ENV}
      tap: ${TAP}
    env:
      MELTANO_ELT_BUFFER_SIZE: 20000000

or this:

Copy code

loaders:
  - name: target-elasticsearch
    namespace: target_elasticsearch
    pip_url: -e ./targets/target-elasticsearch
    executable: target-elasticsearch
    config:
      url: ${ES_URL}
      scheme: https
      slack_channel: ${SLACK_CHANNEL}
      slack_token: ${SLACK_TOKEN}
      dagit_url: ${DAGIT_BASE_URL}
      environment: ${ENV}
      tap: ${TAP}
    elt:
      buffer_size: 20000000

BuzzCutNorman

10/02/2024, 3:40 PM

I use the second version.

Copy code

elt:
  buffer_size: 26214400

or

elt.buffer_size: 26214400

This might be the wrong way of looking at it but I always thought of your first option as how you can set this per server via system or user using environment variables. Then you don't have to control that in your meltano config allowing you have a different value per server. You can set it via env in the meltano config but I usually default to using the normal config option.

Nir Diwakar (Nir)

10/02/2024, 4:02 PM

Thanks, I wasn't able to find the correct usage in documentation

Edgar Ramírez (Arch.dev)

10/02/2024, 4:23 PM

fwiw that should be a top-level setting, and you can set it with

Copy code

meltano config meltano set elt.buffer_size 26214400

Nir Diwakar (Nir)

10/02/2024, 4:28 PM

Thanks, I corrected it

👍 1

Nir Diwakar (Nir)

10/02/2024, 4:28 PM

This should be: loaders: - name: target-elasticsearch namespace: target_elasticsearch pip_url: -e ./targets/target-elasticsearch executable: target-elasticsearch config: url: ${ES_URL} scheme: https slack_channel: ${SLACK_CHANNEL} slack_token: ${SLACK_TOKEN} dagit_url: ${DAGIT_BASE_URL} environment: ${ENV} tap: ${TAP} elt: buffer_size: 20000000

19 Views

Open in Slack

Previous Next