Hello Everyone, I need some help. I am trying to customize a Tap; however, I am a data analyst with minimal to no development skills and only some basic knowledge of Python, so I am feeling a bit lost. I managed to figure out how to make changes to some parts of the tap-clickup code and even added a few new features that worked locally. However, I am struggling to fetch data from the meltano.yml file. To solve this issue, I resorted to using YAML to read the file and retrieve the data I needed, as it was the only method I was familiar with. While this approach works, it has its limitations because it doesn't recognize the different environments in which Meltano is running. It simply retrieves the raw data from the YAML file, locking it to a single configuration. So, my question is, how can I modify the Tap code to access the meltano.yml settings dynamically during runtime? I want it to use the appropriate configuration saved in meltano.yml. Here is a snippet of the code I added to one of the tap-clickup files (which currently statically retrieves the configuration from the file). I need to fetch the values from workspace_id,spaces_id and list_ids so it can have different sources configured for each env

streams.py

import yaml

# Load the YAML config file from meltano
with open("meltano.yml", "r") as yaml_file:
    cu_config = yaml.safe_load(yaml_file)

def extract_and_convert_list(config, key):
    value = config.get(key, "")
    if value:
        split_values = value.split(',')
        return [int(item) for item in split_values]
    return []

def find_tap_clickup_config(plugins):
    for plugin in plugins:
        if plugin.get("name") == "tap-clickup":
            return plugin.get("config", {})
    return {}

# Find the tap-clickup configuration
tap_clickup_config = find_tap_clickup_config(cu_config["plugins"]["extractors"])

# Extract and convert workspace ID
cu_workspace = tap_clickup_config.get("workspace_id")
# Extract and convert spaces and lists
spaces_id_list = extract_and_convert_list(tap_clickup_config, "spaces_id")
lists_id_list = extract_and_convert_list(tap_clickup_config, "list_ids")

And here is the

meltano.yml

version: 1
default_environment: dev
project_id: 8f483ca8-e13c-4abb-8d24-45666666
environments:
- name: dev
- name: staging
- name: prod
  config:
    plugins:
      extractors:
      - name: tap-clickup
        variant: autoidm
        pip_url: tap-clickup
        config:
          flattening_enabled: true
          flattening_max_depth: 3
          workspace_id: 12345
          spaces_id: '11111111,22222222'
          list_ids: ''
        select:
        - space.name
        - space.id
        - task.*
      loaders:
plugins:
  extractors:
  - name: tap-clickup
    variant: autoidm
    pip_url: tap-clickup
    config:
      flattening_enabled: true
      flattening_max_depth: 3
      workspace_id: 12345
      spaces_id: '33333333,44444444'
      list_ids: ''
    select:
    - space.name
    - space.id
    - task.*

I wrote up something similar here https://github.com/AutoIDM/tap-clickup/issues/143 for someone asking how to do this (maybe this was you / someone from your team?) A much easier way to do this is to submit a PR to the tap, and then direct the questions there as I can talk about code in a PR much more efficiently. https://github.com/AutoIDM/tap-clickup/blob/main/tap_clickup/tap.py Add properties here for

space_ids

,

workspace_ids

etc Copy the code from here for the stream https://github.com/AutoIDM/tap-indeed/blob/main/tap_indeedsponsoredjobs/streams.py#L80-L87 (plopping in

client.py

is probably the easiest way )

Yes, that was me. I ended up not seeing your reply on that sorry. I am new to all of this so I'll try to figure out how to do a PR. qnd then we talk over there. I managed to add this new feature (besides the point that i stated here that it is hardcoded so it is not reading configs from the yml file) and also found a bug when fetching custom fields that ill post overthere as well (figured out how to solve it so it is also fixed)

@talles_lessa nice, could you push up a PR to help other folks? We try to help everyone out here!