https://linen.dev logo
Join Slack
Powered by
hey! i'm developing a tap and am trying to split c...
# troubleshooting
p
hey! i'm developing a tap and am trying to split config across multiple meltano.yml files but meltano doesn't seem to be reading config from the other file. threading the contents of both files - i bet it's something subtle with how the yml is formatted but i can't figure it out
this is the default meltano.yml in the home directory of the tap
Copy code
version: 1
send_anonymous_usage_stats: true
project_id: "tap-facebook-pages"
include_paths:
  - ./*.meltano.yml
default_environment: test
environments:
- name: test
plugins:
  extractors:
  - name: "tap-facebook-pages"
    namespace: "tap_facebook_pages"
    pip_url: -e .
    capabilities:
    - state
    - catalog
    - discover
    - about
    - stream-maps
    config:
      start_date: '2010-01-01T00:00:00Z'
    settings:
    - name: user_token
      kind: password
    - name: page_ids
    - name: start_date
      value: '2022-10-01T00:00:00Z'
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
and then i have another file 
private.meltano.yml
in the same directory
Copy code
environments:
- name: test
  config:
    plugins:
      extractors:
      - name: "tap-facebook-pages"
        config:
          page_ids:
          - 'xxxxxxxxxxxx'
but meltano complains about 
page_ids
missing 
Config validation failed: 'page_ids' is a required property
c
The root element of your 
private.meltano.yml
should be the 
plugins:
key.
p
huh okay i was following the meltano squared repo as an example https://github.com/meltano/squared/blob/main/data/environments/prod.meltano.yml
and changing to this returns the same error
Copy code
plugins:
  extractors:
  - name: "tap-facebook-pages"
    config:
      page_ids:
      - 'xxxxxxxxx'
c
Ok. Might not be related to the error then. 😁
p
haha thanks for trying to help though
c
What does the output of 
meltano config tap-facebook-pages list
say? Does it show the 
page_ids
value?
p
nope, it reads my access token set with an env var but not 
page_ids
from the other file
Untitled.txt
c
Ah. I forgot that config ignores the default env ... what about about 
meltano --environment=test config tap-facebook-pages list
?
p
beat you to it, same output there too except for this first line naturally 🙃
Copy code
2022-11-02T21:28:52.628657Z [info     ] Environment 'test' is active
t
@prratek_ramchandani what version of Meltano are you on?
p
2.8.0
t
Hmm - I wonder if https://github.com/meltano/meltano/issues/6727 is related at all. @edgar_ramirez_mondragon have you seen this?
@prratek_ramchandani might be good to make an issue on this as well since it’s getting later in the day for some of the team
p
setting a reminder to come back to this and create an issue tomorrow morning!
e
@taylor that issue may indeed be related. I see in @prratek_ramchandani's project files that the environment 
test
is defined in both. Removing it from 
meltano.yml
might fix it.
p
@edgar_ramirez_mondragon is this what you're suggesting? meltano still doesn't see the 
page_ids
config
Copy code
# meltano.yml

version: 1
send_anonymous_usage_stats: true
project_id: "tap-facebook-pages"
include_paths:
  - ./*.meltano.yml
#default_environment: test
#environments:
#- name: test
plugins:
  extractors:
  - name: "tap-facebook-pages"
    namespace: "tap_facebook_pages"
    pip_url: -e .
    capabilities:
    - state
    - catalog
    - discover
    - about
    - stream-maps
    config:
      start_date: '2010-01-01T00:00:00Z'
    settings:
    - name: user_token
      kind: password
    - name: page_ids
    - name: start_date
      value: '2022-10-01T00:00:00Z'
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
Copy code
# private.meltano.yml

#environments:
#- name: test
#  config:
plugins:
  extractors:
  - name: "tap-facebook-pages"
    config:
      page_ids:
      - 'xxxxxxxxx'
t
I’m not able to reproduce this using a very similar setup and a different tap. @prratek_ramchandani try removing the enviornments key form the main meltano.yml file but keeping it in 
private.meltano.yml
. The environments key in the main meltano.yml isn’t for listing what enviornments are available, it’s for defining actual config in them. So drop it from the main one and only define it once in 
private.meltano.yml
p
that didn't seem to fix it either so i created this issue https://github.com/meltano/meltano/issues/6970
for context this is what i tried
Copy code
# meltano.yml 

version: 1
send_anonymous_usage_stats: true
project_id: "tap-facebook-pages"
include_paths:
  - ./*.meltano.yml
#default_environment: test
#environments:
#- name: test
plugins:
  extractors:
  - name: "tap-facebook-pages"
    namespace: "tap_facebook_pages"
    pip_url: -e .
    capabilities:
    - state
    - catalog
    - discover
    - about
    - stream-maps
    config:
      start_date: '2010-01-01T00:00:00Z'
    settings:
    - name: user_token
      kind: password
    - name: page_ids
    - name: start_date
      value: '2022-10-01T00:00:00Z'
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
Copy code
# private.meltano.yml

environments:
- name: test
  config:
    plugins:
      extractors:
      - name: "tap-facebook-pages"
        config:
          page_ids:
          - 'xxxxxxxx'
t
@prratek_ramchandani and doing 
meltano config tap-facebook-pages
does not show the proper config list? I’m wondering if this is specific to 
meltano elt
p
correct, it shows that 
page_ids
is None
Copy code
❯ meltano config tap-facebook-pages list
user_token [env: TAP_FACEBOOK_PAGES_USER_TOKEN] current value: 'xxxxxxxxxxxx' (from the TAP_FACEBOOK_PAGES_USER_TOKEN variable in the environment)
page_ids [env: TAP_FACEBOOK_PAGES_PAGE_IDS] current value: None (default)
start_date [env: TAP_FACEBOOK_PAGES_START_DATE] (default: '2022-10-01T00:00:00Z') current value: '2010-01-01T00:00:00Z' (from `meltano.yml`)
t
@prratek_ramchandani we may need to hop on a quick call to debug b/c I don’t know that we can reproduce this on our end. @edgar_ramirez_mondragon commented in the issue that he wasn’t able to repro.
c
I think I might be able to reproduce this now ... confirming a few more things.
Yeah. I can reproduce it:
Copy code
meltano init meltenvtest

cd meltenvtest

meltano add extractor tap-github

mkdir environments
add contents to `environments/dev.meltano.yml`:
Copy code
environments:
- name: dev
  config:
    plugins:
      extractors:
      - name: tap-github
        config:
          user_agent: meltano
update meltano.yml with `include_paths`:
Copy code
version: 1
default_environment: dev
project_id: c3284a57-63c8-486c-8a3d-c6114642dea6
environments:
- name: dev
include_paths:
- ./environments/*.meltano.yml
plugins:
  extractors:
  - name: tap-github
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-github.git>
Run `meltano config`:
Copy code
MELTANO_ENVIRONMENT=dev meltano config tap-github list
Output:
Copy code
2022-11-11T07:06:41.693904Z [info     ] Environment 'dev' is active
user_agent [env: TAP_GITHUB_USER_AGENT] current value: None (default)
metrics_log_level [env: TAP_GITHUB_METRICS_LOG_LEVEL] current value: None (default)
        Metrics Log Level: The log level of the API response metrics.
auth_token [env: TAP_GITHUB_AUTH_TOKEN] current value: None (default)
        Auth Token: GitHub token to authenticate with.
additional_auth_tokens [env: TAP_GITHUB_ADDITIONAL_AUTH_TOKENS] current value: None (default)
        Additional Auth Tokens: List of GitHub tokens to authenticate with. Streams will loop through them when hitting rate limits.
rate_limit_buffer [env: TAP_GITHUB_RATE_LIMIT_BUFFER] current value: None (default)
        Rate Limit Buffer: Add a buffer to avoid consuming all query points for the token at hand. Defaults to 1000.
searches [env: TAP_GITHUB_SEARCHES] current value: None (default)
        Searches: An array of search descriptor objects with the following properties. "name" - a human readable name for the search query. "query" -  a github search string (generally the same as would come after ?q= in the URL)
organizations [env: TAP_GITHUB_ORGANIZATIONS] current value: None (default)
        Organizations: An array of strings containing the github organizations to be included
repositories [env: TAP_GITHUB_REPOSITORIES] current value: None (default)
        Repositories: An array of strings containing the github repos to be included
user_usernames [env: TAP_GITHUB_USER_USERNAMES] current value: None (default)
        User Usernames: A list of GithHub usernames.
user_ids [env: TAP_GITHUB_USER_IDS] current value: None (default)
        User IDs: A list of GitHub user ids.
start_date [env: TAP_GITHUB_START_DATE] current value: None (default)
stream_maps [env: TAP_GITHUB_STREAM_MAPS] current value: None (default)
stream_map_config [env: TAP_GITHUB_STREAM_MAP_CONFIG] current value: None (default)
skip_parent_streams [env: TAP_GITHUB_SKIP_PARENT_STREAMS] current value: None (default)
        Skip Parent Streams: Set to true to skip API calls for the parent streams (such as repositories) if it is not selected but children are.

To learn more about extractor 'tap-github' and its settings, visit <https://hub.meltano.com/extractors/tap-github--meltanolabs>
user_agent
is 
None
Ah! And found the issue ... it is indeed the problem of environments specified in multiple places ... 😂
I didn't get that error message from the Github issue though ...
Open in Slack
PreviousNext
Powered by