Hi i am trying the meltano as poc and looking for ...
# getting-starteds
sk
10/31/2023, 8:04 PMHi i am trying the meltano as poc and looking for a help on setting up the dev environment on maltano.yml
sk
10/31/2023, 8:05 PMmeltano config --environment list
sk
10/31/2023, 8:05 PMError: No such option: --environment
sk
10/31/2023, 8:09 PMi am running meltano, version 3.1.0
u
user
10/31/2023, 8:52 PMI dont think that is a valid command, check out https://docs.meltano.com/reference/command-line-interface/. You can try
meltano --environment=dev config target-bigquery set key valuemeltano environment lists
sk
10/31/2023, 8:56 PMThank you Pat
sk
10/31/2023, 9:03 PMi trying to figure out whats wrong with my config so i dont get this kind of config parameter in my meltano.yml (that i copied previously) specially under "environments:
- name: dev"
sk
10/31/2023, 9:03 PMplugins:
extractors:
- name: tap-gitlab
variant: meltanolabs
pip_url: git+https://github.com/MeltanoLabs/tap-gitlab.git
environments:
- name: dev
config:
plugins:
extractors:
- name: tap-gitlab
config:
projects: meltano/meltano meltano/tap-gitlab
start_date: "2022-03-01T000000Z"
select:
- commits.id
- commits.project_id
- commits.created_at
- commits.author_name
- commits.message
- tags.*
- "!*.*_url"
sk
10/31/2023, 9:03 PMi am following this doc : https://docs.meltano.com/guide/complete_tutorial
sk
10/31/2023, 9:04 PMin place of tab-gitlap i chose to use tap-rest-api-msdk and target-bigquery
sk
10/31/2023, 9:41 PMTrying to resolve this Error ---.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.11/site-packages/tap_rest_api_msdk/tap.py", line 538, in get_schema
raise ValueError(r.text)
ValueError: {"message":null}
p
pat_nadolny
11/01/2023, 1:29 PMWe usually recommend getting a simple example working first before trying to customize to use more advanced plugins, tap-rest-api-msdk is pretty complex since it can be configured to do a lot. I'm not sure I follow the issue youre running into right now, sharing your full meltano.yml might help. Additionally if you run the config command it should print your current json config to stdout https://docs.meltano.com/reference/command-line-interface#config
s
sk
11/01/2023, 6:37 PMHI Pat, thanks for helping me here, and letting me know "tap-rest-api-msdk" is bit advanced to try first. But our requirement mainly focuses to show how well can we use meltano to get rest api data into BQ, SO i picked up a simple API that we use
sk
11/01/2023, 6:37 PMthe ouput of the api is 2 columns
sk
11/01/2023, 6:37 PM{
"code": 200,
"data": [
{
"account_id": 2376,
"account_name": "AAA"
},
{
"account_id": 2354,
"account_name": "BBBBBBBB"
},
]
}
sk
11/01/2023, 6:42 PMand here is my full meltano.yml
sk
11/01/2023, 6:42 PM Copy code
version: 1
default_environment: dev
project_id: f349f92a-9f42-40ff-bb48-434eef411caa
environments:
- name: dev
- name: staging
- name: prod
plugins:
extractors:
- name: tap-rest-api-msdk
variant: widen
pip_url: tap-rest-api-msdk
config:
api_url: <https://api.somesteamname.com/accounts>
auth_method: basic
loaders:
- name: target-bigquery
variant: z3z1ma
pip_url: git+<https://github.com/z3z1ma/target-bigquery.git>
config:
credentials_json: "{\n \" unt\"\n}"sk
11/01/2023, 6:44 PMi used the interactive "`meltano config tap-rest-api-msdk set --interactive"`
sk
11/01/2023, 6:45 PMto answer most of the questions , like api_url, using basic authrization token,
sk
11/01/2023, 6:45 PMwhen i do meltano config tap-rest-api-msdk test
sk
11/01/2023, 6:46 PMi endup getting this error
sk
11/01/2023, 6:46 PM Copy code
.meltano/extractors/tap-rest-api-msdk/venv/lib/python3.11/site-packages/tap_rest_api_msdk/tap.py", line 538, in get_schema
raise ValueError(r.text)
ValueError: {"message":null}