thomas_karbe
01/16/2023, 1:11 PMtap-gitlab
to tap-jsonl
pipeline work, but I’m now having trouble with target-postgres
. The connection is up, but I’m getting Loader Failed
. The traceback seems to be hold pretty useless information, but in the log before that, I stumbled over the following lines:
2023-01-15T07:04:47.084980Z [info ] File "/workspaces/engineering-metrics/meltano/engineering-metrics/.meltano/loaders/target-postgres/venv/lib/python3.10/site-packages/singer_sdk/connectors/sql.py", line 668, in prepare_table cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2023-01-15T07:04:47.087739Z [info ] self.create_empty_table( cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2023-01-15T07:04:47.088341Z [info ] File "/workspaces/engineering-metrics/meltano/engineering-metrics/.meltano/loaders/target-postgres/venv/lib/python3.10/site-packages/target_postgres/connector.py", line 140, in create_empty_table cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2023-01-15T07:04:47.089901Z [info ] self.to_sql_type(property_jsonschema), cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2023-01-15T07:04:47.090177Z [info ] File "/workspaces/engineering-metrics/meltano/engineering-metrics/.meltano/loaders/target-postgres/venv/lib/python3.10/site-packages/target_postgres/connector.py", line 89, in to_sql_type cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2023-01-15T07:04:47.098893Z [info ] if "integer" in jsonschema_type["type"]: cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2023-01-15T07:04:47.099580Z [info ] KeyError: 'type' cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2023-01-15T07:04:47.215360Z [error ] Loader failed
thomas_karbe
01/16/2023, 1:12 PMKeyError: 'type'
thomas_karbe
01/16/2023, 1:15 PMmeltano.yml
for what its worth:
version: 1
default_environment: dev
project_id: 3d9554d1-42dd-42d0-b2e1-914ca4237494
environments:
- name: dev
- name: staging
- name: prod
plugins:
extractors:
- name: tap-gitlab
variant: meltanolabs
pip_url: git+<https://github.com/MeltanoLabs/tap-gitlab.git>
config:
api_url: <https://gitlab.company.com>
groups: group1 group2
start_date: '2022-12-01'
select:
- commits.url
- commits.created_at
- commits.sha
- commits.author_name
- commits.authored_date
- commits.message
loaders:
- name: target-jsonl
variant: andyh1203
pip_url: target-jsonl
- name: target-postgres
variant: meltanolabs
pip_url: git+<https://github.com/MeltanoLabs/target-postgres.git>
config:
user: meltano
database: postgres
add_metadata_columns: 'True'
host: 192.168.123.45
thomas_karbe
01/17/2023, 9:31 AM{
"type": "SCHEMA",
"stream": "commits",
"schema": {
"properties": {
"id": {
"type": [
"null",
"string"
]
},
"project_id": {
"type": [
"null",
"integer"
]
},
"short_id": {
"type": [
"null",
"string"
]
},
"title": {
"type": [
"null",
"string"
]
},
"author_name": {
"type": [
"null",
"string"
]
},
"author_email": {
"type": [
"null",
"string"
]
},
"authored_date": {
"anyOf": [
{
"type": "string",
"format": "date-time"
},
{
"type": "null"
}
]
},
"committer_name": {
"type": [
"null",
"string"
]
},
"committer_email": {
"type": [
"null",
"string"
]
},
"committed_date": {
"anyOf": [
{
"type": "string",
"format": "date-time"
},
{
"type": "null"
}
]
},
"created_at": {
"anyOf": [
{
"type": "string",
"format": "date-time"
},
{
"type": "null"
}
]
},
"message": {
"type": [
"null",
"string"
]
},
"allow_failure": {
"type": [
"null",
"boolean"
]
},
"parent_ids": {
"anyOf": [
{
"type": "array",
"items": {
"type": [
"null",
"string"
]
}
},
{
"type": "null"
}
]
},
"stats": {
"properties": {
"additions": {
"type": [
"null",
"integer"
]
},
"deletions": {
"type": [
"null",
"integer"
]
},
"total": {
"type": [
"null",
"integer"
]
}
},
"type": "object"
}
},
"type": "object"
},
"key_properties": [
"id"
]
}
thomas_karbe
01/17/2023, 9:31 AM"type": ["null", …
seems offthomas_karbe
01/17/2023, 10:31 AMnull
is probably not the issue, since the error complains about type
not being a key in the dict. After scanning through the above, I found that some `type`s are nested below a anyOf
, which probably is an issue with the implementationthomas_karbe
01/17/2023, 10:42 AMJSON Schema combinations such as anyOf and oneOf are not supported.
on https://github.com/datamill-co/target-postgresthomas_karbe
01/17/2023, 10:43 AMthomas_karbe
01/18/2023, 1:12 PMvisch
01/18/2023, 1:13 PMvisch
01/18/2023, 1:14 PMthomas_karbe
01/18/2023, 1:14 PMthomas_karbe
01/18/2023, 1:14 PMvisch
01/18/2023, 1:15 PMthomas_karbe
01/18/2023, 1:15 PManyOf
?visch
01/18/2023, 1:15 PMthomas_karbe
01/18/2023, 1:15 PMvisch
01/18/2023, 1:15 PMthomas_karbe
01/18/2023, 1:16 PManyOf
, so it sounds to me more like a new feature than a fixvisch
01/18/2023, 1:16 PMmeltano.yml
file I think you're using the meltanolabs
variant, and the repo link you sent is for the datamill
variant 🤷thomas_karbe
01/18/2023, 1:17 PMthomas_karbe
01/18/2023, 1:17 PMvisch
01/18/2023, 1:18 PMtransferwise
variant as well you could try. I'm going to vouch for the meltanolabs
variant as we've put a bunch of work into it but obviously there's still some bugs left!thomas_karbe
01/18/2023, 1:19 PMthomas_karbe
01/18/2023, 3:22 PMthomas_karbe
01/18/2023, 3:33 PMmeltano invoke tap-gitlab