Hi folks I am trying to use a schema file to ingest data fro Meltano #troubleshooting

Hi folks, I am trying to use a schema file to inge...

Chris

01/29/2024, 5:05 PM

Hi folks, I am trying to use a schema file to ingest data from REST API. If I declare it like this https://sdk.meltano.com/en/latest/typing.html it works. If I use the following it throws AttributeError

Copy code

SCHEMAS_DIR = importlib_resources.files(__package__) / "schemas"

✅ 1

Andy Carter

01/29/2024, 5:59 PM

Can you share the json files you put into the

schemas

folder?

Chris

01/29/2024, 6:17 PM

calls.json

Chris

01/29/2024, 6:18 PM

I had

schema_filepath = SCHEMAS_DIR / "calls.json"

. Not sure if I need to do anything else. If I declare the schema dict manually it will work.

Copy code

schema = th.PropertiesList(
        th.Property(
            "metaData",
            th.ObjectType(
                th.Property("id", th.StringType),
                th.Property("url", th.StringType),
                th.Property("title", th.StringType),
                th.Property("scheduled", th.DateTimeType),
                th.Property("started", th.DateTimeType),
                th.Property("duration", th.IntegerType),
                th.Property("primaryUserId", th.StringType),
                th.Property("direction", th.StringType),
                th.Property("system", th.StringType),
                th.Property("scope", th.StringType),
                th.Property("media", th.StringType),
                th.Property("language", th.StringType),
                th.Property("meetingUrl", th.StringType),
                th.Property("isPrivate", th.BooleanType),
                th.Property("calendarEventId", th.StringType),
            )
        ),
    ).to_dict()

Chris

01/29/2024, 6:59 PM

just double checking, where should i place the schema files? under the tap? tap-rest-api |--schemas --schema.json init.py main.py client.py stream.py tap.py

Edgar Ramírez (Arch.dev)

01/29/2024, 8:49 PM

The

schemas

directory should be live inside the package:

Copy code

tap_bitso
├── __init__.py
├── auth.py
├── client.py
├── schemas
│   ├── __init__.py
│   ├── book.json
│   ├── ledger.json
│   ├── open_order.json
│   ├── ticker.json
│   └── trade.json
├── streams.py
└── tap.py

I'll add that to the cookiecutter.

Chris

01/29/2024, 11:57 PM

i had an import issue and a json schema declaration issue. thank you for sharing! 🙇

🙌 1

2 Views

Open in Slack

Previous Next