👋 Hello! I’ve been using meltano for about an year now and absolutely loving it! Was using to extract data from

tap-mysql

to

target-bigquery

till now which was working great, but now we’re using DynamoDB as well. Tried to set up

tap-dynamodb

and after a lot of struggle with roles and policies permissions, was finally able to extract data, but then I got an error that loader failed.

load_job = client.load_table_from_file(\n', '  File "/home/ubuntu/meltano-projects/erp-to-bigquery-2/.meltano/loaders/target-bigquery/venv/lib/python3.8/site-packages/google/cloud/bigquery/client.py", line 2431, in load_table_from_file\n    raise exceptions.from_http_response(exc.response)\n', 'google.api_core.exceptions.BadRequest: 400 POST <https://bigquery.googleapis.com/upload/bigquery/v2/projects/turing-audio-266016/jobs?uploadType=resumable>: Empty schema specified for the load job. Please specify a schema that describes the data being loaded.\n']

Wasn’t sure how to specify the schema apart from catalog (?), so used

meltano invoke

to dump catalog into a JSON file, and then`--catalog` flag to specify that as the schema but I guess that doesn’t work either, got the same error. So does anyone know to specify the schema, or any other way for this data to reach BigQuery?

Hi, @abhishek_ajmera! So glad to have you as a happy Meltano user. The DynamoDB is probably one of the hardest to configure so you're in good company for first time experience trying to onboard with that tap.

Hi @aaronsteers, perfect timing - I think I JUST figured out how to define the schema in catalog JSON. Did the first successful EL for DynamoDB to BigQuery 2 min ago! 😬

Glad it is working out 👍