Hello guys G DAY me again smiley I got this error when tryin Meltano #troubleshooting

Hello guys! G’ DAY, me again :smiley:. I got this...

lin_liu

08/25/2022, 4:11 AM

Hello guys! G’ DAY, me again 😃. I got this error when trying to got my local dynamodb into target postgres:

Run invocation could not be completed as block failed: Loader failed

not sure which part of my config is wrong, can I have some 👀s on it! Thank you ! 🙏

- name: tap-test

inherit_from: tap-dynamodb

config:

account_id: ‘<YOURAWSACCOUNTID>’

external_id: ‘NOT_REQUIRED_BUT_NEEDS_TO_HAVE_MORE_THAN_2_CHARACTERS’

role_name: <‘NameOfAnIamRoleWithSufficientAccess’

region_name: ‘ap-southeast-2’

use_local_dynamo: True

load_schema: test_raw_20220822

select:

- testTable.*

metadata:

testTable:

replication-method: INCREMENTAL

lin_liu

08/25/2022, 4:13 AM

not too many err logs tho:

Traceback (most recent call last):

File "…/meltano-pipeline/.venv/lib/python3.8/site-packages/meltano/core/logging/output_logger.py", line 137, in redirect_logging

yield

File "…/meltano-pipeline/.venv/lib/python3.8/site-packages/meltano/core/block/extract_load.py", line 438, in run

await self.run_with_job()

File "…/meltano-pipeline/.venv/lib/python3.8/site-packages/meltano/core/block/extract_load.py", line 464, in run_with_job

await self.execute()

File "…/meltano-pipeline/.venv/lib/python3.8/site-packages/meltano/core/block/extract_load.py", line 430, in execute

await manager.run()

File "…/meltano-pipeline/.venv/lib/python3.8/site-packages/meltano/core/block/extract_load.py", line 630, in run

_check_exit_codes(

File "/…/meltano-pipeline/.venv/lib/python3.8/site-packages/meltano/core/block/extract_load.py", line 782, in _check_exit_codes

raise RunnerError("Loader failed", {PluginType.LOADERS: consumer_code})

meltano.core.runner.RunnerError: Loader failed

2022-08-25T04:08:38.411867Z [error    ] Block run completed.           block_type=ExtractLoadBlocks err=RunnerError('Loader failed') exit_codes={<PluginType.LOADERS: 'loaders'>: 1} set_number=0 success=False

Run invocation could not be completed as block failed: Loader failed

christoph

08/25/2022, 6:03 AM

At a first glance it smells like the postgres target fails to invoke properly (likely due to configuration errors). You should get some good error details with

meltano --log-level=debug

https://docs.meltano.com/reference/command-line-interface#debugging

lin_liu

08/25/2022, 6:10 AM

Hi Christoph thanks! is this cmd like this: meltano --log-level=debug elt tap-XXX target-warehouse-postgres?

christoph

08/25/2022, 6:24 AM

Yup. That's how to run it.

christoph

08/25/2022, 6:27 AM

target-warehouse-postgres

needs to exist as a plugin in your meltano project though. (normally the postgres targets from the hub are named

target-postgres

)

lin_liu

08/25/2022, 6:30 AM

yes that is the one inherited from target-postgres, actually we got some source postgres to target postgres working well. However, for this one tap-dynamodb, I met some troubles these days, i suspect there are some configs wrong, but not sure where. 🤔

christoph

08/25/2022, 6:31 AM

Makes sense. I suspected it was an inherited plugin. Interesting that it works for a different source. Let's hope the debug will be useful.

christoph

08/25/2022, 6:54 AM

Copy code

2022-08-25T06:26:10.709566Z [info     ] psycopg2.errors.UndefinedColumn: column "hash_key" named in key does not exist cmd_type=loader name=target-warehouse-postgres run_id=8a128b5d-12db-484c-aa61-1d1e894337a8 state_id=2022-08-25T062605--tap-kamprad--target-warehouse-postgres stdio=stderr

At a first glance, this sounds like the dynamodb tap is not emitting the

hash_key

column for some reason, so the postgres target just bails when it is instructed to create a new table with a primary key that doesn't exist in the source stream. You could just run the dynamodb tap alone with