https://linen.dev logo
Join Slack
Powered by
Hello :wave: I am build a custom tap for Greenhou...
# singer-tap-development
a
Hello 👋 I am build a custom tap for Greenhouse and trying to get Activity Feed data into 
Redshift
but unfortunately running into error, more details in the thread. Any help would be appreciated! 🙂
Error: ```2022-08-01T171542.496694Z [info ] time=2022-08-01 224542 name=target_redshift level=INFO message=Getting catalog objects from table cache... cmd_type=loader job_id=greenhouse-to-redshift name=target-redshift run_id=03f08b22-98ac-477a-a3dc-df983d9458a3 stdio=stderr 2022-08-01T171542.497501Z [info ] time=2022-08-01 224542 name=target_redshift level=ERROR message=Invalid configuration: cmd_type=loader job_id=greenhouse-to-redshift name=target-redshift run_id=03f08b22-98ac-477a-a3dc-df983d9458a3 stdio=stderr 2022-08-01T171542.497619Z [info ] * Required key is missing from config: [dbname] cmd_type=loader job_id=greenhouse-to-redshift name=target-redshift run_id=03f08b22-98ac-477a-a3dc-df983d9458a3 stdio=stderr 2022-08-01T171542.528102Z [debug ] Deleted configuration at /Users/ananyasingh/Code/Remote/etl/.meltano/run/elt/greenhouse-to-redshift/03f08b22-98ac-477a-a3dc-df983d9458a3/target.99d54437-f2d3-4323-a2a1-42410fc57078.config.json 2022-08-01T171542.528685Z [debug ] Deleted configuration at /Users/ananyasingh/Code/Remote/etl/.meltano/run/elt/greenhouse-to-redshift/03f08b22-98ac-477a-a3dc-df983d9458a3/tap.d44ac0c0-b1f2-4110-91a9-9236bc192ef9.config.json 2022-08-01T171542.528793Z [error ] Loading failed code=1 job_id=greenhouse-to-redshift message= * Required key is missing from config: [dbname] name=meltano run_id=03f08b22-98ac-477a-a3dc-df983d9458a3 2022-08-01T171542.530232Z [debug ] ELT could not be completed: Loader failed Traceback (most recent call last): File "/Users/ananyasingh/Code/Personal/meltano-projects/.venv/meltano/lib/python3.9/site-packages/meltano/cli/elt.py", line 240, in _run_elt await _run_extract_load(log, elt_context, output_logger) File "/Users/ananyasingh/Code/Personal/meltano-projects/.venv/meltano/lib/python3.9/site-packages/meltano/cli/elt.py", line 296, in _run_extract_load await singer_runner.run( File "/Users/ananyasingh/Code/Personal/meltano-projects/.venv/meltano/lib/python3.9/site-packages/meltano/core/runner/singer.py", line 224, in run await self.invoke( File "/Users/ananyasingh/Code/Personal/meltano-projects/.venv/meltano/lib/python3.9/site-packages/meltano/core/runner/singer.py", line 206, in invoke raise RunnerError("Loader failed", {PluginType.LOADERS: target_code}) meltano.core.runner.RunnerError: Loader failed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/Users/ananyasingh/Code/Personal/meltano-projects/.venv/meltano/lib/python3.9/site-packages/meltano/cli/elt.py", line 228, in _redirect_output yield File "/Users/ananyasingh/Code/Personal/meltano-projects/.venv/meltano/lib/python3.9/site-packages/meltano/cli/elt.py", line 249, in _run_elt raise CliError(f"ELT could not be completed: {err}") from err meltano.cli.utils.CliError: ELT could not be completed: Loader failed 2022-08-01T171542.530675Z [info ] ELT could not be completed: Loader failed cmd_type=elt job_id=greenhouse-to-redshift name=meltano run_id=03f08b22-98ac-477a-a3dc-df983d9458a3 stdio=stderr 2022-08-01T171542.542675Z [debug ] ELT could not be completed: Loader failed Traceback (most recent call last): File "/Users/ananyasingh/Code/Personal/meltano-projects/.venv/meltano/lib/python3.9/site-packages/meltano/cli/elt.py", line 240, in _run_elt await _run_extract_load(log, elt_context, output_logger) File "/Users/ananyasingh/Code/Personal/meltano-projects/.venv/meltano/lib/python3.9/site-packages/meltano/cli/elt.py", line 296, in _run_extract_load await singer_runner.run( File "/Users/ananyasingh/Code/Personal/meltano-projects/.venv/meltano/lib/python3.9/site-packages/meltano/core/runner/singer.py", line 224, in run await self.invoke( File "/Users/ananyasingh/Code/Personal/meltano-projects/.venv/meltano/lib/python3.9/site-packages/meltano/core/runner/singer.py", line 206, in invoke raise RunnerError("Loader failed", {Plugin…
I tried putting 
primary_keys = None
/ 
primary_keys = []
in the defined stream but still giving the same error.
e
it’s not finding 
dbname
in the target config
a
umm it is working fine with rest of the streams except this one. so, I am not sure if it’s actually 
dbname
responsible for the error. apologies, checked the code you shared. is there a way to handle this?
my understanding is redshift is not allowing to have a stream without a primary_key defined in it. I saw similar queries on SF here but not on Redshift.
e
well there are a few clues pointing to redshift config being the problem: This log line is coming from the line of code I linked above:
Copy code
2022-08-01T17:15:42.497619Z [info     ]    * Required key is missing from config: [dbname] cmd_type=loader job_id=greenhouse-to-redshift name=target-redshift run_id=03f08b22-98ac-477a-a3dc-df983d9458a3 stdio=stderr
This message actually says the configuration is invalid:
Copy code
2022-08-01T17:15:42.497501Z [info     ] time=2022-08-01 22:45:42 name=target_redshift level=ERROR message=Invalid configuration: cmd_type=loader job_id=greenhouse-to-redshift name=target-redshift run_id=03f08b22-98ac-477a-a3dc-df983d9458a3 stdio=stderr
Can you confirm that you’re setting 
dbname
? You can use 
meltano config target-redshift list
a
yes, 
dbname
is not none.
ahh okay! got it! 🙇‍♀️ Thanks for all the help 🙏 although I am still getting error for 
primary key
basically this one was my original error. 😟 error:
Copy code
2022-08-01T17:54:55.443843Z [info     ]     return super(DictCursor, self).execute(query, vars) cmd_type=loader job_id=greenhouse-to-redshift name=target-redshift run_id=751eb7f9-b34a-425b-8387-407048c68824 stdio=stderr
2022-08-01T17:54:55.443963Z [info     ] psycopg2.errors.UndefinedColumn: column "i" named in key does not exist cmd_type=loader job_id=greenhouse-to-redshift name=target-redshift run_id=751eb7f9-b34a-425b-8387-407048c68824 stdio=stderr
2022-08-01T17:54:55.444076Z [info     ]                                cmd_type=loader job_id=greenhouse-to-redshift name=target-redshift run_id=751eb7f9-b34a-425b-8387-407048c68824 stdio=stderr
2022-08-01T17:54:55.493055Z [debug    ] Deleted configuration at /Users/ananyasingh/Code/Remote/etl/.meltano/run/elt/greenhouse-to-redshift/751eb7f9-b34a-425b-8387-407048c68824/target.aab7e180-99da-473b-99f7-f480e57fa5d0.config.json
2022-08-01T17:54:55.493533Z [debug    ] Deleted configuration at /Users/ananyasingh/Code/Remote/etl/.meltano/run/elt/greenhouse-to-redshift/751eb7f9-b34a-425b-8387-407048c68824/tap.2cfa280f-188a-4f21-8c4d-3a60252d45b2.config.json
2022-08-01T17:54:55.493634Z [error    ] Loading failed                 code=1 job_id=greenhouse-to-redshift message=(see above) name=meltano run_id=751eb7f9-b34a-425b-8387-407048c68824
2022-08-01T17:54:55.496981Z [debug    ] ELT could not be completed: Loader failed
In stream, I have set 
primary_keys = []
and in meltano.yml file
Copy code
- name: tap-greenhouse
    namespace: tap_greenhouse
    pip_url: -e extract/tap-greenhouse
    executable: tap-greenhouse
    capabilities:
    - state
    - catalog
    - discover
    settings:
    - name: api_key
    - name: start_date
    metadata:
      activityfeed:
        key_properties: null
I think I am doing something wrong here as am still a noob here.. 😅
e
@ananya is this a tap you’re creating using the SDK?
a
@edgar_ramirez_mondragon yes
hey-hey @edgar_ramirez_mondragon! just a nudge here 🙂
e
column "i" named in key does not exist
@ananya I’m not sure where column 
i
might be coming from 🤔
Open in Slack
PreviousNext
Powered by