https://linen.dev logo
Join Slack
Powered by
Hi, I'm making progress with getting Meltano set u...
# announcements
a
Hi, I'm making progress with getting Meltano set up and have been trying to get the 
tap-surveymonkey
Singer tap working. I've installed it but when I run the pipeline from the UI I get an error. The error log says:
Copy code
Traceback (most recent call last):
  File "/Code/meltano_test/testproj/.meltano/loaders/target-postgres/venv/bin/target-postgres", line 11, in <module>
    load_entry_point('target-postgres==1.1.3', 'console_scripts', 'target-postgres')()
  File "/Users/allanwhatmough/Code/meltano_test/testproj/.meltano/loaders/target-postgres/venv/lib/python3.7/site-packages/target_postgres/__init__.py", line 138, in main
    state = persist_lines(config, input)
  File "/Code/meltano_test/testproj/.meltano/loaders/target-postgres/venv/lib/python3.7/site-packages/target_postgres/__init__.py", line 100, in persist_lines
    stream_to_sync[stream] = DbSync(config, o)
  File "/Code/meltano_test/testproj/.meltano/loaders/target-postgres/venv/lib/python3.7/site-packages/target_postgres/db_sync.py", line 118, in __init__
    self.flatten_schema = flatten_schema(stream_schema_message['schema'])
  File "/Code/meltano_test/testproj/.meltano/loaders/target-postgres/venv/lib/python3.7/site-packages/target_postgres/db_sync.py", line 65, in flatten_schema
    for k, v in d['properties'].items():
KeyError: 'properties'
CRITICAL [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/Code/meltano_test/testproj/.meltano/extractors/tap-surveymonkey/venv/bin/tap-surveymonkey", line 11, in <module>
    load_entry_point('tap-surveymonkey==0.1.0', 'console_scripts', 'tap-surveymonkey')()
  File "/Code/meltano_test/testproj/.meltano/extractors/tap-surveymonkey/venv/lib/python3.7/site-packages/singer/utils.py", line 229, in wrapped
    return fnc(*args, **kwargs)
  File "/Code/meltano_test/testproj/.meltano/extractors/tap-surveymonkey/venv/lib/python3.7/site-packages/tap_surveymonkey/__init__.py", line 290, in main
    sync(args.config, state, catalog)
  File "/Code/meltano_test/testproj/.meltano/extractors/tap-surveymonkey/venv/lib/python3.7/site-packages/tap_surveymonkey/__init__.py", line 149, in sync
    mdata)
  File "/Code/meltano_test/testproj/.meltano/extractors/tap-surveymonkey/venv/lib/python3.7/site-packages/tap_surveymonkey/__init__.py", line 174, in sync_survey_details
    [survey_details]
  File "/Code/meltano_test/testproj/.meltano/extractors/tap-surveymonkey/venv/lib/python3.7/site-packages/singer/messages.py", line 248, in write_records
    write_record(stream_name, record)
  File "/Code/meltano_test/testproj/.meltano/extractors/tap-surveymonkey/venv/lib/python3.7/site-packages/singer/messages.py", line 237, in write_record
    time_extracted=time_extracted))
  File "/Code/meltano_test/testproj/.meltano/extractors/tap-surveymonkey/venv/lib/python3.7/site-packages/singer/messages.py", line 227, in write_message
    sys.stdout.flush()
BrokenPipeError: [Errno 32] Broken pipe
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe
ELT could not complete, an error happened during the process: Subprocesses didn't exit cleanly: tap(120), target(1)
I think the tap isn't configured properly i.e. the properties aren't being set. I ran 
meltano select tap-surveymonkey --list
and got this output:
Copy code
Legend:
	selected
	excluded
	automatic

Enabled patterns:
	survey_details.*

Selected properties:
I've managed to run this tap successfully outside of Meltano but had to do some manual editing of the 
properties.json
file to set certain values e.g. 
additionalProperties
on the 
survey_details
JSON object.
Is there a way to copy an existing properties JSON file which Meltano will then pick up and use to set the config used when running the tap?
y
@al_whatmough you can check the settings you have set for your custom tap by running:
Copy code
# List the available settings for the plugin.
meltano config tap-surveymonkey list
You should be able to see all the configuration parameters you have set. And then you should be able to see values for all those parameters by running the following:
Copy code
# Displays the plugin's configuration.
meltano config tap-surveymonkey
For more info on that: https://www.meltano.com/docs/command-line-interface.html#config
Finally, the names of the parameters on the settings should match the ones required by 
tap-surveymonkey
For example, if you are using the official Singer.io singerio-tap-surveymonkey: https://github.com/SurveyMonkey/singerio-tap-surveymonkey I'd expect your meltano.yml to look more or less like the following:
Copy code
plugins:
  extractors:
  - capabilities:
    - discover
    - properties
    - state
    executable: tap-surveymonkey
    name: tap-surveymonkey
    namespace: tap_surveymonkey
    pip_url: 'git+<https://github.com/SurveyMonkey/singerio-tap-surveymonkey.git>'
    settings:
      - name: access_token
        env: SURVEYMONKEY_ACCESS_TOKEN
      - name: start_date
        env: SURVEYMONKEY_START_DATE
      - name: survey_id
        env: SURVEYMONKEY_SURVEY_ID
and you have a .env
Copy code
export SURVEYMONKEY_ACCESS_TOKEN="EW..."
export SURVEYMONKEY_START_DATE="2017-01-01T00:00:00Z"
export SURVEYMONKEY_SURVEY_ID="112213614"
With respect to the 
properties.json
file: (1) You can check the properties file generated by meltano on 
.meltano/run/tap-surveymonkey/tap.properties.json
In that way you can check for issues with setting up your tap. (2) You can copy paste your own working 
properties.json
in there and rename it to 
tap.properties.json
and I think that Meltano will skip on creating a new one and use that instead. But if you have not properly set the selection rules, we may go in your file and unselect selected streams
If you keep on having issues after checking my comments, can you please post here your 
meltano.yml
?
And also the 
config.json
you used to make 
tap-surveymonkey
to work on its own? (with the access token value removed of course)
Please test everything using the cli to make sure that this one works and we'll then figure out any issues we may have with Meltano UI
a
Hi @yannis_roussos, thanks for you reply - sorry that mine was so slow! I was away
I checked and the .env file and meltano.yml file looks OK
Here's a snippet from my 
meltano.yml
Copy code
plugins:
  extractors:
  - capabilities:
    - discover
    - properties
    - state
    executable: tap-surveymonkey
    name: tap-surveymonkey
    namespace: tap_surveymonkey
    pip_url: git+<https://github.com/SurveyMonkey/singerio-tap-surveymonkey>
    settings:
    - env: ACCESS_TOKEN
      name: access_token
    - env: START_DATE
      name: start_date
    - env: SURVEY_ID
      name: survey_id
however, I'm still getting the same error. I copied my existing (working) tap properties.json to 
.meltano/run/tap-surveymonkey/tap.properties.json
and re-ran it using the CLI, but I'm still seeing this error:
Copy code
File "/Users/allanwhatmough/Code/meltano_test/testproj/.meltano/loaders/target-postgres/venv/lib/python3.7/site-packages/target_postgres/db_sync.py", line 65, in flatten_schema
    for k, v in d['properties'].items():
KeyError: 'properties'
y
Ok then this seems like something is not properly sent by 
tap-surveymonkey
. The error you get is when 
target-postgres
goes through the schema provided by 
tap-surveymonkey
for one of the extracted Entities and it seems to get an improperly formatted schema (it should have properties). Can you please run 
tap-surveymonkey
by itself and provide here the schema messages sent?
Copy code
meltano invoke tap-surveymonkey
a
Hi @yannis_roussos, I'm still having trouble with this - I just created an issue here https://gitlab.com/meltano/meltano/issues/1332 which links to a Gitlab snippet with the output of 
meltano invoke tap-surveymonkey
y
Thank you @al_whatmough, I will check it and continue the discussion in the issue
18 Views
Open in Slack
PreviousNext
Powered by