I was able to get everything running except now I ...
# announcements
j
I was able to get everything running except now I get this error:
Copy code
meltano       | Connection lost
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/meltano/core/logging/utils.py", line 69, in capture_subprocess_output
    writer.writeline(line.decode())
AttributeError: 'StreamWriter' object has no attribute 'writeline'

tap-github    | INFO Filtered 2 paths during transforms as they were unsupported or not selected:
Connection lost
ERROR: 1
d
@jacob_schatz What version of Meltano are you using? That
AttributeError
should be getting caught: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/logging/utils.py#L70
j
Latest docker so
meltano:latest
Will find exact version in a second
@douwe_maan meltano, version 1.71.0
via docker
which uses in my docker-compose.yml file:
image: meltano/meltano:latest
Which looks like the latest version
So it appears to be working and the problem was (and this is probably something that can be caught better) was that when I was passing the
state.json
in the
--state
param, if I didn’t have all properties I was selecting in the state file it threw that error
So my
select
was
Copy code
select:
    - pull_requests.id
    - pull_request_reviews.state
    - pull_request_reviews.commit_id
    - pr_commits.commit.committer.email
    - pr_commits.commit.committer.name
    - pr_commits.files
    - pr_commits.commit.message
    - pr_commits.commit.comment_count
    - assignees.id
    - assignees.login
My state.json was:
Copy code
{
  "bookmarks": {
    "tactivos/murally": {
      "pull_requests": {
        "since": "2021-03-11T12:00:00.000000Z"
      },
      "pull_request_reviews": {
        "since": "2021-03-11T12:00:00.000000Z"
      },
      "review_comments": {
        "since": "2021-03-11T12:00:00.000000Z"
      },
      "pr_commits": {
        "since": "2021-03-11T12:00:00.000000Z"
      }
    }
  }
}
once I changed it to
Copy code
{
  "bookmarks": {
    "tactivos/murally": {
      "pull_requests": {
        "since": "2021-03-11T12:00:00.000000Z"
      },
      "pull_request_reviews": {
        "since": "2021-03-11T12:00:00.000000Z"
      },
      "review_comments": {
        "since": "2021-03-11T12:00:00.000000Z"
      },
      "pr_commits": {
        "since": "2021-03-11T12:00:00.000000Z"
      },
      "assignees": {
        "since": "2021-03-11T12:00:00.000000Z"
      }
    }
  }
}
Everything worked
d
Ah, interesting. I'm not sure why that would trigger the error that it did, but glad you figure it out
j
I mean I tried to target target-csv and I get the same error
So IDK
d
I wonder if we should be focusing on that "Connection lost" in the logs. That could indicate that the target died unexpectedly. Is there anything else in the logs that may be relevant?
j
```tap-github | WARNING Removed paths list: ['_links', 'active_lock_reason', 'assignee', 'assignees', 'author_association', 'auto_merge', 'base', 'comments_url', 'commits_url', 'diff_url', 'draft', 'head', 'html_url', 'issue_url', 'locked', 'merge_commit_sha', 'milestone', 'node_id', 'patch_url', 'requested_reviewers', 'requested_teams', 'review_comment_url', 'review_comments_url', 'statuses_url'] meltano | ERROR Connection lost Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/meltano/core/logging/utils.py", line 69, in capture_subprocess_output writer.writeline(line.decode()) AttributeError: 'StreamWriter' object has no attribute 'writeline' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/meltano/core/logging/output_logger.py", line 135, in redirect_logging yield File "/usr/local/lib/python3.6/site-packages/meltano/cli/elt.py", line 224, in _redirect_output yield File "/usr/local/lib/python3.6/site-packages/meltano/cli/elt.py", line 236, in _run_elt await _run_extract_load(elt_context, output_logger) File "/usr/local/lib/python3.6/site-packages/meltano/cli/elt.py", line 279, in _run_extract_load loader_out=loader_out_writer, File "/usr/local/lib/python3.6/site-packages/meltano/core/runner/singer.py", line 259, in run loader_out=loader_out, File "/usr/local/lib/python3.6/site-packages/meltano/core/runner/singer.py", line 178, in invoke raise failed_future.exception() File "/usr/local/lib/python3.6/site-packages/meltano/core/logging/utils.py", line 72, in capture_subprocess_output await writer.drain() File "/usr/local/lib/python3.6/asyncio/streams.py", line 339, in drain yield from self._protocol._drain_helper() File "/usr/local/lib/python3.6/asyncio/streams.py", line 210, in _drain_helper raise ConnectionResetError('Connection lost') ConnectionResetError: Connection lost tap-github | INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.2377016544342041, "tags": {"endpoint": "pr_commits", "http_status_code": 200, "status": "succeeded"}} [2021-03-31 202424,730] [1|MainThread|meltano.cli.utils] [DEBUG] Connection lost Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/meltano/core/logging/utils.py", line 69, in capture_subprocess_output writer.writeline(line.decode()) AttributeError: 'StreamWriter' object has no attribute 'writeline' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/meltano/cli/__init__.py", line 43, in main cli(obj={"project": None}) File "/usr/local/lib/python3.6/site-packages/click/core.py", line 829, in call return self.main(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/meltano/cli/params.py", line 23, in decorate return func(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/meltano/cli/params.py", line 57, in decorate func(project, *args, **kwargs) File "/usr/local/lib/python3.6/site-packages/meltano/cli/elt.py", line 136, in elt run_async(_run_job(project, job, session, context_builder, force=force)) File "/usr/local/lib/python3.6/site-packages/meltano/cli/utils.py", line 49, in run_async loop.run_until_complete(coro) File "/usr/local/lib/python3.6/asyncio/base_events.py", l…
d
OK, so
AttributeError: 'StreamWriter' object has no attribute 'writeline'
looks like a red herring
Any log lines related to the target? Starting with
target-foo |
?
j
target-foo?
d
target-jsonl, target-csv, what have you 🙂
j
oh ok one second
Copy code
[32mtarget-csv       | [0mINFO Sending version information to <http://singer.io|singer.io>. To disable sending anonymous usage data, set the config parameter "disable_collection" to true
[32mtarget-csv       | [0mTraceback (most recent call last):
[32mtarget-csv       | [0m  File "/project/testing_initative/.meltano/loaders/target-csv/venv/bin/target-csv", line 8, in <module>
[32mtarget-csv       | [0m    sys.exit(main())
[32mtarget-csv       | [0m  File "/project/testing_initative/.meltano/loaders/target-csv/venv/lib/python3.6/site-packages/target_csv.py", line 144, in main
[32mtarget-csv       | [0m    config.get('destination_path', ''))
[32mtarget-csv       | [0m  File "/project/testing_initative/.meltano/loaders/target-csv/venv/lib/python3.6/site-packages/target_csv.py", line 77, in persist_messages
[32mtarget-csv       | [0m    with open(filename, 'a') as csvfile:
[32mtarget-csv       | [0mFileNotFoundError: [Errno 2] No such file or directory: 'output/assignees-20210331T203436.csv'
So I don’t have an output directory maybe?
d
Ah yes I think that's it!
And the fact that you're seeing the ConnectionResetError is because of this bug: https://gitlab.com/meltano/meltano/-/issues/2526
j
yeah the error reporting leaves a lot to be desired 🙂
d
Issues please 😄
j
Will do
d
Thanks!