https://meltano.com/ logo
Channels
Powered by
#announcements
Title
# announcements
p

powerful-businessperson-37571

03/31/2021, 1:39 PM
I was able to get everything running except now I get this error:
Copy code
meltano       | Connection lost
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/meltano/core/logging/utils.py", line 69, in capture_subprocess_output
    writer.writeline(line.decode())
AttributeError: 'StreamWriter' object has no attribute 'writeline'

tap-github    | INFO Filtered 2 paths during transforms as they were unsupported or not selected:
Connection lost
ERROR: 1
r

ripe-musician-59933

03/31/2021, 5:07 PM
@powerful-businessperson-37571 What version of Meltano are you using? That 
AttributeError
should be getting caught: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/logging/utils.py#L70
p

powerful-businessperson-37571

03/31/2021, 5:15 PM
Latest docker so 
meltano:latest
Will find exact version in a second
@ripe-musician-59933 meltano, version 1.71.0
via docker
which uses in my docker-compose.yml file: 
image: meltano/meltano:latest
Which looks like the latest version
So it appears to be working and the problem was (and this is probably something that can be caught better) was that when I was passing the 
state.json
in the 
--state
param, if I didn’t have all properties I was selecting in the state file it threw that error
So my 
select
was
Copy code
select:
    - pull_requests.id
    - pull_request_reviews.state
    - pull_request_reviews.commit_id
    - pr_commits.commit.committer.email
    - pr_commits.commit.committer.name
    - pr_commits.files
    - pr_commits.commit.message
    - pr_commits.commit.comment_count
    - assignees.id
    - assignees.login
My state.json was:
Copy code
{
  "bookmarks": {
    "tactivos/murally": {
      "pull_requests": {
        "since": "2021-03-11T12:00:00.000000Z"
      },
      "pull_request_reviews": {
        "since": "2021-03-11T12:00:00.000000Z"
      },
      "review_comments": {
        "since": "2021-03-11T12:00:00.000000Z"
      },
      "pr_commits": {
        "since": "2021-03-11T12:00:00.000000Z"
      }
    }
  }
}
once I changed it to
Copy code
{
  "bookmarks": {
    "tactivos/murally": {
      "pull_requests": {
        "since": "2021-03-11T12:00:00.000000Z"
      },
      "pull_request_reviews": {
        "since": "2021-03-11T12:00:00.000000Z"
      },
      "review_comments": {
        "since": "2021-03-11T12:00:00.000000Z"
      },
      "pr_commits": {
        "since": "2021-03-11T12:00:00.000000Z"
      },
      "assignees": {
        "since": "2021-03-11T12:00:00.000000Z"
      }
    }
  }
}
Everything worked
r

ripe-musician-59933

03/31/2021, 7:55 PM
Ah, interesting. I'm not sure why that would trigger the error that it did, but glad you figure it out
p

powerful-businessperson-37571

03/31/2021, 8:27 PM
I mean I tried to target target-csv and I get the same error
So IDK
r

ripe-musician-59933

03/31/2021, 8:31 PM
I wonder if we should be focusing on that "Connection lost" in the logs. That could indicate that the target died unexpectedly. Is there anything else in the logs that may be relevant?
p

powerful-businessperson-37571

03/31/2021, 8:31 PM
Copy code
tap-github       | WARNING Removed paths list: ['_links', 'active_lock_reason', 'assignee', 'assignees', 'author_association', 'auto_merge', 'base', 'comments_url', 'commits_url', 'diff_url', 'draft', 'head', 'html_url', 'issue_url', 'locked', 'merge_commit_sha', 'milestone', 'node_id', 'patch_url', 'requested_reviewers', 'requested_teams', 'review_comment_url', 'review_comments_url', 'statuses_url']
meltano          | ERROR Connection lost
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/meltano/core/logging/utils.py", line 69, in capture_subprocess_output
    writer.writeline(line.decode())
AttributeError: 'StreamWriter' object has no attribute 'writeline'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/meltano/core/logging/output_logger.py", line 135, in redirect_logging
    yield
  File "/usr/local/lib/python3.6/site-packages/meltano/cli/elt.py", line 224, in _redirect_output
    yield
  File "/usr/local/lib/python3.6/site-packages/meltano/cli/elt.py", line 236, in _run_elt
    await _run_extract_load(elt_context, output_logger)
  File "/usr/local/lib/python3.6/site-packages/meltano/cli/elt.py", line 279, in _run_extract_load
    loader_out=loader_out_writer,
  File "/usr/local/lib/python3.6/site-packages/meltano/core/runner/singer.py", line 259, in run
    loader_out=loader_out,
  File "/usr/local/lib/python3.6/site-packages/meltano/core/runner/singer.py", line 178, in invoke
    raise failed_future.exception()
  File "/usr/local/lib/python3.6/site-packages/meltano/core/logging/utils.py", line 72, in capture_subprocess_output
    await writer.drain()
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 339, in drain
    yield from self._protocol._drain_helper()
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 210, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
tap-github       | INFO METRIC: {"type": "timer", "metric": "http_request_duration", "value": 0.2377016544342041, "tags": {"endpoint": "pr_commits", "http_status_code": 200, "status": "succeeded"}}
[2021-03-31 20:24:24,730] [1|MainThread|meltano.cli.utils] [DEBUG] Connection lost
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/meltano/core/logging/utils.py", line 69, in capture_subprocess_output
    writer.writeline(line.decode())
AttributeError: 'StreamWriter' object has no attribute 'writeline'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/meltano/cli/__init__.py", line 43, in main
    cli(obj={"project": None})
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/meltano/cli/params.py", line 23, in decorate
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/meltano/cli/params.py", line 57, in decorate
    func(project, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/meltano/cli/elt.py", line 136, in elt
    run_async(_run_job(project, job, session, context_builder, force=force))
  File "/usr/local/lib/python3.6/site-packages/meltano/cli/utils.py", line 49, in run_async
    loop.run_until_complete(coro)
  File "/usr/local/lib/python3.6/asyncio/base_events.py", line 488, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.6/site-packages/meltano/cli/elt.py", line 213, in _run_job
    await _run_elt(project, context_builder, output_logger)
  File "/usr/local/lib/python3.6/site-packages/meltano/cli/elt.py", line 236, in _run_elt
    await _run_extract_load(elt_context, output_logger)
  File "/usr/local/lib/python3.6/site-packages/meltano/cli/elt.py", line 279, in _run_extract_load
    loader_out=loader_out_writer,
  File "/usr/local/lib/python3.6/site-packages/meltano/core/runner/singer.py", line 259, in run
    loader_out=loader_out,
  File "/usr/local/lib/python3.6/site-packages/meltano/core/runner/singer.py", line 178, in invoke
    raise failed_future.exception()
  File "/usr/local/lib/python3.6/site-packages/meltano/core/logging/utils.py", line 72, in capture_subprocess_output
    await writer.drain()
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 339, in drain
    yield from self._protocol._drain_helper()
  File "/usr/local/lib/python3.6/asyncio/streams.py", line 210, in _drain_helper
    raise ConnectionResetError('Connection lost')
r

ripe-musician-59933

03/31/2021, 8:33 PM
OK, so 
AttributeError: 'StreamWriter' object has no attribute 'writeline'
looks like a red herring
Any log lines related to the target? Starting with 
target-foo |
?
p

powerful-businessperson-37571

03/31/2021, 8:33 PM
target-foo?
r

ripe-musician-59933

03/31/2021, 8:33 PM
target-jsonl, target-csv, what have you 🙂
p

powerful-businessperson-37571

03/31/2021, 8:34 PM
oh ok one second
Copy code
[32mtarget-csv       | [0mINFO Sending version information to <http://singer.io|singer.io>. To disable sending anonymous usage data, set the config parameter "disable_collection" to true
[32mtarget-csv       | [0mTraceback (most recent call last):
[32mtarget-csv       | [0m  File "/project/testing_initative/.meltano/loaders/target-csv/venv/bin/target-csv", line 8, in <module>
[32mtarget-csv       | [0m    sys.exit(main())
[32mtarget-csv       | [0m  File "/project/testing_initative/.meltano/loaders/target-csv/venv/lib/python3.6/site-packages/target_csv.py", line 144, in main
[32mtarget-csv       | [0m    config.get('destination_path', ''))
[32mtarget-csv       | [0m  File "/project/testing_initative/.meltano/loaders/target-csv/venv/lib/python3.6/site-packages/target_csv.py", line 77, in persist_messages
[32mtarget-csv       | [0m    with open(filename, 'a') as csvfile:
[32mtarget-csv       | [0mFileNotFoundError: [Errno 2] No such file or directory: 'output/assignees-20210331T203436.csv'
So I don’t have an output directory maybe?
r

ripe-musician-59933

03/31/2021, 8:36 PM
Ah yes I think that's it!
And the fact that you're seeing the ConnectionResetError is because of this bug: https://gitlab.com/meltano/meltano/-/issues/2526
p

powerful-businessperson-37571

03/31/2021, 8:37 PM
yeah the error reporting leaves a lot to be desired 🙂
r

ripe-musician-59933

03/31/2021, 8:37 PM
Issues please 😄
p

powerful-businessperson-37571

03/31/2021, 8:37 PM
Will do
r

ripe-musician-59933

03/31/2021, 8:51 PM
Thanks!
Powered by