Hi all! I have few questions around singer tap development which I was not able find in the documentation or sample codes. 1. Does

poetry install &&  poetry run my-tap --config .secrets/config.json

refer to meltano.yml file in anyway internally? From my understanding it doesn’t. If so, then how to I mark my streams as selected ? I am getting this error ->

Skipping deselected stream <my-stream>

every time(from tap_base.py). 2. When I try to run the same project with meltano instead of poetry, I face issues.

ImportError: cannot import name 'cached_property' from 'functools' (/Users/me/.pyenv/versions/3.7.15/lib/python3.7/functools.py)

. If I remove this cached_property also, I face some other issues. I am guessing its due to difference in python versions or poetry environments(running 3.11)? I see that meltano is trying to use python 3.7.15, but my global python version is managed by

pyenv

and is set to 3.9.2.

python3 --version

= Python 3.9.2.

python --version

= Python 3.9.2. How can I fix this conflict ? 3. When running

poetry run my-tap --test

, then I get

MaxRecordsLimitException

, which looks like it happens when I reach

ABORT_AT_RECORD_COUNT

. In my case, it is set to 1(not sure how), so I don’t get any records in the output, straight away this exception. Thanks in advance. 🙂

1. No, try

poetry run my-tap --help

/

poetry run my-tap --config config.json --catalog catalog.json

gen the catalog.json with meltano via

meltano invoke --dump=catalog my-tap > catalog.json

2. You need to fix the python version issue, if you give the command you're running that hits this error we can point to things to help, but it's your environment, a full reinstall would work too 3. Can you show how you're running the tap? maybe it's a dry run? re https://github.com/search?q=repo%3Ameltano%2Fsdk%20ABORT_AT_RECORD_COUNT&type=code

Hi Derek, For 2, I am running

meltano invoke

command only. Here is the complete log.

meltano invoke --dump=catalog tap-redshift > catalog.json
2023-06-27T12:52:49.526054Z [info     ] Environment 'test' is active
Need help fixing this problem? Visit <http://melta.no/> for troubleshooting steps, or to
join our friendly Slack community.

Could not dump catalog: Catalog discovery failed: command ['/Users/me/Documents/opensource/tap-redshift/.meltano/extractors/tap-redshift/venv/bin/tap-redshift', '--config', '/Users/me/Documents/opensource/tap-redshift/.meltano/run/tap-redshift/tap.fa054854-fc36-44be-a96d-79c22d0b4029.config.json', '--discover'] returned 1 with stderr:
 Traceback (most recent call last):
  File "/Users/me/Documents/opensource/tap-redshift/.meltano/extractors/tap-redshift/venv/bin/tap-redshift", line 5, in <module>
    from tap_redshift.tap import TapRedshift
  File "/Users/me/Documents/opensource/tap-redshift/tap_redshift/tap.py", line 6, in <module>
    from functools import cached_property
ImportError: cannot import name 'cached_property' from 'functools' (/Users/me/.pyenv/versions/3.7.15/lib/python3.7/functools.py)

This is my local environment, not sure why its referencing to 3.7.15 specifically. For 3, yes I ran

poetry run my-tap --test

, is that expected behaviour ? I wanted to test if i’m getting some records.

There's a ton of questions wrapped up into your first question and follow ups! Maybe seperate them out into individual's. For your environment there's a large number of things to try sorry I can't run you through them all 😕 , but yes you need to figure out why it has python 3.7 installed

poetry run my-tap --test

I'm not sure exactly what the behavior should be as I don't know how your tap is setup I personally don't run test too often but it is nice to use to check that your credentials are working. Hard to say if it's "expected" or not, probably not but it depends on your tap development 🤷

Okay, i uninstalled all the other versions of py and it seems to be fine now. So let’s try to limit the scope of this thread to 1st question itself. Do I need to generate catalog with poetry or meltano?

meltano invoke --dump=catalog my-tap > catalog.json

-> gives

Could not find catalog

I am able to generate with

poetry run my-tap --config .secrets/config.json --discover > ./catalog.json

.

Both work, without meltano you have to manually update your catalog

Then why is it saying Could not find catalog 🤔 with meltano way

Hard to know for sure without seeing your code, meltano.yml

File "/usr/local/lib/python3.7/site-packages/meltano/cli/invoke.py", line 214, in dump_file
    content = await invoker.dump(file_id)
  File "/usr/local/lib/python3.7/site-packages/meltano/core/plugin_invoker.py", line 511, in dump
    raise err.__cause__ from None  # noqa: WPS469, WPS609
  File "/usr/local/lib/python3.7/site-packages/meltano/core/plugin_invoker.py", line 426, in _invoke
    yield (popen_args, popen_options, popen_env)
  File "/usr/local/lib/python3.7/site-packages/meltano/core/plugin_invoker.py", line 451, in invoke_async
    env=popen_env,
  File "/usr/local/Cellar/python@3.7/3.7.17/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/subprocess.py", line 217, in create_subprocess_exec
    stderr=stderr, **kwds)
  File "/usr/local/Cellar/python@3.7/3.7.17/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 1544, in subprocess_exec
    bufsize, **kwargs)
  File "/usr/local/Cellar/python@3.7/3.7.17/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/unix_events.py", line 190, in _make_subprocess_transport
    **kwargs)
  File "/usr/local/Cellar/python@3.7/3.7.17/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_subprocess.py", line 37, in __init__
    stderr=stderr, bufsize=bufsize, **kwargs)
  File "/usr/local/Cellar/python@3.7/3.7.17/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/unix_events.py", line 775, in _start
    universal_newlines=False, bufsize=bufsize, **kwargs)
  File "/usr/local/Cellar/python@3.7/3.7.17/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/usr/local/Cellar/python@3.7/3.7.17/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 1567, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/me/Documents/opensource/tap-redshift/.meltano/extractors/tap-redshift/venv/bin/tap-redshift': '/Users/me/Documents/opensource/tap-redshift/.meltano/extractors/tap-redshift/venv/bin/tap-redshift'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/meltano/cli/invoke.py", line 131, in invoke
    print_var=print_var,
  File "/usr/local/Cellar/python@3.7/3.7.17/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/local/Cellar/python@3.7/3.7.17/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.7/site-packages/meltano/cli/invoke.py", line 165, in _invoke
    await dump_file(invoker, dump)
  File "/usr/local/lib/python3.7/site-packages/meltano/cli/invoke.py", line 216, in dump_file
    raise CliError(f"Could not find {file_id}") from err
meltano.cli.utils.CliError: Could not find catalog

I see the following error when running with debug log

FileNotFoundError: [Errno 2] No such file or directory: '/Users/me/Documents/opensource/tap-redshift/.meltano/extractors/tap-redshift/venv/bin/tap-redshift': '/Users/me/Documents/opensource/tap-redshift/.meltano/extractors/tap-redshift/venv/bin/tap-redshift'

Share your

meltano.yml

as well

this is pretty much self generated

after you added that config to your meltano.yml did you run

meltano install extractor tap-redshift --clean

? It seems like the venv thats created when you install a plugin isnt present in your project yet

Yup, that was spot on @pat_nadolny thanks 🙂