https://linen.dev logo
Join Slack
Powered by
Hello all I'm worling with meltano 3.1.0 and it wa...
# troubleshooting
i
Hello all I'm worling with meltano 3.1.0 and it was working witthout problems - I"m using tap-salesforce and target-postgres and it gives me a strange error - posted the error in the thread
2023-12-01T17:03:24.073009Z [*info*   ] *TypeError: PostgresConnector.prepare_column() missing 1 required positional argument: 'column_name'* cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
2023-12-01T17:03:24.256505Z [*error*  ]
╭─────────────────────────────── *Traceback (most recent call last)* ────────────────────────────────╮
│ /home/airflow/.local/pipx/venvs/meltano/lib/python3.10/site-packages/meltano/core/logging/*output* │
│ *_logger.py*:207 in redirect_logging                                │
│                                                 │
│  204 │  │  │  *ignore_errors,                                │
│  205 │  │  )                                         │
│  206 │  │  try:                                        │
│ ❱ 207 │  │  │  yield                                     │
│  208 │  │  except ignored_errors: # noqa: WPS329                       │
│  209 │  │  │  raise                                     │
│  210 │  │  except RunnerError as err:                             │
│                                                 │
│ /home/airflow/.local/pipx/venvs/meltano/lib/python3.10/site-packages/meltano/core/block/*extract_* │
│ *load.py*:444 in run                                        │
│                                                 │
`│ 441 │ │ │ # TODO: legacy 
meltano elt
style logging should be deprecated │` 
│  442 │  │  │  legacy_log_handler = self.output_logger.out("meltano", logger)         │
│  443 │  │  │  with legacy_log_handler.redirect_logging():                  │
│ ❱ 444 │  │  │  │  await self.run_with_job()                         │
│  445 │  │  │  │  return                                   │
│  446 │  │  else:                                       │
│  447 │  │  │  logger.warning(                                │
│                                                 │
│ /home/airflow/.local/pipx/venvs/meltano/lib/python3.10/site-packages/meltano/core/block/*extract_* │
│ *load.py*:476 in run_with_job                                   │
│                                                 │
│  473 │  │                                            │
│  474 │  │  with closing(self.context.session) as session:                   │
│  475 │  │  │  async with job.run(session):                          │
│ ❱ 476 │  │  │  │  await self.execute()                            │
│  477 │                                              │
│  478 │  async def terminate(self, graceful: bool = False) -> None:               │
│  479 │  │  """Terminate an in flight ExtractLoad execution, potentially disruptive.      │
│                                                 │
│ /home/airflow/.local/pipx/venvs/meltano/lib/python3.10/site-packages/meltano/core/block/*extract_* │
│ *load.py*:436 in execute                                      │
│                                                 │
│  433 │  │  async with self._start_blocks():                          │
│  434 │  │  │  await self._link_io()                             │
│  435 │  │  │  manager = ELBExecutionManager(self)                      │
│ ❱ 436 │  │  │  await manager.run()                              │
│  437 │                                              │
│  438 │  async def run(self) -> None:                              │
│  439 │  │  """Run the ELT task."""                              │
`│ …
e
@Isoctcolo is your plugin pointing to the 
main
branch in 
pip_url
? Some breaking changes may have landed on that branch recently. cc @visch @sebastian in case you folks know what's going on
i
I have it installed via pipx
normally installed via meltano add
@edgar_ramirez_mondragon where can I check which version of the plugin I have installed
v
looks like bad code, uhhh peeking now
I ran the latest version of the code and a few quick tests that definitely call 
prepare_column
and I'm not seeing that
@Isoctcolo can you share your 
meltano.yml
, and then run 
meltano install --clean
just to be sure we have you on the latest version
i
sure
v
Our tests pass and they "should" do almost all operations especially these "straight forward" ones but it's always a possible we're missing a code path doesn't seem likely with that stack trace though
It does look like meltanohub is now pointed to 
pypi
which is an older version (we haven't done a release in a bit) but that error still shouldn't be getting hit. I'll try using the pypi version
i
here is the meltano.yml
meltano.yml
v
meltanolabs-target-postgres~=0.0.7
is not from 
meltano add
so maybe you hard coded that version? Try removing that version modifier
Even with that version I can't hit the bug so I'm a bit confused
~=
I'm not sure exactly how that solves for 0.0.* with semver it might still pull the latest. I'm definitely confused as I can't replicate the issue. I should definietly release the latest version to pypi regardless but I doubt that fixes this
i
I removed the 0.0.7 version and try it again with install --clean
v
We haven't released a new version (we have been making updates) so I don't think those changes are related to your issue Also can you print out the full stack trace we are missing the 
target-postgres
portion of the trace here
i
I think I found the issue 
2023-12-01T19:02:28.458822Z [*debug*  ] *Project is read-only.*
2023-12-01T19:02:28.473831Z [*debug*  ] */etc/timezone found, contents:*
Etc/UTC
 
2023-12-01T19:02:28.474209Z [*debug*  ] */etc/localtime found*
2023-12-01T19:02:28.474904Z [*debug*  ] *2 found:*
{'/etc/timezone': 'Etc/UTC', '/etc/localtime is a symlink to': 'Etc/UTC'}
 
2023-12-01T19:02:28.481480Z [*info*   ] *Environment 'dev' is active*
2023-12-01T19:02:28.497959Z [*debug*  ] *Creating DB engine for project at '/home/airflow/builds/93f21a7/meltano/dvload' with DB xxxxx*
2023-12-01T19:02:28.508433Z [*debug*  ] *couldn't import psycopg 'c' implementation: No module named 'psycopg_c'*
2023-12-01T19:02:28.687301Z [*debug*  ] *Found plugin parent*       parent=tap-salesforce plugin=tap-salesforce source=<DefinitionSource.LOCKFILE: 4>
2023-12-01T19:02:28.687801Z [*debug*  ] *Found plugin parent*       parent=tap-salesforce plugin=tap-salesforce--bulk source=<DefinitionSource.INHERITED: 8>
2023-12-01T19:02:29.313010Z [*debug*  ] *Found plugin parent*       parent=target-postgres plugin=target-postgres source=<DefinitionSource.LOCKFILE: 4>
2023-12-01T19:02:29.583072Z [*debug*  ] *Variable '$TARGET_POSTGRES_SCHEMA' is not set in the provided env dictionary.*
2023-12-01T19:02:29.584589Z [*info*   ] *Running extract & load...*    name=meltano run_id=e243b37c-8f8f-403d-b844-1b41fb00c826 state_id=2023-12-01T190228--tap-salesforce--bulk--target-postgres
I insalled it via pipx and in the virtual environment is the psycopg-binary installed
v
yeah that make sense but I don't get where
Copy code
2023-12-01T17:03:24.073009Z [info   ] TypeError: PostgresConnector.prepare_column() missing 1 required positional argument: 'column_name' cmd_type=elb consumer=True name=target-postgres producer=False stdio=stderr string_id=target-postgres
would come from but I"ll just let it go, glad it's not the target!
i
I printed all the stack trace I got
Just one basic question when I install it via pipx it create a pyenv and installs all the necessary packages - the strange thing it was working before - and the psycopg-binary is installed airflow@airflow:*~/.local/pipx/venvs*$ source meltano/bin/activate (meltano) airflow@airflow:*~/.local/pipx/venvs*$ pip freeze | grep psycopg psycopg==3.1.8 psycopg-binary==3.1.8 *psycopg*2==2.9.5 *psycopg*2-binary==2.9.5
in my opinion it shouldn't ask for the c file
e
fwiw 
couldn't import psycopg 'c' implementation: No module named 'psycopg_c'
 is not an error, just a debug log: https://github.com/psycopg/psycopg/issues/665#issuecomment-1772281448
i
ok got it - but at the end I have the error and the load doesn't work
I found some more traces
v
That's very helpful.
@edgar_ramirez_mondragon against releasing target-postgres as is? I don't think there's any new major bugs? The big switch is JSONB
Let me dive a bit this is triggered from Activate Version it looks like we may have a bug back on that version and on the main branch version, I'm a bit confused as I think we have this tested going to dive
e
I think the root issue is that the SDK and target-postgres have incompatible implementations of `prepare_column`: SDK:
Copy code
def prepare_column(
        self,
        full_table_name: str,
        column_name: str,
        sql_type: sqlalchemy.types.TypeEngine,
    ) -> None:
target-postgres:
Copy code
def prepare_column(  # type: ignore[override]
        self,
        schema_name: str,
        table: sqlalchemy.Table,
        column_name: str,
        sql_type: sqlalchemy.types.TypeEngine,
        connection: sqlalchemy.engine.Connection,
        column_object: sqlalchemy.Column | None = None,
    ) -> None:
First one has 2 positional-only, second one has 3. The 
# type: ignore[override]
is probably hiding a real error.
against releasing target-postgres as is? I don't think there's any new major bugs? The big switch is JSONB.
Not at all against 🙂
v
sebastian is diving. It looks like the code is just wrong, but the pytest passing and no one having an issue for a while is confusing. We'll figure it out and get a release out @Isoctcolo
i
ok perfect just let me know and I test it
Just FYI it seems to be a corner case because I deploy it automatically and on my 3 other environments it is running without issues also just one env it is throwing me the error
s
Status update: identified and replicated the issue; working on a fix.
v
@Isoctcolo if you set the 
add_record_metadata
config to true your stuff will work
i
ok
@visch Just FYI it worked switching to add_record_metadata
@sebastian Just out of curiosity can you point me to the fix - As I want to get more involved in development
v
There wasn't a fix we put up, it's documented in the readme that it needs to be on, but the "fix" would be we should probably fail more clearly when that happens
i
ok perfect understood thanks a lot
Open in Slack
PreviousNext
Powered by