https://linen.dev logo
Join Slack
Powered by
Hi, can someone help me understand if I like to us...
# troubleshooting
m
Hi, can someone help me understand if I like to use Elementary within Meltano, I would need to fork https://github.com/potloc/elementary-ext and adjust how it is needed? If yes, what should I update to add the following Elementary command 
edr run-operation upload-source-freshness --project-dir <path to your project>
?
I have tried various option. The last configuration of my meltano.yml is;
Copy code
- name: elementary
    variant: elementary
    pip_url: elementary-data[athena]==0.16.0 git+<https://github.com/datarts-tech/elementary-ext.git@create_monitor_command>
    config:
      file-path: $MELTANO_PROJECT_ROOT/transform/dbt_athena/edr_target/elementary_report.html
      profiles-dir: $MELTANO_PROJECT_ROOT/transform/dbt_athena
      project_dir: $MELTANO_PROJECT_ROOT/transform/dbt_athena
    # TODO: should be updated <https://github.com/meltano/hub/blob/main/_data/meltano/utilities/elementary/elementary.yml>
    commands:
      monito:
        args: monitor
        executable: elementary_extension
      freshness:
        args: "run-operation upload-source-freshness --project-dir $MELTANO_PROJECT_ROOT/transform/dbt_athena"
        executable: elementary_extension
But if I run freshness, I get the following:
Copy code
root@6f669d784cdd:/project# meltano invoke elementary:freshness
2024-11-12T16:50:18.477728Z [info     ] Environment 'dev' is active
Usage: elementary_extension [OPTIONS] COMMAND [ARGS]...
Try 'elementary_extension --help' for help.

Error: No such command 'run-operation'.
v
Does 
meltano invoke elementary --help
give you a list of commands you can run? Try running this directly to see if it works 
meltano invoke elementary run-operation upload-source-freshness --project-dir ./transform/dbt_athena
Just make sure your pwd is in the root meltano dir
If that give you the same error then it's definietly an issue with the extension not passing through or whatever it passing through to not having 
run-operation
e
Can you try setting 
executable
to 
elementary_invoker
?
m
Tried the first one:
Copy code
root@5b27527642c1:/project# meltano invoke elementary run-operation upload-source-freshness --profiles-dir ./transform/dbt_athena/ --project-dir ./transform/dbt_athena/
2024-11-13T09:51:55.983767Z [info     ] Environment 'dev' is active
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

You are using Elementary 0.16.0, however version 0.16.1 is available.
Consider upgrading by running: "/project/.meltano/utilities/elementary/venv/bin/python -m pip install --upgrade elementary-data"

Any feedback and suggestions are welcomed! join our community here - <https://bit.ly/slack-elementary>

2024-11-13 09:52:00 — INFO — Running with edr=0.16.0
Traceback (most recent call last):
  File "/project/.meltano/utilities/elementary/venv/bin/edr", line 8, in <module>
    sys.exit(cli())
  File "/project/.meltano/utilities/elementary/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/project/.meltano/utilities/elementary/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/project/.meltano/utilities/elementary/venv/lib/python3.10/site-packages/elementary/cli/cli.py", line 67, in invoke
    return super().invoke(ctx)
  File "/project/.meltano/utilities/elementary/venv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/project/.meltano/utilities/elementary/venv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/project/.meltano/utilities/elementary/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/project/.meltano/utilities/elementary/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/project/.meltano/utilities/elementary/venv/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/project/.meltano/utilities/elementary/venv/lib/python3.10/site-packages/elementary/operations/cli.py", line 63, in upload_source_freshness
    UploadSourceFreshnessOperation(config).run(rows_per_insert)
  File "/project/.meltano/utilities/elementary/venv/lib/python3.10/site-packages/elementary/operations/upload_source_freshness.py", line 26, in run
    self.upload_results(results, metadata, rows_per_insert)
  File "/project/.meltano/utilities/elementary/venv/lib/python3.10/site-packages/elementary/operations/upload_source_freshness.py", line 50, in upload_results
    response = dbt_runner.run_operation(
  File "/project/.meltano/utilities/elementary/venv/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 175, in run_operation
    result = self._run_command(
  File "/project/.meltano/utilities/elementary/venv/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 112, in _run_command
    result = self._inner_run_command(
  File "/project/.meltano/utilities/elementary/venv/lib/python3.10/site-packages/elementary/clients/dbt/api_dbt_runner.py", line 53, in _inner_run_command
    raise DbtCommandError(
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.
Path './transform/dbt_athena/' does not exist.
And the second one, change executable to `elementary_invoker`:
Copy code
root@5b27527642c1:/project# meltano invoke elementary:freshness
2024-11-13T09:52:36.715368Z [info     ] Environment 'dev' is active
Extension executing `dbt clean`...
Traceback (most recent call last):
  File "/project/.meltano/utilities/elementary/venv/lib/python3.10/site-packages/meltano/edk/extension.py", line 122, in pass_through_invoker
    self.pre_invoke(None, *command_args)
  File "/project/.meltano/utilities/elementary/venv/lib/python3.10/site-packages/elementary_ext/extension.py", line 77, in pre_invoke
    self.dbt_invoker.run_and_log("clean")
AttributeError: 'elementary' object has no attribute 'dbt_invoker'
pre_invoke failed with uncaught exception, please report to maintainer
e
The first one failed only because the directory doesn't exist so that's a bit of progress 😅
➕ 2
9 Views
Open in Slack
PreviousNext
Powered by