I am running into an issue with a custom meltano dag in airflow. I'm trying to run the elt command with a

--select

flag to rerun one table if needed (basically to fix problems if they come up. The command is

.meltano/run/bin --log-level=debug schedule run core-mysql-postgres --select "$table"'

(The

$table

variable is an airflow conf variable that we pass in). This is the error I am getting:

ELT could not be completed: Cannot start extractor: 'PATH'

```[2022-02-15 140452,056] {subprocess.py:79} INFO - Traceback (most recent call last): [2022-02-15 140452,057] {subprocess.py:79} INFO - File "/Users/larskoester/Knack/core-bi/.venv/lib/python3.8/site-packages/meltano/cli/__init__.py", line 47, in main [2022-02-15 140452,057] {subprocess.py:79} INFO - cli(obj={"project": None}) [2022-02-15 140452,057] {subprocess.py:79} INFO - File "/Users/larskoester/Knack/core-bi/.venv/lib/python3.8/site-packages/click/core.py", line 829, in call [2022-02-15 140452,058] {subprocess.py:79} INFO - return self.main(*args, **kwargs) [2022-02-15 140452,058] {subprocess.py:79} INFO - File "/Users/larskoester/Knack/core-bi/.venv/lib/python3.8/site-packages/click/core.py", line 782, in main [2022-02-15 140452,058] {subprocess.py:79} INFO - rv = self.invoke(ctx) [2022-02-15 140452,059] {subprocess.py:79} INFO - File "/Users/larskoester/Knack/core-bi/.venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke [2022-02-15 140452,059] {subprocess.py:79} INFO - return _process_result(sub_ctx.command.invoke(sub_ctx)) [2022-02-15 140452,059] {subprocess.py:79} INFO - File "/Users/larskoester/Knack/core-bi/.venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke [2022-02-15 140452,059] {subprocess.py:79} INFO - return ctx.invoke(self.callback, **ctx.params) [2022-02-15 140452,060] {subprocess.py:79} INFO - File "/Users/larskoester/Knack/core-bi/.venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke [2022-02-15 140452,060] {subprocess.py:79} INFO - return callback(*args, **kwargs) [2022-02-15 140452,060] {subprocess.py:79} INFO - File "/Users/larskoester/Knack/core-bi/.venv/lib/python3.8/site-packages/meltano/cli/params.py", line 23, in decorate [2022-02-15 140452,061] {subprocess.py:79} INFO - return func(*args, **kwargs) [2022-02-15 140452,061] {subprocess.py:79} INFO - File "/Users/larskoester/Knack/core-bi/.venv/lib/python3.8/site-packages/meltano/cli/params.py", line 56, in decorate [2022-02-15 140452,061] {subprocess.py:79} INFO - func(project, *args, **kwargs) [2022-02-15 140452,061] {subprocess.py:79} INFO - File "/Users/larskoester/Knack/core-bi/.venv/lib/python3.8/site-packages/meltano/core/utils/__init__.py", line 71, in wrapper [2022-02-15 140452,062] {subprocess.py:79} INFO - return run_async(func(*args, **kwargs)) [2022-02-15 140452,062] {subprocess.py:79} INFO - File "/Users/larskoester/Knack/core-bi/.venv/lib/python3.8/site-packages/meltano/core/utils/__init__.py", line 50, in run_async [2022-02-15 140452,062] {subprocess.py:79} INFO - loop.run_until_complete(future) [2022-02-15 140452,063] {subprocess.py:79} INFO - File "/usr/local/Cellar/python@3.8/3.8.12_1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete [2022-02-15 140452,063] {subprocess.py:79} INFO - return future.result() [2022-02-15 140452,063] {subprocess.py:79} INFO - File "/Users/larskoester/Knack/core-bi/.venv/lib/python3.8/site-packages/meltano/cli/elt.py", line 130, in elt [2022-02-15 140452,063] {subprocess.py:79} INFO - await _run_job(project, job, session, context_builder, force=force) [2022-02-15 140452,064] {subprocess.py:79} INFO - File "/Users/larskoester/Knack/core-bi/.venv/lib/python3.8/site-packages/meltano/cli/elt.py", line 212, in _run_job [2022-02-15 140452,064] {subprocess.py:79} INFO - await _run_elt(log, context_builder, ou…

Hi, there may be a much more elegant ways to achieve this. I however have a little wee shell script called load_one_table.sh. I call this if I want to re-load a single table.

#!/bin/bash
printf -v CURRENT_DATE '%(%Y%m%d)T' -1

echo BEGIN
date '+%Y-%m-%d %H:%M:%S'

########################################
# Checking for correct arguments       #
########################################

if [ $# -le 2 ] || [ $# -gt 6 ]
then
  echo "Usage   : `basename $0` <tap> <target> <table_name> <Optional: job_id> <Optional: force parameter to load the job>"
  echo ""
  echo "  e.g. `basename $0` tap-salesforce target-snowflake Cluster__c"
  echo "       `basename $0` tap-salesforce target-snowflake Cluster__c salesforce-border-to-snowflake"
  echo "       `basename $0` tap-salesforce target-snowflake Cluster__c salesforce-border-to-snowflake --force"
  echo "       `basename $0` tap-salesforce target-snowflake Cluster__c salesforce-border-to-snowflake --full-refresh --force"  
  echo ""
  exit $E_BADARGS
fi

echo "Parameter tap                   : $1"
echo "Parameter target                : $2"
echo "Parameter source table          : $3"
echo "Parameter job id                : $4"
echo "Parameter force or full refresh : $5"
echo "Parameter force load            : $6"
echo ""

#Set an environment variable to allow a change to the meltano.yml file
export MELTANO_PROJECT_READONLY=0
meltano select $1 $3

# Test if the force parameter is supplied as there are five arguments supplied. If there are four parameters supply a job_id, else load without state.
if [ $# -eq 6 ]
then
  if [ "$6" != "--force" ] && [ "$5" != "--full-refresh" ]
  then
    echo "Final arguments should be --full-refresh --force"
          exit $E_BADARGS
  fi
  meltano elt $1 $2 --select $3.* --job_id $4 $5 $6
  
elif [ $# -eq 5 ]
then
  if [ "$5" != "--full-refresh" ] && [ "$5" != "--force" ]
  then
    echo "Incorrect refresh argument use --full-refresh or --force"
          exit $E_BADARGS
  fi
  meltano elt $1 $2 --select $3.* --job_id $4 $5  

elif  [ $# -eq 4 ]
then
  meltano elt $1 $2 --select $3.* --job_id $4
else
  meltano elt $1 $2 --select $3.*
fi

date '+%Y-%m-%d %H:%M:%S'
echo END

Thanks for the script. I'll take a look if that works for our needs.