Why the Meltano complains of missing environment variable if state is found in system database? I was assuming that environment variable is used to set an initial value for the first run.

- name: tap-exchangeratesapi
    namespace: tap_exchangeratesapi
    pip_url: tap-exchangeratesapi
    executable: tap-exchangeratesapi
    config:
      start_date: '${EXCHANGE_START_DATE}'

First run:

EXCHANGE_START_DATE='2020-07-13' meltano elt tap-exchangeratesapi target-csv --job_id=extract_rates

Subsequent run:

meltano elt tap-exchangeratesapi target-csv --job_id=extract_rates

Log output:

Variable EXCHANGE_START_DATE is missing from the environment.
Variable EXCHANGE_START_DATE is missing from the environment.
Running extract & load...
Variable EXCHANGE_START_DATE is missing from the environment.
Variable EXCHANGE_START_DATE is missing from the environment.
Found state from 2020-07-14 19:25:28.973045.
INFO Replicating exchange rate data from 2020-07-14 using base USD
INFO Tap exiting normally
Incremental state has been updated at 2020-07-14 19:26:15.840289.
Extract & load complete!
Transformation skipped.

If I pass the

EXCHANGE_START_DATE

the tap starts from the passed date ignoring the state. How can I silent these noise around missing variables?

@rahul_anand Taps are expected to ignore the

start_date

setting when state is provided, and

tap-exchangeratesapi

does so as well: https://github.com/singer-io/tap-exchangeratesapi/blob/master/tap_exchangeratesapi/__init__.py#L112

Running this:

EXCHANGE_START_DATE=2020-07-13 meltano elt tap-exchangeratesapi target-csv --job_id=extract_rates

Overrides the state and the tap starts re-downloading from ' 2020-07-13'.

Yep, but that's behavior that is to be implemented by the tap itself 🙂 tap-

exchangesratesapi

appears to do so correctly (https://github.com/singer-io/tap-exchangeratesapi/blob/master/tap_exchangeratesapi/__init__.py#L112), so I think we'll want to debug why that's not actually behaving as the code would suggest

Yes I can see

{"singer_state": {"start_date": "2020-07-14"}}

All right, that's a good sign! Do you think you could dig into

.meltano/extractors/tap-exchangeratesapi/venv/lib/python3.7/site-packages/tap-exchangeratesapi/tap_exchangeratesapi/__init__.py

and add some print-based debugging around the line at https://github.com/singer-io/tap-exchangeratesapi/blob/master/tap_exchangeratesapi/__init__.py#L112, so that we can see whether it's reading the state correctly, and why it's still preferring the config over the state?

So @douwe_maan you are saying in this case when environment variable is set, the meltano is passing

--state

without overriding the value and the

--config

as usual. The tap needs to honor the state and disregard the value from

--config

.

Correct, Meltano will always pass all (env var-expanded) settings in the generated

config.json

, regardless of state, and it's up to the tap to prefer state over configured

start_date

Ok. Is there a way to get the meltano command generated in log where it passes the config and state?

Yes!

meltano --log-level=debug elt ...

@douwe_maan Adding the print statements confirmed the state is not being passed to the tap. Added:

<http://logger.info|logger.info>(f'state: {state}')
<http://logger.info|logger.info>(f'config: {config}')

Log output:

INFO state: {}
INFO config: {'start_date': '2020-07-13'}

In system database I see the state:

41|extract_rates|SUCCESS|2020-07-14 20:17:09.885764|2020-07-14 20:17:14.440576|{"singer_state": {"start_date": "2020-07-14"}}|1|c473429584834c5f9f501682293ba530|cli

Interesting. And when you don't set

EXCHANGE_START_DATE

, is any

state

logged then?

No I did not. That may be the culprit then.

Ah yes, that's definitely it

Without setting the env variable I see this:

INFO state: {}
INFO config: {'start_date': None}

Yep, that's consistent with us never actually passing the state because plugin is not known to have that capability... That's definitely confusing though, since we do log other state related stuff

And yes I confirm, adding

state

under

capabilities

resolved the issue.

Awesome 🙂 Sorry, I should've suggested that earlier. I'm still figuring out the best troubleshooting strategy as well!

Yes it will definitely make it clear

If you have any interest in contributing, that may actually be a nice small one to start off with 😉

Sure