Hi all, i'm having some issues with tap-xero. it seems that when it uses the refresh token to get a new access token, its not saving the new refresh token in the config. i can see the code where it dumps json to a 'config file' but no idea where that is. If i do

meltano invoke --dump=config tap-xero

i get the settings i have provided as environment variables. I'm running dockerised with the .env file being passed in via the compose config. I'm using postgres as my meltano db uri, its written the tables but most of them are empty. I'm not sure where its meant to be storing the auth information, or if it is storing it and my env file is overriding it somehow? If anyone can shed some light that would be appreciated.

My suggestion would be to pay the extra fee to be able to use OAuth 2.0 Client Credential Grant flow.

what xero calls a custom connection? we do, i cant see how to configure that for meltano at all. documentation welcome :)

Yup. The "custom connection" should allow for client credential grant flow. According to the documentation the settings should be

client_id

and

client_secret

https://hub.meltano.com/extractors/tap-xero

yeah hence my confusion about it. like i said, it uses the refresh token, it stores the new one in a 'config file' somewhere (no idea where). problem is you go to execute again and get 'invalid_grant' because its using the first refresh_token again which is now invalidated. I didnt see any docs on how to do the client credential grant flow. I have it configured with client id, client secret and refresh token. i get the token by using the postman collection xero publishes for oauth2.0 flow.

My colleague @steve_clarke implemented a big update to https://github.com/s7clarke10/tap-rest-api-msdk to handle all sorts of pagination and authentication styles - maybe that would work better than the variant of

tap-xero

you're using?

i would have to adapt that to the xero api? thats a lot more work than i was hoping to do 😞

Yeah, you would unfortunately, but it's probably not heaps of work. I know the generic tap suggested is quite well documented on the README, just a thought 🤷‍♂️

seems easier to debug the xero tap refresh token tbh

Theres a few issues related to this https://github.com/meltano/meltano/issues/2660. Taps that alter the config file are pretty rare but in those cases meltano doesnt really have a good way to persist the changes back to the project

ah so this is a more core issue. hmmm. meltano might not be quite ready for what i'm doing then?

While state may not be a best practice for this scenario, technically you could persist the refresh token alongside other bookmarks. If there is state discovered, it could pull out the refresh_token bookmark and initialize the refresh_token from that prior to the OAuth call. If there is no state discovered, i.e. it is the first run it would retrieve the refresh_token value from the config.json. When a request is made for OAuth Credentials, save the new refresh_token to an appropriately named bookmark to be saved along with other bookmarks like the replication_key when the state is written by the tap. In some ways I prefer this to writing back to a config.json file because some implementations of Meltano are ephemeral and the only persisted settings come from the state database. I run in a docker container and the container is destroyed at the end of the ingestion. If you are wondering how you could read and write a custom bookmark to the state database, here is an example. Example 1: Reading a custom bookmark from tap-mssql. https://github.com/s7clarke10/pipelinewise-tap-mssql/blob/f7a0bf18c01d24aa55ce5fbc8b3bc7ee9c928fc7/tap_mssql/sync_strategies/log_based.py#L289 Example 2: Saving a custom bookmark from tap-mssql https://github.com/s7clarke10/pipelinewise-tap-mssql/blob/f7a0bf18c01d24aa55ce5fbc8b3bc7ee9c928fc7/tap_mssql/sync_strategies/log_based.py#L38[…]9C93

I should also note that ¹https://github.com/MeltanoLabs/tap-salesforce/tree/main (tap-salesforce) uses a refresh_token for OAuth.

Probably another case where refresh tokens should not be stored in the config to begin with. Salesforce fully supports the OAuth 2.0 Client Credentials Grant flow.