<https://github.com/NangoHQ/nango> looks kind of i...
# random
v
https://github.com/NangoHQ/nango looks kind of interesting. I want something like this but as some kind of spec that multiple providers can implement. Then we could implement that spec for any tap that has an oauth dance required
👀 1
c
Well, there is RFC9068 for the industry at large to agree on standard claims naming. The owner of the RFC (former Azure AD PM) did a neat write up on it: https://auth0.com/blog/how-the-jwt-profile-for-oauth-20-access-tokens-became-rfc9068/
v
Out of the (dozens+?) singer taps I've written I've only had to implement a JWT once, and as a tap we're only on the client side. Small sample size but generally that seems to be the case here. This isn't what I'm talking about but maybe you're making a different point and I'm missing it! My end goal (may not fit with nango directly) is to have a spec that defines the oauth dance. Today we handle this like (these are cases where we're being nice and giving a step by step, some taps don't even do this) 1. https://github.com/AutoIDM/tap-googleads#get-refresh-token 2. https://github.com/AutoIDM/tap-gmail#source-authentication-and-authorization 3. https://github.com/AutoIDM/tap-clickup#getting-an-api-token (Pretty common pattern, but no oauth dance, the spec here would literally just give you a description maybe in markdown format for how to do this, maybe even with helpful links) 4. https://github.com/AutoIDM/tap-zohosprints#setup-authentication These can be handled with a spec (some day we can name it) with something like filename
spec.specerton
for tap-googleds
Copy code
inputs:
  - client_id
  - client_secret 
  - scope

dance_flow:
  get: <https://accounts.google.com/o/oauth2/v2/auth?response_type=code&client_id={inputs.clietn_id}&redirect_uri={specerton.uri}&scope={inputs.scope}&state=autoidm&access_type=offline&prompt=select_account&include_granted_scopes=true>
    await_response: true
    response_data: url_param("code")
  post: <https://www.googleapis.com/oauth2/v4/token?code={dance_flow[0].code}&client_id={inputs.client_id}&client_secret={inputs.client_secrett}&redirect_uri={specerton.uri}&grant_type=authorization_code>
Just floating this idea out there in the wilderness as I keep thinking about it so I wanted to write it down
c
Well, the majority of taps I've written are for HTTP APIs that don't even use OAuth, but instead use their own mechanisms. So, there's no "OAuth dance" at all. 😁 But for the ones that do use OAuth, I haven't come across one that doesn't encode their Access Tokens in the form of a JWT. In the context of a singer tap, the tap never really needs to worry about the claims inside the Access Tokens though, so there's probably not much overlap between the meltano ecosystem and RFC9068
v
Yeah as I was making that list there were 4-5 that were just manually generated API keys in the source app. Which honestly would be niceto have some standard way of documenting the steps somewhere
But not nearly as big of a need (we have a readme already) as a webserver that gets spun up to do flows for you (not needing to copy the code in the url paramater manually 🤷 ) . Where this really can help people isn't so much tap developers / individual tap users but folks who want a front end. Like the Meltano UI or other folks that are trying to front different Singer Taps authentication
c
I believe what you are thinking of are all the OAuth 2 token grant flows that are not meant for server-to-server interactions. (e.g. auth code grant flow)
v
I wouldn't limit it to just oauth 2 token grant flows, and yes anytime we have to do a
dance
with another server where we are sending requests back and forth. The end goal for the tap side of all this is to just get that sweet refresh token in most cases.
https://github.com/NangoHQ/nango/blob/master/packages/auth/providers.yaml is the spec they chose for this. Seems fairly close
c
Nice find! I was trying to find where Nango was keeping all their definitions.
That's really cool. It's a collection of OAUTH2 Auth and Token URLs, which is pretty handy. TBH, I never had any problems finding those URLs quickly from the vendor's docs though. 😁
v
The problem isn't finding the URLs really. The problem is something like: I have a front end service that allows folks to pull data from gmail service (Let's say gmail for now just to keep it simple but in reality you're talking about 10-20 services). To pull data from gmail I need a refresh token from each of my end users. To get that refresh token I have to have the users go through an auth flow of some kind. How do I define the oauth dance without creating 2 sources of truth for auth. The 2 sources of truth would be the tap, and the web service you create for gmail auth.
https://github.com/meltano/meltano/discussions/7257 is another description of the use case
Everyone I know of just makes a seperate web service and deals with the drift between the tap and the auth service via tickets and monitoring of logs
Really this isn't even just Meltano It's pretty much every service I"ve seen that pulls external data from systems 🤷 I'm sure some are more cohesive but they sacrifice the flexibility of having "taps" generated very quickly and with a standard spec
c
Interesting. I guess I have been lucky to only ever use the OAUTH2 client_credentials token grant flow (i.e. "a service account") ... Never really had to use any other OAUTH2 flows in a meltano tap. I see how a "service account" might be problematic in the GMail example and you might actually need every single user to agree to having their emails ingested. I wonder if this is a B2C vs B2B challenge?
v
It's just as hard in B2B, that's where I hit this most of the time
jira, docusign, azuread, gsuite, etc etc
c
Rigth, I agree. 'Jira' doesn't seem to have support for client credentials grant flows. Azure AD is a piece of cake though. I use it a lot and never had to deviate from the client credentials grant flow in any azure ad cases.
Haven't had to deal with Gsuite yet ...
v
At some point you have to get creds. service account, just means you give a client steps to do and then you have them pass over the creds in your UI. And you have a similar issue but maybe not as much worry about a refresh token
c
Yes. So basically the way I see it, there are two challenges: 1. Some providers don't have support for all grant types as specified in rfc6749, specifically the lack of client credentials grant support. 2. Meltano currently works well for client credentials grant flow, but not for authorization code grant.
a
Love this conversation. We deal with the 'dance' generically similar to Nango, but I really like the way they simplified adding the app. As an example, our google taps all use the same authorization + refresh endpoint in our service (which we call a proxy OAuth endpoint) https://github.com/Matatika/utility-gdrive/blob/master/utility_gdrive/auth.py#L8 Some others call it a proxy too: https://oauth2-proxy.github.io/oauth2-proxy/docs/configuration/oauth_provider/ I think Auth0 and Okta offer a service like this. Finally, I prefer to think of this as the consent flow. When it is a simple HTTP API it's almost a 'who cares' situation. Managing the credentials is fairly easy. When you have to get client consent and manage all those separately and refresh, it gets harder.
v
@aaron_phethean in your case you all had to develop similar "consent flows" for each of the apps right. So you had to develop flows for each tap/target that you wanted in a seperate part of your app (seperate meaning not in the tap/target). I think the "right" abstraction layer is in the tap/target itself but maybe I'm dreaming? Oauth proxies make sense in certain scenario's but the tap having any idea about where oauth is being proxied from seems a bit much but 🤷
It's one of those things that's "easy" to do but from what I've seen it's one of the biggest pain points people have. Maybe the pain is always there but dealing with the pain in a standard way would be helpful I think
a
Actually not entirely separate from the tap, the plugin.yaml indicates to the tap that it should use the proxy - (so Bearer token configuration works with the same tap) The consent flow does need a UI or something to get user sign in / scope consent, and our approach relies on the UI understanding these
oauth_credentials.
configs (something the meltano UI once did I think)
Copy code
- name: oauth_credentials.authorization_url
    kind: hidden
    label: OAuth identity provider authorization endpoint used create and refresh tokens
    value: <https://oauth2.googleapis.com/token>
    required: true
  - name: oauth_credentials.scope
    kind: hidden
    label: OAuth scopes we need to request access to
    value: <https://www.googleapis.com/auth/drive.readonly>
    required: true
  - name: oauth_credentials.refresh_proxy_url
    kind: hidden
    label: Optional - will be called with 'oauth_credentials.refresh_token' to refresh the access token
  - name: oauth_credentials.refresh_proxy_url_auth
    kind: hidden
    label: Optional - Sets Authorization header on 'oauth_credentials.refresh_url' request
  - name: oauth_credentials.client_id
    kind: hidden
    label: Optional - OAuth Client ID used if refresh_proxy_url not supplied
    env: GDRIVE_CLIENT_ID
  - name: oauth_credentials.client_secret
    kind: hidden
    label: Optional - OAuth Client Secret used if refresh_proxy_url not supplied
    env: GDRIVE_CLIENT_SECRET
  - name: oauth_credentials.refresh_token
    kind: hidden
    label: OAuth Refresh Token
    env: GDRIVE_REFRESH_TOKEN
  - name: oauth_credentials.access_token
    kind: hidden
    label: Access Token
    env: GDRIVE_ACCESS_TOKEN
    required: true
I don't see how this could be in the tap / target, because fundamentally you need and app + the authorization, which is different for each case the tap is used in. Defeating the purpose of OAuth2 flow?