Can some one advice how do I run 'metano ui' over https

Do you have experience with nginx or another reverse proxy? You can use it to handle SSL termination on port 443 and then forward the requests to the regular

meltano ui

server at port 5000

yes I do have some experience in that, i am trying to get the POC production env. for now i don't mind running it in http but i am having problems, I have

MELTANO_UI_AUTHENTICATION=true

i get

Authentication is enabled, but your configuration is currently insecure:

so i added

MELTANO_UI_SESSION_COOKIE_DOMAIN="<http://sub.domain.co.uk|sub.domain.co.uk>"

but now i get

The CSRF tokens do not match.

All right, you're getting close 🙂 Did you see https://meltano.com/docs/production.html#meltano-ui? Specifically:

If you will be running Meltano UI behind a front-end (reverse) proxy that will be responsible for SSL termination (HTTPS), it is recommended that you enable the `ui.session_cookie_secure` setting so that session cookies used for authentication are only sent along with secure requests.

You may also need to change the `ui.forwarded_allow_ips` setting to get Meltano UI to realize it should use the

https

URL scheme rather than

http

in the URLs it builds.

yes i am 🙂

meltano, version 1.58.0

i have tried it with

MELTANO_UI_SESSION_COOKIE_SECURE="true"

MELTANO_UI_FORWARDED_ALLOW_IPS="*"

Its just i am not sure about which combination of things work (i think) With regards to https://gitlab.com/meltano/meltano/-/issues/2307 it is only if you want to serve it over https?

What symptoms are you seeing right now? Still the CSRF error? And can you please share the value of all of your

ui.*

settings? You can find them in

meltano config meltano

So my current setting are

MELTANO_DATABASE_URI="postgresql://......"
MELTANO_UI_SECRET_KEY="8cb......"
MELTANO_UI_PASSWORD_SALT="a0....."
MELTANO_UI_AUTHENTICATION="true"
MELTANO_UI_SESSION_COOKIE_DOMAIN="<http://SUB.DOMAIN.co.uk|SUB.DOMAIN.co.uk>"
MELTANO_UI_SESSION_COOKIE_SECURE="true"
MELTANO_UI_FORWARDED_ALLOW_IPS="*"

and i am still getting CSRF token do not match

OK, that's the same error we saw in https://gitlab.com/meltano/meltano/-/issues/2307, and there the reason was that Flask could not find its own session cookie because the secure flag was set even though the connection was not over https, which was causing Chrome to refuse to store the cookie

the default 5000 so its http://sub.domain.co.uk:5000

OK! Then try adding

:5000

to the

ui.server_name

and/or

ui.session_cookie_name

i have added

MELTANO_UI_SESSION_COOKIE_DOMAIN="<http://sub.domain.co.uk:5000|sub.domain.co.uk:5000>"

MELTANO_UI_SERVER_NAME="<http://sub.domain.co.uk:5000|sub.domain.co.uk:5000>"

still CSRF

Set-Cookie

in the

/

and login page response headers,

Cookies

in the error page request headers

😞 i do not have

Cookies

  in the error page request headers

OK, so that suggests that indeed the

Set-Cookie

didn't "work"

First meltano.m-eat.co.uk response headers Set-Cookie: session=eyJfZmxhc2hlcyI6W3siIHQiOlsiaW5mbyIsIlBsZWFzZSBsb2cgaW4gdG8gYWNjZXNzIHRoaXMgcGFnZS4iXX1dfQ.X7RTCA.0uS5g_4YDkqHxYSQHhWNHp83xxk; Domain=meltano.m-eat.co.uk; Secure; HttpOnly; Path=/ login?next=%2F response headers Set-Cookie: session=eyJjc3JmX3Rva2VuIjoiNmE1ZGU3YTM0MjJjYmQwZDFlNzYyYTViOTgyYWZmMGY1OWM2MzFkYSJ9.X7RTCA.xq8yy1EjyjxSWXY0-4e82kH2CmA; Domain=meltano.m-eat.co.uk; Secure; HttpOnly; Path=/ ============ after log in (error page): response headers Set-Cookie: session=eyJjc3JmX3Rva2VuIjoiYWRkMGIxOTFhMmI2Yjk5NmFjNTQ3MjczOTAyYzdjMjU3ZDY5NDMyMiJ9.X7RTrA

OK, so the fact that it's different each time is also consistent with the fact that it couldn't read back the current cookie

if i remove

MELTANO_UI_SESSION_COOKIE_DOMAIN="<http://meltano.m-eat.co.uk:5000|meltano.m-eat.co.uk:5000>"
MELTANO_UI_SERVER_NAME="<http://meltano.m-eat.co.uk:5000|meltano.m-eat.co.uk:5000>"
MELTANO_UI_SESSION_COOKIE_SECURE="true"
MELTANO_UI_FORWARDED_ALLOW_IPS="*"

The i can log in normally without any problem (apart from "Authentication is enabled, but your configuration is currently insecure:")

What does the Set-Cookie header look like now?

http://meltano.m-eat.co.uk:5000 (i omitted domain earlier to not to advertise 🙂 but i think keeping actual domain i am using makes thing easier to understand ) These are the cookies (can login with out errors) Set-Cookie: session=eyJfZmxhc2hlcyI6W3siIHQiOlsiaW5mbyIsIlBsZWFzZSBsb2cgaW4gdG8gYWNjZXNzIHRoaXMgcGFnZS4iXX1dfQ.X7RW2A.wrmlduJKwlII1m9dnzb8zugB9_c; HttpOnly; Path=/ Set-Cookie: session=eyJfZnJlc2giOmZhbHNlLCJjc3JmX3Rva2VuIjoiMjA3ZGQ4YTY0YjcyNTg5ZThmZmI1ZmNiMWRkMTRhNmUwNDU4MTYxMyJ9.X7RW2Q._S1LXAbfx0xlGQs_vdj_0BO-BD0; HttpOnly; Path=/ After log in (no error) Cookie: session=eyJfZnJlc2giOmZhbHNlLCJjc3JmX3Rva2VuIjoiMjA3ZGQ4YTY0YjcyNTg5ZThmZmI1ZmNiMWRkMTRhNmUwNDU4MTYxMyJ9.X7RW2Q._S1LXAbfx0xlGQs_vdj_0BO-BD0

OK, so it looks like the

Domain

in the "secured"

Set-Cookie

is the problem

If i look at

cookies

tab i can see the domain

Yeah that all looks right. I assume it's showing the default if it's not explicitly included in the

Set-Cookie

definition

i am not using HTTPS, currently its HTTP. let me check which one breaks again 🙂

At least make sure the Secure flag matches whether you're currently using HTTPS or not 🙂

🙂 ok

MELTANO_UI_SESSION_COOKIE_SECURE="true"

own its own breaks things (the secure box is ticked, but in cmd i see "Authentication is enabled, but your configuration is currently insecure" first image)

MELTANO_UI_SESSION_COOKIE_DOMAIN="<http://meltano.m-eat.co.uk|meltano.m-eat.co.uk>"

Does not break things (with this i do not see the tick in secure box, second image)

Does the "i" say anything on hover?

This cookie was blocked because it had the "secure" attribute but was not received over secure connection

😄 OK, I think I was assuming you were accessing over HTTPS the whole time since that's what sparked the conversation

but in order to set secure we need reverse proxy? (I thought there was some confusion that is why i said i was using HTTP) mean we were trying to solve problem that does not exist 😂

Yeah we need a reverse proxy to actually get https (SSL) to work on that subdomain of yours, unless it's already behind cloudflare or something

I guess so 😄 (sorry for wasting your time, but i learned a lot about cookies today)

No worries, happy to help 😛

ok lets do this you choose a issue that is pending (may be some low hanging fruit that i can do 😁) and assign it to me 🙂

@nil How about https://gitlab.com/meltano/meltano/-/issues/2387?

awesome I like things in order.

Haha good

yes I saw it everywhere (on blog, in singer...) thanks. I will stick with Nil (by the way full name is Nilesh) as I use that almost all the time.

OK, that works, just wanted to make sure you get the chance to have your last name included!

ah ok, i tend to leave out last name as it contains lots of

a

and majority of them silent 🤣

😄

There is mysql-logo.png is missing shell I add that when I create MR for https://gitlab.com/meltano/meltano/-/issues/2387 or do you want it separately? (i can send the logo to you if you wish)

Separately please! Let's keeps MRs focused

@douwe_maan I’m thinking I have the same issue. I have set up Meltano using Docker. Set up reverse proxy with Nginx and SSL. I’m having an issue like the following error on Chrome:

Mixed Content: The page at '<https://example.com/extractors>' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint '<http://example.com/api/v1/version?include_latest=true>'. This request has been blocked; the content must be served over HTTPS.

Your help is greatly appreciated.

@ed_bergavera That suggests that Meltano doesn't realize it's behind SSL and should be building its URLs with

https://

instead of

http://

. Did you see the suggestion under https://meltano.com/docs/production.html#meltano-ui?

If you will be running Meltano UI behind a front-end (reverse) proxy that will be responsible for SSL termination (HTTPS), it is recommended that you enable the `ui.session_cookie_secure` setting so that session cookies used for authentication are only sent along with secure requests.

You may also need to change the `ui.forwarded_allow_ips` setting to get Meltano UI to realize it should use the

https

URL scheme rather than

http

in the URLs it builds.

Specifically, look into https://meltano.com/docs/settings.html#ui-forwarded-allow-ips

@douwe_maan Thanks for the response. Yes, we all set the options by putting them into an environment file and referencing it with Docker run. However, we turned our attention to PoC to run FB ads to Postgres, but we haven’t been successful yet. We keep getting failed output to store information to Postgres. I have attached a recent log.

Hmm, looks like it's tripping over the

age

column even before `target-postgres | psycopg2.errors.UndefinedColumn: column "age" does not exist`:

target-postgres | INFO Table Schema Change [`ads_insights_age_and_gender`.`('age',)`:`age`] New column (took 4 millis)
target-postgres | WARNING FIELD COLLISION: Field `('age',)` exists in remote already. No compatible type found. Appending type suffix: `age__s`
target-postgres | WARNING FIELD COLLISION: Field `('age',)` exists in remote already. No compatible type found. Appending type suffix: `age__i`
target-postgres | INFO Table Schema Change [`ads_insights_age_and_gender`.`('age',)`:`age__s`] Splitting `age` into `age__i` and `age__s`. New column matches existing column path, but the types are incompatible. (took 13 millis)

@douwe_maan thanks for your insights. I'm in the impression that Meltano could just populate automatically the fields regardless of the structure of your source.

Taps inform targets of the schema of the records they'll be emitting using a SCHEMA message, which the target then uses to create the corresponding database tables, but in this case the tap outputs multiple types for the column in question which the target doesn't understand.

I mean, you don't have to worry about the structure of the source data the fact that Meltano can automatically fill those data to your target.

Meltano handles the communication between tap and target, but it's still the tap itself that's responsible for the structure of the data it emits, which needs to be compatible with the target that will create the appropriate database tables.

Oh, I see..that explains the issue here.

So in this case we need to either fix the tap to not claim that this column has 2 different types, or we can add support for this to the target, or use a different variant of the target that may already support it

We'll check that out. I appreciate your time and providing helpful suggestions 🙂

I've created a new issue that would allow us to override the broken schema for

age

using the existing

schema

extra in

meltano.yml

I linked to a few messages up: https://gitlab.com/meltano/meltano/-/issues/2469

Will do. Thanks again @douwe_maan

Hello @douwe_maan @ed_bergavera I am also getting two types "integer and string" columns for one column and i saw this discussion over this issue. @douwe_maan As you specify the specific reasons: "So in this case we need to either fix the tap to not claim that this column has 2 different types, or we can add support for this to the target, or use a different variant of the target that may already support it" I checked the types on schema but it's just single type specified( Integer) For the reference i am using tap-shopify and target-redshift. @edward_ryan @connor_flynn