@here getting error while using airflow scheduler root@meltano-licensingeservice-156-402-868f7c67dc-kk7fx:/mel_v1# meltano install 2024-07-03T073654.821279Z [warning ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))': /meltano/api/v1/plugins/orchestrators/index 2024-07-03T073654.875173Z [warning ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))': /meltano/api/v1/plugins/orchestrators/index 2024-07-03T073654.930343Z [warning ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))': /meltano/api/v1/plugins/orchestrators/index Could not connect to Meltano Hub. Could not reach Meltano Hub.. Run

meltano discover orchestrator

to explore available plugins.

which version of meltano are you running, with lock files they shouldn't need to reach out to the hub anymore

2.20.0

try upgrading v3.0.0 requires lock files which will force you to fix this, another way with that version is to probably implement lock files. I'd just bump to the latest

Did airflow required internet in every execution of scheduler

With that version without lock files yes, if you upgrade or use lock files, no. Just upgrade it'll tell you to run something like

meltano lock

(it'll say in the error when you go to run) then you have to commit those lock files to your repo. Bam problem gone!

currently we are doing we create a meltano project with adding all files and command and running in a pod , where no internet is there

Trust me on this one @ashish_soni, upgrade to the latest version of meltano*. It'll force you to run meltano lock which will make it so internet won't be required

ok ok

https://docs.meltano.com/reference/command-line-interface/#lock I'm not sure which version introduced locking but this should do it

Extractor tap-cloudwatch_logs is a custom plugin Loader kafka-loader is a custom plugin Locked definition for orchestrator airflow Locked definition for file bundle files-airflow this i got when i run meltano lock --all

will this restrict meltano invoke airflow to stop scanning on internet ?

Yes.

but i didn't find an lock file in meltano project

They exist under the

plugins

directory in your project.

@Reuben (Matatika) @visch Thank You ! Problem Resolved