Hi community Could anyone share the way to run airbyte conne Meltano #getting-started

Hi community! Could anyone share the way to run ai...

ivan

04/28/2023, 6:31 PM

Hi community! Could anyone share the way to run airbyte connectors in meltano docker? I have created docker image via meltano command. So docker image starts with that

Copy code

ARG MELTANO_IMAGE=meltano/meltano:latest
FROM $MELTANO_IMAGE

Next, I've read about running airbyte connectors in docker: https://docs.meltano.com/guide/advanced-topics#am-i-able-to-put-my-meltano-project-with-airbyte-connectors-in-production As first way to run that connector I go with the most easiest way mentioned there Docker in Docker Using [/var/run/docker.sock]. SO I just try to run meltano build image with option

--volume /var/run/docker.sock:/var/run/docker.sock

and it does not work because there is not docker cli inside meltano image. Could you advise if meltano has docker image with preinstalled docker cli inside or how could I install docker cli via comman in

Dockerfile

pat_nadolny

04/28/2023, 9:12 PM

@ivan_kedrin I dont have much docker in docker experience but I think you'd probably need to build your own image. The default meltano image wouldnt support docker in docker as it stands today. I'm interested in this and I bet a bunch of others would benefit if someone figured it out, happy to help.

alexander_butler

04/29/2023, 6:58 AM

You can just add

RUN apt install docker-cli

alexander_butler

04/29/2023, 7:02 AM

I put this together as a starting point back when I developed the tap. Anyone is welcome to take it and play with it too. It is supposed to run a docker enabled job on a k8s cluster (obviously the

priviledged: true

flag is unavoidable unless maybe you spin up a VM and serve docker over TCP which is actually a valid path. If you have all your cloud resources on a VPC, then its decent and drops the priviledged requirement within the cluster

Copy code

apiVersion: batch/v1
kind: Job
metadata:
  name: tap-airbyte-job
spec:
  template:
	spec:
	    containers: 
	      - name: etl-command
	        image: your-meltano-docker-image-with-docker-cli:latest
	        command: ['meltano', 'run', 'tap-airbyte-wrapper', 'target-whatever'] 
	        resources: 
	            requests: 
	                cpu: 250m
	                memory: 512Mi
	        env:
	          - name: DOCKER_HOST 
	            value: <tcp://localhost:2375> 
	      - name: dind-sidecar
	        image: docker:dind
	        resources:
	            requests:
	                cpu: 750m
	                memory: 512Mi
	        securityContext: 
	            privileged: true 
	        volumeMounts:
	          - name: docker-graph-storage 
	            mountPath: /var/lib/docker
	    volumes:
	      - name: docker-graph-storage 
	        emptyDir: {}

ivan

04/29/2023, 9:28 AM

That helps, but not I get errors inside of running container: airbyte cdk could not find config file during discovery run:

/tmp/config.json

Am I missed some volume mapping related to config files?

Copy code

tap_airbyte.tap.AirbyteException: Traceback (most recent call last):
  File "/airbyte/integration_code/main.py", line 13, in <module>
    launch(source, sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 131, in launch
    for message in source_entrypoint.run(parsed_args):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 85, in run
    raw_config = self.source.read_config(parsed_args.config)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/connector.py", line 51, in read_config
    config = BaseConnector._read_json_file(config_path)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/connector.py", line 61, in _read_json_file
    with open(file_path, "r") as file:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/config.json'

pat_nadolny

04/29/2023, 2:11 PM

Yeah looks like a volume issue to me

ivan

04/29/2023, 3:25 PM

Looks like it is becose env setting not available inside. Need to connect .env file or setup env variables inside docker.

Open in Slack

Previous Next