Hey everyone We are looking to use <http Astronomer io|Astro Meltano #troubleshooting

Hey everyone! We are looking to use <Astronomer.io...

ricky_renner

09/20/2021, 5:57 PM

Hey everyone! We are looking to use Astronomer.io, and I was wondering if anyone has played around with baking all of the Meltano dependencies into the Astronomer base image. Thank you!

ricky_renner

09/21/2021, 12:50 PM

<!subteam^S02BCD9FFEF> just including the Meltano python package should be all we need right? Or do we need to install Git or something like that

ken_payne

09/21/2021, 1:33 PM

Hey Ricky 👋 The meltano base image is here. We don't require git, but

build-essential

node

and

yarn

are installed, I believe to support the UI. This is possibly not required for running

meltano elt

(the others will correct me if I'm wrong). You should then be able to

pip3 install meltano

inside your container and have everything run as expected 🙂

ricky_renner

09/21/2021, 1:40 PM

Hey @ken_payne! I guess I'm wondering if Meltano will work standalone with just the python package (i.e. is it required to use the image?)

ken_payne

09/21/2021, 2:06 PM

It definitely is not a requirement to use the image - I just thought you'd be interested to see it as a reference. You should be able to simply

pip3 install meltano

(or include it in your

requirements.txt

) and add

meltano install

to your Dockerfile (inheriting from the Astronomer one) and then

meltano elt ...

(called from the Airflow DAG generator) should just work 🙂 I haven't used Astronomer yet, but would be interested to hear about your experience with it once you get going!

ricky_renner

09/21/2021, 2:15 PM

Thank you @ken_payne. I'm getting this error after trying to run `meltano install`:

Copy code

Loader 'target-bigquery' could not be installed: failed to install plugin 'target-bigquery'.
  Running command git clone -q <https://github.com/adswerve/target-bigquery.git> /tmp/pip-req-build-gtvxkdxt
  ERROR: Command errored out with exit status 1:
   command: /usr/local/airflow/.meltano/loaders/target-bigquery/venv/bin/python /usr/local/airflow/.meltano/loaders/target-bigquery/venv/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmpim4fm1fs
       cwd: /tmp/pip-install-amhjf1fh/ciso8601_1dc5d4fd9cba4f00810c2faf1b57bfb0
  Complete output (22 lines):
  running bdist_wheel
  running build
  running build_py
  package init file 'ciso8601/__init__.py' not found (or not a regular file)
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/ciso8601
  copying ciso8601/__init__.pyi -> build/lib.linux-x86_64-3.7/ciso8601
  copying ciso8601/py.typed -> build/lib.linux-x86_64-3.7/ciso8601
  running build_ext
  building 'ciso8601' extension
  creating build/temp.linux-x86_64-3.7
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DCISO8601_VERSION=2.2.0 -DCISO8601_CACHING_ENABLED=1 -I/usr/local/airflow/.meltano/loaders/target-bigquery/venv/include -I/usr/local/include/python3.7m -c module.c -o build/temp.linux-x86_64-3.7/module.o
  In file included from /usr/lib/gcc/x86_64-linux-gnu/8/include-fixed/syslimits.h:7,
                   from /usr/lib/gcc/x86_64-linux-gnu/8/include-fixed/limits.h:34,
                   from /usr/local/include/python3.7m/Python.h:11,
                   from module.c:1:
  /usr/lib/gcc/x86_64-linux-gnu/8/include-fixed/limits.h:194:15: fatal error: limits.h: No such file or directory
   #include_next <limits.h>  /* recurse down to the real one */
                 ^~~~~~~~~~
  compilation terminated.
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for ciso8601
ERROR: Could not build wheels for ciso8601 which use PEP 517 and cannot be installed directly

ricky_renner

09/21/2021, 2:15 PM

Similar issues with the other taps/targets

ken_payne

09/21/2021, 2:18 PM

Ok, this looks like

build-essentials

is a core requirement then 😅 I thought it might just be for node/yarn, but that error would suggest otherwise.

ken_payne

09/21/2021, 2:43 PM

Looking a bit deeper, while

build-essential

may still work (it installs a collection of build-related packages), it may not give you everything you need 🤔 The Astronomer base image inherits from the

python:buster-slim

images, which does include

gcc

by default, but not

python-dev

python3-dev

and a few other common python linux dependencies. If you run into any other trouble let us know - I'd love to arrive at a base set of OS/python requirements for Meltano 🙂

ricky_renner

09/21/2021, 2:58 PM

Thank you! I think we have another issue because I think Astronomer and Meltano require different versions of

jsonschema

. Meltano defines the version it needs here, and I am looking for where Astronomer defines this.

ken_payne

09/21/2021, 3:05 PM

So there are a couple of options to get around this: 1. Install meltano into a virtual env inside the container and modify the meltano DAG generator to activate the venv before calling

meltano elt

. This will work with Astronomers LocalExecutor deployment option. 2. Use the KunernetesExecutor or the KubernetesPodOperator in Astronomer; both allow you to specify the task image you want to run, allowing you to build an image specifically for meltano.

ken_payne

09/21/2021, 3:10 PM

1. will definitely be easiest - there are some logging implications with the k8s options. Let me know if you go down either of 2. - I have worked through adding meltano logs to the Airflow UI before. We hope to have a supported way of running on Astronomer in the future, and are tracking all the deployment options in this epic. Would love your feedback and request for Astronomer as a supported platform in a ticket if you don't mind 🙏

ricky_renner

09/21/2021, 7:33 PM

Thank you Ken! Yes where do I submit a ticket? We actually current use option 2, but we're exploring a different way of doing things.

Open in Slack

Previous Next