ran Meltano on kubernetes via argo workflows today Meltano #show-us-what-you-got

Join Slack

ran Meltano on kubernetes via argo workflows today...

# show-us-what-you-got

Matt Menzenski

02/24/2023, 9:23 PM

ran Meltano on kubernetes via argo workflows today!

umar_saleem

02/25/2023, 12:22 PM

Could you share the method pleeeease ? 🤓

Matt Menzenski

02/27/2023, 4:41 PM

@umar_saleem I’m using hera-workflows to write the Argo Workflow definition in Python.

Copy code

from hera import Env, Task, Workflow

with Workflow("test_workflow") as w:
    load_meltano_runs = Task(
        "load-meltano-runs",
        image="<dockerhub_organization>/<dockerhub_image>:latest",
        command=["meltano", "run", "tap-meltano", "target-redshift-staging-meltano"],
        env=[
            Env(name="AWS_ACCESS_KEY_ID", value=""),
            Env(name="AWS_SECRET_ACCESS_KEY", value=""),
            Env(name="AWS_DEFAULT_REGION", value=""),
            Env(name="MELTANO_DATABASE_URI", value=""),
            Env(name="MELTANO_ENVIRONMENT", value=""),
            Env(name="TAP_MELTANO_MELTANO_DATABASE_URI", value=""),
            Env(name="TARGET_REDSHIFT_HOST", value=""),
            Env(name="TARGET_REDSHIFT_PORT", value=""),
            Env(name="TARGET_REDSHIFT_S3_BUCKET", value=""),
            Env(name="TARGET_REDSHIFT_USER", value=""),
            Env(name="TARGET_REDSHIFT_PASSWORD", value=""),
        ],
    )

if __name__ == "__main__":
    print(w.to_yaml())

For testing, I provided the values in that file, then executed that script and piped the output to a YAML file that I submitted in the Argo UI.

Matt Menzenski

02/27/2023, 4:42 PM

our docker image is a thin wrapper around the meltano/meltano image:

Copy code

ARG MELTANO_IMAGE=meltano/meltano:v2.15.3-python3.11
FROM $MELTANO_IMAGE

WORKDIR /project

# Install any additional requirements
COPY ./requirements.txt .
RUN pip install -r requirements.txt

# Copy over Meltano project directory
COPY . .
RUN meltano install

# Install dbt dependencies
RUN meltano invoke dbt-redshift deps

# Don't allow changes to containerized project files
# see <https://docs.meltano.com/reference/settings#project_readonly>
ENV MELTANO_PROJECT_READONLY "true"

umar_saleem

02/28/2023, 8:00 AM

Thanks @Matt Menzenski! will try that !

10 Views

Open in Slack

Previous Next