Not sure the best place to put this question, but this seems like a decent option. I’m curious if anyone has suggestions about ways to optimize the build process for a meltano docker container to help make it easier to make iterative changes quickly? Here’s what I mean: This is the standard meltano dockerfile:

ARG MELTANO_IMAGE=meltano/meltano:latest
FROM $MELTANO_IMAGE

WORKDIR /project

# Install any additional requirements
COPY ./requirements.txt .
RUN pip install -r requirements.txt

# Copy over Meltano project directory
COPY . .
RUN meltano install

# Don't allow changes to containerized project files
ENV MELTANO_PROJECT_READONLY 1

# Expose default port used by `meltano ui`
EXPOSE 5000

ENTRYPOINT ["meltano"]

The

meltano install

line needs

meltano.yml

(and any other yml files if they’ve been broken out) to know which plugins to install, and so the line before it copies all those files into the image. This means that if one makes a change to

meltano.yml

, like changing the `select`ed fields for a particular tap, the image will fully reinstall all the plugins when the container is rebuilt. This can really slow down the iteration process, so I’m wondering if there’s a way to refactor this dockerfile to avoid that. For example, might there be a way to get

meltano install

to run based on a single yml file that gets copied in first, then restructure some other yml files that capture all the field level configuration so that we can take more mindful advantage of the image layers? Any ideas/suggestions from others who have run into this?

fwiw I've hit this as well. I just eat the time loss as its' so nice things just rebuild and work 🤷 Where I've hit some "show stoppers" here is when my project is installing 20+ plugins and a few of the PIP HTTP calls retry too much, then I have to rerun meltano install 1hr plus and then a retry on top of that 😮

Soooo I’ve been screwing around with this too and have just started passing flags on small subsets of what I want to install with make

ah interesting, does that mean instead of doing the

meltano install

type commands explicitly in your dockerfile you’ve moved the installation steps into Makefiles and have the docker file run those as you need them?

Lol yes

got it, that seems like a viable option

I’m really digging it!

ohh yeah, I bet that gets gnarly quickly in a situation like this

Love the simplicity of just

meltano install tap-1

tap...N, simple easy fix I like it

I’m working on 3 different viz options so I’m sort of splitting options in the make file so you can build only the pieces you need

@visch, that might work too! it’s really only the taps that change frequently, so I could probably copy over the yaml files for all the others first, do a generic

meltano install

, then explicitly install all the taps separately, THEN copy over the yaml files for the taps after they’ve been installed so that they’re added in a downstream layer

Meltano could probably help here but it doesn't seem simple on that side. Would need a cache system, but maybe there's something magic in

pip

that could be used 🤷

hmm yeah, thanks for those pointers/callouts!

@nick_hamlin I think you should create an issue for this. I think your instinct that it feels wrong means its a missing feature.

It’s also possible that there’s a way to get multiple yml files configured in such a way that the field selection is handled separately from the core config, which would fix the “wrongness” I’m feeling. I’m going to tinker a bit more with that approach, but will put in an issue if that turns out to be a dead end.

Okay, I have had this discussion as well. What I have done is not place any table selects or metadata like the type of replication into my meltano.yml file. Reason, what changes mostly is what tables I want to ingest not adding new taps or targets. To work out what tables to select and what the metadata should be for each table we use two Meltano environment variables for each tap to set the tables we want to select and how to replicate them. These are set at run-time via Chamber which grabs the values and sets them when running the pipeline. The key environment variable to use are: _SELECT and _METADATA Example: export TAP_SYBASE__NMD__SELECT='["dwh_extract-table1.*","dwh_extract-dwh_table2.*","dbo-table3.*","dbo-table4.*","dbo-table5.*"]' export TAP_SYBASE__NMD__METADATA='{"dwh_extract-dwh-table1": {"is-view":false, "replication-method": "FULL_TABLE", "table-key-properties": ["column_1"]}}' here is another example of the _METADATA environment variable with a default replication method of full table, and overriding the primary key to be empty. This means that target_snowflake will insert the data rather than merge the data. export TAP_SYBASE__NMD__METADATA='{"*": {"replication-method": "FULL_TABLE"}, "dwh_extract-dwh-table1": {"replication-method": "FULL_TABLE", "table-key-properties": []}}'