Meltano

Anyone have any recommendations on a relatively simple - preferably global - way to limit the number of extractions for each extractor ; say, for setting up test environment ?

My go-to would be to use the <https://docs.meltano.com/concepts/environments|environments> feature to override some configuration to limit the data pull (if there’s a `start_date` config for example).

If it’s an SDK-based tap you can --test too to just get one record <https://sdk.meltano.com/en/latest/cli_commands.html?highlight=test#test-connectivity>

Speaking of environments - are there any good example meltano projects (specifically their meltano.yml ) showcasing environments ?

(community sourcing some ‘best practices’ Meltano projects in the MeltanoLabs org might be a fun resource, if something like that doesn’t exist yet)

Yes! Our `squared` project uses environments quite a bit <https://gitlab.com/meltano/squared/-/blob/master/data/environments.meltano.yml> :smile:

We use this to power internal metrics and the Hub data

We’ve had a few requests for example public projects. <@U06C8MQ6JNR> we could put a section like this in the docs and make it easy for folks to add theirs!

Ah, ok, I was poking around the squared project but missed that section.

<https://gitlab.com/meltano/meltano/-/issues/3327>

<@U06CE59HKBN> it’s in a separate file to show-off the multi-file support too :smile:

Yeah, I think it’s a lot clearer using the multi-file support as well

In fact this particular setup I intuitively recognize as being analogous to `docker-compose.yml` vs `docker-compose.override.yml` (as an example)