In Meltano Office Hours on March 28, Mattitika demoed their Matte plugin that had just been added to the Hub. The example project was inspired by Chapstick data and it took some data from GitHub and put it in a postgres database. They showed how to use the UI utility available for running pipelines without needing Python or Multeno installed. The utility also allowed users to configure pipelines and run jobs with schedules, as well as view results in a postgres database through an API provided by the lab. They discussed ways of making this experience simpler for analytics users who don’t want to deal with cloud configurations.

Aaron, Ruben and Alex discussed the possibility of using Pex files to speed up installation of plugins. Pex are Python executable bundles that package python packages into a single file which is much faster than running a full Pip install at Runtime. They discussed potential barriers such as supporting multiple versions and version pinning, cross section compatibility for different platforms, security scanning processes for hosted code on the Hub, and an implementation strategy for middle tenor install or ad. As well as committing pre compiled text files to Meltano repos in order to skip download operations.

Meltano had a discussion about creating platform-specific wheels, informative exit codes, and return codes. It was suggested that the best solution would be for each project to manage itself at its own pace rather than trying to get official Hub support. They discussed using non-zero exit codes as a way of communicating information between taps and targets, while avoiding interference with existing code. The team also discussed having Maltamo create an error message when there is a user configuration error or tap failure, which could help users troubleshoot better. Finally, they agreed on not touching the first 128 exit codes and instead starting from 1280 onward in order to maintain consistency and document any custom errors properly.

This is an example Meltano project, and we’ve written a little article, and we will send that around. It’ll be nice to get this on the main Meltano examples because it’s pretty simple. It just takes some data from GitHub and puts it in a postgres database. And this was inspired by Chapstick data. We really liked what they demoed for the analytics, but we saw that the whole run the pipeline part was a little bit difficult, or at least it’d be a little bit difficult if you were just analytics user.