https://linen.dev logo
Join Slack
Powered by
Quick question... and my advance apologies if I mi...
# announcements
a
Quick question... and my advance apologies if I missed this is in the docs... : How does one recreate the 
.meltano
directory, since this directory is hidden from git and not exposed to source control?
y
Don't we recreate it the moment you run any meltano command?
If not, then that's a bug for sure
a
Let me try that and report back... 🙂
y
If you pushed a meltano project on a git repo and then got it back somewhere else, start by first running 
meltano install
a
@yannis_roussos - Thanks so much. Yes, 
meltano install
seems to be doing the trick. Previously I was getting messages like the below, and there were no 
venv
folders under 
.meltano
. For reference, the below is what I was seeing without the install command:
Copy code
Failed to start plugin <meltano.core.plugin.singer.tap.SingerTap object at 0x7f32278799d0>.
before_invoke hook 'run_discovery_hook' has failed: [Errno 2] No such file or directory: '/projects/meltano-project/.meltano/extractors/tap-salesforce/venv/bin/tap-salesforce': '/projects/meltano-project/.meltano/extractors/tap-salesforce/venv/bin/tap-salesforce'
Thanks!
@yannis_roussos For my future reference, (1) would 
meltano upgrade
also work for this purpose? (2) would I also run 
meltano install/upgrade
if I'm not 100% sure everything is properly installed - for instance, if my container might have been killed after partial install? These are relevant because I notice 
meltano install
takes quite a long time to execute.
I think I answered my questions above - (1) 
meltano upgrade
does not appear to work, as it appears to upgrade just meltano itself (via pip) and does not install the virtual environments for the plugins (2) subsequent calls to 
meltano install
appear to start over from scratch, re-installing all software, and can take over 20+ minutes to complete. as such, this isn't easily rerunnable and should probably only be executed by exception, or when 
.meltano
folder is missing. Happy to be corrected if any of the above are mistaken. Thanks!
For ref, here's my log of a pretty bare-bones 
meltano install
. Duration was 41 minutes when using salesforce and postgres. Is it expected that this would take so long, and is there anything on the roadmap to reduce this or run them in parrallel?
Copy code
Wed Nov 13 23:22:25 UTC 2019
Installed 'tap-salesforce'.
Installed 'target-postgres'.
Installed 'airflow'.
Installed 'dbt'.
4/4 plugins installed.
Install complete Thu Nov 14 00:03:47 UTC 2019
y
@aaronsteers yes 
meltano install
is the way to go. At the moment it installs everything and should not take more than a minute or two for the 4 packages above. Most probably something is not properly running on your environment. Do you have enough memory / bandwidth?
With respect to doing everything from scratch: We currently fully depend on pip for the package management (we do not incorporate a package management layer in Meltano to check for versions, etc) But pip should have not taken that much time ansd should have used the locally cached packages from your previous 
pip install
a
Thanks @yannis_roussos. I now believe the reason 
meltano install
is taking so much longer on my machine is that I am using the new WSL2 Backend for docker (using the latest Windows insider builds and the latest Edge version of Docker Desktop). My project folder is hosted on windows and shared as a volume to the linux container. Anyway, there are apparently known performance issues related to WSL performance when running linux apps on Windows filesystem and those issue would explain the much slower performance I'm seeing during installs. At least in initial builds ow WSL2, the recommendation from Microsoft is to host the files on the Linux filesystem, which is now also accessible from Windows. Since the older/stable Docker Desktop backend is Windows-based, I could also try reverting to the non-WSL version of the Docker backend. Refs: - https://docs.microsoft.com/en-us/windows/wsl/wsl2-ux-changes#place-your-linux-files-in-your-linux-root-file-system - https://docs.microsoft.com/en-us/windows/wsl/wsl2-ux-changes#cross-os-file-speed-will-be-slower-in-initial-preview-builds
y
Great info @aaronsteers, thank you for sharing that! I am personally using pure linux and a Mac so I have never been exposed to those
a
Posting my latest update back on this thread. I've confirmed that reverting to the standard/stable (Hyper-V-based) docker backend resolved the performance issues I was seeing with 
meltano install
on Windows Docker over WSL2. The install process now just takes a few (5 to 7) minutes to complete instead of 40+.
3 Views
Open in Slack
PreviousNext
Powered by