Hi everyone, I have successfully added a new extractor (tap) and loader (target) and the ELT process works fine in command line. Now I am trying to use it in the Meltano User Interface. How can I integrate new extractor and loader with Meltano UI ? I made changes to the settings property in

meltano.yml

which is located in project folder. But when I start Meltano UI (

meltano ui

), I don't see any changes in UI. I think that is supposed to generate a configuration form. I also tried to add the settings property of tap and target to another

discovery.yml

file which is located in

myproject/.meltano/cache/discovery.yml

. But the same, this didn't help. In this case, after running

meltano ui

,

discovery.yml

is updated to previous version.

Did you add the extractor using the

--custom

flag as per https://meltano.com/tutorials/create-a-custom-extractor.html#create-the-plugin-s-package?

@zafar Try copying

.meltano/cache/discovery.yml

to the root of your project directory. Per https://meltano.com/docs/contributor-guide.html#known-plugins, that file (if it exists) always takes precedence over the cached copy or the one downloaded from meltano.com/discovery.yml 🙂

@al_whatmough yes, I added the extractor using the 

--custom

 flag. yes I see references to the extractor in your 

meltano.yml

@zafar Did you also copy

<project>/.meltano/cache/discovery.yml

to

<project>/discovery.yml

and add the new extractor in there? The API endpoint and UI list all known plugins, so plugins only added to

meltano.yml

won't show up. Copying the plugin definition you provided into that

discovery.yml

made it show up in the API endpoint and UI for me, but without a logo (understandably). Clicking "Connect" in the UI fails with some errors related to a missing

docs

option on the plugin definition (issue to fix this: https://gitlab.com/meltano/meltano/-/issues/2042), which can be worked around by adding

docs: "<https://meltano.com/plugins/extractors/firstexample.html>"

, even though that URL currently 404s. The result looks like this:

@douwe_maan Thank you for your detailed and clear explanation. Now it is working. I believe, Meltano had loader settings in UI, but as you said now it is not available. Tomorrow we also want to talk with you and Mark about loader settings in UI. I think loader in the UI is very important in this project

@zafar Sounds good! Feel free to weigh in on the related discussion in https://meltano.slack.com/archives/CFG3C3C66/p1589895254131000 🙂

@douwe_maan How we can add logo for custom taps? is it possible?

@zafar The logos all live in this directory in the Meltano project: https://gitlab.com/meltano/meltano/-/tree/master/src/webapp/public/static/logos and are named

X-logo.png

for

tap-X

. Do you have a development environment for Meltano UI set up, or are you using

meltano ui

from the

meltano

package?

I am using

meltano ui

right now

All right, you'll want to follow the steps under https://meltano.com/docs/contributor-guide.html#setting-up-your-environment and https://meltano.com/docs/contributor-guide.html#ui-development to get a development build of the Meltano UI running at http://localhost:8080/. Then, you can add the logos to the

src/webapp/public/static/logos

directory of your local clone of the meltano/meltano repo

@douwe_maan

make bundle

is giving me error:

rm -rf ./build ./dist ./src/meltano/api/static/js ./src/meltano/api/static/css ./src/webapp/dist

cd src/webapp && yarn && yarn build

00h00m00s 0/0: : ERROR: There are no scenarios; must have at least one.

Makefile:117: recipe for target 'ui' failed

make: *** [ui] Error 1

@zafar I've seen that issue before: https://gitlab.com/meltano/meltano/-/issues/1944 The error message suggests that you have the wrong

yarn

installed; see that issue for more details!

@douwe_maan thank you, this https://github.com/yarnpkg/yarn/issues/2821#issuecomment-284181365 and upgrading the node helped for me. All process was successful, but now I don't see taps and loading http://localhost:8080/ never ends. Only http://localhost:5000/ works fine without logos

@zafar All right, let's see what's up with

localhost:8080

then. Do you see anything in your web inspector console?

allowing CORS in browser doesn't help

@zafar Are you running

meltano ui

with

FLASK_ENV=development

as suggested under https://meltano.com/docs/contributor-guide.html#api-development?

I missed that step. After

make bundle

, now what I did is: 1. Started

FLASK_ENV=development meltano ui

in project directory 2. Opened a new terminal tab,

yarn setup

;

export MELTANO_EMBED=1

;

yarn serve

in cloned meltano folder

@zafar Hmm, that's weird. You shouldn't need

MELTANO_EMBED=1

, but other than that what you're doing is working locally for me. Can you inspect the Network tab and have a look at the response headers for any of the CORS-preflight OPTIONS requests?

after turning off CORS extension in browser, it works now. Thank you!