https://linen.dev logo
Join Slack
Powered by
Hello everyone, Ive been trying the meltano projec...
# announcements
v
Hello everyone, Ive been trying the meltano project for the past days and I'm really liking it so far, thank you! I have a question about the elt command, does it always run the tap in discover mode before running the extaction itself? I have a very large database that makes this discovery slow, anyway to skip discovering at etl time?
d
@vitor_avancini Glad you've been liking it so far! 
meltano elt
does always run discovery mode first, because it dynamically generates the appropriate catalog file for sync mode based on the specified selection and metadata rules (and soon schema rules as well: https://gitlab.com/meltano/meltano/-/issues/2145). Are you using selection and metadata rules, or always using the discovered catalog file as-is?
Implementing some kind of automatic discovery/catalog caching mechanism would be tricky, because we'd only know when to invalidate the cache by running discovery mode to see if anything changed, which defeats the point... In your case, you may want to explicitly say you're fine with caching it, but that'd also mean you'd need to explicitly inform Meltano to invalidate the cache when the schema changes, and until you would, your 
meltano elt
would likely be failing.
v
Im using select config, im only extracting one table
For testing purposes
d
Which tap are you using?
v
TapMssql
I understand that the cache invalidation would be tricky
Ans the i wont die waiting for the discovery but if there was a way of chosing to use cache like you said would be nice
d
Yep, makes sense. I could imagine something like an opt-in 
--catalog-cache
or 
--discovery-cache
argument on 
meltano elt
. Then to invalidate the cache, you'd just need to run 
meltano elt
once without that argument, so that the new catalog gets discovered and stored, to be used by subsequent runs with the cache argument.
v
Yeah i thought of something like that.
d
It'd be up to the user to know when the cache needs to be invalidated, probably as a result of some catalog related failures in their pipeline.
v
Makes sense
But ill keep going without it for now
d
Would you like to create an issue in https://gitlab.com/meltano/meltano/issues for this? 🙂 It's more of a nice-to-have than an absolute must, so I don't expect I'll be able to get to it any time soon, but it'll be good to at least start tracking it, in case some member of the community (you, perhaps? 😉) will feel like contributing it!
v
Also if anyone needs to extract from sql server I managed to do it
Its a hacky way buts its working so far
d
v
Yeah sure! Ill creatw the issue
Im using a wrapper ive made for the singer one
d
Ah, interesting. What kind of wrapping was necessary?
v
The singer tap is written in clojuere
d
Yeah I saw that
v
Wrote a python package that runs the clojure tap as a subprocess
Hacky hahah
d
Haha yeah, but I'm glad it works for you 🙂 Not something we could support out of the box with 
meltano add extractor tap-mssql
, though, since that can only do 
pip install
, not also set up the whole clojure env and dependencies
I had been thinking about adding native support for Dockerfile-based taps as opposed to pypi package-based ones at some point
v
It would make the world of extraction bigger
Would be nice
The meltano add extractoe works for me with mssql
d
Right, because you already have the Clojure tap installed, right?
v
But you need a jvm set in your environment, its the only req ive noticed so far
d
Ah, right
v
Meltano installs it for me from local disk
-e extract/tapmssql
d
But I assume your Python code depends on the actual Clojure 
tap-mssql
executable already being present on the system / in the $PATH?
v
No ive packaged the clj code with my python entry point
It runs relatively to the python file
d
Interesting, so in theory you could put it up on PyPi, and someone running 
pip install tap-mssql
would get all the necessary bits, including the Clojure source, so everything would work as expected assuming the JVM is installed?
v
Yeah
Actually right now you also need lein
The clj executor
But im pretty sure we couls build the uberjar
And skip lein lart
Part
d
Ah yeah. Still though, we should be able to add this as a known plugin (https://meltano.com/docs/contributor-guide.html#known-plugins) so that users can run 
meltano add extractor tap-mssql
and be off to the races, assuming they read the documentation stating the JVM needs to be installed
v
Yes we should, i havent used it enough to be comfortable adverising it
But its definitely possible
d
No worries, I'm getting ahead of ourselves anyway 🙂
But once you're comfortably running this in production, it'd be awesome to see if we can bring that power to all Meltano users!
v
Yeah i intend to do that
Actually if things go my way, ill be adding 5-10 taps to meltano
From Brazilian sources
d
😍
v
Meltano fits very well with our tooling
We use airflow dbt and singer taps
In our prod pipelines
d
Oh that's great! I'd love to make Meltano work for you, even if you may still run into some limitations today, especially around our dbt integration. But we want it all to work, and you'll find that we're very responsive to feedback and suggestions 🙂
v
Nice, Ill be around! Thank you Douwe
d
My pleasure, looking forward to chatting more
v
Douwe, I'm not beeing able to figure out how to add a custom dbt project
Can you point me to docs where explains how to do that or somewhere in the source code where would clarify that
d
@vitor_avancini The conversation in this thread may contain some useful bits of information, but I haven't translated that into better docs yet: https://meltano.slack.com/archives/CFG3C3C66/p1593308187353200
Once you've read that thread, please let me know what else I can clear up 🙂
v
allright
thanks!
ah one more question, do custom taps shows up in explore section of the webapp?
Im trying to look at at source code but im not fully aware of where things happen there, so im sorry for the many questions
d
@vitor_avancini If transforms and models are present, yes: https://meltano.slack.com/archives/CFG3C3C66/p1594047795439800
No worries about the questions, it's my job to answer them and/or make sure the answers are documented so that people won't have them 😛
v
cool!
I have one more then
haha
Do meltano use the singer metrics for any purposes?
Ive been trying to figure a way to monitor and get metrics of this singer pipelines for almost an year
didnt achive much yet
d
@vitor_avancini Not yet, but @nehil_jain had the same question 2 days ago: https://meltano.slack.com/archives/CFG3C3C66/p1594158421009100 which led to the creation of these 2 issues: https://gitlab.com/meltano/meltano/-/issues/2148 https://gitlab.com/meltano/meltano/-/issues/2150
v
nice
ill try to look at it this weekend
d
@vitor_avancini Awesome! This is definitely the direction we want to see the UI go in, so any feedback would be greatly appreciated 🙂
v
Another thing, the pipeline schedule, is it dependant on airflow or another scheduler/orchestrator to run?
meltano itself doesnt have a scheduler right?
d
@vitor_avancini That's correct. 
meltano schedule
adds to 
meltano.yml
under 
schedules
, and some kind of orchestrator is then expected to run 
meltano schedule list --format=json
to get those schedules with their metadata, and schedule specific 
meltano elt
jobs for each. That's exactly what the Airflow DAG generator does: https://gitlab.com/meltano/files-airflow/-/blob/master/bundle/orchestrate/dags/meltano.py, but you can write something similar for any other orchestrator if you want to use 
meltano schedule
When you run 
meltano add orchestrator airflow
, Meltano will install Airflow, add that file to your project, and point Airflow to that directory to look for DAGs
(Note that you can also use an existing Airflow installation: https://meltano.com/docs/orchestration.html#using-an-existing-airflow-installation)
v
nice
thanks again!
do you know if its possible to do glob negation for selecting metadata? Im trying to set replciation method for 'a' and 'b' incremetnal, and the rest full_table something like: select: - table_a.* - table_b.* - table_c.* - table_d.* - table_e.* - table_f.* metadata: table_a: replication-method: INCREMENTAL replication-key: cda table_b: replication-method: INCREMENTAL replication-key: cdb '*!(table_a|table_b)': replication-method: FULL_TABLE
it lost identation, sorry for that
d
@vitor_avancini Meltano iterates over every stream and then applies the metadata rules in order, so this may just do it:
Copy code
metadata:
  "*":
    replication-method: FULL_TABLE
  table_a:
    replication-method: INCREMENTAL
    replication-key: cda
  table_b:
    replication-method: INCREMENTAL
    replication-key: cdb
v
awesome
ive tried the other way around
haha
it all became full table
d
yeah, order is important here 😉
9 Views
Open in Slack
PreviousNext
Powered by