https://linen.dev logo
Join Slack
Powered by
Hi! I am completely new to meltano, having just di...
# announcements
s
Hi! I am completely new to meltano, having just discovered the application. Hopefully this is the right place to ask this question. For testing we self-hosted, used tap-postgres to pull some data from our app db into target-postgres, then got stuck. Do we need to write any custom transforms/models for our data although it is postgres to postgres? Is Meltano not suited for pulling data from datasets outside of those listed here - https://meltano.com/plugins/extractors/?
d
...then got stuck
@stephanie Can you elaborate?
If you're saying that you expected Meltano's Analyze UI "to just work", then yes, you need to create custom transforms and models. You can learn more about that here https://www.meltano.com/tutorials/create-custom-transforms-and-models.html#creating-custom-transforms-and-models
Though not currently an immediate priority, we anticipate assisting in the generation of custom transforms/models (possibly through a UI wizard of some sort)
s
@derek_knox I was wondering how much you could natively 'probe' the data through the ui without having to write a custom model - because the data was moving from postgres to postgres, I assumed it would not need a custom transform. We wanted to explore what kind of visuals we could get from the data so didn't really have a plan or a model in mind as yet.
m
@stephanie the Model defines what you can see in the UI, so it is currently necessary to create one.
s
@micael_bergeron thanks for the clarification - the tool looks so cool, we really were hoping to use it! Unfortunately i am a mere devops engineer, not a data analyst, so this might be a bit beyond the time I have available to get this working at the moment 😅
m
@stephanie no worries, this is very valuable feedback for us. I know @yannis_roussos had an idea about auto-generating basic models so that might be a good use-case for it.
d
This use case may be worth exploring however. @stephanie @micael_bergeron I just created this issue to track it https://gitlab.com/meltano/meltano/issues/1663
This would be valuable generally I think when new extractors are added too (before a dedicated effort of more custom models occurs)
s
That sounds interesting, if you would like any other input from us please let me know! We would be looking to interpret data from a postgres db holding sales and order data from an opensource product called Odoo.
Oh also @micael_bergeron, after the tap I attempted to select some of our data streams but after selecting the first one, subsequent ones kept getting this error. I tried fresh installs a couple of times but each time got the same behaviour. Unsure if you care about this particular error or if its due to something in our data.
Copy code
meltano invoke tap-postgres --discover > hello
for line in `grep tap_stream_id hello | grep order | cut -d " " -f8 | tr -d '"' | tr -d ","` ; do meltano select tap-postgres $line ; done
---
[MainThread][INFO][2020-02-03 14:24:56,473] Context impl SQLiteImpl.
[MainThread][INFO][2020-02-03 14:24:56,474] Will assume non-transactional DDL.
System database up-to-date.
Error: 'list' object has no attribute 'add'
m
Humm that seems like a fair bug, could you please export the value of
Copy code
meltano invoke tap-postgres --discover > hello
for line in `grep tap_stream_id hello | grep order | cut -d " " -f8 | tr -d '"' | tr -d ","` ; do meltano select tap-postgres $line ; done
I'm curious to see what the invocations looks like.
s
Copy code
odoo-public-product_template_attribute_value_sale_order_line_rel
odoo-public-purchase_order_stock_picking_rel
odoo-public-sale_order_line_invoice_rel
odoo-public-store_store_order_rel
odoo-public-account_analytic_tag_purchase_order_line_rel
odoo-public-account_analytic_tag_sale_order_line_rel
odoo-public-account_invoice_purchase_order_rel
odoo-public-purchase_order
odoo-public-purchase_order_line
odoo-public-account_tax_purchase_order_line_rel
odoo-public-account_tax_sale_order_line_rel
odoo-public-sale_order_line
odoo-public-brand_sale_orders
odoo-public-daily_order
odoo-public-mrp_workorder
odoo-public-sale_order_option
odoo-public-sale_order_tag_rel
odoo-public-sale_order_template_line
odoo-public-sale_order_template_option
odoo-public-sale_order_template
odoo-public-sale_order_transaction_rel
odoo-public-sale_order
odoo-public-stock_backorder_confirmation
odoo-public-stock_picking_backorder_rel
odoo-public-stock_warehouse_orderpoint
odoo-public-store_order_service
odoo-public-store_order
m
Okay and doing 
meltano select tap-postgres odoo-public-purchase_order
fails?
s
Yeah, the very first one works, but anything after that fails
m
Oh, alright that might be a bug — let me open an issue.
s
Copy code
meltano select tap-postgres --list
[MainThread][INFO][2020-02-03 14:16:38,017] Context impl SQLiteImpl.
[MainThread][INFO][2020-02-03 14:16:38,018] Will assume non-transactional DDL.
System database up-to-date.
[MainThread][INFO][2020-02-03 14:16:38,171] Catalog not found, trying to run the tap with --discover.
Legend:
	selected
	excluded
	automatic

Enabled patterns:
	odoo-public-product_template_attribute_value_sale_order_line_rel.*

Selected properties:
	[selected ] odoo-public-product_template_attribute_value_sale_order_line_rel.sale_order_line_id
	[selected ] odoo-public-product_template_attribute_value_sale_order_line_rel.product_template_attribute_value_id
m
For the time being, you can the select patterns manually in the 
meltano.yml
file.
Look for the 
select:
key underneath the 
tap-postgres
block, you can add them there.
They are glob patterns, so don't forget the 
.*
at the end.
s
@micael_bergeron thanks, if i get some more time to play around, ill try reading up more on the models and try build one using the file directly as a workaround 🙂
m
Cool, you can find inspiration in our own models at https://gitlab.com/meltano?utf8=%E2%9C%93&filter=model-
7 Views
Open in Slack
PreviousNext
Powered by