Hi, I am writing a shopify ELT with plugins available in meltano. I succeeded in EL but having issues in transformer. extractor: tap-shopify loader: target-postgres transform: tap-shopify transformer: dbt shopify model's git: https://gitlab.com/meltano/dbt-tap-shopify

Hi @mubeen! What warehouse and loader plugin are you using? Depending on your type of warehouse (snowflake, bigquery, etc.) and which variant (pipelinewise, etc.), some columns may be landed as raw json objects or flattened into

object__nested

columns. That and the fact that it's nearly impossible to write warehouse-agnostic SQL are the reasons we're moving away from recommending

transform

packages like

dbt-tap-shopify

.

my loader/database is postgres.

for loader target-postgres im using transferwise variant

ok, so that might explain things. Afaik the

dbt-tap-shopify

package was developed with the target-postgres Meltano variant. The transferwise variant supports a data_flattening_max_level setting that produce similar results.

thanks im trying meltano variant

Hi Mubeen. We've recently ported several of the dbt-tap-* transforms from target-postgres--meltano to target-postgres--pipelinewise, and target-snowflake--meltano. I'll get @daniel_walker to ping you a list of things we ran into, but from memory: • column naming between the target-postgres variants was different (we actually forked the pipelinewise variant to add a setting that would name the columns the same as the meltano variant, but I'm going to look again at the data_flattinging_max_level setting today) • dates handling between postgres and snowflake was different • dbt settings are different, I think we decided to create a new dbt profile for the target-postgres pipelinewise variant. We are think about building the tap-shopify in the SDK and also converting the dbt-tap-shopify. Keen to see if we can collaborate on this if you want to!

The main issue I ran into was the column names being output differently for the different

target-postgres

variants. The

Meltano

variant breaks apart CamelCase field names with underscores, then lower cases the whole name, while the

Transferwise

variant just lower cases the names.