Still facing issues with tap-shopify, particularly with orders. I’ve found if I do tap-shopify to target-jsonl, I get the data in a reasonable enough state. But if I use target-athena or target-csv the columns are offset.

plugins:
  extractors:
  - name: tap-shopify
    variant: singer-io
    pip_url: tap-shopify
    select:
    - orders.*
  loaders:
  - name: target-athena
    namespace: target_athena
    pip_url: git+<https://github.com/dataops-tk/target-athena.git>
    executable: target-athena
    settings:
    - name: s3_bucket
    - name: athena_database
    - name: aws_region
    - name: aws_access_key_id
    - name: aws_secret_access_key
    - name: s3_staging_dir
    - name: compression

I know there’s some issues with the schema per https://meltano.slack.com/archives/C013EKWA2Q1/p1621895322025000 and https://github.com/singer-io/tap-shopify/blob/master/tap_shopify/schemas/orders.json#L9-L14 have been trying to update catalog.json to override schema but no luck yet. Is this an issue with the delimiter and quotchar on Athena side? In a related note, I’m unable to experiment with \t delimiter. Any help appreciated at this point!

Hi @nicholas_degiacomo

I’m unable to experiment with \t delimiter

Do you mean the target is not picking it up from configuration, so something else? Another user went through the process of overriding the schema a few weeks ago: https://meltano.slack.com/archives/C01TCRBBJD7/p1631054350102000 As for the misalignment problem, I'm almost sure it's a bug on `target-athena`'s side not escaping some characters in the data. cc @aaronsteers

thanks @edgar_ramirez_mondragon, I’ve connected with @royzac and he’s facing the same issue that i don’t believe was resolved As for CSV delimiter, if I configure within the yaml and run, I get a “delimiter must be a 1-character string” error. I’ve tried \t, b\t, and “\t”

hmm. quoting like

"\t"

in yaml should pass it to Python as the right character afaik

Hi, @nicholas_degiacomo. It's very possible that you are right and that something is causing the cell shift. I've certainly seen something like this before in csv handlers. Do you mind opening an issue there in target-athena with specifics of the issue you are seeing? And if you have time to start a PR or propose a fix, that would also be very helpful.

@aaronsteers since the issue seems to happen for both target-csv and target-Athena for the same tap, should I make issues in both?

That would be great. You can link them to save some copy pasting. 😅 For my info, which target-csv are you using?

git+https://github.com/hotgluexyz/target-csv.git@0.3.3

Hello, Does anyone have an update about this problem?

Hi @andre_anastacio! @visch came up with a workaround for the misalignment issue

Created this for your pain @andre_anastacio https://github.com/MeltanoLabs/target-athena/pull/38/files 😄

And... merged. shipitparrot

💥