Andres Felipe Huertas Suarez
11/25/2024, 4:32 PMtarget-parquet
files, so I also added that to the project. I run the meltano invoke tap-awin
and I get the data that I want, special attention to the networkFee
field. But if I run the process complete as meltano run tap-awin target-parquet
the field are not kept, ( second screenshot) I need to keep track of the networkFee object, I know this object has two properties amount
and currency
but the tap-parquet is unable to pick on this, The meltano.yml looks roughly as follows, any ideas what might be the issue? ( if I run the tap-jsonl then I get the data, but I'd prefer to use parquet format so I can later do some analysis using duckdb)
plugins:
extractors:
- name: tap-awin
capabilities:
- state
- catalog
- discover
variant: gthesheep
pip_url: /Users/andres/melt/tap-awin
config:
lookback_days: 15
request_batch_size_days: 10
start_date: "2024-10-01"
select:
- transactions.transactionDate
- transactions.networkFee
loaders:
- name: target-jsonl
variant: andyh1203
pip_url: target-jsonl
- name: target-parquet
variant: automattic
pip_url: git+https://github.com/Automattic/target-parquet.git
config:
destination_path: output/awin-parquet
``````Edgar RamĂrez (Arch.dev)
11/25/2024, 5:37 PMnetworkFee__amount
and networkFee__currrency
. I believe networkFee
is a field you added to your fork of the tap. If so, how did you define it?Andres Felipe Huertas Suarez
11/25/2024, 6:24 PMamount
and currency
in the tap repo, I get empty objects in return. I think it has something to do with the flattening, as I'm testing now target-csv
with flattening, and I get these errors, so it canÂżt find a definition for networkFee__amount
and networkFee__currency
but I don't know where to define those. Thanks for your help!Edgar RamĂrez (Arch.dev)
11/25/2024, 6:54 PMth.Property(
"networkFee",
th.ObjectType(
th.Property("amount", th.NumberType),
th.Property("currency", th.StringType),
),
)
?Andres Felipe Huertas Suarez
11/26/2024, 7:31 AM{}
I did manage to solve my problem, using the tap in discover mode I got the catalog.json
then I modified it to add the properties amount
and currency
to the schema, and then using meltano el tap-awin target-csv --catalog catalog.json
thanks for your time and help! 🙂