Hi ! Not sure where to start with my question, I h...
# getting-started
a
Hi ! Not sure where to start with my question, I hope posting here its fine, if not please let me know the appropriated channel 🙂 I'm trying to I have set up a clone of the tap, and I want to use the
target-parquet
files, so I also added that to the project. I run the
meltano invoke tap-awin
and I get the data that I want, special attention to the
networkFee
field. But if I run the process complete as
meltano run tap-awin target-parquet
the field are not kept, ( second screenshot) I need to keep track of the networkFee object, I know this object has two properties
amount
and
currency
but the tap-parquet is unable to pick on this, The meltano.yml looks roughly as follows, any ideas what might be the issue? ( if I run the tap-jsonl then I get the data, but I'd prefer to use parquet format so I can later do some analysis using duckdb) plugins: extractors: - name: tap-awin capabilities: - state - catalog - discover variant: gthesheep pip_url: /Users/andres/melt/tap-awin config: lookback_days: 15 request_batch_size_days: 10 start_date: "2024-10-01" select: - transactions.transactionDate - transactions.networkFee loaders: - name: target-jsonl variant: andyh1203 pip_url: target-jsonl - name: target-parquet variant: automattic pip_url: git+https://github.com/Automattic/target-parquet.git config: destination_path: output/awin-parquet ``````
âś… 1
e
I believe the target does automatic flattening, so those fields should end up as
networkFee__amount
and
networkFee__currrency
. I believe
networkFee
is a field you added to your fork of the tap. If so, how did you define it?
a
Hi!, yes in my fork of the tap I added like this, as it is not supported out of the box, the interesting part is that if I try to be more specific and define the properties
amount
and
currency
in the tap repo, I get empty objects in return. I think it has something to do with the flattening, as I'm testing now
target-csv
with flattening, and I get these errors, so it canÂżt find a definition for
networkFee__amount
and
networkFee__currency
but I don't know where to define those. Thanks for your help!
e
so you already tried something like
Copy code
th.Property(
    "networkFee",
    th.ObjectType(
        th.Property("amount", th.NumberType),
        th.Property("currency", th.StringType),
    ),
)
?
a
Yes I tried that, and for some reason the results from the tap were empty objects
{}
I did manage to solve my problem, using the tap in discover mode I got the
catalog.json
then I modified it to add the properties
amount
and
currency
to the schema, and then using
meltano el tap-awin target-csv --catalog catalog.json
thanks for your time and help! 🙂
🙌 1