Hi ! Not sure where to start with my question, I hope posting here its fine, if not please let me know the appropriated channel 🙂 I'm trying to I have set up a clone of the tap, and I want to use the

target-parquet

files, so I also added that to the project. I run the

meltano invoke tap-awin

and I get the data that I want, special attention to the

networkFee

field. But if I run the process complete as

meltano run tap-awin target-parquet

the field are not kept, ( second screenshot) I need to keep track of the networkFee object, I know this object has two properties

amount

and

currency

but the tap-parquet is unable to pick on this, The meltano.yml looks roughly as follows, any ideas what might be the issue? ( if I run the tap-jsonl then I get the data, but I'd prefer to use parquet format so I can later do some analysis using duckdb) plugins: extractors: - name: tap-awin capabilities: - state - catalog - discover variant: gthesheep pip_url: /Users/andres/melt/tap-awin config: lookback_days: 15 request_batch_size_days: 10 start_date: "2024-10-01" select: - transactions.transactionDate - transactions.networkFee loaders: - name: target-jsonl variant: andyh1203 pip_url: target-jsonl - name: target-parquet variant: automattic pip_url: git+https://github.com/Automattic/target-parquet.git config: destination_path: output/awin-parquet ``````

I believe the target does automatic flattening, so those fields should end up as

networkFee__amount

and

networkFee__currrency

. I believe

networkFee

is a field you added to your fork of the tap. If so, how did you define it?

Hi!, yes in my fork of the tap I added like this, as it is not supported out of the box, the interesting part is that if I try to be more specific and define the properties

amount

and

currency

in the tap repo, I get empty objects in return. I think it has something to do with the flattening, as I'm testing now

target-csv

with flattening, and I get these errors, so it can¿t find a definition for

networkFee__amount

and

networkFee__currency

but I don't know where to define those. Thanks for your help!

so you already tried something like

th.Property(
    "networkFee",
    th.ObjectType(
        th.Property("amount", th.NumberType),
        th.Property("currency", th.StringType),
    ),
)

?

Yes I tried that, and for some reason the results from the tap were empty objects

{}

I did manage to solve my problem, using the tap in discover mode I got the

catalog.json

then I modified it to add the properties

amount

and

currency

to the schema, and then using

meltano el tap-awin target-csv --catalog catalog.json

thanks for your time and help! 🙂