One of our engineers is working on a target for ic...
# troubleshootingt
taeef_najib
11/29/2023, 1:24 PMOne of our engineers is working on a target for iceberg. He's using pyspark and struggling with it. Can meltano target use pyspark? I think it has java dependencies. Meltano does not support java, right? I know airbyte does. Any idea how to deal with this? Can we use pyspark inside meltano? If not, how can he write data in Iceberg format?
v
visch
11/29/2023, 2:01 PMMeltano can run any executable and pipe data to it so it can target java
e
edgar_ramirez_mondragon
11/29/2023, 3:36 PMThere's also https://py.iceberg.apache.org/
PyIceberg is a Python implementation for accessing Iceberg tables, without the need of a JVM.
t
taeef_najib
11/29/2023, 5:56 PM@visch thank you so much
@edgar_ramirez_mondragon I'll ask him to try pyiceberg instead
e
edgar_ramirez_mondragon
11/29/2023, 6:48 PMDo let me know how it goes!
t
taeef_najib
11/29/2023, 10:05 PM@edgar_ramirez_mondragon I think PyIceberg does not support writing data into Iceberg table. I could create table, but didn't find any way to write data from csv to Iceberg. Do you know any workaround??
e
edgar_ramirez_mondragon
11/29/2023, 10:21 PMAh, I saw references to pyarrow in their docs and thought it would simply use that to write to Iceberg: https://github.com/apache/iceberg-python/issues/23. That's unfortunate 😕.