Hi everyone, I’m happy to share that I have completed a working version of a new Netsuite SOAP tap. This tap is more robust than the existing ones because it does not rely on external libraries and it has a dynamic discovery. The Netsuite REST API is still lacking some features and the SOAP one is more stable, so I chose to use that one. I’m planning to work on improving the logging, backoff, pagination and overall code quality. If you have any feedback or suggestions, please let me know. I hope to have this tap ready for production by next week. You can check out the code here: https://github.com/sehnem/tap-netsuite Next week, I’m also thinking of creating a Shopify GraphQL tap. If you know of any other APIs that need a better tap, please let me know as well. I’m trying to build a portfolio before looking for opportunities.

It must be a Netsuite-y kinda day! I agree that the SOAP one is more stable, but I started work on a tap for the REST API today also if you're interested (very bare rn but I'll keep going): https://github.com/GtheSheep/tap-netsuite-rest

Excited to dig into both! Thanks!

@dan_ladd great, please let me know if you find anything that can be improved, I made some adjustments and now the tap is in a good shape.

@josue_sehnem this is excellent! Would you be interested in creating an issue for a generic

SOAPStream

class and

Connection

for the SDK? It would be epic to collect your feedback and learnings on SOAP with the SDK, as a starting point to adding native SOAP support 🙂 👍 Looks like there is an old issue for 🧼 already, but feel free to create a fresh one to point at the work you’ve done on tap-netsuite 🙌 🚀

@ken_payne sure, I can work on that, I am not very familiar with soap, just learned it for doing this tap, but I can study it further. Do you have any other soap api that I can develop a tap for? I think it would help me to get more familiar with this kind of API before putting together the common parts, I think that many of the methods used there are specific to the netsuite API and I need to figure out how things can be solved in a more generic way.

I am no expert either, but have worked with one or two soap endpoints before 😅 From what I remember, the WSDL file details the available endpoints, the supported methods and the returned object schemas. So a generic implementation would ideally handle discovery (list and create streams for each getable endpoint advertised) and schema translation from XML to JSONSchema for each created stream. This is similar to how the SQL support works (listing remote schemas and tables during discovery and converting db table schema to JSONSchema). Some compromises usually have to be made on typing, defaulting to simple string/object/number types for compatibility, but developers always have to option to override the default typing resolver in their own subclasses 🙂

I’m working on creating a new

Connection

and

SOAPStream

(or

SoapStream

) classes for the netsuite tap, based on the sql cookiecutter and sdk code. I think the conversion to JSONSchema should be the same, since I’m using the zeep types. However, I’m wondering if adding zeep as a dependency would cause any issues for the SDK. I saw in the issue discussion that you shared that some people suggested using only the request library, but I’m not sure if that’s a good idea, as it might make things too complex. The list of endpoint that will be fetched may not be totally dynamic, in cases as netsuite just some of the elements are fetchable and rely on other complex type that are used to fetch it. Will try to look at other API and see how it works in there (I don’t think it should be more complex than netsuite) Once I have basic POC I will let create new issue describing my approach.

Amazing @josue_sehnem 🙌 Your

tap-netsuite

is a great place to start a generic implementation, thank you 👍 As to dependencies, we can add zeep to the SDK as an optional dependency, so future soap taps can depend on

singer-sdk[soap]

if they want to leverage the soap base classes 🙂 We are planning to do the same for

sqlalchemy

in the next major version of the SDK for the same reason 🚀