Hey, Team. I have an ELT project in Meltano, where I collect data from APIs and various databases. We created our DW, and everything is great so far. However, now we need to develop RPA (robot automation process) to automate some activities using the DW data that Meltano generates.
Can we write these robots (using Python code with Selenium stuff, etc.) and schedule them for the Meltano project? I see two options: 1. Write the code inside the airflow folder (plugins, for example) and create a custom dag to use it, and 2. Create a Python project outside of Meltano, configure it in Meltano using the utilities CLI/folder, and schedule its airflow job as a utility.
The last option is to develop and schedule the RPAs in another infra/architecture. But, I would like to use Meltano (especially at the beginning) to avoid adding another infrastructure (orchestrator, ci/cd, etc.) at this moment.