I have a feeling that many of the problems I'm having, such as state not persisting, or scheduler env vars not propagating etc. may be related to the fact that I'm using an airbyte variant of extractor, and maybe it's not set-up correctly. I just added an extractor and stated using it. I did not have docker installed but I was able to do full sync, etc. So, my question is, is there a detailed guide on what needs to be downloaded and ran in docker, and how the extractor needs to be set-up? I tried the configuration found on this page but now, I can't even pass in the required configurations (it throws an error). I may need a step-by-step guide on this. For example, the extractor page has a setting for 'Airbyte Spec Image' with default value

airbyte/source-hubspot

. Where do I get this spec image and how do I set it up to be ran? Thank you.

Maybe @Conner Panarella knows 🙂

@Kyunghwan Choi The documentation for the airbyte-wrapper needs to be updated to reflect this. But basically 99% of Airbyte sources are available to get via

pip

and don't need to be run in containers. There was an undocumented update to the repo that added a check so see if the source can be run locally instead of in a container. That's why you were able to run it without docker.

Oh, I see. Thank you. Then.. I guess the issues I was seeing are not related to that. Thank you.