Do i right understant, that if i change all logger.info in tap-facebook, or remove they, it’s not break tap-facebook?

@anton_parfenyuk Correct, those are just log messages, not something either tap-facebook or Meltano actually depends on

@douwe_maan Nice! Also, maybe you know, why tap-facebook not work in multithreads mode for load data from multiple ads accounts?

Also, maybe you know, why tap-facebook not work in multithreads mode for load data from multiple ads accounts?

Just because no one has implemented that yet 🙂 If you do so, I'm sure many users would be very grateful! Note that you can also just run multiple

meltano elt

pipelines with

tap-facebook

in parallel, if you provide a different

TAP_FACEBOOK_ACCOUNT_ID

each time (see https://meltano.com/docs/command-line-interface.html#pipeline-specific-configuration).

So i modify code for multithreading. It can break anything? Maybe facebook can block API access?

You'd need to make sure you handle rate limiting correctly: https://developers.facebook.com/docs/marketing-apis/rate-limiting#adaccountlimit, but I wouldn't be surprised if that's already handled by the

facebook_business

package the tap uses (which we're in the progress of bumping to the latest version in https://gitlab.com/meltano/tap-facebook/-/merge_requests/5).

Also, do it possible make in meltano option for change log type from Details (current log) to easy log with info only about current progress?

Right now,

meltano elt

prints all tap and target output, but we could definitely add an option to silence the tap or target or both! If that's something you'd like, would you mind creating an issue for it? 🙂

About output log i mean next: When we load data from multiple ads accounts we have next problem - we don’t know estimate progress. So if we got log in format: -Founded 300 accounts Start loading for ac_1, stream name ,current_date / end_date , streams_left for each account.

Ah, interesting. What the tap logs is not something Meltano usually has influence over, unless the tap provides some kind of setting to manage the log level. Meltano can really only print all of the tap's logs, or nothing; it doesn't really know what log messages correspond to progress or debug information.

How variant we can send additional parameter to Log. For example: INFO advance(ads_insights, 2020-04-04) it’s full log INFO [SIMPLE] advance(ads_insights, 2020-04-04)

Something like that should be relatively easy to implement, you're right, but I don't think Meltano should be making assumptions around how taps and targets behave, and how they intend for their log messages to be interpreted, that aren't explicitly documented in the Singer specification 😬 I think a better option to address your specific problem would be to add a

verbosity

or

log_level

setting to

tap-facebook

, which you can configure just like any other setting. The tap itself would then adjust its logging behavior accordingly.

Yeah, it’s look better.

Are you suggesting a Meltano setting to change the log directory from

.meltano/run/elt/...

to something else, or a

tap-facebook

setting, so that we can output simple logs, and store full logs?

I think that for now I’ll try create own solution for tap-facebook, because now understand how it works. And in future can try add to another taps. I’ll work with it and when complete - send merge request to repository.

Sounds good!