Hey. I'm running an initial run for a large MySQL instance containing of 56 databases with two tables each. The tables contain up to two million rows each. The MySQL instance is running in Google Cloud SQL and the pipeline connects to it through a Cloud SQL Proxy. The pipeline targets a Google BigQuery dataset located in the same zone, but a different project, as the Cloud SQL database. The pipeline runs in a container in a google compute instance, also in the same zone as the database. After a few hours I get this error message

meltano         | Incremental state has been updated at 2021-08-24 07:01:41.185403.
meltano         | Extraction failed (1): AttributeError: 'NoneType' object has no attribute 'settimeout'
meltano         | ELT could not be completed: Extractor failed

After restarting the pipeline it seems to pick up where it crashed. Any ideas on what might be happening?

@par_degerman Can you share your schema with the settimeout field definition?

Is a thread with this problem in common

Extraction failed (1): AttributeError: 'NoneType' object has no attribute 'settimeout'

https://meltano.slack.com/archives/CFG3C3C66/p1610570608209000?thread_ts=1610560265.206500&cid=CFG3C3C66

@charley_guillaume There's actually no column or table in any of the databases named settimeout. @boggdan_barrientos Thanks for pointing me in the right direction, should have searched before. 🙏

@par_degerman u welcome! I have the same issue in one elt. Increment the buffer size works for me, but I'm still debuging it.

Interesting indeed. The circumstances do support the buffer size theory since it crashes "at random", but always when syncing a table that is both long and wide. I will tweak buffer size and also deselect a few columns which are unnecessary. I'll report back here with my findings.

We just went through this - we just let it run, as data was still flowing even if we hit the settimeout error. Once we had fully backfilled, the issue effectively goes away because you aren't saturating the pipe out of the db anymore

@tj_murphy so, you only run again the elt? to complete the backfill.

Yeah I just set the meltano pipelines to re-run a bunch of times until the backfill was complete

I believe I had a similar issue at one point. If I remember right I fixed it by messing around with the config in my mysql tap:

session_sqls:
      - SET @@session.max_execution_time=0
      - SET @@session.net_read_timeout=3600
      - SET @@session.net_write_timeout=3600
      - SET @@session.wait_timeout=28800
      - SET @@session.innodb_lock_wait_timeout=3600

those are what I’m currently set to