Discovery timeouts, emergency read operations throttled by discovery timeouts

[shlomi-noach]

Followup to an issue reported outside this repo.

Scenario:

• Instance becomes unavailable, and causing connections to hang
• ReadTopologyInstanceBufferable (the main discovery function) runs again and again, totaling 3 concurrent executions, exhausting the connection pool limit for this instance.
• any further queries, such as api/discover/the-instance/3306 are blocked on this instance
• In a orchestrator/raft setup, someone calls api/discover/the-instance/3306
• Now the raft protocol is blocked
• All further rat communications are blocked
• No failovers possible

Description

Timeouts already configured at DSN level. We add:

• Explicit context.WithTimeout() in ReadTopologyInstanceBufferable, now using db.QueryRowContext() and sqlutils.QueryRowsMapContext()
• in topology_recovery.go, emergentlyReadTopologyInstance() only runs at most 1 read (for a given instance) per MySQLDiscoveryReadTimeoutSeconds, to avoid further spamming the instance.

Expected result: worst case scenario an operation will hang up to MySQLDiscoveryReadTimeoutSeconds (configurable, default: 10)

[MOON-CLJ]

@shlomi-noach should publishDiscoverMasters also use async-discover?
because sync discover can slow down the raft log apply。
cc @yangeagle