Store bandwidth characteristic of Manager restore process

[mikliapko]

Since evaluating the restore process performance by bandwidth seems like the most reasonable approach - especially when comparing across different clusters and datasets - it makes sense to store and display this data in Manager.

It would be a great improvement for restore benchmarking tests.

[karol-kokoszka]

The suggestion is to include bandwidth of the restore into the sctool progress.
The current output is (example from other issue):

To achieve the goal of this task, we need to store an information about the duration of load&stream, download and idle. Right now, it's only indicated by the scylla_manager_restore_state metric.

The restore_progress that is saved to DB is of the following form

type RunProgress struct {
	ClusterID uuid.UUID
	TaskID    uuid.UUID
	RunID     uuid.UUID

	ManifestPath string
	Keyspace     string `db:"keyspace_name"`
	Table        string `db:"table_name"`
	Host         string // IP of the node to which SSTables are downloaded.
	AgentJobID   int64

	SSTableID           []string `db:"sstable_id"`
	DownloadStartedAt   *time.Time
	DownloadCompletedAt *time.Time
	RestoreStartedAt    *time.Time
	RestoreCompletedAt  *time.Time
	Error               string
	Downloaded          int64
	Skipped             int64
	Failed              int64
	VersionedProgress   int64
}

It looks that we already store the needed data in DB. It's per batch of the sstables (SSTableID column), and we store an information of when the download has started/ended (DownloadStartedAt/DownloadCompletedAt), when the l&s started/ended (RestoreStartedAt, RestoreCompletedAt) and what was the host owning this batch (Host). Besides, we save info of how much of the bytes was downloaded/restored in downloaded column.
It gives us a possibility of querying the "restore_run_progress" table when the restore completes to create a map for every host that includes how much time it took for a given host to download and load&stream. The idle state is the duration - (download + load&stream).
Then we have formula:

downloaded (per host) / duration of the download (DownloadedCompletedAt - DownloadedStartedAt)
downloaded (per host) / duration of the load&stream (RestoreCompletedAt - RestoreStartedAt)

We can show the idle time in the summary as well by outputing (per host)

Restore duration - (time reported as download) - (time reported as load & stream)

The output can be:

<host>:
   download: BW
   load&stream: BW
   idle: time

As the output may be quite big for x nodes cluster, we can include it into the restore progress with the --details flag.
https://manager.docs.scylladb.com/stable/sctool/progress.html#details

[karol-kokoszka]

Possibly it's worth to include it into 3.4 release.

[dorlaor]

Nice!

[mykaul]

Isn't it why God created Metrics and we implemented Monitoring?

[Michal-Leszczynski]

I will add it in both metrics and progress display.