ci-entrypoint.sh log collection misses rotated logs

[nojnhuh]

/kind bug

What steps did you take and what happened:

In long-running, high-volume tests where components generate lots of logs and those logs get rotated, invoking log collection at the end of a ci-entrypoint.sh run will only collect the most recent set of logs, ignoring any that have been rotated.

Instead, we should start streaming logs immediately before running the user's real test, continue streaming in the background during the test, then stop streaming once the test is done.

There are some points that will make this non-trivial to implement:

• We need a way to orchestrate running log collection in the background while a user's test is running, then signal log collection to end when the test is over.
• The log collector currently also collects things like Azure activity logs and k8s resources which aren't streamed like pod logs and should only be captured at the end of a test.

What did you expect to happen:

All logs are captured.

Anything else you would like to add:

Environment:

• cluster-api-provider-azure version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[nojnhuh]

fyi @alaypatel07

[alaypatel07]

+1 this is absolutely critical to debug any CI failures from logs.