Best practices for batch exporting large volume of Trace data (daily dump) from BanyanDB

[Silvengs]

Hi SkyWalking Team and Community,

I am currently using SkyWalking OAP with BanyanDB as the storage solution. I have a requirement to export all Trace data for a specific Service ID on a daily basis for offline analysis. The data volume is approximately 10 million traces per day.

Is batch exporting raw data considered a recommended use case for BanyanDB?What is the best practice for this scenario?

Environment:
SkyWalking OAP Version: [10.3.0]
BanyanDB Version: [0.9.0]
Client: [BanyanDB gRPC Client Proto]

Thanks in advance for your guidance!

[wu-sheng]

From database perspective, exporting is not supported, especially in cluster mode.
OAP provides exporting but they are trace segments oriented, instead of trace per service.
Why do you need this? Why can't query via our API directly?

[Silvengs]

I will take your advice into consideration and try to adjust my architecture. Appreciate your time and guidance!

[wu-sheng]

BanyanDB optimized its kernel very specifically for trace query and storage, and a lot of ongoing for this in coming versions.
No matter where you plan to export these traces, I am highly doubt it is more efficient there.