Can we search/delete when compacting segments

[KendrickChou]

I've read Milvus architecture documents and understand that segment compaction is intended to combine small segments into larger ones and clean up logical deletions. This can be triggered manually by the user or automatically in the background.

My question is, can we search/delete when compacting segments? E.g., if segment A and B are compacting into C, can A and B still handle search/delete requests during compaction? If so, how does Miluvs ensure concurrency between these requests and the compaction process?

Thank you in advance for your answer :)

[xiaofan-luan]

Hi Kendrick,

All the delete is put into a new segment named L0 segments.
L0 segments is compact to multiple deltalogs to different sealed segments, and this L0 compaction is mutually exclusive with all the mix compaction so during L0 compaction all the mix compaction will be hold.

So under your case, while A and B compacting into C, all the L0 segment will wait until the compaction done. Mean time you can still do insertion(only on a new Segment, say D) and deletion (only on a new L0 segment,. say F)

[KendrickChou]

Thanks for your clear answer :)
I still have another question: could we search vectors in A and B while compacting them into C? If search is blocked, how long will it last?

[yhmo]

Compaction doesn't block search.

Let's say there are two segments: A and B.

1. A and B are loaded into memory of query nodes. Search requests are applied on A and B.
2. Data node begin to compact A and B to C.
3. Before the compaction is finished, all the search requests are applied on A and B.
4. Compaction done, C is ready(persisted and indexed) and loaded into memory of query nodes.
5. All the search requests are redirected to C
6. A and B are released from memory of query nodes.
7. data files of A and B are marked as "soft-delete" and wait GC process to clean from storage.

The step 5 is very fast, the client doesn't know the A and B is changed to C.

[KendrickChou]

I got it! Thanks very much