Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: [GoSDK] fp32 <-> fp16/bf16 vector conversion #37978

Merged
merged 1 commit into from
Nov 29, 2024
Merged

feat: [GoSDK] fp32 <-> fp16/bf16 vector conversion #37978

merged 1 commit into from
Nov 29, 2024

Conversation

jiangyinzuo
Copy link
Contributor

@jiangyinzuo jiangyinzuo commented Nov 25, 2024 •
edited
Loading

Add the following methods for convenient fp32 vector <-> fp16/bf16
vector conversion

fp32 <-> fp16/bf16 vector conversion:

  • func (fv FloatVector) ToFloat16Vector() Float16Vector
  • func (fv FloatVector) ToBFloat16Vector() BFloat16Vector
  • func (fv Float16Vector) ToFloat32Vector() FloatVector
  • func (fv BFloat16Vector) ToFloat32Vector() FloatVector

columnBasedDataOption:

  • func (opt *columnBasedDataOption) WithFloat16VectorColumn(colName string, dim int, data [][]float32) *columnBasedDataOption
  • func (opt *columnBasedDataOption) WithBFloat16VectorColumn(colName string, dim int, data [][]float32) *columnBasedDataOption

ColumnFloat16Vector/ColumnBFloat16Vector:

  • func NewColumnFloat16VectorFromFp32Vector(fieldName string, dim int, data [][]float32) *ColumnFloat16Vector
  • func NewColumnBFloat16VectorFromFp32Vector(fieldName string, dim int, data [][]float32) *ColumnBFloat16Vector
  • support []float32 or entity.FloatVector in
    • func (c *ColumnFloat16Vector) AppendValue(i interface{}) error
    • func (c *ColumnFloat16Vector) AppendValue(i interface{}) error

issue: #37448

@sre-ci-robot sre-ci-robot added area/dependency Pull requests that update a dependency file area/test sig/testing size/L Denotes a PR that changes 100-499 lines. labels Nov 25, 2024
@mergify mergify bot added dco-passed DCO check passed. kind/feature Issues related to feature request from users labels Nov 25, 2024
@jiangyinzuo jiangyinzuo force-pushed the feat/go-sdk-fp32-to-fp16-or-bf16 branch from 285580e to 63e028c Compare November 25, 2024 06:04
@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 25, 2024

@jiangyinzuo go-sdk check failed, comment rerun go-sdk can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 25, 2024

@jiangyinzuo E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@codecov Codecov
Copy link

codecov bot commented Nov 25, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 91.48936% with 4 lines in your changes missing coverage. Please review.

Project coverage is 81.06%. Comparing base (6aaa926) to head (e90721b).
Report is 5 commits behind head on master.

Files with missing lines Patch % Lines
client/column/vector.go 77.77% 4 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           master   #37978       +/-   ##
===========================================
+ Coverage   68.87%   81.06%   +12.19%     
===========================================
  Files         289     1360     +1071     
  Lines       25484   190879   +165395     
===========================================
+ Hits        17551   154734   +137183     
- Misses       7933    30648    +22715     
- Partials        0     5497     +5497
Components Coverage Δ
Client 72.60% <91.48%> (∅)
Core 68.87% <ø> (ø)
Go 83.23% <ø> (∅)
Files with missing lines Coverage Δ
client/entity/vectors.go 68.42% <100.00%> (ø)
client/milvusclient/write_options.go 64.73% <100.00%> (ø)
client/column/vector.go 81.98% <77.77%> (ø)

... and 1068 files with indirect coverage changes

@jiangyinzuo jiangyinzuo force-pushed the feat/go-sdk-fp32-to-fp16-or-bf16 branch from 63e028c to 2d1f984 Compare November 26, 2024 06:38
@jiangyinzuo jiangyinzuo changed the title feat: [GoSDK] fp32 to fp16/bf16 vector conversion feat: [GoSDK] fp32 <-> fp16/bf16 vector conversion Nov 26, 2024
@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 26, 2024

@jiangyinzuo go-sdk check failed, comment rerun go-sdk can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 26, 2024

@jiangyinzuo E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@jiangyinzuo jiangyinzuo force-pushed the feat/go-sdk-fp32-to-fp16-or-bf16 branch from 2d1f984 to 2d280cf Compare November 26, 2024 07:27
@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 26, 2024

@jiangyinzuo go-sdk check failed, comment rerun go-sdk can trigger the job again.

@jiangyinzuo jiangyinzuo force-pushed the feat/go-sdk-fp32-to-fp16-or-bf16 branch from 2d280cf to f213224 Compare November 26, 2024 07:51
@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 26, 2024

@jiangyinzuo go-sdk check failed, comment rerun go-sdk can trigger the job again.

@jiangyinzuo jiangyinzuo force-pushed the feat/go-sdk-fp32-to-fp16-or-bf16 branch from f213224 to 14fb31b Compare November 27, 2024 07:01
@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 27, 2024

@jiangyinzuo go-sdk check failed, comment rerun go-sdk can trigger the job again.

@jiangyinzuo jiangyinzuo force-pushed the feat/go-sdk-fp32-to-fp16-or-bf16 branch from 14fb31b to 703423a Compare November 27, 2024 07:23
@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 27, 2024

@jiangyinzuo go-sdk check failed, comment rerun go-sdk can trigger the job again.

@jiangyinzuo jiangyinzuo force-pushed the feat/go-sdk-fp32-to-fp16-or-bf16 branch from 703423a to 9a3e2f5 Compare November 27, 2024 07:46
@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 27, 2024

@jiangyinzuo go-sdk check failed, comment rerun go-sdk can trigger the job again.

@jiangyinzuo jiangyinzuo force-pushed the feat/go-sdk-fp32-to-fp16-or-bf16 branch from 9a3e2f5 to 13d883c Compare November 27, 2024 08:23
@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 27, 2024

@jiangyinzuo go-sdk check failed, comment rerun go-sdk can trigger the job again.

@jiangyinzuo jiangyinzuo force-pushed the feat/go-sdk-fp32-to-fp16-or-bf16 branch 2 times, most recently from 6fcba1a to bfe98b9 Compare November 27, 2024 11:11
@jiangyinzuo jiangyinzuo force-pushed the feat/go-sdk-fp32-to-fp16-or-bf16 branch 2 times, most recently from bf67626 to 4ac02d0 Compare November 27, 2024 13:24
@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 27, 2024

@jiangyinzuo E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@jiangyinzuo
Copy link
Contributor Author

/run-cpu-e2e

@jiangyinzuo jiangyinzuo force-pushed the feat/go-sdk-fp32-to-fp16-or-bf16 branch from 4ac02d0 to 1ddb04d Compare November 27, 2024 15:22
@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 27, 2024

@jiangyinzuo go-sdk check failed, comment rerun go-sdk can trigger the job again.

@jiangyinzuo
Copy link
Contributor Author

rerun go-sdk

@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 27, 2024

@jiangyinzuo E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@jiangyinzuo
Copy link
Contributor Author

/run-cpu-e2e

@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 27, 2024

@jiangyinzuo E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@jiangyinzuo jiangyinzuo force-pushed the feat/go-sdk-fp32-to-fp16-or-bf16 branch from 1ddb04d to e90721b Compare November 28, 2024 02:53
@jiangyinzuo
feat: [GoSDK] fp32 <-> fp16/bf16 vector conversion
e90721b 
Add the following methods for convenient fp32 vector <-> fp16/bf16
vector conversion

fp32 <-> fp16/bf16 vector conversion:

- `func (fv FloatVector) ToFloat16Vector() Float16Vector`
- `func (fv FloatVector) ToBFloat16Vector() BFloat16Vector`
- `func (fv Float16Vector) ToFloat32Vector() FloatVector`
- `func (fv BFloat16Vector) ToFloat32Vector() FloatVector`

`columnBasedDataOption`:

- `func (opt *columnBasedDataOption) WithFloat16VectorColumn(colName string, dim int, data [][]float32) *columnBasedDataOption`
- `func (opt *columnBasedDataOption) WithBFloat16VectorColumn(colName string, dim int, data [][]float32) *columnBasedDataOption`

`ColumnFloat16Vector`/`ColumnBFloat16Vector`:

- `func NewColumnFloat16VectorFromFp32Vector(fieldName string, dim int, data [][]float32) *ColumnFloat16Vector`
- `func NewColumnBFloat16VectorFromFp32Vector(fieldName string, dim int, data [][]float32) *ColumnBFloat16Vector`
- support []float32 or `entity.FloatVector` in
    - `func (c *ColumnFloat16Vector) AppendValue(i interface{}) error`
    - `func (c *ColumnFloat16Vector) AppendValue(i interface{}) error`

issue: milvus-io#37448

Signed-off-by: Yinzuo Jiang <[email protected]>
Signed-off-by: Yinzuo Jiang <[email protected]>
@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 28, 2024

@jiangyinzuo E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@jiangyinzuo
Copy link
Contributor Author

/run-cpu-e2e

@mergify Mergify
Copy link
Contributor

mergify bot commented Nov 28, 2024

@jiangyinzuo E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@jiangyinzuo
Copy link
Contributor Author

/run-cpu-e2e

@mergify mergify bot added the ci-passed label Nov 28, 2024
congqixia
congqixia approved these changes Nov 29, 2024
View reviewed changes
Copy link
Contributor

@congqixia congqixia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: congqixia, jiangyinzuo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sre-ci-robot sre-ci-robot merged commit 1ce6098 into milvus-io:master Nov 29, 2024
20 checks passed
congqixia added a commit to congqixia/milvus that referenced this pull request Nov 29, 2024
@congqixia
enhance: [GoSDK] Refine float point conversion precision loss ut
1fb51a2 
Previous PR: milvus-io#37978

This unit test is unable due to dim is a random number. When dim is
large enough precision loss will be greater than 0.04

Signed-off-by: Congqi Xia <[email protected]>
@congqixia congqixia mentioned this pull request Nov 29, 2024
Merged
sre-ci-robot pushed a commit that referenced this pull request Nov 29, 2024
@congqixia
enhance: [GoSDK] Refine float point conversion precision loss ut (#38097
fdea7a7 
)

Previous PR: #37978

This unit test is unable due to dim is a random number. When dim is
large enough precision loss will be greater than 0.04

Signed-off-by: Congqi Xia <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@congqixia congqixia congqixia approved these changes

@wangting0128 wangting0128 Awaiting requested review from wangting0128

Assignees

@congqixia congqixia

Labels
approved area/dependency Pull requests that update a dependency file area/test ci-passed dco-passed DCO check passed. kind/feature Issues related to feature request from users lgtm sig/testing size/L Denotes a PR that changes 100-499 lines.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jiangyinzuo @sre-ci-robot @congqixia