doc: graph: add document for sdpa with compressed key and value #2301
Conversation
wzt1997
added
the
documentation
Thank you @mgouicem ! That's also my question. Initially when i added the floating-point section, i was thinking to include all these fp, int8 quantized, and only kv quantized patterns together in a single page. But with this PR, it seems there will be too much information. Maybe we need @ranukund 's input for which is a better format. |
wzt1997
force-pushed
the
zhitao/doc-compressed-sdpa
branch
from
51d1971 to
9c91ae9
Compare
Thanks for the question! From my perspective, I think it's better for us to put the quantized sdpa patterns in a separate document as it requires much more information compared with pure floating-point patterns, such as fpmath mode setting, group quantization and adding extra dynamic quantization ops. It's also worth noting that this PR only includes the document for quantization sdpa patterns with compressed KV, but not the pure quantized sdpa. We may need to think about fusing them together in the future. |
doc/graph/sdpa_with_compressed_kv.md
Outdated
| | dt_fp | dt_fp | N/A | N/A | dt_int | dt_fp | u4,s4,u8,s8,s32 | dt_fp | | ||
|
|
||
| Notes: | ||
| - dt_fp can be either: f16, bf16 and f32. |
There was a problem hiding this comment.
what's the problem if we directly expand dt_fp as f32, bf16, f16 in the table above?
There was a problem hiding this comment.
If we directly use f32, bf16, f16 in the table, it would result in an excessively lengthy table. Furthermore, it could lead users to misinterpret that these data types can be mixed, such as using f16 for Query with bf16 for Key scale. However, all sections involving dt_fp should utilize the same data type.
wzt1997
force-pushed
the
zhitao/doc-compressed-sdpa
branch
from
9c91ae9 to
2f307af
Compare
|
@ranukund, please help with the review. |
|
|
||
| ## Overview | ||
|
|
||
| The KV Cache mechanism was developed to improve the efficiency of models by |
|
|
||
| ## Overview | ||
|
|
||
| The KV Cache mechanism was developed to improve the efficiency of models by |
There was a problem hiding this comment.
| The KV Cache mechanism was developed to improve the efficiency of models by | |
| The KV Cache mechanism is developed to improve the efficiency of models by |
| usage, and are subsequently de-quantized to wider floating point types such as | ||
| f16 and bf16 for computation. | ||
|
|
||
| It's worth noting that grouped quantization is required to improve model |
There was a problem hiding this comment.
| It's worth noting that grouped quantization is required to improve model | |
| Note that grouped quantization is required to improve the model |
| f16 and bf16 for computation. | ||
|
|
||
| It's worth noting that grouped quantization is required to improve model | ||
| accurarcy, especially for int4 data types. In this case, group size is needed |
There was a problem hiding this comment.
| accurarcy, especially for int4 data types. In this case, group size is needed | |
| accuracy, especially for int4 data types. In this case, group size is needed |
| as an attribute for quantization, which indicates the number of elements that | ||
| share the same scaling factor and zero-points in each quantization group. | ||
|
|
||
| The notations used in the document: |
There was a problem hiding this comment.
| The notations used in the document: | |
| The notations used in this topic are: |
| intermediate results of the dot products between Query and Key which takes | ||
| \f$O(S^2)\f$ memory. It may lead to Out-of-Memory error when computing long | ||
| sequence length input on platforms with limited memory. | ||
| 2. The compressed SDPA patterns functionally support all input shapes meeting |
There was a problem hiding this comment.
| 2. The compressed SDPA patterns functionally support all input shapes meeting | |
| - The compressed SDPA patterns functionally support all input shapes meeting |
| sequence length input on platforms with limited memory. | ||
| 2. The compressed SDPA patterns functionally support all input shapes meeting | ||
| the shape requirements of each operation in the graph. | ||
| 3. CPU |
There was a problem hiding this comment.
| 3. CPU | |
| - CPU |
| - oneDNN does not provide optimized implementation on CPU currently. All | ||
| executions will be implemented with the primitive-based reference | ||
| computation. | ||
| 4. GPU |
There was a problem hiding this comment.
| 4. GPU | |
| - GPU |
| 4. GPU | ||
| - Optimized implementation is available for 4D Q/K/V tensors with the shape | ||
| defined as (N, H, S, D) for Query, Key and Value, and (N, H, S, D/G) for | ||
| scales and zero-points( if available ). |
There was a problem hiding this comment.
| scales and zero-points( if available ). | |
| scales and zero-points (if available). |
| computation data type on Intel Graphics Products with Intel(R) Xe Matrix | ||
| Extensions (Intel(R) XMX) support. | ||
| - If int4 zero-points are specified, optimized implementation will be only | ||
| avaibable when group size equals to 16. |
There was a problem hiding this comment.
| avaibable when group size equals to 16. | |
| available when the group size equals 16. |
General