Will module output not be quantized when the model is directly trained after Calibration?

[tusiqi1]

No description provided.

[tusiqi1]

1. model structure and train data is:
   
2. clibration code is:
   
3. train quant model with initial input_scale and output_scale code is:
   

I found that when I executed step 3, the output quant would not execute, and the reason is when model first passes

with Clibration():
    model(input)

the hook for callibrate_output for the entire model executes the code in the red box:

Why should the code in the red box disable output quant?
Or my use of calibration + QAT use errors.

[dacorvo]

As you can see in the comment on the line you highlighted, the quantization of the outputs is disabled because the operation immediately following is not compatible with quantized inputs. This means that when the Tensor reaches that operation, it will be immediately dequantized: the streamline optimization policy removes the spurious quantize/dequantize.

If you want to disable this behaviour, just pass streamline=False during calibration.

[tusiqi1]

As you can see in the comment on the line you highlighted, the quantization of the outputs is disabled because the operation immediately following is not compatible with quantized inputs. This means that when the Tensor reaches that operation, it will be immediately dequantized: the streamline optimization policy removes the spurious quantize/dequantize.

If you want to disable this behaviour, just pass streamline=False during calibration.

When I press the break point in the image above, the entire model is passed to the calibrate_output() function, which causes the quantization of the output to be turned off.

Is this right?

[dacorvo]

Just use with Calibration(streamline=False): to disable this behaviour.

[tusiqi1]

Just use with Calibration(streamline=False): to disable this behaviour.

Thank you very much, I'll think more about the purpose of streamline myself.