Quantization guide
When choosing a learning rate to use within CCO it is good practice to set it to the last learning rate the optimizer was had at the end of the original (fp32/fp16) model training.
Selective and semi-automatic quantization
By default, only the model tail (layers from the output node upstream to the last weighted layer(s) in the model) will be skipped for quantization automatically.
If you wish to customize which layers will be skipped you can use the following methods:
- To avoid the automatic skipping of the model tail you can set
global_quantization_settings.automatic_skip_quantization
=False
in yourSetting
object. - To skip quantization in the interval between two layers you can use
set_quantization_settings_for_layer
(see example #1 below). - To skip quantization downstream you can skip quantization from a layer in the graph all the way to the last layer(s) by setting
LayerQuantizationSettings.skip_quantization_downstream
=True
(see example #2 below).
For this kind of customization, it is recommended to examine the generated graph visualization in model_init.svg
(the original model graph)
and model_post_preprocessing.svg
(the quantized model graph).
To make sure those file will be generated follow the relevant instructions here. You may also refer to the "parsing layers" section in the last section of CCO setup in the Output Log Breakdown page.
Example #1
If we want to skip all layers between the layer named adaptive_avg_pool_1
and the layer named linear_5
:
from clika_compression import LayerQuantizationSettings, Settings
settings = Settings() # Default settings
settings.set_quantization_settings_for_layer(
"adaptive_avg_pool_1",
LayerQuantizationSettings(skip_quantization=True, skip_quantization_until=["linear_5"])
)
You can also specify more than one destination in `skip_quantization_until.
Example #2
If we want to skip all layers from the layer named adaptive_avg_pool_1
to the last layer:
from clika_compression import LayerQuantizationSettings, Settings
settings = Settings() # Default settings
settings.set_quantization_settings_for_layer(
"adaptive_avg_pool_1",
LayerQuantizationSettings(skip_quantization=True, skip_quantization_downstream=True)
)
Graph visualization
If you follow the instructions to install the optional requirements, the files:
model_init.{svg, dot}
- Architecture of the original model before compressionmodel_post_preprocessing.{svg, dot}
- Architecture of the quantized model after compression
will be generated in your outputs
folder.
The color-coding for the layers is as follows:
- Blue - is an input or output node
- Green - is a quantized layer
- Yellow - is non-quantized layer
Graph Visualization includes the following information about each layer:
- Name and type
- Input and output shapes
- Layer attributes of the layer such as kernel size, strides etc.
- Quantization Sensitivity (QS)
Quantization sensitivity (QS) is a measurement of the difference between original and quantized outputs of each layer. It is used to determine which layers should be skipped. The QS is computed as the L2-norm between the quantized and float layer outputs; the higher the number, the harder it is to quantize.
It is recommended to skip quantization layers with a QS value above 10,000, since the higher the QS value, the longer it will take to for CCO to compress the model.