8.1.1.8. blueoil.quantizations

This Package provides a set of quantizations.

How to use:

quantizer = binary_mean_scaling_quantizer() # initialize quantizer
weights = tf.get_variable("kernel", shape=[1, 2, 3, 4]) # prepare variable.
quantized_weights = quantizer(weights) # use quantizer to quantize variable
tf.nn.conv2d(inputs, quantized_weights)
blueoil.quantizations.linear_mid_tread_half_quantizer()
blueoil.quantizations.binary_mean_scaling_quantizer()
blueoil.quantizations.binary_channel_wise_mean_scaling_quantizer()
blueoil.quantizations.ttq_weight_quantizer()
blueoil.quantizations.twn_weight_quantizer()

8.1.1.8.2. Package Contents

8.1.1.8.2.1. Functions

binary_channel_wise_mean_scaling_quantizer(backward=None, dtype=tf.float32)

Binary channel wise mean scaling quantizer.

binary_mean_scaling_quantizer(backward=None, dtype=tf.float32)

Binary mean scaling quantizer.

linear_mid_tread_half_quantizer(bit=None, max_value=None, backward=None, dtype=tf.float32)

Linear mid tread half quantizer.

ttq_weight_quantizer(threshold=0.05, dtype=tf.float32)

Trained Ternary Quantization (TTQ)

twn_weight_quantizer(threshold=0.7, dtype=tf.float32)

Ternary Weight Networks (TWN)

blueoil.quantizations.binary_channel_wise_mean_scaling_quantizer(backward=None, dtype=tf.float32)

Binary channel wise mean scaling quantizer.

This quantization creates a binary channel wise mean scaling quantizer. If backward is provided, this backward will be used in backpropagation.

This method is varient of XNOR-Net [1]_ weight quantization, the difference from XNOR-Net [1]_ is backward function.

op_type is BinaryChannelWiseMeanScalingQuantizer.

Forward is:

\[\begin{split}\begin{align} \bar{\mathbf{x}} & = \frac{1}{n}||\mathbf{X}||_{\ell1} & \text{$\bar{\mathbf{x}}$ is a $c$-channels vector} \\ & & \text{$n$ is number of elements in each channel of $\mathbf{X}$} \\\\ \mathbf{Y} & = \text{sign}\big(\mathbf{X}\big) \times \bar{\mathbf{x}} &\\ \end{align}\end{split}\]

Default backward is:

\[\frac{\partial Loss}{\partial \mathbf{X}} = \frac{\partial Loss}{\partial \mathbf{Y}}\]
Parameters
  • backward (callable) – Be used in backpropagation.

  • dtype (tf.DType) – Define the data type of args of forward and backward.

Returns

forward function (grad_func defined).

Return type

callable

Reference:
1

XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks

blueoil.quantizations.binary_mean_scaling_quantizer(backward=None, dtype=tf.float32)

Binary mean scaling quantizer.

This quantization creates a binary mean scaling quantizer. If backward is provided, this backward will be used in backpropagation.

This method is DoReFa-Net 2 weight quantization.

op_type is BinaryMeanScalingQuantizer.

Forward is:

\[\begin{split}\begin{align} \bar{x} & = \frac{1}{N}||\mathbf{X}||_{\ell1} & \text{$\bar{x}$ is a scalar} \\ & & \text{$N$ is number of elements in all channels of $\mathbf{X}$}\\ \mathbf{Y} & = \text{sign}\big(\mathbf{X}\big) \cdot \bar{x} &\\ \end{align}\end{split}\]

Default backward is:

\[\frac{\partial Loss}{\partial \mathbf{X}} = \frac{\partial Loss}{\partial \mathbf{Y}}\]
Parameters
  • backward (callable) – Be used in backpropagation.

  • dtype (tf.DType) – Define the data type of args of forward and backward.

Returns

forward function (grad_func defined).

Return type

callable

Reference:
2

DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients

blueoil.quantizations.linear_mid_tread_half_quantizer(bit=None, max_value=None, backward=None, dtype=tf.float32)

Linear mid tread half quantizer.

This quantization creates a linear mid tread half quantizer. If backward is provided, this backward will be used in backpropagation.

This quantization method is DoReFa-Net [1]_ activation quantization variant, the difference from DoReFa-Net [1]_ is to be able to change max_value.

op_type is LinearMidTreadHalfQuantizer.

Forward is:

\[\begin{split}\mathbf{X} & = \text{clip}\big(\mathbf{X}, 0, max\_value\big)\\ \mathbf{Y} & = \begin{cases} \mathbf{X}, & \text{if $bit$ is 32} \\ \frac{\text{round}\big(\frac{\mathbf{X}}{max\_value} \cdot (2^{bit}-1)\big)}{2^{bit}-1} \cdot max\_value, & otherwise \end{cases}\end{split}\]

Default backward is:

\[\begin{split}\frac{\partial Loss}{\partial \mathbf{X}} = \begin{cases} \frac{\partial Loss}{\partial y}, & \text{if $0 < x < max\_value$}\\ 0, & otherwise \end{cases}\end{split}\]
Parameters
  • bit (int) – Specify the bit of quantization.

  • max_value (int) – Be used for shift and clip.

  • backward (callable) – Be used in backpropagation.

  • dtype (tf.DType) – Define the data type of args of forward and backward.

Returns

forward function (grad_func defined).

Return type

callable

Reference:
1

DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients

blueoil.quantizations.ttq_weight_quantizer(threshold=0.05, dtype=tf.float32)

Trained Ternary Quantization (TTQ)

Ref: https://arxiv.org/pdf/1612.01064v1.pdf

blueoil.quantizations.twn_weight_quantizer(threshold=0.7, dtype=tf.float32)

Ternary Weight Networks (TWN)

Ref: https://arxiv.org/abs/1605.04711