Optimizer Passes and Flows
hls4ml library will parse models from Keras, PyTorch or ONNX into an internal execution graph. This model graph is represented with the
ModelGraph class. The nodes in this graph, corresponding to the layer and operations of the input model are represented
by classes derived from the
Layer base class.
Layers are required to have defined inputs and outputs that define how they are connected in the graph and what is the shape of their output. All information about the layer’s state and configuration is stored in its attributes. All weights, variables and data types are attributes and there are mapping views to sort through them. Layers can define expected attributes and can be verified for correctness, or to produce a list of configurable attributes that user can tweak.
To reach a state from which the code can be generated, the internal model graph undergoes a series of optimizations (transformations), dubbed
optimization passes. All transformations of the model and any modification to any layer’s attributes must be implemented through an optimization
pass. All optimizer passes derive from the
OptimizerPass class. Optimizer passes are usually applied to
nodes/layers; however, a special class
ModelOptimizerPass exists that is applied on the full model. An
example of a layer optimizer is
fuse_biasadd, which adds a bias to a
Conv2D layer, while an example of
an optimizer pass that runs on the full model is
MakeStamp, which creates a unique number (stamp).
OptimizerPass must provide a criteria in
match function that, if satisfied, will
perform the transformation from
transform function. The boolean return value of
transform indicates if the optimizer pass made changes to the
model graph that may require running the optimizers again. In that case, optimizers in a flow are run again.
Optimizers can be general, independent of the backend, in which case they are located in
hls4ml.model.optimizer.passes, or they may be backend-specific,
in which case they are located in a folder dependent on the backend, e.g.,
hls4ml.backends.quartus.passes. A common set of optimizers that are used by FPGA backends are located in
Certain optimizers are used frequently enough that it makes sense to define special classes, which inherit from
GlobalOptimizerPass: An optimizer pass that matches each node. This is useful, for example, to transform the types for a particular backend.
LayerOptimizerPass: An optimizer pass that matches each node of a particular layer type. This is useful, for example, to write out the HLS code for a particular node that remains in the final graph.
ConfigurableOptimizerPass: An optimizer pass that has some configurable parameters.
Template: An optimizer pass that populates a code template and assigns it to an attribute of a given layer. This is commonly used to generate code blocks in later stages of the conversion.
New optimizers can be registered with the
register_pass(). Optimizers should be assigned to a flow (see below).
Flow is an ordered set of optimizers that represents a single stage in the conversion process. The optimizers
from a flow are applied in sequence until they no longer make changes to the model graph (controlled by the
transform return value), after which
the next flow (stage) can start. Flows may require that other flows are applied before them, ensuring the model graph is in a desired state before a
flow starts. The function
register_flow() is used to register a new flow. Flows are applied on a model graph with
There are common model-level flows that can run regardless of the backend, and there are backend-specific flows. The convert and optimize flows do not depend on a backend.
Each backend provides provides a default flow that defines the default target for that backend. For example, the Vivado backend defaults to an IP flow that requires additional flows and produces an IP. It runs no optimizers itself, but it requires that many other flows (sub-flows) to have run. The convert and optimize flows defined above are some of these required sub-flows.
Another example is FIFO buffer depth optimization explained in the FIFO Buffer Depth Optimization section.