FAQ

Why is tracing the model necessary?

By nature, PyTorch models are represented as "dynamic graphs." However, for frameworks to execute the model, they must know how the model is constructed. A key benefit of static graphs is that they allow for more optimizations once the model's structure is clear.

note

A future release of clika-ace will support partial compilation, even if full-graph tracing is not possible. This means that models with control flow operations will be supported without modification.

My model fails to parse

If the model is parsed successfully but ACE reports unsupported operators, contact us and we will gladly implement the missing operator.

If torch.dynamo fails to parse the model, a model modification may be required. In most cases, the fix is straightforward, as torch.dynamo works well for most models.

Why is the tracing_inputs argument necessary for clika_compile?

By default, if tracing_inputs is not provided to clika_compile, the calibration_inputs will be used for tracing (see Compile API).

However, providing explicit tracing_inputs is often necessary to ensure the compiled model supports dynamic shapes (e.g., variable batch sizes, image resolutions, or sequence lengths).

The underlying torch.dynamo tracer tends to specialize on the input shapes it sees. If you only provide a single input (or inputs with identical shapes), Dynamo may generate a graph that is hard-coded (specialized) to those specific dimensions. This causes the model to fail or re-compile if it encounters a different shape during inference.

To prevent this specialization and enable dynamic behavior, you must provide multiple inputs that vary along the dimensions you want to be dynamic. This forces Dynamo to treat those dimensions as symbolic (variable) rather than static constants.

How to enable Dynamic Shapes?

For more information about torch.dynamo, see:

• Dynamo Overview
• Dynamo Deepdive

As a general rule, you must provide at least two examples that differ along the dimension(s) you wish to make dynamic. The tracer compares all provided inputs to determine the full set of dynamic axes.

Examples:

• Dynamic Batch Size: Provide inputs with different batch sizes.
  • tracing_inputs shapes: [(1, 3, 224, 224), (2, 3, 224, 224)]
• Dynamic Spatial Dimensions: Provide inputs with different heights/widths.
  • tracing_inputs shapes: [(1, 3, 224, 224), (1, 3, 320, 640)]
• Dynamic Batch & Spatial: Provide inputs covering the range of variability.
  • tracing_inputs shapes: [(1, 3, 224, 224), (2, 3, 320, 640)]
• LLM Sequence Length: Provide inputs with different sequence lengths to handle both "prefill" (long) and "decode" (short) steps:
  • tracing_inputs shapes (conceptual): [{"input_ids": tensor(1, 1)}, {"input_ids": tensor(1, 256)}]

How it works (conceptual):
The tracer finds differences between all inputs to mark axes as dynamic.

• Compares (1, 3, 224, 224) and (2, 3, 224, 224) -> Marks dim 0 as dynamic.
• Compares (1, 3, 224, 224) and (1, 3, 320, 224) -> Marks dim 2 as dynamic.
• ...and so on. This is why Example 3 works for three dynamic axes.

If the calibration_inputs already cover this variability (e.g., they contain images of different sizes or batches of different lengths), you do not need to provide separate tracing_inputs.

The compiling phase takes a long time

The first potential fix is to update PyTorch (see requirements). The clika-ace compiling phase depends on torch.dynamo, and the improvements made to torch.dynamo with each new PyTorch release are usually significant.

If that does not help, your model may have very complicated shape inferencing. An example of this was the maxvit model from the torchvision package. We're happy to assist in any case.