onnx2tf

A tool for converting ONNX files to LiteRT/TFLite/TensorFlow, PyTorch native code (nn.Module), TorchScript (.pt), state_dict (.pt), Exported Program (.pt2), and Dynamo ONNX. It also supports direct conversion from LiteRT to PyTorch.

You should use LiteRT Torch rather than onnx2tf. https://github.com/google-ai-edge/litert-torch and https://github.com/google-ai-edge/ai-edge-quantizer

tf_converter supported layers

• https://github.com/onnx/onnx/blob/main/docs/Operators.md
• :heavy_check_mark:: Supported　:white_check_mark:: Partial support　Help wanted: Pull Request are welcome

See the list of supported layers

flatbuffer_direct execution path

flatbuffer_direct is now the default backend. It is faster and has a higher success rate than tf_converter for the supported direct path. The simplest conversion command now outputs only a LiteRT model by default, but if you add --flatbuffer_direct_output_saved_model, it will also output a saved_model. Unlike the legacy tf_converter path, this SavedModel is built from the LiteRT-side ModelIR.

[!IMPORTANT] flatbuffer_direct is the current default backend. Use --tflite_backend tf_converter only when you explicitly need the legacy TensorFlow Lite Converter compatibility path.

With the default flatbuffer_direct backend, onnx2tf uses a direct fast path for both ONNX input and -it/--input_tflite_file_path input:

1. ONNX graph preprocessing (tflite_builder.preprocess) and direct lowering (lower_onnx_to_ir)
2. Direct FlatBuffer export (*_float32.tflite, *_float16.tflite, and optional quantized variants)
3. Optional direct reports/evaluation (*_op_coverage_report.json, tensor correspondence, ONNX/TFLite check)

In this fast path, the per-node TensorFlow conversion (op.make_node() over all ONNX nodes) is skipped. This removes the long debug traces such as:

• INFO: /
• INFO: onnx_op_type: ...
• INFO: tf_op_type: ...

Measured example (same model, float32 TFLite write stage):

• tf_converter: ~24.947s
• flatbuffer_direct: ~0.239s
• flatbuffer_direct was approximately 107x faster than tf_converter in this case.

Actual speedup depends on model structure, enabled options, and runtime environment.

Direct export can also generate TF-side artifacts without falling back to tf_converter:

• --output_h5
• --output_keras_v3
• --output_tfv1_pb
• --flatbuffer_direct_output_pytorch

These artifacts are generated from an internal SavedModel bridge built from float32 ModelIR. If direct export fails, conversion stops with an explicit error.

-inimc / -onimc also stay on the direct path in flatbuffer_direct. For ONNX input and -it input, these options crop the imported/lowered ModelIR at the specified boundary tensor names instead of splitting the ONNX graph.

-dgc, -ebu, and -eru also stay on the direct path in flatbuffer_direct. For ONNX input they are applied during lowering or as post-lowering ModelIR rewrites. For -it input they are applied to imported ModelIR before SavedModel bridge, split planning, or rewritten TFLite export. If the requested rewrite cannot be applied safely, conversion stops with an explicit error.

-me also stays on the direct path in flatbuffer_direct. For ONNX MeanVarianceNormalization, direct lowering uses primitive builtin ops and applies mvn_epsilon to the internal variance + epsilon term without falling back to tf_converter.

--disable_model_save also stays on the direct path. In flatbuffer_direct, it means the conversion can still run internal validation and temporary staging, but no final artifacts are left in the requested output directory.

Invalid combinations are rejected explicitly:

• --disable_model_save with --output_h5, --output_keras_v3, or --output_tfv1_pb
• --enable_auto_split_model with --output_h5, --output_keras_v3, or --output_tfv1_pb

SavedModel direct export from flatbuffer_direct ModelIR is available with --flatbuffer_direct_output_saved_model. PyTorch package direct export is available with --flatbuffer_direct_output_pytorch. These options have the following constraints:

• --tflite_backend flatbuffer_direct is required for both
• --flatbuffer_direct_output_saved_model cannot be combined with --disable_model_save
• CUSTOM ops are rejected with an explicit error

INT8 ONNX	INT8 TFLite(LiteRT)

• e.g. LiteRT only output

  onnx2tf \
  -i iat_llie_180x320.onnx \
  -tb flatbuffer_direct
  

• e.g. Generate additional saved_model from LiteRT after LiteRT output

  onnx2tf \
  -i iat_llie_180x320.onnx \
  -tb flatbuffer_direct \
  -fdosm
  

• e.g. Generate saved_model directly from an existing LiteRT (.tflite) file

  onnx2tf \
  -it iat_llie_180x320_float32.tflite \
  -tb flatbuffer_direct
  

• e.g. Generate .h5 directly from an existing LiteRT (.tflite) file without tf_converter fallback

  onnx2tf \
  -it iat_llie_180x320_float32.tflite \
  -tb flatbuffer_direct \
  -oh5
  

  

• e.g. Generate a PyTorch package directly from an existing LiteRT (.tflite) file

  onnx2tf \
  -it iat_llie_180x320_float32.tflite \
  -o tmp_iat_llie_180x320_from_tflite \
  -tb flatbuffer_direct \
  -fdopt
  

• e.g. Compare the input LiteRT model and the generated PyTorch package with the same seeded inputs

  onnx2tf \
  -it iat_llie_180x320_float32.tflite \
  -o tmp_iat_llie_180x320_from_tflite \
  -tb flatbuffer_direct \
  -fdopt \
  -cotof
  

  This outputs:

  • iat_llie_180x320_float32_pytorch/
  • iat_llie_180x320_float32_pytorch_accuracy_report.json (TFLite↔PyTorch)
  • iat_llie_180x320_float32_accuracy_comparison_report.json

[Ultra experimental] PyTorch export example (yolox_s.onnx)

flatbuffer_direct can emit a native PyTorch package together with optional TorchScript, Dynamo ONNX, and ExportedProgram artifacts in one run.

Generate all PyTorch-side artifacts plus TFLite and accuracy reports:

onnx2tf \
-i yolox_s.onnx \
-o tmp_yolox_s \
-tb flatbuffer_direct \
-cotof \
-fdopt \
-fdots \
-fdodo \
-fdoep

The output directory contains:

• yolox_s_float32.tflite
• yolox_s_float16.tflite
• yolox_s_accuracy_report.json (ONNX↔TFLite)
• yolox_s_pytorch_accuracy_report.json (ONNX↔PyTorch)
• yolox_s_accuracy_comparison_report.json
• yolox_s_pytorch/
  • model.py
  • runtime.py
  • state_dict.pth
  • metadata.json
  • yolox_s_pytorch/yolox_s_jit.pt
  • yolox_s_pytorch/yolox_s_dynamo.onnx
  • yolox_s_pytorch/yolox_s_ep.pt2

The generated PyTorch package is a normal torch.nn.Module package. You can load it and run eager inference directly:

import sys

import torch

sys.path.append("tmp_yolox_s")
from yolox_s_pytorch import load_model

model = load_model(device="cpu", eval_mode=True)
x = torch.zeros((1, 3, 640, 640), dtype=torch.float32)

with torch.no_grad():
    output = model(x)

print("input :", tuple(x.shape))
print("output:", tuple(output.shape))
print(model.forward_named(x).keys())

Expected output for the current yolox_s package:

input : (1, 3, 640, 640)
output: (1, 8400, 85)
dict_keys(['output'])

You can also load the bundled state_dict.pth explicitly with standard PyTorch APIs:

import sys
from pathlib import Path

import torch

sys.path.append("tmp_yolox_s")
from yolox_s_pytorch.model import Model

package_dir = Path("tmp_yolox_s/yolox_s_pytorch")
model = Model(load_weights=False, eval_mode=True)
state_dict = torch.load(package_dir / "state_dict.pth", map_location="cpu")
model.load_state_dict(state_dict, strict=True)

The generated state_dict.pth is saved in load_state_dict-compatible format for native PyTorch packages.

For native packages, raw torch.onnx.export(..., dynamo=True) and raw torch.export.save(torch.export.export(...)) are intended to produce the same graph structure as the helper-generated *_dynamo.onnx and *_ep.pt2.

Example: raw torch.onnx.export

from pathlib import Path
import importlib
import logging
import sys
import torch

package_dir = Path("tmp_yolox_s/yolox_s_pytorch").resolve()
sys.path.insert(0, str(package_dir.parent))
pkg = importlib.import_module(package_dir.name)

model = pkg.load_model(device="cpu", eval_mode=True)
model.eval()

example_inputs = (torch.randn(1, 3, 640, 640),)
logging.getLogger("torch.onnx._internal.exporter._registration").setLevel(logging.ERROR)

with torch.no_grad():
    torch.onnx.export(
        model,
        example_inputs,
        str(package_dir / "raw_dynamo.onnx"),
        dynamo=True,
        input_names=model.input_names,
        output_names=model.output_names,
    )

Example: raw torch.export.save

from pathlib import Path
import importlib
import sys
import torch

package_dir = Path("tmp_yolox_s/yolox_s_pytorch").resolve()
sys.path.insert(0, str(package_dir.parent))
pkg = importlib.import_module(package_dir.name)

model = pkg.load_model(device="cpu", eval_mode=True)
model.eval()

example_inputs = (torch.randn(1, 3, 640, 640),)

with torch.no_grad():
    exported_program = torch.export.export(model, example_inputs)

torch.export.save(
    exported_program,
    str(package_dir / "raw_exported_program.pt2"),
)

Both examples require a concrete example input shape. For dynamic public inputs, use the same concrete shape/data that you would provide to -fdodo or -fdoep via --shape_hints, --test_data_nhwc_path, or -cind.

[!CAUTION] Native PyTorch packages generated by --flatbuffer_direct_output_pytorch are intended primarily for inference, not for training as-is.

Detailed reasons:

• The exporter is designed to preserve inference behavior of the converted TFLite/ModelIR graph, not to reconstruct the original training-time PyTorch model semantics.
• The generated graph may include inference-oriented rewrites such as layout normalization, constant folding, reshapes/transposes inserted for runtime compatibility, and decomposition into primitive ops. These are correct for forward inference but are not guaranteed to be ideal or even stable for gradient-based optimization.
• Some generated models include postprocessing or task-specific inference logic such as argmax, non_max_suppression, score filtering, indexing-heavy tensor selection, or shape-control branches. These are typically non-differentiable or poor training targets.
• Native export may emit helper paths that are semantically equivalent for inference but were not designed with training ergonomics in mind, such as strict shape/layout alignment, bridge shims for 1D/2D/3D convolution compatibility, or graph-local runtime helpers.
• Fallback-backed packages (tflite, saved_model, string_normalizer) are wrappers around non-PyTorch execution backends and therefore should be treated as inference-only.
• Even native packages are emitted with eval_mode=True as the normal usage path. The exporter does not currently guarantee training-safe reconstruction of modules such as normalization layers, recurrent state handling, or control-flow-heavy blocks in the same form expected by optimizers and schedulers.
• state_dict.pth is load_state_dict-compatible for native packages, but compatibility of weight loading does not imply that the generated module is a faithful training architecture suitable for fine-tuning.

Practical guidance:

• Use generated PyTorch packages for inference validation, packaging, and side-by-side output comparison.
• If you want to train or fine-tune a model, treat the generated package as a reference implementation only, and rebuild or simplify the architecture into a training-oriented PyTorch model before optimization.

Click to expand

• Scope: ONNX ops listed in the ``tf_converter supported layers table above.
• Source of truth: onnx2tf/tflite_builder/op_registry.py and --report_op_coverage output.
• Current summary:
  • Listed ONNX ops in the builtin table below: 192
  • Policy counts are generated in *_op_coverage_report.json (schema_policy_counts).
  • Check each conversion run with --report_op_coverage for the latest numbers.

Notes:

• flatbuffer_direct supports only a subset of ONNX ops as TFLite builtins.
• Some ops are conditionally supported (rank/attribute/constant-input constraints).
• For model-specific results, use --report_op_coverage and check *_op_coverage_report.json.

Builtin supported (ONNX -> TFLite) in flatbuffer_direct

Custom-op candidates in flatbuffer_direct (opt-in)

Notes:

• Einsum is now treated as builtin_supported when it matches builtin constraints; unsupported Einsum patterns may still fallback to CUSTOM if custom-op mode is enabled.
• QLinearConv is treated as builtin_supported for regular/depthwise patterns; unsupported grouped patterns may still fallback to CUSTOM when custom-op mode is enabled.
• LogSoftmax is now treated as builtin_supported when builtin constraints pass; unsupported patterns may still fallback to CUSTOM if custom-op mode is enabled.
• LSTM is now treated as builtin_supported for constrained forward/reverse/bidirectional patterns; unsupported patterns may still fallback to CUSTOM if custom-op mode is enabled.
• NonMaxSuppression is now treated as builtin_supported when builtin constraints pass; unsupported patterns may still fallback to CUSTOM if custom-op mode is enabled.
• DynamicQuantizeLinear is now treated as builtin_supported for constrained float-input/uint8-output patterns; unsupported patterns may still fallback to CUSTOM if custom-op mode is enabled.
• If is now treated as builtin_supported for constrained patterns (NMS-guard, axis0 Add-branch, SequenceConstruct Add-branch, and nested ReduceMin/Add); unsupported patterns may still fallback to CUSTOM if custom-op mode is enabled.
• Loop is now treated as builtin_supported for constrained static-unroll/WHILE loop-carried patterns; unsupported patterns may still fallback to CUSTOM if custom-op mode is enabled.
• StringNormalizer is now treated as builtin_supported under constrained runtime patterns (case_change_action=NONE, locale ''/en_US, rank1/2). Unsupported runtime patterns (e.g. LOWER/UPPER or case-insensitive stopword filtering) may still fallback to CUSTOM if custom-op mode is enabled.
• StringNormalizer constant-input graphs are now folded at conversion time with string-constant buffer serialization support in flatbuffer_direct (including stopword filtering and LOWER/UPPER case conversion).
• OptionalHasElement is now treated as builtin_supported for determinable-presence cases (non-optional inputs and Optional producer lineage), reducing ONNX_OPTIONALHASELEMENT custom-op fallbacks.
• OneHot, MatMulInteger, Pow, Reciprocal, and Inverse are now treated as builtin_supported when builtin constraints pass.
• ReduceMin is now treated as builtin_supported under builtin constraints.
• Min and TopK are now treated as builtin_supported under builtin constraints, reducing ONNX_MIN / ONNX_TOPK custom-op fallbacks.
• DepthToSpace and HardSwish are now treated as builtin_supported under builtin constraints (HardSwish is lowered directly as TFLite HARD_SWISH).
• Pad builtin path now supports dynamic pads input (rank-1 length 2*rank) via CAST + RESHAPE + TRANSPOSE bridge before PAD.
• Pad builtin path now supports mode=reflect by lowering to TFLite MIRROR_PAD (REFLECT mode).
• Pad builtin path now supports constant non-zero padding values by lowering to TFLite PADV2 (non-quantized tensors).
• Unsqueeze builtin path now normalizes axes using output-rank semantics (output_rank = input_rank + len(axes)), supporting opset8-style cases such as axes=[2,3] for rank-2 input and reducing ONNX_UNSQUEEZE custom-op fallback.
• ConvTranspose builtin path has been expanded: constrained 1D lowering (EXPAND_DIMS -> TRANSPOSE_CONV -> SQUEEZE), relaxed output_padding handling (0 <= output_padding < stride), and explicit non-zero pad handling via post-crop when output shape is static.
• Conv now includes rank-5 CONV_3D builtin lowering (group=1, constant weights), reducing ONNX_CONV custom-op fallbacks on 3D conv subgraphs.
• ConvTranspose now includes rank-5 CONV_3D_TRANSPOSE builtin lowering (group=1, dilations=[1,1,1], constrained output_padding), reducing ONNX_CONVTRANSPOSE custom-op fallbacks on 3D deconv subgraphs.
• NonMaxSuppression builtin path now supports class dim >1 without forcing --output_nms_with_argmax by using per-class NON_MAX_SUPPRESSION_V4/V5 (selected by --switch_nms_version) and concatenating [batch, class, index] triplets (matching default onnx2tf/ops/NonMaxSuppression.py behavior when -onwa is not set).
• AveragePool builtin path now supports explicit pads and count_include_pad in {0,1}; when count_include_pad=0 with non-zero effective pads, divisor correction is applied to keep ONNX semantics.
• Leading input transpose passthrough optimization now treats CAST as passthrough, reducing redundant Transpose -> Cast -> (Sub/Mul/...) -> Transpose chains.
• Newly added builtin-covered ops in this update include: Abs, Acos, Acosh, And, ArgMin, Asin, Asinh, Atan, Atanh, BitShift, BitwiseAnd, BitwiseNot, BitwiseOr, BitwiseXor, Ceil, Celu, Cos, Cosh, Elu, Equal, EyeLike, Floor, GatherND, Gelu, Greater, GRU, Hardmax, Less, LessOrEqual, Mish, NonZero, Not, Or, Range, ReduceL1, ReduceL2, RNN, Round, Selu, Sign, Sin, Sinh, Softplus, Softsign, Tan, Trilu, Where, and Xor.
• Additional builtin-covered ops added in subsequent commits include: Erf, GlobalAveragePool, GlobalMaxPool, QLinearLeakyRelu, QLinearSoftmax, ScatterND, Slice, Split, and Tile.
• Newly builtin-covered ops added in this update include: Log, Max, RoiAlign, and ScatterElements.
• Resize builtin path now accepts dynamic rank-1 integer sizes input in addition to constant scales/sizes.
• Resize(cubic) now uses strict ONNX cubic semantics (including cubic_coeff_a and exclude_outside) in both tf_converter and flatbuffer_direct.
• GreaterOrEqual, RandomNormalLike, and constrained GridSample (rank-4/5, bilinear, padding_mode in {zeros,border}, align_corners in {0,1}) are now treated as builtin_supported under builtin constraints.
• Slice builtin path now supports additional constrained patterns: rank-1 dynamic-end prefix slice and full-axis reverse (step=-1) via REVERSE_V2.
• Abs builtin path now avoids unsupported ABS(INT64) by lowering INT64 input to NEG + MAXIMUM.
• tf_converter now prefers a non-Flex cubic lowering (RESHAPE + BATCH_MATMUL + RESHAPE + BATCH_MATMUL) when rank-4 input and static spatial sizes are available, reducing FlexResizeBicubic generation.
• flatbuffer_direct cubic lowering now preserves batch metadata on intermediate tensors and output tensors (no batch-dimension drop in RESHAPE/BATCH_MATMUL chain).
• NHWC propagation around Conv-family outputs has been tightened: CONV_2D / DEPTHWISE_CONV_2D / TRANSPOSE_CONV outputs now avoid redundant immediate post-transpose insertion when layout is already NHWC.
• Added direct NHWC passthrough optimization for TRANSPOSE(0,3,1,2) -> MEAN(keepDims=True) -> TRANSPOSE(0,2,3,1) by removing both transposes and remapping reduction axes.
• HardSigmoid-related transpose passthrough has been strengthened (including expanded MUL+ADD+RELU_0_TO_1 form), reducing redundant transpose wrappers in activation/residual chains.
• Added PReLU transpose passthrough optimization for TRANSPOSE(0,3,1,2) -> PRELU -> TRANSPOSE(0,2,3,1) style chains (including per-channel slope remap).
• Extended pre-concat transpose-chain optimization to handle broader unary ops and singleton-channel reshape adapters before CONCATENATION.
• Added ShuffleNet-style transpose-shuffle optimization to reduce long Transpose/Reshape/Transpose/Reshape/Gather chains while preserving downstream layout contracts.
• Added SiNet-tail NHWC optimization for Softmax-mask residual blocks: removes redundant pre/post transpose adapters around MUL/ADD/PRELU + SOFTMAX + REDUCE_MAX + RESHAPE + MUL + ADD and remaps axis/shape constants to NHWC to avoid terminal transpose bridges.
• Added affine fold for single-path conv chains: CONV_2D -> MUL(const) -> ADD(const) is folded into Conv weights/bias when the Conv output is not multi-fanout.
• Added clamp canonicalization: MAXIMUM(0.0) -> MINIMUM(1.0) is rewritten to RELU_0_TO_1 to reduce op count and improve downstream transpose/activation fusion opportunities.
• Added unary clamp canonicalization: MAXIMUM(x, 0.0) is rewritten to RELU(x) when the second input is singleton zero (input2=0) to reduce op count.
• Recurrent lowering practical notes:
  • LSTM builtin supports direction in {forward, reverse, bidirectional} and supports optional runtime initial_h/initial_c inputs and Y_h/Y_c outputs (under builtin shape constraints).
  • GRU builtin supports forward/reverse/bidirectional, but requires activations [Sigmoid,Tanh], clip=0, and no sequence_lens.
  • RNN builtin supports direction in {forward, reverse, bidirectional} (layout=0, no sequence_lens).

tf_converter vs flatbuffer_direct (operational differences)

flatbuffer_direct preprocess absorption scope

flatbuffer_direct runs staged preprocess rules before lowering. Current major coverage:

1. pattern_fusion_wave2
   • Relu -> Clip(min=0,max=6) chain normalization
   • GELU chain fusion (Div -> Erf -> Add -> Mul -> Mul)
   • Reshape -> Transpose -> Reshape to SpaceToDepth
2. quant_chain_fusion_wave3
   • DequantizeLinear -> BatchNormalization -> PRelu -> QuantizeLinear chain rewrite
   • BatchNormalization parameter folding into Mul + Add
3. pseudo_ops_wave1
   • LeakyRelu, limited Pow, and MatMulInteger rewrites to builtin-friendly forms
4. constant_fold_a5
   • Limited constant folding for shape/axes and arithmetic helper chains
   • Includes DequantizeLinear (axis/block-size aware) and downstream Reshape folding for constant-weight subgraphs
5. normalize_attrs_a5
   • Normalize perm/axes/negative-axis forms and softmax-axis bridge rewrites

Notes:

1. This reduces, but does not fully match, the TF-path replacement coverage.
2. To inspect what was applied, use --report_op_coverage and check preprocess_report.applied_rules.

Known constraints and workaround options

Demo

Video speed is adjusted approximately 50 times slower than actual speed.

Environment

• Linux / Windows
• Python>=3.12
• numpy==2.2.6
• onnx==1.20.1
• onnxruntime==1.26.0
• opencv-python==4.13.0.92
• onnxsim==0.6.5
• onnxoptimizer==0.4.2
• onnxscript==0.6.2
• ai-edge-litert==2.1.2
• sne4onnx==2.0.1
• sng4onnx==2.0.1
• psutil==5.9.5
• protobuf==7.35.1
• h5py==3.14.0
• ml_dtypes==0.5.4
• setuptools==81.0.0
• flatbuffers==25.12.19
• tqdm==4.67.1
• pytest==9.0.2

Optional:

• tensorflow==2.21.0 (TensorFlow-backed export / tf_converter only)
• tf-keras==2.21.0 (TensorFlow-backed export / tf_converter only)
• keras==3.15.0 (TensorFlow-backed export / tf_converter only)
• torch==2.11.0 (PyTorch-backed export / validation only)

Sample Usage

1. Install

Note:

1. If you are using TensorFlow v2.13.0 or earlier, use a version older than onnx2tf v1.17.5. onnx2tf v1.17.6 or later will not work properly due to changes in TensorFlow's API.

2. The latest onnx2tf implementation is based on Keras API 3 and will not work properly if you install TensorFlow v2.15.0 or earlier.

3. Starting with onnx2tf v2.0.0, due to onnxruntime issues, onnx2tf will no longer support environments older than Python 3.10. Accordingly, the Docker Image has been upgraded to Ubuntu 24.04. The dependency on onnx-graphsurgeon has also been completely removed. onnxruntime v1.24.1: https://github.com/microsoft/onnxruntime/releases/tag/v1.24.1

• HostPC

Click to expand

• When using GHCR, see Authenticating to the Container registry

  https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-to-the-container-registry

# PAT authentication is required to pull from GHCR.
docker login ghcr.io

Username (xxxx): {Enter}
Password: {Personal Access Token}
Login Succeeded

# Start an interactive session on the terminal.
docker run --rm -it \
-v `pwd`:/workdir \
-w /workdir \
ghcr.io/pinto0309/onnx2tf:2.6.9

or

# Authentication is not required for pulls from Docker Hub.
# Start an interactive session on the terminal.
docker run --rm -it \
-v `pwd`:/workdir \
-w /workdir \
docker.io/pinto0309/onnx2tf:2.6.9
or

# Direct execution in Docker
# The model conversion is performed within Docker,
# but the model is output to the host PC's storage.
docker run --rm \
--user $(id -u):$(id -g) \
-v $(pwd):/work \
docker.io/pinto0309/onnx2tf:2.6.9 \
onnx2tf -i /work/densenet-12.onnx -o /work/saved_model

or

curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.12.12
uv venv -p 3.12.12 .venv
source .venv/bin/activate
uv pip install -U onnx2tf

or

# Install TensorFlow-backed features too (tf_converter, SavedModel/H5/Keras exports).
uv pip install -U 'onnx2tf[tensorflow]'

or

# Install PyTorch-backed features too (native package / TorchScript / Dynamo ONNX / ExportedProgram).
uv pip install -U 'onnx2tf[torch]'

or

# Install all optional features at once.
uv pip install -U 'onnx2tf[tensorflow,torch]'

or

curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.12.12
uv venv -p 3.12.12 .venv
source .venv/bin/activate
uv sync

or

# Sync with TensorFlow-backed features enabled.
uv sync --extra tensorflow

or

# Sync with PyTorch-backed features enabled.
uv sync --extra torch

or

# Sync with all optional features enabled.
uv sync --all-extras

or

pip install -e .

or

pip install -e '.[tensorflow]'

or

pip install -e '.[torch]'

or

pip install -e '.[tensorflow,torch]'

or

docker buildx build \
--platform linux/amd64 \
--build-arg BUILD_ARCH=linux/amd64 \
--progress=plain \
-t onnx2tf:amd64 \
--load .

or

# It is possible to cross-compile an arm64 environment on an x64 environment.
docker buildx build \
--platform linux/arm64 \
--build-arg BUILD_ARCH=linux/arm64 \
--progress=plain \
-t onnx2tf:arm64 \
--load .

2. Run test

Only patterns that are considered to be used particularly frequently are described. In addition, there are several other options, such as disabling Flex OP and additional options to improve inference performance. See: CLI Parameter

# Float32, Float16
# This is the fastest way to generate tflite.
# Add `-fdosm` if you also want SavedModel output on the default backend.
# Also, starting from v1.24.0, efficient TFLite can be generated
# without unrolling `GroupConvolution`. e.g. YOLOv9, YOLOvN
# Conversion to other frameworks. e.g. TensorFlow.js, CoreML, etc
# https://github.com/PINTO0309/onnx2tf#19-conversion-to-tensorflowjs
# https://github.com/PINTO0309/onnx2tf#20-conversion-to-coreml
wget https://github.com/PINTO0309/onnx2tf/releases/download/0.0.2/resnet18-v1-7.onnx
onnx2tf -i resnet18-v1-7.onnx -fdosm

ls -lh saved_model/

assets
fingerprint.pb
resnet18-v1-7_float16.tflite
resnet18-v1-7_float32.tflite
saved_model.pb
variables

TF_CPP_MIN_LOG_LEVEL=3 \
saved_model_cli show \
--dir saved_model \
--signature_def serving_default \
--tag_set serve

The given SavedModel SignatureDef contains the following input(s):
  inputs['data'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, 224, 224, 3)
      name: serving_default_data:0
The given SavedModel SignatureDef contains the following output(s):
  outputs['output_0'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, 1000)
      name: PartitionedCall:0
Method name is: tensorflow/serving/predict

# In the interest of efficiency for my development and debugging of onnx2tf,
# the default configuration shows a large amount of debug level logs.
# However, for most users, a large number of debug logs are unnecessary.
# If you want to reduce the amount of information displayed in the conversion log,
# you can change the amount of information in the log by specifying the
# `--verbosity` or `-v` option as follows.
# Possible values are "debug", "info", "warn", and "error".
wget https://github.com/PINTO0309/onnx2tf/releases/download/0.0.2/resnet18-v1-7.onnx
onnx2tf -i resnet18-v1-7.onnx -v info

# Override undefined batch size or other dimensions with static values.
# If the model has undefined dimensions, rewriting them to a static size will significantly
# improve the success rate of the conversion.
# The `-b` option overwrites the zero-dimensional batch size with the number specified
# without input OP name.
# Note that if there are multiple input OPs, the zero dimension of all input OPs is
# forced to be rewritten.
# The `-sh/--shape-hints` option provides shape hints for input tensors with undefined
# dimensions, significantly improving the conversion success rate for models with dynamic
# input shapes. Specifying this option in combination with the `-b` option will further
# improve the success rate of model conversion. The `-sh` option does not change ONNX
# input OPs to static shapes.
# The `-ois/--overwrite_input_shape` option allows undefined dimensions in all dimensions,
# including the zero dimensionality, to be overwritten to a static shape, but requires
# the input OP name to be specified.
# e.g. -ois data1:1,3,224,224 data2:1,255 data3:1,224,6
wget https://github.com/PINTO0309/onnx2tf/releases/download/0.0.2/resnet18-v1-7.onnx
onnx2tf -i resnet18-v1-7.onnx -b 1
or
onnx2tf -i resnet18-v1-7.onnx -sh data:1,3,224,224 -b 1
or
onnx2tf -i resnet18-v1-7.onnx -ois data:1,3,224,224

# Suppress automatic transposition of input OPs from NCW, NCHW, NCDHW to NWC, NHWC, NDHWC.
# onnx2tf is a specification that automatically transposes the input OP to [N,H,W,C] format
# before converting the model. However, since onnx2tf cannot determine from the structure of
# the model whether the input data is image, audio data, or something else, it unconditionally
# transposes the channels. Therefore, it is the models of STT/TTS models where the input is
# not NHWC that tend to have particular problems with the automatic transposition of the
# input OP.
# If you do not want input OPs to be automatically transposed, you can disable automatic
# transposition of input OPs by specifying the `-kat` option.
wget https://github.com/PINTO0309/onnx2tf/releases/download/1.1.28/double_gru.onnx
# INPUT OPs: "spec": float32[1,3,257,1], "states_in": float32[2,1,32]
# The following command suppresses the automatic transposition of "states_in" and converts it.
onnx2tf -i double_gru.onnx -kat states_in

# Keras h5 format
# .h5, .json, .keras, .weights.h5, .weights.keras, .data-00000-of-00001, .index
wget https://github.com/PINTO0309/onnx2tf/releases/download/0.0.2/resnet18-v1-7.onnx
onnx2tf -i resnet18-v1-7.onnx -oh5

# Keras keras_v3 format (TensorFlow v2.12.0 or later only)
wget https://github.com/PINTO0309/onnx2tf/releases/download/0.0.2/resnet18-v1-7.onnx
onnx2tf -i resnet18-v1-7.onnx -okv3

# TensorFlow v1 (.pb) format
wget https://github.com/PINTO0309/onnx2tf/releases/download/0.0.2/resnet18-v1-7.onnx
onnx2tf -i resnet18-v1-7.onnx -otfv1pb

# Automatic JSON generation only
# Generates an optimal parameter replacement JSON file for model conversion.
# The JSON file is saved to {model_name}_auto.json when conversion errors occur
# or accuracy issues are detected and the feature is explicitly enabled.
onnx2tf -i model.onnx -agj

# Accuracy validation only (no JSON generation)
# Validates the accuracy between ONNX and TensorFlow outputs without generating
# any parameter replacement JSON file.
onnx2tf -i model.onnx -cotof

# Accuracy validation + automatic JSON generation
# First generates an optimal parameter replacement JSON file, then uses it
# to validate the model accuracy. This ensures the best possible conversion accuracy.
onnx2tf -i model.onnx -agj -cotof

# Accuracy validation with opt-in JSON generation on error
# Generates a parameter replacement JSON only when accuracy errors greater than 1e-2
# are detected during validation.
onnx2tf -i model.onnx -cotof -agje

# INT8 Quantization, Full INT8 Quantization
# INT8 Quantization with INT16 activation, Full INT8 Quantization with INT16 activation
# Dynamic Range Quantization
wget https://github.com/PINTO0309/onnx2tf/releases/download/1.1.1/emotion-ferplus-8.onnx
# INT8 Quantization (per-channel)
onnx2tf -i emotion-ferplus-8.onnx -oiqt
# INT8 Quantization (per-tensor)
onnx2tf -i emotion-ferplus-8.onnx -oiqt -qt per-tensor

# Split the model at the middle position for debugging
# Specify the input name of the OP
wget https://github.com/PINTO0309/onnx2tf/releases/download/1.25.0/cf_fus.onnx
onnx2tf -i cf_fus.onnx -inimc 448

# Split the model at the middle position for debugging
# Specify the output name of the OP
wget https://github.com/PINTO0309/onnx2tf/releases/download/1.25.0/cf_fus.onnx
onnx2tf -i cf_fus.onnx -onimc dep_sec

# Split the model at the middle position for debugging
# Specify the input/output name of the OP
wget https://github.com/PINTO0309/onnx2tf/releases/download/1.25.0/cf_fus.onnx
onnx2tf -i cf_fus.onnx -inimc 448 -onimc velocity

# Suppress generation of Flex OP and replace with Pseudo-Function
# [
#     Asin, Acos, Atan, Abs, PReLU,
#     LeakyReLU, Power, GatherND,
#     Neg, HardSwish, Erf, GeLU, MatMulInteger,
# ]
# Below is a sample of replacing Erf with another set of operations.
wget https://s3.ap-northeast-2.wasabisys.com/temp-models/onnx2tf_readme/Erf_11.onnx
onnx2tf -i Erf_11.onnx -rtpo Erf

# High-dimensional Transpose decomposition
# If you do not like FlexTranspose being generated, try `-nodaftc`.
# Suppresses the generation of FlexTranspose by decomposing Transpose
# to the specified number of dimensions.
# In TensorFlow v2.12.0 and later, up to 6 dimensions are converted to normal Transpose;
# in v2.11.0 and earlier, up to 5 dimensions are converted to normal Transpose.
# Note that specifying `2` for the `-nodaftc` option causes all Transpose OPs to disappear
# from the model structure.
# Below is an example of decomposing a Transpose of 5 or more dimensions into a Transpose
# of 4 dimensions.
onnx2tf -i xxxx.onnx -nodaftc 4

# High-dimensional Slice(StridedSlice) decomposition
# If your special circumstances do not allow you to deploy a `StridedSlice` with more than
# 5 dimensions to a device, you can use the `-nodafsc` option to decompose the `StridedSlice`
# into a process with 4 or fewer dimensions.
# Below is an example of decomposing a `StridedSlice` of 5 or more dimensions into a
# `StridedSlice` of 4 dimensions.
onnx2tf -i xxxx.onnx -nodafsc 4

# Float16 inference doubling on devices with ARM64 ARMv8.2 or higher instruction set
# Double the inference speed with Float16 precision tflite models on devices with
# high-performance CPUs such as Snapdragon.
# (Pixel 3a, Pixel 5a, Pixel 7, Galaxy M12 and Galaxy S22, ...)
# XNNPACK float16 inference on certain ARM64 cores is 2x faster.
# Unfortunately, Float16 inference cannot be accelerated when using the RaspberryPi4's
# ARM64 CPU.
onnx2tf -i xxxx.onnx -eatfp16

# Parameter replacement (Resize,Transpose,Softmax)
rm replace.json
wget https://github.com/PINTO0309/onnx2tf/releases/download/1.1.27/human_segmentation_pphumanseg_2021oct.onnx
wget https://github.com/PINTO0309/onnx2tf/releases/download/1.1.27/replace.json
onnx2tf -i human_segmentation_pphumanseg_2021oct.onnx -prf replace.json

3. Accuracy check

Click to expand

Perform error checking of ONNX output and TensorFlow output. Verify that the error of all outputs, one operation at a time, is below a certain threshold. Automatically determines before and after which OPs the tool's automatic conversion of the model failed. Know where dimensional compression, dimensional expansion, and dimensional transposition by Reshape and Traspose are failing. Once you have identified the problem area, you can refer to the tutorial on Parameter replacement to modify the tool's behavior.

After many upgrades, the need for JSON parameter correction has become much less common, but there are still some edge cases where JSON correction is required. If the PC has sufficient free space in its RAM, onnx2tf will convert the model while carefully performing accuracy checks on all OPs. Thus, at the cost of successful model conversion, the conversion speed is a little slower. If the amount of RAM required for the accuracy check is expected to exceed 80% of the total available RAM capacity of the entire PC, the conversion operation will be performed without an accuracy check. Therefore, if the accuracy of the converted model is found to be significantly degraded, the accuracy may be automatically corrected by re-conversion on a PC with a large amount of RAM. For example, my PC has 128GB of RAM, but the StableDiffusion v1.5 model is too complex in its structure and consumed about 180GB of RAM in total with 50GB of SWAP space.

-ois an option to overwrite the input OP to a static size if it has undefined dimensions. -cotof option checks the accuracy of all OPs one by one. -cotoa is the error value of the threshold for determining an accuracy error. If there are undefined dimensions in the input OP, it is better to fix them to the static geometry to improve the accuracy of the accuracy measurement.

Also, you can use the -cind option to specify custom input for -cotof, instead of using the default dummy input. Otherwise, all input values will be set to 1. You can override the dummy input values with --value_hints (scalar only, *:default supported). For more information about the -cind option, please refer to here. If your input is image data in NHWC format, you can also use --test_data_nhwc_path to provide fixed test samples for validation. For -fdots, the recommended way to resolve dynamic trace shapes is --shape_hints. --test_data_nhwc_path is also accepted for eligible 4D RGB inputs, and -cind remains available when per-input custom data is needed.

Quick difference between -tdnp and -cind:

• -tdnp (--test_data_nhwc_path): Validation-only test data for accuracy checks. Expects one NHWC RGB .npy ([N,H,W,3]). No mean/std. For multi-input models, this single array is reused across inputs (per-input mapping is not supported). Also accepted by -fdots for eligible 4D RGB inputs.
• -cind (--custom_input_op_name_np_data_path): Per-input custom data mapping by input name. Supports multi-input/non-image inputs. Also used for -fdots trace inputs and INT8 calibration (-oiqt) with optional mean/std.

The -cotof option evaluates Float32 accuracy only. On the default flatbuffer_direct path, the base report is ONNX↔TFLite. When --tflite_backend tf_converter is explicitly used, it checks ONNX against TensorFlow/TFLite outputs. If --flatbuffer_direct_output_pytorch is also enabled, onnx2tf additionally emits ONNX↔PyTorch and combined comparison reports using the same input samples.

onnx2tf -i mobilenetv2-12.onnx -ois input:1,3,224,224 -cotof -cotoa 1e-1

or

onnx2tf -i mobilenetv2-12.onnx -b 1 -cotof -cotoa 1e-1

or

onnx2tf -i mobilenetv2-12.onnx -cotof -cotoa 1e-1 -cind "input" "/your/path/x.npy"

or

onnx2tf -i mobilenetv2-12.onnx -cotof -cotoa 1e-1 -tdnp "/your/path/test_data_nhwc.npy"

or

onnx2tf -i mobilenetv2-12.onnx -cotof -cotoa 1e-1 --value_hints "input:0.5" "*:1.0"

4. Match tflite input/output names and input/output order to ONNX

Click to expand

If you want to match tflite's input/output OP names and the order of input/output OPs with ONNX, you can use the interpreter.get_signature_runner() to infer this after using the -coion / --copy_onnx_input_output_names_to_tflite option to output tflite file. See: https://github.com/PINTO0309/onnx2tf/issues/228

onnx2tf automatically compares the final input/output shapes of ONNX and the generated TFLite and tries to automatically correct the input/output order as much as possible if there is a difference. However, if INT8 quantization is used and there are multiple inputs and outputs with the same shape, automatic correction may fail. This is because TFLiteConverter shuffles the input-output order by itself only when INT8 quantization is performed.

import torch
import onnxruntime
import numpy as np
import onnx2tf
import tensorflow as tf
from ai_edge_litert.interpreter import Interpreter

class Model(torch.nn.Module):
    def forward(self, x, y):
        return {
            "add": x + y,
            "sub": x - y,
        }

# Let's double check what PyTorch gives us
model = Model()
pytorch_output = model.forward(10, 2)
print("[PyTorch] Model Predictions:", pytorch_output)

# First, export the above model to ONNX
torch.onnx.export(
    Model(),
    {"x": 10, "y": 2},
    "model.onnx",
    opset_version=16,
    input_names=["x", "y"],
    output_names=["add", "sub"],
)

# And check its output
session = onnxruntime.InferenceSession("model.onnx")
onnx_output = session.run(["add", "sub"], {"x": np.array(10), "y": np.array(2)})
print("[ONNX] Model Outputs:", [o.name for o in session.get_outputs()])
print("[ONNX] Model Predictions:", onnx_output)

# Now, let's convert the ONNX model to TF
onnx2tf.convert(
    input_onnx_file_path="model.onnx",
    output_folder_path="model.tf",
    copy_onnx_input_output_names_to_tflite=True,
    non_verbose=True,
)

# Now, test the newer TFLite model
interpreter = Interpreter(model_path="model.tf/model_float32.tflite")
tf_lite_model = interpreter.get_signature_runner()
inputs = {
  'x': np.asarray([10], dtype=np.int64),
  'y': np.asarray([2], dtype=np.int64),
}
tf_lite_output = tf_lite_model(**inputs)
print("[TFLite] Model Predictions:", tf_lite_output)

[PyTorch] Model Predictions:
  {
    'add': 12,
    'sub': 8
  }
[ONNX] Model Outputs:
  [
    'add',
    'sub'
  ]
[ONNX] Model Predictions:
  [
    array(12, dtype=int64),
    array(8, dtype=int64)
  ]
[TFLite] Model Predictions:
  {
    'add': array([12]),
    'sub': array([8])
  }

5. Rewriting of tflite input/output OP names and signature_defs

Click to expand

If you do not like tflite input/output names such as serving_default_*:0 or StatefulPartitionedCall:0, you can rewrite them using the following tools and procedures. It can be rewritten from any name to any name, so it does not have to be serving_default_*:0 or StatefulPartitionedCall:0.

https://github.com/PINTO0309/tflite-input-output-rewriter

# Install tfliteiorewriter
pip install -U tfliteiorewriter

• Before

  

  tfliteiorewriter \
  -i xxxx.tflite \
  -r serving_default_input_1:0 aaa \
  -r StatefulPartitionedCall:0 bbb
  

  

• After

  

6. Embed metadata in tflite

Click to expand

If you want to embed label maps, quantization parameters, descriptions, etc. into your tflite file, you can refer to the official tutorial and try it yourself. For now, this tool does not plan to implement the ability to append metadata, as I do not want to write byte arrays to the tflite file that are not essential to its operation.

• Adding metadata to TensorFlow Lite models

  https://www.tensorflow.org/lite/models/convert/metadata

7. If the accuracy of the INT8 quantized model degrades significantly

Click to expand

It is a matter of model structure. The activation function (SiLU/Swish), kernel size and stride for Pooling, and kernel size and stride for Conv should be completely revised. See: https://github.com/PINTO0309/onnx2tf/issues/269

If you want to see the difference in quantization error between SiLU and ReLU, please check this Gist by @motokimura who helped us in our research. Thanks Motoki!

Gist: Quantization error simulation of SiLU (Swish) activation

The accuracy error rates after quantization for different activation functions are shown in the figure below. The graph plots the distribution of absolute error, so a position with a higher value on the horizontal axis indicates a larger error. The vertical axis is the number of samples. SiLU (Swish) produces catastrophic errors after INT8 quantization.

• e.g. YOLOX-Nano
  • https://github.com/motokimura/yolox-ti-lite_tflite
  • https://github.com/TexasInstruments/edgeai-yolox

    Before	After
    Swish/SiLU

|ReLU
| |DepthwiseConv2D
|Conv2D
| |MaxPool, kernel_size=5x5,9x9,13x13
|MaxPool, kernel_size=3x3
|

```
### Float32 - YOLOX-Nano
(1, 52, 52, 85)
array([[[
    [ 0.971787,  0.811184,  0.550566, ..., -5.962632, -7.403673, -6.735206],
    [ 0.858804,  1.351296,  1.231673, ..., -6.479690, -8.277064, -7.664936],
    [ 0.214827,  1.035119,  1.458006, ..., -6.291425, -8.229385, -7.761562],
        ...,
    [ 0.450116,  1.391900,  1.533354, ..., -5.672194, -7.121591, -6.880231],
    [ 0.593133,  2.112723,  0.968755, ..., -6.150078, -7.370633, -6.874294],
    [ 0.088263,  1.985220,  0.619998, ..., -5.507928, -6.914980, -6.234259]]]]),

### INT8 - YOLOX-Nano
(1, 52, 52, 85)
array([[[
    [ 0.941908,  0.770652,  0.513768, ..., -5.993958, -7.449634, -6.850238],
    [ 0.856280,  1.284420,  1.198792, ..., -6.507727, -8.391542, -7.792146],
    [ 0.256884,  0.941908,  1.455676, ..., -6.336471, -8.305914, -7.877774],
        ...,
    [ 0.342512,  1.370048,  1.541304, ..., -5.737075, -7.192750, -7.107122],
    [ 0.513768,  2.226327,  1.027536, ..., -6.165215, -7.449634, -7.021494],
    [ 0.085628,  2.055072,  0.685024, ..., -5.480191, -7.021494, -6.422099]]]]),
```

• Other recommended replacement OP

  Before	After
  HardSwish

|ReLU
| |ReLU6

Paper: A Quantization-Friendly Separable Convolution for MobileNets https://arxiv.org/pdf/1803.08607.pdf|`ReLU`
|

• Quantization range collapse due to non-zero constant padding

  If padding is performed with a constant other than zero, the padding value may destroy the quantization range of the input tensor. For example, the pattern is shown in the figure below. The MaxPool2D is done after padding the 4 sides of the input tensor with the minimum value of Float32. It seems that if INT8 quantization is performed with this structure, the quantization range is determined by MaxPool2D during quantization, including the values padded to the tensor. See: #444

  Therefore, the following two similar examples are equally likely to result in divergent output values for the model after INT8 quantization, with all output values being Nan or zero.

  1. Pattern with fixed value -255.0 padded on 4 sides of tensor

  2. Pattern with fixed value -128.0 padded on 4 sides of tensor

8. Calibration data creation for INT8 quantization

Click to expand

Calibration data (.npy) for INT8 quantization (-cind) is generated as follows. This is a sample when the data used for training is image data. See: https://github.com/PINTO0309/onnx2tf/issues/222

https://www.tensorflow.org/lite/performance/post_training_quantization

import cv2
import glob
import numpy as np

# Not used during data generation ################################
# You will need to do the calculations yourself using the test data
MEAN = np.asarray([[[[0.485, 0.456, 0.406]]]], dtype=np.float32) # [1,1,1,3]
STD = np.asarray([[[[0.229, 0.224, 0.225]]]], dtype=np.float32) # [1,1,1,3]
# Not used during data generation ################################

files = glob.glob("data/*.png")
img_datas = []
for idx, file in enumerate(files):
    bgr_img = cv2.imread(file)
    rgb_img = cv2.cvtColor(bgr_img, cv2.COLOR_BGR2RGB)
    resized_img = cv2.resize(rgb_img, dsize=(200,112))
    extend_batch_size_img = resized_img[np.newaxis, :]
    normalized_img = extend_batch_size_img / 255.0 # 0.0 - 1.0
    print(
        f'{str(idx+1).zfill(2)}. extend_batch_size_img.shape: {extend_batch_size_img.shape}'
    ) # [1,112,200,3]
    img_datas.append(extend_batch_size_img)
calib_datas = np.vstack(img_datas)
print(f'calib_datas.shape: {calib_datas.shape}') # [10,112,200,3]
np.save(file='data/calibdata.npy', arr=calib_datas)

loaded_data = np.load('data/calibdata.npy')
print(f'loaded_data.shape: {loaded_data.shape}') # [10,112,200,3]

"""
-cind INPUT_NAME NUMPY_FILE_PATH MEAN STD
int8_calib_datas = (loaded_data - MEAN) / STD # -1.0 - 1.0

e.g. How to specify calibration data in CLI or Script respectively.
1. CLI
  -cind "pc_dep" "data/calibdata.npy" "[[[[0.485,0.456,0.406]]]]" "[[[[0.229,0.224,0.225]]]]"
  -cind "feat" "data/calibdata2.npy" "[[[[0.123,...,0.321]]]]" "[[[[0.112,...,0.451]]]]"

2. Script
  custom_input_op_name_np_data_path=[
      ["pc_dep", "data/calibdata.npy", [[[[0.485,0.456,0.406]]]], [[[[0.229,0.224,0.225]]]]],
      ["feat", "data/calibdata2.npy", [[[[0.123,...,0.321]]]], [[[[0.112,...,0.451]]]],
  ]
"""

9. INT8 quantization of models with multiple inputs requiring non-image data

Click to expand

If you do not need to perform INT8 quantization with this tool alone, the following method is the easiest.

On the default flatbuffer_direct backend, combine -fdosm with -osd to output a saved_model.pb in the saved_model folder with the full size required for quantization. That is, a default signature named serving_default is embedded in .pb. If you need the legacy behavior instead, use --tflite_backend tf_converter explicitly. The -b option is used to convert the batch size by rewriting it as a static integer.

Note: INT8 TFLite generated by following this procedure as is will result in a model with significantly degraded accuracy. This tutorial only demonstrates the INT8 quantization procedure; if you wish to correct for accuracy, please refer to Parameter replacement to correct for transposition errors in the operation.

# Ref: https://github.com/onnx/models/tree/main/text/machine_comprehension/bert-squad
wget https://s3.ap-northeast-2.wasabisys.com/temp-models/onnx2tf_248/bertsquad-12.onnx

onnx2tf -i bertsquad-12.onnx -b 1 -fdosm -osd -cotof

Use the saved_model_cli command to check the saved_model signature. INT8 quantization calibration using signatures allows correct control of the input order of data for calibration. Therefore, calibration with signatures is recommended for INT8 quantization of models with multiple inputs.

saved_model_cli show --dir saved_model/ --tag_set serve --signature_def serving_default

The given SavedModel SignatureDef contains the following input(s):
  inputs['input_ids_0'] tensor_info:
      dtype: DT_INT64
      shape: (1, 256)
      name: serving_default_input_ids_0:0
  inputs['input_mask_0'] tensor_info:
      dtype: DT_INT64
      shape: (1, 256)
      name: serving_default_input_mask_0:0
  inputs['segment_ids_0'] tensor_info:
      dtype: DT_INT64
      shape: (1, 256)
      name: serving_default_segment_ids_0:0
  inputs['unique_ids_raw_output___9_0'] tensor_info:
      dtype: DT_INT64
      shape: (1)
      name: serving_default_unique_ids_raw_output___9_0:0

Calibrate by specifying the input OP name displayed in inputs. The np.ones([xxx], dtype=np.int64) part must be replaced with the correct calibration test data. In practice, several pieces of data used for training are extracted and used.

import tensorflow as tf
import numpy as np

def representative_dataset():
    unique_ids = np.ones([10, 256], dtype=np.int64)
    segment_ids = np.ones([10, 256], dtype=np.int64)
    input_masks = np.ones([10, 256], dtype=np.int64)
    input_ids = np.ones([10], dtype=np.int64)

    for unique_id, segment_id, input_mask, input_id \
        in zip(unique_ids, segment_ids, input_masks, input_ids):

        yield {
            "unique_ids_raw_output___9_0": unique_id,
            "segment_ids_0": segment_id,
            "input_mask_0": input_mask,
            "input_ids_0": input_id,
        }

converter = tf.lite.TFLiteConverter.from_saved_model('saved_model')
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8  # or tf.uint8
converter.inference_output_type = tf.int8  # or tf.uint8
tflite_quant_model = converter.convert()

with open('saved_model/int8_model.tflite', 'wb') as w:
    w.write(tflite_quant_model)

https://www.tensorflow.org/lite/performance/post_training_quantization

See: https://github.com/PINTO0309/onnx2tf/issues/248

10. Fixing the output of NonMaxSuppression (NMS)

Click to expand

PyTorch's NonMaxSuppression (torchvision.ops.nms) and ONNX's NonMaxSuppression are not fully compatible. TorchVision's NMS is very inefficient. Therefore, it is inevitable that converting ONNX using NMS in object detection models and other models will be very redundant and will be converted with a structure that is difficult for TensorFlow.js and TFLite models to take advantage of in devices. This is due to the indefinite number of tensors output by the NMS. In this chapter, I share how to easily tune the ONNX generated using TorchVision's redundant NMS to generate an optimized NMS.

1. There are multiple issues with TorchVision's NMS. First, the batch size specification is not supported; second, the max_output_boxes_per_class parameter cannot be specified. Please see the NMS sample ONNX part I generated. The max_output_boxes_per_class has been changed to 896 instead of -Infinity. The biggest problem with TorchVision NMS is that it generates ONNX with max_output_boxes_per_class set to -Infinity or 9223372036854775807 (Maximum value of INT64), resulting in a variable number of NMS outputs from zero to infinite. Thus, by rewriting -Infinity or 9223372036854775807 (Maximum value of INT64) to a constant value, it is possible to output an NMS that can be effortlessly inferred by TFJS or TFLite.

   Here you will find committed ONNX components optimized for various devices. https://github.com/PINTO0309/components_of_onnx/tree/main/components_of_onnx/ops

2. In the following example, the max_output_boxes_per_class of NMS in the post-processing generated by YOLOv7 is changed from -Infinity or 9223372036854775807 (Maximum value of INT64) to 20, as shown in the figure below. The name main01_max_output_boxes_per_class has been rewritten by me for clarity, but it originally appears as max_output_boxes_per_class.

   Simply execute the following command. The command rewrites the specified attribute value of the OP specified by ONNX.

   pip install sam4onnx
   
   sam4onnx \
   --op_name main01_nonmaxsuppression11 \
   --input_onnx_file_path yolov7.onnx \
   --output_onnx_file_path nms_yolov7_update.onnx \
   --input_constants main01_max_output_boxes_per_class int64 [20]
   

   A tutorial on one of my ONNX modification tools, sam4onnx, can be found here.

   https://github.com/PINTO0309/sam4onnx

   Many detailed tutorials are provided below, so if you are interested, please play with them.

   https://github.com/PINTO0309/PINTO_model_zoo/tree/main/307_YOLOv7/post_process_gen_tools

3. Finally, simply convert ONNX to TFLite or saved_model or TFJS using onnx2tf. onnx2tf performs an internal operation to automatically optimize the NMS output to a fixed shape if max_output_boxes_per_class is set to a value other than -Infinity and 9223372036854775807 (Maximum value of INT64). Specify --output_nms_with_dynamic_tensor or -onwdt if you do not want to optimize for a fixed shape. If you want to shrink class scores in NMS from [B, C, N] to [B, 1, N], enable --output_nms_with_argmax or -onwa.

   onnx2tf -i nms_yolov7_update.onnx -fdosm -osd -cotof
   

   I would be happy if this is a reference for Android + Java or TFJS implementations. There are tons more tricky model optimization techniques described in my blog posts, so you'll have to find them yourself. I don't dare to list the URL here because it is annoying to see so many issues being posted. And unfortunately, all articles are in Japanese.

11. RNN (RNN, GRU, LSTM) Inference Acceleration

Click to expand

TensorFlow's RNN has a speedup option called unroll. The network will be unrolled, else a symbolic loop will be used. Unrolling can speed-up a RNN, although it tends to be more memory-intensive. Unrolling is only suitable for short sequences. onnx2tf allows you to deploy RNNs into memory-intensive operations by specifying the --enable_rnn_unroll or -eru options. The --enable_rnn_unroll option is available for RNN, GRU, and LSTM.

• Keras https://keras.io/api/layers/recurrent_layers/lstm/
• TensorFlow https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM

An example of BidirectionalLSTM conversion with the --enable_rnn_unroll option is shown below. Please ignore that the shapes of the input and output tensors do not match, since the samples are shown by picking up separate models.

• ONNX LSTM (Bidirectional)

  

• BidirectionalLSTM with --enable_rnn_unroll option unspecified

  Recurrent layer is implemented from scratch.

  

• BidirectionalLSTM with --enable_rnn_unroll option

  

12. If the accuracy of the Float32 model degrades significantly

Click to expand

The pattern of accuracy degradation of the converted model does not only occur when INT8 quantization is performed. A special edge case is when there is a problem with the implementation of a particular OP on the TFLite runtime side. Below, I will reproduce the problem by means of a very simple CNN model and further explain its workaround. Here is the issue that prompted me to add this explanation. [Conv-TasNet] Facing issue in converting Conv-TasNet model #447

Download a sample model for validation.

curl \
-L https://github.com/PINTO0309/onnx2tf/files/12367312/prelu_check.onnx.zip \
-o prelu_check.onnx.zip

unzip prelu_check.onnx.zip

The part of the downloaded model where the problem occurs is the PRelu part in the figure below.

• ONNX

  

Reproduce the problem. The following command converts an ONNX file to a TFLite file.

onnx2tf -i prelu_check.onnx -cotof

The conversion was successful and, as shown in the figure below, the inference test results from ONNX and the inference results for the Float32 model in TensorFlow (Keras) match perfectly. It is important to note that the comparison of inference results between ONNX and TensorFlow transformed models is comparing ONNX models with TensorFlow (Keras) models, not ONNX models with TFLite models.

• Conversion results

  

• tflite

  

Now, let's try inference with the TFLite runtime instead of the TensorFlow runtime.

• test.py

  import time
  import numpy as np
  np.random.seed(0)
  from ai_edge_litert.interpreter import Interpreter
  
  # Load TFLite model
  interpreter = Interpreter(model_path="./saved_model/prelu_check_float32.tflite")
  interpreter.allocate_tensors()
  tensor_shape = (256, 20)
  input_data = {'waveform': np.random.randn(*tensor_shape).astype(np.float32)}
  
  # Load and preprocess
  input_details = interpreter.get_input_details()
  input_shape = input_details[0]['shape']
  print(input_shape)
  
  # Run inference
  interpreter.set_tensor(input_details[0]['index'], input_data["waveform"])
  separate_time = time.time()
  interpreter.invoke()
  print("Done! {:.3f} s".format(time.time() - separate_time))
  output_details = interpreter.get_output_details()
  output_data = interpreter.get_tensor(output_details[0]['index'])
  
  output_data = []
  for output_detail in output_details:
      output_data.append(interpreter.get_tensor(output_detail['index']))
  
  print(output_data)
  

Oddly enough, the output value of PReLU contains multiple nan. However, as can be seen by converting the ONNX model to the middle of the model using the -onimc option, nan does not occur until just before PReLU. Thus, it is clear that the PReLU OP in the TFLite runtime has a problem with divergent inference results.

• TFLite inference results

  

The following is a work-around to avoid this problem. Use the -rtpo option to replace PReLU with a similar primitive operation when transforming a model, and then perform the model transformation.

onnx2tf -i prelu_check.onnx -cotof -rtpo PReLU

As before, the inference results from ONNX and TensorFlow (Keras) match perfectly.

• Conversion results

  

However, -rtpo PReLU will generate a .tflite file with the PRelu OP replaced by a primitive OP combination.

• tflite

  

Again, run the test code to check the inference results. The figure below shows that no nan occurs when inference is performed by replacing the PReLU OP with only combinations of primitive operations. In other words, it is important to know that large arithmetic errors are not only due to the broken structure of the model, but can also be caused by internal implementations such as the TFLite runtime. I have implemented the -rtpo option to replace operators as a work-around to avoid such runtime problems.

• TFLite inference results

  

13. Problem of extremely large calculation error in InstanceNormalization

Click to expand

Even if the conversion is successful, InstanceNormalization tends to have very large errors. This is an ONNX specification.

• See.1: https://discuss.pytorch.org/t/understanding-instance-normalization-2d-with-running-mean-and-running-var/144139
• See.2: https://github.com/pytorch/pytorch/issues/72057

I verified this with a very simple sample model. There are more than 8 million elements, and the calculation error reached 1e-2.

14. Inference with dynamic tensors in TFLite

Click to expand

For some time now, TFLite runtime has supported inference by dynamic tensors. However, the existence of this important function is not widely recognized. In this chapter, I will show how I can convert an ONNX file that contains dynamic geometry in batch size directly into a TFLite file that contains dynamic geometry and then further infer it in variable batch conditions. The issue that inspired me to add this tutorial is here. [Dynamic batch / Dynamic shape] onnx model with dynamic input is converted to tflite with static input 1 #441, or Cannot use converted model with dynamic input shape #521

First, download the sample ONNX file.

wget https://s3.ap-northeast-2.wasabisys.com/temp-models/onnx2tf_441/osnet_x0_25_msmt17.onnx

This model calculates the similarity of features by cosine similarity. The batch size dimension of the input tensor is batch, allowing various numbers of images to be input simultaneously. This is often used, for example, to achieve tracking by calculating the similarity of people or objects reflected between successive video frames. However, the total number of objects to be tracked changes rapidly with each video frame because the number of people and objects in the image constantly increases and decreases. Therefore, there is a very significant use case for generating models with variable settings for the number of input images (batch size) of the model.

Convert the downloaded OSNet to tflite and saved_model as a variable batch. If you do not specify the -b or -ois options, onnx2tf does not change the batch size as N. The only important point is to convert the model with the -osd and -coion options.

onnx2tf -i osnet_x0_25_msmt17.onnx -fdosm -osd -coion

• .tflite

  When viewing tflite in Netron, the batch size appears to be fixed at 1.

• saved_model

  However, checking the structure of saved_model, the batch size is correctly set to -1.

  saved_model_cli show --dir saved_model/ --all
  
  MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:
  
  signature_def['__saved_model_init_op']:
    The given SavedModel SignatureDef contains the following input(s):
    The given SavedModel SignatureDef contains the following output(s):
      outputs['__saved_model_init_op'] tensor_info:
          dtype: DT_INVALID
          shape: unknown_rank
          name: NoOp
    Method name is:
  
  signature_def['serving_default']:
    The given SavedModel SignatureDef contains the following input(s):
      inputs['images'] tensor_info:
          dtype: DT_FLOAT
          shape: (-1, 256, 128, 3)
          name: serving_default_images:0
    The given SavedModel SignatureDef contains the following output(s):
      outputs['output'] tensor_info:
          dtype: DT_FLOAT
          shape: (-1, 512)
          name: PartitionedCall:0
    Method name is: tensorflow/serving/predict
  

To prove that the tflite structure has been converted correctly, I will convert the tflite to JSON and look at the structure. (The flatc command below is only for manual inspection in the tflite2json2tflite container, not a requirement for onnx2tf conversion.)

docker run --rm -it \
-v `pwd`:/home/user/workdir \
ghcr.io/pinto0309/tflite2json2tflite:latest

./flatc -t \
--strict-json \
--defaults-json \
-o workdir \
./schema.fbs -- workdir/saved_model/osnet_x0_25_msmt17_float32.tflite

ls -l workdir

-rw-rw-r-- 1 user user   921564 Aug  4 10:24 osnet_x0_25_msmt17.onnx
-rw-r--r-- 1 user user 10369524 Aug  4 10:30 osnet_x0_25_msmt17_float32.json
drwxrwxr-x 4 user user     4096 Aug  4 10:26 saved_model

• osnet_x0_25_msmt17_float32.json

  "shape_signature" is correctly set to -1. However, "shape" is set to 1. This could be a problem with TFLiteConverter, or it could be a problem with Netron's graphical display capabilities.

In other words, although onnx2tf converts TFLiteConverer as specified, with the batch size of -1 without any model processing, only Netron's display is broken. This is a problem I have known for quite some time. However, the inference itself does not cause the problem.

If you want to infer in variable batches, you need to infer using signature. In such cases, the -coion option must be specified when converting the model. Note that I have identified a problem with quantization with the -coion option, which can corrupt tflite files. https://github.com/PINTO0309/onnx2tf/issues/429

https://github.com/PINTO0309/onnx2tf#4-match-tflite-inputoutput-names-and-inputoutput-order-to-onnx

You can use signature_runner to handle dynamic input tensors by performing inference using signature. Below I show that both batch_size=5 and batch_size=3 tensors can be inferred with the same model.

• test.py - Batch size: 5

  import numpy as np
  from ai_edge_litert.interpreter import Interpreter
  from pprint import pprint
  
  interpreter = Interpreter(model_path="saved_model/osnet_x0_25_msmt17_float32.tflite")
  tf_lite_model = interpreter.get_signature_runner()
  inputs = {
      'images': np.ones([5,256,128,3], dtype=np.float32),
  }
  tf_lite_output = tf_lite_model(**inputs)
  print(f"[TFLite] Model Predictions shape: {tf_lite_output['output'].shape}")
  print(f"[TFLite] Model Predictions:")
  pprint(tf_lite_output)
  

• Results

  [TFLite] Model Pr