Popular DL frameworks, such as TensorFlow, PyTorch, and Caffe, are designed primarily for training deep neural networks. They focus on offering features that are more useful for researchers to experiment easily with different types of network structures, training regimens, and techniques to achieve optimum training accuracy to solve a particular problem in the real world. After a neural network model has been successfully trained, practitioners could continue to use the same DL framework for deploying the trained model for inference. However, there are more efficient deployment solutions for inference. These are pieces of inference software that compile a trained model into a computation engine that is most efficient in latency or throughput on the accelerator hardware used for deployment.

Much like a C or C++ compiler, inference engines take the trained model...