C++ Runtime Note

Model v.s. ModelContainer

These are the two main classes involved in the C++ runtime implementation:

  • The bulk of the runtime implementation is in the Model class.

  • The ModelContainer class stores a set of shared constants and a collection of Model instances. Almost all functions in model_interface.h forward to a method in ModelContainer. When Run is invoked, ModelContainer looks for an available Model, or blocks until one becomes available (see the section on asynchronous predictions). It then forwards the run request to the runtime.

Code Structure

Some important files:

  1. include/model_interface.h: The interface that we expose in the compiled .so.

  2. include/model_container.h: The bulk of the ModelContainer implementation.

Some files are generated at compile time. These include:

  • model-generated.h: The implementation of the Model.

  • model_container_base.cu: A small part of the implementation for ModelContainer that needs to be generated. ModelContainer inherits from ModelContainerBase, and ModelContainerBase’s implementation lives in this file. See model_container.h for more details.

All codegen templates can be found in backend/main_templates.py. The codegen implementation is in backend/codegen.py.

Note that many of the headers in this directory rely on generated code and thus cannot be #include -d in external projects. model_interface.h is an exception.