octoml-profile

Installation and Getting Started

• Create and activate a python virtual environment. Python 3.8 is recommended and tested on both Ubuntu and macOS. Python 3.10.9 is tested on macOS with Apple silicon.

  python3 -m venv env
  source env/bin/activate
  

• Install dependencies

  PyTorch 2.0 and above is required. Below we install the cpu version for simplicity; CUDA version works too.

  pip install --upgrade pip
  pip install "torch>=2.0.0" torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
  pip install "octoml-profile>=0.2.0"
  

  You've completed installation! (If you have trouble, see issues with installation)

• Next, try running this very simple example that shows how to integrate octoml-profile into your model code.

  import torch
  from torch.nn import Linear, ReLU, Sequential
  from octoml_profile import accelerate, remote_profile
  
  model = Sequential(Linear(100, 200), ReLU())
  
  @accelerate
  def predict(x: torch.Tensor):
      return model(x)
  
  with remote_profile():
      for _ in range(3):
          x = torch.randn(1, 100)
          predict(x)
  

• The first time you run this, you'll be prompted to supply your API key.

      ,-""-.
    /      \    Welcome to OctoML Profiler!
    :        ;
    \      /    It looks like you don't have an access token configured.
      `.  .'     Please go to https://profiler.app.octoml.ai/ to generate one
    '._.'`._.'   and then paste it here.
  
  Access token: 
  

  (Sign up so that you can generate an API token when prompted)

• Once you've provided credentials, running this results in the following output that shows times of the function being executed remotely on each backend.

  Function `predict` has 1 profile:
  - Profile `predict[1/1]` ran 3 times. (1 discarded because compilation happened)
  
  Instance     Processor           Backend              Backend Time (ms)  Total Time (ms)  Cost ($/MReq)
  =======================================================================================================
  r6i.large    Intel Ice Lake CPU  torch-eager-cpu                  0.024            0.086          $0.00
  g4dn.xlarge  Nvidia T4 GPU       torch-eager-cuda                 0.097            0.159          $0.02
  g4dn.xlarge  Nvidia T4 GPU       torch-inductor-cuda              0.177            0.239          $0.03
  -------------------------------------------------------------------------------------------------------
  Total time above is `remote backend time + local python code time`,
  in which local python code run time is 0.062 ms.
  Graph level profile is located at /tmp/octoml_profile_8o45fe39/0/predict_1*
  

  To see more examples, see examples/.

Issues with installation

• If you are on macOS with Apple silicon and seeing symbol not found in flat namespace '_CFRelease', it is likely that you created a venv with python installed by conda. Please make sure to deactivate any conda environment(s) and use the system-shipped python on macOS to create venv. Or follow the instructions below to create a conda environment.

  conda create -n octoml python=3.8
  conda activate octoml
  

• If you see a version conflict, please install the pip dependencies above with --force-reinstall.

• For any other problems, please file a github issue.