Fixed typo

Author: orangetin

inference/README.md

@@ -9,17 +9,17 @@ This directory contains code for OpenChatKit's inference.
 - `--temperature`: temperature for the LM. Default: `0.6`
 - `--top-k`: top-k for the LM. Default: `40`
 - `--retrieval`: augment queries with context from the retrieval index. Default `False`
-- `-g` `--gpu-vram`: GPU ID and vRAM to allocate to loading the model, separated by a `:` in the format `ID:RAM` where ID is the CUDA ID and RAM is in GiB. `gpu-id` must be present in this list to avoid errors. Accepts multiple values, for example, `-g ID_0:RAM_0 ID_1:RAM_1 ID_N:RAM_N`
+- `-g` `--gpu-vram`: GPU ID and VRAM to allocate to loading the model, separated by a `:` in the format `ID:RAM` where ID is the CUDA ID and RAM is in GiB. `gpu-id` must be present in this list to avoid errors. Accepts multiple values, for example, `-g ID_0:RAM_0 ID_1:RAM_1 ID_N:RAM_N`
 - `-r` `--cpu-ram`: CPU RAM overflow allocation for loading the model. Optional, and only used if the model does not fit onto the GPUs given.
 
 ## Hardware requirements for inference
-The GPT-NeoXT-Chat-Base-20B model requires at least 41GB of free vRAM. Used vRAM also goes up by ~100-200 MB per prompt. 
+The GPT-NeoXT-Chat-Base-20B model requires at least 41GB of free VRAM. Used VRAM also goes up by ~100-200 MB per prompt. 
 
 - A **minimum of 80 GB is recommended** 
 
-- A **minimum of 48 GB in vRAM is recommended** for fast responses.
+- A **minimum of 48 GB in VRAM is recommended** for fast responses.
 
-If you'd like to run inference on a GPU with <48 GB vRAM, refer to this section on [running on consumer hardware](#running-on-consumer-hardware).
+If you'd like to run inference on a GPU with <48 GB VRAM, refer to this section on [running on consumer hardware](#running-on-consumer-hardware).
 
 By default, inference uses only CUDA Device 0.
 
@@ -28,19 +28,19 @@ By default, inference uses only CUDA Device 0.
 ## Running on multiple GPUs
 Add the argument 
 
-```-g ID0:MAX_vRAM ID1:MAX_vRAM ID2:MAX_vRAM ...``` 
+```-g ID0:MAX_VRAM ID1:MAX_VRAM ID2:MAX_VRAM ...``` 
 
-where IDx is the CUDA ID of the device and MAX_vRAM is the amount of vRAM you'd like to allocate to the device.
+where IDx is the CUDA ID of the device and MAX_VRAM is the amount of VRAM you'd like to allocate to the device.
 
 For example, if you are running this on 4x 48 GB GPUs and want to distribute the model across all devices, add ```-g 0:10 1:12 2:12 3:12 4:12```. In this example, the first device gets loaded to a max of 10 GiB while the others are loaded with a max of 12 GiB.
 
-How it works: The model fills up the max available vRAM on the first device passed and then overflows into the next until the whole model is loaded.
+How it works: The model fills up the max available VRAM on the first device passed and then overflows into the next until the whole model is loaded.
 
-**IMPORTANT: This MAX_vRAM is only for loading the model. It does not account for the additional inputs that are added to the device. It is recommended to set the MAX_vRAM to be at least 1 or 2 GiB less than the max available vRAM on each device, and at least 3GiB less than the max available vRAM on the primary device (set by `gpu-id` default=0).**
+**IMPORTANT: This MAX_VRAM is only for loading the model. It does not account for the additional inputs that are added to the device. It is recommended to set the MAX_VRAM to be at least 1 or 2 GiB less than the max available VRAM on each device, and at least 3GiB less than the max available VRAM on the primary device (set by `gpu-id` default=0).**
 
-**Decrease MAX_vRAM if you run into CUDA OOM. This happens because each input takes up additional space on the device.**
+**Decrease MAX_VRAM if you run into CUDA OOM. This happens because each input takes up additional space on the device.**
 
-**NOTE: Total MAX_vRAM across all devices must be > size of the model in GB. If not, `bot.py` automatically offloads the rest of the model to RAM and disk. It will use up all available RAM. To allocate a specified amount of RAM: [refer to this section on running on consumer hardware](#running-on-consumer-hardware).**
+**NOTE: Total MAX_VRAM across all devices must be > size of the model in GB. If not, `bot.py` automatically offloads the rest of the model to RAM and disk. It will use up all available RAM. To allocate a specified amount of RAM: [refer to this section on running on consumer hardware](#running-on-consumer-hardware).**
 
 ## Running on specific GPUs
 If you have multiple GPUs but would only like to use a specific device(s), [use the same steps as in this section on running on multiple devices](#running-on-multiple-gpus) and only specify the devices you'd like to use. 
@@ -53,16 +53,16 @@ Also, if needed, add the argument `--gpu-id ID` where ID is the CUDA ID of the d
 
 
 ## Running on consumer hardware
-If you have multiple GPUs, each <48 GB vRAM, [the steps mentioned in this section on running on multiple GPUs](#running-on-multiple-gpus) still apply, unless, any of these apply:
-- Running on just 1x GPU with <48 GB vRAM,
-- <48 GB vRAM combined across multiple GPUs
+If you have multiple GPUs, each <48 GB VRAM, [the steps mentioned in this section on running on multiple GPUs](#running-on-multiple-gpus) still apply, unless, any of these apply:
+- Running on just 1x GPU with <48 GB VRAM,
+- <48 GB VRAM combined across multiple GPUs
 - Running into Out-Of-Memory (OOM) issues
 
 In which case, add the flag `-r CPU_RAM` where CPU_RAM is the maximum amount of RAM you'd like to allocate to loading model. Note: This significantly reduces inference speeds. 
 
 The model will load without specifying `-r`, however, it is not recommended because it will allocate all available RAM to the model. To limit how much RAM the model can use, add `-r`.
 
-If the total vRAM + CPU_RAM < the size of the model in GiB, the rest of the model will be offloaded to a folder "offload" at the root of the directory. Note: This significantly reduces inference speeds.
+If the total VRAM + CPU_RAM < the size of the model in GiB, the rest of the model will be offloaded to a folder "offload" at the root of the directory. Note: This significantly reduces inference speeds.
 
 - Example: `-g 0:12 -r 20` will first load up to 12 GiB of the model into the CUDA device 0, then load up to 20 GiB into RAM, and load the rest into the "offload" directory.
 

inference/bot.py

@@ -22,13 +22,13 @@ class ChatModel:
     def __init__(self, model_name, gpu_id, max_memory):
         device = torch.device('cuda', gpu_id)   # TODO: allow sending to cpu
 
-        # recommended default for devices with > 40 GB vRAM
+        # recommended default for devices with > 40 GB VRAM
         # load model onto one device
         if max_memory is None:
             self._model = AutoModelForCausalLM.from_pretrained(
                 model_name, torch_dtype=torch.float16, device_map="auto")
             self._model.to(device)
-        # load the model with the given max_memory config (for devices with insufficient vRAM or multi-gpu)
+        # load the model with the given max_memory config (for devices with insufficient VRAM or multi-gpu)
         else:
             config = AutoConfig.from_pretrained(model_name)
             # load empty weights
@@ -203,7 +203,7 @@ def main():
         '-g',
         '--gpu-vram',
         action='store',
-        help='max vRAM to allocate per GPU',
+        help='max VRAM to allocate per GPU',
         nargs='+',
         required=False,
     )