[usage] Tensorize Lora Adapters

[paulliwog]

We are starting to look into implementing tensorizer with VLLM, but didn't see any specific support for Lora Adapters. We use a rank of up to 128 with adapters trained on llama3-70B which results in ~3.5GB safetensors file per adapter. Is there support for serializing and deploying these from tensorizer?

[sangstar]

Hi @paulliwog

Work to integrate Tensorizer with LoRA adapters was largely done with help from @Yard1 . Let me know if you have any trouble using it. I'm unaware of any issues with using the two at this time. An example of using both Tensorizer with LoRA adapters can be found in one of the Tensorizer tests.

[amromidhat]

Hi @sangstar

Is there any way to serialize the LoRa files itself (with encryption), and deserialize/load it at runtime?

[sangstar]

@amromidhat

As LoRA adapters are nothing more lower-rank matrices and thereby a collection of tensors, this should theoretically be serializable and encryptable with tensorizer. This is an underexplored area for our team, but we welcome you to try serializing a LoRA matrix with encryption, and letting us know if there are any issues with that.