Add audio to llm

[ribenamaplesyrup]

Would be great to add specialised models like text-to-speech (gpt-4o-mini-tts) to adulib so I can monitor ongoing costs of generating speech. I've used them before through the client interface as documented but not sure how this would work in adulib.

[lukastk]

Yes that is possible. There are submodules for both speech and transcripts in litellm:

• https://docs.litellm.ai/docs/audio_transcription
• https://docs.litellm.ai/docs/text_to_speech

So it would be fairly straightforward to add them to adulib.

You can also use litellm directly of course. The main reason why I've gone through the trouble of creating a wrapper around litellm in adulib.llm is that I wanted to include automatic caching and call logging.

[ribenamaplesyrup]

Cool, just tried this. Although what I'm trying to do is use get_total_costs() after generating speech. Not sure if its possible to do this without integrating into adulib.

[lukastk]

You just tried it just using litellm?

Yes that would not link up with adulib automatically. The requests would neither be cached nor call logged (and therefore cost logged)

[ribenamaplesyrup]

Yes I tried with litellm like this:

def text_to_speech(
    text: str,
    voice: str = "nova",
    model: str = "gpt-4o-mini-tts",
    out_path: Union[str, Path] = None,
    instructions: str = "Speak in a calm, clear, and informative tone."
) -> Path:
    client = OpenAI()
    data_dir = get_data_dir()
    data_dir.mkdir(parents=True, exist_ok=True)

    if out_path is None:
        out_path = data_dir / "speech.mp3"
    else:
        out_path = Path(out_path)
        if not out_path.is_absolute():
            out_path = data_dir / out_path

    response = speech(
        model=model,
        voice=voice,
        input=text,
        instructions=instructions
    ) 
    response.stream_to_file(out_path)

    return out_path

[lukastk]

can you paste response.model_dump() here. I want to see if it returns costing