Consider offloading LLM call retrying

[eb8680]

Currently handlers.llm.providers includes a somewhat half-baked handler for retrying failed Template calls. It's not clear that what we have implemented is semantically correct, and it's certainly not full-featured. Rather than handling Template.__apply__ with a catch-all RetryHandler, we should use LiteLLM's built-in functionality for retrying failed API calls, and handle other failure modes (e.g. failed tool calls or structured outputs) separately.

[datvo06]

@eb8680 If we use the LiteLLM, we would only handle completion errors and not validation errors (happening at the decoding time, for example), right?

[eb8680]

Probably, although LiteLLM may do some of that as well - it seems to have a built-in setting for validating outputs against a JSON schema, though I don't know what happens when it fails. For things like function synthesis where validation logic lives outside a Pydantic-derived JSON schema (e.g. calling a type checker #361), we'll still have to manage failures ourselves (#495).