Skip to content

Enable compatibility with OpenAI-compatible APIs#65

Open
thuanha-groove wants to merge 2 commits into
nraiden:mainfrom
thuanha-groove:openai-like
Open

Enable compatibility with OpenAI-compatible APIs#65
thuanha-groove wants to merge 2 commits into
nraiden:mainfrom
thuanha-groove:openai-like

Conversation

@thuanha-groove

@thuanha-groove thuanha-groove commented Nov 5, 2024

Copy link
Copy Markdown

This PR enables compatibility with OpenAI-compatible APIs. It also adds the missing nodemon dependency and an ignore rule for locally generated data.

I have tested with:

  • LM Studio
  • OpenRouter
  • Jan

@nraiden

nraiden commented Nov 8, 2024

Copy link
Copy Markdown
Owner
  • aiming to move all of inference outside of the main api
  • to have langsmith-like eval ops that can also trace back to which architecture module (and which architecture) called the inference
  • prompts to be at ^ abstraction level , to run benchmarks in batch against prompts

@EddieBSN

Copy link
Copy Markdown

I tried it with JAN but i dont seem to get it working. A tutorial might be good

@kntjspr

kntjspr commented Nov 29, 2024

Copy link
Copy Markdown

It's working on OpenRouter but it doesn't work on other OpenAI compatible api like https://glhf.chat/.
It returns 'Failed validation: model must begin with valid prefix (eg. hf:)',

I assume this is also the same case on any huggingface models.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants