on How much gpu do i need to run a 90b model · c/localllama · 2 pts · 1yYou may try Harbor. The description claims to provide an OpenAI-compatible API.
on How much gpu do i need to run a 90b model · c/localllama · 3 pts · 1yIn case of LLM's you should look at AirLLM. I suppose there is no conviniet integrations to local chat tools, but issue at Ollama already started.
You may try Harbor. The description claims to provide an OpenAI-compatible API.
In case of LLM's you should look at AirLLM. I suppose there is no conviniet integrations to local chat tools, but issue at Ollama already started.