Compare commits

...

3 Commits

Author SHA1 Message Date
Benson Wong 13d4552edc Add FreeBSD/amd64 to auto built releases (#51) 2025-02-13 16:44:31 -08:00
Benson Wong 6667e307a2 Update README.md 2025-02-08 10:28:35 -08:00
Benson Wong 7ac446e6a9 Update README.md 2025-02-08 10:26:11 -08:00
3 changed files with 10 additions and 6 deletions
+5 -1
View File
@@ -6,6 +6,10 @@ builds:
goos:
- linux
- darwin
- freebsd
goarch:
- amd64
- arm64
- arm64
ignore:
- goos: freebsd
goarch: arm64
+1 -1
View File
@@ -60,4 +60,4 @@ release:
git tag "$$new_tag";
# Phony targets
.PHONY: all clean osx linux
.PHONY: all clean mac linux
+4 -4
View File
@@ -1,8 +1,6 @@
# llama-swap
![llama-swap header image](header.jpeg)
# Introduction
# llama-swap
llama-swap is a light weight, transparent proxy server that provides automatic model swapping to llama.cpp's server.
Written in golang, it is very easy to install (single binary with no dependancies) and configure (single yaml file).
@@ -15,7 +13,9 @@ When a request is made to an OpenAI compatible endpoint, lama-swap will extract
In the most basic configuration llama-swap handles one model at a time. For more advanced use cases, the `profiles` feature can load multiple models at the same time. You have complete control over how your system resources are used.
## Do I need to use llama.cpp's server (llama-server)?
Any OpenAI compatible server would work. llama-swap was originally designed for llama-server and it is the best supported. For Python based inference servers like vllm or tabbyAPI it is recommended to run them via podman. This provides clean environment isolation as well as responding correctly to `SIGTERM` signals to shutdown.
Any OpenAI compatible server would work. llama-swap was originally designed for llama-server and it is the best supported.
For Python based inference servers like vllm or tabbyAPI it is recommended to run them via podman or docker. This provides clean environment isolation as well as responding correctly to `SIGTERM` signals to shutdown.
## Features: