steve/llama-swap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
245 Commits 3 Branches 175 Tags
78b2bc3dbc98b9ffcc8f0a8916f757f7f16ff73d
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Benson WongandGitHub 78b2bc3dbc add toggle to hide/show unlisted models (#187) 2025-07-02 16:14:20 -07:00
Benson Wong 6a058e4191 Change fsnotify to watch config directory instead of file 
The fsnotify library suggests watching a directory and checking that the
name matches the configuration file.
 2025-07-02 10:23:52 -07:00
Benson WongandGitHub 1921e570d7 Add Event Bus (#184) 
Major internal refactor to use an event bus to pass event/messages along. These changes are largely invisible user facing but sets up internal design for real time stats and information.

- `--watch-config` logic refactored for events
- remove multiple SSE api endpoints, replaced with /api/events
- keep all functionality essentially the same
- UI/backend sync is in near real time now
 2025-07-01 22:17:35 -07:00
Benson WongandGitHub c867a6c9a2 Add name and description to v1/models list (#179) 
* Add support for name and description in v1/models list
* add configuration example for name and description
 2025-06-30 23:02:44 -07:00
Benson WongandGitHub fefd14903d improve log display and add a small stats table in ui (#178) 2025-06-25 12:27:49 -07:00
Benson WongandGitHub 717d64e336 update GUI image in README [skip ci] 2025-06-24 10:38:28 -07:00
Benson WongandGitHub 285191e655 Various UI improvements (#176) 
* add retry/backoff to reconnecting log streams
* update favicons
 2025-06-23 16:17:21 -07:00
Benson WongandGitHub 4236cec03a Add Filters to Model Configuration (#174) 
llama-swap can strip specific keys in JSON requests. This is useful for removing the ability for clients to set sampling parameters like temperature, top_k, top_p, etc.
 2025-06-23 10:52:29 -07:00
Benson WongandGitHub a6b2e930d8 Update README.md [skip ci] 2025-06-18 11:47:08 -07:00
Benson WongandGitHub 9e02c22ff8 stopCmd should use same environment as p.cmd.Env (#171, #172) 2025-06-18 11:36:59 -07:00
Benson Wong 0bdbf2fdc1 fix more goreleaser deprecation warnings [skip ci] 2025-06-18 11:15:12 -07:00
Benson WongandGitHub 49035e2e8e Append custom env vars instead of replace in Process (#171) 
Append custom env vars instead of replace in Process (#168, #169)

PR #162 refactored the default configuration code. This
introduced a subtle bug where `env` became `[]string{}` instead of the
default of `nil`.

In golang, `exec.Cmd.Env == nil` means to use the "current process's
environment". By setting it to `[]string{}` as a default the Process's
environment was emptied out which caused an array of strange and
difficult to troubleshoot behaviour. See issues #168 and #169

This commit changes the behaviour to append model configured environment
variables to the default list rather than replace them.
 2025-06-18 11:09:13 -07:00
Benson Wong 9963ae18bf fix? deprecation warning in .goreleaser.yaml [skip-ci] 2025-06-18 07:49:33 -07:00
Benson Wong 2ae48c713b add debug output for start command 2025-06-18 07:43:23 -07:00
Benson Wong 54c519e365 update Makefile to install ui deps 2025-06-17 09:54:01 -07:00
Benson WongandGitHub 3fce9ee0e9 Update README.md [skip ci] 2025-06-17 09:53:22 -07:00
Benson WongandGitHub 5899ae7966 Update README.md [skip ci] 2025-06-17 09:52:47 -07:00
Benson Wong 591a9cdf4d update release.yml 2025-06-16 16:50:25 -07:00
Benson WongandGitHub 9a3c656738 New UI (#157, #164) 
- Add a react UI to replace the plain HTML one. 
- Serve as a foundation for better GUI interactions
 2025-06-16 16:45:19 -07:00
Benson WongandGitHub 75015f82ea fix bug caused by macro replacement order (#166) 
User defined macros should be applied before checking for ${PORT} constraint in model.cmd and model.proxy.
 2025-06-16 15:32:09 -07:00
Benson WongandGitHub 4fa12a429c Refactor all default config values into config.go (#162) 
- Move all default values into one place.
- Update tests to be more cross platform
 2025-06-15 12:32:00 -07:00
Benson WongandGitHub 2dc0ca0663 improve llama-swap upstream process recovery and restarts (#155) 
Refactor internal upstream process life cycle management to recover better from unexpected situations. With this change llama-swap should never need to be restarted due to a crashed upstream child process.  The `StateFailed` state was removed in favour of always trying to start/restart a process.
 2025-06-05 16:24:55 -07:00
Benson WongandGitHub 4d02ccd26a Update README.md [skip ci] 2025-05-30 09:38:45 -07:00
Benson Wong dfd47eeac4 Readme updates [skip ci] 2025-05-30 09:19:08 -07:00
Benson WongandGitHub 1ac6499c08 Add macros to Configuration schema (#149) 
* Add macros to Configuration schema
* update docs
 2025-05-29 21:51:25 -07:00
Benson Wong 25f3dc25e7 small doc update [skip ci] 2025-05-26 16:03:27 -07:00
Benson Wong 8422e4e6a1 move some docs to the wiki [no-ci] 2025-05-26 15:46:08 -07:00
Benson Wong 02ee29d881 increase default healthCheckTimeout to 120s 2025-05-26 09:57:53 -07:00
Benson WongandGitHub b2a891f8f4 Disable building of intel container until it's fixed upstream 2025-05-23 22:54:43 -07:00
Benson Wong 02aee4e86d remove noisy debug print message 2025-05-20 10:43:10 -07:00
Benson Wong f45896d395 add guard to avoid unnecessary logic in Process.Shutdown 2025-05-20 10:43:09 -07:00
Benson WongandGitHub b83a5fa291 make Failed stated recoverable (#137) 
A process in the failed state can transition to stopped either by calling /unload or swapping to another model.
 2025-05-16 19:54:44 -07:00
Benson Wong 6e2ff28d59 improve cmdStop docs [no ci] 2025-05-16 13:52:04 -07:00
Benson WongandGitHub a8b81f2799 Add stopCmd for custom stopping instructions (#136) 
Allow configuration of how a model is stopped before swapping. Setting `cmdStop` in the configuration will override the default behaviour and enables better integration with other process/container managers like docker or podman.
 2025-05-16 13:48:42 -07:00
Benson Wong f9ee7156dc update configuration examples for multiline yaml commands #133 2025-05-16 11:45:39 -07:00
Benson WongandGitHub afc9aef058 Fix #133 SanitizeCommand removes comments (#134) 2025-05-15 15:28:50 -07:00
Benson WongandGitHub d7b390df74 Add GH Action for Testing on Windows (#132) 
* Add windows specific test changes
* Change the command line parsing library - Possible breaking changes for windows users!
 2025-05-14 21:51:53 -07:00
Benson Wong 5025c2f1f3 Add GH windows tests (not working yet) 2025-05-14 19:58:22 -07:00
Benson Wong e3a0b013c1 add content length test for #131 2025-05-14 19:50:01 -07:00
Benson WongandGitHub 8ada72eb57 Update issue templates 2025-05-14 16:36:32 -07:00
Benson Wong 2441b383d3 Make checking for process killed status more robust 2025-05-14 16:26:56 -07:00
Benson WongandGitHub 25f251699c Prevent StateFailed after SIGKILL (#129) 
Closes #125
 2025-05-14 10:47:35 -07:00
Benson WongandGitHub 7f37bcc6eb Improve testing around using SIGKILL (#127) 
* Add test for SIGKILL of process
* silent TestProxyManager_RunningEndpoint debug output
* Ref #125
 2025-05-13 21:21:52 -07:00
Benson WongandGitHub 519c3a4d22 Change /unload to not wait for inflight requests (#125) 
Sometimes upstreams can accept HTTP but never respond causing requests
to build up waiting for a response. This can block Process.Stop() as
that waits for inflight requests to finish. This change refactors the
code to not wait when attempting to shutdown the process.
 2025-05-13 11:39:19 -07:00
Benson WongandGitHub 9dc4bcb46c Add a concurrency limit to Process.ProxyRequest (#123) 2025-05-12 18:12:52 -07:00
Benson Wong cb876c143b update example config 2025-05-12 10:20:18 -07:00
Benson WongandGitHub 5c5a5da664 Update README.md 
removed extra section.
 2025-05-06 06:59:15 -07:00
Benson WongandGitHub aa9ef59aa5 Create .coderabbit.yaml 2025-05-05 19:47:23 -07:00
Benson WongandGitHub 09e52c0500 Automatic Port Numbers (#105) 
Add automatic port numbers assignment in configuration file. The string `${PORT}` will be substituted in model.cmd and model.proxy for an actual port number. This also allows model.proxy to be omitted from the configuration.
 2025-05-05 17:07:43 -07:00
Benson WongandGitHub ca9063ffbe ensure aliases are unique (#116) 2025-05-05 15:34:18 -07:00
Benson WongandGitHub 21d7973d11 Improve content-length handling (#115) 
ref: See #114

* Improve content-length handling
- Content length was not always being sent
- Add tests for content-length
 2025-05-05 10:46:26 -07:00
Benson Wong 27465fe053 bug fix with missing early return statements fix #112 2025-05-05 09:32:44 -07:00
Benson WongandGitHub 9667989727 Disabling intel container build since it's been broken for weeks. 2025-05-04 21:39:42 -07:00
Benson WongandGitHub d9a1ddea0d Truncate web logs to 100K characters (#111) 
* set log limit to 100K in browser
 2025-05-02 23:43:21 -07:00
Benson Wong e7ab024ca0 small locking optimization 2025-05-02 23:18:07 -07:00
Benson WongandGitHub 448ccae959 Introduce Groups Feature (#107) 
Groups allows more control over swapping behaviour when a model is requested. The new groups feature provides three ways to control swapping: within the group, swapping out other groups or keep the models in the group loaded persistently (never swapped out). 

Closes #96, #99 and #106.
 2025-05-02 22:35:38 -07:00
Benson WongandGitHub ec0348e431 Reduce stale time for issues 2025-04-29 21:16:34 -07:00
Benson WongandGitHub 06eda7f591 tag all process logs with its ID (#103) 
Makes identifying Process of log messages easier
 2025-04-25 12:58:25 -07:00
Benson WongandGitHub 5fad24c16f Make checkHealthTimeout Interruptable during startup (#102) 
interrupt and exit Process.start() early if the upstream process exits prematurely or unexpectedly.
 2025-04-24 14:39:33 -07:00
Benson Wong 8404244fab Moderate security update for golang/x/net -> v0.38.0 2025-04-24 09:58:40 -07:00
Benson Wong 712cd01081 fix confusing INFO message [no ci] 2025-04-24 09:56:20 -07:00
Benson Wong 1f7aa359b1 Update header image 
AI has finally made my dreams of llamas in funny clothing and stuck in
a claw machine waiting to be picked come true!
 2025-04-23 13:02:12 -07:00
Benson WongandGitHub b138d6cf25 fix starhistory in README 2025-04-15 20:23:46 -07:00
Benson WongandGitHub fb7c808082 add timing for Process start, stop, total request time (#91) 2025-04-14 14:34:59 -07:00
Benson Wong a7e640b0f7 add aider example 2025-04-07 12:37:14 -07:00
Benson Wong 593604dfdc Show proxy and upstream logs in separate columns in logs UI 2025-04-05 10:36:54 -07:00
Benson WongandGitHub b8f888f864 Logging Improvements (#88) 
This change revamps the internal logging architecture to be more flexible and descriptive. Previously all logs from both llama-swap and upstream services were mixed together. This makes it harder to troubleshoot and identify problems. This PR adds these new endpoints: 

- `/logs/stream/proxy` - just llama-swap's logs
- `/logs/stream/upstream` - stdout output from the upstream server
 2025-04-04 21:01:33 -07:00
Benson Wong 192b2ae621 Remove no longer needed test 2025-04-04 14:46:01 -07:00
Benson Wong b7f8cb5094 Limit Access-Control-Allow-Origin to OPTIONS preflight requests #85 2025-04-04 14:44:35 -07:00
Benson WongandGitHub a23da6eb57 Sanitize CORS headers (#85) 
Add sanitation step for `Access-Control-Allow-Headers` when echoing back user supplied headers
 2025-04-01 08:43:53 -07:00
Benson WongandGitHub 84e2c07a7e Refactor wildcard out of CORS headers (#81) 
Changes to CORS functionality: 

- `Access-Control-Allow-Origin: *` is set for all requests 
- for pre-flight OPTIONS requests
  - specify methods: `Access-Control-Allow-Methods: GET, POST, PUT, PATCH, DELETE, OPTIONS`
  - if the client sent `Access-Control-Request-Headers` then echo back the same value in `Access-Control-Allow-Headers`. If no `Access-Control-Request-Headers` were sent, then send back a default set
  - set `Access-Control-Max-Age: 86400` to that may improve performance 
- Add CORS tests to the proxy-manager
 2025-03-25 15:24:43 -07:00
Benson Wong 680af28bcc Allow very permissive CORS headers (#77) 2025-03-20 15:50:21 -07:00
Benson Wong d94db42ffe fix bug checking incorrect error 2025-03-20 15:49:36 -07:00
Benson WongandGitHub 93cd83c55c add override for windows (#76) 2025-03-20 13:23:04 -07:00
Benson WongandGitHub 5565fca3ac add some badges to README 2025-03-19 11:25:06 -07:00
Benson WongandGitHub d625ab8d92 Refactor process state management (#70) (#73) 
* add isValidStateTransition helper function
* Replace Process.setState() with Process.swapState()
* Refactor locking logic in Process
 2025-03-15 17:14:03 -07:00
Benson Wong a3f82c140b tidy up config examples in README 2025-03-15 10:36:45 -07:00
Benson WongandGitHub 5c97299e7b Add support for sending a custom model name to upstream (#69) (#71) 
* add test for splitRequestedModel()
* Add `useModelName` parameter to model configuration
* add docs to README
 2025-03-14 21:07:52 -07:00
Benson Wong 671c1a5a7b update deps 2025-03-13 14:00:15 -07:00
Benson Wong 52c0196e0f clean up feature list in readme 2025-03-13 13:55:20 -07:00
Benson WongandGitHub 3201a68a04 Add /v1/audio/transcriptions support (#41) 
* add support for /v1/audio/transcriptions
 2025-03-13 13:49:39 -07:00
Benson Wong 60355bf74a fix some potentially confusing Process.start() comment 2025-03-11 11:00:45 -07:00
Benson WongandGitHub 9b2ed244e2 Improve Continuous integration and fix concurrency bugs (#66) 
- improvements to the continuous GH actions
- fix edge case concurrency bugs with Process.start() and state transitions discovered setting up CI.
 2025-03-11 10:39:14 -07:00
Benson Wong eeb72297f7 add first version of CI for go 2025-03-11 08:45:56 -07:00
Benson Wong eabfe70cc6 add GH action to close inactive issues 2025-03-09 19:51:48 -07:00
Benson Wong 29cd98878d better container build logic when upstream containers do not exist 2025-03-09 13:02:06 -07:00
Benson WongandGitHub b3d331da0d Properly strip profile name slug from models fixes (#62) 
The profile slug in a model name, `profile:model`, is specific to
llama-swap. This strips `profile:` out of the model name request so
upstreams that expect just `model` work and do not require knowing about
the profile slug.
 2025-03-09 12:41:52 -07:00
Benson Wong 62275e078d add examples to restart on config change #59 2025-03-06 10:50:29 -08:00
Benson Wong 88916059e1 add /unload to docs 2025-03-03 10:44:16 -08:00
Benson Wong 082d5d0fc5 Add /unload endpoint (#58) to unload all currently running models 2025-03-03 10:33:36 -08:00
Benson Wong 53338938bd increase health check to a minimum of 5 seconds 2025-03-03 10:04:08 -08:00
Benson WongandGitHub af653347ae Update README.md w/ starhistory graph 2025-02-27 16:43:34 -08:00
Benson Wong 1e25b44a06 add workflow_dispatch to release action 2025-02-18 17:27:43 -08:00
Benson Wong 0815bb4cc3 Add windows to goreleaser #54 2025-02-18 17:26:43 -08:00
Benson WongandGitHub 24089d2d9c remove "no musa container" note from README 2025-02-18 16:38:48 -08:00
Benson WongandGitHub ebabe55ff3 Delete untagged packages after build and push (#55) 2025-02-18 10:32:32 -08:00
Benson Wong 41a338297c deletion of untagged containers happen after build-and-push 2025-02-18 10:11:59 -08:00
Benson Wong 7e3353efeb add action step to remove untagged containers 2025-02-18 10:08:41 -08:00
Benson Wong 4ed58fb173 update container build action 2025-02-18 09:59:06 -08:00
Benson Wong f5a2be698d revert package src until new ggml-org has them 2025-02-15 18:23:58 -08:00