steve/llama-swap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
292 Commits 3 Branches 175 Tags
831a90d3b0f5aa0fda842e79e6f5d213b5a766cc
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Benson WongandGitHub 831a90d3b0 Add different timeout scenarios to Process.checkHealthEndpoint #276 (#278) 
- add a TCP connection timeout of 500ms
- increase HTTP client timeout to 5000ms

In this new behaviour the upstream has 500ms to accept a tcp connection
and 5000ms to respond to the HTTP request.
 2025-08-28 22:03:14 -07:00
Benson Wong 52b329f7bc Fix #277 race condition in ProcessGroup.ProxyRequest when swap=true 2025-08-28 21:38:40 -07:00
Benson WongandGitHub 57803fd3aa Support llama-server's /infill endpoint (#272) 
Add support for llama-server's /infill endpoint and metrics gathering on the Activities page.
 2025-08-27 08:36:05 -07:00
Benson Wong c55d0cc842 Add docs for model.concurrencyLimit #263 [skip ci] 2025-08-22 16:08:37 -07:00
Benson WongandGitHub 7acbaf4712 Add connection status indicator in UI (#260) 
* show connection status as icon in UI title
* make connection status event driven
 2025-08-20 13:58:24 -07:00
Benson Wong fcc5ad135a UI: Allow editing of title (#246) 
- make <h1> title contentEditable
- title setting persists across reloads in localStorage
 2025-08-17 09:42:06 -07:00
Benson Wong 305e5a0031 improve example config [skip ci] 2025-08-17 09:19:04 -07:00
Benson WongandGitHub 04fc67354a Improve Activity event handling in the UI (#254) 
Improve Activity event handling in the UI

- fixes #252 found that the Activity page showed activity inconsistent
  with /api/metrics
- Change data structure for event metrics to array.
- Add Event stream connections status indicator
 2025-08-15 21:44:08 -07:00
Benson WongandGitHub 4662cf7699 add 'unconfirmed bug' as default label in bug-report.md 2025-08-15 15:38:12 -07:00
Benson WongandGitHub 5dc6b3e6d9 Add barebones but working implementation of model preload (#209, #235) 
Add barebones but working implementation of model preload

* add config test for Preload hook
* improve TestProxyManager_StartupHooks
* docs for new hook configuration
* add a .dev to .gitignore
 2025-08-14 10:27:28 -07:00
Benson WongandGitHub 74c69f39ef Add prompt processing metrics (#250) 
- capture prompt processing metrics
- display prompt processing metrics on UI Activity page
 2025-08-14 10:02:16 -07:00
Benson WongandGitHub a186318892 Update Readme, Add screenshot for Activities page [skip ci] 2025-08-08 13:39:46 -07:00
Benson WongandGitHub c4e4d5e1e9 Update Readme UI Screenshot [skip ci] 2025-08-08 13:33:47 -07:00
Benson Wong 7985e94ba4 add tokens processed to ui models page 2025-08-08 13:28:39 -07:00
Benson WongandGitHub 74556c3a36 Update bug-report.md [skip ci] 2025-08-08 09:52:05 -07:00
Benson Wong 5c381e4b30 Add gofmt linting to ci 2025-08-07 20:29:18 -07:00
Benson WongandGitHub 10569ed546 Fix model alias usage in upstream path (#230) 
Model alias values are not properly resolved and work in upstream/ path.

Related to #229.
 2025-08-07 20:16:56 -07:00
Benson WongandGitHub 5b10b3c23f UI Tweaks (#228) 
* sort model names in UI

* add toggle to show model id/name on UI model page
 2025-08-07 11:07:03 -07:00
Benson Wong 45ea792a3a Fix UI panel not saving position correctly 2025-08-06 14:02:22 -07:00
Benson Wong 1bc2802353 fix panels not saving sizing state 2025-08-06 14:00:21 -07:00
Benson WongandGitHub 701476c0c4 Update README.md - remove contributor block [skip ci] 
Contributor information available on the Github page's sidebar. Redundant.
 2025-08-06 11:11:47 -07:00
Benson WongandGitHub 558a72de17 UI Improvements (#219) 
- use react-resizable-panels for UI
- improve icons for buttons
- improve mobile layout with drag/resize panels
 2025-08-03 17:49:13 -07:00
Benson WongandGitHub 574fdfabb4 UI improvements (#213) 
* use two column for logs view on wider screens

* hide log controls when panel is minimized
 2025-07-31 11:59:21 -07:00
Benson WongandGitHub 5172cb2e12 Update docs in Readme [skip ci] 2025-07-30 11:51:14 -07:00
Benson WongandGitHub 5672cb03fd Update github actions for notifying homebrew build (#212) 
Combine homebrew-llama-swap event with the release action
 2025-07-30 11:29:03 -07:00
Benson WongandGitHub 0f583163f7 add /health (#211) 2025-07-30 10:37:10 -07:00
Benson WongandGitHub 7905fa9ea3 Update trigger-homebrew-update.yml [skip ci] 2025-07-30 10:13:49 -07:00
Benson WongandGitHub fd50932dbc Decouple MetricsMiddleware from downstream handlers (#206) 
* Decouple MetricsMiddleware from downstream handlers

Remove ls-real-model-name optimization. Within proxyOAIHandler the
request body's bytes are required for various rewriting features
anyways. This negated any benefits from trying not to parse it twice.
 2025-07-27 10:36:06 -07:00
Benson Wong 8f2af26a41 fix stats on model page 2025-07-23 13:57:33 -07:00
Benson WongandGitHub 01d4838fb3 Fix token metrics parsing (#199) 
Fix #198

- use llama-server's `timings` info if available in response body
- send "-1" for token/sec when not able to accurately calculate
  performance
- optimize streaming body search for metrics information
 2025-07-22 23:10:14 -07:00
Benson WongandGitHub accd65294b add contributors to README [skip ci] 2025-07-21 23:16:48 -07:00
Benson WongandGitHub 7472a25864 Update README.md [skip ci] 
update screenshot for web UI
 2025-07-21 23:08:19 -07:00
Benson Wong cce0bc6aa1 add guard to ensure ls-real-model-name is set in context 2025-07-21 22:59:41 -07:00
Benson Wong 36e25125e8 UI tidy [skip ci] 2025-07-21 22:47:55 -07:00
Benson Wong 9a54273d15 Update UI with new Activity event stream from #195 
- use new metrics data instead of log parsing
- auto-start events connection to server, improves responsiveness
- remove unnecessary libraries and code
 2025-07-21 22:42:30 -07:00
Benson Wong 307e619521 remove old eventsources from UI 2025-07-19 15:36:40 -07:00
Benson WongandGitHub 6299c1b874 Fix High CPU (#189) 
* vendor in kelindar/event lib and refactor to remove time.Ticker
 2025-07-15 18:04:30 -07:00
Benson WongandGitHub 78b2bc3dbc add toggle to hide/show unlisted models (#187) 2025-07-02 16:14:20 -07:00
Benson Wong 6a058e4191 Change fsnotify to watch config directory instead of file 
The fsnotify library suggests watching a directory and checking that the
name matches the configuration file.
 2025-07-02 10:23:52 -07:00
Benson WongandGitHub 1921e570d7 Add Event Bus (#184) 
Major internal refactor to use an event bus to pass event/messages along. These changes are largely invisible user facing but sets up internal design for real time stats and information.

- `--watch-config` logic refactored for events
- remove multiple SSE api endpoints, replaced with /api/events
- keep all functionality essentially the same
- UI/backend sync is in near real time now
 2025-07-01 22:17:35 -07:00
Benson WongandGitHub c867a6c9a2 Add name and description to v1/models list (#179) 
* Add support for name and description in v1/models list
* add configuration example for name and description
 2025-06-30 23:02:44 -07:00
Benson WongandGitHub fefd14903d improve log display and add a small stats table in ui (#178) 2025-06-25 12:27:49 -07:00
Benson WongandGitHub 717d64e336 update GUI image in README [skip ci] 2025-06-24 10:38:28 -07:00
Benson WongandGitHub 285191e655 Various UI improvements (#176) 
* add retry/backoff to reconnecting log streams
* update favicons
 2025-06-23 16:17:21 -07:00
Benson WongandGitHub 4236cec03a Add Filters to Model Configuration (#174) 
llama-swap can strip specific keys in JSON requests. This is useful for removing the ability for clients to set sampling parameters like temperature, top_k, top_p, etc.
 2025-06-23 10:52:29 -07:00
Benson WongandGitHub a6b2e930d8 Update README.md [skip ci] 2025-06-18 11:47:08 -07:00
Benson WongandGitHub 9e02c22ff8 stopCmd should use same environment as p.cmd.Env (#171, #172) 2025-06-18 11:36:59 -07:00
Benson Wong 0bdbf2fdc1 fix more goreleaser deprecation warnings [skip ci] 2025-06-18 11:15:12 -07:00
Benson WongandGitHub 49035e2e8e Append custom env vars instead of replace in Process (#171) 
Append custom env vars instead of replace in Process (#168, #169)

PR #162 refactored the default configuration code. This
introduced a subtle bug where `env` became `[]string{}` instead of the
default of `nil`.

In golang, `exec.Cmd.Env == nil` means to use the "current process's
environment". By setting it to `[]string{}` as a default the Process's
environment was emptied out which caused an array of strange and
difficult to troubleshoot behaviour. See issues #168 and #169

This commit changes the behaviour to append model configured environment
variables to the default list rather than replace them.
 2025-06-18 11:09:13 -07:00
Benson Wong 9963ae18bf fix? deprecation warning in .goreleaser.yaml [skip-ci] 2025-06-18 07:49:33 -07:00
Benson Wong 2ae48c713b add debug output for start command 2025-06-18 07:43:23 -07:00
Benson Wong 54c519e365 update Makefile to install ui deps 2025-06-17 09:54:01 -07:00
Benson WongandGitHub 3fce9ee0e9 Update README.md [skip ci] 2025-06-17 09:53:22 -07:00
Benson WongandGitHub 5899ae7966 Update README.md [skip ci] 2025-06-17 09:52:47 -07:00
Benson Wong 591a9cdf4d update release.yml 2025-06-16 16:50:25 -07:00
Benson WongandGitHub 9a3c656738 New UI (#157, #164) 
- Add a react UI to replace the plain HTML one. 
- Serve as a foundation for better GUI interactions
 2025-06-16 16:45:19 -07:00
Benson WongandGitHub 75015f82ea fix bug caused by macro replacement order (#166) 
User defined macros should be applied before checking for ${PORT} constraint in model.cmd and model.proxy.
 2025-06-16 15:32:09 -07:00
Benson WongandGitHub 4fa12a429c Refactor all default config values into config.go (#162) 
- Move all default values into one place.
- Update tests to be more cross platform
 2025-06-15 12:32:00 -07:00
Benson WongandGitHub 2dc0ca0663 improve llama-swap upstream process recovery and restarts (#155) 
Refactor internal upstream process life cycle management to recover better from unexpected situations. With this change llama-swap should never need to be restarted due to a crashed upstream child process.  The `StateFailed` state was removed in favour of always trying to start/restart a process.
 2025-06-05 16:24:55 -07:00
Benson WongandGitHub 4d02ccd26a Update README.md [skip ci] 2025-05-30 09:38:45 -07:00
Benson Wong dfd47eeac4 Readme updates [skip ci] 2025-05-30 09:19:08 -07:00
Benson WongandGitHub 1ac6499c08 Add macros to Configuration schema (#149) 
* Add macros to Configuration schema
* update docs
 2025-05-29 21:51:25 -07:00
Benson Wong 25f3dc25e7 small doc update [skip ci] 2025-05-26 16:03:27 -07:00
Benson Wong 8422e4e6a1 move some docs to the wiki [no-ci] 2025-05-26 15:46:08 -07:00
Benson Wong 02ee29d881 increase default healthCheckTimeout to 120s 2025-05-26 09:57:53 -07:00
Benson WongandGitHub b2a891f8f4 Disable building of intel container until it's fixed upstream 2025-05-23 22:54:43 -07:00
Benson Wong 02aee4e86d remove noisy debug print message 2025-05-20 10:43:10 -07:00
Benson Wong f45896d395 add guard to avoid unnecessary logic in Process.Shutdown 2025-05-20 10:43:09 -07:00
Benson WongandGitHub b83a5fa291 make Failed stated recoverable (#137) 
A process in the failed state can transition to stopped either by calling /unload or swapping to another model.
 2025-05-16 19:54:44 -07:00
Benson Wong 6e2ff28d59 improve cmdStop docs [no ci] 2025-05-16 13:52:04 -07:00
Benson WongandGitHub a8b81f2799 Add stopCmd for custom stopping instructions (#136) 
Allow configuration of how a model is stopped before swapping. Setting `cmdStop` in the configuration will override the default behaviour and enables better integration with other process/container managers like docker or podman.
 2025-05-16 13:48:42 -07:00
Benson Wong f9ee7156dc update configuration examples for multiline yaml commands #133 2025-05-16 11:45:39 -07:00
Benson WongandGitHub afc9aef058 Fix #133 SanitizeCommand removes comments (#134) 2025-05-15 15:28:50 -07:00
Benson WongandGitHub d7b390df74 Add GH Action for Testing on Windows (#132) 
* Add windows specific test changes
* Change the command line parsing library - Possible breaking changes for windows users!
 2025-05-14 21:51:53 -07:00
Benson Wong 5025c2f1f3 Add GH windows tests (not working yet) 2025-05-14 19:58:22 -07:00
Benson Wong e3a0b013c1 add content length test for #131 2025-05-14 19:50:01 -07:00
Benson WongandGitHub 8ada72eb57 Update issue templates 2025-05-14 16:36:32 -07:00
Benson Wong 2441b383d3 Make checking for process killed status more robust 2025-05-14 16:26:56 -07:00
Benson WongandGitHub 25f251699c Prevent StateFailed after SIGKILL (#129) 
Closes #125
 2025-05-14 10:47:35 -07:00
Benson WongandGitHub 7f37bcc6eb Improve testing around using SIGKILL (#127) 
* Add test for SIGKILL of process
* silent TestProxyManager_RunningEndpoint debug output
* Ref #125
 2025-05-13 21:21:52 -07:00
Benson WongandGitHub 519c3a4d22 Change /unload to not wait for inflight requests (#125) 
Sometimes upstreams can accept HTTP but never respond causing requests
to build up waiting for a response. This can block Process.Stop() as
that waits for inflight requests to finish. This change refactors the
code to not wait when attempting to shutdown the process.
 2025-05-13 11:39:19 -07:00
Benson WongandGitHub 9dc4bcb46c Add a concurrency limit to Process.ProxyRequest (#123) 2025-05-12 18:12:52 -07:00
Benson Wong cb876c143b update example config 2025-05-12 10:20:18 -07:00
Benson WongandGitHub 5c5a5da664 Update README.md 
removed extra section.
 2025-05-06 06:59:15 -07:00
Benson WongandGitHub aa9ef59aa5 Create .coderabbit.yaml 2025-05-05 19:47:23 -07:00
Benson WongandGitHub 09e52c0500 Automatic Port Numbers (#105) 
Add automatic port numbers assignment in configuration file. The string `${PORT}` will be substituted in model.cmd and model.proxy for an actual port number. This also allows model.proxy to be omitted from the configuration.
 2025-05-05 17:07:43 -07:00
Benson WongandGitHub ca9063ffbe ensure aliases are unique (#116) 2025-05-05 15:34:18 -07:00
Benson WongandGitHub 21d7973d11 Improve content-length handling (#115) 
ref: See #114

* Improve content-length handling
- Content length was not always being sent
- Add tests for content-length
 2025-05-05 10:46:26 -07:00
Benson Wong 27465fe053 bug fix with missing early return statements fix #112 2025-05-05 09:32:44 -07:00
Benson WongandGitHub 9667989727 Disabling intel container build since it's been broken for weeks. 2025-05-04 21:39:42 -07:00
Benson WongandGitHub d9a1ddea0d Truncate web logs to 100K characters (#111) 
* set log limit to 100K in browser
 2025-05-02 23:43:21 -07:00
Benson Wong e7ab024ca0 small locking optimization 2025-05-02 23:18:07 -07:00
Benson WongandGitHub 448ccae959 Introduce Groups Feature (#107) 
Groups allows more control over swapping behaviour when a model is requested. The new groups feature provides three ways to control swapping: within the group, swapping out other groups or keep the models in the group loaded persistently (never swapped out). 

Closes #96, #99 and #106.
 2025-05-02 22:35:38 -07:00
Benson WongandGitHub ec0348e431 Reduce stale time for issues 2025-04-29 21:16:34 -07:00
Benson WongandGitHub 06eda7f591 tag all process logs with its ID (#103) 
Makes identifying Process of log messages easier
 2025-04-25 12:58:25 -07:00
Benson WongandGitHub 5fad24c16f Make checkHealthTimeout Interruptable during startup (#102) 
interrupt and exit Process.start() early if the upstream process exits prematurely or unexpectedly.
 2025-04-24 14:39:33 -07:00
Benson Wong 8404244fab Moderate security update for golang/x/net -> v0.38.0 2025-04-24 09:58:40 -07:00
Benson Wong 712cd01081 fix confusing INFO message [no ci] 2025-04-24 09:56:20 -07:00
Benson Wong 1f7aa359b1 Update header image 
AI has finally made my dreams of llamas in funny clothing and stuck in
a claw machine waiting to be picked come true!
 2025-04-23 13:02:12 -07:00
Benson WongandGitHub b138d6cf25 fix starhistory in README 2025-04-15 20:23:46 -07:00