proxy: add support for user defined metadata in model configs (#333)

Changes: 

- add Metadata key to ModelConfig
- include metadata in /v1/models under meta.llamaswap key
- add recursive macro substitution into Metadata
- change macros at global and model level to be any scalar type

Note: 

This is the first mostly AI generated change to llama-swap. See #333 for notes about the workflow and approach to AI going forward.

.github/ISSUE_TEMPLATE/bug-report.md

@@ -1,11 +1,13 @@
 ---
 name: Bug Report
-about: Something is not working as expected...
+about: I found a defect
 title: ''
-labels: bug
+labels: 'unconfirmed bug'
 assignees: ''
 
 ---
+> [!IMPORTANT]
+> If you have questions about llama-swap please post in the Q&A in Discussions. Use bug reports when you've found a defect and wish to discuss a fix.
 
 **Describe the bug**
 A clear and concise description of what the bug is.

.github/workflows/go-ci.yml

@@ -22,6 +22,13 @@ jobs:
       with:
         go-version: '1.23'
 
+    # Only run in this linux based runner
+    - name: Check Formatting
+      run: |
+        if [ "$(gofmt -l . | grep -v 'event/.*_test.go' | wc -l)" -gt 0 ]; then
+          gofmt -l . | grep -v 'event/.*_test.go'
+          exit 1
+        fi
     # cache simple-responder to save the build time
     - name: Restore Simple Responder
       id: restore-simple-responder

.gitignore

@@ -4,3 +4,4 @@ build/
 dist/
 .vscode
 .DS_Store
+.dev/

CLAUDE.md

@@ -0,0 +1,43 @@
+# Project: llama-swap
+
+## Project Description:
+
+llama-swap is a light weight, transparent proxy server that provides automatic model swapping to llama.cpp's server.
+
+## Tech stack
+
+- golang
+- typescript, vite and react for UI (ui/)
+
+## Testing
+
+- `make test-dev` - Use this when making iterative changes. Runs `go test` and `staticcheck`. Fix any static checking errors.
+- `make test-all` - runs at the end before completing work. Includes long running concurrency tests.
+
+## Workflow Tasks
+
+### Plan Improvements
+
+Work plans are located in ai-plans/. Plans written by the user may be incomplete, contain inconsistencies or errors.
+
+When the user asks to improve a plan follow these guidelines for expanding and improving it.
+
+- Identify any inconsistencies.
+- Expand plans out to be detailed specification of requirements and changes to be made.
+- Plans should have at least these sections:
+  - Title - very short, describes changes
+  - Overview: A more detailed summary of goal and outcomes desired
+  - Design Requirements: Detailed descriptions of what needs to be done
+  - Testing Plan: Tests to be implemented
+  - Checklist: A detailed list of changes to be made
+
+Look for "plan expansion" as explicit instructions to improve a plan.
+
+### Implementation of plans
+
+When the user says "paint it", respond with "commencing automated assembly". Then implement the changes as described by the plan. Update the checklist as you complete items.
+
+## General Rules
+
+- when summarizing changes only include details that require further action (action items)
+- when there are no action items, just say "Done."

Makefile

@@ -23,11 +23,17 @@ proxy/ui_dist/placeholder.txt:
 	mkdir -p proxy/ui_dist
 	touch $@
 
-test: proxy/ui_dist/placeholder.txt
+# use cached test results while developing
-	go test -short -v -count=1 ./proxy
+test-dev: proxy/ui_dist/placeholder.txt
+	go test -short ./proxy/...
+	staticcheck ./proxy/... || true
 
+test: proxy/ui_dist/placeholder.txt
+	go test -short -count=1 ./proxy/...
+
+# for CI - full test (takes longer)
 test-all: proxy/ui_dist/placeholder.txt
-	go test -v -count=1 ./proxy
+	go test -count=1 ./proxy/...
 
 ui/node_modules:
 	cd ui && npm install
@@ -45,6 +51,7 @@ mac: ui
 linux: ui
 	@echo "Building Linux binary..."
 	GOOS=linux GOARCH=amd64 go build -ldflags="-X main.commit=${GIT_HASH} -X main.version=local_${GIT_HASH} -X main.date=${BUILD_DATE}" -o $(BUILD_DIR)/$(APP_NAME)-linux-amd64
+	GOOS=linux GOARCH=arm64 go build -ldflags="-X main.commit=${GIT_HASH} -X main.version=local_${GIT_HASH} -X main.date=${BUILD_DATE}" -o $(BUILD_DIR)/$(APP_NAME)-linux-arm64
 
 # Build Windows binary
 windows: ui
@@ -80,4 +87,4 @@ release:
 	git tag "$$new_tag";
 
 # Phony targets
-.PHONY: all clean ui mac linux windows simple-responder
+.PHONY: all clean ui mac linux windows simple-responder test test-all test-dev

README.md

@@ -7,7 +7,7 @@
 
 llama-swap is a light weight, transparent proxy server that provides automatic model swapping to llama.cpp's server.
 
-Written in golang, it is very easy to install (single binary with no dependencies) and configure (single yaml file). To get started, download a pre-built binary or use the provided docker images.
+Written in golang, it is very easy to install (single binary with no dependencies) and configure (single yaml file). To get started, download a pre-built binary, a provided docker images or Homebrew.
 
 ## Features:
 
@@ -18,9 +18,12 @@ Written in golang, it is very easy to install (single binary with no dependencie
   - `v1/completions`
   - `v1/chat/completions`
   - `v1/embeddings`
-  - `v1/rerank`, `v1/reranking`, `rerank`
   - `v1/audio/speech` ([#36](https://github.com/mostlygeek/llama-swap/issues/36))
   - `v1/audio/transcriptions` ([docs](https://github.com/mostlygeek/llama-swap/issues/41#issuecomment-2722637867))
+- ✅ llama-server (llama.cpp) supported endpoints:
+  - `v1/rerank`, `v1/reranking`, `/rerank`
+  - `/infill` - for code infilling
+  - `/completion` - for completion endpoint
 - ✅ llama-swap custom API endpoints
   - `/ui` - web UI
   - `/log` - remote log monitoring
@@ -31,20 +34,21 @@ Written in golang, it is very easy to install (single binary with no dependencie
 - ✅ Run multiple models at once with `Groups` ([#107](https://github.com/mostlygeek/llama-swap/issues/107))
 - ✅ Automatic unloading of models after timeout by setting a `ttl`
 - ✅ Use any local OpenAI compatible server (llama.cpp, vllm, tabbyAPI, etc)
-- ✅ Docker and Podman support
+- ✅ Reliable Docker and Podman support using `cmd` and `cmdStop` together
 - ✅ Full control over server settings per model
+- ✅ Preload models on startup with `hooks` ([#235](https://github.com/mostlygeek/llama-swap/pull/235))
 
 ## How does llama-swap work?
 
-When a request is made to an OpenAI compatible endpoint, lama-swap will extract the `model` value and load the appropriate server configuration to serve it. If the wrong upstream server is running, it will be replaced with the correct one. This is where the "swap" part comes in. The upstream server is automatically swapped to the correct one to serve the request.
+When a request is made to an OpenAI compatible endpoint, llama-swap will extract the `model` value and load the appropriate server configuration to serve it. If the wrong upstream server is running, it will be replaced with the correct one. This is where the "swap" part comes in. The upstream server is automatically swapped to the correct one to serve the request.
 
 In the most basic configuration llama-swap handles one model at a time. For more advanced use cases, the `groups` feature allows multiple models to be loaded at the same time. You have complete control over how your system resources are used.
 
 ## config.yaml
 
-llama-swap is managed entirely through a yaml configuration file. 
+llama-swap is managed entirely through a yaml configuration file.
 
-It can be very minimal to start: 
+It can be very minimal to start:
 
 ```yaml
 models:
@@ -55,7 +59,7 @@ models:
       --port ${PORT}
 ```
 
-However, there are many more capabilities that llama-swap supports: 
+However, there are many more capabilities that llama-swap supports:
 
 - `groups` to run multiple models at once
 - `ttl` to automatically unload models
@@ -69,11 +73,39 @@ However, there are many more capabilities that llama-swap supports:
 
 See the [configuration documentation](https://github.com/mostlygeek/llama-swap/wiki/Configuration) in the wiki all options and examples.
 
+## Reverse Proxy Configuration (nginx)
+
+If you deploy llama-swap behind nginx, disable response buffering for streaming endpoints. By default, nginx buffers responses which breaks Server‑Sent Events (SSE) and streaming chat completion. ([#236](https://github.com/mostlygeek/llama-swap/issues/236))
+
+Recommended nginx configuration snippets:
+
+```nginx
+# SSE for UI events/logs
+location /api/events {
+    proxy_pass http://your-llama-swap-backend;
+    proxy_buffering off;
+    proxy_cache off;
+}
+
+# Streaming chat completions (stream=true)
+location /v1/chat/completions {
+    proxy_pass http://your-llama-swap-backend;
+    proxy_buffering off;
+    proxy_cache off;
+}
+```
+
+As a safeguard, llama-swap also sets `X-Accel-Buffering: no` on SSE responses. However, explicitly disabling `proxy_buffering` at your reverse proxy is still recommended for reliable streaming behavior.
+
 ## Web UI
 
-llama-swap ships with a real time web interface to monitor logs and status of models:
+llama-swap includes a real time web interface for monitoring logs and models:
 
-<img width="1786" height="1334" alt="image" src="https://github.com/user-attachments/assets/d6258cb9-1dad-40db-828f-2be860aec8fe" />
+<img width="1360" height="963" alt="image" src="https://github.com/user-attachments/assets/adef4a8e-de0b-49db-885a-8f6dedae6799" />
+
+The Activity Page shows recent requests:
+
+<img width="1360" height="963" alt="image" src="https://github.com/user-attachments/assets/5f3edee6-d03a-4ae5-ae06-b20ac1f135bd" />
 
 ## Installation
 
@@ -86,7 +118,7 @@ llama-swap can be installed in multiple ways
 
 ### Docker Install ([download images](https://github.com/mostlygeek/llama-swap/pkgs/container/llama-swap))
 
-Docker images with llama-swap and llama-server are built nightly. 
+Docker images with llama-swap and llama-server are built nightly.
 
 ```shell
 # use CPU inference comes with the example config above
@@ -133,10 +165,10 @@ $ docker run -it --rm --runtime nvidia -p 9292:8080 \
 
 ### Homebrew Install (macOS/Linux)
 
-The latest release of `llama-swap` can be installed via [Homebrew](https://brew.sh). 
+The latest release of `llama-swap` can be installed via [Homebrew](https://brew.sh).
 
 ```shell
-# Set up tap and install formula 
+# Set up tap and install formula
 brew tap mostlygeek/llama-swap
 brew install llama-swap
 # Run llama-swap
@@ -161,7 +193,7 @@ Binaries are available for Linux, Mac, Windows and FreeBSD. These are automatica
 ### Building from source
 
 1. Build requires golang and nodejs for the user interface.
-1. `git clone git@github.com:mostlygeek/llama-swap.git`
+1. `git clone https://github.com/mostlygeek/llama-swap.git`
 1. `make clean all`
 1. Binaries will be in `build/` subdirectory
 
@@ -197,13 +229,9 @@ Any OpenAI compatible server would work. llama-swap was originally designed for
 
 For Python based inference servers like vllm or tabbyAPI it is recommended to run them via podman or docker. This provides clean environment isolation as well as responding correctly to `SIGTERM` signals to shutdown.
 
-## Contributors 
-<a href="https://github.com/mostlygeek/llama-swap/graphs/contributors">
-  <img src="https://contrib.rocks/image?repo=mostlygeek/llama-swap" />
-</a>
-
-Made with [contrib.rocks](https://contrib.rocks).
-
 ## Star History
 
+> [!NOTE]
+> ⭐️ Star this project to help others discover it! 
+
 [![Star History Chart](https://api.star-history.com/svg?repos=mostlygeek/llama-swap&type=Date)](https://www.star-history.com/#mostlygeek/llama-swap&Date)

ai-plans/issue-264-add-metadata.md

@@ -0,0 +1,292 @@
+# Add Model Metadata Support with Typed Macros
+
+## Overview
+
+Implement support for arbitrary metadata on model configurations that can be exposed through the `/v1/models` API endpoint. This feature extends the existing macro system to support scalar types (string, int, float, bool) instead of only strings, enabling type-safe metadata values.
+
+The metadata will be schemaless, allowing users to define any key-value pairs they need. Macro substitution will work within metadata values, preserving types when macros are used directly and converting to strings when macros are interpolated within strings.
+
+## Design Requirements
+
+### 1. Enhanced Macro System
+
+**Current State:**
+
+- Macros are defined as `map[string]string` at both global and model levels
+- Only string substitution is supported
+- Macros are replaced in: `cmd`, `cmdStop`, `proxy`, `checkEndpoint`, `filters.stripParams`
+
+**Required Changes:**
+
+- Change `MacroList` type from `map[string]string` to `map[string]any`
+- Support scalar types: `string`, `int`, `float64`, `bool`
+- Implement type-preserving macro substitution:
+  - Direct macro usage (`key: ${macro}`) preserves the macro's type
+  - Interpolated usage (`key: "text ${macro}"`) converts to string
+- Add validation to ensure macro values are scalar types only
+- Update existing macro substitution logic in [proxy/config/config.go](proxy/config/config.go) to handle `any` types
+
+**Implementation Details:**
+
+- Create a generic helper function to perform macro substitution that:
+  - Takes a value of type `any`
+  - Recursively processes maps, slices, and scalar values
+  - Replaces `${macro_name}` patterns with macro values
+  - Preserves types for direct substitution
+  - Converts to strings for interpolated substitution
+- Update `validateMacro()` function to accept `any` type and validate scalar types
+- Maintain backward compatibility with existing string-only macros
+
+### 2. Metadata Field in ModelConfig
+
+**Location:** [proxy/config/model_config.go](proxy/config/model_config.go)
+
+**Required Changes:**
+
+- Add `Metadata map[string]any` field to `ModelConfig` struct
+- Support YAML unmarshaling of arbitrary structures (maps, arrays, scalars)
+- Apply macro substitution to metadata values during config loading
+
+**Schema Requirements:**
+
+- Metadata is optional (default: empty/nil map)
+- Supports nested structures (objects within objects, arrays, etc.)
+- All string values within metadata undergo macro substitution
+- Type preservation rules apply as described above
+
+### 3. Macro Substitution in Metadata
+
+**Location:** [proxy/config/config.go](proxy/config/config.go) in `LoadConfigFromReader()`
+
+**Process Flow:**
+
+1. After loading YAML configuration
+2. After model-level and global macro merging
+3. Apply macro substitution to `ModelConfig.Metadata` field
+4. Use the same merged macros available to `cmd`, `proxy`, etc.
+5. Process recursively through all nested structures
+
+**Substitution Rules:**
+
+- `port: ${PORT}` → keeps integer type from PORT macro
+- `temperature: ${temp}` → keeps float type from temp macro
+- `note: "Running on ${PORT}"` → converts to string `"Running on 10001"`
+- Arrays and nested objects are processed recursively
+- Unknown macros should cause configuration load error (consistent with existing behavior)
+
+### 4. API Response Updates
+
+**Location:** [proxy/proxymanager.go:350](proxy/proxymanager.go#L350) `listModelsHandler()`
+
+**Current Behavior:**
+
+- Returns model records with: `id`, `object`, `created`, `owned_by`
+- Optionally includes: `name`, `description`
+
+**Required Changes:**
+
+- Add metadata to each model record under the key `llamaswap_meta`
+- Only include `llamaswap_meta` if metadata is non-empty
+- Preserve all types when marshaling to JSON
+- Maintain existing sorting by model ID
+
+**Example Response:**
+
+```json
+{
+  "object": "list",
+  "data": [
+    {
+      "id": "llama",
+      "object": "model",
+      "created": 1234567890,
+      "owned_by": "llama-swap",
+      "name": "llama 3.1 8B",
+      "description": "A small but capable model",
+      "llamaswap_meta": {
+        "port": 10001,
+        "temperature": 0.7,
+        "note": "The llama is running on port 10001 temp=0.7, context=16384",
+        "a_list": [1, 1.23, "macros are OK in list and dictionary types: llama"],
+        "an_obj": {
+          "a": "1",
+          "b": 2,
+          "c": [0.7, false, "model: llama"]
+        }
+      }
+    }
+  ]
+}
+```
+
+### 5. Validation and Error Handling
+
+**Macro Validation:**
+
+- Extend `validateMacro()` to accept values of type `any`
+- Verify macro values are scalar types: `string`, `int`, `float64`, `bool`
+- Reject complex types (maps, slices, structs) as macro values
+- Maintain existing validation for macro names and lengths
+
+**Configuration Loading:**
+
+- Fail fast if unknown macros are found in metadata
+- Provide clear error messages indicating which model and field contains errors
+- Ensure macros in metadata follow same rules as macros in cmd/proxy fields
+
+## Testing Plan
+
+### Test 1: Model-Level Macros with Different Types
+
+**File:** [proxy/config/model_config_test.go](proxy/config/model_config_test.go)
+
+**Test Cases:**
+
+- Define model with macros of each scalar type
+- Verify metadata correctly substitutes and preserves types
+- Test direct substitution (`port: ${PORT}`)
+- Test string interpolation (`note: "Port is ${PORT}"`)
+- Verify nested objects and arrays work correctly
+
+### Test 2: Global and Model Macro Precedence
+
+**File:** [proxy/config/config_test.go](proxy/config/config_test.go)
+
+**Test Cases:**
+
+- Define same macro at global and model level with different types
+- Verify model-level macro takes precedence
+- Test metadata uses correct macro value
+- Verify type is preserved from the winning macro
+
+### Test 3: Macro Validation
+
+**File:** [proxy/config/config_test.go](proxy/config/config_test.go)
+
+**Test Cases:**
+
+- Test that complex types (maps, arrays) are rejected as macro values
+  - Verify error message includes: macro name and type that was rejected
+- Test that scalar types (string, int, float, bool) are accepted
+  - Each type should load without error
+- Test macro name validation still works with `any` types
+  - Invalid characters, reserved names, length limits should still be enforced
+
+### Test 4: Metadata in API Response
+
+**File:** [proxy/proxymanager_test.go](proxy/proxymanager_test.go)
+
+**Existing Test:** `TestProxyManager_ListModelsHandler`
+
+**Test Cases:**
+
+- Model with metadata → verify `llamaswap_meta` key appears
+- Model without metadata → verify `llamaswap_meta` key is absent
+- Verify all types are correctly marshaled to JSON
+- Verify nested structures are preserved
+- Verify macro substitution has occurred before serialization
+
+### Test 5: Unknown Macros in Metadata
+
+**File:** [proxy/config/config_test.go](proxy/config/config_test.go)
+
+**Test Cases:**
+
+- Use undefined macro in metadata
+- Verify configuration loading fails with clear error
+- Error should indicate model name and that macro is undefined
+
+### Test 6: Recursive Substitution
+
+**File:** [proxy/config/config_test.go](proxy/config/config_test.go)
+
+**Test Cases:**
+
+- Metadata with deeply nested structures
+- Arrays containing objects with macros
+- Objects containing arrays with macros
+- Mixed string interpolation and direct substitution at various nesting levels
+
+## Checklist
+
+### Configuration Schema Changes
+
+- [x] Change `MacroList` type from `map[string]string` to `map[string]any` in [proxy/config/config.go:19](proxy/config/config.go#L19)
+- [x] Add `Metadata map[string]any` field to `ModelConfig` struct in [proxy/config/model_config.go:37](proxy/config/model_config.go#L37)
+- [x] Update `validateMacro()` function signature to accept `any` type for values
+- [x] Add validation logic to ensure macro values are scalar types only
+
+### Macro Substitution Logic
+
+- [x] Create generic recursive function `substituteMetadataMacros()` to handle `any` types
+- [x] Implement type-preserving direct substitution logic
+- [x] Implement string interpolation with type conversion
+- [x] Handle maps: recursively process all values
+- [x] Handle slices: recursively process all elements
+- [x] Handle scalar types: perform string-based macro substitution if value is string
+- [x] Integrate macro substitution into `LoadConfigFromReader()` after existing macro expansion
+- [x] Update existing macro substitution calls to use merged macros with correct types
+
+### API Response Changes
+
+- [x] Modify `listModelsHandler()` in [proxy/proxymanager.go:350](proxy/proxymanager.go#L350)
+- [x] Add `llamaswap_meta` field to model records when metadata exists
+- [x] Ensure empty metadata results in omitted `llamaswap_meta` key
+- [x] Verify JSON marshaling preserves all types correctly
+
+### Testing - Config Package
+
+- [x] Add test for string macros in metadata: [proxy/config/config_test.go](proxy/config/config_test.go)
+- [x] Add test for int macros in metadata: [proxy/config/config_test.go](proxy/config/config_test.go)
+- [x] Add test for float macros in metadata: [proxy/config/config_test.go](proxy/config/config_test.go)
+- [x] Add test for bool macros in metadata: [proxy/config/config_test.go](proxy/config/config_test.go)
+- [x] Add test for string interpolation in metadata: [proxy/config/config_test.go](proxy/config/config_test.go)
+- [x] Add test for model-level macro precedence: [proxy/config/config_test.go](proxy/config/config_test.go)
+- [x] Add test for nested structures in metadata: [proxy/config/config_test.go](proxy/config/config_test.go)
+- [x] Add test for unknown macro in metadata (should error): [proxy/config/config_test.go](proxy/config/config_test.go)
+- [x] Add test for invalid macro type validation: [proxy/config/config_test.go](proxy/config/config_test.go)
+
+### Testing - Model Config Package
+
+- [x] Add test cases to [proxy/config/model_config_test.go](proxy/config/model_config_test.go) for metadata unmarshaling
+- [x] Test metadata with various scalar types
+- [x] Test metadata with nested objects and arrays
+
+### Testing - Proxy Manager
+
+- [x] Update `TestProxyManager_ListModelsHandler` in [proxy/proxymanager_test.go](proxy/proxymanager_test.go)
+- [x] Add test case for model with metadata
+- [x] Add test case for model without metadata
+- [x] Verify `llamaswap_meta` key presence/absence
+- [x] Verify type preservation in JSON output
+- [x] Verify macro substitution has occurred
+
+### Documentation
+
+- [x] Verify [config.example.yaml](config.example.yaml) already has complete metadata examples (lines 149-171)
+- [x] No additional documentation needed per project instructions
+
+## Known Issues and Considerations
+
+### Inconsistencies
+
+None identified. The plan references the correct existing example in [config.example.yaml:149-171](config.example.yaml#L149-L171).
+
+### Design Decisions
+
+1. **Why `llamaswap_meta` instead of merging into record?**
+
+   - Avoids potential collisions with OpenAI API standard fields
+   - Makes it clear this is llama-swap specific metadata
+   - Easier for clients to distinguish standard vs. custom fields
+
+2. **Why support nested structures?**
+
+   - Provides maximum flexibility for users
+   - Aligns with the schemaless design principle
+   - Example config already demonstrates this capability
+
+3. **Why validate macro types?**
+   - Prevents confusing behavior (e.g., substituting a map)
+   - Makes configuration errors explicit at load time
+   - Simpler implementation and testing

config.example.yaml

@@ -1,9 +1,17 @@
 # llama-swap YAML configuration example
 # -------------------------------------
 #
+# 💡 Tip - Use an LLM with this file!
+# ====================================
+#  This example configuration is written to be LLM friendly. Try
+#  copying this file into an LLM and asking it to explain or generate
+#  sections for you.
+# ====================================
+
+# Usage notes:
 # - Below are all the available configuration options for llama-swap.
-# - Settings with a default value, or noted as optional can be omitted.
+# - Settings noted as "required" must be in your configuration file
-# - Settings that are marked required must be in your configuration file
+# - Settings noted as "optional" can be omitted
 
 # healthCheckTimeout: number of seconds to wait for a model to be ready to serve requests
 # - optional, default: 120
@@ -27,26 +35,41 @@ metricsMaxInMemory: 1000
 # - it is automatically incremented for every model that uses it
 startPort: 10001
 
-# macros: sets a dictionary of string:string pairs
+# macros: a dictionary of string substitutions
 # - optional, default: empty dictionary
-# - these are reusable snippets
+# - macros are reusable snippets
-# - used in a model's cmd, cmdStop, proxy and checkEndpoint
+# - used in a model's cmd, cmdStop, proxy, checkEndpoint, filters.stripParams
 # - useful for reducing common configuration settings
+# - macro names are strings and must be less than 64 characters
+# - macro names must match the regex ^[a-zA-Z0-9_-]+$
+# - macro names must not be a reserved name: PORT or MODEL_ID
+# - macro values must be less than 1024 characters
+#
+# Important: do not nest macros inside other macros; expansion is single-pass
 macros:
   "latest-llama": >
     /path/to/llama-server/llama-server-ec9e0301
     --port ${PORT}
+  "default_ctx": "4096"
 
 # models: a dictionary of model configurations
 # - required
 # - each key is the model's ID, used in API requests
 # - model settings have default values that are used if they are not defined here
-# - below are examples of the various settings a model can have:
+# - the model's ID is available in the ${MODEL_ID} macro, also available in macros defined above
-# - available model settings: env, cmd, cmdStop, proxy, aliases, checkEndpoint, ttl, unlisted
+# - below are examples of the all the settings a model can have
 models:
 
   # keys are the model names used in API requests
   "llama":
+    # macros: a dictionary of string substitutions specific to this model
+    # - optional, default: empty dictionary
+    # - macros defined here override macros defined in the global macros section
+    # - model level macros follow the same rules as global macros
+    macros:
+      "default_ctx": 16384
+      "temp": 0.7
+
     # cmd: the command to run to start the inference server.
     # - required
     # - it is just a string, similar to what you would run on the CLI
@@ -56,6 +79,8 @@ models:
       # ${latest-llama} is a macro that is defined above
       ${latest-llama}
       --model path/to/llama-8B-Q4_K_M.gguf
+      --ctx-size ${default_ctx}
+      --temperature ${temp}
 
     # name: a display name for the model
     # - optional, default: empty string
@@ -92,49 +117,84 @@ models:
 
     # checkEndpoint: URL path to check if the server is ready
     # - optional, default: /health
-    # - use "none" to skip endpoint ready checking
     # - endpoint is expected to return an HTTP 200 response
-    # - all requests wait until the endpoint is ready (or fails)
+    # - all requests wait until the endpoint is ready or fails
+    # - use "none" to skip endpoint health checking
     checkEndpoint: /custom-endpoint
 
-    # ttl: automatically unload the model after this many seconds
+    # ttl: automatically unload the model after ttl seconds
     # - optional, default: 0
     # - ttl values must be a value greater than 0
     # - a value of 0 disables automatic unloading of the model
     ttl: 60
 
-    # useModelName: overrides the model name that is sent to upstream server
+    # useModelName: override the model name that is sent to upstream server
     # - optional, default: ""
-    # - useful when the upstream server expects a specific model name or format
+    # - useful for when the upstream server expects a specific model name that
+    #   is different from the model's ID
     useModelName: "qwen:qwq"
 
     # filters: a dictionary of filter settings
     # - optional, default: empty dictionary
+    # - only stripParams is currently supported
     filters:
-      # strip_params: a comma separated list of parameters to remove from the request
+      # stripParams: a comma separated list of parameters to remove from the request
       # - optional, default: ""
-      # - useful for preventing overriding of default server params by requests
+      # - useful for server side enforcement of sampling parameters
-      # - `model` parameter is never removed
+      # - the `model` parameter can never be removed
       # - can be any JSON key in the request body
       # - recommended to stick to sampling parameters
-      strip_params: "temperature, top_p, top_k"
+      stripParams: "temperature, top_p, top_k"
+
+    # metadata: a dictionary of arbitrary values that are included in /v1/models
+    # - optional, default: empty dictionary
+    # - while metadata can contains complex types it is recommended to keep it simple
+    # - metadata is only passed through in /v1/models responses
+    metadata:
+      # port will remain an integer
+      port: ${PORT}
+
+      # the ${temp} macro will remain a float
+      temperature: ${temp}
+      note: "The ${MODEL_ID} is running on port ${PORT} temp=${temp}, context=${default_ctx}"
+
+      a_list:
+        - 1
+        - 1.23
+        - "macros are OK in list and dictionary types: ${MODEL_ID}"
+
+      an_obj:
+        a: "1"
+        b: 2
+        # objects can contain complex types with macro substitution
+        # becomes: c: [0.7, false, "model: llama"]
+        c: ["${temp}", false, "model: ${MODEL_ID}"]
+
+    # concurrencyLimit: overrides the allowed number of active parallel requests to a model
+    # - optional, default: 0
+    # - useful for limiting the number of active parallel requests a model can process
+    # - must be set per model
+    # - any number greater than 0 will override the internal default value of 10
+    # - any requests that exceeds the limit will receive an HTTP 429 Too Many Requests response
+    # - recommended to be omitted and the default used
+    concurrencyLimit: 0
 
   # Unlisted model example:
   "qwen-unlisted":
-    # unlisted: true or false
+    # unlisted: boolean, true or false
     # - optional, default: false
-    # - unlisted models do not show up in /v1/models or /upstream lists
+    # - unlisted models do not show up in /v1/models api requests
     # - can be requested as normal through all apis
     unlisted: true
     cmd: llama-server --port ${PORT} -m Llama-3.2-1B-Instruct-Q4_K_M.gguf -ngl 0
 
   # Docker example:
-  # container run times like Docker and Podman can also be used with a
+  # container runtimes like Docker and Podman can be used reliably with
-  # a combination of cmd and cmdStop.
+  # a combination of cmd, cmdStop, and ${MODEL_ID}
   "docker-llama":
     proxy: "http://127.0.0.1:${PORT}"
     cmd: |
-      docker run --name dockertest
+      docker run --name ${MODEL_ID}
       --init --rm -p ${PORT}:8080 -v /mnt/nvme/models:/models
       ghcr.io/ggml-org/llama.cpp:server
       --model '/models/Qwen2.5-Coder-0.5B-Instruct-Q4_K_M.gguf'
@@ -142,24 +202,26 @@ models:
     # cmdStop: command to run to stop the model gracefully
     # - optional, default: ""
     # - useful for stopping commands managed by another system
-    # - on POSIX systems: a SIGTERM is sent for graceful shutdown
-    # - on Windows, taskkill is used
-    # - processes are given 5 seconds to shutdown until they are forcefully killed
     # - the upstream's process id is available in the ${PID} macro
-    cmdStop: docker stop dockertest
+    #
+    # When empty, llama-swap has this default behaviour:
+    # - on POSIX systems: a SIGTERM signal is sent
+    # - on Windows, calls taskkill to stop the process
+    # - processes have 5 seconds to shutdown until forceful termination is attempted
+    cmdStop: docker stop ${MODEL_ID}
 
 # groups: a dictionary of group settings
 # - optional, default: empty dictionary
-# - provide advanced controls over model swapping behaviour.
+# - provides advanced controls over model swapping behaviour
-# - Using groups some models can be kept loaded indefinitely, while others are swapped out.
+# - using groups some models can be kept loaded indefinitely, while others are swapped out
-# - model ids must be defined in the Models section
+# - model IDs must be defined in the Models section
 # - a model can only be a member of one group
 # - group behaviour is controlled via the `swap`, `exclusive` and `persistent` fields
 # - see issue #109 for details
 #
 # NOTE: the example below uses model names that are not defined above for demonstration purposes
 groups:
-  # group1 is same as the default behaviour of llama-swap where only one model is allowed
+  # group1 works the same as the default behaviour of llama-swap where only one model is allowed
   # to run a time across the whole llama-swap instance
   "group1":
     # swap: controls the model swapping behaviour in within the group
@@ -181,10 +243,13 @@ groups:
       - "qwen-unlisted"
 
   # Example:
-  # - in this group all the models can run at the same time
+  # - in group2 all models can run at the same time
-  # - when a different group loads all running models in this group are unloaded
+  # - when a different group is loaded it causes all running models in this group to unload
   "group2":
     swap: false
+
+    # exclusive: false does not unload other groups when a model in group2 is requested
+    # - the models in group2 will be loaded but will not unload any other groups
     exclusive: false
     members:
       - "docker-llama"
@@ -207,3 +272,19 @@ groups:
       - "forever-modelA"
       - "forever-modelB"
       - "forever-modelc"
+
+# hooks: a dictionary of event triggers and actions
+# - optional, default: empty dictionary
+# - the only supported hook is on_startup
+hooks:
+  # on_startup: a dictionary of actions to perform on startup
+  # - optional, default: empty dictionary
+  # - the only supported action is preload
+  on_startup:
+        # preload: a list of model ids to load on startup
+        # - optional, default: empty list
+        # - model names must match keys in the models sections
+        # - when preloading multiple models at once, define a group
+        #   otherwise models will be loaded and swapped out
+    preload:
+      - "llama"

llama-swap.go

@@ -16,6 +16,7 @@ import (
 	"github.com/gin-gonic/gin"
 	"github.com/mostlygeek/llama-swap/event"
 	"github.com/mostlygeek/llama-swap/proxy"
+	"github.com/mostlygeek/llama-swap/proxy/config"
 )
 
 var (
@@ -38,13 +39,13 @@ func main() {
 		os.Exit(0)
 	}
 
-	config, err := proxy.LoadConfig(*configPath)
+	conf, err := config.LoadConfig(*configPath)
 	if err != nil {
 		fmt.Printf("Error loading config: %v\n", err)
 		os.Exit(1)
 	}
 
-	if len(config.Profiles) > 0 {
+	if len(conf.Profiles) > 0 {
 		fmt.Println("WARNING: Profile functionality has been removed in favor of Groups. See the README for more information.")
 	}
 
@@ -67,7 +68,7 @@ func main() {
 	// Support for watching config and reloading when it changes
 	reloadProxyManager := func() {
 		if currentPM, ok := srv.Handler.(*proxy.ProxyManager); ok {
-			config, err = proxy.LoadConfig(*configPath)
+			conf, err = config.LoadConfig(*configPath)
 			if err != nil {
 				fmt.Printf("Warning, unable to reload configuration: %v\n", err)
 				return
@@ -75,7 +76,7 @@ func main() {
 
 			fmt.Println("Configuration Changed")
 			currentPM.Shutdown()
-			srv.Handler = proxy.New(config)
+			srv.Handler = proxy.New(conf)
 			fmt.Println("Configuration Reloaded")
 
 			// wait a few seconds and tell any UI to reload
@@ -85,12 +86,12 @@ func main() {
 				})
 			})
 		} else {
-			config, err = proxy.LoadConfig(*configPath)
+			conf, err = config.LoadConfig(*configPath)
 			if err != nil {
 				fmt.Printf("Error, unable to load configuration: %v\n", err)
 				os.Exit(1)
 			}
-			srv.Handler = proxy.New(config)
+			srv.Handler = proxy.New(conf)
 		}
 	}
 
@@ -132,6 +133,11 @@ func main() {
 						event.Emit(proxy.ConfigFileChangedEvent{
 							ReloadingState: proxy.ReloadingStateStart,
 						})
+					} else if changeEvent.Name == filepath.Join(configDir, "..data") && changeEvent.Has(fsnotify.Create) {
+						// the change for k8s configmap
+						event.Emit(proxy.ConfigFileChangedEvent{
+							ReloadingState: proxy.ReloadingStateStart,
+						})
 					}
 
 				case err := <-watcher.Errors:

misc/benchmark-chatcompletion/main.go

@@ -0,0 +1,159 @@
+package main
+
+// created for issue: #252 https://github.com/mostlygeek/llama-swap/issues/252
+// this simple benchmark tool sends a lot of small chat completion requests to llama-swap
+// to make sure all the requests are accounted for.
+//
+// requests can be sent in parallel, and the tool will report the results.
+// usage: go run main.go -baseurl http://localhost:8080/v1 -model llama3 -requests 1000 -par 5
+
+import (
+	"bytes"
+	"flag"
+	"fmt"
+	"io"
+	"log"
+	"net/http"
+	"os"
+	"sync"
+	"time"
+)
+
+func main() {
+	// ----- CLI arguments ----------------------------------------------------
+	var (
+		baseurl         string
+		modelName       string
+		totalRequests   int
+		parallelization int
+	)
+
+	flag.StringVar(&baseurl, "baseurl", "http://localhost:8080/v1", "Base URL of the API (e.g., https://api.example.com)")
+	flag.StringVar(&modelName, "model", "", "Model name to use")
+	flag.IntVar(&totalRequests, "requests", 1, "Total number of requests to send")
+	flag.IntVar(&parallelization, "par", 1, "Maximum number of concurrent requests")
+	flag.Parse()
+
+	if baseurl == "" || modelName == "" {
+		fmt.Println("Error: both -baseurl and -model are required.")
+		flag.Usage()
+		os.Exit(1)
+	}
+	if totalRequests <= 0 {
+		fmt.Println("Error: -requests must be greater than 0.")
+		os.Exit(1)
+	}
+	if parallelization <= 0 {
+		fmt.Println("Error: -parallelization must be greater than 0.")
+		os.Exit(1)
+	}
+
+	// ----- HTTP client -------------------------------------------------------
+	client := &http.Client{
+		Timeout: 30 * time.Second,
+	}
+
+	// ----- Tracking response codes -------------------------------------------
+	statusCounts := make(map[int]int) // map[statusCode]count
+	var mu sync.Mutex                 // protects statusCounts
+
+	// ----- Request queue (buffered channel) ----------------------------------
+	requests := make(chan int, 10) // Buffered channel with capacity 10
+
+	// Goroutine to fill the request queue
+	go func() {
+		for i := 0; i < totalRequests; i++ {
+			requests <- i + 1
+		}
+		close(requests)
+	}()
+
+	// ----- Worker pool -------------------------------------------------------
+	var wg sync.WaitGroup
+	for i := 0; i < parallelization; i++ {
+		wg.Add(1)
+		go func(workerID int) {
+			defer wg.Done()
+
+			for reqID := range requests {
+				// Build request payload as a single line JSON string
+				payload := `{"model":"` + modelName + `","max_tokens":100,"stream":false,"messages":[{"role":"user","content":"write a snake game in python"}]}`
+
+				// Send POST request
+				req, err := http.NewRequest(http.MethodPost,
+					fmt.Sprintf("%s/chat/completions", baseurl),
+					bytes.NewReader([]byte(payload)))
+				if err != nil {
+					log.Printf("[worker %d][req %d] request creation error: %v", workerID, reqID, err)
+					mu.Lock()
+					statusCounts[-1]++
+					mu.Unlock()
+					continue
+				}
+				req.Header.Set("Content-Type", "application/json")
+
+				resp, err := client.Do(req)
+				if err != nil {
+					log.Printf("[worker %d][req %d] HTTP request error: %v", workerID, reqID, err)
+					mu.Lock()
+					statusCounts[-1]++
+					mu.Unlock()
+					continue
+				}
+				io.Copy(io.Discard, resp.Body)
+				resp.Body.Close()
+
+				// Record status code
+				mu.Lock()
+				statusCounts[resp.StatusCode]++
+				mu.Unlock()
+			}
+		}(i + 1)
+	}
+
+	// ----- Status ticker (prints every second) -------------------------------
+	done := make(chan struct{})
+	tickerDone := make(chan struct{})
+	go func() {
+		ticker := time.NewTicker(1 * time.Second)
+		startTime := time.Now()
+		for {
+			select {
+			case <-ticker.C:
+				mu.Lock()
+				// Compute how many requests have completed so far
+				completed := 0
+				for _, cnt := range statusCounts {
+					completed += cnt
+				}
+				// Calculate duration and progress
+				duration := time.Since(startTime)
+				progress := completed * 100 / totalRequests
+				fmt.Printf("Duration: %v, Completed: %d%% requests\n", duration, progress)
+				mu.Unlock()
+			case <-done:
+				duration := time.Since(startTime)
+				fmt.Printf("Duration: %v, Completed: %d%% requests\n", duration, 100)
+				close(tickerDone)
+				return
+			}
+		}
+	}()
+
+	// Wait for all workers to finish
+	wg.Wait()
+	close(done)  // stops the status-update goroutine
+	<-tickerDone // give ticker time to finish / print
+
+	// ----- Summary ------------------------------------------------------------
+	fmt.Println("\n\n=== HTTP response code summary ===")
+	mu.Lock()
+	for code, cnt := range statusCounts {
+		if code == -1 {
+			fmt.Printf("Client-side errors (no HTTP response): %d\n", cnt)
+		} else {
+			fmt.Printf("%d : %d\n", code, cnt)
+		}
+	}
+	mu.Unlock()
+}

misc/simple-responder/simple-responder.go

@@ -153,6 +153,19 @@ func main() {
 
 	})
 
+	// llama-server compatibility: /completion
+	r.POST("/completion", func(c *gin.Context) {
+		c.Header("Content-Type", "application/json")
+		c.JSON(http.StatusOK, gin.H{
+			"responseMessage": *responseMessage,
+			"usage": gin.H{
+				"completion_tokens": 10,
+				"prompt_tokens":     25,
+				"total_tokens":      35,
+			},
+		})
+	})
+
 	// issue #41
 	r.POST("/v1/audio/transcriptions", func(c *gin.Context) {
 		// Parse the multipart form

proxy/config/config.go

@@ -1,4 +1,4 @@
-package proxy
+package config
 
 import (
 	"fmt"
@@ -6,7 +6,6 @@ import (
 	"os"
 	"regexp"
 	"runtime"
-	"slices"
 	"sort"
 	"strconv"
 	"strings"
@@ -17,101 +16,7 @@ import (
 
 const DEFAULT_GROUP_ID = "(default)"
 
-type ModelConfig struct {
+type MacroList map[string]any
-	Cmd           string   `yaml:"cmd"`
-	CmdStop       string   `yaml:"cmdStop"`
-	Proxy         string   `yaml:"proxy"`
-	Aliases       []string `yaml:"aliases"`
-	Env           []string `yaml:"env"`
-	CheckEndpoint string   `yaml:"checkEndpoint"`
-	UnloadAfter   int      `yaml:"ttl"`
-	Unlisted      bool     `yaml:"unlisted"`
-	UseModelName  string   `yaml:"useModelName"`
-
-	// #179 for /v1/models
-	Name        string `yaml:"name"`
-	Description string `yaml:"description"`
-
-	// Limit concurrency of HTTP requests to process
-	ConcurrencyLimit int `yaml:"concurrencyLimit"`
-
-	// Model filters see issue #174
-	Filters ModelFilters `yaml:"filters"`
-}
-
-func (m *ModelConfig) UnmarshalYAML(unmarshal func(interface{}) error) error {
-	type rawModelConfig ModelConfig
-	defaults := rawModelConfig{
-		Cmd:              "",
-		CmdStop:          "",
-		Proxy:            "http://localhost:${PORT}",
-		Aliases:          []string{},
-		Env:              []string{},
-		CheckEndpoint:    "/health",
-		UnloadAfter:      0,
-		Unlisted:         false,
-		UseModelName:     "",
-		ConcurrencyLimit: 0,
-		Name:             "",
-		Description:      "",
-	}
-
-	// the default cmdStop to taskkill /f /t /pid ${PID}
-	if runtime.GOOS == "windows" {
-		defaults.CmdStop = "taskkill /f /t /pid ${PID}"
-	}
-
-	if err := unmarshal(&defaults); err != nil {
-		return err
-	}
-
-	*m = ModelConfig(defaults)
-	return nil
-}
-
-func (m *ModelConfig) SanitizedCommand() ([]string, error) {
-	return SanitizeCommand(m.Cmd)
-}
-
-// ModelFilters see issue #174
-type ModelFilters struct {
-	StripParams string `yaml:"strip_params"`
-}
-
-func (m *ModelFilters) UnmarshalYAML(unmarshal func(interface{}) error) error {
-	type rawModelFilters ModelFilters
-	defaults := rawModelFilters{
-		StripParams: "",
-	}
-
-	if err := unmarshal(&defaults); err != nil {
-		return err
-	}
-
-	*m = ModelFilters(defaults)
-	return nil
-}
-
-func (f ModelFilters) SanitizedStripParams() ([]string, error) {
-	if f.StripParams == "" {
-		return nil, nil
-	}
-
-	params := strings.Split(f.StripParams, ",")
-	cleaned := make([]string, 0, len(params))
-
-	for _, param := range params {
-		trimmed := strings.TrimSpace(param)
-		if trimmed == "model" || trimmed == "" {
-			continue
-		}
-		cleaned = append(cleaned, trimmed)
-	}
-
-	// sort cleaned
-	slices.Sort(cleaned)
-	return cleaned, nil
-}
 
 type GroupConfig struct {
 	Swap       bool     `yaml:"swap"`
@@ -120,6 +25,11 @@ type GroupConfig struct {
 	Members    []string `yaml:"members"`
 }
 
+var (
+	macroNameRegex    = regexp.MustCompile(`^[a-zA-Z0-9_-]+$`)
+	macroPatternRegex = regexp.MustCompile(`\$\{([a-zA-Z0-9_-]+)\}`)
+)
+
 // set default values for GroupConfig
 func (c *GroupConfig) UnmarshalYAML(unmarshal func(interface{}) error) error {
 	type rawGroupConfig GroupConfig
@@ -138,6 +48,14 @@ func (c *GroupConfig) UnmarshalYAML(unmarshal func(interface{}) error) error {
 	return nil
 }
 
+type HooksConfig struct {
+	OnStartup HookOnStartup `yaml:"on_startup"`
+}
+
+type HookOnStartup struct {
+	Preload []string `yaml:"preload"`
+}
+
 type Config struct {
 	HealthCheckTimeout int                    `yaml:"healthCheckTimeout"`
 	LogRequests        bool                   `yaml:"logRequests"`
@@ -148,13 +66,16 @@ type Config struct {
 	Groups             map[string]GroupConfig `yaml:"groups"` /* key is group ID */
 
 	// for key/value replacements in model's cmd, cmdStop, proxy, checkEndPoint
-	Macros map[string]string `yaml:"macros"`
+	Macros MacroList `yaml:"macros"`
 
 	// map aliases to actual model IDs
 	aliases map[string]string
 
 	// automatic port assignments
 	StartPort int `yaml:"startPort"`
+
+	// hooks, see: #209
+	Hooks HooksConfig `yaml:"hooks"`
 }
 
 func (c *Config) RealModelName(search string) (string, bool) {
@@ -226,23 +147,12 @@ func LoadConfigFromReader(r io.Reader) (Config, error) {
 
 	- name must fit the regex ^[a-zA-Z0-9_-]+$
 	- names must be less than 64 characters (no reason, just cause)
-	- name can not be any reserved macros: PORT
+	- name can not be any reserved macros: PORT, MODEL_ID
 	- macro values must be less than 1024 characters
 	*/
-	macroNameRegex := regexp.MustCompile(`^[a-zA-Z0-9_-]+$`)
 	for macroName, macroValue := range config.Macros {
-		if len(macroName) >= 64 {
+		if err = validateMacro(macroName, macroValue); err != nil {
-			return Config{}, fmt.Errorf("macro name '%s' exceeds maximum length of 63 characters", macroName)
+			return Config{}, err
-		}
-		if !macroNameRegex.MatchString(macroName) {
-			return Config{}, fmt.Errorf("macro name '%s' contains invalid characters, must match pattern ^[a-zA-Z0-9_-]+$", macroName)
-		}
-		if len(macroValue) >= 1024 {
-			return Config{}, fmt.Errorf("macro value for '%s' exceeds maximum length of 1024 characters", macroName)
-		}
-		switch macroName {
-		case "PORT":
-			return Config{}, fmt.Errorf("macro name '%s' is reserved and cannot be used", macroName)
 		}
 	}
 
@@ -261,14 +171,34 @@ func LoadConfigFromReader(r io.Reader) (Config, error) {
 		modelConfig.Cmd = StripComments(modelConfig.Cmd)
 		modelConfig.CmdStop = StripComments(modelConfig.CmdStop)
 
+		// validate model macros
+		for macroName, macroValue := range modelConfig.Macros {
+			if err = validateMacro(macroName, macroValue); err != nil {
+				return Config{}, fmt.Errorf("model %s: %s", modelId, err.Error())
+			}
+		}
+
+		// Merge global config and model macros. Model macros take precedence
+		mergedMacros := make(MacroList)
+		for k, v := range config.Macros {
+			mergedMacros[k] = v
+		}
+		for k, v := range modelConfig.Macros {
+			mergedMacros[k] = v
+		}
+
+		mergedMacros["MODEL_ID"] = modelId
+
 		// go through model config fields: cmd, cmdStop, proxy, checkEndPoint and replace macros with macro values
-		for macroName, macroValue := range config.Macros {
+		for macroName, macroValue := range mergedMacros {
 			macroSlug := fmt.Sprintf("${%s}", macroName)
-			modelConfig.Cmd = strings.ReplaceAll(modelConfig.Cmd, macroSlug, macroValue)
+			// Convert macro value to string for command/string field substitution
-			modelConfig.CmdStop = strings.ReplaceAll(modelConfig.CmdStop, macroSlug, macroValue)
+			macroStr := fmt.Sprintf("%v", macroValue)
-			modelConfig.Proxy = strings.ReplaceAll(modelConfig.Proxy, macroSlug, macroValue)
+			modelConfig.Cmd = strings.ReplaceAll(modelConfig.Cmd, macroSlug, macroStr)
-			modelConfig.CheckEndpoint = strings.ReplaceAll(modelConfig.CheckEndpoint, macroSlug, macroValue)
+			modelConfig.CmdStop = strings.ReplaceAll(modelConfig.CmdStop, macroSlug, macroStr)
-			modelConfig.Filters.StripParams = strings.ReplaceAll(modelConfig.Filters.StripParams, macroSlug, macroValue)
+			modelConfig.Proxy = strings.ReplaceAll(modelConfig.Proxy, macroSlug, macroStr)
+			modelConfig.CheckEndpoint = strings.ReplaceAll(modelConfig.CheckEndpoint, macroSlug, macroStr)
+			modelConfig.Filters.StripParams = strings.ReplaceAll(modelConfig.Filters.StripParams, macroSlug, macroStr)
 		}
 
 		// enforce ${PORT} used in both cmd and proxy
@@ -282,20 +212,24 @@ func LoadConfigFromReader(r io.Reader) (Config, error) {
 			modelConfig.Cmd = strings.ReplaceAll(modelConfig.Cmd, "${PORT}", nextPortStr)
 			modelConfig.CmdStop = strings.ReplaceAll(modelConfig.CmdStop, "${PORT}", nextPortStr)
 			modelConfig.Proxy = strings.ReplaceAll(modelConfig.Proxy, "${PORT}", nextPortStr)
+
+			// add port to merged macros so it can be used in metadata
+			mergedMacros["PORT"] = nextPort
+
 			nextPort++
 		}
 
 		// make sure there are no unknown macros that have not been replaced
-		macroPattern := regexp.MustCompile(`\$\{([a-zA-Z0-9_-]+)\}`)
 		fieldMap := map[string]string{
-			"cmd":           modelConfig.Cmd,
+			"cmd":                 modelConfig.Cmd,
-			"cmdStop":       modelConfig.CmdStop,
+			"cmdStop":             modelConfig.CmdStop,
-			"proxy":         modelConfig.Proxy,
+			"proxy":               modelConfig.Proxy,
-			"checkEndpoint": modelConfig.CheckEndpoint,
+			"checkEndpoint":       modelConfig.CheckEndpoint,
+			"filters.stripParams": modelConfig.Filters.StripParams,
 		}
 
 		for fieldName, fieldValue := range fieldMap {
-			matches := macroPattern.FindAllStringSubmatch(fieldValue, -1)
+			matches := macroPatternRegex.FindAllStringSubmatch(fieldValue, -1)
 			for _, match := range matches {
 				macroName := match[1]
 				if macroName == "PID" && fieldName == "cmdStop" {
@@ -307,6 +241,15 @@ func LoadConfigFromReader(r io.Reader) (Config, error) {
 			}
 		}
 
+		// Apply macro substitution to metadata
+		if len(modelConfig.Metadata) > 0 {
+			substitutedMetadata, err := substituteMetadataMacros(modelConfig.Metadata, mergedMacros)
+			if err != nil {
+				return Config{}, fmt.Errorf("model %s metadata: %s", modelId, err.Error())
+			}
+			modelConfig.Metadata = substitutedMetadata.(map[string]any)
+		}
+
 		config.Models[modelId] = modelConfig
 	}
 
@@ -330,6 +273,22 @@ func LoadConfigFromReader(r io.Reader) (Config, error) {
 		}
 	}
 
+	// clean up hooks preload
+	if len(config.Hooks.OnStartup.Preload) > 0 {
+		var toPreload []string
+		for _, modelID := range config.Hooks.OnStartup.Preload {
+			modelID = strings.TrimSpace(modelID)
+			if modelID == "" {
+				continue
+			}
+			if real, found := config.RealModelName(modelID); found {
+				toPreload = append(toPreload, real)
+			}
+		}
+
+		config.Hooks.OnStartup.Preload = toPreload
+	}
+
 	return config, nil
 }
 
@@ -353,7 +312,7 @@ func AddDefaultGroupToConfig(config Config) Config {
 		}
 	} else {
 		// iterate over existing group members and add non-grouped models into the default group
-		for modelName, _ := range config.Models {
+		for modelName := range config.Models {
 			foundModel := false
 		found:
 			// search for the model in existing groups
@@ -425,3 +384,92 @@ func StripComments(cmdStr string) string {
 	}
 	return strings.Join(cleanedLines, "\n")
 }
+
+// validateMacro validates macro name and value constraints
+func validateMacro(name string, value any) error {
+	if len(name) >= 64 {
+		return fmt.Errorf("macro name '%s' exceeds maximum length of 63 characters", name)
+	}
+	if !macroNameRegex.MatchString(name) {
+		return fmt.Errorf("macro name '%s' contains invalid characters, must match pattern ^[a-zA-Z0-9_-]+$", name)
+	}
+
+	// Validate that value is a scalar type
+	switch v := value.(type) {
+	case string:
+		if len(v) >= 1024 {
+			return fmt.Errorf("macro value for '%s' exceeds maximum length of 1024 characters", name)
+		}
+	case int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64, float32, float64, bool:
+		// These types are allowed
+	default:
+		return fmt.Errorf("macro '%s' has invalid type %T, must be a scalar type (string, int, float, or bool)", name, value)
+	}
+
+	switch name {
+	case "PORT", "MODEL_ID":
+		return fmt.Errorf("macro name '%s' is reserved", name)
+	}
+
+	return nil
+}
+
+// substituteMetadataMacros recursively substitutes macros in metadata structures
+// Direct substitution (key: ${macro}) preserves the macro's type
+// Interpolated substitution (key: "text ${macro}") converts to string
+func substituteMetadataMacros(value any, macros MacroList) (any, error) {
+	switch v := value.(type) {
+	case string:
+		// Check if this is a direct macro substitution
+		if strings.HasPrefix(v, "${") && strings.HasSuffix(v, "}") && strings.Count(v, "${") == 1 {
+			macroName := v[2 : len(v)-1]
+			if macroValue, exists := macros[macroName]; exists {
+				return macroValue, nil
+			}
+			return nil, fmt.Errorf("unknown macro '${%s}' in metadata", macroName)
+		}
+
+		// Handle string interpolation
+		matches := macroPatternRegex.FindAllStringSubmatch(v, -1)
+		result := v
+		for _, match := range matches {
+			macroName := match[1]
+			macroValue, exists := macros[macroName]
+			if !exists {
+				return nil, fmt.Errorf("unknown macro '${%s}' in metadata", macroName)
+			}
+			// Convert macro value to string for interpolation
+			macroStr := fmt.Sprintf("%v", macroValue)
+			result = strings.ReplaceAll(result, match[0], macroStr)
+		}
+		return result, nil
+
+	case map[string]any:
+		// Recursively process map values
+		newMap := make(map[string]any)
+		for key, val := range v {
+			newVal, err := substituteMetadataMacros(val, macros)
+			if err != nil {
+				return nil, err
+			}
+			newMap[key] = newVal
+		}
+		return newMap, nil
+
+	case []any:
+		// Recursively process slice elements
+		newSlice := make([]any, len(v))
+		for i, val := range v {
+			newVal, err := substituteMetadataMacros(val, macros)
+			if err != nil {
+				return nil, err
+			}
+			newSlice[i] = newVal
+		}
+		return newSlice, nil
+
+	default:
+		// Return scalar types as-is
+		return value, nil
+	}
+}

proxy/config/config_posix_test.go

@@ -1,6 +1,6 @@
 //go:build !windows
 
-package proxy
+package config
 
 import (
 	"os"
@@ -100,6 +100,9 @@ func TestConfig_LoadPosix(t *testing.T) {
 	content := `
 macros:
   svr-path: "path/to/server"
+hooks:
+  on_startup:
+    preload: ["model1", "model2"]
 models:
   model1:
     cmd: path/to/cmd --arg1 one
@@ -160,9 +163,14 @@ groups:
 	expected := Config{
 		LogLevel:  "info",
 		StartPort: 5800,
-		Macros: map[string]string{
+		Macros: MacroList{
 			"svr-path": "path/to/server",
 		},
+		Hooks: HooksConfig{
+			OnStartup: HookOnStartup{
+				Preload: []string{"model1", "model2"},
+			},
+		},
 		Models: map[string]ModelConfig{
 			"model1": {
 				Cmd:           "path/to/cmd --arg1 one",

proxy/config/config_test.go

@@ -1,4 +1,4 @@
-package proxy
+package config
 
 import (
 	"slices"
@@ -65,18 +65,6 @@ models:
 	assert.Contains(t, err.Error(), "duplicate alias m1 found in model: model")
 }
 
-func TestConfig_ModelConfigSanitizedCommand(t *testing.T) {
-	config := &ModelConfig{
-		Cmd: `python model1.py \
-    --arg1 value1 \
-    --arg2 value2`,
-	}
-
-	args, err := config.SanitizedCommand()
-	assert.NoError(t, err)
-	assert.Equal(t, []string{"python", "model1.py", "--arg1", "value1", "--arg2", "value2"}, args)
-}
-
 func TestConfig_FindConfig(t *testing.T) {
 
 	// TODO?
@@ -207,15 +195,19 @@ macros:
   argOne: "--arg1"
   argTwo: "--arg2"
   autoPort: "--port ${PORT}"
+  overriddenByModelMacro: failed
 
 models:
   model1:
+    macros:
+      overriddenByModelMacro: success
     cmd: |
       ${svr-path} ${argTwo}
       # the automatic ${PORT} is replaced
       ${autoPort}
       ${argOne}
       --arg3 three
+      --overridden ${overriddenByModelMacro}
     cmdStop: |
       /path/to/stop.sh --port ${PORT} ${argTwo}
 `
@@ -224,13 +216,68 @@ models:
 	assert.NoError(t, err)
 	sanitizedCmd, err := SanitizeCommand(config.Models["model1"].Cmd)
 	assert.NoError(t, err)
-	assert.Equal(t, "path/to/server --arg2 --port 9990 --arg1 --arg3 three", strings.Join(sanitizedCmd, " "))
+	assert.Equal(t, "path/to/server --arg2 --port 9990 --arg1 --arg3 three --overridden success", strings.Join(sanitizedCmd, " "))
 
 	sanitizedCmdStop, err := SanitizeCommand(config.Models["model1"].CmdStop)
 	assert.NoError(t, err)
 	assert.Equal(t, "/path/to/stop.sh --port 9990 --arg2", strings.Join(sanitizedCmdStop, " "))
 }
 
+func TestConfig_MacroReservedNames(t *testing.T) {
+
+	tests := []struct {
+		name          string
+		config        string
+		expectedError string
+	}{
+		{
+			name: "global macro named PORT",
+			config: `
+macros:
+  PORT: "1111"
+`,
+			expectedError: "macro name 'PORT' is reserved",
+		},
+		{
+			name: "global macro named MODEL_ID",
+			config: `
+macros:
+  MODEL_ID: model1
+`,
+			expectedError: "macro name 'MODEL_ID' is reserved",
+		},
+		{
+			name: "model macro named PORT",
+			config: `
+models:
+  model1:
+    macros:
+      PORT: 1111
+`,
+			expectedError: "model model1: macro name 'PORT' is reserved",
+		},
+
+		{
+			name: "model macro named MODEL_ID",
+			config: `
+models:
+  model1:
+    macros:
+      MODEL_ID: model1
+`,
+			expectedError: "model model1: macro name 'MODEL_ID' is reserved",
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			_, err := LoadConfigFromReader(strings.NewReader(tt.config))
+			assert.NotNil(t, err)
+			assert.Equal(t, tt.expectedError, err.Error())
+		})
+	}
+}
+
 func TestConfig_MacroErrorOnUnknownMacros(t *testing.T) {
 	tests := []struct {
 		name    string
@@ -288,6 +335,20 @@ models:
   model1:
     cmd: "${svr-path} --port ${PORT}"
     checkEndpoint: "http://localhost:${unknownMacro}/health"
+`,
+		},
+		{
+			name:  "unknown macro in filters.stripParams",
+			field: "filters.stripParams",
+			content: `
+startPort: 9990
+macros:
+  svr-path: "path/to/server"
+models:
+  model1:
+    cmd: "${svr-path} --port ${PORT}"
+    filters:
+      stripParams: "model,${unknownMacro}"
 `,
 		},
 	}
@@ -295,38 +356,13 @@ models:
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
 			_, err := LoadConfigFromReader(strings.NewReader(tt.content))
-			assert.Error(t, err)
+			if assert.Error(t, err) {
-			assert.Contains(t, err.Error(), "unknown macro '${unknownMacro}' found in model1."+tt.field)
+				assert.Contains(t, err.Error(), "unknown macro '${unknownMacro}' found in model1."+tt.field)
+			}
 			//t.Log(err)
 		})
 	}
 }
-
-func TestConfig_ModelFilters(t *testing.T) {
-	content := `
-macros:
-  default_strip: "temperature, top_p"
-models:
-  model1:
-    cmd: path/to/cmd --port ${PORT}
-    filters:
-      strip_params: "model, top_k, ${default_strip}, , ,"
-`
-	config, err := LoadConfigFromReader(strings.NewReader(content))
-	assert.NoError(t, err)
-	modelConfig, ok := config.Models["model1"]
-	if !assert.True(t, ok) {
-		t.FailNow()
-	}
-
-	// make sure `model` and enmpty strings are not in the list
-	assert.Equal(t, "model, top_k, temperature, top_p, , ,", modelConfig.Filters.StripParams)
-	sanitized, err := modelConfig.Filters.SanitizedStripParams()
-	if assert.NoError(t, err) {
-		assert.Equal(t, []string{"temperature", "top_k", "top_p"}, sanitized)
-	}
-}
-
 func TestStripComments(t *testing.T) {
 	tests := []struct {
 		name     string
@@ -440,3 +476,284 @@ models:
 	expectedCmd := "/user/llama.cpp/build/bin/llama-server --port 9990 --model /path/to/model.gguf -ngl 99"
 	assert.Equal(t, expectedCmd, cmdStr, "Final command does not match expected structure")
 }
+
+func TestConfig_MacroModelId(t *testing.T) {
+	content := `
+startPort: 9000
+macros:
+  "docker-llama": docker run --name ${MODEL_ID} -p ${PORT}:8080 docker_img
+  "docker-stop": docker stop ${MODEL_ID}
+
+models:
+  model1:
+    cmd: /path/to/server -p ${PORT} -hf ${MODEL_ID}
+
+  model2:
+    cmd: ${docker-llama}
+    cmdStop: ${docker-stop}
+
+  author/model:F16:
+    cmd: /path/to/server -p ${PORT} -hf ${MODEL_ID}
+    cmdStop: stop
+`
+
+	config, err := LoadConfigFromReader(strings.NewReader(content))
+	assert.NoError(t, err)
+	sanitizedCmd, err := SanitizeCommand(config.Models["model1"].Cmd)
+	assert.NoError(t, err)
+	assert.Equal(t, "/path/to/server -p 9001 -hf model1", strings.Join(sanitizedCmd, " "))
+
+	assert.Equal(t, "docker stop ${MODEL_ID}", config.Macros["docker-stop"])
+
+	sanitizedCmd2, err := SanitizeCommand(config.Models["model2"].Cmd)
+	assert.NoError(t, err)
+	assert.Equal(t, "docker run --name model2 -p 9002:8080 docker_img", strings.Join(sanitizedCmd2, " "))
+
+	sanitizedCmdStop, err := SanitizeCommand(config.Models["model2"].CmdStop)
+	assert.NoError(t, err)
+	assert.Equal(t, "docker stop model2", strings.Join(sanitizedCmdStop, " "))
+
+	sanitizedCmd3, err := SanitizeCommand(config.Models["author/model:F16"].Cmd)
+	assert.NoError(t, err)
+	assert.Equal(t, "/path/to/server -p 9000 -hf author/model:F16", strings.Join(sanitizedCmd3, " "))
+}
+
+func TestConfig_TypedMacrosInMetadata(t *testing.T) {
+	content := `
+startPort: 10000
+macros:
+  PORT_NUM: 10001
+  TEMP: 0.7
+  ENABLED: true
+  NAME: "llama model"
+  CTX: 16384
+
+models:
+  test-model:
+    cmd: /path/to/server -p ${PORT}
+    metadata:
+      port: ${PORT_NUM}
+      temperature: ${TEMP}
+      enabled: ${ENABLED}
+      model_name: ${NAME}
+      context: ${CTX}
+      note: "Running on port ${PORT_NUM} with temp ${TEMP} and context ${CTX}"
+`
+
+	config, err := LoadConfigFromReader(strings.NewReader(content))
+	assert.NoError(t, err)
+
+	meta := config.Models["test-model"].Metadata
+	assert.NotNil(t, meta)
+
+	// Verify direct substitution preserves types
+	assert.Equal(t, 10001, meta["port"])
+	assert.Equal(t, 0.7, meta["temperature"])
+	assert.Equal(t, true, meta["enabled"])
+	assert.Equal(t, "llama model", meta["model_name"])
+	assert.Equal(t, 16384, meta["context"])
+
+	// Verify string interpolation converts to string
+	assert.Equal(t, "Running on port 10001 with temp 0.7 and context 16384", meta["note"])
+}
+
+func TestConfig_NestedStructuresInMetadata(t *testing.T) {
+	content := `
+startPort: 10000
+macros:
+  TEMP: 0.7
+
+models:
+  test-model:
+    cmd: /path/to/server -p ${PORT}
+    metadata:
+      config:
+        port: ${PORT}
+        temperature: ${TEMP}
+      tags: ["model:${MODEL_ID}", "port:${PORT}"]
+      nested:
+        deep:
+          value: ${TEMP}
+`
+
+	config, err := LoadConfigFromReader(strings.NewReader(content))
+	assert.NoError(t, err)
+
+	meta := config.Models["test-model"].Metadata
+	assert.NotNil(t, meta)
+
+	// Verify nested objects
+	configMap := meta["config"].(map[string]any)
+	assert.Equal(t, 10000, configMap["port"])
+	assert.Equal(t, 0.7, configMap["temperature"])
+
+	// Verify arrays
+	tags := meta["tags"].([]any)
+	assert.Equal(t, "model:test-model", tags[0])
+	assert.Equal(t, "port:10000", tags[1])
+
+	// Verify deeply nested structures
+	nested := meta["nested"].(map[string]any)
+	deep := nested["deep"].(map[string]any)
+	assert.Equal(t, 0.7, deep["value"])
+}
+
+func TestConfig_ModelLevelMacroPrecedenceInMetadata(t *testing.T) {
+	content := `
+startPort: 10000
+macros:
+  TEMP: 0.5
+  GLOBAL_VAL: "global"
+
+models:
+  test-model:
+    cmd: /path/to/server -p ${PORT}
+    macros:
+      TEMP: 0.9
+      LOCAL_VAL: "local"
+    metadata:
+      temperature: ${TEMP}
+      global: ${GLOBAL_VAL}
+      local: ${LOCAL_VAL}
+`
+
+	config, err := LoadConfigFromReader(strings.NewReader(content))
+	assert.NoError(t, err)
+
+	meta := config.Models["test-model"].Metadata
+	assert.NotNil(t, meta)
+
+	// Model-level macro should override global
+	assert.Equal(t, 0.9, meta["temperature"])
+	// Global macro should be accessible
+	assert.Equal(t, "global", meta["global"])
+	// Model-level macro should be accessible
+	assert.Equal(t, "local", meta["local"])
+}
+
+func TestConfig_UnknownMacroInMetadata(t *testing.T) {
+	content := `
+startPort: 10000
+models:
+  test-model:
+    cmd: /path/to/server -p ${PORT}
+    metadata:
+      value: ${UNKNOWN_MACRO}
+`
+
+	_, err := LoadConfigFromReader(strings.NewReader(content))
+	assert.Error(t, err)
+	assert.Contains(t, err.Error(), "test-model")
+	assert.Contains(t, err.Error(), "UNKNOWN_MACRO")
+}
+
+func TestConfig_InvalidMacroType(t *testing.T) {
+	content := `
+startPort: 10000
+macros:
+  INVALID:
+    nested: value
+
+models:
+  test-model:
+    cmd: /path/to/server -p ${PORT}
+`
+
+	_, err := LoadConfigFromReader(strings.NewReader(content))
+	assert.Error(t, err)
+	assert.Contains(t, err.Error(), "INVALID")
+	assert.Contains(t, err.Error(), "must be a scalar type")
+}
+
+func TestConfig_MacroTypeValidation(t *testing.T) {
+	tests := []struct {
+		name      string
+		yaml      string
+		shouldErr bool
+	}{
+		{
+			name: "string macro",
+			yaml: `
+startPort: 10000
+macros:
+  STR: "test"
+models:
+  test-model:
+    cmd: /path/to/server -p ${PORT}
+`,
+			shouldErr: false,
+		},
+		{
+			name: "int macro",
+			yaml: `
+startPort: 10000
+macros:
+  NUM: 42
+models:
+  test-model:
+    cmd: /path/to/server -p ${PORT}
+`,
+			shouldErr: false,
+		},
+		{
+			name: "float macro",
+			yaml: `
+startPort: 10000
+macros:
+  FLOAT: 3.14
+models:
+  test-model:
+    cmd: /path/to/server -p ${PORT}
+`,
+			shouldErr: false,
+		},
+		{
+			name: "bool macro",
+			yaml: `
+startPort: 10000
+macros:
+  BOOL: true
+models:
+  test-model:
+    cmd: /path/to/server -p ${PORT}
+`,
+			shouldErr: false,
+		},
+		{
+			name: "array macro (invalid)",
+			yaml: `
+startPort: 10000
+macros:
+  ARR: [1, 2, 3]
+models:
+  test-model:
+    cmd: /path/to/server -p ${PORT}
+`,
+			shouldErr: true,
+		},
+		{
+			name: "map macro (invalid)",
+			yaml: `
+startPort: 10000
+macros:
+  MAP:
+    key: value
+models:
+  test-model:
+    cmd: /path/to/server -p ${PORT}
+`,
+			shouldErr: true,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			_, err := LoadConfigFromReader(strings.NewReader(tt.yaml))
+			if tt.shouldErr {
+				assert.Error(t, err)
+			} else {
+				assert.NoError(t, err)
+			}
+		})
+	}
+}

proxy/config/config_windows_test.go

@@ -1,6 +1,6 @@
 //go:build windows
 
-package proxy
+package config
 
 import (
 	"os"
@@ -155,7 +155,7 @@ groups:
 	expected := Config{
 		LogLevel:  "info",
 		StartPort: 5800,
-		Macros: map[string]string{
+		Macros: MacroList{
 			"svr-path": "path/to/server",
 		},
 		Models: map[string]ModelConfig{

proxy/config/model_config.go

@@ -0,0 +1,125 @@
+package config
+
+import (
+	"errors"
+	"runtime"
+	"slices"
+	"strings"
+)
+
+type ModelConfig struct {
+	Cmd           string   `yaml:"cmd"`
+	CmdStop       string   `yaml:"cmdStop"`
+	Proxy         string   `yaml:"proxy"`
+	Aliases       []string `yaml:"aliases"`
+	Env           []string `yaml:"env"`
+	CheckEndpoint string   `yaml:"checkEndpoint"`
+	UnloadAfter   int      `yaml:"ttl"`
+	Unlisted      bool     `yaml:"unlisted"`
+	UseModelName  string   `yaml:"useModelName"`
+
+	// #179 for /v1/models
+	Name        string `yaml:"name"`
+	Description string `yaml:"description"`
+
+	// Limit concurrency of HTTP requests to process
+	ConcurrencyLimit int `yaml:"concurrencyLimit"`
+
+	// Model filters see issue #174
+	Filters ModelFilters `yaml:"filters"`
+
+	// Macros: see #264
+	// Model level macros take precedence over the global macros
+	Macros MacroList `yaml:"macros"`
+
+	// Metadata: see #264
+	// Arbitrary metadata that can be exposed through the API
+	Metadata map[string]any `yaml:"metadata"`
+}
+
+func (m *ModelConfig) UnmarshalYAML(unmarshal func(interface{}) error) error {
+	type rawModelConfig ModelConfig
+	defaults := rawModelConfig{
+		Cmd:              "",
+		CmdStop:          "",
+		Proxy:            "http://localhost:${PORT}",
+		Aliases:          []string{},
+		Env:              []string{},
+		CheckEndpoint:    "/health",
+		UnloadAfter:      0,
+		Unlisted:         false,
+		UseModelName:     "",
+		ConcurrencyLimit: 0,
+		Name:             "",
+		Description:      "",
+	}
+
+	// the default cmdStop to taskkill /f /t /pid ${PID}
+	if runtime.GOOS == "windows" {
+		defaults.CmdStop = "taskkill /f /t /pid ${PID}"
+	}
+
+	if err := unmarshal(&defaults); err != nil {
+		return err
+	}
+
+	*m = ModelConfig(defaults)
+	return nil
+}
+
+func (m *ModelConfig) SanitizedCommand() ([]string, error) {
+	return SanitizeCommand(m.Cmd)
+}
+
+// ModelFilters see issue #174
+type ModelFilters struct {
+	StripParams string `yaml:"stripParams"`
+}
+
+func (m *ModelFilters) UnmarshalYAML(unmarshal func(interface{}) error) error {
+	type rawModelFilters ModelFilters
+	defaults := rawModelFilters{
+		StripParams: "",
+	}
+
+	if err := unmarshal(&defaults); err != nil {
+		return err
+	}
+
+	// Try to unmarshal with the old field name for backwards compatibility
+	if defaults.StripParams == "" {
+		var legacy struct {
+			StripParams string `yaml:"strip_params"`
+		}
+		if legacyErr := unmarshal(&legacy); legacyErr != nil {
+			return errors.New("failed to unmarshal legacy filters.strip_params: " + legacyErr.Error())
+		}
+		defaults.StripParams = legacy.StripParams
+	}
+
+	*m = ModelFilters(defaults)
+	return nil
+}
+
+func (f ModelFilters) SanitizedStripParams() ([]string, error) {
+	if f.StripParams == "" {
+		return nil, nil
+	}
+
+	params := strings.Split(f.StripParams, ",")
+	cleaned := make([]string, 0, len(params))
+	seen := make(map[string]bool)
+
+	for _, param := range params {
+		trimmed := strings.TrimSpace(param)
+		if trimmed == "model" || trimmed == "" || seen[trimmed] {
+			continue
+		}
+		seen[trimmed] = true
+		cleaned = append(cleaned, trimmed)
+	}
+
+	// sort cleaned
+	slices.Sort(cleaned)
+	return cleaned, nil
+}

proxy/config/model_config_test.go

@@ -0,0 +1,54 @@
+package config
+
+import (
+	"fmt"
+	"strings"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestConfig_ModelConfigSanitizedCommand(t *testing.T) {
+	config := &ModelConfig{
+		Cmd: `python model1.py \
+    --arg1 value1 \
+    --arg2 value2`,
+	}
+
+	args, err := config.SanitizedCommand()
+	assert.NoError(t, err)
+	assert.Equal(t, []string{"python", "model1.py", "--arg1", "value1", "--arg2", "value2"}, args)
+}
+
+func TestConfig_ModelFilters(t *testing.T) {
+	content := `
+macros:
+  default_strip: "temperature, top_p"
+models:
+  model1:
+    cmd: path/to/cmd --port ${PORT}
+    filters:
+      # macros inserted and list is cleaned of duplicates and empty strings
+      stripParams: "model, top_k, top_k, temperature, ${default_strip}, , ,"
+  # check for strip_params (legacy field name) compatibility
+  legacy:
+    cmd: path/to/cmd --port ${PORT}
+    filters:
+      strip_params: "model, top_k, top_k, temperature, ${default_strip}, , ,"
+`
+	config, err := LoadConfigFromReader(strings.NewReader(content))
+	assert.NoError(t, err)
+	for modelId, modelConfig := range config.Models {
+		t.Run(fmt.Sprintf("Testing macros in filters for model %s", modelId), func(t *testing.T) {
+			assert.Equal(t, "model, top_k, top_k, temperature, temperature, top_p, , ,", modelConfig.Filters.StripParams)
+			sanitized, err := modelConfig.Filters.SanitizedStripParams()
+			if assert.NoError(t, err) {
+				// model has been removed
+				// empty strings have been removed
+				// duplicates have been removed
+				assert.Equal(t, []string{"temperature", "top_k", "top_p"}, sanitized)
+			}
+		})
+	}
+
+}

proxy/discardWriter.go

@@ -0,0 +1,27 @@
+package proxy
+
+import "net/http"
+
+// Custom discard writer that implements http.ResponseWriter but just discards everything
+type DiscardWriter struct {
+	header http.Header
+	status int
+}
+
+func (w *DiscardWriter) Header() http.Header {
+	if w.header == nil {
+		w.header = make(http.Header)
+	}
+	return w.header
+}
+
+func (w *DiscardWriter) Write(data []byte) (int, error) {
+	return len(data), nil
+}
+
+func (w *DiscardWriter) WriteHeader(code int) {
+	w.status = code
+}
+
+// Satisfy the http.Flusher interface for streaming responses
+func (w *DiscardWriter) Flush() {}

proxy/events.go

@@ -7,6 +7,7 @@ const ChatCompletionStatsEventID = 0x02
 const ConfigFileChangedEventID = 0x03
 const LogDataEventID = 0x04
 const TokenMetricsEventID = 0x05
+const ModelPreloadedEventID = 0x06
 
 type ProcessStateChangeEvent struct {
 	ProcessName string
@@ -48,3 +49,12 @@ type LogDataEvent struct {
 func (e LogDataEvent) Type() uint32 {
 	return LogDataEventID
 }
+
+type ModelPreloadedEvent struct {
+	ModelName string
+	Success   bool
+}
+
+func (e ModelPreloadedEvent) Type() uint32 {
+	return ModelPreloadedEventID
+}

proxy/helpers_test.go

@@ -9,13 +9,15 @@ import (
 	"testing"
 
 	"github.com/gin-gonic/gin"
+	"github.com/mostlygeek/llama-swap/proxy/config"
 	"gopkg.in/yaml.v3"
 )
 
 var (
-	nextTestPort int = 12000
+	nextTestPort        int = 12000
-	portMutex    sync.Mutex
+	portMutex           sync.Mutex
-	testLogger   = NewLogMonitorWriter(os.Stdout)
+	testLogger          = NewLogMonitorWriter(os.Stdout)
+	simpleResponderPath = getSimpleResponderPath()
 )
 
 // Check if the binary exists
@@ -64,20 +66,18 @@ func getTestPort() int {
 	return port
 }
 
-func getTestSimpleResponderConfig(expectedMessage string) ModelConfig {
+func getTestSimpleResponderConfig(expectedMessage string) config.ModelConfig {
 	return getTestSimpleResponderConfigPort(expectedMessage, getTestPort())
 }
 
-func getTestSimpleResponderConfigPort(expectedMessage string, port int) ModelConfig {
+func getTestSimpleResponderConfigPort(expectedMessage string, port int) config.ModelConfig {
-	binaryPath := getSimpleResponderPath()
-
 	// Create a YAML string with just the values we want to set
 	yamlStr := fmt.Sprintf(`
 cmd: '%s --port %d --silent --respond %s'
 proxy: "http://127.0.0.1:%d"
-`, binaryPath, port, expectedMessage, port)
+`, simpleResponderPath, port, expectedMessage, port)
 
-	var cfg ModelConfig
+	var cfg config.ModelConfig
 	if err := yaml.Unmarshal([]byte(yamlStr), &cfg); err != nil {
 		panic(fmt.Sprintf("failed to unmarshal test config: %v in [%s]", err, yamlStr))
 	}

proxy/metrics_middleware.go

@@ -5,12 +5,20 @@ import (
 	"fmt"
 	"io"
 	"net/http"
+	"strings"
 	"time"
 
 	"github.com/gin-gonic/gin"
 	"github.com/tidwall/gjson"
 )
 
+type MetricsRecorder struct {
+	metricsMonitor *MetricsMonitor
+	realModelName  string
+	//	isStreaming    bool
+	startTime time.Time
+}
+
 // MetricsMiddleware sets up the MetricsResponseWriter for capturing upstream requests
 func MetricsMiddleware(pm *ProxyManager) gin.HandlerFunc {
 	return func(c *gin.Context) {
@@ -41,57 +49,63 @@ func MetricsMiddleware(pm *ProxyManager) gin.HandlerFunc {
 			metricsRecorder: &MetricsRecorder{
 				metricsMonitor: pm.metricsMonitor,
 				realModelName:  realModelName,
-				isStreaming:    gjson.GetBytes(bodyBytes, "stream").Bool(),
 				startTime:      time.Now(),
 			},
 		}
 		c.Writer = writer
 		c.Next()
 
-		rec := writer.metricsRecorder
+		// check for streaming response
-		rec.processBody(writer.body)
+		if strings.Contains(c.Writer.Header().Get("Content-Type"), "text/event-stream") {
-	}
+			writer.metricsRecorder.processStreamingResponse(writer.body)
-}
+		} else {
-
+			writer.metricsRecorder.processNonStreamingResponse(writer.body)
-type MetricsRecorder struct {
+		}
-	metricsMonitor *MetricsMonitor
-	realModelName  string
-	isStreaming    bool
-	startTime      time.Time
-}
-
-// processBody handles response processing after request completes
-func (rec *MetricsRecorder) processBody(body []byte) {
-	if rec.isStreaming {
-		rec.processStreamingResponse(body)
-	} else {
-		rec.processNonStreamingResponse(body)
 	}
 }
 
 func (rec *MetricsRecorder) parseAndRecordMetrics(jsonData gjson.Result) bool {
 	usage := jsonData.Get("usage")
-	if !usage.Exists() {
+	timings := jsonData.Get("timings")
+	if !usage.Exists() && !timings.Exists() {
 		return false
 	}
 
 	// default values
-	outputTokens := int(jsonData.Get("usage.completion_tokens").Int())
+	cachedTokens := -1 // unknown or missing data
-	inputTokens := int(jsonData.Get("usage.prompt_tokens").Int())
+	outputTokens := 0
+	inputTokens := 0
+
+	// timings data
 	tokensPerSecond := -1.0
+	promptPerSecond := -1.0
 	durationMs := int(time.Since(rec.startTime).Milliseconds())
 
+	if usage.Exists() {
+		outputTokens = int(jsonData.Get("usage.completion_tokens").Int())
+		inputTokens = int(jsonData.Get("usage.prompt_tokens").Int())
+	}
+
 	// use llama-server's timing data for tok/sec and duration as it is more accurate
-	if timings := jsonData.Get("timings"); timings.Exists() {
+	if timings.Exists() {
+		inputTokens = int(jsonData.Get("timings.prompt_n").Int())
+		outputTokens = int(jsonData.Get("timings.predicted_n").Int())
+		promptPerSecond = jsonData.Get("timings.prompt_per_second").Float()
 		tokensPerSecond = jsonData.Get("timings.predicted_per_second").Float()
 		durationMs = int(jsonData.Get("timings.prompt_ms").Float() + jsonData.Get("timings.predicted_ms").Float())
+
+		if cachedValue := jsonData.Get("timings.cache_n"); cachedValue.Exists() {
+			cachedTokens = int(cachedValue.Int())
+		}
 	}
 
 	rec.metricsMonitor.addMetrics(TokenMetrics{
 		Timestamp:       time.Now(),
 		Model:           rec.realModelName,
+		CachedTokens:    cachedTokens,
 		InputTokens:     inputTokens,
 		OutputTokens:    outputTokens,
+		PromptPerSecond: promptPerSecond,
 		TokensPerSecond: tokensPerSecond,
 		DurationMs:      durationMs,
 	})

proxy/metrics_monitor.go

@@ -6,6 +6,7 @@ import (
 	"time"
 
 	"github.com/mostlygeek/llama-swap/event"
+	"github.com/mostlygeek/llama-swap/proxy/config"
 )
 
 // TokenMetrics represents parsed token statistics from llama-server logs
@@ -13,8 +14,10 @@ type TokenMetrics struct {
 	ID              int       `json:"id"`
 	Timestamp       time.Time `json:"timestamp"`
 	Model           string    `json:"model"`
+	CachedTokens    int       `json:"cache_tokens"`
 	InputTokens     int       `json:"input_tokens"`
 	OutputTokens    int       `json:"output_tokens"`
+	PromptPerSecond float64   `json:"prompt_per_second"`
 	TokensPerSecond float64   `json:"tokens_per_second"`
 	DurationMs      int       `json:"duration_ms"`
 }
@@ -36,7 +39,7 @@ type MetricsMonitor struct {
 	nextID     int
 }
 
-func NewMetricsMonitor(config *Config) *MetricsMonitor {
+func NewMetricsMonitor(config *config.Config) *MetricsMonitor {
 	maxMetrics := config.MetricsMaxInMemory
 	if maxMetrics <= 0 {
 		maxMetrics = 1000 // Default fallback
@@ -60,7 +63,6 @@ func (mp *MetricsMonitor) addMetrics(metric TokenMetrics) {
 	if len(mp.metrics) > mp.maxMetrics {
 		mp.metrics = mp.metrics[len(mp.metrics)-mp.maxMetrics:]
 	}
-
 	event.Emit(TokenMetricsEvent{Metrics: metric})
 }
 

proxy/process.go

@@ -5,6 +5,7 @@ import (
 	"errors"
 	"fmt"
 	"io"
+	"net"
 	"net/http"
 	"net/url"
 	"os/exec"
@@ -15,6 +16,7 @@ import (
 	"time"
 
 	"github.com/mostlygeek/llama-swap/event"
+	"github.com/mostlygeek/llama-swap/proxy/config"
 )
 
 type ProcessState string
@@ -38,7 +40,7 @@ const (
 
 type Process struct {
 	ID     string
-	config ModelConfig
+	config config.ModelConfig
 	cmd    *exec.Cmd
 
 	// PR #155 called to cancel the upstream process
@@ -73,7 +75,7 @@ type Process struct {
 	failedStartCount int
 }
 
-func NewProcess(ID string, healthCheckTimeout int, config ModelConfig, processLogger *LogMonitor, proxyLogger *LogMonitor) *Process {
+func NewProcess(ID string, healthCheckTimeout int, config config.ModelConfig, processLogger *LogMonitor, proxyLogger *LogMonitor) *Process {
 	concurrentLimit := 10
 	if config.ConcurrencyLimit > 0 {
 		concurrentLimit = config.ConcurrencyLimit
@@ -363,8 +365,18 @@ func (p *Process) stopCommand() {
 }
 
 func (p *Process) checkHealthEndpoint(healthURL string) error {
+
 	client := &http.Client{
-		Timeout: 500 * time.Millisecond,
+		// wait a short time for a tcp connection to be established
+		Transport: &http.Transport{
+			DialContext: (&net.Dialer{
+				Timeout: 500 * time.Millisecond,
+			}).DialContext,
+		},
+
+		// give a long time to respond to the health check endpoint
+		// after the connection is established. See issue: 276
+		Timeout: 5000 * time.Millisecond,
 	}
 
 	req, err := http.NewRequest("GET", healthURL, nil)
@@ -447,6 +459,10 @@ func (p *Process) ProxyRequest(w http.ResponseWriter, r *http.Request) {
 			w.Header().Add(k, v)
 		}
 	}
+	// prevent nginx from buffering streaming responses (e.g., SSE)
+	if strings.Contains(strings.ToLower(resp.Header.Get("Content-Type")), "text/event-stream") {
+		w.Header().Set("X-Accel-Buffering", "no")
+	}
 	w.WriteHeader(resp.StatusCode)
 
 	// faster than io.Copy when streaming
@@ -524,7 +540,7 @@ func (p *Process) cmdStopUpstreamProcess() error {
 
 	if p.config.CmdStop != "" {
 		// replace ${PID} with the pid of the process
-		stopArgs, err := SanitizeCommand(strings.ReplaceAll(p.config.CmdStop, "${PID}", fmt.Sprintf("%d", p.cmd.Process.Pid)))
+		stopArgs, err := config.SanitizeCommand(strings.ReplaceAll(p.config.CmdStop, "${PID}", fmt.Sprintf("%d", p.cmd.Process.Pid)))
 		if err != nil {
 			p.proxyLogger.Errorf("<%s> Failed to sanitize stop command: %v", p.ID, err)
 			return err

proxy/process_test.go

@@ -10,6 +10,7 @@ import (
 	"testing"
 	"time"
 
+	"github.com/mostlygeek/llama-swap/proxy/config"
 	"github.com/stretchr/testify/assert"
 )
 
@@ -90,7 +91,7 @@ func TestProcess_WaitOnMultipleStarts(t *testing.T) {
 // test that the automatic start returns the expected error type
 func TestProcess_BrokenModelConfig(t *testing.T) {
 	// Create a process configuration
-	config := ModelConfig{
+	config := config.ModelConfig{
 		Cmd:           "nonexistent-command",
 		Proxy:         "http://127.0.0.1:9913",
 		CheckEndpoint: "/health",
@@ -325,7 +326,7 @@ func TestProcess_ExitInterruptsHealthCheck(t *testing.T) {
 
 	// should run and exit but interrupt the long checkHealthTimeout
 	checkHealthTimeout := 5
-	config := ModelConfig{
+	config := config.ModelConfig{
 		Cmd:           "sleep 1",
 		Proxy:         "http://127.0.0.1:9913",
 		CheckEndpoint: "/health",
@@ -402,7 +403,7 @@ func TestProcess_ForceStopWithKill(t *testing.T) {
 	binaryPath := getSimpleResponderPath()
 	port := getTestPort()
 
-	config := ModelConfig{
+	conf := config.ModelConfig{
 		// note --ignore-sig-term which ignores the SIGTERM signal so a SIGKILL must be sent
 		// to force the process to exit
 		Cmd:           fmt.Sprintf("%s --port %d --respond %s --silent --ignore-sig-term", binaryPath, port, expectedMessage),
@@ -410,7 +411,7 @@ func TestProcess_ForceStopWithKill(t *testing.T) {
 		CheckEndpoint: "/health",
 	}
 
-	process := NewProcess("stop_immediate", 2, config, debugLogger, debugLogger)
+	process := NewProcess("stop_immediate", 2, conf, debugLogger, debugLogger)
 	defer process.Stop()
 
 	// reduce to make testing go faster
@@ -450,15 +451,15 @@ func TestProcess_ForceStopWithKill(t *testing.T) {
 }
 
 func TestProcess_StopCmd(t *testing.T) {
-	config := getTestSimpleResponderConfig("test_stop_cmd")
+	conf := getTestSimpleResponderConfig("test_stop_cmd")
 
 	if runtime.GOOS == "windows" {
-		config.CmdStop = "taskkill /f /t /pid ${PID}"
+		conf.CmdStop = "taskkill /f /t /pid ${PID}"
 	} else {
-		config.CmdStop = "kill -TERM ${PID}"
+		conf.CmdStop = "kill -TERM ${PID}"
 	}
 
-	process := NewProcess("testStopCmd", 2, config, debugLogger, debugLogger)
+	process := NewProcess("testStopCmd", 2, conf, debugLogger, debugLogger)
 	defer process.Stop()
 
 	err := process.start()
@@ -470,15 +471,15 @@ func TestProcess_StopCmd(t *testing.T) {
 
 func TestProcess_EnvironmentSetCorrectly(t *testing.T) {
 	expectedMessage := "test_env_not_emptied"
-	config := getTestSimpleResponderConfig(expectedMessage)
+	conf := getTestSimpleResponderConfig(expectedMessage)
 
 	// ensure that the the default config does not blank out the inherited environment
-	configWEnv := config
+	configWEnv := conf
 
 	// ensure the additiona variables are appended to the process' environment
 	configWEnv.Env = append(configWEnv.Env, "TEST_ENV1=1", "TEST_ENV2=2")
 
-	process1 := NewProcess("env_test", 2, config, debugLogger, debugLogger)
+	process1 := NewProcess("env_test", 2, conf, debugLogger, debugLogger)
 	process2 := NewProcess("env_test", 2, configWEnv, debugLogger, debugLogger)
 
 	process1.start()

proxy/processgroup.go

@@ -5,12 +5,14 @@ import (
 	"net/http"
 	"slices"
 	"sync"
+
+	"github.com/mostlygeek/llama-swap/proxy/config"
 )
 
 type ProcessGroup struct {
 	sync.Mutex
 
-	config     Config
+	config     config.Config
 	id         string
 	swap       bool
 	exclusive  bool
@@ -24,7 +26,7 @@ type ProcessGroup struct {
 	lastUsedProcess string
 }
 
-func NewProcessGroup(id string, config Config, proxyLogger *LogMonitor, upstreamLogger *LogMonitor) *ProcessGroup {
+func NewProcessGroup(id string, config config.Config, proxyLogger *LogMonitor, upstreamLogger *LogMonitor) *ProcessGroup {
 	groupConfig, ok := config.Groups[id]
 	if !ok {
 		panic("Unable to find configuration for group id: " + id)
@@ -60,10 +62,20 @@ func (pg *ProcessGroup) ProxyRequest(modelID string, writer http.ResponseWriter,
 	if pg.swap {
 		pg.Lock()
 		if pg.lastUsedProcess != modelID {
+
+			// is there something already running?
 			if pg.lastUsedProcess != "" {
 				pg.processes[pg.lastUsedProcess].Stop()
 			}
+
+			// wait for the request to the new model to be fully handled
+			// and prevent race conditions see issue #277
+			pg.processes[modelID].ProxyRequest(writer, request)
 			pg.lastUsedProcess = modelID
+
+			// short circuit and exit
+			pg.Unlock()
+			return nil
 		}
 		pg.Unlock()
 	}
@@ -76,6 +88,29 @@ func (pg *ProcessGroup) HasMember(modelName string) bool {
 	return slices.Contains(pg.config.Groups[pg.id].Members, modelName)
 }
 
+func (pg *ProcessGroup) StopProcess(modelID string, strategy StopStrategy) error {
+	pg.Lock()
+
+	process, exists := pg.processes[modelID]
+	if !exists {
+		pg.Unlock()
+		return fmt.Errorf("process not found for %s", modelID)
+	}
+
+	if pg.lastUsedProcess == modelID {
+		pg.lastUsedProcess = ""
+	}
+	pg.Unlock()
+
+	switch strategy {
+	case StopImmediately:
+		process.StopImmediately()
+	default:
+		process.Stop()
+	}
+	return nil
+}
+
 func (pg *ProcessGroup) StopProcesses(strategy StopStrategy) {
 	pg.Lock()
 	defer pg.Unlock()

proxy/processgroup_test.go

@@ -4,21 +4,23 @@ import (
 	"bytes"
 	"net/http"
 	"net/http/httptest"
+	"sync"
 	"testing"
 
+	"github.com/mostlygeek/llama-swap/proxy/config"
 	"github.com/stretchr/testify/assert"
 )
 
-var processGroupTestConfig = AddDefaultGroupToConfig(Config{
+var processGroupTestConfig = config.AddDefaultGroupToConfig(config.Config{
 	HealthCheckTimeout: 15,
-	Models: map[string]ModelConfig{
+	Models: map[string]config.ModelConfig{
 		"model1": getTestSimpleResponderConfig("model1"),
 		"model2": getTestSimpleResponderConfig("model2"),
 		"model3": getTestSimpleResponderConfig("model3"),
 		"model4": getTestSimpleResponderConfig("model4"),
 		"model5": getTestSimpleResponderConfig("model5"),
 	},
-	Groups: map[string]GroupConfig{
+	Groups: map[string]config.GroupConfig{
 		"G1": {
 			Swap:      true,
 			Exclusive: true,
@@ -33,7 +35,7 @@ var processGroupTestConfig = AddDefaultGroupToConfig(Config{
 })
 
 func TestProcessGroup_DefaultHasCorrectModel(t *testing.T) {
-	pg := NewProcessGroup(DEFAULT_GROUP_ID, processGroupTestConfig, testLogger, testLogger)
+	pg := NewProcessGroup(config.DEFAULT_GROUP_ID, processGroupTestConfig, testLogger, testLogger)
 	assert.True(t, pg.HasMember("model5"))
 }
 
@@ -44,32 +46,49 @@ func TestProcessGroup_HasMember(t *testing.T) {
 	assert.False(t, pg.HasMember("model3"))
 }
 
-func TestProcessGroup_ProxyRequestSwapIsTrue(t *testing.T) {
+// TestProcessGroup_ProxyRequestSwapIsTrueParallel tests that when swap is true
+// and multiple requests are made in parallel, only one process is running at a time.
+func TestProcessGroup_ProxyRequestSwapIsTrueParallel(t *testing.T) {
+	var processGroupTestConfig = config.AddDefaultGroupToConfig(config.Config{
+		HealthCheckTimeout: 15,
+		Models: map[string]config.ModelConfig{
+			// use the same listening so if a model is already running, it will fail
+			// this is a way to test that swap isolation is working
+			// properly when there are parallel requests made at the
+			// same time.
+			"model1": getTestSimpleResponderConfigPort("model1", 9832),
+			"model2": getTestSimpleResponderConfigPort("model2", 9832),
+			"model3": getTestSimpleResponderConfigPort("model3", 9832),
+			"model4": getTestSimpleResponderConfigPort("model4", 9832),
+			"model5": getTestSimpleResponderConfigPort("model5", 9832),
+		},
+		Groups: map[string]config.GroupConfig{
+			"G1": {
+				Swap:    true,
+				Members: []string{"model1", "model2", "model3", "model4", "model5"},
+			},
+		},
+	})
+
 	pg := NewProcessGroup("G1", processGroupTestConfig, testLogger, testLogger)
 	defer pg.StopProcesses(StopWaitForInflightRequest)
 
-	tests := []string{"model1", "model2"}
+	tests := []string{"model1", "model2", "model3", "model4", "model5"}
 
+	var wg sync.WaitGroup
+
+	wg.Add(len(tests))
 	for _, modelName := range tests {
-		t.Run(modelName, func(t *testing.T) {
+		go func(modelName string) {
-			reqBody := `{"x", "y"}`
+			defer wg.Done()
-			req := httptest.NewRequest("POST", "/v1/chat/completions", bytes.NewBufferString(reqBody))
+			req := httptest.NewRequest("POST", "/v1/chat/completions", nil)
 			w := httptest.NewRecorder()
-
 			assert.NoError(t, pg.ProxyRequest(modelName, w, req))
 			assert.Equal(t, http.StatusOK, w.Code)
 			assert.Contains(t, w.Body.String(), modelName)
-
+		}(modelName)
-			// make sure only one process is in the running state
-			count := 0
-			for _, process := range pg.processes {
-				if process.CurrentState() == StateReady {
-					count++
-				}
-			}
-			assert.Equal(t, 1, count)
-		})
 	}
+	wg.Wait()
 }
 
 func TestProcessGroup_ProxyRequestSwapIsFalse(t *testing.T) {

proxy/proxymanager.go

@@ -8,12 +8,15 @@ import (
 	"mime/multipart"
 	"net/http"
 	"os"
+	"sort"
 	"strconv"
 	"strings"
 	"sync"
 	"time"
 
 	"github.com/gin-gonic/gin"
+	"github.com/mostlygeek/llama-swap/event"
+	"github.com/mostlygeek/llama-swap/proxy/config"
 	"github.com/tidwall/gjson"
 	"github.com/tidwall/sjson"
 )
@@ -25,7 +28,7 @@ const (
 type ProxyManager struct {
 	sync.Mutex
 
-	config    Config
+	config    config.Config
 	ginEngine *gin.Engine
 
 	// logging
@@ -42,7 +45,7 @@ type ProxyManager struct {
 	shutdownCancel context.CancelFunc
 }
 
-func New(config Config) *ProxyManager {
+func New(config config.Config) *ProxyManager {
 	// set up loggers
 	stdoutLogger := NewLogMonitorWriter(os.Stdout)
 	upstreamLogger := NewLogMonitorWriter(stdoutLogger)
@@ -95,6 +98,35 @@ func New(config Config) *ProxyManager {
 	}
 
 	pm.setupGinEngine()
+
+	// run any startup hooks
+	if len(config.Hooks.OnStartup.Preload) > 0 {
+		// do it in the background, don't block startup -- not sure if good idea yet
+		go func() {
+			discardWriter := &DiscardWriter{}
+			for _, realModelName := range config.Hooks.OnStartup.Preload {
+				proxyLogger.Infof("Preloading model: %s", realModelName)
+				processGroup, _, err := pm.swapProcessGroup(realModelName)
+
+				if err != nil {
+					event.Emit(ModelPreloadedEvent{
+						ModelName: realModelName,
+						Success:   false,
+					})
+					proxyLogger.Errorf("Failed to preload model %s: %v", realModelName, err)
+					continue
+				} else {
+					req, _ := http.NewRequest("GET", "/", nil)
+					processGroup.ProxyRequest(realModelName, discardWriter, req)
+					event.Emit(ModelPreloadedEvent{
+						ModelName: realModelName,
+						Success:   true,
+					})
+				}
+			}
+		}()
+	}
+
 	return pm
 }
 
@@ -160,11 +192,20 @@ func (pm *ProxyManager) setupGinEngine() {
 	// Support legacy /v1/completions api, see issue #12
 	pm.ginEngine.POST("/v1/completions", mm, pm.proxyOAIHandler)
 
-	// Support embeddings
+	// Support embeddings and reranking
 	pm.ginEngine.POST("/v1/embeddings", mm, pm.proxyOAIHandler)
+
+	// llama-server's /reranking endpoint + aliases
+	pm.ginEngine.POST("/reranking", mm, pm.proxyOAIHandler)
+	pm.ginEngine.POST("/rerank", mm, pm.proxyOAIHandler)
 	pm.ginEngine.POST("/v1/rerank", mm, pm.proxyOAIHandler)
 	pm.ginEngine.POST("/v1/reranking", mm, pm.proxyOAIHandler)
-	pm.ginEngine.POST("/rerank", mm, pm.proxyOAIHandler)
+
+	// llama-server's /infill endpoint for code infilling
+	pm.ginEngine.POST("/infill", mm, pm.proxyOAIHandler)
+
+	// llama-server's /completion endpoint
+	pm.ginEngine.POST("/completion", mm, pm.proxyOAIHandler)
 
 	// Support audio/speech endpoint
 	pm.ginEngine.POST("/v1/audio/speech", pm.proxyOAIHandler)
@@ -187,8 +228,7 @@ func (pm *ProxyManager) setupGinEngine() {
 	pm.ginEngine.GET("/upstream", func(c *gin.Context) {
 		c.Redirect(http.StatusFound, "/ui/models")
 	})
-	pm.ginEngine.Any("/upstream/:model_id/*upstreamPath", pm.proxyToUpstream)
+	pm.ginEngine.Any("/upstream/*upstreamPath", pm.proxyToUpstream)
-
 	pm.ginEngine.GET("/unload", pm.unloadAllModelsHandler)
 	pm.ginEngine.GET("/running", pm.listRunningProcessesHandler)
 	pm.ginEngine.GET("/health", func(c *gin.Context) {
@@ -330,9 +370,23 @@ func (pm *ProxyManager) listModelsHandler(c *gin.Context) {
 			record["description"] = desc
 		}
 
+		// Add metadata if present
+		if len(modelConfig.Metadata) > 0 {
+			record["meta"] = gin.H{
+				"llamaswap": modelConfig.Metadata,
+			}
+		}
+
 		data = append(data, record)
 	}
 
+	// Sort by the "id" key
+	sort.Slice(data, func(i, j int) bool {
+		si, _ := data[i]["id"].(string)
+		sj, _ := data[j]["id"].(string)
+		return si < sj
+	})
+
 	// Set CORS headers if origin exists
 	if origin := c.GetHeader("Origin"); origin != "" {
 		c.Header("Access-Control-Allow-Origin", origin)
@@ -346,22 +400,69 @@ func (pm *ProxyManager) listModelsHandler(c *gin.Context) {
 }
 
 func (pm *ProxyManager) proxyToUpstream(c *gin.Context) {
-	requestedModel := c.Param("model_id")
+	upstreamPath := c.Param("upstreamPath")
 
-	if requestedModel == "" {
+	// split the upstream path by / and search for the model name
+	parts := strings.Split(strings.TrimSpace(upstreamPath), "/")
+	if len(parts) == 0 {
 		pm.sendErrorResponse(c, http.StatusBadRequest, "model id required in path")
 		return
 	}
 
-	processGroup, _, err := pm.swapProcessGroup(requestedModel)
+	modelFound := false
+	searchModelName := ""
+	var modelName, remainingPath string
+	for i, part := range parts {
+		if parts[i] == "" {
+			continue
+		}
+
+		if searchModelName == "" {
+			searchModelName = part
+		} else {
+			searchModelName = searchModelName + "/" + parts[i]
+		}
+
+		if real, ok := pm.config.RealModelName(searchModelName); ok {
+			modelName = real
+			remainingPath = "/" + strings.Join(parts[i+1:], "/")
+			modelFound = true
+
+			// Check if this is exactly a model name with no additional path
+			// and doesn't end with a trailing slash
+			if remainingPath == "/" && !strings.HasSuffix(upstreamPath, "/") {
+				// Build new URL with query parameters preserved
+				newPath := "/upstream/" + searchModelName + "/"
+				if c.Request.URL.RawQuery != "" {
+					newPath += "?" + c.Request.URL.RawQuery
+				}
+
+				// Use 308 for non-GET/HEAD requests to preserve method
+				if c.Request.Method == http.MethodGet || c.Request.Method == http.MethodHead {
+					c.Redirect(http.StatusMovedPermanently, newPath)
+				} else {
+					c.Redirect(http.StatusPermanentRedirect, newPath)
+				}
+				return
+			}
+			break
+		}
+	}
+
+	if !modelFound {
+		pm.sendErrorResponse(c, http.StatusBadRequest, "model id required in path")
+		return
+	}
+
+	processGroup, realModelName, err := pm.swapProcessGroup(modelName)
 	if err != nil {
 		pm.sendErrorResponse(c, http.StatusInternalServerError, fmt.Sprintf("error swapping process group: %s", err.Error()))
 		return
 	}
 
 	// rewrite the path
-	c.Request.URL.Path = c.Param("upstreamPath")
+	c.Request.URL.Path = remainingPath
-	processGroup.ProxyRequest(requestedModel, c.Writer, c.Request)
+	processGroup.ProxyRequest(realModelName, c.Writer, c.Request)
 }
 
 func (pm *ProxyManager) proxyOAIHandler(c *gin.Context) {

proxy/proxymanager_api.go

@@ -3,8 +3,10 @@ package proxy
 import (
 	"context"
 	"encoding/json"
+	"fmt"
 	"net/http"
 	"sort"
+	"strings"
 
 	"github.com/gin-gonic/gin"
 	"github.com/mostlygeek/llama-swap/event"
@@ -23,6 +25,7 @@ func addApiHandlers(pm *ProxyManager) {
 	apiGroup := pm.ginEngine.Group("/api")
 	{
 		apiGroup.POST("/models/unload", pm.apiUnloadAllModels)
+		apiGroup.POST("/models/unload/*model", pm.apiUnloadSingleModelHandler)
 		apiGroup.GET("/events", pm.apiSendEvents)
 		apiGroup.GET("/metrics", pm.apiGetMetrics)
 	}
@@ -100,6 +103,8 @@ func (pm *ProxyManager) apiSendEvents(c *gin.Context) {
 	c.Header("Cache-Control", "no-cache")
 	c.Header("Connection", "keep-alive")
 	c.Header("X-Content-Type-Options", "nosniff")
+	// prevent nginx from buffering SSE
+	c.Header("X-Accel-Buffering", "no")
 
 	sendBuffer := make(chan messageEnvelope, 25)
 	ctx, cancel := context.WithCancel(c.Request.Context())
@@ -132,7 +137,7 @@ func (pm *ProxyManager) apiSendEvents(c *gin.Context) {
 		}
 	}
 
-	sendMetrics := func(metrics TokenMetrics) {
+	sendMetrics := func(metrics []TokenMetrics) {
 		jsonData, err := json.Marshal(metrics)
 		if err == nil {
 			select {
@@ -168,16 +173,14 @@ func (pm *ProxyManager) apiSendEvents(c *gin.Context) {
 	 * Send Metrics data
 	 */
 	defer event.On(func(e TokenMetricsEvent) {
-		sendMetrics(e.Metrics)
+		sendMetrics([]TokenMetrics{e.Metrics})
 	})()
 
 	// send initial batch of data
 	sendLogData("proxy", pm.proxyLogger.GetHistory())
 	sendLogData("upstream", pm.upstreamLogger.GetHistory())
 	sendModels()
-	for _, metrics := range pm.metricsMonitor.GetMetrics() {
+	sendMetrics(pm.metricsMonitor.GetMetrics())
-		sendMetrics(metrics)
-	}
 
 	for {
 		select {
@@ -202,3 +205,25 @@ func (pm *ProxyManager) apiGetMetrics(c *gin.Context) {
 	}
 	c.Data(http.StatusOK, "application/json", jsonData)
 }
+
+func (pm *ProxyManager) apiUnloadSingleModelHandler(c *gin.Context) {
+	requestedModel := strings.TrimPrefix(c.Param("model"), "/")
+	realModelName, found := pm.config.RealModelName(requestedModel)
+	if !found {
+		pm.sendErrorResponse(c, http.StatusNotFound, "Model not found")
+		return
+	}
+
+	processGroup := pm.findGroupByModelName(realModelName)
+	if processGroup == nil {
+		pm.sendErrorResponse(c, http.StatusInternalServerError, fmt.Sprintf("process group not found for model %s", requestedModel))
+		return
+	}
+
+	if err := processGroup.StopProcess(realModelName, StopImmediately); err != nil {
+		pm.sendErrorResponse(c, http.StatusInternalServerError, fmt.Sprintf("error stopping process: %s", err.Error()))
+		return
+	} else {
+		c.String(http.StatusOK, "OK")
+	}
+}

proxy/proxymanager_loghandlers.go

@@ -28,6 +28,8 @@ func (pm *ProxyManager) streamLogsHandler(c *gin.Context) {
 	c.Header("Content-Type", "text/plain")
 	c.Header("Transfer-Encoding", "chunked")
 	c.Header("X-Content-Type-Options", "nosniff")
+	// prevent nginx from buffering streamed logs
+	c.Header("X-Accel-Buffering", "no")
 
 	logMonitorId := c.Param("logMonitorID")
 	logger, err := pm.getLogger(logMonitorId)

proxy/proxymanager_test.go

@@ -2,6 +2,7 @@ package proxy
 
 import (
 	"bytes"
+	"context"
 	"encoding/json"
 	"fmt"
 	"math/rand"
@@ -9,18 +10,21 @@ import (
 	"net/http"
 	"net/http/httptest"
 	"strconv"
+	"strings"
 	"sync"
 	"testing"
 	"time"
 
+	"github.com/mostlygeek/llama-swap/event"
+	"github.com/mostlygeek/llama-swap/proxy/config"
 	"github.com/stretchr/testify/assert"
 	"github.com/tidwall/gjson"
 )
 
 func TestProxyManager_SwapProcessCorrectly(t *testing.T) {
-	config := AddDefaultGroupToConfig(Config{
+	config := config.AddDefaultGroupToConfig(config.Config{
 		HealthCheckTimeout: 15,
-		Models: map[string]ModelConfig{
+		Models: map[string]config.ModelConfig{
 			"model1": getTestSimpleResponderConfig("model1"),
 			"model2": getTestSimpleResponderConfig("model2"),
 		},
@@ -40,16 +44,15 @@ func TestProxyManager_SwapProcessCorrectly(t *testing.T) {
 		assert.Contains(t, w.Body.String(), modelName)
 	}
 }
-
 func TestProxyManager_SwapMultiProcess(t *testing.T) {
-	config := AddDefaultGroupToConfig(Config{
+	config := config.AddDefaultGroupToConfig(config.Config{
 		HealthCheckTimeout: 15,
-		Models: map[string]ModelConfig{
+		Models: map[string]config.ModelConfig{
 			"model1": getTestSimpleResponderConfig("model1"),
 			"model2": getTestSimpleResponderConfig("model2"),
 		},
 		LogLevel: "error",
-		Groups: map[string]GroupConfig{
+		Groups: map[string]config.GroupConfig{
 			"G1": {
 				Swap:      true,
 				Exclusive: false,
@@ -87,14 +90,14 @@ func TestProxyManager_SwapMultiProcess(t *testing.T) {
 // Test that a persistent group is not affected by the swapping behaviour of
 // other groups.
 func TestProxyManager_PersistentGroupsAreNotSwapped(t *testing.T) {
-	config := AddDefaultGroupToConfig(Config{
+	config := config.AddDefaultGroupToConfig(config.Config{
 		HealthCheckTimeout: 15,
-		Models: map[string]ModelConfig{
+		Models: map[string]config.ModelConfig{
 			"model1": getTestSimpleResponderConfig("model1"), // goes into the default group
 			"model2": getTestSimpleResponderConfig("model2"),
 		},
 		LogLevel: "error",
-		Groups: map[string]GroupConfig{
+		Groups: map[string]config.GroupConfig{
 			// the forever group is persistent and should not be affected by model1
 			"forever": {
 				Swap:       true,
@@ -131,9 +134,9 @@ func TestProxyManager_SwapMultiProcessParallelRequests(t *testing.T) {
 		t.Skip("skipping slow test")
 	}
 
-	config := AddDefaultGroupToConfig(Config{
+	config := config.AddDefaultGroupToConfig(config.Config{
 		HealthCheckTimeout: 15,
-		Models: map[string]ModelConfig{
+		Models: map[string]config.ModelConfig{
 			"model1": getTestSimpleResponderConfig("model1"),
 			"model2": getTestSimpleResponderConfig("model2"),
 			"model3": getTestSimpleResponderConfig("model3"),
@@ -194,9 +197,9 @@ func TestProxyManager_ListModelsHandler(t *testing.T) {
 	model2Config.Name = "     " // empty whitespace only strings will get ignored
 	model2Config.Description = "  "
 
-	config := Config{
+	config := config.Config{
 		HealthCheckTimeout: 15,
-		Models: map[string]ModelConfig{
+		Models: map[string]config.ModelConfig{
 			"model1": model1Config,
 			"model2": model2Config,
 			"model3": getTestSimpleResponderConfig("model3"),
@@ -279,6 +282,135 @@ func TestProxyManager_ListModelsHandler(t *testing.T) {
 	assert.Empty(t, expectedModels, "not all expected models were returned")
 }
 
+func TestProxyManager_ListModelsHandler_WithMetadata(t *testing.T) {
+	// Process config through LoadConfigFromReader to apply macro substitution
+	configYaml := `
+healthCheckTimeout: 15
+logLevel: error
+startPort: 10000
+models:
+  model1:
+    cmd: /path/to/server -p ${PORT}
+    macros:
+      PORT_NUM: 10001
+      TEMP: 0.7
+      NAME: "llama"
+    metadata:
+      port: ${PORT_NUM}
+      temperature: ${TEMP}
+      enabled: true
+      note: "Running on port ${PORT_NUM}"
+      nested:
+        value: ${TEMP}
+  model2:
+    cmd: /path/to/server -p ${PORT}
+`
+	processedConfig, err := config.LoadConfigFromReader(strings.NewReader(configYaml))
+	assert.NoError(t, err)
+
+	proxy := New(processedConfig)
+
+	req := httptest.NewRequest("GET", "/v1/models", nil)
+	w := httptest.NewRecorder()
+	proxy.ServeHTTP(w, req)
+
+	assert.Equal(t, http.StatusOK, w.Code)
+
+	var response struct {
+		Data []map[string]any `json:"data"`
+	}
+
+	err = json.Unmarshal(w.Body.Bytes(), &response)
+	assert.NoError(t, err)
+	assert.Len(t, response.Data, 2)
+
+	// Find model1 and model2 in response
+	var model1Data, model2Data map[string]any
+	for _, model := range response.Data {
+		if model["id"] == "model1" {
+			model1Data = model
+		} else if model["id"] == "model2" {
+			model2Data = model
+		}
+	}
+
+	// Verify model1 has llamaswap_meta
+	assert.NotNil(t, model1Data)
+	meta, exists := model1Data["meta"]
+	if !assert.True(t, exists, "model1 should have meta key") {
+		t.FailNow()
+	}
+
+	metaMap := meta.(map[string]any)
+
+	lsmeta, exists := metaMap["llamaswap"]
+	if !assert.True(t, exists, "model1 should have meta.llamaswap key") {
+		t.FailNow()
+	}
+
+	lsmetamap := lsmeta.(map[string]any)
+
+	// Verify type preservation
+	assert.Equal(t, float64(10001), lsmetamap["port"]) // JSON numbers are float64
+	assert.Equal(t, 0.7, lsmetamap["temperature"])
+	assert.Equal(t, true, lsmetamap["enabled"])
+	// Verify string interpolation
+	assert.Equal(t, "Running on port 10001", lsmetamap["note"])
+	// Verify nested structure
+	nested := lsmetamap["nested"].(map[string]any)
+	assert.Equal(t, 0.7, nested["value"])
+
+	// Verify model2 does NOT have llamaswap_meta
+	assert.NotNil(t, model2Data)
+	_, exists = model2Data["llamaswap_meta"]
+	assert.False(t, exists, "model2 should not have llamaswap_meta")
+}
+
+func TestProxyManager_ListModelsHandler_SortedByID(t *testing.T) {
+	// Intentionally add models in non-sorted order and with an unlisted model
+	config := config.Config{
+		HealthCheckTimeout: 15,
+		Models: map[string]config.ModelConfig{
+			"zeta":  getTestSimpleResponderConfig("zeta"),
+			"alpha": getTestSimpleResponderConfig("alpha"),
+			"beta":  getTestSimpleResponderConfig("beta"),
+			"hidden": func() config.ModelConfig {
+				mc := getTestSimpleResponderConfig("hidden")
+				mc.Unlisted = true
+				return mc
+			}(),
+		},
+		LogLevel: "error",
+	}
+
+	proxy := New(config)
+
+	// Request models list
+	req := httptest.NewRequest("GET", "/v1/models", nil)
+	w := httptest.NewRecorder()
+	proxy.ServeHTTP(w, req)
+
+	assert.Equal(t, http.StatusOK, w.Code)
+
+	var response struct {
+		Data []map[string]interface{} `json:"data"`
+	}
+	if err := json.Unmarshal(w.Body.Bytes(), &response); err != nil {
+		t.Fatalf("Failed to parse JSON response: %v", err)
+	}
+
+	// We expect only the listed models in sorted order by id
+	expectedOrder := []string{"alpha", "beta", "zeta"}
+	if assert.Len(t, response.Data, len(expectedOrder), "unexpected number of listed models") {
+		got := make([]string, 0, len(response.Data))
+		for _, m := range response.Data {
+			id, _ := m["id"].(string)
+			got = append(got, id)
+		}
+		assert.Equal(t, expectedOrder, got, "models should be sorted by id ascending")
+	}
+}
+
 func TestProxyManager_Shutdown(t *testing.T) {
 	// make broken model configurations
 	model1Config := getTestSimpleResponderConfigPort("model1", 9991)
@@ -290,15 +422,15 @@ func TestProxyManager_Shutdown(t *testing.T) {
 	model3Config := getTestSimpleResponderConfigPort("model3", 9993)
 	model3Config.Proxy = "http://localhost:10003/"
 
-	config := AddDefaultGroupToConfig(Config{
+	config := config.AddDefaultGroupToConfig(config.Config{
 		HealthCheckTimeout: 15,
-		Models: map[string]ModelConfig{
+		Models: map[string]config.ModelConfig{
 			"model1": model1Config,
 			"model2": model2Config,
 			"model3": model3Config,
 		},
 		LogLevel: "error",
-		Groups: map[string]GroupConfig{
+		Groups: map[string]config.GroupConfig{
 			"test": {
 				Swap:    false,
 				Members: []string{"model1", "model2", "model3"},
@@ -333,38 +465,92 @@ func TestProxyManager_Shutdown(t *testing.T) {
 }
 
 func TestProxyManager_Unload(t *testing.T) {
-	config := AddDefaultGroupToConfig(Config{
+	conf := config.AddDefaultGroupToConfig(config.Config{
 		HealthCheckTimeout: 15,
-		Models: map[string]ModelConfig{
+		Models: map[string]config.ModelConfig{
 			"model1": getTestSimpleResponderConfig("model1"),
 		},
 		LogLevel: "error",
 	})
 
-	proxy := New(config)
+	proxy := New(conf)
 	reqBody := fmt.Sprintf(`{"model":"%s"}`, "model1")
 	req := httptest.NewRequest("POST", "/v1/chat/completions", bytes.NewBufferString(reqBody))
 	w := httptest.NewRecorder()
 	proxy.ServeHTTP(w, req)
 
-	assert.Equal(t, proxy.processGroups[DEFAULT_GROUP_ID].processes["model1"].CurrentState(), StateReady)
+	assert.Equal(t, proxy.processGroups[config.DEFAULT_GROUP_ID].processes["model1"].CurrentState(), StateReady)
 	req = httptest.NewRequest("GET", "/unload", nil)
 	w = httptest.NewRecorder()
 	proxy.ServeHTTP(w, req)
 	assert.Equal(t, http.StatusOK, w.Code)
 	assert.Equal(t, w.Body.String(), "OK")
 
-	// give it a bit of time to stop
+	select {
-	<-time.After(time.Millisecond * 250)
+	case <-proxy.processGroups[config.DEFAULT_GROUP_ID].processes["model1"].cmdWaitChan:
-	assert.Equal(t, proxy.processGroups[DEFAULT_GROUP_ID].processes["model1"].CurrentState(), StateStopped)
+		// good
+	case <-time.After(2 * time.Second):
+		t.Fatal("timeout waiting for model1 to stop")
+	}
+	assert.Equal(t, proxy.processGroups[config.DEFAULT_GROUP_ID].processes["model1"].CurrentState(), StateStopped)
+}
+
+func TestProxyManager_UnloadSingleModel(t *testing.T) {
+	const testGroupId = "testGroup"
+	config := config.AddDefaultGroupToConfig(config.Config{
+		HealthCheckTimeout: 15,
+		Models: map[string]config.ModelConfig{
+			"model1": getTestSimpleResponderConfig("model1"),
+			"model2": getTestSimpleResponderConfig("model2"),
+		},
+		Groups: map[string]config.GroupConfig{
+			testGroupId: {
+				Swap:    false,
+				Members: []string{"model1", "model2"},
+			},
+		},
+		LogLevel: "error",
+	})
+
+	proxy := New(config)
+	defer proxy.StopProcesses(StopImmediately)
+
+	// start both model
+	for _, modelName := range []string{"model1", "model2"} {
+		reqBody := fmt.Sprintf(`{"model":"%s"}`, modelName)
+		req := httptest.NewRequest("POST", "/v1/chat/completions", bytes.NewBufferString(reqBody))
+		w := httptest.NewRecorder()
+		proxy.ServeHTTP(w, req)
+	}
+
+	assert.Equal(t, StateReady, proxy.processGroups[testGroupId].processes["model1"].CurrentState())
+	assert.Equal(t, StateReady, proxy.processGroups[testGroupId].processes["model2"].CurrentState())
+
+	req := httptest.NewRequest("POST", "/api/models/unload/model1", nil)
+	w := httptest.NewRecorder()
+	proxy.ServeHTTP(w, req)
+	assert.Equal(t, http.StatusOK, w.Code)
+	if !assert.Equal(t, w.Body.String(), "OK") {
+		t.FailNow()
+	}
+
+	select {
+	case <-proxy.processGroups[testGroupId].processes["model1"].cmdWaitChan:
+		// good
+	case <-time.After(2 * time.Second):
+		t.Fatal("timeout waiting for model1 to stop")
+	}
+
+	assert.Equal(t, proxy.processGroups[testGroupId].processes["model1"].CurrentState(), StateStopped)
+	assert.Equal(t, proxy.processGroups[testGroupId].processes["model2"].CurrentState(), StateReady)
 }
 
 // Test issue #61 `Listing the current list of models and the loaded model.`
 func TestProxyManager_RunningEndpoint(t *testing.T) {
 	// Shared configuration
-	config := AddDefaultGroupToConfig(Config{
+	config := config.AddDefaultGroupToConfig(config.Config{
 		HealthCheckTimeout: 15,
-		Models: map[string]ModelConfig{
+		Models: map[string]config.ModelConfig{
 			"model1": getTestSimpleResponderConfig("model1"),
 			"model2": getTestSimpleResponderConfig("model2"),
 		},
@@ -427,9 +613,9 @@ func TestProxyManager_RunningEndpoint(t *testing.T) {
 }
 
 func TestProxyManager_AudioTranscriptionHandler(t *testing.T) {
-	config := AddDefaultGroupToConfig(Config{
+	config := config.AddDefaultGroupToConfig(config.Config{
 		HealthCheckTimeout: 15,
-		Models: map[string]ModelConfig{
+		Models: map[string]config.ModelConfig{
 			"TheExpectedModel": getTestSimpleResponderConfig("TheExpectedModel"),
 		},
 		LogLevel: "error",
@@ -480,15 +666,15 @@ func TestProxyManager_UseModelName(t *testing.T) {
 	modelConfig := getTestSimpleResponderConfig(upstreamModelName)
 	modelConfig.UseModelName = upstreamModelName
 
-	config := AddDefaultGroupToConfig(Config{
+	conf := config.AddDefaultGroupToConfig(config.Config{
 		HealthCheckTimeout: 15,
-		Models: map[string]ModelConfig{
+		Models: map[string]config.ModelConfig{
 			"model1": modelConfig,
 		},
 		LogLevel: "error",
 	})
 
-	proxy := New(config)
+	proxy := New(conf)
 	defer proxy.StopProcesses(StopWaitForInflightRequest)
 
 	requestedModel := "model1"
@@ -543,9 +729,9 @@ func TestProxyManager_UseModelName(t *testing.T) {
 }
 
 func TestProxyManager_CORSOptionsHandler(t *testing.T) {
-	config := AddDefaultGroupToConfig(Config{
+	config := config.AddDefaultGroupToConfig(config.Config{
 		HealthCheckTimeout: 15,
-		Models: map[string]ModelConfig{
+		Models: map[string]config.ModelConfig{
 			"model1": getTestSimpleResponderConfig("model1"),
 		},
 		LogLevel: "error",
@@ -611,27 +797,40 @@ func TestProxyManager_CORSOptionsHandler(t *testing.T) {
 }
 
 func TestProxyManager_Upstream(t *testing.T) {
-	config := AddDefaultGroupToConfig(Config{
+	configStr := fmt.Sprintf(`
-		HealthCheckTimeout: 15,
+logLevel: error
-		Models: map[string]ModelConfig{
+models:
-			"model1": getTestSimpleResponderConfig("model1"),
+  model1:
-		},
+    cmd: %s -port ${PORT} -silent -respond model1
-		LogLevel: "error",
+    aliases: [model-alias]
-	})
+`, getSimpleResponderPath())
+
+	config, err := config.LoadConfigFromReader(strings.NewReader(configStr))
+	assert.NoError(t, err)
 
 	proxy := New(config)
 	defer proxy.StopProcesses(StopWaitForInflightRequest)
-	req := httptest.NewRequest("GET", "/upstream/model1/test", nil)
+	t.Run("main model name", func(t *testing.T) {
-	rec := httptest.NewRecorder()
+		req := httptest.NewRequest("GET", "/upstream/model1/test", nil)
-	proxy.ServeHTTP(rec, req)
+		rec := httptest.NewRecorder()
-	assert.Equal(t, http.StatusOK, rec.Code)
+		proxy.ServeHTTP(rec, req)
-	assert.Equal(t, "model1", rec.Body.String())
+		assert.Equal(t, http.StatusOK, rec.Code)
+		assert.Equal(t, "model1", rec.Body.String())
+	})
+
+	t.Run("model alias", func(t *testing.T) {
+		req := httptest.NewRequest("GET", "/upstream/model-alias/test", nil)
+		rec := httptest.NewRecorder()
+		proxy.ServeHTTP(rec, req)
+		assert.Equal(t, http.StatusOK, rec.Code)
+		assert.Equal(t, "model1", rec.Body.String())
+	})
 }
 
 func TestProxyManager_ChatContentLength(t *testing.T) {
-	config := AddDefaultGroupToConfig(Config{
+	config := config.AddDefaultGroupToConfig(config.Config{
 		HealthCheckTimeout: 15,
-		Models: map[string]ModelConfig{
+		Models: map[string]config.ModelConfig{
 			"model1": getTestSimpleResponderConfig("model1"),
 		},
 		LogLevel: "error",
@@ -654,14 +853,14 @@ func TestProxyManager_ChatContentLength(t *testing.T) {
 
 func TestProxyManager_FiltersStripParams(t *testing.T) {
 	modelConfig := getTestSimpleResponderConfig("model1")
-	modelConfig.Filters = ModelFilters{
+	modelConfig.Filters = config.ModelFilters{
 		StripParams: "temperature, model, stream",
 	}
 
-	config := AddDefaultGroupToConfig(Config{
+	config := config.AddDefaultGroupToConfig(config.Config{
 		HealthCheckTimeout: 15,
 		LogLevel:           "error",
-		Models: map[string]ModelConfig{
+		Models: map[string]config.ModelConfig{
 			"model1": modelConfig,
 		},
 	})
@@ -687,9 +886,9 @@ func TestProxyManager_FiltersStripParams(t *testing.T) {
 }
 
 func TestProxyManager_MiddlewareWritesMetrics_NonStreaming(t *testing.T) {
-	config := AddDefaultGroupToConfig(Config{
+	config := config.AddDefaultGroupToConfig(config.Config{
 		HealthCheckTimeout: 15,
-		Models: map[string]ModelConfig{
+		Models: map[string]config.ModelConfig{
 			"model1": getTestSimpleResponderConfig("model1"),
 		},
 		LogLevel: "error",
@@ -722,9 +921,9 @@ func TestProxyManager_MiddlewareWritesMetrics_NonStreaming(t *testing.T) {
 }
 
 func TestProxyManager_MiddlewareWritesMetrics_Streaming(t *testing.T) {
-	config := AddDefaultGroupToConfig(Config{
+	config := config.AddDefaultGroupToConfig(config.Config{
 		HealthCheckTimeout: 15,
-		Models: map[string]ModelConfig{
+		Models: map[string]config.ModelConfig{
 			"model1": getTestSimpleResponderConfig("model1"),
 		},
 		LogLevel: "error",
@@ -757,9 +956,9 @@ func TestProxyManager_MiddlewareWritesMetrics_Streaming(t *testing.T) {
 }
 
 func TestProxyManager_HealthEndpoint(t *testing.T) {
-	config := AddDefaultGroupToConfig(Config{
+	config := config.AddDefaultGroupToConfig(config.Config{
 		HealthCheckTimeout: 15,
-		Models: map[string]ModelConfig{
+		Models: map[string]config.ModelConfig{
 			"model1": getTestSimpleResponderConfig("model1"),
 		},
 		LogLevel: "error",
@@ -773,3 +972,148 @@ func TestProxyManager_HealthEndpoint(t *testing.T) {
 	assert.Equal(t, http.StatusOK, rec.Code)
 	assert.Equal(t, "OK", rec.Body.String())
 }
+
+// Ensure the custom llama-server /completion endpoint proxies correctly
+func TestProxyManager_CompletionEndpoint(t *testing.T) {
+	config := config.AddDefaultGroupToConfig(config.Config{
+		HealthCheckTimeout: 15,
+		Models: map[string]config.ModelConfig{
+			"model1": getTestSimpleResponderConfig("model1"),
+		},
+		LogLevel: "error",
+	})
+
+	proxy := New(config)
+	defer proxy.StopProcesses(StopWaitForInflightRequest)
+
+	reqBody := `{"model":"model1"}`
+	req := httptest.NewRequest("POST", "/completion", bytes.NewBufferString(reqBody))
+	w := httptest.NewRecorder()
+
+	proxy.ServeHTTP(w, req)
+	assert.Equal(t, http.StatusOK, w.Code)
+	assert.Contains(t, w.Body.String(), "model1")
+}
+
+func TestProxyManager_StartupHooks(t *testing.T) {
+
+	// using real YAML as the configuration has gotten more complex
+	// is the right approach as LoadConfigFromReader() does a lot more
+	// than parse YAML now. Eventually migrate all tests to use this approach
+	configStr := strings.Replace(`
+logLevel: error
+hooks:
+  on_startup:
+    preload:
+      - model1
+      - model2
+groups:
+  preloadTestGroup:
+    swap: false
+    members:
+       - model1
+       - model2
+models:
+  model1:
+    cmd: ${simpleresponderpath} --port ${PORT} --silent --respond model1
+  model2:
+      cmd: ${simpleresponderpath} --port ${PORT} --silent --respond model2
+`, "${simpleresponderpath}", simpleResponderPath, -1)
+
+	// Create a test model configuration
+	config, err := config.LoadConfigFromReader(strings.NewReader(configStr))
+	if !assert.NoError(t, err, "Invalid configuration") {
+		return
+	}
+
+	preloadChan := make(chan ModelPreloadedEvent, 2) // buffer for 2 expected events
+
+	unsub := event.On(func(e ModelPreloadedEvent) {
+		preloadChan <- e
+	})
+
+	defer unsub()
+
+	// Create the proxy which should trigger preloading
+	proxy := New(config)
+	defer proxy.StopProcesses(StopWaitForInflightRequest)
+
+	for i := 0; i < 2; i++ {
+		select {
+		case <-preloadChan:
+		case <-time.After(5 * time.Second):
+			t.Fatal("timed out waiting for models to preload")
+		}
+	}
+	// make sure they are both loaded
+	_, foundGroup := proxy.processGroups["preloadTestGroup"]
+	if !assert.True(t, foundGroup, "preloadTestGroup should exist") {
+		return
+	}
+	assert.Equal(t, StateReady, proxy.processGroups["preloadTestGroup"].processes["model1"].CurrentState())
+	assert.Equal(t, StateReady, proxy.processGroups["preloadTestGroup"].processes["model2"].CurrentState())
+}
+
+func TestProxyManager_StreamingEndpointsReturnNoBufferingHeader(t *testing.T) {
+	config := config.AddDefaultGroupToConfig(config.Config{
+		HealthCheckTimeout: 15,
+		Models: map[string]config.ModelConfig{
+			"model1": getTestSimpleResponderConfig("model1"),
+		},
+		LogLevel: "error",
+	})
+
+	proxy := New(config)
+	defer proxy.StopProcesses(StopWaitForInflightRequest)
+
+	endpoints := []string{
+		"/api/events",
+		"/logs/stream",
+		"/logs/stream/proxy",
+		"/logs/stream/upstream",
+	}
+
+	for _, endpoint := range endpoints {
+		t.Run(endpoint, func(t *testing.T) {
+			ctx, cancel := context.WithCancel(context.Background())
+			defer cancel()
+
+			req := httptest.NewRequest("GET", endpoint, nil)
+			req = req.WithContext(ctx)
+			rec := httptest.NewRecorder()
+
+			// We don't need the handler to fully complete, just to set the headers
+			// so run it in a goroutine and check the headers after a short delay
+			go proxy.ServeHTTP(rec, req)
+			time.Sleep(10 * time.Millisecond) // give it time to start and write headers
+
+			assert.Equal(t, http.StatusOK, rec.Code)
+			assert.Equal(t, "no", rec.Header().Get("X-Accel-Buffering"))
+		})
+	}
+}
+
+func TestProxyManager_ProxiedStreamingEndpointReturnsNoBufferingHeader(t *testing.T) {
+	config := config.AddDefaultGroupToConfig(config.Config{
+		HealthCheckTimeout: 15,
+		Models: map[string]config.ModelConfig{
+			"streaming-model": getTestSimpleResponderConfig("streaming-model"),
+		},
+		LogLevel: "error",
+	})
+
+	proxy := New(config)
+	defer proxy.StopProcesses(StopWaitForInflightRequest)
+
+	// Make a streaming request
+	reqBody := `{"model":"streaming-model"}`
+	// simple-responder will return text/event-stream when stream=true is in the query
+	req := httptest.NewRequest("POST", "/v1/chat/completions?stream=true", bytes.NewBufferString(reqBody))
+	rec := httptest.NewRecorder()
+
+	proxy.ServeHTTP(rec, req)
+
+	assert.Equal(t, http.StatusOK, rec.Code)
+	assert.Equal(t, "no", rec.Header().Get("X-Accel-Buffering"))
+	assert.Contains(t, rec.Header().Get("Content-Type"), "text/event-stream")
+}

ui/package-lock.json

@@ -8,15 +8,15 @@
       "name": "ui",
       "version": "0.0.0",
       "dependencies": {
-        "@tailwindcss/vite": "^4.1.8",
-        "@tanstack/react-query": "^5.80.6",
         "react": "^19.1.0",
         "react-dom": "^19.1.0",
-        "react-router-dom": "^7.6.2",
+        "react-icons": "^5.5.0",
-        "tailwindcss": "^4.1.8"
+        "react-resizable-panels": "^3.0.4",
+        "react-router-dom": "^7.6.2"
       },
       "devDependencies": {
         "@eslint/js": "^9.25.0",
+        "@tailwindcss/vite": "^4.1.8",
         "@types/react": "^19.1.2",
         "@types/react-dom": "^19.1.2",
         "@vitejs/plugin-react": "^4.4.1",
@@ -24,6 +24,7 @@
         "eslint-plugin-react-hooks": "^5.2.0",
         "eslint-plugin-react-refresh": "^0.4.19",
         "globals": "^16.0.0",
+        "tailwindcss": "^4.1.8",
         "typescript": "~5.8.3",
         "typescript-eslint": "^8.30.1",
         "vite": "^6.3.5"
@@ -33,6 +34,7 @@
       "version": "2.3.0",
       "resolved": "https://registry.npmjs.org/@ampproject/remapping/-/remapping-2.3.0.tgz",
       "integrity": "sha512-30iZtAPgz+LTIYoeivqYo853f02jBYSd5uGnGpkFV0M3xOt9aN73erkgYAmZU43x4VfqcnLxW9Kpg3R5LC4YYw==",
+      "dev": true,
       "license": "Apache-2.0",
       "dependencies": {
         "@jridgewell/gen-mapping": "^0.3.5",
@@ -331,6 +333,7 @@
       "cpu": [
         "ppc64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -347,6 +350,7 @@
       "cpu": [
         "arm"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -363,6 +367,7 @@
       "cpu": [
         "arm64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -379,6 +384,7 @@
       "cpu": [
         "x64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -395,6 +401,7 @@
       "cpu": [
         "arm64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -411,6 +418,7 @@
       "cpu": [
         "x64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -427,6 +435,7 @@
       "cpu": [
         "arm64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -443,6 +452,7 @@
       "cpu": [
         "x64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -459,6 +469,7 @@
       "cpu": [
         "arm"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -475,6 +486,7 @@
       "cpu": [
         "arm64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -491,6 +503,7 @@
       "cpu": [
         "ia32"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -507,6 +520,7 @@
       "cpu": [
         "loong64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -523,6 +537,7 @@
       "cpu": [
         "mips64el"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -539,6 +554,7 @@
       "cpu": [
         "ppc64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -555,6 +571,7 @@
       "cpu": [
         "riscv64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -571,6 +588,7 @@
       "cpu": [
         "s390x"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -587,6 +605,7 @@
       "cpu": [
         "x64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -603,6 +622,7 @@
       "cpu": [
         "arm64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -619,6 +639,7 @@
       "cpu": [
         "x64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -635,6 +656,7 @@
       "cpu": [
         "arm64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -651,6 +673,7 @@
       "cpu": [
         "x64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -667,6 +690,7 @@
       "cpu": [
         "x64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -683,6 +707,7 @@
       "cpu": [
         "arm64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -699,6 +724,7 @@
       "cpu": [
         "ia32"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -715,6 +741,7 @@
       "cpu": [
         "x64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -948,6 +975,7 @@
       "version": "4.0.1",
       "resolved": "https://registry.npmjs.org/@isaacs/fs-minipass/-/fs-minipass-4.0.1.tgz",
       "integrity": "sha512-wgm9Ehl2jpeqP3zw/7mo3kRHFp5MEDhqAdwy1fTGkHAwnkGOVsgpvQhL8B5n1qlb01jV3n/bI0ZfZp5lWA1k4w==",
+      "dev": true,
       "license": "ISC",
       "dependencies": {
         "minipass": "^7.0.4"
@@ -960,6 +988,7 @@
       "version": "0.3.8",
       "resolved": "https://registry.npmjs.org/@jridgewell/gen-mapping/-/gen-mapping-0.3.8.tgz",
       "integrity": "sha512-imAbBGkb+ebQyxKgzv5Hu2nmROxoDOXHh80evxdoXNOrvAnVx7zimzc1Oo5h9RlfV4vPXaE2iM5pOFbvOCClWA==",
+      "dev": true,
       "license": "MIT",
       "dependencies": {
         "@jridgewell/set-array": "^1.2.1",
@@ -974,6 +1003,7 @@
       "version": "3.1.2",
       "resolved": "https://registry.npmjs.org/@jridgewell/resolve-uri/-/resolve-uri-3.1.2.tgz",
       "integrity": "sha512-bRISgCIjP20/tbWSPWMEi54QVPRZExkuD9lJL+UIxUKtwVJA8wW1Trb1jMs1RFXo1CBTNZ/5hpC9QvmKWdopKw==",
+      "dev": true,
       "license": "MIT",
       "engines": {
         "node": ">=6.0.0"
@@ -983,6 +1013,7 @@
       "version": "1.2.1",
       "resolved": "https://registry.npmjs.org/@jridgewell/set-array/-/set-array-1.2.1.tgz",
       "integrity": "sha512-R8gLRTZeyp03ymzP/6Lil/28tGeGEzhx1q2k703KGWRAI1VdvPIXdG70VJc2pAMw3NA6JKL5hhFu1sJX0Mnn/A==",
+      "dev": true,
       "license": "MIT",
       "engines": {
         "node": ">=6.0.0"
@@ -992,12 +1023,14 @@
       "version": "1.5.0",
       "resolved": "https://registry.npmjs.org/@jridgewell/sourcemap-codec/-/sourcemap-codec-1.5.0.tgz",
       "integrity": "sha512-gv3ZRaISU3fjPAgNsriBRqGWQL6quFx04YMPW/zD8XMLsU32mhCCbfbO6KZFLjvYpCZ8zyDEgqsgf+PwPaM7GQ==",
+      "dev": true,
       "license": "MIT"
     },
     "node_modules/@jridgewell/trace-mapping": {
       "version": "0.3.25",
       "resolved": "https://registry.npmjs.org/@jridgewell/trace-mapping/-/trace-mapping-0.3.25.tgz",
       "integrity": "sha512-vNk6aEwybGtawWmy/PzwnGDOjCkLWSD2wqvjGGAgOAwCGWySYXfYoxt00IJkTF+8Lb57DwOb3Aa0o9CApepiYQ==",
+      "dev": true,
       "license": "MIT",
       "dependencies": {
         "@jridgewell/resolve-uri": "^3.1.0",
@@ -1056,6 +1089,7 @@
       "cpu": [
         "arm"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1069,6 +1103,7 @@
       "cpu": [
         "arm64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1082,6 +1117,7 @@
       "cpu": [
         "arm64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1095,6 +1131,7 @@
       "cpu": [
         "x64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1108,6 +1145,7 @@
       "cpu": [
         "arm64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1121,6 +1159,7 @@
       "cpu": [
         "x64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1134,6 +1173,7 @@
       "cpu": [
         "arm"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1147,6 +1187,7 @@
       "cpu": [
         "arm"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1160,6 +1201,7 @@
       "cpu": [
         "arm64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1173,6 +1215,7 @@
       "cpu": [
         "arm64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1186,6 +1229,7 @@
       "cpu": [
         "loong64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1199,6 +1243,7 @@
       "cpu": [
         "ppc64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1212,6 +1257,7 @@
       "cpu": [
         "riscv64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1225,6 +1271,7 @@
       "cpu": [
         "riscv64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1238,6 +1285,7 @@
       "cpu": [
         "s390x"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1251,6 +1299,7 @@
       "cpu": [
         "x64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1264,6 +1313,7 @@
       "cpu": [
         "x64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1277,6 +1327,7 @@
       "cpu": [
         "arm64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1290,6 +1341,7 @@
       "cpu": [
         "ia32"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1303,6 +1355,7 @@
       "cpu": [
         "x64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1313,6 +1366,7 @@
       "version": "4.1.8",
       "resolved": "https://registry.npmjs.org/@tailwindcss/node/-/node-4.1.8.tgz",
       "integrity": "sha512-OWwBsbC9BFAJelmnNcrKuf+bka2ZxCE2A4Ft53Tkg4uoiE67r/PMEYwCsourC26E+kmxfwE0hVzMdxqeW+xu7Q==",
+      "dev": true,
       "license": "MIT",
       "dependencies": {
         "@ampproject/remapping": "^2.3.0",
@@ -1328,6 +1382,7 @@
       "version": "4.1.8",
       "resolved": "https://registry.npmjs.org/@tailwindcss/oxide/-/oxide-4.1.8.tgz",
       "integrity": "sha512-d7qvv9PsM5N3VNKhwVUhpK6r4h9wtLkJ6lz9ZY9aeZgrUWk1Z8VPyqyDT9MZlem7GTGseRQHkeB1j3tC7W1P+A==",
+      "dev": true,
       "hasInstallScript": true,
       "license": "MIT",
       "dependencies": {
@@ -1359,6 +1414,7 @@
       "cpu": [
         "arm64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1375,6 +1431,7 @@
       "cpu": [
         "arm64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1391,6 +1448,7 @@
       "cpu": [
         "x64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1407,6 +1465,7 @@
       "cpu": [
         "x64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1423,6 +1482,7 @@
       "cpu": [
         "arm"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1439,6 +1499,7 @@
       "cpu": [
         "arm64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1455,6 +1516,7 @@
       "cpu": [
         "arm64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1471,6 +1533,7 @@
       "cpu": [
         "x64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1487,6 +1550,7 @@
       "cpu": [
         "x64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1511,6 +1575,7 @@
       "cpu": [
         "wasm32"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "dependencies": {
@@ -1532,6 +1597,7 @@
       "cpu": [
         "arm64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1548,6 +1614,7 @@
       "cpu": [
         "x64"
       ],
+      "dev": true,
       "license": "MIT",
       "optional": true,
       "os": [
@@ -1561,6 +1628,7 @@
       "version": "4.1.8",
       "resolved": "https://registry.npmjs.org/@tailwindcss/vite/-/vite-4.1.8.tgz",
       "integrity": "sha512-CQ+I8yxNV5/6uGaJjiuymgw0kEQiNKRinYbZXPdx1fk5WgiyReG0VaUx/Xq6aVNSUNJFzxm6o8FNKS5aMaim5A==",
+      "dev": true,
       "license": "MIT",
       "dependencies": {
         "@tailwindcss/node": "4.1.8",
@@ -1571,32 +1639,6 @@
         "vite": "^5.2.0 || ^6"
       }
     },
-    "node_modules/@tanstack/query-core": {
-      "version": "5.80.6",
-      "resolved": "https://registry.npmjs.org/@tanstack/query-core/-/query-core-5.80.6.tgz",
-      "integrity": "sha512-nl7YxT/TAU+VTf+e2zTkObGTyY8YZBMnbgeA1ee66lIVqzKlYursAII6z5t0e6rXgwUMJSV4dshBTNacNpZHbQ==",
-      "license": "MIT",
-      "funding": {
-        "type": "github",
-        "url": "https://github.com/sponsors/tannerlinsley"
-      }
-    },
-    "node_modules/@tanstack/react-query": {
-      "version": "5.80.6",
-      "resolved": "https://registry.npmjs.org/@tanstack/react-query/-/react-query-5.80.6.tgz",
-      "integrity": "sha512-izX+5CnkpON3NQGcEm3/d7LfFQNo9ZpFtX2QsINgCYK9LT2VCIdi8D3bMaMSNhrAJCznRoAkFic76uvLroALBw==",
-      "license": "MIT",
-      "dependencies": {
-        "@tanstack/query-core": "5.80.6"
-      },
-      "funding": {
-        "type": "github",
-        "url": "https://github.com/sponsors/tannerlinsley"
-      },
-      "peerDependencies": {
-        "react": "^18 || ^19"
-      }
-    },
     "node_modules/@types/babel__core": {
       "version": "7.20.5",
       "resolved": "https://registry.npmjs.org/@types/babel__core/-/babel__core-7.20.5.tgz",
@@ -2146,6 +2188,7 @@
       "version": "3.0.0",
       "resolved": "https://registry.npmjs.org/chownr/-/chownr-3.0.0.tgz",
       "integrity": "sha512-+IxzY9BZOQd/XuYPRmrvEVjF/nqj5kgT4kEq7VofrDoM1MxoRjEWkrCC3EtLi59TVawxTAn+orJwFQcrqEN1+g==",
+      "dev": true,
       "license": "BlueOak-1.0.0",
       "engines": {
         "node": ">=18"
@@ -2245,6 +2288,7 @@
       "version": "2.0.4",
       "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.0.4.tgz",
       "integrity": "sha512-3UDv+G9CsCKO1WKMGw9fwq/SWJYbI0c5Y7LU1AXYoDdbhE2AHQ6N6Nb34sG8Fj7T5APy8qXDCKuuIHd1BR0tVA==",
+      "dev": true,
       "license": "Apache-2.0",
       "engines": {
         "node": ">=8"
@@ -2261,6 +2305,7 @@
       "version": "5.18.1",
       "resolved": "https://registry.npmjs.org/enhanced-resolve/-/enhanced-resolve-5.18.1.tgz",
       "integrity": "sha512-ZSW3ma5GkcQBIpwZTSRAI8N71Uuwgs93IezB7mf7R60tC8ZbJideoDNKjHn2O9KIlx6rkGTTEk1xUCK2E1Y2Yg==",
+      "dev": true,
       "license": "MIT",
       "dependencies": {
         "graceful-fs": "^4.2.4",
@@ -2274,6 +2319,7 @@
       "version": "0.25.5",
       "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.25.5.tgz",
       "integrity": "sha512-P8OtKZRv/5J5hhz0cUAdu/cLuPIKXpQl1R9pZtvmHWQvrAUVd0UNIPT4IB4W3rNOqVO0rlqHmCIbSwxh/c9yUQ==",
+      "dev": true,
       "hasInstallScript": true,
       "license": "MIT",
       "bin": {
@@ -2640,6 +2686,7 @@
       "version": "2.3.3",
       "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz",
       "integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==",
+      "dev": true,
       "hasInstallScript": true,
       "license": "MIT",
       "optional": true,
@@ -2690,6 +2737,7 @@
       "version": "4.2.11",
       "resolved": "https://registry.npmjs.org/graceful-fs/-/graceful-fs-4.2.11.tgz",
       "integrity": "sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ==",
+      "dev": true,
       "license": "ISC"
     },
     "node_modules/graphemer": {
@@ -2790,6 +2838,7 @@
       "version": "2.4.2",
       "resolved": "https://registry.npmjs.org/jiti/-/jiti-2.4.2.tgz",
       "integrity": "sha512-rg9zJN+G4n2nfJl5MW3BMygZX56zKPNVEYYqq7adpmMh4Jn2QNEwhvQlFy6jPVdcod7txZtKHWnyZiA3a0zP7A==",
+      "dev": true,
       "license": "MIT",
       "bin": {
         "jiti": "lib/jiti-cli.mjs"
@@ -2890,6 +2939,7 @@
       "version": "1.30.1",
       "resolved": "https://registry.npmjs.org/lightningcss/-/lightningcss-1.30.1.tgz",
       "integrity": "sha512-xi6IyHML+c9+Q3W0S4fCQJOym42pyurFiJUHEcEyHS0CeKzia4yZDEsLlqOFykxOdHpNy0NmvVO31vcSqAxJCg==",
+      "dev": true,
       "license": "MPL-2.0",
       "dependencies": {
         "detect-libc": "^2.0.3"
@@ -2921,6 +2971,7 @@
       "cpu": [
         "arm64"
       ],
+      "dev": true,
       "license": "MPL-2.0",
       "optional": true,
       "os": [
@@ -2941,6 +2992,7 @@
       "cpu": [
         "x64"
       ],
+      "dev": true,
       "license": "MPL-2.0",
       "optional": true,
       "os": [
@@ -2961,6 +3013,7 @@
       "cpu": [
         "x64"
       ],
+      "dev": true,
       "license": "MPL-2.0",
       "optional": true,
       "os": [
@@ -2981,6 +3034,7 @@
       "cpu": [
         "arm"
       ],
+      "dev": true,
       "license": "MPL-2.0",
       "optional": true,
       "os": [
@@ -3001,6 +3055,7 @@
       "cpu": [
         "arm64"
       ],
+      "dev": true,
       "license": "MPL-2.0",
       "optional": true,
       "os": [
@@ -3021,6 +3076,7 @@
       "cpu": [
         "arm64"
       ],
+      "dev": true,
       "license": "MPL-2.0",
       "optional": true,
       "os": [
@@ -3041,6 +3097,7 @@
       "cpu": [
         "x64"
       ],
+      "dev": true,
       "license": "MPL-2.0",
       "optional": true,
       "os": [
@@ -3061,6 +3118,7 @@
       "cpu": [
         "x64"
       ],
+      "dev": true,
       "license": "MPL-2.0",
       "optional": true,
       "os": [
@@ -3081,6 +3139,7 @@
       "cpu": [
         "arm64"
       ],
+      "dev": true,
       "license": "MPL-2.0",
       "optional": true,
       "os": [
@@ -3101,6 +3160,7 @@
       "cpu": [
         "x64"
       ],
+      "dev": true,
       "license": "MPL-2.0",
       "optional": true,
       "os": [
@@ -3151,6 +3211,7 @@
       "version": "0.30.17",
       "resolved": "https://registry.npmjs.org/magic-string/-/magic-string-0.30.17.tgz",
       "integrity": "sha512-sNPKHvyjVf7gyjwS4xGTaW/mCnF8wnjtifKBEhxfZ7E/S8tQ0rssrwGNn6q8JH/ohItJfSQp9mBtQYuTlH5QnA==",
+      "dev": true,
       "license": "MIT",
       "dependencies": {
         "@jridgewell/sourcemap-codec": "^1.5.0"
@@ -3197,6 +3258,7 @@
       "version": "7.1.2",
       "resolved": "https://registry.npmjs.org/minipass/-/minipass-7.1.2.tgz",
       "integrity": "sha512-qOOzS1cBTWYF4BH8fVePDBOO9iptMnGUEZwNc/cMWnTV2nVLZ7VoNWEPHkYczZA0pdoA7dl6e7FL659nX9S2aw==",
+      "dev": true,
       "license": "ISC",
       "engines": {
         "node": ">=16 || 14 >=14.17"
@@ -3206,6 +3268,7 @@
       "version": "3.0.2",
       "resolved": "https://registry.npmjs.org/minizlib/-/minizlib-3.0.2.tgz",
       "integrity": "sha512-oG62iEk+CYt5Xj2YqI5Xi9xWUeZhDI8jjQmC5oThVH5JGCTgIjr7ciJDzC7MBzYd//WvR1OTmP5Q38Q8ShQtVA==",
+      "dev": true,
       "license": "MIT",
       "dependencies": {
         "minipass": "^7.1.2"
@@ -3218,6 +3281,7 @@
       "version": "3.0.1",
       "resolved": "https://registry.npmjs.org/mkdirp/-/mkdirp-3.0.1.tgz",
       "integrity": "sha512-+NsyUUAZDmo6YVHzL/stxSu3t9YS1iljliy3BSDrXJ/dkn1KYdmtZODGGjLcc9XLgVVpH4KshHB8XmZgMhaBXg==",
+      "dev": true,
       "license": "MIT",
       "bin": {
         "mkdirp": "dist/cjs/src/bin.js"
@@ -3240,6 +3304,7 @@
       "version": "3.3.11",
       "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.11.tgz",
       "integrity": "sha512-N8SpfPUnUp1bK+PMYW8qSWdl9U+wwNWI4QKxOYDy9JAro3WMX7p2OeVRF9v+347pnakNevPmiHhNmZ2HbFA76w==",
+      "dev": true,
       "funding": [
         {
           "type": "github",
@@ -3355,6 +3420,7 @@
       "version": "1.1.1",
       "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz",
       "integrity": "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==",
+      "dev": true,
       "license": "ISC"
     },
     "node_modules/picomatch": {
@@ -3374,6 +3440,7 @@
       "version": "8.5.4",
       "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.4.tgz",
       "integrity": "sha512-QSa9EBe+uwlGTFmHsPKokv3B/oEMQZxfqW0QqNCyhpa6mB1afzulwn8hihglqAb2pOw+BJgNlmXQ8la2VeHB7w==",
+      "dev": true,
       "funding": [
         {
           "type": "opencollective",
@@ -3460,6 +3527,15 @@
         "react": "^19.1.0"
       }
     },
+    "node_modules/react-icons": {
+      "version": "5.5.0",
+      "resolved": "https://registry.npmjs.org/react-icons/-/react-icons-5.5.0.tgz",
+      "integrity": "sha512-MEFcXdkP3dLo8uumGI5xN3lDFNsRtrjbOEKDLD7yv76v4wpnEq2Lt2qeHaQOr34I/wPN3s3+N08WkQ+CW37Xiw==",
+      "license": "MIT",
+      "peerDependencies": {
+        "react": "*"
+      }
+    },
     "node_modules/react-refresh": {
       "version": "0.17.0",
       "resolved": "https://registry.npmjs.org/react-refresh/-/react-refresh-0.17.0.tgz",
@@ -3470,6 +3546,16 @@
         "node": ">=0.10.0"
       }
     },
+    "node_modules/react-resizable-panels": {
+      "version": "3.0.4",
+      "resolved": "https://registry.npmjs.org/react-resizable-panels/-/react-resizable-panels-3.0.4.tgz",
+      "integrity": "sha512-8Y4KNgV94XhUvI2LeByyPIjoUJb71M/0hyhtzkHaqpVHs+ZQs8b627HmzyhmVYi3C9YP6R+XD1KmG7hHjEZXFQ==",
+      "license": "MIT",
+      "peerDependencies": {
+        "react": "^16.14.0 || ^17.0.0 || ^18.0.0 || ^19.0.0 || ^19.0.0-rc",
+        "react-dom": "^16.14.0 || ^17.0.0 || ^18.0.0 || ^19.0.0 || ^19.0.0-rc"
+      }
+    },
     "node_modules/react-router": {
       "version": "7.6.2",
       "resolved": "https://registry.npmjs.org/react-router/-/react-router-7.6.2.tgz",
@@ -3533,6 +3619,7 @@
       "version": "4.42.0",
       "resolved": "https://registry.npmjs.org/rollup/-/rollup-4.42.0.tgz",
       "integrity": "sha512-LW+Vse3BJPyGJGAJt1j8pWDKPd73QM8cRXYK1IxOBgL2AGLu7Xd2YOW0M2sLUBCkF5MshXXtMApyEAEzMVMsnw==",
+      "dev": true,
       "license": "MIT",
       "dependencies": {
         "@types/estree": "1.0.7"
@@ -3572,6 +3659,7 @@
       "version": "1.0.7",
       "resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.7.tgz",
       "integrity": "sha512-w28IoSUCJpidD/TGviZwwMJckNESJZXFu7NBZ5YJ4mEUnNraUn9Pm8HSZm/jDF1pDWYKspWE7oVphigUPRakIQ==",
+      "dev": true,
       "license": "MIT"
     },
     "node_modules/run-parallel": {
@@ -3647,6 +3735,7 @@
       "version": "1.2.1",
       "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz",
       "integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==",
+      "dev": true,
       "license": "BSD-3-Clause",
       "engines": {
         "node": ">=0.10.0"
@@ -3682,12 +3771,14 @@
       "version": "4.1.8",
       "resolved": "https://registry.npmjs.org/tailwindcss/-/tailwindcss-4.1.8.tgz",
       "integrity": "sha512-kjeW8gjdxasbmFKpVGrGd5T4i40mV5J2Rasw48QARfYeQ8YS9x02ON9SFWax3Qf616rt4Cp3nVNIj6Hd1mP3og==",
+      "dev": true,
       "license": "MIT"
     },
     "node_modules/tapable": {
       "version": "2.2.2",
       "resolved": "https://registry.npmjs.org/tapable/-/tapable-2.2.2.tgz",
       "integrity": "sha512-Re10+NauLTMCudc7T5WLFLAwDhQ0JWdrMK+9B2M8zR5hRExKmsRDCBA7/aV/pNJFltmBFO5BAMlQFi/vq3nKOg==",
+      "dev": true,
       "license": "MIT",
       "engines": {
         "node": ">=6"
@@ -3697,6 +3788,7 @@
       "version": "7.4.3",
       "resolved": "https://registry.npmjs.org/tar/-/tar-7.4.3.tgz",
       "integrity": "sha512-5S7Va8hKfV7W5U6g3aYxXmlPoZVAwUMy9AOKyF2fVuZa2UD3qZjg578OrLRt8PcNN1PleVaL/5/yYATNL0ICUw==",
+      "dev": true,
       "license": "ISC",
       "dependencies": {
         "@isaacs/fs-minipass": "^4.0.0",
@@ -3714,6 +3806,7 @@
       "version": "5.0.0",
       "resolved": "https://registry.npmjs.org/yallist/-/yallist-5.0.0.tgz",
       "integrity": "sha512-YgvUTfwqyc7UXVMrB+SImsVYSmTS8X/tSrtdNZMImM+n7+QTriRXyXim0mBrTXNeqzVF0KWGgHPeiyViFFrNDw==",
+      "dev": true,
       "license": "BlueOak-1.0.0",
       "engines": {
         "node": ">=18"
@@ -3723,6 +3816,7 @@
       "version": "0.2.14",
       "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.14.tgz",
       "integrity": "sha512-tX5e7OM1HnYr2+a2C/4V0htOcSQcoSTH9KgJnVvNm5zm/cyEWKJ7j7YutsH9CxMdtOkkLFy2AHrMci9IM8IPZQ==",
+      "dev": true,
       "license": "MIT",
       "dependencies": {
         "fdir": "^6.4.4",
@@ -3739,6 +3833,7 @@
       "version": "6.4.5",
       "resolved": "https://registry.npmjs.org/fdir/-/fdir-6.4.5.tgz",
       "integrity": "sha512-4BG7puHpVsIYxZUbiUE3RqGloLaSSwzYie5jvasC4LWuBWzZawynvYouhjbQKw2JuIGYdm0DzIxl8iVidKlUEw==",
+      "dev": true,
       "license": "MIT",
       "peerDependencies": {
         "picomatch": "^3 || ^4"
@@ -3753,6 +3848,7 @@
       "version": "4.0.2",
       "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.2.tgz",
       "integrity": "sha512-M7BAV6Rlcy5u+m6oPhAPFgJTzAioX/6B0DxyvDlo9l8+T3nLKbrczg2WLUyzd45L8RqfUMyGPzekbMvX2Ldkwg==",
+      "dev": true,
       "license": "MIT",
       "engines": {
         "node": ">=12"
@@ -3882,6 +3978,7 @@
       "version": "6.3.5",
       "resolved": "https://registry.npmjs.org/vite/-/vite-6.3.5.tgz",
       "integrity": "sha512-cZn6NDFE7wdTpINgs++ZJ4N49W2vRp8LCKrn3Ob1kYNtOo21vfDoaV5GzBfLU4MovSAB8uNRm4jgzVQZ+mBzPQ==",
+      "dev": true,
       "license": "MIT",
       "dependencies": {
         "esbuild": "^0.25.0",
@@ -3956,6 +4053,7 @@
       "version": "6.4.5",
       "resolved": "https://registry.npmjs.org/fdir/-/fdir-6.4.5.tgz",
       "integrity": "sha512-4BG7puHpVsIYxZUbiUE3RqGloLaSSwzYie5jvasC4LWuBWzZawynvYouhjbQKw2JuIGYdm0DzIxl8iVidKlUEw==",
+      "dev": true,
       "license": "MIT",
       "peerDependencies": {
         "picomatch": "^3 || ^4"
@@ -3970,6 +4068,7 @@
       "version": "4.0.2",
       "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.2.tgz",
       "integrity": "sha512-M7BAV6Rlcy5u+m6oPhAPFgJTzAioX/6B0DxyvDlo9l8+T3nLKbrczg2WLUyzd45L8RqfUMyGPzekbMvX2Ldkwg==",
+      "dev": true,
       "license": "MIT",
       "engines": {
         "node": ">=12"

ui/package.json

@@ -4,21 +4,21 @@
   "version": "0.0.0",
   "type": "module",
   "scripts": {
-    "dev": "vite",
+    "start": "vite",
     "build": "tsc -b && vite build --emptyOutDir",
     "lint": "eslint .",
     "preview": "vite preview"
   },
   "dependencies": {
-    "@tailwindcss/vite": "^4.1.8",
-    "@tanstack/react-query": "^5.80.6",
     "react": "^19.1.0",
     "react-dom": "^19.1.0",
-    "react-router-dom": "^7.6.2",
+    "react-icons": "^5.5.0",
-    "tailwindcss": "^4.1.8"
+    "react-resizable-panels": "^3.0.4",
+    "react-router-dom": "^7.6.2"
   },
   "devDependencies": {
     "@eslint/js": "^9.25.0",
+    "@tailwindcss/vite": "^4.1.8",
     "@types/react": "^19.1.2",
     "@types/react-dom": "^19.1.2",
     "@vitejs/plugin-react": "^4.4.1",
@@ -26,8 +26,9 @@
     "eslint-plugin-react-hooks": "^5.2.0",
     "eslint-plugin-react-refresh": "^0.4.19",
     "globals": "^16.0.0",
+    "tailwindcss": "^4.1.8",
     "typescript": "~5.8.3",
     "typescript-eslint": "^8.30.1",
     "vite": "^6.3.5"
   }
-}
+}

ui/src/App.tsx

@@ -1,48 +1,36 @@
-import { BrowserRouter as Router, Routes, Route, Navigate, NavLink } from "react-router-dom";
+import { useEffect } from "react";
+import { Navigate, Route, BrowserRouter as Router, Routes } from "react-router-dom";
+import { Header } from "./components/Header";
+import { useAPI } from "./contexts/APIProvider";
 import { useTheme } from "./contexts/ThemeProvider";
-import { APIProvider } from "./contexts/APIProvider";
+import ActivityPage from "./pages/Activity";
 import LogViewerPage from "./pages/LogViewer";
 import ModelPage from "./pages/Models";
-import ActivityPage from "./pages/Activity";
 
 function App() {
-  const theme = useTheme();
+  const { setConnectionState } = useTheme();
+
+  const { connectionStatus } = useAPI();
+
+  // Synchronize the window.title connections state with the actual connection state
+  useEffect(() => {
+    setConnectionState(connectionStatus);
+  }, [connectionStatus]);
+
   return (
     <Router basename="/ui/">
-      <APIProvider>
+      <div className="flex flex-col h-screen">
-        <div>
+        <Header />
-          <nav className="bg-surface border-b border-border p-2 h-[75px]">
-            <div className="flex items-center justify-between mx-auto px-4 h-full">
-              <h1 className="flex items-center p-0">llama-swap</h1>
-              <div className="flex items-center space-x-4">
-                <NavLink to="/" className={({ isActive }) => (isActive ? "navlink active" : "navlink")}>
-                  Logs
-                </NavLink>
 
-                <NavLink to="/models" className={({ isActive }) => (isActive ? "navlink active" : "navlink")}>
+        <main className="flex-1 overflow-auto p-4">
-                  Models
+          <Routes>
-                </NavLink>
+            <Route path="/" element={<LogViewerPage />} />
-
+            <Route path="/models" element={<ModelPage />} />
-                <NavLink to="/activity" className={({ isActive }) => (isActive ? "navlink active" : "navlink")}>
+            <Route path="/activity" element={<ActivityPage />} />
-                  Activity
+            <Route path="*" element={<Navigate to="/" replace />} />
-                </NavLink>
+          </Routes>
-                <button className="btn btn--sm" onClick={theme.toggleTheme}>
+        </main>
-                  {theme.isDarkMode ? "🌙" : "☀️"}
+      </div>
-                </button>
-              </div>
-            </div>
-          </nav>
-
-          <main className="mx-auto py-4 px-4">
-            <Routes>
-              <Route path="/" element={<LogViewerPage />} />
-              <Route path="/models" element={<ModelPage />} />
-              <Route path="/activity" element={<ActivityPage />} />
-              <Route path="*" element={<Navigate to="/" replace />} />
-            </Routes>
-          </main>
-        </div>
-      </APIProvider>
     </Router>
   );
 }

ui/src/components/ConnectionStatus.tsx

@@ -0,0 +1,26 @@
+import { useAPI } from "../contexts/APIProvider";
+import { useMemo } from "react";
+
+const ConnectionStatusIcon = () => {
+  const { connectionStatus } = useAPI();
+
+  const eventStatusColor = useMemo(() => {
+    switch (connectionStatus) {
+      case "connected":
+        return "bg-emerald-500";
+      case "connecting":
+        return "bg-amber-500";
+      case "disconnected":
+      default:
+        return "bg-red-500";
+    }
+  }, [connectionStatus]);
+
+  return (
+    <div className="flex items-center" title={`event stream: ${connectionStatus}`}>
+      <span className={`inline-block w-3 h-3 rounded-full ${eventStatusColor} mr-2`}></span>
+    </div>
+  );
+};
+
+export default ConnectionStatusIcon;

ui/src/components/Header.tsx

@@ -0,0 +1,56 @@
+import { useCallback } from "react";
+import { RiMoonFill, RiSunFill } from "react-icons/ri";
+import { NavLink, type NavLinkRenderProps } from "react-router-dom";
+import { useTheme } from "../contexts/ThemeProvider";
+import ConnectionStatusIcon from "./ConnectionStatus";
+
+export function Header() {
+  const { screenWidth, toggleTheme, isDarkMode, appTitle, setAppTitle } = useTheme();
+  const handleTitleChange = useCallback(
+    (newTitle: string) => {
+      setAppTitle(newTitle.replace(/\n/g, "").trim().substring(0, 64) || "llama-swap");
+    },
+    [setAppTitle]
+  );
+
+  const navLinkClass = ({ isActive }: NavLinkRenderProps) =>
+    `text-gray-600 hover:text-black dark:text-gray-300 dark:hover:text-gray-100 p-1 ${isActive ? "font-semibold" : ""}`;
+
+  return (
+    <header className="flex items-center justify-between bg-surface border-b border-border p-2 px-4 h-[75px]">
+      {screenWidth !== "xs" && screenWidth !== "sm" && (
+        <h1
+          contentEditable
+          suppressContentEditableWarning
+          className="p-0 outline-none hover:bg-gray-100 dark:hover:bg-gray-700 rounded"
+          onBlur={(e) => handleTitleChange(e.currentTarget.textContent || "(set title)")}
+          onKeyDown={(e) => {
+            if (e.key === "Enter") {
+              e.preventDefault();
+              handleTitleChange(e.currentTarget.textContent || "(set title)");
+              e.currentTarget.blur();
+            }
+          }}
+        >
+          {appTitle}
+        </h1>
+      )}
+
+      <menu className="flex items-center gap-4">
+        <NavLink to="/" className={navLinkClass} type="button">
+          Logs
+        </NavLink>
+        <NavLink to="/models" className={navLinkClass} type="button">
+          Models
+        </NavLink>
+        <NavLink to="/activity" className={navLinkClass} type="button">
+          Activity
+        </NavLink>
+        <button className="" onClick={toggleTheme}>
+          {isDarkMode ? <RiMoonFill /> : <RiSunFill />}
+        </button>
+        <ConnectionStatusIcon />
+      </menu>
+    </header>
+  );
+}

ui/src/contexts/APIProvider.tsx

@@ -1,4 +1,5 @@
-import { useRef, createContext, useState, useContext, useEffect, useCallback, useMemo, type ReactNode } from "react";
+import { createContext, useState, useContext, useEffect, useCallback, useMemo, type ReactNode } from "react";
+import type { ConnectionState } from "../lib/types";
 
 type ModelStatus = "ready" | "starting" | "stopping" | "stopped" | "shutdown" | "unknown";
 const LOG_LENGTH_LIMIT = 1024 * 100; /* 100KB of log data */
@@ -15,19 +16,23 @@ interface APIProviderType {
   models: Model[];
   listModels: () => Promise<Model[]>;
   unloadAllModels: () => Promise<void>;
+  unloadSingleModel: (model: string) => Promise<void>;
   loadModel: (model: string) => Promise<void>;
   enableAPIEvents: (enabled: boolean) => void;
   proxyLogs: string;
   upstreamLogs: string;
   metrics: Metrics[];
+  connectionStatus: ConnectionState;
 }
 
 interface Metrics {
   id: number;
   timestamp: string;
   model: string;
+  cache_tokens: number;
   input_tokens: number;
   output_tokens: number;
+  prompt_per_second: number;
   tokens_per_second: number;
   duration_ms: number;
 }
@@ -47,11 +52,14 @@ type APIProviderProps = {
   autoStartAPIEvents?: boolean;
 };
 
+let apiEventSource: EventSource | null = null;
+
 export function APIProvider({ children, autoStartAPIEvents = true }: APIProviderProps) {
   const [proxyLogs, setProxyLogs] = useState("");
   const [upstreamLogs, setUpstreamLogs] = useState("");
   const [metrics, setMetrics] = useState<Metrics[]>([]);
-  const apiEventSource = useRef<EventSource | null>(null);
+  const [connectionStatus, setConnectionState] = useState<ConnectionState>("disconnected");
+  //const apiEventSource = useRef<EventSource | null>(null);
 
   const [models, setModels] = useState<Model[]>([]);
 
@@ -64,8 +72,8 @@ export function APIProvider({ children, autoStartAPIEvents = true }: APIProvider
 
   const enableAPIEvents = useCallback((enabled: boolean) => {
     if (!enabled) {
-      apiEventSource.current?.close();
+      apiEventSource?.close();
-      apiEventSource.current = null;
+      apiEventSource = null;
       setMetrics([]);
       return;
     }
@@ -74,15 +82,34 @@ export function APIProvider({ children, autoStartAPIEvents = true }: APIProvider
     const initialDelay = 1000; // 1 second
 
     const connect = () => {
-      const eventSource = new EventSource("/api/events");
+      apiEventSource?.close();
+      apiEventSource = new EventSource("/api/events");
 
-      eventSource.onmessage = (e: MessageEvent) => {
+      setConnectionState("connecting");
+
+      apiEventSource.onopen = () => {
+        // clear everything out on connect to keep things in sync
+        setProxyLogs("");
+        setUpstreamLogs("");
+        setMetrics([]); // clear metrics on reconnect
+        setModels([]); // clear models on reconnect
+        retryCount = 0;
+        setConnectionState("connected");
+      };
+
+      apiEventSource.onmessage = (e: MessageEvent) => {
         try {
           const message = JSON.parse(e.data) as APIEventEnvelope;
           switch (message.type) {
             case "modelStatus":
               {
                 const models = JSON.parse(message.data) as Model[];
+
+                // sort models by name and id
+                models.sort((a, b) => {
+                  return (a.name + a.id).localeCompare(b.name + b.id);
+                });
+
                 setModels(models);
               }
               break;
@@ -101,9 +128,9 @@ export function APIProvider({ children, autoStartAPIEvents = true }: APIProvider
 
             case "metrics":
               {
-                const newMetric = JSON.parse(message.data) as Metrics;
+                const newMetrics = JSON.parse(message.data) as Metrics[];
                 setMetrics((prevMetrics) => {
-                  return [newMetric, ...prevMetrics];
+                  return [...newMetrics, ...prevMetrics];
                 });
               }
               break;
@@ -112,14 +139,14 @@ export function APIProvider({ children, autoStartAPIEvents = true }: APIProvider
           console.error(e.data, err);
         }
       };
-      eventSource.onerror = () => {
+
-        eventSource.close();
+      apiEventSource.onerror = () => {
+        apiEventSource?.close();
         retryCount++;
         const delay = Math.min(initialDelay * Math.pow(2, retryCount - 1), 5000);
+        setConnectionState("disconnected");
         setTimeout(connect, delay);
       };
-
-      apiEventSource.current = eventSource;
     };
 
     connect();
@@ -151,7 +178,7 @@ export function APIProvider({ children, autoStartAPIEvents = true }: APIProvider
 
   const unloadAllModels = useCallback(async () => {
     try {
-      const response = await fetch(`/api/models/unload/`, {
+      const response = await fetch(`/api/models/unload`, {
         method: "POST",
       });
       if (!response.ok) {
@@ -163,6 +190,20 @@ export function APIProvider({ children, autoStartAPIEvents = true }: APIProvider
     }
   }, []);
 
+  const unloadSingleModel = useCallback(async (model: string) => {
+    try {
+      const response = await fetch(`/api/models/unload/${model}`, {
+        method: "POST",
+      });
+      if (!response.ok) {
+        throw new Error(`Failed to unload model: ${response.status}`);
+      }
+    } catch (error) {
+      console.error("Failed to unload model", model, error);
+      throw error;
+    }
+  }, []);
+
   const loadModel = useCallback(async (model: string) => {
     try {
       const response = await fetch(`/upstream/${model}/`, {
@@ -182,11 +223,13 @@ export function APIProvider({ children, autoStartAPIEvents = true }: APIProvider
       models,
       listModels,
       unloadAllModels,
+      unloadSingleModel,
       loadModel,
       enableAPIEvents,
       proxyLogs,
       upstreamLogs,
       metrics,
+      connectionStatus,
     }),
     [models, listModels, unloadAllModels, loadModel, enableAPIEvents, proxyLogs, upstreamLogs, metrics]
   );

ui/src/contexts/ThemeProvider.tsx

@@ -1,9 +1,18 @@
-import { createContext, useContext, useEffect, type ReactNode } from "react";
+import { createContext, useContext, useEffect, type ReactNode, useMemo, useState } from "react";
 import { usePersistentState } from "../hooks/usePersistentState";
+import type { ConnectionState } from "../lib/types";
 
+type ScreenWidth = "xs" | "sm" | "md" | "lg" | "xl" | "2xl";
 type ThemeContextType = {
   isDarkMode: boolean;
+  screenWidth: ScreenWidth;
+  isNarrow: boolean;
   toggleTheme: () => void;
+
+  // for managing the window title and connection state information
+  appTitle: string;
+  setAppTitle: (title: string) => void;
+  setConnectionState: (state: ConnectionState) => void;
 };
 
 const ThemeContext = createContext<ThemeContextType | undefined>(undefined);
@@ -13,15 +22,70 @@ type ThemeProviderProps = {
 };
 
 export function ThemeProvider({ children }: ThemeProviderProps) {
+  const [appTitle, setAppTitle] = usePersistentState("app-title", "llama-swap");
+  const [connectionState, setConnectionState] = useState<ConnectionState>("disconnected");
+
+  /**
+   * Set the document.title with informative information
+   */
+  useEffect(() => {
+    const connectionIcon = connectionState === "connecting" ? "🟡" : connectionState === "connected" ? "🟢" : "🔴";
+    document.title = connectionIcon + " " + appTitle; // Set initial title
+  }, [appTitle, connectionState]);
+
   const [isDarkMode, setIsDarkMode] = usePersistentState<boolean>("theme", false);
+  const [screenWidth, setScreenWidth] = useState<ScreenWidth>("md"); // Default to md
+
+  // matches tailwind classes
+  // https://tailwindcss.com/docs/responsive-design
+  useEffect(() => {
+    const checkInnerWidth = () => {
+      const innerWidth = window.innerWidth;
+      if (innerWidth < 640) {
+        setScreenWidth("xs");
+      } else if (innerWidth < 768) {
+        setScreenWidth("sm");
+      } else if (innerWidth < 1024) {
+        setScreenWidth("md");
+      } else if (innerWidth < 1280) {
+        setScreenWidth("lg");
+      } else if (innerWidth < 1536) {
+        setScreenWidth("xl");
+      } else {
+        setScreenWidth("2xl");
+      }
+    };
+
+    checkInnerWidth();
+    window.addEventListener("resize", checkInnerWidth);
+
+    return () => window.removeEventListener("resize", checkInnerWidth);
+  }, []);
 
   useEffect(() => {
     document.documentElement.setAttribute("data-theme", isDarkMode ? "dark" : "light");
   }, [isDarkMode]);
 
   const toggleTheme = () => setIsDarkMode((prev) => !prev);
+  const isNarrow = useMemo(() => {
+    return screenWidth === "xs" || screenWidth === "sm" || screenWidth === "md";
+  }, [screenWidth]);
 
-  return <ThemeContext.Provider value={{ isDarkMode, toggleTheme }}>{children}</ThemeContext.Provider>;
+  return (
+    <ThemeContext.Provider
+      value={{
+        isDarkMode,
+        toggleTheme,
+        screenWidth,
+        isNarrow,
+        appTitle,
+        setAppTitle,
+        setConnectionState,
+      }}
+    >
+      {children}
+    </ThemeContext.Provider>
+  );
 }
 
 export function useTheme(): ThemeContextType {

ui/src/index.css

@@ -93,6 +93,14 @@
     @apply px-4;
   }
 
+  /* Tables */
+  table th {
+    @apply p-2 font-semibold;
+  }
+  table td {
+    @apply p-2;
+  }
+
   /* Navigation Header */
 
   .navlink {
@@ -122,7 +130,7 @@
 
   /* Status Badges */
   .status {
-    @apply inline-block px-2 py-1 text-xs font-medium rounded-full;
+    @apply inline-block px-2 py-1 text-xs font-medium rounded-lg;
   }
 
   .status--ready {
@@ -140,7 +148,7 @@
 
   /* Buttons */
   .btn {
-    @apply bg-surface p-2 px-4 text-sm rounded-full border border-2 transition-colors duration-200 border-btn-border;
+    @apply bg-surface py-2 px-4 text-sm rounded-md border transition-colors duration-200 border-btn-border;
   }
 
   .btn:hover {

ui/src/lib/types.ts

@@ -0,0 +1 @@
+export type ConnectionState = "connected" | "connecting" | "disconnected";

ui/src/main.tsx

@@ -3,11 +3,14 @@ import { createRoot } from "react-dom/client";
 import "./index.css";
 import App from "./App.tsx";
 import { ThemeProvider } from "./contexts/ThemeProvider";
+import { APIProvider } from "./contexts/APIProvider";
 
 createRoot(document.getElementById("root")!).render(
   <StrictMode>
     <ThemeProvider>
-      <App />
+      <APIProvider>
+        <App />
+      </APIProvider>
     </ThemeProvider>
   </StrictMode>
 );

ui/src/pages/Activity.tsx

@@ -1,10 +1,6 @@
-import { useState, useEffect } from "react";
+import { useMemo } from "react";
 import { useAPI } from "../contexts/APIProvider";
 
-const formatTimestamp = (timestamp: string): string => {
-  return new Date(timestamp).toLocaleString();
-};
-
 const formatSpeed = (speed: number): string => {
   return speed < 0 ? "unknown" : speed.toFixed(2) + " t/s";
 };
@@ -13,57 +9,80 @@ const formatDuration = (ms: number): string => {
   return (ms / 1000).toFixed(2) + "s";
 };
 
-const ActivityPage = () => {
+const formatRelativeTime = (timestamp: string): string => {
-  const { metrics } = useAPI();
+  const now = new Date();
-  const [error, setError] = useState<string | null>(null);
+  const date = new Date(timestamp);
+  const diffInSeconds = Math.floor((now.getTime() - date.getTime()) / 1000);
 
-  useEffect(() => {
+  // Handle future dates by returning "just now"
-    if (metrics.length > 0) {
+  if (diffInSeconds < 5) {
-      setError(null);
+    return "now";
-    }
-  }, [metrics]);
-
-  if (error) {
-    return (
-      <div className="p-6">
-        <h1 className="text-2xl font-bold mb-4">Activity</h1>
-        <div className="bg-red-50 border border-red-200 rounded-md p-4">
-          <p className="text-red-800">{error}</p>
-        </div>
-      </div>
-    );
   }
 
-  return (
+  if (diffInSeconds < 60) {
-    <div className="p-6">
+    return `${diffInSeconds}s ago`;
-      <h1 className="text-2xl font-bold mb-4">Activity</h1>
+  }
 
-      {metrics.length === 0 ? (
+  const diffInMinutes = Math.floor(diffInSeconds / 60);
+  if (diffInMinutes < 60) {
+    return `${diffInMinutes}m ago`;
+  }
+
+  const diffInHours = Math.floor(diffInMinutes / 60);
+  if (diffInHours < 24) {
+    return `${diffInHours}h ago`;
+  }
+
+  return "a while ago";
+};
+
+const ActivityPage = () => {
+  const { metrics } = useAPI();
+  const sortedMetrics = useMemo(() => {
+    return [...metrics].sort((a, b) => b.id - a.id);
+  }, [metrics]);
+
+  return (
+    <div className="p-2">
+      <h1 className="text-2xl font-bold">Activity</h1>
+
+      {metrics.length === 0 && (
         <div className="text-center py-8">
           <p className="text-gray-600">No metrics data available</p>
         </div>
-      ) : (
+      )}
-        <div className="overflow-x-auto">
+      {metrics.length > 0 && (
+        <div className="card overflow-auto">
           <table className="min-w-full divide-y">
-            <thead>
+            <thead className="border-gray-200 dark:border-white/10">
-              <tr>
+              <tr className="text-left text-xs uppercase tracking-wider">
-                <th className="px-6 py-3 text-left text-xs font-medium uppercase tracking-wider">Timestamp</th>
+                <th className="px-6 py-3">ID</th>
-                <th className="px-6 py-3 text-left text-xs font-medium uppercase tracking-wider">Model</th>
+                <th className="px-6 py-3">Time</th>
-                <th className="px-6 py-3 text-left text-xs font-medium uppercase tracking-wider">Input Tokens</th>
+                <th className="px-6 py-3">Model</th>
-                <th className="px-6 py-3 text-left text-xs font-medium uppercase tracking-wider">Output Tokens</th>
+                <th className="px-6 py-3">
-                <th className="px-6 py-3 text-left text-xs font-medium uppercase tracking-wider">Generation Speed</th>
+                  Cached <Tooltip content="prompt tokens from cache" />
-                <th className="px-6 py-3 text-left text-xs font-medium uppercase tracking-wider">Duration</th>
+                </th>
+                <th className="px-6 py-3">
+                  Prompt <Tooltip content="new prompt tokens processed" />
+                </th>
+                <th className="px-6 py-3">Generated</th>
+                <th className="px-6 py-3">Prompt Processing</th>
+                <th className="px-6 py-3">Generation Speed</th>
+                <th className="px-6 py-3">Duration</th>
               </tr>
             </thead>
             <tbody className="divide-y">
-              {metrics.map((metric, index) => (
+              {sortedMetrics.map((metric) => (
-                <tr key={`${metric.id}-${index}`}>
+                <tr key={metric.id} className="whitespace-nowrap text-sm border-gray-200 dark:border-white/10">
-                  <td className="px-6 py-4 whitespace-nowrap text-sm">{formatTimestamp(metric.timestamp)}</td>
+                  <td className="px-4 py-4">{metric.id + 1 /* un-zero index */}</td>
-                  <td className="px-6 py-4 whitespace-nowrap text-sm">{metric.model}</td>
+                  <td className="px-6 py-4">{formatRelativeTime(metric.timestamp)}</td>
-                  <td className="px-6 py-4 whitespace-nowrap text-sm">{metric.input_tokens.toLocaleString()}</td>
+                  <td className="px-6 py-4">{metric.model}</td>
-                  <td className="px-6 py-4 whitespace-nowrap text-sm">{metric.output_tokens.toLocaleString()}</td>
+                  <td className="px-6 py-4">{metric.cache_tokens > 0 ? metric.cache_tokens.toLocaleString() : "-"}</td>
-                  <td className="px-6 py-4 whitespace-nowrap text-sm">{formatSpeed(metric.tokens_per_second)}</td>
+                  <td className="px-6 py-4">{metric.input_tokens.toLocaleString()}</td>
-                  <td className="px-6 py-4 whitespace-nowrap text-sm">{formatDuration(metric.duration_ms)}</td>
+                  <td className="px-6 py-4">{metric.output_tokens.toLocaleString()}</td>
+                  <td className="px-6 py-4">{formatSpeed(metric.prompt_per_second)}</td>
+                  <td className="px-6 py-4">{formatSpeed(metric.tokens_per_second)}</td>
+                  <td className="px-6 py-4">{formatDuration(metric.duration_ms)}</td>
                 </tr>
               ))}
             </tbody>
@@ -74,4 +93,28 @@ const ActivityPage = () => {
   );
 };
 
+interface TooltipProps {
+  content: string;
+}
+
+const Tooltip: React.FC<TooltipProps> = ({ content }) => {
+  return (
+    <div className="relative group inline-block">
+      ⓘ
+      <div
+        className="absolute top-full left-1/2 transform -translate-x-1/2 mt-2
+                     px-3 py-2 bg-gray-900 text-white text-sm rounded-md
+                     opacity-0 group-hover:opacity-100 transition-opacity
+                     duration-200 pointer-events-none whitespace-nowrap z-50 normal-case"
+      >
+        {content}
+        <div
+          className="absolute bottom-full left-1/2 transform -translate-x-1/2
+                       border-4 border-transparent border-b-gray-900"
+        ></div>
+      </div>
+    </div>
+  );
+};
+
 export default ActivityPage;

ui/src/pages/LogViewer.tsx

@@ -1,15 +1,38 @@
 import { useState, useEffect, useRef, useMemo, useCallback } from "react";
 import { useAPI } from "../contexts/APIProvider";
 import { usePersistentState } from "../hooks/usePersistentState";
+import { Panel, PanelGroup, PanelResizeHandle } from "react-resizable-panels";
+import {
+  RiTextWrap,
+  RiAlignJustify,
+  RiFontSize,
+  RiMenuSearchLine,
+  RiMenuSearchFill,
+  RiCloseCircleFill,
+} from "react-icons/ri";
+import { useTheme } from "../contexts/ThemeProvider";
 
 const LogViewer = () => {
   const { proxyLogs, upstreamLogs } = useAPI();
+  const { screenWidth } = useTheme();
+  const direction = screenWidth === "xs" || screenWidth === "sm" ? "vertical" : "horizontal";
 
   return (
-    <div className="flex flex-col lg:flex-row gap-5" style={{ height: "calc(100vh - 125px)" }}>
+    <PanelGroup direction={direction} className="gap-2" autoSaveId="logviewer-panel-group">
-      <LogPanel id="proxy" title="Proxy Logs" logData={proxyLogs} />
+      <Panel id="proxy" defaultSize={50} minSize={5} maxSize={100} collapsible={true}>
-      <LogPanel id="upstream" title="Upstream Logs" logData={upstreamLogs} />
+        <LogPanel id="proxy" title="Proxy Logs" logData={proxyLogs} />
-    </div>
+      </Panel>
+      <PanelResizeHandle
+        className={
+          direction === "horizontal"
+            ? "w-2 h-full bg-primary hover:bg-success transition-colors rounded"
+            : "w-full h-2 bg-primary hover:bg-success transition-colors rounded"
+        }
+      />
+      <Panel id="upstream" defaultSize={50} minSize={5} maxSize={100} collapsible={true}>
+        <LogPanel id="upstream" title="Upstream Logs" logData={upstreamLogs} />
+      </Panel>
+    </PanelGroup>
   );
 };
 
@@ -17,17 +40,15 @@ interface LogPanelProps {
   id: string;
   title: string;
   logData: string;
-  className?: string;
 }
-
+export const LogPanel = ({ id, title, logData }: LogPanelProps) => {
-export const LogPanel = ({ id, title, logData, className }: LogPanelProps) => {
-  const [isCollapsed, setIsCollapsed] = usePersistentState(`logPanel-${id}-isCollapsed`, false);
   const [filterRegex, setFilterRegex] = useState("");
   const [fontSize, setFontSize] = usePersistentState<"xxs" | "xs" | "small" | "normal">(
     `logPanel-${id}-fontSize`,
     "normal"
   );
   const [wrapText, setTextWrap] = usePersistentState(`logPanel-${id}-wrapText`, false);
+  const [showFilter, setShowFilter] = usePersistentState(`logPanel-${id}-showFilter`, false);
 
   const textWrapClass = useMemo(() => {
     return wrapText ? "whitespace-pre-wrap" : "whitespace-pre";
@@ -48,6 +69,19 @@ export const LogPanel = ({ id, title, logData, className }: LogPanelProps) => {
     });
   }, []);
 
+  const toggleWrapText = useCallback(() => {
+    setTextWrap((prev) => !prev);
+  }, []);
+
+  const toggleFilter = useCallback(() => {
+    if (showFilter) {
+      setShowFilter(false);
+      setFilterRegex(""); // Clear filter when closing
+    } else {
+      setShowFilter(true);
+    }
+  }, [filterRegex, setFilterRegex, showFilter]);
+
   const fontSizeClass = useMemo(() => {
     switch (fontSize) {
       case "xxs":
@@ -81,58 +115,47 @@ export const LogPanel = ({ id, title, logData, className }: LogPanelProps) => {
   }, [filteredLogs]);
 
   return (
-    <div
+    <div className="rounded-lg overflow-hidden flex flex-col bg-gray-950/5 dark:bg-white/10 h-full p-1">
-      className={`bg-surface border border-border rounded-lg overflow-hidden flex flex-col ${
+      <div className="p-4">
-        !isCollapsed && "h-full"
+        <div className="flex items-center justify-between">
-      } ${className || ""}`}
+          <h3 className="m-0 text-lg p-0">{title}</h3>
-    >
+
-      <div className="p-4 border-b border-border bg-secondary">
+          <div className="flex gap-2 items-center">
-        <div className="flex flex-col md:flex-row md:items-center md:justify-between gap-4">
+            <button className="btn border-0" onClick={toggleFontSize}>
-          {/* Title - Always full width on mobile, normal on desktop */}
+              <RiFontSize />
-          <div className="w-full md:w-auto" onClick={() => setIsCollapsed(!isCollapsed)}>
+            </button>
-            <h3 className="m-0 text-lg p-0">{title}</h3>
+            <button className="btn border-0" onClick={toggleWrapText}>
+              {wrapText ? <RiTextWrap /> : <RiAlignJustify />}
+            </button>
+            <button className="btn border-0" onClick={toggleFilter}>
+              {showFilter ? <RiMenuSearchFill /> : <RiMenuSearchLine />}
+            </button>
           </div>
+        </div>
 
-          {!isCollapsed && (
+        {/* Filtering Options - Full width on mobile, normal on desktop */}
-            <div className="flex flex-col sm:flex-row gap-4 w-full md:w-auto">
+        {showFilter && (
-              {/* Sizing Buttons - Stacks vertically on mobile */}
+          <div className="mt-2 w-full">
-              <div className="flex flex-wrap gap-2">
+            <div className="flex gap-2 items-center w-full">
-                <button className="btn" onClick={toggleFontSize}>
+              <input
-                  font: {fontSize}
+                type="text"
-                </button>
+                className="w-full text-sm border border-gray-950/10 dark:border-white/5 p-2 rounded outline-none"
-                <button className="btn" onClick={() => setTextWrap((prev) => !prev)}>
+                placeholder="Filter logs..."
-                  {wrapText ? "wrap" : "wrap off"}
+                value={filterRegex}
-                </button>
+                onChange={(e) => setFilterRegex(e.target.value)}
-              </div>
+              />
-
+              <button className="pl-2" onClick={() => setFilterRegex("")}>
-              {/* Filtering Options - Full width on mobile, normal on desktop */}
+                <RiCloseCircleFill size="24" />
-              <div className="flex flex-1 min-w-0 gap-2">
+              </button>
-                <input
-                  type="text"
-                  className="flex-1 min-w-[120px] text-sm border p-2 rounded"
-                  placeholder="Filter logs..."
-                  value={filterRegex}
-                  onChange={(e) => setFilterRegex(e.target.value)}
-                />
-                <button className="btn" onClick={() => setFilterRegex("")}>
-                  Clear
-                </button>
-              </div>
             </div>
-          )}
+          </div>
-        </div>
+        )}
+      </div>
+      <div className="rounded-lg bg-background font-mono text-sm flex-1 overflow-hidden">
+        <pre ref={preTagRef} className={`${textWrapClass} ${fontSizeClass} h-full overflow-auto p-4`}>
+          {filteredLogs}
+        </pre>
       </div>
-
-      {!isCollapsed && (
-        <div className="flex-1 bg-background font-mono text-sm p-3 overflow-hidden">
-          <pre
-            ref={preTagRef}
-            className={`h-full p-4 overflow-y-auto whitespace-pre min-h-0 ${textWrapClass} ${fontSizeClass}`}
-          >
-            {filteredLogs}
-          </pre>
-        </div>
-      )}
     </div>
   );
 };

ui/src/pages/Models.tsx

@@ -2,11 +2,45 @@ import { useState, useCallback, useMemo } from "react";
 import { useAPI } from "../contexts/APIProvider";
 import { LogPanel } from "./LogViewer";
 import { usePersistentState } from "../hooks/usePersistentState";
+import { Panel, PanelGroup, PanelResizeHandle } from "react-resizable-panels";
+import { useTheme } from "../contexts/ThemeProvider";
+import { RiEyeFill, RiEyeOffFill, RiSwapBoxFill, RiEjectLine } from "react-icons/ri";
 
 export default function ModelsPage() {
-  const { models, unloadAllModels, loadModel, upstreamLogs, metrics } = useAPI();
+  const { isNarrow } = useTheme();
+  const direction = isNarrow ? "vertical" : "horizontal";
+  const { upstreamLogs } = useAPI();
+
+  return (
+    <PanelGroup direction={direction} className="gap-2" autoSaveId={"models-panel-group"}>
+      <Panel id="models" defaultSize={50} minSize={isNarrow ? 0 : 25} maxSize={100} collapsible={isNarrow}>
+        <ModelsPanel />
+      </Panel>
+
+      <PanelResizeHandle
+        className={
+          direction === "horizontal"
+            ? "w-2 h-full bg-primary hover:bg-success transition-colors rounded"
+            : "w-full h-2 bg-primary hover:bg-success transition-colors rounded"
+        }
+      />
+      <Panel collapsible={true} defaultSize={50} minSize={0}>
+        <div className="flex flex-col h-full space-y-4">
+          {direction === "horizontal" && <StatsPanel />}
+          <div className="flex-1 min-h-0">
+            <LogPanel id="modelsupstream" title="Upstream Logs" logData={upstreamLogs} />
+          </div>
+        </div>
+      </Panel>
+    </PanelGroup>
+  );
+}
+
+function ModelsPanel() {
+  const { models, loadModel, unloadAllModels, unloadSingleModel } = useAPI();
   const [isUnloading, setIsUnloading] = useState(false);
   const [showUnlisted, setShowUnlisted] = usePersistentState("showUnlisted", true);
+  const [showIdorName, setShowIdorName] = usePersistentState<"id" | "name">("showIdorName", "id"); // true = show ID, false = show name
 
   const filteredModels = useMemo(() => {
     return models.filter((model) => showUnlisted || !model.unlisted);
@@ -19,103 +53,137 @@ export default function ModelsPage() {
     } catch (e) {
       console.error(e);
     } finally {
-      // at least give it a second to show the unloading message
       setTimeout(() => {
         setIsUnloading(false);
       }, 1000);
     }
-  }, []);
+  }, [unloadAllModels]);
 
-  const [totalRequests, totalTokens, avgTokensPerSecond] = useMemo(() => {
+  const toggleIdorName = useCallback(() => {
-    const totalRequests = metrics.length;
+    setShowIdorName((prev) => (prev === "name" ? "id" : "name"));
-    if (totalRequests === 0) {
+  }, [showIdorName]);
-      return [0, 0, 0];
-    }
-    const totalTokens = metrics.reduce((sum, m) => sum + m.output_tokens, 0);
-    const avgTokensPerSecond = (metrics.reduce((sum, m) => sum + m.tokens_per_second, 0) / totalRequests).toFixed(2);
-    return [totalRequests, totalTokens, avgTokensPerSecond];
-  }, [metrics]);
 
   return (
-    <div>
+    <div className="card h-full flex flex-col">
-      <div className="flex flex-col md:flex-row gap-4">
+      <div className="shrink-0">
-        {/* Left Column */}
+        <h2>Models</h2>
-        <div className="w-full md:w-1/2 flex items-top">
+        <div className="flex justify-between">
-          <div className="card w-full">
+          <div className="flex gap-2">
-            <h2 className="">Models</h2>
+            <button
-            <div className="flex justify-between">
+              className="btn text-base flex items-center gap-2"
-              <button className="btn" onClick={() => setShowUnlisted(!showUnlisted)} style={{ lineHeight: "1.2" }}>
+              onClick={toggleIdorName}
-                {showUnlisted ? "🟢 unlisted" : "⚫️ unlisted"}
+              style={{ lineHeight: "1.2" }}
-              </button>
+            >
-              <button className="btn" onClick={handleUnloadAllModels} disabled={isUnloading}>
+              <RiSwapBoxFill size="20" /> {showIdorName === "id" ? "ID" : "Name"}
-                {isUnloading ? "Stopping ..." : "Stop All"}
+            </button>
-              </button>
-            </div>
 
-            <table className="w-full mt-4">
+            <button
-              <thead>
+              className="btn text-base flex items-center gap-2"
-                <tr className="border-b border-primary">
+              onClick={() => setShowUnlisted(!showUnlisted)}
-                  <th className="text-left p-2">Name</th>
+              style={{ lineHeight: "1.2" }}
-                  <th className="text-left p-2"></th>
+            >
-                  <th className="text-left p-2">State</th>
+              {showUnlisted ? <RiEyeFill size="20" /> : <RiEyeOffFill size="20" />} unlisted
-                </tr>
+            </button>
-              </thead>
-              <tbody>
-                {filteredModels.map((model) => (
-                  <tr key={model.id} className="border-b hover:bg-secondary-hover border-border">
-                    <td className="p-2">
-                      <a href={`/upstream/${model.id}/`} className="underline" target="_blank">
-                        {model.name !== "" ? model.name : model.id}
-                      </a>
-                      {model.description != "" && (
-                        <p>
-                          <em>{model.description}</em>
-                        </p>
-                      )}
-                    </td>
-                    <td className="p-2 w-[50px]">
-                      <button
-                        className="btn btn--sm"
-                        disabled={model.state !== "stopped"}
-                        onClick={() => loadModel(model.id)}
-                      >
-                        Load
-                      </button>
-                    </td>
-                    <td className="p-2 w-[75px]">
-                      <span className={`status status--${model.state}`}>{model.state}</span>
-                    </td>
-                  </tr>
-                ))}
-              </tbody>
-            </table>
           </div>
+          <button
+            className="btn text-base flex items-center gap-2"
+            onClick={handleUnloadAllModels}
+            disabled={isUnloading}
+          >
+            <RiEjectLine size="24" /> {isUnloading ? "Unloading..." : "Unload All"}
+          </button>
         </div>
+      </div>
 
-        {/* Right Column */}
+      <div className="flex-1 overflow-y-auto">
-        <div className="w-full md:w-1/2 flex flex-col" style={{ height: "calc(100vh - 125px)" }}>
+        <table className="w-full">
-          <div className="card mb-4 min-h-[225px]">
+          <thead className="sticky top-0 bg-card z-10">
-            <h2>Chat Activity</h2>
+            <tr className="text-left border-b border-gray-200 dark:border-white/10 bg-surface">
-            <table className="w-full border border-gray-200">
+              <th>{showIdorName === "id" ? "Model ID" : "Name"}</th>
-              <tbody>
+              <th></th>
-                <tr className="border-b border-gray-200">
+              <th>State</th>
-                  <td className="py-2 px-4 font-medium border-r border-gray-200">Requests</td>
+            </tr>
-                  <td className="py-2 px-4 text-right">{totalRequests}</td>
+          </thead>
-                </tr>
+          <tbody>
-                <tr className="border-b border-gray-200">
+            {filteredModels.map((model) => (
-                  <td className="py-2 px-4 font-medium border-r border-gray-200">Total Tokens Generated</td>
+              <tr key={model.id} className="border-b hover:bg-secondary-hover border-gray-200">
-                  <td className="py-2 px-4 text-right">{totalTokens}</td>
+                <td className={`${model.unlisted ? "text-txtsecondary" : ""}`}>
-                </tr>
+                  <a href={`/upstream/${model.id}/`} className="font-semibold" target="_blank">
-                <tr>
+                    {showIdorName === "id" ? model.id : model.name !== "" ? model.name : model.id}
-                  <td className="py-2 px-4 font-medium border-r border-gray-200">Average Tokens/Second</td>
+                  </a>
-                  <td className="py-2 px-4 text-right">{avgTokensPerSecond}</td>
-                </tr>
-              </tbody>
-            </table>
-          </div>
 
-          <LogPanel id="modelsupstream" title="Upstream Logs" logData={upstreamLogs} />
+                  {!!model.description && (
-        </div>
+                    <p className={model.unlisted ? "text-opacity-70" : ""}>
+                      <em>{model.description}</em>
+                    </p>
+                  )}
+                </td>
+                <td className="w-12">
+                  {model.state === "stopped" ? (
+                    <button className="btn btn--sm" onClick={() => loadModel(model.id)}>
+                      Load
+                    </button>
+                  ) : (
+                    <button
+                      className="btn btn--sm"
+                      onClick={() => unloadSingleModel(model.id)}
+                      disabled={model.state !== "ready"}
+                    >
+                      Unload
+                    </button>
+                  )}
+                </td>
+                <td className="w-20">
+                  <span className={`w-16 text-center status status--${model.state}`}>{model.state}</span>
+                </td>
+              </tr>
+            ))}
+          </tbody>
+        </table>
+      </div>
+    </div>
+  );
+}
+
+function StatsPanel() {
+  const { metrics } = useAPI();
+
+  const [totalRequests, totalInputTokens, totalOutputTokens, avgTokensPerSecond] = useMemo(() => {
+    const totalRequests = metrics.length;
+    if (totalRequests === 0) {
+      return [0, 0, 0];
+    }
+    const totalInputTokens = metrics.reduce((sum, m) => sum + m.input_tokens, 0);
+    const totalOutputTokens = metrics.reduce((sum, m) => sum + m.output_tokens, 0);
+    const avgTokensPerSecond = (metrics.reduce((sum, m) => sum + m.tokens_per_second, 0) / totalRequests).toFixed(2);
+    return [totalRequests, totalInputTokens, totalOutputTokens, avgTokensPerSecond];
+  }, [metrics]);
+
+  return (
+    <div className="card">
+      <div className="rounded-lg overflow-hidden border border-gray-200 dark:border-white/10">
+        <table className="w-full">
+          <thead>
+            <tr className="border-b border-gray-200 dark:border-white/10 text-right">
+              <th>Requests</th>
+              <th className="border-l border-gray-200 dark:border-white/10">Processed</th>
+              <th className="border-l border-gray-200 dark:border-white/10">Generated</th>
+              <th className="border-l border-gray-200 dark:border-white/10">Tokens/Sec</th>
+            </tr>
+          </thead>
+          <tbody>
+            <tr className="text-right">
+              <td className="border-r border-gray-200 dark:border-white/10">{totalRequests}</td>
+              <td className="border-r border-gray-200 dark:border-white/10">
+                {new Intl.NumberFormat().format(totalInputTokens)}
+              </td>
+              <td className="border-r border-gray-200 dark:border-white/10">
+                {new Intl.NumberFormat().format(totalOutputTokens)}
+              </td>
+              <td>{avgTokensPerSecond}</td>
+            </tr>
+          </tbody>
+        </table>
       </div>
     </div>
   );

ui/vite.config.ts

@@ -15,6 +15,7 @@ export default defineConfig({
       "/api": "http://localhost:8080", // Proxy API calls to Go backend during development
       "/logs": "http://localhost:8080",
       "/upstream": "http://localhost:8080",
+      "/unload": "http://localhost:8080",
     },
   },
 });