UI styling / code improvements (#307 )

Clean up and improve UI styling * fix: UI - dependency cleanup * chore: UI - start script * refactor: UI - Extract Header * fix: UI - Header styling * fix: UI - LogViewer styling * fix: UI - Models styling * fix: UI - Activity styling * fix: UI - ConnectionStatus colors * review: UI - table border colors
upstream handler support for model names with forward slash (#298 )
2025-09-19 10:47:17 -07:00 · 2025-09-13 13:37:03 -07:00 · 2025-09-09 16:07:46 -07:00 · 2025-09-07 21:48:58 -07:00 · 2025-09-06 13:58:02 -07:00 · 2025-09-04 11:57:28 -07:00
24 changed files with 671 additions and 224 deletions
@@ -7,7 +7,7 @@

 llama-swap is a light weight, transparent proxy server that provides automatic model swapping to llama.cpp's server.

-Written in golang, it is very easy to install (single binary with no dependencies) and configure (single yaml file). To get started, download a pre-built binary or use the provided docker images.
+Written in golang, it is very easy to install (single binary with no dependencies) and configure (single yaml file). To get started, download a pre-built binary, a provided docker images or Homebrew.

 ## Features:

@@ -18,9 +18,12 @@ Written in golang, it is very easy to install (single binary with no dependencie
  - `v1/completions`
  - `v1/chat/completions`
  - `v1/embeddings`
-  - `v1/rerank`, `v1/reranking`, `rerank`
  - `v1/audio/speech` ([#36](https://github.com/mostlygeek/llama-swap/issues/36))
  - `v1/audio/transcriptions` ([docs](https://github.com/mostlygeek/llama-swap/issues/41#issuecomment-2722637867))
+- ✅ llama-server (llama.cpp) supported endpoints:
+  - `v1/rerank`, `v1/reranking`, `/rerank`
+  - `/infill` - for code infilling
+  - `/completion` - for completion endpoint
 - ✅ llama-swap custom API endpoints
  - `/ui` - web UI
  - `/log` - remote log monitoring
@@ -31,7 +34,7 @@ Written in golang, it is very easy to install (single binary with no dependencie
 - ✅ Run multiple models at once with `Groups` ([#107](https://github.com/mostlygeek/llama-swap/issues/107))
 - ✅ Automatic unloading of models after timeout by setting a `ttl`
 - ✅ Use any local OpenAI compatible server (llama.cpp, vllm, tabbyAPI, etc)
- ✅ Reliable Docker and Podman support with `cmdStart` and `cmdStop`
+- ✅ Reliable Docker and Podman support using `cmd` and `cmdStop` together
 - ✅ Full control over server settings per model
 - ✅ Preload models on startup with `hooks` ([#235](https://github.com/mostlygeek/llama-swap/pull/235))

@@ -70,6 +73,30 @@ However, there are many more capabilities that llama-swap supports:

 See the [configuration documentation](https://github.com/mostlygeek/llama-swap/wiki/Configuration) in the wiki all options and examples.

+## Reverse Proxy Configuration (nginx)
+
+If you deploy llama-swap behind nginx, disable response buffering for streaming endpoints. By default, nginx buffers responses which breaks Server‑Sent Events (SSE) and streaming chat completion. ([#236](https://github.com/mostlygeek/llama-swap/issues/236))
+
+Recommended nginx configuration snippets:
+
+```nginx
+# SSE for UI events/logs
+location /api/events {
+    proxy_pass http://your-llama-swap-backend;
+    proxy_buffering off;
+    proxy_cache off;
+}
+
+# Streaming chat completions (stream=true)
+location /v1/chat/completions {
+    proxy_pass http://your-llama-swap-backend;
+    proxy_buffering off;
+    proxy_cache off;
+}
+```
+
+As a safeguard, llama-swap also sets `X-Accel-Buffering: no` on SSE responses. However, explicitly disabling `proxy_buffering` at your reverse proxy is still recommended for reliable streaming behavior.
+
 ## Web UI

 llama-swap includes a real time web interface for monitoring logs and models:
@@ -204,4 +231,7 @@ For Python based inference servers like vllm or tabbyAPI it is recommended to ru

 ## Star History

+> [!NOTE]
+> ⭐️ Star this project to help others discover it! 
+
 [![Star History Chart](https://api.star-history.com/svg?repos=mostlygeek/llama-swap&type=Date)](https://www.star-history.com/#mostlygeek/llama-swap&Date)
@@ -49,8 +49,8 @@ macros:
 # - required
 # - each key is the model's ID, used in API requests
 # - model settings have default values that are used if they are not defined here
-# - below are examples of the various settings a model can have:
-# - available model settings: env, cmd, cmdStop, proxy, aliases, checkEndpoint, ttl, unlisted
+# - the model's ID is available in the ${MODEL_ID} macro, also available in macros defined above
+# - below are examples of the all the settings a model can have
 models:

  # keys are the model names used in API requests
@@ -129,6 +129,15 @@ models:
      # - recommended to stick to sampling parameters
      strip_params: "temperature, top_p, top_k"

+    # concurrencyLimit: overrides the allowed number of active parallel requests to a model
+    # - optional, default: 0
+    # - useful for limiting the number of active parallel requests a model can process
+    # - must be set per model
+    # - any number greater than 0 will override the internal default value of 10
+    # - any requests that exceeds the limit will receive an HTTP 429 Too Many Requests response
+    # - recommended to be omitted and the default used
+    concurrencyLimit: 0
+
  # Unlisted model example:
  "qwen-unlisted":
    # unlisted: boolean, true or false
@@ -139,12 +148,12 @@ models:
    cmd: llama-server --port ${PORT} -m Llama-3.2-1B-Instruct-Q4_K_M.gguf -ngl 0

  # Docker example:
-  # container run times like Docker and Podman can be used reliably with a
-  # a combination of cmd and cmdStop.
+  # container runtimes like Docker and Podman can be used reliably with
+  # a combination of cmd, cmdStop, and ${MODEL_ID}
  "docker-llama":
    proxy: "http://127.0.0.1:${PORT}"
    cmd: |
-      docker run --name dockertest
+      docker run --name ${MODEL_ID}
      --init --rm -p ${PORT}:8080 -v /mnt/nvme/models:/models
      ghcr.io/ggml-org/llama.cpp:server
      --model '/models/Qwen2.5-Coder-0.5B-Instruct-Q4_K_M.gguf'
@@ -158,7 +167,7 @@ models:
    # - on POSIX systems: a SIGTERM signal is sent
    # - on Windows, calls taskkill to stop the process
    # - processes have 5 seconds to shutdown until forceful termination is attempted
-    cmdStop: docker stop dockertest
+    cmdStop: docker stop ${MODEL_ID}

 # groups: a dictionary of group settings
 # - optional, default: empty dictionary
@@ -153,6 +153,19 @@ func main() {

 	})

+	// llama-server compatibility: /completion
+	r.POST("/completion", func(c *gin.Context) {
+		c.Header("Content-Type", "application/json")
+		c.JSON(http.StatusOK, gin.H{
+			"responseMessage": *responseMessage,
+			"usage": gin.H{
+				"completion_tokens": 10,
+				"prompt_tokens":     25,
+				"total_tokens":      35,
+			},
+		})
+	})
+
 	// issue #41
 	r.POST("/v1/audio/transcriptions", func(c *gin.Context) {
 		// Parse the multipart form
@@ -237,7 +237,7 @@ func LoadConfigFromReader(r io.Reader) (Config, error) {

 	- name must fit the regex ^[a-zA-Z0-9_-]+$
 	- names must be less than 64 characters (no reason, just cause)
-	- name can not be any reserved macros: PORT
+	- name can not be any reserved macros: PORT, MODEL_ID
 	- macro values must be less than 1024 characters
 	*/
 	macroNameRegex := regexp.MustCompile(`^[a-zA-Z0-9_-]+$`)
@@ -253,6 +253,7 @@ func LoadConfigFromReader(r io.Reader) (Config, error) {
 		}
 		switch macroName {
 		case "PORT":
+		case "MODEL_ID":
 			return Config{}, fmt.Errorf("macro name '%s' is reserved and cannot be used", macroName)
 		}
 	}
@@ -296,6 +297,11 @@ func LoadConfigFromReader(r io.Reader) (Config, error) {
 			nextPort++
 		}

+		if strings.Contains(modelConfig.Cmd, "${MODEL_ID}") || strings.Contains(modelConfig.CmdStop, "${MODEL_ID}") {
+			modelConfig.Cmd = strings.ReplaceAll(modelConfig.Cmd, "${MODEL_ID}", modelId)
+			modelConfig.CmdStop = strings.ReplaceAll(modelConfig.CmdStop, "${MODEL_ID}", modelId)
+		}
+
 		// make sure there are no unknown macros that have not been replaced
 		macroPattern := regexp.MustCompile(`\$\{([a-zA-Z0-9_-]+)\}`)
 		fieldMap := map[string]string{
@@ -440,3 +440,44 @@ models:
 	expectedCmd := "/user/llama.cpp/build/bin/llama-server --port 9990 --model /path/to/model.gguf -ngl 99"
 	assert.Equal(t, expectedCmd, cmdStr, "Final command does not match expected structure")
 }
+
+func TestConfig_MacroModelId(t *testing.T) {
+	content := `
+startPort: 9000
+macros:
+  "docker-llama": docker run --name ${MODEL_ID} -p ${PORT}:8080 docker_img
+  "docker-stop": docker stop ${MODEL_ID}
+
+models:
+  model1:
+    cmd: /path/to/server -p ${PORT} -hf ${MODEL_ID}
+
+  model2:
+    cmd: ${docker-llama}
+    cmdStop: ${docker-stop}
+
+  author/model:F16:
+    cmd: /path/to/server -p ${PORT} -hf ${MODEL_ID}
+    cmdStop: stop
+`
+
+	config, err := LoadConfigFromReader(strings.NewReader(content))
+	assert.NoError(t, err)
+	sanitizedCmd, err := SanitizeCommand(config.Models["model1"].Cmd)
+	assert.NoError(t, err)
+	assert.Equal(t, "/path/to/server -p 9001 -hf model1", strings.Join(sanitizedCmd, " "))
+
+	assert.Equal(t, "docker stop ${MODEL_ID}", config.Macros["docker-stop"])
+
+	sanitizedCmd2, err := SanitizeCommand(config.Models["model2"].Cmd)
+	assert.NoError(t, err)
+	assert.Equal(t, "docker run --name model2 -p 9002:8080 docker_img", strings.Join(sanitizedCmd2, " "))
+
+	sanitizedCmdStop, err := SanitizeCommand(config.Models["model2"].CmdStop)
+	assert.NoError(t, err)
+	assert.Equal(t, "docker stop model2", strings.Join(sanitizedCmdStop, " "))
+
+	sanitizedCmd3, err := SanitizeCommand(config.Models["author/model:F16"].Cmd)
+	assert.NoError(t, err)
+	assert.Equal(t, "/path/to/server -p 9000 -hf author/model:F16", strings.Join(sanitizedCmd3, " "))
+}
@@ -5,12 +5,20 @@ import (
 	"fmt"
 	"io"
 	"net/http"
+	"strings"
 	"time"

 	"github.com/gin-gonic/gin"
 	"github.com/tidwall/gjson"
 )

+type MetricsRecorder struct {
+	metricsMonitor *MetricsMonitor
+	realModelName  string
+	//	isStreaming    bool
+	startTime time.Time
+}
+
 // MetricsMiddleware sets up the MetricsResponseWriter for capturing upstream requests
 func MetricsMiddleware(pm *ProxyManager) gin.HandlerFunc {
 	return func(c *gin.Context) {
@@ -41,57 +49,60 @@ func MetricsMiddleware(pm *ProxyManager) gin.HandlerFunc {
 			metricsRecorder: &MetricsRecorder{
 				metricsMonitor: pm.metricsMonitor,
 				realModelName:  realModelName,
-				isStreaming:    gjson.GetBytes(bodyBytes, "stream").Bool(),
 				startTime:      time.Now(),
 			},
 		}
 		c.Writer = writer
 		c.Next()

-		rec := writer.metricsRecorder
-		rec.processBody(writer.body)
-	}
-}
-
-type MetricsRecorder struct {
-	metricsMonitor *MetricsMonitor
-	realModelName  string
-	isStreaming    bool
-	startTime      time.Time
-}
-
-// processBody handles response processing after request completes
-func (rec *MetricsRecorder) processBody(body []byte) {
-	if rec.isStreaming {
-		rec.processStreamingResponse(body)
-	} else {
-		rec.processNonStreamingResponse(body)
+		// check for streaming response
+		if strings.Contains(c.Writer.Header().Get("Content-Type"), "text/event-stream") {
+			writer.metricsRecorder.processStreamingResponse(writer.body)
+		} else {
+			writer.metricsRecorder.processNonStreamingResponse(writer.body)
+		}
 	}
 }

 func (rec *MetricsRecorder) parseAndRecordMetrics(jsonData gjson.Result) bool {
 	usage := jsonData.Get("usage")
-	if !usage.Exists() {
+	timings := jsonData.Get("timings")
+	if !usage.Exists() && !timings.Exists() {
 		return false
 	}

 	// default values
-	outputTokens := int(jsonData.Get("usage.completion_tokens").Int())
-	inputTokens := int(jsonData.Get("usage.prompt_tokens").Int())
+	cachedTokens := -1 // unknown or missing data
+	outputTokens := 0
+	inputTokens := 0
+
+	// timings data
 	tokensPerSecond := -1.0
 	promptPerSecond := -1.0
 	durationMs := int(time.Since(rec.startTime).Milliseconds())

+	if usage.Exists() {
+		outputTokens = int(jsonData.Get("usage.completion_tokens").Int())
+		inputTokens = int(jsonData.Get("usage.prompt_tokens").Int())
+	}
+
 	// use llama-server's timing data for tok/sec and duration as it is more accurate
-	if timings := jsonData.Get("timings"); timings.Exists() {
+	if timings.Exists() {
+		inputTokens = int(jsonData.Get("timings.prompt_n").Int())
+		outputTokens = int(jsonData.Get("timings.predicted_n").Int())
 		promptPerSecond = jsonData.Get("timings.prompt_per_second").Float()
 		tokensPerSecond = jsonData.Get("timings.predicted_per_second").Float()
 		durationMs = int(jsonData.Get("timings.prompt_ms").Float() + jsonData.Get("timings.predicted_ms").Float())
+
+		if cachedValue := jsonData.Get("timings.cache_n"); cachedValue.Exists() {
+			cachedTokens = int(cachedValue.Int())
+		}
 	}

 	rec.metricsMonitor.addMetrics(TokenMetrics{
 		Timestamp:       time.Now(),
 		Model:           rec.realModelName,
+		CachedTokens:    cachedTokens,
 		InputTokens:     inputTokens,
 		OutputTokens:    outputTokens,
 		PromptPerSecond: promptPerSecond,
@@ -13,6 +13,7 @@ type TokenMetrics struct {
 	ID              int       `json:"id"`
 	Timestamp       time.Time `json:"timestamp"`
 	Model           string    `json:"model"`
+	CachedTokens    int       `json:"cache_tokens"`
 	InputTokens     int       `json:"input_tokens"`
 	OutputTokens    int       `json:"output_tokens"`
 	PromptPerSecond float64   `json:"prompt_per_second"`
@@ -61,7 +62,6 @@ func (mp *MetricsMonitor) addMetrics(metric TokenMetrics) {
 	if len(mp.metrics) > mp.maxMetrics {
 		mp.metrics = mp.metrics[len(mp.metrics)-mp.maxMetrics:]
 	}
-
 	event.Emit(TokenMetricsEvent{Metrics: metric})
 }

@@ -5,6 +5,7 @@ import (
 	"errors"
 	"fmt"
 	"io"
+	"net"
 	"net/http"
 	"net/url"
 	"os/exec"
@@ -363,8 +364,18 @@ func (p *Process) stopCommand() {
 }

 func (p *Process) checkHealthEndpoint(healthURL string) error {
+
 	client := &http.Client{
-		Timeout: 500 * time.Millisecond,
+		// wait a short time for a tcp connection to be established
+		Transport: &http.Transport{
+			DialContext: (&net.Dialer{
+				Timeout: 500 * time.Millisecond,
+			}).DialContext,
+		},
+
+		// give a long time to respond to the health check endpoint
+		// after the connection is established. See issue: 276
+		Timeout: 5000 * time.Millisecond,
 	}

 	req, err := http.NewRequest("GET", healthURL, nil)
@@ -447,6 +458,10 @@ func (p *Process) ProxyRequest(w http.ResponseWriter, r *http.Request) {
 			w.Header().Add(k, v)
 		}
 	}
+	// prevent nginx from buffering streaming responses (e.g., SSE)
+	if strings.Contains(strings.ToLower(resp.Header.Get("Content-Type")), "text/event-stream") {
+		w.Header().Set("X-Accel-Buffering", "no")
+	}
 	w.WriteHeader(resp.StatusCode)

 	// faster than io.Copy when streaming
@@ -60,10 +60,20 @@ func (pg *ProcessGroup) ProxyRequest(modelID string, writer http.ResponseWriter,
 	if pg.swap {
 		pg.Lock()
 		if pg.lastUsedProcess != modelID {
+
+			// is there something already running?
 			if pg.lastUsedProcess != "" {
 				pg.processes[pg.lastUsedProcess].Stop()
 			}
+
+			// wait for the request to the new model to be fully handled
+			// and prevent race conditions see issue #277
+			pg.processes[modelID].ProxyRequest(writer, request)
 			pg.lastUsedProcess = modelID
+
+			// short circuit and exit
+			pg.Unlock()
+			return nil
 		}
 		pg.Unlock()
 	}
@@ -4,6 +4,7 @@ import (
 	"bytes"
 	"net/http"
 	"net/http/httptest"
+	"sync"
 	"testing"

 	"github.com/stretchr/testify/assert"
@@ -44,32 +45,49 @@ func TestProcessGroup_HasMember(t *testing.T) {
 	assert.False(t, pg.HasMember("model3"))
 }

-func TestProcessGroup_ProxyRequestSwapIsTrue(t *testing.T) {
+// TestProcessGroup_ProxyRequestSwapIsTrueParallel tests that when swap is true
+// and multiple requests are made in parallel, only one process is running at a time.
+func TestProcessGroup_ProxyRequestSwapIsTrueParallel(t *testing.T) {
+	var processGroupTestConfig = AddDefaultGroupToConfig(Config{
+		HealthCheckTimeout: 15,
+		Models: map[string]ModelConfig{
+			// use the same listening so if a model is already running, it will fail
+			// this is a way to test that swap isolation is working
+			// properly when there are parallel requests made at the
+			// same time.
+			"model1": getTestSimpleResponderConfigPort("model1", 9832),
+			"model2": getTestSimpleResponderConfigPort("model2", 9832),
+			"model3": getTestSimpleResponderConfigPort("model3", 9832),
+			"model4": getTestSimpleResponderConfigPort("model4", 9832),
+			"model5": getTestSimpleResponderConfigPort("model5", 9832),
+		},
+		Groups: map[string]GroupConfig{
+			"G1": {
+				Swap:    true,
+				Members: []string{"model1", "model2", "model3", "model4", "model5"},
+			},
+		},
+	})
+
 	pg := NewProcessGroup("G1", processGroupTestConfig, testLogger, testLogger)
 	defer pg.StopProcesses(StopWaitForInflightRequest)

-	tests := []string{"model1", "model2"}
+	tests := []string{"model1", "model2", "model3", "model4", "model5"}

+	var wg sync.WaitGroup
+
+	wg.Add(len(tests))
 	for _, modelName := range tests {
-		t.Run(modelName, func(t *testing.T) {
-			reqBody := `{"x", "y"}`
-			req := httptest.NewRequest("POST", "/v1/chat/completions", bytes.NewBufferString(reqBody))
+		go func(modelName string) {
+			defer wg.Done()
+			req := httptest.NewRequest("POST", "/v1/chat/completions", nil)
 			w := httptest.NewRecorder()
-
 			assert.NoError(t, pg.ProxyRequest(modelName, w, req))
 			assert.Equal(t, http.StatusOK, w.Code)
 			assert.Contains(t, w.Body.String(), modelName)
-
-			// make sure only one process is in the running state
-			count := 0
-			for _, process := range pg.processes {
-				if process.CurrentState() == StateReady {
-					count++
-				}
-			}
-			assert.Equal(t, 1, count)
-		})
+		}(modelName)
 	}
+	wg.Wait()
 }

 func TestProcessGroup_ProxyRequestSwapIsFalse(t *testing.T) {
@@ -191,11 +191,20 @@ func (pm *ProxyManager) setupGinEngine() {
 	// Support legacy /v1/completions api, see issue #12
 	pm.ginEngine.POST("/v1/completions", mm, pm.proxyOAIHandler)

-	// Support embeddings
+	// Support embeddings and reranking
 	pm.ginEngine.POST("/v1/embeddings", mm, pm.proxyOAIHandler)
+
+	// llama-server's /reranking endpoint + aliases
+	pm.ginEngine.POST("/reranking", mm, pm.proxyOAIHandler)
+	pm.ginEngine.POST("/rerank", mm, pm.proxyOAIHandler)
 	pm.ginEngine.POST("/v1/rerank", mm, pm.proxyOAIHandler)
 	pm.ginEngine.POST("/v1/reranking", mm, pm.proxyOAIHandler)
-	pm.ginEngine.POST("/rerank", mm, pm.proxyOAIHandler)
+
+	// llama-server's /infill endpoint for code infilling
+	pm.ginEngine.POST("/infill", mm, pm.proxyOAIHandler)
+
+	// llama-server's /completion endpoint
+	pm.ginEngine.POST("/completion", mm, pm.proxyOAIHandler)

 	// Support audio/speech endpoint
 	pm.ginEngine.POST("/v1/audio/speech", pm.proxyOAIHandler)
@@ -218,7 +227,7 @@ func (pm *ProxyManager) setupGinEngine() {
 	pm.ginEngine.GET("/upstream", func(c *gin.Context) {
 		c.Redirect(http.StatusFound, "/ui/models")
 	})
-	pm.ginEngine.Any("/upstream/:model_id/*upstreamPath", pm.proxyToUpstream)
+	pm.ginEngine.Any("/upstream/*upstreamPath", pm.proxyToUpstream)

 	pm.ginEngine.GET("/unload", pm.unloadAllModelsHandler)
 	pm.ginEngine.GET("/running", pm.listRunningProcessesHandler)
@@ -384,24 +393,52 @@ func (pm *ProxyManager) listModelsHandler(c *gin.Context) {
 }

 func (pm *ProxyManager) proxyToUpstream(c *gin.Context) {
-	requestedModel := c.Param("model_id")
+	upstreamPath := c.Param("upstreamPath")

-	if requestedModel == "" {
+	// split the upstream path by / and search for the model name
+	parts := strings.Split(strings.TrimSpace(upstreamPath), "/")
+	if len(parts) == 0 {
 		pm.sendErrorResponse(c, http.StatusBadRequest, "model id required in path")
 		return
 	}

-	processGroup, realModelName, err := pm.swapProcessGroup(requestedModel)
+	modelFound := false
+	searchModelName := ""
+	var modelName, remainingPath string
+	for i, part := range parts {
+		if parts[i] == "" {
+			continue
+		}
+
+		if searchModelName == "" {
+			searchModelName = part
+		} else {
+			searchModelName = searchModelName + "/" + parts[i]
+		}
+
+		if real, ok := pm.config.RealModelName(searchModelName); ok {
+			modelName = real
+			remainingPath = "/" + strings.Join(parts[i+1:], "/")
+			modelFound = true
+			break
+		}
+	}
+
+	if !modelFound {
+		pm.sendErrorResponse(c, http.StatusBadRequest, "model id required in path")
+		return
+	}
+
+	processGroup, realModelName, err := pm.swapProcessGroup(modelName)
 	if err != nil {
 		pm.sendErrorResponse(c, http.StatusInternalServerError, fmt.Sprintf("error swapping process group: %s", err.Error()))
 		return
 	}

 	// rewrite the path
-	c.Request.URL.Path = c.Param("upstreamPath")
+	c.Request.URL.Path = remainingPath
 	processGroup.ProxyRequest(realModelName, c.Writer, c.Request)
 }
-
 func (pm *ProxyManager) proxyOAIHandler(c *gin.Context) {
 	bodyBytes, err := io.ReadAll(c.Request.Body)
 	if err != nil {
@@ -100,6 +100,8 @@ func (pm *ProxyManager) apiSendEvents(c *gin.Context) {
 	c.Header("Cache-Control", "no-cache")
 	c.Header("Connection", "keep-alive")
 	c.Header("X-Content-Type-Options", "nosniff")
+	// prevent nginx from buffering SSE
+	c.Header("X-Accel-Buffering", "no")

 	sendBuffer := make(chan messageEnvelope, 25)
 	ctx, cancel := context.WithCancel(c.Request.Context())
@@ -28,6 +28,8 @@ func (pm *ProxyManager) streamLogsHandler(c *gin.Context) {
 	c.Header("Content-Type", "text/plain")
 	c.Header("Transfer-Encoding", "chunked")
 	c.Header("X-Content-Type-Options", "nosniff")
+	// prevent nginx from buffering streamed logs
+	c.Header("X-Accel-Buffering", "no")

 	logMonitorId := c.Param("logMonitorID")
 	logger, err := pm.getLogger(logMonitorId)
@@ -2,6 +2,7 @@ package proxy

 import (
 	"bytes"
+	"context"
 	"encoding/json"
 	"fmt"
 	"math/rand"
@@ -42,7 +43,6 @@ func TestProxyManager_SwapProcessCorrectly(t *testing.T) {
 		assert.Contains(t, w.Body.String(), modelName)
 	}
 }
-
 func TestProxyManager_SwapMultiProcess(t *testing.T) {
 	config := AddDefaultGroupToConfig(Config{
 		HealthCheckTimeout: 15,
@@ -834,6 +834,28 @@ func TestProxyManager_HealthEndpoint(t *testing.T) {
 	assert.Equal(t, "OK", rec.Body.String())
 }

+// Ensure the custom llama-server /completion endpoint proxies correctly
+func TestProxyManager_CompletionEndpoint(t *testing.T) {
+	config := AddDefaultGroupToConfig(Config{
+		HealthCheckTimeout: 15,
+		Models: map[string]ModelConfig{
+			"model1": getTestSimpleResponderConfig("model1"),
+		},
+		LogLevel: "error",
+	})
+
+	proxy := New(config)
+	defer proxy.StopProcesses(StopWaitForInflightRequest)
+
+	reqBody := `{"model":"model1"}`
+	req := httptest.NewRequest("POST", "/completion", bytes.NewBufferString(reqBody))
+	w := httptest.NewRecorder()
+
+	proxy.ServeHTTP(w, req)
+	assert.Equal(t, http.StatusOK, w.Code)
+	assert.Contains(t, w.Body.String(), "model1")
+}
+
 func TestProxyManager_StartupHooks(t *testing.T) {

 	// using real YAML as the configuration has gotten more complex
@@ -892,3 +914,67 @@ models:
 	assert.Equal(t, StateReady, proxy.processGroups["preloadTestGroup"].processes["model1"].CurrentState())
 	assert.Equal(t, StateReady, proxy.processGroups["preloadTestGroup"].processes["model2"].CurrentState())
 }
+
+func TestProxyManager_StreamingEndpointsReturnNoBufferingHeader(t *testing.T) {
+	config := AddDefaultGroupToConfig(Config{
+		HealthCheckTimeout: 15,
+		Models: map[string]ModelConfig{
+			"model1": getTestSimpleResponderConfig("model1"),
+		},
+		LogLevel: "error",
+	})
+
+	proxy := New(config)
+	defer proxy.StopProcesses(StopWaitForInflightRequest)
+
+	endpoints := []string{
+		"/api/events",
+		"/logs/stream",
+		"/logs/stream/proxy",
+		"/logs/stream/upstream",
+	}
+
+	for _, endpoint := range endpoints {
+		t.Run(endpoint, func(t *testing.T) {
+			ctx, cancel := context.WithCancel(context.Background())
+			defer cancel()
+
+			req := httptest.NewRequest("GET", endpoint, nil)
+			req = req.WithContext(ctx)
+			rec := httptest.NewRecorder()
+
+			// We don't need the handler to fully complete, just to set the headers
+			// so run it in a goroutine and check the headers after a short delay
+			go proxy.ServeHTTP(rec, req)
+			time.Sleep(10 * time.Millisecond) // give it time to start and write headers
+
+			assert.Equal(t, http.StatusOK, rec.Code)
+			assert.Equal(t, "no", rec.Header().Get("X-Accel-Buffering"))
+		})
+	}
+}
+
+func TestProxyManager_ProxiedStreamingEndpointReturnsNoBufferingHeader(t *testing.T) {
+	config := AddDefaultGroupToConfig(Config{
+		HealthCheckTimeout: 15,
+		Models: map[string]ModelConfig{
+			"streaming-model": getTestSimpleResponderConfig("streaming-model"),
+		},
+		LogLevel: "error",
+	})
+
+	proxy := New(config)
+	defer proxy.StopProcesses(StopWaitForInflightRequest)
+
+	// Make a streaming request
+	reqBody := `{"model":"streaming-model"}`
+	// simple-responder will return text/event-stream when stream=true is in the query
+	req := httptest.NewRequest("POST", "/v1/chat/completions?stream=true", bytes.NewBufferString(reqBody))
+	rec := httptest.NewRecorder()
+
+	proxy.ServeHTTP(rec, req)
+
+	assert.Equal(t, http.StatusOK, rec.Code)
+	assert.Equal(t, "no", rec.Header().Get("X-Accel-Buffering"))
+	assert.Contains(t, rec.Header().Get("Content-Type"), "text/event-stream")
+}
@@ -8,17 +8,15 @@
      "name": "ui",
      "version": "0.0.0",
      "dependencies": {
-        "@tailwindcss/vite": "^4.1.8",
-        "@tanstack/react-query": "^5.80.6",
        "react": "^19.1.0",
        "react-dom": "^19.1.0",
        "react-icons": "^5.5.0",
        "react-resizable-panels": "^3.0.4",
-        "react-router-dom": "^7.6.2",
-        "tailwindcss": "^4.1.8"
+        "react-router-dom": "^7.6.2"
      },
      "devDependencies": {
        "@eslint/js": "^9.25.0",
+        "@tailwindcss/vite": "^4.1.8",
        "@types/react": "^19.1.2",
        "@types/react-dom": "^19.1.2",
        "@vitejs/plugin-react": "^4.4.1",
@@ -26,6 +24,7 @@
        "eslint-plugin-react-hooks": "^5.2.0",
        "eslint-plugin-react-refresh": "^0.4.19",
        "globals": "^16.0.0",
+        "tailwindcss": "^4.1.8",
        "typescript": "~5.8.3",
        "typescript-eslint": "^8.30.1",
        "vite": "^6.3.5"
@@ -35,6 +34,7 @@
      "version": "2.3.0",
      "resolved": "https://registry.npmjs.org/@ampproject/remapping/-/remapping-2.3.0.tgz",
      "integrity": "sha512-30iZtAPgz+LTIYoeivqYo853f02jBYSd5uGnGpkFV0M3xOt9aN73erkgYAmZU43x4VfqcnLxW9Kpg3R5LC4YYw==",
+      "dev": true,
      "license": "Apache-2.0",
      "dependencies": {
        "@jridgewell/gen-mapping": "^0.3.5",
@@ -333,6 +333,7 @@
      "cpu": [
        "ppc64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -349,6 +350,7 @@
      "cpu": [
        "arm"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -365,6 +367,7 @@
      "cpu": [
        "arm64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -381,6 +384,7 @@
      "cpu": [
        "x64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -397,6 +401,7 @@
      "cpu": [
        "arm64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -413,6 +418,7 @@
      "cpu": [
        "x64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -429,6 +435,7 @@
      "cpu": [
        "arm64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -445,6 +452,7 @@
      "cpu": [
        "x64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -461,6 +469,7 @@
      "cpu": [
        "arm"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -477,6 +486,7 @@
      "cpu": [
        "arm64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -493,6 +503,7 @@
      "cpu": [
        "ia32"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -509,6 +520,7 @@
      "cpu": [
        "loong64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -525,6 +537,7 @@
      "cpu": [
        "mips64el"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -541,6 +554,7 @@
      "cpu": [
        "ppc64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -557,6 +571,7 @@
      "cpu": [
        "riscv64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -573,6 +588,7 @@
      "cpu": [
        "s390x"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -589,6 +605,7 @@
      "cpu": [
        "x64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -605,6 +622,7 @@
      "cpu": [
        "arm64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -621,6 +639,7 @@
      "cpu": [
        "x64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -637,6 +656,7 @@
      "cpu": [
        "arm64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -653,6 +673,7 @@
      "cpu": [
        "x64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -669,6 +690,7 @@
      "cpu": [
        "x64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -685,6 +707,7 @@
      "cpu": [
        "arm64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -701,6 +724,7 @@
      "cpu": [
        "ia32"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -717,6 +741,7 @@
      "cpu": [
        "x64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -950,6 +975,7 @@
      "version": "4.0.1",
      "resolved": "https://registry.npmjs.org/@isaacs/fs-minipass/-/fs-minipass-4.0.1.tgz",
      "integrity": "sha512-wgm9Ehl2jpeqP3zw/7mo3kRHFp5MEDhqAdwy1fTGkHAwnkGOVsgpvQhL8B5n1qlb01jV3n/bI0ZfZp5lWA1k4w==",
+      "dev": true,
      "license": "ISC",
      "dependencies": {
        "minipass": "^7.0.4"
@@ -962,6 +988,7 @@
      "version": "0.3.8",
      "resolved": "https://registry.npmjs.org/@jridgewell/gen-mapping/-/gen-mapping-0.3.8.tgz",
      "integrity": "sha512-imAbBGkb+ebQyxKgzv5Hu2nmROxoDOXHh80evxdoXNOrvAnVx7zimzc1Oo5h9RlfV4vPXaE2iM5pOFbvOCClWA==",
+      "dev": true,
      "license": "MIT",
      "dependencies": {
        "@jridgewell/set-array": "^1.2.1",
@@ -976,6 +1003,7 @@
      "version": "3.1.2",
      "resolved": "https://registry.npmjs.org/@jridgewell/resolve-uri/-/resolve-uri-3.1.2.tgz",
      "integrity": "sha512-bRISgCIjP20/tbWSPWMEi54QVPRZExkuD9lJL+UIxUKtwVJA8wW1Trb1jMs1RFXo1CBTNZ/5hpC9QvmKWdopKw==",
+      "dev": true,
      "license": "MIT",
      "engines": {
        "node": ">=6.0.0"
@@ -985,6 +1013,7 @@
      "version": "1.2.1",
      "resolved": "https://registry.npmjs.org/@jridgewell/set-array/-/set-array-1.2.1.tgz",
      "integrity": "sha512-R8gLRTZeyp03ymzP/6Lil/28tGeGEzhx1q2k703KGWRAI1VdvPIXdG70VJc2pAMw3NA6JKL5hhFu1sJX0Mnn/A==",
+      "dev": true,
      "license": "MIT",
      "engines": {
        "node": ">=6.0.0"
@@ -994,12 +1023,14 @@
      "version": "1.5.0",
      "resolved": "https://registry.npmjs.org/@jridgewell/sourcemap-codec/-/sourcemap-codec-1.5.0.tgz",
      "integrity": "sha512-gv3ZRaISU3fjPAgNsriBRqGWQL6quFx04YMPW/zD8XMLsU32mhCCbfbO6KZFLjvYpCZ8zyDEgqsgf+PwPaM7GQ==",
+      "dev": true,
      "license": "MIT"
    },
    "node_modules/@jridgewell/trace-mapping": {
      "version": "0.3.25",
      "resolved": "https://registry.npmjs.org/@jridgewell/trace-mapping/-/trace-mapping-0.3.25.tgz",
      "integrity": "sha512-vNk6aEwybGtawWmy/PzwnGDOjCkLWSD2wqvjGGAgOAwCGWySYXfYoxt00IJkTF+8Lb57DwOb3Aa0o9CApepiYQ==",
+      "dev": true,
      "license": "MIT",
      "dependencies": {
        "@jridgewell/resolve-uri": "^3.1.0",
@@ -1058,6 +1089,7 @@
      "cpu": [
        "arm"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1071,6 +1103,7 @@
      "cpu": [
        "arm64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1084,6 +1117,7 @@
      "cpu": [
        "arm64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1097,6 +1131,7 @@
      "cpu": [
        "x64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1110,6 +1145,7 @@
      "cpu": [
        "arm64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1123,6 +1159,7 @@
      "cpu": [
        "x64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1136,6 +1173,7 @@
      "cpu": [
        "arm"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1149,6 +1187,7 @@
      "cpu": [
        "arm"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1162,6 +1201,7 @@
      "cpu": [
        "arm64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1175,6 +1215,7 @@
      "cpu": [
        "arm64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1188,6 +1229,7 @@
      "cpu": [
        "loong64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1201,6 +1243,7 @@
      "cpu": [
        "ppc64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1214,6 +1257,7 @@
      "cpu": [
        "riscv64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1227,6 +1271,7 @@
      "cpu": [
        "riscv64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1240,6 +1285,7 @@
      "cpu": [
        "s390x"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1253,6 +1299,7 @@
      "cpu": [
        "x64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1266,6 +1313,7 @@
      "cpu": [
        "x64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1279,6 +1327,7 @@
      "cpu": [
        "arm64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1292,6 +1341,7 @@
      "cpu": [
        "ia32"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1305,6 +1355,7 @@
      "cpu": [
        "x64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1315,6 +1366,7 @@
      "version": "4.1.8",
      "resolved": "https://registry.npmjs.org/@tailwindcss/node/-/node-4.1.8.tgz",
      "integrity": "sha512-OWwBsbC9BFAJelmnNcrKuf+bka2ZxCE2A4Ft53Tkg4uoiE67r/PMEYwCsourC26E+kmxfwE0hVzMdxqeW+xu7Q==",
+      "dev": true,
      "license": "MIT",
      "dependencies": {
        "@ampproject/remapping": "^2.3.0",
@@ -1330,6 +1382,7 @@
      "version": "4.1.8",
      "resolved": "https://registry.npmjs.org/@tailwindcss/oxide/-/oxide-4.1.8.tgz",
      "integrity": "sha512-d7qvv9PsM5N3VNKhwVUhpK6r4h9wtLkJ6lz9ZY9aeZgrUWk1Z8VPyqyDT9MZlem7GTGseRQHkeB1j3tC7W1P+A==",
+      "dev": true,
      "hasInstallScript": true,
      "license": "MIT",
      "dependencies": {
@@ -1361,6 +1414,7 @@
      "cpu": [
        "arm64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1377,6 +1431,7 @@
      "cpu": [
        "arm64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1393,6 +1448,7 @@
      "cpu": [
        "x64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1409,6 +1465,7 @@
      "cpu": [
        "x64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1425,6 +1482,7 @@
      "cpu": [
        "arm"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1441,6 +1499,7 @@
      "cpu": [
        "arm64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1457,6 +1516,7 @@
      "cpu": [
        "arm64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1473,6 +1533,7 @@
      "cpu": [
        "x64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1489,6 +1550,7 @@
      "cpu": [
        "x64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1513,6 +1575,7 @@
      "cpu": [
        "wasm32"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "dependencies": {
@@ -1534,6 +1597,7 @@
      "cpu": [
        "arm64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1550,6 +1614,7 @@
      "cpu": [
        "x64"
      ],
+      "dev": true,
      "license": "MIT",
      "optional": true,
      "os": [
@@ -1563,6 +1628,7 @@
      "version": "4.1.8",
      "resolved": "https://registry.npmjs.org/@tailwindcss/vite/-/vite-4.1.8.tgz",
      "integrity": "sha512-CQ+I8yxNV5/6uGaJjiuymgw0kEQiNKRinYbZXPdx1fk5WgiyReG0VaUx/Xq6aVNSUNJFzxm6o8FNKS5aMaim5A==",
+      "dev": true,
      "license": "MIT",
      "dependencies": {
        "@tailwindcss/node": "4.1.8",
@@ -1573,32 +1639,6 @@
        "vite": "^5.2.0 || ^6"
      }
    },
-    "node_modules/@tanstack/query-core": {
-      "version": "5.80.6",
-      "resolved": "https://registry.npmjs.org/@tanstack/query-core/-/query-core-5.80.6.tgz",
-      "integrity": "sha512-nl7YxT/TAU+VTf+e2zTkObGTyY8YZBMnbgeA1ee66lIVqzKlYursAII6z5t0e6rXgwUMJSV4dshBTNacNpZHbQ==",
-      "license": "MIT",
-      "funding": {
-        "type": "github",
-        "url": "https://github.com/sponsors/tannerlinsley"
-      }
-    },
-    "node_modules/@tanstack/react-query": {
-      "version": "5.80.6",
-      "resolved": "https://registry.npmjs.org/@tanstack/react-query/-/react-query-5.80.6.tgz",
-      "integrity": "sha512-izX+5CnkpON3NQGcEm3/d7LfFQNo9ZpFtX2QsINgCYK9LT2VCIdi8D3bMaMSNhrAJCznRoAkFic76uvLroALBw==",
-      "license": "MIT",
-      "dependencies": {
-        "@tanstack/query-core": "5.80.6"
-      },
-      "funding": {
-        "type": "github",
-        "url": "https://github.com/sponsors/tannerlinsley"
-      },
-      "peerDependencies": {
-        "react": "^18 || ^19"
-      }
-    },
    "node_modules/@types/babel__core": {
      "version": "7.20.5",
      "resolved": "https://registry.npmjs.org/@types/babel__core/-/babel__core-7.20.5.tgz",
@@ -2148,6 +2188,7 @@
      "version": "3.0.0",
      "resolved": "https://registry.npmjs.org/chownr/-/chownr-3.0.0.tgz",
      "integrity": "sha512-+IxzY9BZOQd/XuYPRmrvEVjF/nqj5kgT4kEq7VofrDoM1MxoRjEWkrCC3EtLi59TVawxTAn+orJwFQcrqEN1+g==",
+      "dev": true,
      "license": "BlueOak-1.0.0",
      "engines": {
        "node": ">=18"
@@ -2247,6 +2288,7 @@
      "version": "2.0.4",
      "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.0.4.tgz",
      "integrity": "sha512-3UDv+G9CsCKO1WKMGw9fwq/SWJYbI0c5Y7LU1AXYoDdbhE2AHQ6N6Nb34sG8Fj7T5APy8qXDCKuuIHd1BR0tVA==",
+      "dev": true,
      "license": "Apache-2.0",
      "engines": {
        "node": ">=8"
@@ -2263,6 +2305,7 @@
      "version": "5.18.1",
      "resolved": "https://registry.npmjs.org/enhanced-resolve/-/enhanced-resolve-5.18.1.tgz",
      "integrity": "sha512-ZSW3ma5GkcQBIpwZTSRAI8N71Uuwgs93IezB7mf7R60tC8ZbJideoDNKjHn2O9KIlx6rkGTTEk1xUCK2E1Y2Yg==",
+      "dev": true,
      "license": "MIT",
      "dependencies": {
        "graceful-fs": "^4.2.4",
@@ -2276,6 +2319,7 @@
      "version": "0.25.5",
      "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.25.5.tgz",
      "integrity": "sha512-P8OtKZRv/5J5hhz0cUAdu/cLuPIKXpQl1R9pZtvmHWQvrAUVd0UNIPT4IB4W3rNOqVO0rlqHmCIbSwxh/c9yUQ==",
+      "dev": true,
      "hasInstallScript": true,
      "license": "MIT",
      "bin": {
@@ -2642,6 +2686,7 @@
      "version": "2.3.3",
      "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz",
      "integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==",
+      "dev": true,
      "hasInstallScript": true,
      "license": "MIT",
      "optional": true,
@@ -2692,6 +2737,7 @@
      "version": "4.2.11",
      "resolved": "https://registry.npmjs.org/graceful-fs/-/graceful-fs-4.2.11.tgz",
      "integrity": "sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ==",
+      "dev": true,
      "license": "ISC"
    },
    "node_modules/graphemer": {
@@ -2792,6 +2838,7 @@
      "version": "2.4.2",
      "resolved": "https://registry.npmjs.org/jiti/-/jiti-2.4.2.tgz",
      "integrity": "sha512-rg9zJN+G4n2nfJl5MW3BMygZX56zKPNVEYYqq7adpmMh4Jn2QNEwhvQlFy6jPVdcod7txZtKHWnyZiA3a0zP7A==",
+      "dev": true,
      "license": "MIT",
      "bin": {
        "jiti": "lib/jiti-cli.mjs"
@@ -2892,6 +2939,7 @@
      "version": "1.30.1",
      "resolved": "https://registry.npmjs.org/lightningcss/-/lightningcss-1.30.1.tgz",
      "integrity": "sha512-xi6IyHML+c9+Q3W0S4fCQJOym42pyurFiJUHEcEyHS0CeKzia4yZDEsLlqOFykxOdHpNy0NmvVO31vcSqAxJCg==",
+      "dev": true,
      "license": "MPL-2.0",
      "dependencies": {
        "detect-libc": "^2.0.3"
@@ -2923,6 +2971,7 @@
      "cpu": [
        "arm64"
      ],
+      "dev": true,
      "license": "MPL-2.0",
      "optional": true,
      "os": [
@@ -2943,6 +2992,7 @@
      "cpu": [
        "x64"
      ],
+      "dev": true,
      "license": "MPL-2.0",
      "optional": true,
      "os": [
@@ -2963,6 +3013,7 @@
      "cpu": [
        "x64"
      ],
+      "dev": true,
      "license": "MPL-2.0",
      "optional": true,
      "os": [
@@ -2983,6 +3034,7 @@
      "cpu": [
        "arm"
      ],
+      "dev": true,
      "license": "MPL-2.0",
      "optional": true,
      "os": [
@@ -3003,6 +3055,7 @@
      "cpu": [
        "arm64"
      ],
+      "dev": true,
      "license": "MPL-2.0",
      "optional": true,
      "os": [
@@ -3023,6 +3076,7 @@
      "cpu": [
        "arm64"
      ],
+      "dev": true,
      "license": "MPL-2.0",
      "optional": true,
      "os": [
@@ -3043,6 +3097,7 @@
      "cpu": [
        "x64"
      ],
+      "dev": true,
      "license": "MPL-2.0",
      "optional": true,
      "os": [
@@ -3063,6 +3118,7 @@
      "cpu": [
        "x64"
      ],
+      "dev": true,
      "license": "MPL-2.0",
      "optional": true,
      "os": [
@@ -3083,6 +3139,7 @@
      "cpu": [
        "arm64"
      ],
+      "dev": true,
      "license": "MPL-2.0",
      "optional": true,
      "os": [
@@ -3103,6 +3160,7 @@
      "cpu": [
        "x64"
      ],
+      "dev": true,
      "license": "MPL-2.0",
      "optional": true,
      "os": [
@@ -3153,6 +3211,7 @@
      "version": "0.30.17",
      "resolved": "https://registry.npmjs.org/magic-string/-/magic-string-0.30.17.tgz",
      "integrity": "sha512-sNPKHvyjVf7gyjwS4xGTaW/mCnF8wnjtifKBEhxfZ7E/S8tQ0rssrwGNn6q8JH/ohItJfSQp9mBtQYuTlH5QnA==",
+      "dev": true,
      "license": "MIT",
      "dependencies": {
        "@jridgewell/sourcemap-codec": "^1.5.0"
@@ -3199,6 +3258,7 @@
      "version": "7.1.2",
      "resolved": "https://registry.npmjs.org/minipass/-/minipass-7.1.2.tgz",
      "integrity": "sha512-qOOzS1cBTWYF4BH8fVePDBOO9iptMnGUEZwNc/cMWnTV2nVLZ7VoNWEPHkYczZA0pdoA7dl6e7FL659nX9S2aw==",
+      "dev": true,
      "license": "ISC",
      "engines": {
        "node": ">=16 || 14 >=14.17"
@@ -3208,6 +3268,7 @@
      "version": "3.0.2",
      "resolved": "https://registry.npmjs.org/minizlib/-/minizlib-3.0.2.tgz",
      "integrity": "sha512-oG62iEk+CYt5Xj2YqI5Xi9xWUeZhDI8jjQmC5oThVH5JGCTgIjr7ciJDzC7MBzYd//WvR1OTmP5Q38Q8ShQtVA==",
+      "dev": true,
      "license": "MIT",
      "dependencies": {
        "minipass": "^7.1.2"
@@ -3220,6 +3281,7 @@
      "version": "3.0.1",
      "resolved": "https://registry.npmjs.org/mkdirp/-/mkdirp-3.0.1.tgz",
      "integrity": "sha512-+NsyUUAZDmo6YVHzL/stxSu3t9YS1iljliy3BSDrXJ/dkn1KYdmtZODGGjLcc9XLgVVpH4KshHB8XmZgMhaBXg==",
+      "dev": true,
      "license": "MIT",
      "bin": {
        "mkdirp": "dist/cjs/src/bin.js"
@@ -3242,6 +3304,7 @@
      "version": "3.3.11",
      "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.11.tgz",
      "integrity": "sha512-N8SpfPUnUp1bK+PMYW8qSWdl9U+wwNWI4QKxOYDy9JAro3WMX7p2OeVRF9v+347pnakNevPmiHhNmZ2HbFA76w==",
+      "dev": true,
      "funding": [
        {
          "type": "github",
@@ -3357,6 +3420,7 @@
      "version": "1.1.1",
      "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz",
      "integrity": "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==",
+      "dev": true,
      "license": "ISC"
    },
    "node_modules/picomatch": {
@@ -3376,6 +3440,7 @@
      "version": "8.5.4",
      "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.4.tgz",
      "integrity": "sha512-QSa9EBe+uwlGTFmHsPKokv3B/oEMQZxfqW0QqNCyhpa6mB1afzulwn8hihglqAb2pOw+BJgNlmXQ8la2VeHB7w==",
+      "dev": true,
      "funding": [
        {
          "type": "opencollective",
@@ -3554,6 +3619,7 @@
      "version": "4.42.0",
      "resolved": "https://registry.npmjs.org/rollup/-/rollup-4.42.0.tgz",
      "integrity": "sha512-LW+Vse3BJPyGJGAJt1j8pWDKPd73QM8cRXYK1IxOBgL2AGLu7Xd2YOW0M2sLUBCkF5MshXXtMApyEAEzMVMsnw==",
+      "dev": true,
      "license": "MIT",
      "dependencies": {
        "@types/estree": "1.0.7"
@@ -3593,6 +3659,7 @@
      "version": "1.0.7",
      "resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.7.tgz",
      "integrity": "sha512-w28IoSUCJpidD/TGviZwwMJckNESJZXFu7NBZ5YJ4mEUnNraUn9Pm8HSZm/jDF1pDWYKspWE7oVphigUPRakIQ==",
+      "dev": true,
      "license": "MIT"
    },
    "node_modules/run-parallel": {
@@ -3668,6 +3735,7 @@
      "version": "1.2.1",
      "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz",
      "integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==",
+      "dev": true,
      "license": "BSD-3-Clause",
      "engines": {
        "node": ">=0.10.0"
@@ -3703,12 +3771,14 @@
      "version": "4.1.8",
      "resolved": "https://registry.npmjs.org/tailwindcss/-/tailwindcss-4.1.8.tgz",
      "integrity": "sha512-kjeW8gjdxasbmFKpVGrGd5T4i40mV5J2Rasw48QARfYeQ8YS9x02ON9SFWax3Qf616rt4Cp3nVNIj6Hd1mP3og==",
+      "dev": true,
      "license": "MIT"
    },
    "node_modules/tapable": {
      "version": "2.2.2",
      "resolved": "https://registry.npmjs.org/tapable/-/tapable-2.2.2.tgz",
      "integrity": "sha512-Re10+NauLTMCudc7T5WLFLAwDhQ0JWdrMK+9B2M8zR5hRExKmsRDCBA7/aV/pNJFltmBFO5BAMlQFi/vq3nKOg==",
+      "dev": true,
      "license": "MIT",
      "engines": {
        "node": ">=6"
@@ -3718,6 +3788,7 @@
      "version": "7.4.3",
      "resolved": "https://registry.npmjs.org/tar/-/tar-7.4.3.tgz",
      "integrity": "sha512-5S7Va8hKfV7W5U6g3aYxXmlPoZVAwUMy9AOKyF2fVuZa2UD3qZjg578OrLRt8PcNN1PleVaL/5/yYATNL0ICUw==",
+      "dev": true,
      "license": "ISC",
      "dependencies": {
        "@isaacs/fs-minipass": "^4.0.0",
@@ -3735,6 +3806,7 @@
      "version": "5.0.0",
      "resolved": "https://registry.npmjs.org/yallist/-/yallist-5.0.0.tgz",
      "integrity": "sha512-YgvUTfwqyc7UXVMrB+SImsVYSmTS8X/tSrtdNZMImM+n7+QTriRXyXim0mBrTXNeqzVF0KWGgHPeiyViFFrNDw==",
+      "dev": true,
      "license": "BlueOak-1.0.0",
      "engines": {
        "node": ">=18"
@@ -3744,6 +3816,7 @@
      "version": "0.2.14",
      "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.14.tgz",
      "integrity": "sha512-tX5e7OM1HnYr2+a2C/4V0htOcSQcoSTH9KgJnVvNm5zm/cyEWKJ7j7YutsH9CxMdtOkkLFy2AHrMci9IM8IPZQ==",
+      "dev": true,
      "license": "MIT",
      "dependencies": {
        "fdir": "^6.4.4",
@@ -3760,6 +3833,7 @@
      "version": "6.4.5",
      "resolved": "https://registry.npmjs.org/fdir/-/fdir-6.4.5.tgz",
      "integrity": "sha512-4BG7puHpVsIYxZUbiUE3RqGloLaSSwzYie5jvasC4LWuBWzZawynvYouhjbQKw2JuIGYdm0DzIxl8iVidKlUEw==",
+      "dev": true,
      "license": "MIT",
      "peerDependencies": {
        "picomatch": "^3 || ^4"
@@ -3774,6 +3848,7 @@
      "version": "4.0.2",
      "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.2.tgz",
      "integrity": "sha512-M7BAV6Rlcy5u+m6oPhAPFgJTzAioX/6B0DxyvDlo9l8+T3nLKbrczg2WLUyzd45L8RqfUMyGPzekbMvX2Ldkwg==",
+      "dev": true,
      "license": "MIT",
      "engines": {
        "node": ">=12"
@@ -3903,6 +3978,7 @@
      "version": "6.3.5",
      "resolved": "https://registry.npmjs.org/vite/-/vite-6.3.5.tgz",
      "integrity": "sha512-cZn6NDFE7wdTpINgs++ZJ4N49W2vRp8LCKrn3Ob1kYNtOo21vfDoaV5GzBfLU4MovSAB8uNRm4jgzVQZ+mBzPQ==",
+      "dev": true,
      "license": "MIT",
      "dependencies": {
        "esbuild": "^0.25.0",
@@ -3977,6 +4053,7 @@
      "version": "6.4.5",
      "resolved": "https://registry.npmjs.org/fdir/-/fdir-6.4.5.tgz",
      "integrity": "sha512-4BG7puHpVsIYxZUbiUE3RqGloLaSSwzYie5jvasC4LWuBWzZawynvYouhjbQKw2JuIGYdm0DzIxl8iVidKlUEw==",
+      "dev": true,
      "license": "MIT",
      "peerDependencies": {
        "picomatch": "^3 || ^4"
@@ -3991,6 +4068,7 @@
      "version": "4.0.2",
      "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.2.tgz",
      "integrity": "sha512-M7BAV6Rlcy5u+m6oPhAPFgJTzAioX/6B0DxyvDlo9l8+T3nLKbrczg2WLUyzd45L8RqfUMyGPzekbMvX2Ldkwg==",
+      "dev": true,
      "license": "MIT",
      "engines": {
        "node": ">=12"
@@ -4,23 +4,21 @@
  "version": "0.0.0",
  "type": "module",
  "scripts": {
-    "dev": "vite",
+    "start": "vite",
    "build": "tsc -b && vite build --emptyOutDir",
    "lint": "eslint .",
    "preview": "vite preview"
  },
  "dependencies": {
-    "@tailwindcss/vite": "^4.1.8",
-    "@tanstack/react-query": "^5.80.6",
    "react": "^19.1.0",
    "react-dom": "^19.1.0",
    "react-icons": "^5.5.0",
    "react-resizable-panels": "^3.0.4",
-    "react-router-dom": "^7.6.2",
-    "tailwindcss": "^4.1.8"
+    "react-router-dom": "^7.6.2"
  },
  "devDependencies": {
    "@eslint/js": "^9.25.0",
+    "@tailwindcss/vite": "^4.1.8",
    "@types/react": "^19.1.2",
    "@types/react-dom": "^19.1.2",
    "@vitejs/plugin-react": "^4.4.1",
@@ -28,6 +26,7 @@
    "eslint-plugin-react-hooks": "^5.2.0",
    "eslint-plugin-react-refresh": "^0.4.19",
    "globals": "^16.0.0",
+    "tailwindcss": "^4.1.8",
    "typescript": "~5.8.3",
    "typescript-eslint": "^8.30.1",
    "vite": "^6.3.5"
@@ -1,21 +1,14 @@
-import { useEffect, useCallback } from "react";
-import { BrowserRouter as Router, Routes, Route, Navigate, NavLink } from "react-router-dom";
-import { useTheme } from "./contexts/ThemeProvider";
+import { useEffect } from "react";
+import { Navigate, Route, BrowserRouter as Router, Routes } from "react-router-dom";
+import { Header } from "./components/Header";
 import { useAPI } from "./contexts/APIProvider";
+import { useTheme } from "./contexts/ThemeProvider";
+import ActivityPage from "./pages/Activity";
 import LogViewerPage from "./pages/LogViewer";
 import ModelPage from "./pages/Models";
-import ActivityPage from "./pages/Activity";
-import ConnectionStatusIcon from "./components/ConnectionStatus";
-import { RiSunFill, RiMoonFill } from "react-icons/ri";

 function App() {
-  const { isNarrow, toggleTheme, isDarkMode, appTitle, setAppTitle, setConnectionState } = useTheme();
-  const handleTitleChange = useCallback(
-    (newTitle: string) => {
-      setAppTitle(newTitle.replace(/\n/g, "").trim().substring(0, 64) || "llama-swap");
-    },
-    [setAppTitle]
-  );
+  const { setConnectionState } = useTheme();

  const { connectionStatus } = useAPI();

@@ -27,42 +20,7 @@ function App() {
  return (
    <Router basename="/ui/">
      <div className="flex flex-col h-screen">
-        <nav className="bg-surface border-b border-border p-2 h-[75px]">
-          <div className="flex items-center justify-between mx-auto px-4 h-full">
-            {!isNarrow && (
-              <h1
-                contentEditable
-                suppressContentEditableWarning
-                className="flex items-center p-0 outline-none hover:bg-gray-100 dark:hover:bg-gray-700 rounded px-1"
-                onBlur={(e) => handleTitleChange(e.currentTarget.textContent || "(set title)")}
-                onKeyDown={(e) => {
-                  if (e.key === "Enter") {
-                    e.preventDefault();
-                    handleTitleChange(e.currentTarget.textContent || "(set title)");
-                    e.currentTarget.blur();
-                  }
-                }}
-              >
-                {appTitle}
-              </h1>
-            )}
-            <div className="flex items-center space-x-4">
-              <NavLink to="/" className={({ isActive }) => (isActive ? "navlink active" : "navlink")}>
-                Logs
-              </NavLink>
-              <NavLink to="/models" className={({ isActive }) => (isActive ? "navlink active" : "navlink")}>
-                Models
-              </NavLink>
-              <NavLink to="/activity" className={({ isActive }) => (isActive ? "navlink active" : "navlink")}>
-                Activity
-              </NavLink>
-              <button className="" onClick={toggleTheme}>
-                {isDarkMode ? <RiMoonFill /> : <RiSunFill />}
-              </button>
-              <ConnectionStatusIcon />
-            </div>
-          </div>
-        </nav>
+        <Header />

        <main className="flex-1 overflow-auto p-4">
          <Routes>
@@ -7,9 +7,9 @@ const ConnectionStatusIcon = () => {
  const eventStatusColor = useMemo(() => {
    switch (connectionStatus) {
      case "connected":
-        return "bg-green-500";
+        return "bg-emerald-500";
      case "connecting":
-        return "bg-yellow-500";
+        return "bg-amber-500";
      case "disconnected":
      default:
        return "bg-red-500";
@@ -0,0 +1,56 @@
+import { useCallback } from "react";
+import { RiMoonFill, RiSunFill } from "react-icons/ri";
+import { NavLink, type NavLinkRenderProps } from "react-router-dom";
+import { useTheme } from "../contexts/ThemeProvider";
+import ConnectionStatusIcon from "./ConnectionStatus";
+
+export function Header() {
+  const { screenWidth, toggleTheme, isDarkMode, appTitle, setAppTitle } = useTheme();
+  const handleTitleChange = useCallback(
+    (newTitle: string) => {
+      setAppTitle(newTitle.replace(/\n/g, "").trim().substring(0, 64) || "llama-swap");
+    },
+    [setAppTitle]
+  );
+
+  const navLinkClass = ({ isActive }: NavLinkRenderProps) =>
+    `text-gray-600 hover:text-black dark:text-gray-300 dark:hover:text-gray-100 p-1 ${isActive ? "font-semibold" : ""}`;
+
+  return (
+    <header className="flex items-center justify-between bg-surface border-b border-border p-2 px-4 h-[75px]">
+      {screenWidth !== "xs" && screenWidth !== "sm" && (
+        <h1
+          contentEditable
+          suppressContentEditableWarning
+          className="p-0 outline-none hover:bg-gray-100 dark:hover:bg-gray-700 rounded"
+          onBlur={(e) => handleTitleChange(e.currentTarget.textContent || "(set title)")}
+          onKeyDown={(e) => {
+            if (e.key === "Enter") {
+              e.preventDefault();
+              handleTitleChange(e.currentTarget.textContent || "(set title)");
+              e.currentTarget.blur();
+            }
+          }}
+        >
+          {appTitle}
+        </h1>
+      )}
+
+      <menu className="flex items-center gap-4">
+        <NavLink to="/" className={navLinkClass} type="button">
+          Logs
+        </NavLink>
+        <NavLink to="/models" className={navLinkClass} type="button">
+          Models
+        </NavLink>
+        <NavLink to="/activity" className={navLinkClass} type="button">
+          Activity
+        </NavLink>
+        <button className="" onClick={toggleTheme}>
+          {isDarkMode ? <RiMoonFill /> : <RiSunFill />}
+        </button>
+        <ConnectionStatusIcon />
+      </menu>
+    </header>
+  );
+}
@@ -1,4 +1,4 @@
-import { useRef, createContext, useState, useContext, useEffect, useCallback, useMemo, type ReactNode } from "react";
+import { createContext, useState, useContext, useEffect, useCallback, useMemo, type ReactNode } from "react";
 import type { ConnectionState } from "../lib/types";

 type ModelStatus = "ready" | "starting" | "stopping" | "stopped" | "shutdown" | "unknown";
@@ -28,6 +28,7 @@ interface Metrics {
  id: number;
  timestamp: string;
  model: string;
+  cache_tokens: number;
  input_tokens: number;
  output_tokens: number;
  prompt_per_second: number;
@@ -50,12 +51,14 @@ type APIProviderProps = {
  autoStartAPIEvents?: boolean;
 };

+let apiEventSource: EventSource | null = null;
+
 export function APIProvider({ children, autoStartAPIEvents = true }: APIProviderProps) {
  const [proxyLogs, setProxyLogs] = useState("");
  const [upstreamLogs, setUpstreamLogs] = useState("");
  const [metrics, setMetrics] = useState<Metrics[]>([]);
  const [connectionStatus, setConnectionState] = useState<ConnectionState>("disconnected");
-  const apiEventSource = useRef<EventSource | null>(null);
+  //const apiEventSource = useRef<EventSource | null>(null);

  const [models, setModels] = useState<Model[]>([]);

@@ -68,8 +71,8 @@ export function APIProvider({ children, autoStartAPIEvents = true }: APIProvider

  const enableAPIEvents = useCallback((enabled: boolean) => {
    if (!enabled) {
-      apiEventSource.current?.close();
-      apiEventSource.current = null;
+      apiEventSource?.close();
+      apiEventSource = null;
      setMetrics([]);
      return;
    }
@@ -78,22 +81,22 @@ export function APIProvider({ children, autoStartAPIEvents = true }: APIProvider
    const initialDelay = 1000; // 1 second

    const connect = () => {
-      apiEventSource.current = null;
-      const eventSource = new EventSource("/api/events");
+      apiEventSource?.close();
+      apiEventSource = new EventSource("/api/events");
+
      setConnectionState("connecting");

-      eventSource.onopen = () => {
+      apiEventSource.onopen = () => {
        // clear everything out on connect to keep things in sync
        setProxyLogs("");
        setUpstreamLogs("");
        setMetrics([]); // clear metrics on reconnect
        setModels([]); // clear models on reconnect
-        apiEventSource.current = eventSource;
        retryCount = 0;
        setConnectionState("connected");
      };

-      eventSource.onmessage = (e: MessageEvent) => {
+      apiEventSource.onmessage = (e: MessageEvent) => {
        try {
          const message = JSON.parse(e.data) as APIEventEnvelope;
          switch (message.type) {
@@ -136,8 +139,8 @@ export function APIProvider({ children, autoStartAPIEvents = true }: APIProvider
        }
      };

-      eventSource.onerror = () => {
-        eventSource.close();
+      apiEventSource.onerror = () => {
+        apiEventSource?.close();
        retryCount++;
        const delay = Math.min(initialDelay * Math.pow(2, retryCount - 1), 5000);
        setConnectionState("disconnected");
@@ -93,6 +93,14 @@
    @apply px-4;
  }

+  /* Tables */
+  table th {
+    @apply p-2 font-semibold;
+  }
+  table td {
+    @apply p-2;
+  }
+
  /* Navigation Header */

  .navlink {
@@ -122,7 +130,7 @@

  /* Status Badges */
  .status {
-    @apply inline-block px-2 py-1 text-xs font-medium rounded-full;
+    @apply inline-block px-2 py-1 text-xs font-medium rounded-lg;
  }

  .status--ready {
@@ -140,7 +148,7 @@

  /* Buttons */
  .btn {
-    @apply bg-surface p-2 px-4 text-sm rounded-full border border-2 transition-colors duration-200 border-btn-border;
+    @apply bg-surface py-2 px-4 text-sm rounded-md border transition-colors duration-200 border-btn-border;
  }

  .btn:hover {
@@ -1,10 +1,6 @@
 import { useMemo } from "react";
 import { useAPI } from "../contexts/APIProvider";

-const formatTimestamp = (timestamp: string): string => {
-  return new Date(timestamp).toLocaleString();
-};
-
 const formatSpeed = (speed: number): string => {
  return speed < 0 ? "unknown" : speed.toFixed(2) + " t/s";
 };
@@ -13,6 +9,33 @@ const formatDuration = (ms: number): string => {
  return (ms / 1000).toFixed(2) + "s";
 };

+const formatRelativeTime = (timestamp: string): string => {
+  const now = new Date();
+  const date = new Date(timestamp);
+  const diffInSeconds = Math.floor((now.getTime() - date.getTime()) / 1000);
+
+  // Handle future dates by returning "just now"
+  if (diffInSeconds < 5) {
+    return "now";
+  }
+
+  if (diffInSeconds < 60) {
+    return `${diffInSeconds}s ago`;
+  }
+
+  const diffInMinutes = Math.floor(diffInSeconds / 60);
+  if (diffInMinutes < 60) {
+    return `${diffInMinutes}m ago`;
+  }
+
+  const diffInHours = Math.floor(diffInMinutes / 60);
+  if (diffInHours < 24) {
+    return `${diffInHours}h ago`;
+  }
+
+  return "a while ago";
+};
+
 const ActivityPage = () => {
  const { metrics } = useAPI();
  const sortedMetrics = useMemo(() => {
@@ -20,39 +43,46 @@ const ActivityPage = () => {
  }, [metrics]);

  return (
-    <div className="p-6">
-      <h1 className="text-2xl font-bold mb-4">Activity</h1>
+    <div className="p-2">
+      <h1 className="text-2xl font-bold">Activity</h1>

-      {metrics.length === 0 ? (
+      {metrics.length === 0 && (
        <div className="text-center py-8">
          <p className="text-gray-600">No metrics data available</p>
        </div>
-      ) : (
-        <div className="overflow-x-auto">
+      )}
+      {metrics.length > 0 && (
+        <div className="card overflow-auto">
          <table className="min-w-full divide-y">
-            <thead>
-              <tr>
-                <th className="px-4 py-3 text-left text-xs font-medium uppercase tracking-wider">Id</th>
-                <th className="px-6 py-3 text-left text-xs font-medium uppercase tracking-wider">Timestamp</th>
-                <th className="px-6 py-3 text-left text-xs font-medium uppercase tracking-wider">Model</th>
-                <th className="px-6 py-3 text-left text-xs font-medium uppercase tracking-wider">Input Tokens</th>
-                <th className="px-6 py-3 text-left text-xs font-medium uppercase tracking-wider">Output Tokens</th>
-                <th className="px-6 py-3 text-left text-xs font-medium uppercase tracking-wider">Prompt Processing</th>
-                <th className="px-6 py-3 text-left text-xs font-medium uppercase tracking-wider">Generation Speed</th>
-                <th className="px-6 py-3 text-left text-xs font-medium uppercase tracking-wider">Duration</th>
+            <thead className="border-gray-200 dark:border-white/10">
+              <tr className="text-left text-xs uppercase tracking-wider">
+                <th className="px-6 py-3">ID</th>
+                <th className="px-6 py-3">Time</th>
+                <th className="px-6 py-3">Model</th>
+                <th className="px-6 py-3">
+                  Cached <Tooltip content="prompt tokens from cache" />
+                </th>
+                <th className="px-6 py-3">
+                  Prompt <Tooltip content="new prompt tokens processed" />
+                </th>
+                <th className="px-6 py-3">Generated</th>
+                <th className="px-6 py-3">Prompt Processing</th>
+                <th className="px-6 py-3">Generation Speed</th>
+                <th className="px-6 py-3">Duration</th>
              </tr>
            </thead>
            <tbody className="divide-y">
              {sortedMetrics.map((metric) => (
-                <tr key={`metric_${metric.id}`}>
-                  <td className="px-4 py-4 whitespace-nowrap text-sm">{metric.id + 1 /* un-zero index */}</td>
-                  <td className="px-6 py-4 whitespace-nowrap text-sm">{formatTimestamp(metric.timestamp)}</td>
-                  <td className="px-6 py-4 whitespace-nowrap text-sm">{metric.model}</td>
-                  <td className="px-6 py-4 whitespace-nowrap text-sm">{metric.input_tokens.toLocaleString()}</td>
-                  <td className="px-6 py-4 whitespace-nowrap text-sm">{metric.output_tokens.toLocaleString()}</td>
-                  <td className="px-6 py-4 whitespace-nowrap text-sm">{formatSpeed(metric.prompt_per_second)}</td>
-                  <td className="px-6 py-4 whitespace-nowrap text-sm">{formatSpeed(metric.tokens_per_second)}</td>
-                  <td className="px-6 py-4 whitespace-nowrap text-sm">{formatDuration(metric.duration_ms)}</td>
+                <tr key={metric.id} className="whitespace-nowrap text-sm border-gray-200 dark:border-white/10">
+                  <td className="px-4 py-4">{metric.id + 1 /* un-zero index */}</td>
+                  <td className="px-6 py-4">{formatRelativeTime(metric.timestamp)}</td>
+                  <td className="px-6 py-4">{metric.model}</td>
+                  <td className="px-6 py-4">{metric.cache_tokens > 0 ? metric.cache_tokens.toLocaleString() : "-"}</td>
+                  <td className="px-6 py-4">{metric.input_tokens.toLocaleString()}</td>
+                  <td className="px-6 py-4">{metric.output_tokens.toLocaleString()}</td>
+                  <td className="px-6 py-4">{formatSpeed(metric.prompt_per_second)}</td>
+                  <td className="px-6 py-4">{formatSpeed(metric.tokens_per_second)}</td>
+                  <td className="px-6 py-4">{formatDuration(metric.duration_ms)}</td>
                </tr>
              ))}
            </tbody>
@@ -63,4 +93,28 @@ const ActivityPage = () => {
  );
 };

+interface TooltipProps {
+  content: string;
+}
+
+const Tooltip: React.FC<TooltipProps> = ({ content }) => {
+  return (
+    <div className="relative group inline-block">
+      ⓘ
+      <div
+        className="absolute top-full left-1/2 transform -translate-x-1/2 mt-2
+                     px-3 py-2 bg-gray-900 text-white text-sm rounded-md
+                     opacity-0 group-hover:opacity-100 transition-opacity
+                     duration-200 pointer-events-none whitespace-nowrap z-50 normal-case"
+      >
+        {content}
+        <div
+          className="absolute bottom-full left-1/2 transform -translate-x-1/2
+                       border-4 border-transparent border-b-gray-900"
+        ></div>
+      </div>
+    </div>
+  );
+};
+
 export default ActivityPage;
@@ -14,8 +14,8 @@ import { useTheme } from "../contexts/ThemeProvider";

 const LogViewer = () => {
  const { proxyLogs, upstreamLogs } = useAPI();
-  const { isNarrow } = useTheme();
-  const direction = isNarrow ? "vertical" : "horizontal";
+  const { screenWidth } = useTheme();
+  const direction = screenWidth === "xs" || screenWidth === "sm" ? "vertical" : "horizontal";

  return (
    <PanelGroup direction={direction} className="gap-2" autoSaveId="logviewer-panel-group">
@@ -115,19 +115,19 @@ export const LogPanel = ({ id, title, logData }: LogPanelProps) => {
  }, [filteredLogs]);

  return (
-    <div className="bg-surface border border-border rounded-lg overflow-hidden flex flex-col h-full">
-      <div className="p-4 border-b border-border bg-secondary">
+    <div className="rounded-lg overflow-hidden flex flex-col bg-gray-950/5 dark:bg-white/10 h-full p-1">
+      <div className="p-4">
        <div className="flex items-center justify-between">
          <h3 className="m-0 text-lg p-0">{title}</h3>

          <div className="flex gap-2 items-center">
-            <button className="btn" onClick={toggleFontSize}>
+            <button className="btn border-0" onClick={toggleFontSize}>
              <RiFontSize />
            </button>
-            <button className="btn" onClick={toggleWrapText}>
+            <button className="btn border-0" onClick={toggleWrapText}>
              {wrapText ? <RiTextWrap /> : <RiAlignJustify />}
            </button>
-            <button className="btn" onClick={toggleFilter}>
+            <button className="btn border-0" onClick={toggleFilter}>
              {showFilter ? <RiMenuSearchFill /> : <RiMenuSearchLine />}
            </button>
          </div>
@@ -139,7 +139,7 @@ export const LogPanel = ({ id, title, logData }: LogPanelProps) => {
            <div className="flex gap-2 items-center w-full">
              <input
                type="text"
-                className="w-full text-sm border p-2 rounded"
+                className="w-full text-sm border border-gray-950/10 dark:border-white/5 p-2 rounded outline-none"
                placeholder="Filter logs..."
                value={filterRegex}
                onChange={(e) => setFilterRegex(e.target.value)}
@@ -151,7 +151,7 @@ export const LogPanel = ({ id, title, logData }: LogPanelProps) => {
          </div>
        )}
      </div>
-      <div className="bg-background font-mono text-sm flex-1 overflow-hidden">
+      <div className="rounded-lg bg-background font-mono text-sm flex-1 overflow-hidden">
        <pre ref={preTagRef} className={`${textWrapClass} ${fontSizeClass} h-full overflow-auto p-4`}>
          {filteredLogs}
        </pre>
@@ -69,19 +69,27 @@ function ModelsPanel() {
        <h2>Models</h2>
        <div className="flex justify-between">
          <div className="flex gap-2">
-            <button className="btn flex items-center gap-2" onClick={toggleIdorName} style={{ lineHeight: "1.2" }}>
-              <RiSwapBoxFill /> {showIdorName === "id" ? "ID" : "Name"}
+            <button
+              className="btn text-base flex items-center gap-2"
+              onClick={toggleIdorName}
+              style={{ lineHeight: "1.2" }}
+            >
+              <RiSwapBoxFill size="20" /> {showIdorName === "id" ? "ID" : "Name"}
            </button>

            <button
-              className="btn flex items-center gap-2"
+              className="btn text-base flex items-center gap-2"
              onClick={() => setShowUnlisted(!showUnlisted)}
              style={{ lineHeight: "1.2" }}
            >
-              {showUnlisted ? <RiEyeFill /> : <RiEyeOffFill />} unlisted
+              {showUnlisted ? <RiEyeFill size="20" /> : <RiEyeOffFill size="20" />} unlisted
            </button>
          </div>
-          <button className="btn flex items-center gap-2" onClick={handleUnloadAllModels} disabled={isUnloading}>
+          <button
+            className="btn text-base flex items-center gap-2"
+            onClick={handleUnloadAllModels}
+            disabled={isUnloading}
+          >
            <RiStopCircleLine size="24" /> {isUnloading ? "Unloading..." : "Unload"}
          </button>
        </div>
@@ -90,26 +98,27 @@ function ModelsPanel() {
      <div className="flex-1 overflow-y-auto">
        <table className="w-full">
          <thead className="sticky top-0 bg-card z-10">
-            <tr className="border-b border-primary bg-surface">
-              <th className="text-left p-2">{showIdorName === "id" ? "Model ID" : "Name"}</th>
-              <th className="text-left p-2"></th>
-              <th className="text-left p-2">State</th>
+            <tr className="text-left border-b border-gray-200 dark:border-white/10 bg-surface">
+              <th>{showIdorName === "id" ? "Model ID" : "Name"}</th>
+              <th></th>
+              <th>State</th>
            </tr>
          </thead>
          <tbody>
            {filteredModels.map((model) => (
-              <tr key={model.id} className="border-b hover:bg-secondary-hover border-border">
-                <td className={`p-2 ${model.unlisted ? "text-txtsecondary" : ""}`}>
-                  <a href={`/upstream/${model.id}/`} className={`underline`} target="_blank">
+              <tr key={model.id} className="border-b hover:bg-secondary-hover border-gray-200">
+                <td className={`${model.unlisted ? "text-txtsecondary" : ""}`}>
+                  <a href={`/upstream/${model.id}/`} className="font-semibold" target="_blank">
                    {showIdorName === "id" ? model.id : model.name !== "" ? model.name : model.id}
                  </a>
-                  {model.description !== "" && (
+
+                  {!!model.description && (
                    <p className={model.unlisted ? "text-opacity-70" : ""}>
                      <em>{model.description}</em>
                    </p>
                  )}
                </td>
-                <td className="p-2 w-[50px]">
+                <td className="w-12">
                  <button
                    className="btn btn--sm"
                    disabled={model.state !== "stopped"}
@@ -118,8 +127,8 @@ function ModelsPanel() {
                    Load
                  </button>
                </td>
-                <td className="p-2 w-[75px]">
-                  <span className={`status status--${model.state}`}>{model.state}</span>
+                <td className="w-20">
+                  <span className={`w-16 text-center status status--${model.state}`}>{model.state}</span>
                </td>
              </tr>
            ))}
@@ -146,24 +155,26 @@ function StatsPanel() {

  return (
    <div className="card">
-      <div className="rounded-lg overflow-hidden border border-gray-200">
+      <div className="rounded-lg overflow-hidden border border-gray-200 dark:border-white/10">
        <table className="w-full">
-          <tbody>
-            <tr>
-              <th className="p-2 font-medium border-b border-gray-200 text-right">Requests</th>
-              <th className="p-2 font-medium border-l border-b border-gray-200 text-right">Processed</th>
-              <th className="p-2 font-medium border-l border-b border-gray-200 text-right">Generated</th>
-              <th className="p-2 font-medium border-l border-b border-gray-200 text-right">Tokens/Sec</th>
+          <thead>
+            <tr className="border-b border-gray-200 dark:border-white/10 text-right">
+              <th>Requests</th>
+              <th className="border-l border-gray-200 dark:border-white/10">Processed</th>
+              <th className="border-l border-gray-200 dark:border-white/10">Generated</th>
+              <th className="border-l border-gray-200 dark:border-white/10">Tokens/Sec</th>
            </tr>
-            <tr>
-              <td className="p-2 text-right border-r border-gray-200">{totalRequests}</td>
-              <td className="p-2 text-right border-r border-gray-200">
+          </thead>
+          <tbody>
+            <tr className="text-right">
+              <td className="border-r border-gray-200 dark:border-white/10">{totalRequests}</td>
+              <td className="border-r border-gray-200 dark:border-white/10">
                {new Intl.NumberFormat().format(totalInputTokens)}
              </td>
-              <td className="p-2 text-right border-r border-gray-200">
+              <td className="border-r border-gray-200 dark:border-white/10">
                {new Intl.NumberFormat().format(totalOutputTokens)}
              </td>
-              <td className="p-2 text-right">{avgTokensPerSecond}</td>
+              <td>{avgTokensPerSecond}</td>
            </tr>
          </tbody>
        </table>
Author	SHA1	Message	Date
Oleg Shulyakov	fc3bb716df	UI styling / code improvements (#307 ) Clean up and improve UI styling * fix: UI - dependency cleanup * chore: UI - start script * refactor: UI - Extract Header * fix: UI - Header styling * fix: UI - LogViewer styling * fix: UI - Models styling * fix: UI - Activity styling * fix: UI - ConnectionStatus colors * review: UI - table border colors	2025-09-19 10:47:17 -07:00
Benson Wong	c36986fef6	upstream handler support for model names with forward slash (#298 ) The upstream handler would break on model IDs that contained a forward slash. Model IDs like "aaa/bbb" called at upstream/aaa/bbb would result in an error. This commit adds support for model IDs with a forward slash by iteratively searching the path for a match. Fixes: #229	2025-09-13 13:37:03 -07:00
Artur Podsiadły	558801db1a	Fix nginx proxy buffering for streaming endpoints (#295 ) * Fix nginx proxy buffering for streaming endpoints - Add X-Accel-Buffering: no header to SSE endpoints (/api/events, /logs/stream) - Add X-Accel-Buffering: no header to proxied text/event-stream responses - Add nginx reverse proxy configuration section to README - Add tests for X-Accel-Buffering header on streaming endpoints Fixes #236 * Fix goroutine cleanup in streaming endpoints test Add context cancellation to TestProxyManager_StreamingEndpointsReturnNoBufferingHeader to ensure the goroutine is properly cleaned up when the test completes.	2025-09-09 16:07:46 -07:00
Benson Wong	b21dee27c1	Fix #288 Vite hot module reloading creating multiple SSE connections (#290 ) - move SSE (EventSource) connection to module level - manage EventSource as a singleton, closing open connection before reopening a new one	2025-09-07 21:48:58 -07:00
Benson Wong	f58c8c8ec5	Support llama.cpp's cache_n in timings info (#287 ) Capture prompt cache metrics and surface them on Activities page in UI	2025-09-06 13:58:02 -07:00
Benson Wong	954e2dee73	Remove `cmdStart` from README [skip ci] cmdStart was in the README but it doesn't exist. Fixed the typo. Oops.	2025-09-04 11:57:28 -07:00
Benson Wong	a533aec736	small tweak to example config	2025-09-01 21:26:58 -07:00
Brett Profitt	97b17fc47d	Add ${MODEL_ID} macro (#226 ) The automatic ${MODEL_ID} macro includes the name of the model and can be used in Cmd and CmdStop.	2025-09-01 21:21:37 -07:00
Benson Wong	2457840698	Update README.md [skip ci]	2025-08-28 23:44:37 -07:00
Benson Wong	7f55494151	Update README.md [skip ci]	2025-08-28 22:47:28 -07:00
Benson Wong	831a90d3b0	Add different timeout scenarios to Process.checkHealthEndpoint #276 (#278 ) - add a TCP connection timeout of 500ms - increase HTTP client timeout to 5000ms In this new behaviour the upstream has 500ms to accept a tcp connection and 5000ms to respond to the HTTP request.	2025-08-28 22:03:14 -07:00
Yandrik	977f1856bb	add /completion endpoint (#275 ) * feat: add /completion endpoint * chore: reformat using gofmt	2025-08-28 21:41:02 -07:00
Benson Wong	52b329f7bc	Fix #277 race condition in ProcessGroup.ProxyRequest when swap=true	2025-08-28 21:38:40 -07:00
Benson Wong	57803fd3aa	Support llama-server's /infill endpoint (#272 ) Add support for llama-server's /infill endpoint and metrics gathering on the Activities page.	2025-08-27 08:36:05 -07:00
Benson Wong	c55d0cc842	Add docs for model.concurrencyLimit #263 [skip ci]	2025-08-22 16:08:37 -07:00