proxy,proxy/config: add global TTL feature (#554)

Add a new configuration parameter globalTTL that all models will
inherit. The default value is 0 which matches the currently
functionality to never automatically unload a model.

The model.ttl's default has changed to -1, which means use the global
TTL value. Any model.ttl >=0 is now value with 0 meaning never unload.
This allows a model to override a globalTTL > 0 and be configured to
never unload.

Fixes #459
Closes #512
This commit is contained in:
Benson Wong
2026-03-01 21:02:12 -08:00
committed by GitHub
parent 390a35bf93
commit cc77139ff8
7 changed files with 115 additions and 18 deletions
+5 -1
View File
@@ -5,6 +5,10 @@ import (
"runtime"
)
const (
MODEL_CONFIG_DEFAULT_TTL = -1
)
type ModelConfig struct {
Cmd string `yaml:"cmd"`
CmdStop string `yaml:"cmdStop"`
@@ -47,7 +51,7 @@ func (m *ModelConfig) UnmarshalYAML(unmarshal func(interface{}) error) error {
Aliases: []string{},
Env: []string{},
CheckEndpoint: "/health",
UnloadAfter: 0,
UnloadAfter: MODEL_CONFIG_DEFAULT_TTL, // use GlobalTTL
Unlisted: false,
UseModelName: "",
ConcurrencyLimit: 0,