steve/go-llm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Comprehensive token usage tracking for V2 #2

New Issue
Closed
opened 2026-03-02 04:17:17 +00:00 by Claude · 2 comments
Claude commented 2026-03-02 04:17:17 +00:00
Collaborator
Copy Link

Problem

V2 callers cannot fully track token consumption across all API surfaces:

  • Streaming: OpenAI and Google streaming don't capture usage in the done event
  • Chat.Send(): Runs a tool-call loop but only returns string — usage from all iterations lost
  • Generate[T]: Returns only the typed result — usage lost
  • Agent.Run(): Delegates to Chat but doesn't expose usage
  • Provider-specific details: Anthropic cache tokens, OpenAI reasoning/cached tokens, Google thoughts tokens — all discarded
  • Cost tracking: No pricing/cost mechanism exists

Solution

  1. Add Details map[string]int to Usage for provider-specific token breakdowns
  2. Fix streaming usage for OpenAI and Google
  3. Return *Usage from Chat.Send*, Generate[T], and Agent.Run*
  4. Enhance UsageTracker middleware to accumulate details
  5. Add ModelPricing / PricingRegistry utility for cost computation
## Problem V2 callers cannot fully track token consumption across all API surfaces: - **Streaming**: OpenAI and Google streaming don't capture usage in the done event - **Chat.Send()**: Runs a tool-call loop but only returns `string` — usage from all iterations lost - **Generate[T]**: Returns only the typed result — usage lost - **Agent.Run()**: Delegates to Chat but doesn't expose usage - **Provider-specific details**: Anthropic cache tokens, OpenAI reasoning/cached tokens, Google thoughts tokens — all discarded - **Cost tracking**: No pricing/cost mechanism exists ## Solution 1. Add `Details map[string]int` to `Usage` for provider-specific token breakdowns 2. Fix streaming usage for OpenAI and Google 3. Return `*Usage` from `Chat.Send*`, `Generate[T]`, and `Agent.Run*` 4. Enhance `UsageTracker` middleware to accumulate details 5. Add `ModelPricing` / `PricingRegistry` utility for cost computation
Claude commented 2026-03-02 04:17:26 +00:00
Author
Collaborator
Copy Link

Starting work on this. Plan of approach:

  1. Phase 1: Add Details map[string]int to provider.Usage and public Usage, extract provider-specific details from OpenAI/Anthropic/Google
  2. Phase 2: Fix streaming usage for OpenAI (enable StreamOptions.IncludeUsage) and Google (capture UsageMetadata from last chunk)
  3. Phase 3: Change signatures — Chat.Send*, Generate[T], Agent.Run* now return *Usage
  4. Phase 4: Enhance UsageTracker middleware to accumulate detail values
  5. Phase 5: Add ModelPricing/PricingRegistry in v2/pricing.go
  6. Phase 6: Tests and verification
Starting work on this. Plan of approach: 1. **Phase 1**: Add `Details map[string]int` to `provider.Usage` and public `Usage`, extract provider-specific details from OpenAI/Anthropic/Google 2. **Phase 2**: Fix streaming usage for OpenAI (enable `StreamOptions.IncludeUsage`) and Google (capture `UsageMetadata` from last chunk) 3. **Phase 3**: Change signatures — `Chat.Send*`, `Generate[T]`, `Agent.Run*` now return `*Usage` 4. **Phase 4**: Enhance `UsageTracker` middleware to accumulate detail values 5. **Phase 5**: Add `ModelPricing`/`PricingRegistry` in `v2/pricing.go` 6. **Phase 6**: Tests and verification
Claude commented 2026-03-02 04:33:50 +00:00
Author
Collaborator
Copy Link

Work finished. PR: #3

Also created steve/steveternet#47 for the V1 audit.

Work finished. PR: https://gitea.stevedudenhoeffer.com/steve/go-llm/pulls/3 Also created steve/steveternet#47 for the V1 audit.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
Claude fizi steve
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: steve/go-llm#2
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?