Token Budget & Context Recovery

Overview

Gravity's agent system maintains conversation history to provide context for LLM interactions. As sessions grow, token counts can exceed model context windows, causing 413 (Payload Too Large) errors. Gravity implements an autonomic token budget system with context recovery to handle this gracefully.

Token Budget

The token budget system monitors conversation length and automatically triggers compression when approaching limits.

Budget Calculation

// Services/TokenBudgetService.cs
public class TokenBudgetService
{
    private readonly int _modelContextWindow;
    private readonly int _reservedTokens = 4096; // reserved for response
    
    public TokenBudgetService(int modelContextWindow)
    {
        _modelContextWindow = modelContextWindow;
    }
    
    public int AvailableTokens => _modelContextWindow - _reservedTokens;
    
    public bool IsOverBudget(int currentTokens) => currentTokens > AvailableTokens * 0.8;
}

Context Recovery

When the budget is exceeded, Gravity compresses history using a hierarchical summarization strategy:

1. Recent messages - Keep last N messages verbatim
2. Tool calls - Preserve all function invocations and results
3. Older messages - Summarize into a single context block

413 Error Recovery

If compression fails and a 413 error occurs, the system automatically retries with aggressive compression:

// Automatic retry with reduced context
catch (HttpRequestException ex) when (ex.StatusCode == HttpStatusCode.RequestEntityTooLarge)
{
    var compressed = await _tokenBudget.CompressAggressively(history);
    return await SendWithRetry(compressed);
}