Trait ModelProvider 

pub trait ModelProvider: Send + Sync {
    // Required method
    fn complete<'life0, 'async_trait>(
        &'life0 self,
        request: CompletionRequest,
    ) -> Pin<Box<dyn Future<Output = Result<CompletionResponse, AgentError>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait;

    // Provided methods
    fn stream_complete<'a>(
        &'a self,
        request: CompletionRequest,
    ) -> Pin<Box<dyn Stream<Item = Result<ProviderEvent, AgentError>> + Send + 'a>> { ... }
    fn max_tokens(&self) -> u32 { ... }
    fn estimate_tokens(&self, text: &str) -> usize { ... }
}

Abstract LLM backend. Implement this to plug in a new model provider.

Required Methods

fn complete<'life0, 'async_trait>( &'life0 self, request: CompletionRequest, ) -> Pin<Box<dyn Future<Output = Result<CompletionResponse, AgentError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Issue a single non-streaming completion request.

Provided Methods

fn stream_complete<'a>( &'a self, request: CompletionRequest, ) -> Pin<Box<dyn Stream<Item = Result<ProviderEvent, AgentError>> + Send + 'a>>

Stream a completion as a sequence of ProviderEvents.

The default implementation calls complete and re-emits the result as one synthetic stream — providers that genuinely stream should override this for incremental delivery.

fn max_tokens(&self) -> u32

Return the maximum number of tokens that can be requested from this provider.

The default implementation returns 128,000 tokens, which is the maximum supported by the cl100k_base tokenizer used by most providers. Providers with a different tokenizer (e.g. Gemini’s SentencePiece) can override this.

fn estimate_tokens(&self, text: &str) -> usize

Estimate the number of tokens for the given text.

The default implementation uses the cl100k_base tokenizer (via tiktoken-rs). Since DeepSeek, Kimi, and most OpenAI-compatible providers all use cl100k_base-compatible BPE tokenizers, this gives ±5% accuracy across all built-in providers.

Providers with a different tokenizer (e.g. Gemini’s SentencePiece) can override this.

Trait Implementations

impl ModelProvider for &(dyn ModelProvider + Send + Sync)

fn complete<'life0, 'async_trait>( &'life0 self, request: CompletionRequest, ) -> Pin<Box<dyn Future<Output = Result<CompletionResponse, AgentError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Issue a single non-streaming completion request.

fn stream_complete<'a>( &'a self, request: CompletionRequest, ) -> Pin<Box<dyn Stream<Item = Result<ProviderEvent, AgentError>> + Send + 'a>>

Stream a completion as a sequence of ProviderEvents.

fn estimate_tokens(&self, text: &str) -> usize

Estimate the number of tokens for the given text.

fn max_tokens(&self) -> u32

Return the maximum number of tokens that can be requested from this provider.

Implementations on Foreign Types

impl ModelProvider for Arc<dyn ModelProvider + Send + Sync>

Implement ModelProvider for Arc<dyn ModelProvider + Send + Sync>.

fn complete<'life0, 'async_trait>( &'life0 self, request: CompletionRequest, ) -> Pin<Box<dyn Future<Output = Result<CompletionResponse, AgentError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

fn stream_complete<'a>( &'a self, request: CompletionRequest, ) -> Pin<Box<dyn Stream<Item = Result<ProviderEvent, AgentError>> + Send + 'a>>

fn estimate_tokens(&self, text: &str) -> usize

Implementors

impl ModelProvider for &(dyn ModelProvider + Send + Sync)

impl ModelProvider for MockProvider

impl ModelProvider for DeepSeekProvider

impl ModelProvider for RoutedProvider

impl ModelProvider for ErasedProvider<'_>