InferenceClient

Namespace: Daisi.SDK.Clients.V1.Host

This is a client that was built to interact with Hosts. There are a few things to remember when using this client:
  1. Host machines vary in speed and overall performance. Hosting your own machines and using your own network is the best way to ensure consistency.
  2. Every interaction with a Host requires an Orc approved session and a client key that has been granted acccess to the Host via the session.
  3. This client will automatically determine the fastest way to connect to the designated Host (Direct Connect (DC) or Fully Orchestrated Connect (FOC)), unless you override that feature.
  4. This client should be closed using client.SessionManager.Close() after you are done using it so that the session doesn't expire. Multiple unclosed sessions will knock your network rating.

Factory

var factory = new InferenceClientFactory();
var infClient = factory.Create();

Properties

  • InferenceSessionManager SessionManager get
    Handles setting up and maintaining the Session.
  • string? InferenceIdget
    Unique Inference Session Id from Orc.

Methods

  • Task<CloseInferenceResponse> CloseAsync(bool closeOrcSession = true)
    Closes the inference session with the Orc and Host. Closes the whole Orc Session by default as well.
  • AsyncServerStreamingCall<SendInferenceResponse> Send(string)
    Sends an inference request using plain text. Requires connected session, but that will be handled by the client.
  • AsyncServerStreamingCall<SendInferenceResponse> Send(SendInferenceRequest)
    Sends an inference request with optional parameters. Requires connected session, but that will be handled by the client.
  • InferenceStatsResponse Stats(InferenceStatsRequest)
    Retrieves the statistics from the last Send request as well as the total Inference Session.

Walkthrough: Send an inference request to a host

  1. Create an InferenceClientFactory.
  2. Create an InferenceClient.
  3. Create a SendInferenceRequest, if you want specific criteria set on the request. Otherwise, you can just send a string with your user input.
  4. Call Send and handle stream.
// Create the factory.
var infClientFactory = new InferenceClientFactory();

// Create a new Inference Session
var infClient = infClientFactory.Create();

// Use CreateDefault so that you don't have to setup all of the settings manually.
var sendInferenceRequest = SendInferenceRequest.CreateDefault();
sendInferenceRequest.Text = "Tell me a joke about an orc";

// Send the request to the Host. 
// The client determines FOC or DC.
var call = infClient.Send(sendInferenceRequest);

// This also works if you don't want to create the full SendInferenceRequest first.
// var call = infClient.Send("Tell me a joke about an orc");

await foreach(var resp in call.ResponseStream.ReadAllAsync())
{
    Console.WriteLine(resp.Content);
}