huggingface.tgi
Module huggingface.tgi
API
Definitions
ballerinax/huggingface.tgi Ballerina library
Overview
Hugging Face Text Generation Inference (TGI) is a server for deploying and serving large language models with high performance.
The Hugging Face TGI connector lets Ballerina applications generate text and stream tokens, create OpenAI-compatible chat and text completions, tokenize text and chat conversations, and inspect the health, metrics and model information of a TGI server. It supports version 3.3.6 of the TGI API.
Key features
- Generate text and stream generated tokens as server-sent events
- Create OpenAI-compatible chat and text completions
- Tokenize plain text and chat conversations with the model's chat template
- Inspect server health, Prometheus metrics and served model information
Setup guide
To use the Hugging Face TGI connector, you need a running TGI server. If your server does not require authentication, any placeholder token is accepted.
Step 1: Start a TGI server
-
Install Docker and, for gated models, create a Hugging Face access token.
-
Launch the server with the model you want to serve:
docker run --gpus all --shm-size 1g -p 8080:80 -v $PWD/data:/data \ -e HF_TOKEN=<your-hugging-face-token> \ ghcr.io/huggingface/text-generation-inference:latest \ --model-id <model-id> -
Wait until the server finishes downloading and loading the model.
Step 2: Note the server URL and token
-
The server URL is the address you published, for example
http://localhost:8080. For a Hugging Face Inference Endpoint, use the endpoint URL. -
If the server sits behind a gateway or is an Inference Endpoint that requires authentication, create an access token and keep it for the connector configuration.
Step 3: Verify your setup
curl http://localhost:8080/info
The response describes the model that the server is running.
Quickstart
To use the Hugging Face TGI connector in your Ballerina application, update the .bal file as follows:
Step 1: Import the module
Import the huggingface.tgi module.
import ballerinax/huggingface.tgi;
Step 2: Instantiate a new connector
- Create a
Config.tomlfile and configure the server URL and token obtained in the steps above:
serviceUrl = "<TGI server URL>" token = "<access token>"
- Create a
tgi:Clientusing the configuration:
configurable string serviceUrl = ?; configurable string token = ?; final tgi:Client tgiClient = check new ({auth: {token}}, serviceUrl);
Step 3: Invoke the connector operation
- Generate text for a prompt:
public function main() returns error? { tgi:GenerateResponse _ = check tgiClient->generateTokens({inputs: "What is deep learning?"}); }
- Use
bal runto compile and run the Ballerina program.
Examples
The Hugging Face TGI connector provides practical examples illustrating usage in various scenarios. Explore these examples, covering the following use cases:
-
Text generation with a health check - Check the health of a TGI server, read the model information and generate text from a prompt.
-
Chat assistant with tokenization - Count the tokens of a templated chat prompt and ask the served model for a chat completion.
Clients
huggingface.tgi: Client
Run text generation, chat completions, tokenization and model inspection against a Hugging Face Text Generation Inference server.
Constructor
Gets invoked to initialize the connector.
init (ConnectionConfig config, string serviceUrl)- config ConnectionConfig - The configurations to be used when initializing the
connector
- serviceUrl string "http://localhost:8080" - URL of the target service
generateCompat
function generateCompat(CompatGenerateRequest payload, map<string|string[]> headers) returns GenerateResponse[]|errorGenerate tokens if stream == false or a stream of token if stream == true
Parameters
- payload CompatGenerateRequest - Input text and parameters for token generation
Return Type
- GenerateResponse[]|error - Generated Text
tokenizeChat
function tokenizeChat(ChatRequest payload, map<string|string[]> headers) returns ChatTokenizeResponse|errorTemplate and tokenize ChatRequest
Parameters
- payload ChatRequest - Chat messages and options to template and tokenize
Return Type
- ChatTokenizeResponse|error - Templated and tokenized ChatRequest
generateTokens
function generateTokens(GenerateRequest payload, map<string|string[]> headers) returns GenerateResponse|errorGenerate tokens
Parameters
- payload GenerateRequest - Input text and parameters for token generation
Return Type
- GenerateResponse|error - Generated Text
streamTokens
function streamTokens(GenerateRequest payload, map<string|string[]> headers) returns stream<SseEvent, error?>|errorGenerate a stream of token using Server-Sent Events
Parameters
- payload GenerateRequest - Input text and parameters for streamed token generation
checkHealth
Health check method
Return Type
- error? - Everything is working fine
getInfo
Text Generation Inference endpoint info
invokeSagemaker
function invokeSagemaker(SagemakerRequest payload, map<string|string[]> headers) returns SagemakerResponse|errorGenerate tokens from Sagemaker request
Parameters
- payload SagemakerRequest - Sagemaker style request for token generation
Return Type
- SagemakerResponse|error - Generated Chat Completion
getMetrics
Prometheus metrics scrape endpoint
tokenizeText
function tokenizeText(GenerateRequest payload, map<string|string[]> headers) returns TokenizeResponse|errorTokenize inputs
Parameters
- payload GenerateRequest - Input text and parameters to tokenize
Return Type
- TokenizeResponse|error - Tokenized ids
createChatCompletion
function createChatCompletion(ChatRequest payload, map<string|string[]> headers) returns ChatCompletion|errorCreate a chat completion
Parameters
- payload ChatRequest - Chat messages and options for the chat completion
Return Type
- ChatCompletion|error - Generated Chat Completion
createCompletion
function createCompletion(CompletionRequest payload, map<string|string[]> headers) returns CompletionFinal|errorCreate a text completion
Parameters
- payload CompletionRequest - Prompt and options for the text completion
Return Type
- CompletionFinal|error - Generated Chat Completion
listModels
function listModels(map<string|string[]> headers) returns ModelsInfo|errorGet model info
Return Type
- ModelsInfo|error - Served model info
Records
huggingface.tgi: BestOfSequence
Result of one of the additional best-of generation sequences
Fields
- finishReason FinishReason - Reason the generation of this sequence stopped
- topTokens? Token[][] - Most likely alternative tokens at each generation step
- seed? int? - Random seed used for this sequence
- generatedTokens Signed32 - Number of tokens generated for this sequence
- prefill PrefillToken[] - Tokens of the processed prompt with their log probabilities
- tokens Token[] - Tokens generated for this sequence
- generatedText string - Text generated for this sequence
huggingface.tgi: ChatCompletion
Complete chat completion response
Fields
- created int - Unix timestamp of when the completion was created
- usage Usage - Token usage statistics of the request
- model string - Name of the model that generated the completion
- id string - Unique identifier of the completion
- choices ChatCompletionComplete[] - List of completion choices generated for the request
- systemFingerprint string - Fingerprint of the server configuration that produced the completion
huggingface.tgi: ChatCompletionComplete
A single complete choice of a chat completion
Fields
- finishReason string - Reason the generation of this choice stopped
- index Signed32 - Index of the choice in the list of choices
- message OutputMessage - Message generated by the model
- logprobs? ChatCompletionLogprobs? - Log probability information for the choice
huggingface.tgi: ChatCompletionLogprob
Log probability information for a generated chat token
Fields
- topLogprobs ChatCompletionTopLogprob[] - Most likely alternative tokens with their log probabilities
- logprob float - Log probability of the token
- token string - The generated token text
huggingface.tgi: ChatCompletionLogprobs
Log probability information for the generated chat tokens
Fields
- content ChatCompletionLogprob[] - Log probability information for each generated token
huggingface.tgi: ChatCompletionTopLogprob
Log probability of one of the most likely alternative tokens
Fields
- logprob float - Log probability of the alternative token
- token string - The alternative token text
huggingface.tgi: ChatRequest
Request payload for generating a chat completion
Fields
- topLogprobs? Signed32? - An integer between 0 and 5 specifying the number of most likely tokens to return at each token position, each with an associated log probability. logprobs must be set to true if this parameter is used
- logitBias? float[]? - UNUSED Modify the likelihood of specified tokens appearing in the completion. Accepts a JSON object that maps tokens (specified by their token ID in the tokenizer) to an associated bias value from -100 to 100. Mathematically, the bias is added to the logits generated by the model prior to sampling. The exact effect will vary per model, but values between -1 and 1 should decrease or increase likelihood of selection; values like -100 or 100 should result in a ban or exclusive selection of the relevant token
- seed? int? - Random seed used to make the generation reproducible
- maxTokens? Signed32? - The maximum number of tokens that can be generated in the chat completion
- presencePenalty? float? - Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far, increasing the model's likelihood to talk about new topics
- toolPrompt? string? - A prompt to be appended before the tools
- tools? Tool[]? - A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for
- n? Signed32? - UNUSED How many chat completion choices to generate for each input message. Note that you will be charged based on the number of generated tokens across all of the choices. Keep n as 1 to minimize costs
- logprobs? boolean? - Whether to return log probabilities of the output tokens or not. If true, returns the log probabilities of each output token returned in the content of message
- topP? float? - An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered
- frequencyPenalty? float? - Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model's likelihood to repeat the same line verbatim
- responseFormat? GrammarType? - Format constraint applied to the generated response
- stop? string[]? - Up to 4 sequences where the API will stop generating further tokens
- 'stream? boolean - Whether to stream the response as server-sent events
- temperature? float? - What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while
lower values like 0.2 will make it more focused and deterministic.
We generally recommend altering this or
top_pbut not both
- messages Message[] - A list of messages comprising the conversation so far
- toolChoice ToolChoice?(default "auto") - Controls whether and which tool the model calls
- model? string? - [UNUSED] ID of the model to use. See the model endpoint compatibility table for details on which models work with the Chat API
- streamOptions? StreamOptions? - Options applied when the response is streamed
huggingface.tgi: ChatTokenizeResponse
Response of tokenizing a chat request, with the templated text and token list
Fields
- templatedText string - Chat text after the chat template was applied
- tokenizeResponse TokenizeResponse - Tokens produced from the templated text
huggingface.tgi: CompatGenerateRequest
Request payload for token generation in the compatibility mode
Fields
- 'stream boolean(default false) - Whether to stream the generated tokens
- inputs string - Input text to generate tokens from
- parameters? GenerateParameters - Parameters controlling the generation
huggingface.tgi: CompletionComplete
A single choice of a text completion
Fields
- finishReason string - Reason the generation of this choice stopped
- index Signed32 - Index of the choice in the list of choices
- text string - Generated completion text
- logprobs? float[]? - Log probabilities of the generated tokens
huggingface.tgi: CompletionFinal
Complete text completion response
Fields
- created int - Unix timestamp of when the completion was created
- usage Usage - Token usage statistics of the request
- model string - Name of the model that generated the completion
- id string - Unique identifier of the completion
- choices CompletionComplete[] - List of completion choices generated for the request
- systemFingerprint string - Fingerprint of the server configuration that produced the completion
huggingface.tgi: CompletionRequest
Request payload for generating a text completion
Fields
- topP? float? - An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered
- frequencyPenalty? float? - Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model's likelihood to repeat the same line verbatim
- seed? int? - Random seed used to make the generation reproducible
- stop? string[]? - Up to 4 sequences where the API will stop generating further tokens
- maxTokens? Signed32? - The maximum number of tokens that can be generated in the chat completion
- 'stream? boolean - Whether to stream the response as server-sent events
- temperature? float? - What sampling temperature to use, between 0 and 2. Higher values like 0.8 will make the output more random, while
lower values like 0.2 will make it more focused and deterministic. We generally recommend altering this or
top_pbut not both
- model? string? - UNUSED ID of the model to use. See the model endpoint compatibility table for details on which models work with the Chat API
- suffix? string? - The text to append to the prompt. This is useful for completing sentences or generating a paragraph of text. please see the completion_template field in the model's tokenizer_config.json file for completion template
- prompt Prompt - Prompt or prompts to generate completions for
- repetitionPenalty? float? - Penalty applied to repeated tokens, where 1.0 means no penalty
huggingface.tgi: ConnectionConfig
Provides a set of configurations for controlling the behaviours when communicating with a remote HTTP endpoint.
Fields
- auth BearerTokenConfig - Configurations related to client authentication
- httpVersion HttpVersion(default http:HTTP_2_0) - The HTTP version understood by the client
- http1Settings ClientHttp1Settings(default {}) - Configurations related to HTTP/1.x protocol
- http2Settings ClientHttp2Settings(default {}) - Configurations related to HTTP/2 protocol
- timeout decimal(default 30) - The maximum time to wait (in seconds) for a response before closing the connection
- forwarded string(default "disable") - The choice of setting
forwarded/x-forwardedheader
- followRedirects? FollowRedirects - Configurations associated with Redirection
- poolConfig? PoolConfiguration - Configurations associated with request pooling
- cache CacheConfig(default {}) - HTTP caching related configurations
- compression Compression(default http:COMPRESSION_AUTO) - Specifies the way of handling compression (
accept-encoding) header
- circuitBreaker? CircuitBreakerConfig - Configurations associated with the behaviour of the Circuit Breaker
- retryConfig? RetryConfig - Configurations associated with retrying
- cookieConfig? CookieConfig - Configurations associated with cookies
- responseLimits ResponseLimitConfigs(default {}) - Configurations associated with inbound response size limits
- secureSocket? ClientSecureSocket - SSL/TLS-related options
- proxy? ProxyConfig - Proxy server related options
- socketConfig ClientSocketConfig(default {}) - Provides settings related to client socket configuration
- validation boolean(default true) - Enables the inbound payload validation functionality which provided by the constraint package. Enabled by default
- laxDataBinding boolean(default true) - Enables relaxed data binding on the client side. When enabled,
nilvalues are treated as optional, and absent fields are handled asnilabletypes. Enabled by default.
huggingface.tgi: Details
Details of a token generation, returned when requested
Fields
- finishReason FinishReason - Reason the generation stopped
- topTokens? Token[][] - Most likely alternative tokens at each generation step
- seed? int? - Random seed used for the generation
- generatedTokens Signed32 - Number of tokens generated
- prefill PrefillToken[] - Tokens of the processed prompt with their log probabilities
- tokens Token[] - Tokens generated for the request
- bestOfSequences? BestOfSequence[]? - Additional best-of sequences generated for the request
huggingface.tgi: FunctionDefinition
Definition of a function the model may call
Fields
- name string - Name of the function
- description? string? - Description of what the function does
- arguments anydata - Parameters the function accepts, described as a JSON schema
huggingface.tgi: FunctionName
Name of a function
Fields
- name string - Name of the function
huggingface.tgi: GenerateParameters
Parameters controlling token generation
Fields
- watermark boolean(default false) - Watermarking with A Watermark for Large Language Models
- seed? int? - Random sampling seed
- doSample boolean(default false) - Activate logits sampling
- topP? float? - Top-p value for nucleus sampling
- frequencyPenalty? float? - The parameter for frequency penalty. 1.0 means no penalty Penalize new tokens based on their existing frequency in the text so far, decreasing the model's likelihood to repeat the same line verbatim
- grammar? GrammarType? - Grammar constraint applied to the generated output
- bestOf? int? - Generate best_of sequences and return the one if the highest token logprobs
- stop? string[] - Stop generating tokens if a member of
stopis generated
- truncate? int? - Truncate inputs tokens to the given size
- topNTokens? Signed32? - The number of highest probability vocabulary tokens to keep for top-n-filtering
- maxNewTokens? Signed32? - Maximum number of tokens to generate
- decoderInputDetails boolean(default false) - Whether to return decoder input token logprobs and ids
- topK? Signed32? - The number of highest probability vocabulary tokens to keep for top-k-filtering
- temperature? float? - The value used to module the logits distribution
- typicalP? float? - Typical Decoding mass See Typical Decoding for Natural Language Generation for more information
- adapterId? string? - Lora adapter id
- details boolean(default true) - Whether to return generation details
- returnFullText? boolean? - Whether to prepend the prompt to the generated text
- repetitionPenalty? float? - The parameter for repetition penalty. 1.0 means no penalty. See this paper for more details
huggingface.tgi: GenerateRequest
Request payload for generating tokens
Fields
- inputs string - Input text to generate tokens from
- parameters? GenerateParameters - Parameters controlling the generation
huggingface.tgi: GenerateResponse
Response of a token generation request
Fields
- details? Details? - Details of the generation, returned when requested
- generatedText string - Text generated by the model
huggingface.tgi: ImageUrlMessageChunk
Fields
- imageUrl Url - Image referenced by the chunk
- 'type "image_url" - Type of the chunk, such as text or image
huggingface.tgi: Info
Information about the running inference server and the loaded model
Fields
- maxConcurrentRequests int - Router Parameters
- validationWorkers int - Number of workers used to validate requests
- maxClientBatchSize int - Maximum number of inputs allowed in a client batch
- maxBestOf int - Maximum number of best-of sequences allowed per request
- maxStopSequences int - Maximum number of stop sequences allowed per request
- modelId string - Model info
- sha? string? - Commit hash of the running server build
- version string - Version of the running server
- maxTotalTokens int - Maximum total number of input and generated tokens per request
- router string - Router Info
- modelPipelineTag? string? - Pipeline tag of the loaded model
- modelSha? string? - Commit hash of the loaded model
- dockerLabel? string? - Docker label of the running server image
- maxInputTokens int - Maximum number of input tokens accepted per request
huggingface.tgi: JsonGrammar
Fields
- 'type "json" - Kind of grammar constraint, such as JSON or regex
- value anydata - A string that represents a JSON Schema. JSON Schema is a declarative language that allows to annotate JSON documents with types and descriptions
huggingface.tgi: JsonSchemaConfig
JSON schema configuration used to constrain the response format
Fields
- schema anydata - The actual JSON schema definition
- name? string? - Optional name identifier for the schema
huggingface.tgi: JsonSchemaGrammar
Fields
- 'type "json_schema" - Kind of grammar constraint, such as JSON or regex
- value JsonSchemaConfig - Grammar definition, such as a JSON schema or a regular expression
huggingface.tgi: Message
A message of a chat conversation
Fields
- Fields Included from *MessageContentFields
- toolCalls ToolCall[]
- content MessageContent
- anydata...
- Fields Included from *MessageRoleFields
huggingface.tgi: MessageContentFields
Fields
- toolCalls? ToolCall[] - Tool calls made by the model
- content? MessageContent - Content of the message
huggingface.tgi: MessageRoleFields
Fields
- role string - Role of the message author
- name? string? - Optional name of the message author
huggingface.tgi: ModelInfo
Information about a served model
Fields
- created int - Unix timestamp of when the model was created
- ownedBy string - Owner of the model
- id string - Identifier of the model
- 'object string - Object type of the model entry
huggingface.tgi: ModelsInfo
List of the models served by the endpoint
Fields
- data ModelInfo[] - Models served by the endpoint
- 'object string - Object type of the response, always
list
huggingface.tgi: PrefillToken
A token of the processed prompt with its log probability
Fields
- logprob float? - Log probability of the token
- id Signed32 - Identifier of the token
- text string - Text of the token
huggingface.tgi: RegexGrammar
Fields
- 'type "regex" - Kind of grammar constraint, such as JSON or regex
- value string - Grammar definition, such as a JSON schema or a regular expression
huggingface.tgi: SimpleToken
A token with its position in the input text
Fields
- stop int - End offset of the token in the input text
- 'start int - Start offset of the token in the input text
- id Signed32 - Identifier of the token
- text string - Text of the token
huggingface.tgi: StreamOptions
Options controlling a streamed response
Fields
- includeUsage? boolean - If set, an additional chunk will be streamed before the data: [DONE] message. The usage field on this chunk shows the token usage statistics for the entire request, and the choices field will always be an empty array. All other chunks will also include a usage field, but with a null value
huggingface.tgi: TextMessage
A text message of a chat conversation
Fields
- role string - Role of the message author
- toolCallId? string? - Identifier of the tool call the message responds to
- content string - Text content of the message
huggingface.tgi: TextMessageChunk
Fields
- text string - Text content of the chunk
- 'type "text" - Type of the chunk, such as text or image
huggingface.tgi: Token
A generated token with its log probability
Fields
- special boolean - Whether the token is a special token
- logprob float? - Log probability of the token
- id Signed32 - Identifier of the token
- text string - Text of the token
huggingface.tgi: Tool
A tool the model may call
Fields
- 'function FunctionDefinition - Definition of the function the tool provides
- 'type string - Type of the tool
huggingface.tgi: ToolCall
A tool call made by the model
Fields
- 'function FunctionDefinition - Function invocation of the tool call
- id string - Identifier of the tool call
- 'type string - Type of the tool call
huggingface.tgi: ToolCallMessage
A message carrying the tool calls made by the model
Fields
- role string - Role of the message author
- toolCalls ToolCall[] - Tool calls made by the model
huggingface.tgi: ToolChoiceFunction
Fields
- 'function FunctionName - Function the model is required to call
huggingface.tgi: Url
A URL reference to an image
Fields
- url string - URL of the image
huggingface.tgi: Usage
Token usage statistics of a request
Fields
- completionTokens Signed32 - Number of tokens in the generated completion
- promptTokens Signed32 - Number of tokens in the prompt
- totalTokens Signed32 - Total number of tokens used by the request
Union types
huggingface.tgi: MessageContent
MessageContent
Content of a chat message, as plain text or a list of chunks
huggingface.tgi: GrammarType
GrammarType
Grammar constraint applied to the generated output
huggingface.tgi: SagemakerResponse
SagemakerResponse
Response of the Sagemaker compatible invocation
huggingface.tgi: FinishReason
FinishReason
Reason the token generation stopped
huggingface.tgi: ToolChoice
ToolChoice
huggingface.tgi: SagemakerRequest
SagemakerRequest
Request payload of the Sagemaker compatible invocation
huggingface.tgi: MessageChunk
MessageChunk
A chunk of message content, either text or an image URL
huggingface.tgi: OutputMessage
OutputMessage
A message produced by the model
Array types
huggingface.tgi: TokenizeResponse
TokenizeResponse
List of tokens produced by tokenizing an input
huggingface.tgi: MessageContentChunks
MessageContentChunks
huggingface.tgi: Prompt
Prompt
Prompt of a completion request, as a single string or a list of strings
String types
huggingface.tgi: MessageContentText
MessageContentText
Import
import ballerinax/huggingface.tgi;Other versions
1.0.0
Metadata
Released date: 1 day ago
Version: 1.0.0
License: Apache-2.0
Security
Direct vulnerabilities: No known direct vulnerabilities
Indirect vulnerabilities:
Checked on 2026-11-21 via OSV.dev
Compatibility
Platform: any
Ballerina version: 2201.12.0
GraalVM compatible: Yes
Pull count
Total: 1
Current verison: 1
Weekly downloads
Keywords
Vendor/Hugging Face
Area/AI & Machine Learning
Type/Connector
Contributors