Underground-Digital/Workflow-Engine
0
1# Interface Methods2 3This section describes the interface methods and parameter explanations that need to be implemented by providers and various model types.4 5## Provider6 7Inherit the `__base.model_provider.ModelProvider` base class and implement the following interfaces:8 9```python10def validate_provider_credentials(self, credentials: dict) -> None:11 """12 Validate provider credentials13 You can choose any validate_credentials method of model type or implement validate method by yourself,14 such as: get model list api15 16 if validate failed, raise exception17 18 :param credentials: provider credentials, credentials form defined in `provider_credential_schema`.19 """20```21 22- `credentials` (object) Credential information23 24 The parameters of credential information are defined by the `provider_credential_schema` in the provider's YAML configuration file. Inputs such as `api_key` are included.25 26If verification fails, throw the `errors.validate.CredentialsValidateFailedError` error.27 28## Model29 30Models are divided into 5 different types, each inheriting from different base classes and requiring the implementation of different methods.31 32All models need to uniformly implement the following 2 methods:33 34- Model Credential Verification35 36 Similar to provider credential verification, this step involves verification for an individual model.37 38 39 ```python40 def validate_credentials(self, model: str, credentials: dict) -> None:41 """42 Validate model credentials43 44 :param model: model name45 :param credentials: model credentials46 :return:47 """48 ```49 50 Parameters:51 52 - `model` (string) Model name53 54 - `credentials` (object) Credential information55 56 The parameters of credential information are defined by either the `provider_credential_schema` or `model_credential_schema` in the provider's YAML configuration file. Inputs such as `api_key` are included.57 58 If verification fails, throw the `errors.validate.CredentialsValidateFailedError` error.59 60- Invocation Error Mapping Table61 62 When there is an exception in model invocation, it needs to be mapped to the `InvokeError` type specified by Runtime. This facilitates Dify's ability to handle different errors with appropriate follow-up actions.63 64 Runtime Errors:65 66 - `InvokeConnectionError` Invocation connection error67 - `InvokeServerUnavailableError` Invocation service provider unavailable68 - `InvokeRateLimitError` Invocation reached rate limit69 - `InvokeAuthorizationError` Invocation authorization failure70 - `InvokeBadRequestError` Invocation parameter error71 72 ```python73 @property74 def _invoke_error_mapping(self) -> dict[type[InvokeError], list[type[Exception]]]:75 """76 Map model invoke error to unified error77 The key is the error type thrown to the caller78 The value is the error type thrown by the model,79 which needs to be converted into a unified error type for the caller.80 81 :return: Invoke error mapping82 """83 ```84 85 You can refer to OpenAI's `_invoke_error_mapping` for an example.86 87### LLM88 89Inherit the `__base.large_language_model.LargeLanguageModel` base class and implement the following interfaces:90 91- LLM Invocation92 93 Implement the core method for LLM invocation, which can support both streaming and synchronous returns.94 95 96 ```python97 def _invoke(self, model: str, credentials: dict,98 prompt_messages: list[PromptMessage], model_parameters: dict,99 tools: Optional[list[PromptMessageTool]] = None, stop: Optional[List[str]] = None,100 stream: bool = True, user: Optional[str] = None) \101 -> Union[LLMResult, Generator]:102 """103 Invoke large language model104 105 :param model: model name106 :param credentials: model credentials107 :param prompt_messages: prompt messages108 :param model_parameters: model parameters109 :param tools: tools for tool calling110 :param stop: stop words111 :param stream: is stream response112 :param user: unique user id113 :return: full response or stream response chunk generator result114 """115 ```116 117 - Parameters:118 119 - `model` (string) Model name120 121 - `credentials` (object) Credential information122 123 The parameters of credential information are defined by either the `provider_credential_schema` or `model_credential_schema` in the provider's YAML configuration file. Inputs such as `api_key` are included.124 125 - `prompt_messages` (array[[PromptMessage](#PromptMessage)]) List of prompts126 127 If the model is of the `Completion` type, the list only needs to include one [UserPromptMessage](#UserPromptMessage) element;128 129 If the model is of the `Chat` type, it requires a list of elements such as [SystemPromptMessage](#SystemPromptMessage), [UserPromptMessage](#UserPromptMessage), [AssistantPromptMessage](#AssistantPromptMessage), [ToolPromptMessage](#ToolPromptMessage) depending on the message.130 131 - `model_parameters` (object) Model parameters132 133 The model parameters are defined by the `parameter_rules` in the model's YAML configuration.134 135 - `tools` (array[[PromptMessageTool](#PromptMessageTool)]) [optional] List of tools, equivalent to the `function` in `function calling`.136 137 That is, the tool list for tool calling.138 139 - `stop` (array[string]) [optional] Stop sequences140 141 The model output will stop before the string defined by the stop sequence.142 143 - `stream` (bool) Whether to output in a streaming manner, default is True144 145 Streaming output returns Generator[[LLMResultChunk](#LLMResultChunk)], non-streaming output returns [LLMResult](#LLMResult).146 147 - `user` (string) [optional] Unique identifier of the user148 149 This can help the provider monitor and detect abusive behavior.150 151 - Returns152 153 Streaming output returns Generator[[LLMResultChunk](#LLMResultChunk)], non-streaming output returns [LLMResult](#LLMResult).154 155- Pre-calculating Input Tokens156 157 If the model does not provide a pre-calculated tokens interface, you can directly return 0.158 159 ```python160 def get_num_tokens(self, model: str, credentials: dict, prompt_messages: list[PromptMessage],161 tools: Optional[list[PromptMessageTool]] = None) -> int:162 """163 Get number of tokens for given prompt messages164 165 :param model: model name166 :param credentials: model credentials167 :param prompt_messages: prompt messages168 :param tools: tools for tool calling169 :return:170 """171 ```172 173 For parameter explanations, refer to the above section on `LLM Invocation`.174 175- Fetch Custom Model Schema [Optional]176 177 ```python178 def get_customizable_model_schema(self, model: str, credentials: dict) -> Optional[AIModelEntity]:179 """180 Get customizable model schema181 182 :param model: model name183 :param credentials: model credentials184 :return: model schema185 """186 ```187 188 When the provider supports adding custom LLMs, this method can be implemented to allow custom models to fetch model schema. The default return null.189 190 191### TextEmbedding192 193Inherit the `__base.text_embedding_model.TextEmbeddingModel` base class and implement the following interfaces:194 195- Embedding Invocation196 197 ```python198 def _invoke(self, model: str, credentials: dict,199 texts: list[str], user: Optional[str] = None) \200 -> TextEmbeddingResult:201 """202 Invoke large language model203 204 :param model: model name205 :param credentials: model credentials206 :param texts: texts to embed207 :param user: unique user id208 :return: embeddings result209 """210 ```211 212 - Parameters:213 214 - `model` (string) Model name215 216 - `credentials` (object) Credential information217 218 The parameters of credential information are defined by either the `provider_credential_schema` or `model_credential_schema` in the provider's YAML configuration file. Inputs such as `api_key` are included.219 220 - `texts` (array[string]) List of texts, capable of batch processing221 222 - `user` (string) [optional] Unique identifier of the user223 224 This can help the provider monitor and detect abusive behavior.225 226 - Returns:227 228 [TextEmbeddingResult](#TextEmbeddingResult) entity.229 230- Pre-calculating Tokens231 232 ```python233 def get_num_tokens(self, model: str, credentials: dict, texts: list[str]) -> int:234 """235 Get number of tokens for given prompt messages236 237 :param model: model name238 :param credentials: model credentials239 :param texts: texts to embed240 :return:241 """242 ```243 244 For parameter explanations, refer to the above section on `Embedding Invocation`.245 246### Rerank247 248Inherit the `__base.rerank_model.RerankModel` base class and implement the following interfaces:249 250- Rerank Invocation251 252 ```python253 def _invoke(self, model: str, credentials: dict,254 query: str, docs: list[str], score_threshold: Optional[float] = None, top_n: Optional[int] = None,255 user: Optional[str] = None) \256 -> RerankResult:257 """258 Invoke rerank model259 260 :param model: model name261 :param credentials: model credentials262 :param query: search query263 :param docs: docs for reranking264 :param score_threshold: score threshold265 :param top_n: top n266 :param user: unique user id267 :return: rerank result268 """269 ```270 271 - Parameters:272 273 - `model` (string) Model name274 275 - `credentials` (object) Credential information276 277 The parameters of credential information are defined by either the `provider_credential_schema` or `model_credential_schema` in the provider's YAML configuration file. Inputs such as `api_key` are included.278 279 - `query` (string) Query request content280 281 - `docs` (array[string]) List of segments to be reranked282 283 - `score_threshold` (float) [optional] Score threshold284 285 - `top_n` (int) [optional] Select the top n segments286 287 - `user` (string) [optional] Unique identifier of the user288 289 This can help the provider monitor and detect abusive behavior.290 291 - Returns:292 293 [RerankResult](#RerankResult) entity.294 295### Speech2text296 297Inherit the `__base.speech2text_model.Speech2TextModel` base class and implement the following interfaces:298 299- Invoke Invocation300 301 ```python302 def _invoke(self, model: str, credentials: dict, file: IO[bytes], user: Optional[str] = None) -> str:303 """304 Invoke large language model305 306 :param model: model name307 :param credentials: model credentials308 :param file: audio file309 :param user: unique user id310 :return: text for given audio file311 """ 312 ```313 314 - Parameters:315 316 - `model` (string) Model name317 318 - `credentials` (object) Credential information319 320 The parameters of credential information are defined by either the `provider_credential_schema` or `model_credential_schema` in the provider's YAML configuration file. Inputs such as `api_key` are included.321 322 - `file` (File) File stream323 324 - `user` (string) [optional] Unique identifier of the user325 326 This can help the provider monitor and detect abusive behavior.327 328 - Returns:329 330 The string after speech-to-text conversion.331 332### Text2speech333 334Inherit the `__base.text2speech_model.Text2SpeechModel` base class and implement the following interfaces:335 336- Invoke Invocation337 338 ```python339 def _invoke(self, model: str, credentials: dict, content_text: str, streaming: bool, user: Optional[str] = None):340 """341 Invoke large language model342 343 :param model: model name344 :param credentials: model credentials345 :param content_text: text content to be translated346 :param streaming: output is streaming347 :param user: unique user id348 :return: translated audio file349 """ 350 ```351 352 - Parameters：353 354 - `model` (string) Model name355 356 - `credentials` (object) Credential information357 358 The parameters of credential information are defined by either the `provider_credential_schema` or `model_credential_schema` in the provider's YAML configuration file. Inputs such as `api_key` are included.359 360 - `content_text` (string) The text content that needs to be converted361 362 - `streaming` (bool) Whether to stream output363 364 - `user` (string) [optional] Unique identifier of the user365 366 This can help the provider monitor and detect abusive behavior.367 368 - Returns：369 370 Text converted speech stream。371 372### Moderation373 374Inherit the `__base.moderation_model.ModerationModel` base class and implement the following interfaces:375 376- Invoke Invocation377 378 ```python379 def _invoke(self, model: str, credentials: dict,380 text: str, user: Optional[str] = None) \381 -> bool:382 """383 Invoke large language model384 385 :param model: model name386 :param credentials: model credentials387 :param text: text to moderate388 :param user: unique user id389 :return: false if text is safe, true otherwise390 """391 ```392 393 - Parameters:394 395 - `model` (string) Model name396 397 - `credentials` (object) Credential information398 399 The parameters of credential information are defined by either the `provider_credential_schema` or `model_credential_schema` in the provider's YAML configuration file. Inputs such as `api_key` are included.400 401 - `text` (string) Text content402 403 - `user` (string) [optional] Unique identifier of the user404 405 This can help the provider monitor and detect abusive behavior.406 407 - Returns:408 409 False indicates that the input text is safe, True indicates otherwise.410 411 412 413## Entities414 415### PromptMessageRole 416 417Message role418 419```python420class PromptMessageRole(Enum):421 """422 Enum class for prompt message.423 """424 SYSTEM = "system"425 USER = "user"426 ASSISTANT = "assistant"427 TOOL = "tool"428```429 430### PromptMessageContentType431 432Message content types, divided into text and image.433 434```python435class PromptMessageContentType(Enum):436 """437 Enum class for prompt message content type.438 """439 TEXT = 'text'440 IMAGE = 'image'441```442 443### PromptMessageContent444 445Message content base class, used only for parameter declaration and cannot be initialized.446 447```python448class PromptMessageContent(BaseModel):449 """450 Model class for prompt message content.451 """452 type: PromptMessageContentType453 data: str454```455 456Currently, two types are supported: text and image. It's possible to simultaneously input text and multiple images.457 458You need to initialize `TextPromptMessageContent` and `ImagePromptMessageContent` separately for input.459 460### TextPromptMessageContent461 462```python463class TextPromptMessageContent(PromptMessageContent):464 """465 Model class for text prompt message content.466 """467 type: PromptMessageContentType = PromptMessageContentType.TEXT468```469 470If inputting a combination of text and images, the text needs to be constructed into this entity as part of the `content` list.471 472### ImagePromptMessageContent473 474```python475class ImagePromptMessageContent(PromptMessageContent):476 """477 Model class for image prompt message content.478 """479 class DETAIL(Enum):480 LOW = 'low'481 HIGH = 'high'482 483 type: PromptMessageContentType = PromptMessageContentType.IMAGE484 detail: DETAIL = DETAIL.LOW # Resolution485```486 487If inputting a combination of text and images, the images need to be constructed into this entity as part of the `content` list.488 489`data` can be either a `url` or a `base64` encoded string of the image.490 491### PromptMessage492 493The base class for all Role message bodies, used only for parameter declaration and cannot be initialized.494 495```python496class PromptMessage(ABC, BaseModel):497 """498 Model class for prompt message.499 """500 role: PromptMessageRole501 content: Optional[str | list[PromptMessageContent]] = None # Supports two types: string and content list. The content list is designed to meet the needs of multimodal inputs. For more details, see the PromptMessageContent explanation.502 name: Optional[str] = None503```504 505### UserPromptMessage506 507UserMessage message body, representing a user's message.508 509```python510class UserPromptMessage(PromptMessage):511 """512 Model class for user prompt message.513 """514 role: PromptMessageRole = PromptMessageRole.USER515```516 517### AssistantPromptMessage518 519Represents a message returned by the model, typically used for `few-shots` or inputting chat history.520 521```python522class AssistantPromptMessage(PromptMessage):523 """524 Model class for assistant prompt message.525 """526 class ToolCall(BaseModel):527 """528 Model class for assistant prompt message tool call.529 """530 class ToolCallFunction(BaseModel):531 """532 Model class for assistant prompt message tool call function.533 """534 name: str # tool name535 arguments: str # tool arguments536 537 id: str # Tool ID, effective only in OpenAI tool calls. It's the unique ID for tool invocation and the same tool can be called multiple times.538 type: str # default: function539 function: ToolCallFunction # tool call information540 541 role: PromptMessageRole = PromptMessageRole.ASSISTANT542 tool_calls: list[ToolCall] = [] # The result of tool invocation in response from the model (returned only when tools are input and the model deems it necessary to invoke a tool).543```544 545Where `tool_calls` are the list of `tool calls` returned by the model after invoking the model with the `tools` input.546 547### SystemPromptMessage548 549Represents system messages, usually used for setting system commands given to the model.550 551```python552class SystemPromptMessage(PromptMessage):553 """554 Model class for system prompt message.555 """556 role: PromptMessageRole = PromptMessageRole.SYSTEM557```558 559### ToolPromptMessage560 561Represents tool messages, used for conveying the results of a tool execution to the model for the next step of processing.562 563```python564class ToolPromptMessage(PromptMessage):565 """566 Model class for tool prompt message.567 """568 role: PromptMessageRole = PromptMessageRole.TOOL569 tool_call_id: str # Tool invocation ID. If OpenAI tool call is not supported, the name of the tool can also be inputted.570```571 572The base class's `content` takes in the results of tool execution.573 574### PromptMessageTool575 576```python577class PromptMessageTool(BaseModel):578 """579 Model class for prompt message tool.580 """581 name: str582 description: str583 parameters: dict584```585 586---587 588### LLMResult589 590```python591class LLMResult(BaseModel):592 """593 Model class for llm result.594 """595 model: str # Actual used modele596 prompt_messages: list[PromptMessage] # prompt messages597 message: AssistantPromptMessage # response message598 usage: LLMUsage # usage info599 system_fingerprint: Optional[str] = None # request fingerprint, refer to OpenAI definition600```601 602### LLMResultChunkDelta603 604In streaming returns, each iteration contains the `delta` entity.605 606```python607class LLMResultChunkDelta(BaseModel):608 """609 Model class for llm result chunk delta.610 """611 index: int612 message: AssistantPromptMessage # response message613 usage: Optional[LLMUsage] = None # usage info614 finish_reason: Optional[str] = None # finish reason, only the last one returns615```616 617### LLMResultChunk618 619Each iteration entity in streaming returns.620 621```python622class LLMResultChunk(BaseModel):623 """624 Model class for llm result chunk.625 """626 model: str # Actual used modele627 prompt_messages: list[PromptMessage] # prompt messages628 system_fingerprint: Optional[str] = None # request fingerprint, refer to OpenAI definition629 delta: LLMResultChunkDelta630```631 632### LLMUsage633 634```python635class LLMUsage(ModelUsage):636 """637 Model class for LLM usage.638 """639 prompt_tokens: int # Tokens used for prompt640 prompt_unit_price: Decimal # Unit price for prompt641 prompt_price_unit: Decimal # Price unit for prompt, i.e., the unit price based on how many tokens642 prompt_price: Decimal # Cost for prompt643 completion_tokens: int # Tokens used for response644 completion_unit_price: Decimal # Unit price for response645 completion_price_unit: Decimal # Price unit for response, i.e., the unit price based on how many tokens646 completion_price: Decimal # Cost for response647 total_tokens: int # Total number of tokens used648 total_price: Decimal # Total cost649 currency: str # Currency unit650 latency: float # Request latency (s)651```652 653---654 655### TextEmbeddingResult656 657```python658class TextEmbeddingResult(BaseModel):659 """660 Model class for text embedding result.661 """662 model: str # Actual model used663 embeddings: list[list[float]] # List of embedding vectors, corresponding to the input texts list664 usage: EmbeddingUsage # Usage information665```666 667### EmbeddingUsage668 669```python670class EmbeddingUsage(ModelUsage):671 """672 Model class for embedding usage.673 """674 tokens: int # Number of tokens used675 total_tokens: int # Total number of tokens used676 unit_price: Decimal # Unit price677 price_unit: Decimal # Price unit, i.e., the unit price based on how many tokens678 total_price: Decimal # Total cost679 currency: str # Currency unit680 latency: float # Request latency (s)681```682 683---684 685### RerankResult686 687```python688class RerankResult(BaseModel):689 """690 Model class for rerank result.691 """692 model: str # Actual model used693 docs: list[RerankDocument] # Reranked document list 694```695 696### RerankDocument697 698```python699class RerankDocument(BaseModel):700 """701 Model class for rerank document.702 """703 index: int # original index704 text: str705 score: float706```707