Underground-Digital/Workflow-Engine

1## Adding a New Provider2 3Providers support three types of model configuration methods:4 5- `predefined-model` Predefined model6 7  This indicates that users only need to configure the unified provider credentials to use the predefined models under the provider.8 9- `customizable-model` Customizable model10 11  Users need to add credential configurations for each model.12 13- `fetch-from-remote` Fetch from remote14 15  This is consistent with the `predefined-model` configuration method. Only unified provider credentials need to be configured, and models are obtained from the provider through credential information.16 17These three configuration methods **can coexist**, meaning a provider can support `predefined-model` + `customizable-model` or `predefined-model` + `fetch-from-remote`, etc. In other words, configuring the unified provider credentials allows the use of predefined and remotely fetched models, and if new models are added, they can be used in addition to the custom models.18 19## Getting Started20 21Adding a new provider starts with determining the English identifier of the provider, such as `anthropic`, and using this identifier to create a `module` in `model_providers`.22 23Under this `module`, we first need to prepare the provider's YAML configuration.24 25### Preparing Provider YAML26 27Here, using `Anthropic` as an example, we preset the provider's basic information, supported model types, configuration methods, and credential rules.28 29```YAML30provider: anthropic  # Provider identifier31label:  # Provider display name, can be set in en_US English and zh_Hans Chinese, zh_Hans will default to en_US if not set.32  en_US: Anthropic33icon_small:  # Small provider icon, stored in the _assets directory under the corresponding provider implementation directory, same language strategy as label34  en_US: icon_s_en.png35icon_large:  # Large provider icon, stored in the _assets directory under the corresponding provider implementation directory, same language strategy as label36  en_US: icon_l_en.png37supported_model_types:  # Supported model types, Anthropic only supports LLM38- llm39configurate_methods:  # Supported configuration methods, Anthropic only supports predefined models40- predefined-model41provider_credential_schema:  # Provider credential rules, as Anthropic only supports predefined models, unified provider credential rules need to be defined42  credential_form_schemas:  # List of credential form items43  - variable: anthropic_api_key  # Credential parameter variable name44    label:  # Display name45      en_US: API Key46    type: secret-input  # Form type, here secret-input represents an encrypted information input box, showing masked information when editing.47    required: true  # Whether required48    placeholder:  # Placeholder information49      zh_Hans: Enter your API Key here50      en_US: Enter your API Key51  - variable: anthropic_api_url52    label:53      en_US: API URL54    type: text-input  # Form type, here text-input represents a text input box55    required: false56    placeholder:57      zh_Hans: Enter your API URL here58      en_US: Enter your API URL59```60 61You can also refer to the YAML configuration information under other provider directories in `model_providers`. The complete YAML rules are available at: [Schema](schema.md#provider).62 63### Implementing Provider Code64 65Providers need to inherit the `__base.model_provider.ModelProvider` base class and implement the `validate_provider_credentials` method for unified provider credential verification. For reference, see [AnthropicProvider](https://github.com/langgenius/dify-runtime/blob/main/lib/model_providers/anthropic/anthropic.py).66> If the provider is the type of `customizable-model`, there is no need to implement the `validate_provider_credentials` method.67 68```python69def validate_provider_credentials(self, credentials: dict) -> None:70    """71    Validate provider credentials72    You can choose any validate_credentials method of model type or implement validate method by yourself,73    such as: get model list api74 75    if validate failed, raise exception76 77    :param credentials: provider credentials, credentials form defined in `provider_credential_schema`.78    """79```80 81Of course, you can also preliminarily reserve the implementation of `validate_provider_credentials` and directly reuse it after the model credential verification method is implemented.82 83---84 85### Adding Models86 87After the provider integration is complete, the next step is to integrate models under the provider.88 89First, we need to determine the type of the model to be integrated and create a `module` for the corresponding model type in the provider's directory.90 91The currently supported model types are as follows:92 93- `llm` Text generation model94- `text_embedding` Text Embedding model95- `rerank` Rerank model96- `speech2text` Speech to text97- `tts` Text to speech98- `moderation` Moderation99 100Continuing with `Anthropic` as an example, since `Anthropic` only supports LLM, we create a `module` named `llm` in `model_providers.anthropic`.101 102For predefined models, we first need to create a YAML file named after the model, such as `claude-2.1.yaml`, under the `llm` `module`.103 104#### Preparing Model YAML105 106```yaml107model: claude-2.1  # Model identifier108# Model display name, can be set in en_US English and zh_Hans Chinese, zh_Hans will default to en_US if not set.109# Alternatively, if the label is not set, use the model identifier content.110label:111  en_US: claude-2.1112model_type: llm  # Model type, claude-2.1 is an LLM113features:  # Supported features, agent-thought for Agent reasoning, vision for image understanding114- agent-thought115model_properties:  # Model properties116  mode: chat  # LLM mode, complete for text completion model, chat for dialogue model117  context_size: 200000  # Maximum supported context size118parameter_rules:  # Model invocation parameter rules, only required for LLM119- name: temperature  # Invocation parameter variable name120  # Default preset with 5 variable content configuration templates: temperature/top_p/max_tokens/presence_penalty/frequency_penalty121  # Directly set the template variable name in use_template, which will use the default configuration in entities.defaults.PARAMETER_RULE_TEMPLATE122  # If additional configuration parameters are set, they will override the default configuration123  use_template: temperature124- name: top_p125  use_template: top_p126- name: top_k127  label:  # Invocation parameter display name128    zh_Hans: Sampling quantity129    en_US: Top k130  type: int  # Parameter type, supports float/int/string/boolean131  help:  # Help information, describing the role of the parameter132    zh_Hans: Only sample from the top K options for each subsequent token.133    en_US: Only sample from the top K options for each subsequent token.134  required: false  # Whether required, can be left unset135- name: max_tokens_to_sample136  use_template: max_tokens137  default: 4096  # Default parameter value138  min: 1  # Minimum parameter value, only applicable for float/int139  max: 4096  # Maximum parameter value, only applicable for float/int140pricing:  # Pricing information141  input: '8.00'  # Input price, i.e., Prompt price142  output: '24.00'  # Output price, i.e., returned content price143  unit: '0.000001'  # Pricing unit, i.e., the above prices are per 100K144  currency: USD  # Currency145```146 147It is recommended to prepare all model configurations before starting the implementation of the model code.148 149Similarly, you can also refer to the YAML configuration information for corresponding model types of other providers in the `model_providers` directory. The complete YAML rules can be found at: [Schema](schema.md#AIModel).150 151#### Implementing Model Invocation Code152 153Next, you need to create a python file named `llm.py` under the `llm` `module` to write the implementation code.154 155In `llm.py`, create an Anthropic LLM class, which we name `AnthropicLargeLanguageModel` (arbitrarily), inheriting the `__base.large_language_model.LargeLanguageModel` base class, and implement the following methods:156 157- LLM Invocation158 159  Implement the core method for LLM invocation, which can support both streaming and synchronous returns.160 161  ```python162  def _invoke(self, model: str, credentials: dict,163              prompt_messages: list[PromptMessage], model_parameters: dict,164              tools: Optional[list[PromptMessageTool]] = None, stop: Optional[list[str]] = None,165              stream: bool = True, user: Optional[str] = None) \166          -> Union[LLMResult, Generator]:167      """168      Invoke large language model169  170      :param model: model name171      :param credentials: model credentials172      :param prompt_messages: prompt messages173      :param model_parameters: model parameters174      :param tools: tools for tool calling175      :param stop: stop words176      :param stream: is stream response177      :param user: unique user id178      :return: full response or stream response chunk generator result179      """180  ```181 182- Pre-calculating Input Tokens183 184  If the model does not provide a pre-calculated tokens interface, you can directly return 0.185 186  ```python187  def get_num_tokens(self, model: str, credentials: dict, prompt_messages: list[PromptMessage],188                   tools: Optional[list[PromptMessageTool]] = None) -> int:189    """190    Get number of tokens for given prompt messages191 192    :param model: model name193    :param credentials: model credentials194    :param prompt_messages: prompt messages195    :param tools: tools for tool calling196    :return:197    """198  ```199 200- Model Credential Verification201 202  Similar to provider credential verification, this step involves verification for an individual model.203 204  ```python205  def validate_credentials(self, model: str, credentials: dict) -> None:206      """207      Validate model credentials208  209      :param model: model name210      :param credentials: model credentials211      :return:212      """213  ```214 215- Invocation Error Mapping Table216 217  When there is an exception in model invocation, it needs to be mapped to the `InvokeError` type specified by Runtime. This facilitates Dify's ability to handle different errors with appropriate follow-up actions.218 219  Runtime Errors:220 221  - `InvokeConnectionError` Invocation connection error222  - `InvokeServerUnavailableError` Invocation service provider unavailable223  - `InvokeRateLimitError` Invocation reached rate limit224  - `InvokeAuthorizationError` Invocation authorization failure225  - `InvokeBadRequestError` Invocation parameter error226 227  ```python228  @property229  def _invoke_error_mapping(self) -> dict[type[InvokeError], list[type[Exception]]]:230      """231      Map model invoke error to unified error232      The key is the error type thrown to the caller233      The value is the error type thrown by the model,234      which needs to be converted into a unified error type for the caller.235  236      :return: Invoke error mapping237      """238  ```239 240For details on the interface methods, see: [Interfaces](interfaces.md). For specific implementations, refer to: [llm.py](https://github.com/langgenius/dify-runtime/blob/main/lib/model_providers/anthropic/llm/llm.py).241 242### Testing243 244To ensure the availability of integrated providers/models, each method written needs corresponding integration test code in the `tests` directory.245 246Continuing with `Anthropic` as an example:247 248Before writing test code, you need to first add the necessary credential environment variables for the test provider in `.env.example`, such as: `ANTHROPIC_API_KEY`.249 250Before execution, copy `.env.example` to `.env` and then execute.251 252#### Writing Test Code253 254Create a `module` with the same name as the provider in the `tests` directory: `anthropic`, and continue to create `test_provider.py` and test py files for the corresponding model types within this module, as shown below:255 256```shell257.258├── __init__.py259├── anthropic260│   ├── __init__.py261│   ├── test_llm.py       # LLM Testing262│   └── test_provider.py  # Provider Testing263```264 265Write test code for all the various cases implemented above and submit the code after passing the tests.266