Underground-Digital/Workflow-Engine

1# Advanced Tool Integration2 3Before starting with this advanced guide, please make sure you have a basic understanding of the tool integration process in Dify. Check out [Quick Integration](./tool_scale_out.md) for a quick runthrough.4 5## Tool Interface6 7We have defined a series of helper methods in the `Tool` class to help developers quickly build more complex tools.8 9### Message Return10 11Dify supports various message types such as `text`, `link`, `json`, `image`, and `file BLOB`. You can return different types of messages to the LLM and users through the following interfaces.12 13Please note, some parameters in the following interfaces will be introduced in later sections.14 15#### Image URL16You only need to pass the URL of the image, and Dify will automatically download the image and return it to the user.17 18```python19    def create_image_message(self, image: str, save_as: str = '') -> ToolInvokeMessage:20        """21            create an image message22 23            :param image: the url of the image24            :return: the image message25        """26```27 28#### Link29If you need to return a link, you can use the following interface.30 31```python32    def create_link_message(self, link: str, save_as: str = '') -> ToolInvokeMessage:33        """34            create a link message35 36            :param link: the url of the link37            :return: the link message38        """39```40 41#### Text42If you need to return a text message, you can use the following interface.43 44```python45    def create_text_message(self, text: str, save_as: str = '') -> ToolInvokeMessage:46        """47            create a text message48 49            :param text: the text of the message50            :return: the text message51        """52```53 54#### File BLOB55If you need to return the raw data of a file, such as images, audio, video, PPT, Word, Excel, etc., you can use the following interface.56 57- `blob` The raw data of the file, of bytes type58- `meta` The metadata of the file, if you know the type of the file, it is best to pass a `mime_type`, otherwise Dify will use `octet/stream` as the default type59 60```python61    def create_blob_message(self, blob: bytes, meta: dict = None, save_as: str = '') -> ToolInvokeMessage:62        """63            create a blob message64 65            :param blob: the blob66            :return: the blob message67        """68```69 70#### JSON71If you need to return a formatted JSON, you can use the following interface. This is commonly used for data transmission between nodes in a workflow, of course, in agent mode, most LLM are also able to read and understand JSON.72 73- `object` A Python dictionary object will be automatically serialized into JSON74 75```python76    def create_json_message(self, object: dict) -> ToolInvokeMessage:77        """78            create a json message79        """80```81 82### Shortcut Tools83 84In large model applications, we have two common needs:85- First, summarize a long text in advance, and then pass the summary content to the LLM to prevent the original text from being too long for the LLM to handle86- The content obtained by the tool is a link, and the web page information needs to be crawled before it can be returned to the LLM87 88To help developers quickly implement these two needs, we provide the following two shortcut tools.89 90#### Text Summary Tool91 92This tool takes in an user_id and the text to be summarized, and returns the summarized text. Dify will use the default model of the current workspace to summarize the long text.93 94```python95    def summary(self, user_id: str, content: str) -> str:96        """97            summary the content98 99            :param user_id: the user id100            :param content: the content101            :return: the summary102        """103```104 105#### Web Page Crawling Tool106 107This tool takes in web page link to be crawled and a user_agent (which can be empty), and returns a string containing the information of the web page. The `user_agent` is an optional parameter that can be used to identify the tool. If not passed, Dify will use the default `user_agent`.108 109```python110    def get_url(self, url: str, user_agent: str = None) -> str:111        """112            get url113        """ the crawled result114```115 116### Variable Pool117 118We have introduced a variable pool in `Tool` to store variables, files, etc. generated during the tool's operation. These variables can be used by other tools during the tool's operation.119 120Next, we will use `DallE3` and `Vectorizer.AI` as examples to introduce how to use the variable pool.121 122- `DallE3` is an image generation tool that can generate images based on text. Here, we will let `DallE3` generate a logo for a coffee shop123- `Vectorizer.AI` is a vector image conversion tool that can convert images into vector images, so that the images can be infinitely enlarged without distortion. Here, we will convert the PNG icon generated by `DallE3` into a vector image, so that it can be truly used by designers.124 125#### DallE3126First, we use DallE3. After creating the image, we save the image to the variable pool. The code is as follows:127 128```python129from typing import Any, Dict, List, Union130from core.tools.entities.tool_entities import ToolInvokeMessage131from core.tools.tool.builtin_tool import BuiltinTool132 133from base64 import b64decode134 135from openai import OpenAI136 137class DallE3Tool(BuiltinTool):138    def _invoke(self, 139                user_id: str, 140               tool_parameters: Dict[str, Any], 141        ) -> Union[ToolInvokeMessage, List[ToolInvokeMessage]]:142        """143            invoke tools144        """145        client = OpenAI(146            api_key=self.runtime.credentials['openai_api_key'],147        )148 149        # prompt150        prompt = tool_parameters.get('prompt', '')151        if not prompt:152            return self.create_text_message('Please input prompt')153 154        # call openapi dalle3155        response = client.images.generate(156            prompt=prompt, model='dall-e-3',157            size='1024x1024', n=1, style='vivid', quality='standard',158            response_format='b64_json'159        )160 161        result = []162        for image in response.data:163            # Save all images to the variable pool through the save_as parameter. The variable name is self.VARIABLE_KEY.IMAGE.value. If new images are generated later, they will overwrite the previous images.164            result.append(self.create_blob_message(blob=b64decode(image.b64_json), 165                                                   meta={ 'mime_type': 'image/png' },166                                                    save_as=self.VARIABLE_KEY.IMAGE.value))167 168        return result169```170 171Note that we used `self.VARIABLE_KEY.IMAGE.value` as the variable name of the image. In order for developers' tools to cooperate with each other, we defined this `KEY`. You can use it freely, or you can choose not to use this `KEY`. Passing a custom KEY is also acceptable.172 173#### Vectorizer.AI174Next, we use Vectorizer.AI to convert the PNG icon generated by DallE3 into a vector image. Let's go through the functions we defined here. The code is as follows:175 176```python177from core.tools.tool.builtin_tool import BuiltinTool178from core.tools.entities.tool_entities import ToolInvokeMessage, ToolParameter179from core.tools.errors import ToolProviderCredentialValidationError180 181from typing import Any, Dict, List, Union182from httpx import post183from base64 import b64decode184 185class VectorizerTool(BuiltinTool):186    def _invoke(self, user_id: str, tool_parameters: Dict[str, Any]) \187        -> Union[ToolInvokeMessage, List[ToolInvokeMessage]]:188        """189        Tool invocation, the image variable name needs to be passed in from here, so that we can get the image from the variable pool190        """191        192    193    def get_runtime_parameters(self) -> List[ToolParameter]:194        """195        Override the tool parameter list, we can dynamically generate the parameter list based on the actual situation in the current variable pool, so that the LLM can generate the form based on the parameter list196        """197        198    199    def is_tool_available(self) -> bool:200        """201        Whether the current tool is available, if there is no image in the current variable pool, then we don't need to display this tool, just return False here202        """     203```204 205Next, let's implement these three functions206 207```python208from core.tools.tool.builtin_tool import BuiltinTool209from core.tools.entities.tool_entities import ToolInvokeMessage, ToolParameter210from core.tools.errors import ToolProviderCredentialValidationError211 212from typing import Any, Dict, List, Union213from httpx import post214from base64 import b64decode215 216class VectorizerTool(BuiltinTool):217    def _invoke(self, user_id: str, tool_parameters: Dict[str, Any]) \218        -> Union[ToolInvokeMessage, List[ToolInvokeMessage]]:219        """220            invoke tools221        """222        api_key_name = self.runtime.credentials.get('api_key_name', None)223        api_key_value = self.runtime.credentials.get('api_key_value', None)224 225        if not api_key_name or not api_key_value:226            raise ToolProviderCredentialValidationError('Please input api key name and value')227 228        # Get image_id, the definition of image_id can be found in get_runtime_parameters229        image_id = tool_parameters.get('image_id', '')230        if not image_id:231            return self.create_text_message('Please input image id')232 233        # Get the image generated by DallE from the variable pool234        image_binary = self.get_variable_file(self.VARIABLE_KEY.IMAGE)235        if not image_binary:236            return self.create_text_message('Image not found, please request user to generate image firstly.')237 238        # Generate vector image239        response = post(240            'https://vectorizer.ai/api/v1/vectorize',241            files={ 'image': image_binary },242            data={ 'mode': 'test' },243            auth=(api_key_name, api_key_value), 244            timeout=30245        )246 247        if response.status_code != 200:248            raise Exception(response.text)249        250        return [251            self.create_text_message('the vectorized svg is saved as an image.'),252            self.create_blob_message(blob=response.content,253                                    meta={'mime_type': 'image/svg+xml'})254        ]255    256    def get_runtime_parameters(self) -> List[ToolParameter]:257        """258        override the runtime parameters259        """260        # Here, we override the tool parameter list, define the image_id, and set its option list to all images in the current variable pool. The configuration here is consistent with the configuration in yaml.261        return [262            ToolParameter.get_simple_instance(263                name='image_id',264                llm_description=f'the image id that you want to vectorize, \265                    and the image id should be specified in \266                        {[i.name for i in self.list_default_image_variables()]}',267                type=ToolParameter.ToolParameterType.SELECT,268                required=True,269                options=[i.name for i in self.list_default_image_variables()]270            )271        ]272    273    def is_tool_available(self) -> bool:274        # Only when there are images in the variable pool, the LLM needs to use this tool275        return len(self.list_default_image_variables()) > 0276```277 278It's worth noting that we didn't actually use `image_id` here. We assumed that there must be an image in the default variable pool when calling this tool, so we directly used `image_binary = self.get_variable_file(self.VARIABLE_KEY.IMAGE)` to get the image. In cases where the model's capabilities are weak, we recommend developers to do the same, which can effectively improve fault tolerance and avoid the model passing incorrect parameters.