MultiTransformer/autogen-tutorials
22
1{2 "cells": [3 {4 "attachments": {},5 "cell_type": "markdown",6 "metadata": {},7 "source": [8 "<a href=\"https://colab.research.google.com/github/microsoft/autogen/blob/main/notebook/agentchat_RetrieveChat.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"9 ]10 },11 {12 "attachments": {},13 "cell_type": "markdown",14 "metadata": {},15 "source": [16 "<a id=\"toc\"></a>\n",17 "# Auto Generated Agent Chat: Using RetrieveChat for Retrieve Augmented Code Generation and Question Answering\n",18 "\n",19 "AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framwork allows tool use and human participance through multi-agent conversation.\n",20 "Please find documentation about this feature [here](https://microsoft.github.io/autogen/docs/Use-Cases/agent_chat).\n",21 "\n",22 "RetrieveChat is a convesational system for retrieve augmented code generation and question answering. In this notebook, we demonstrate how to utilize RetrieveChat to generate code and answer questions based on customized documentations that are not present in the LLM's training dataset. RetrieveChat uses the `RetrieveAssistantAgent` and `RetrieveUserProxyAgent`, which is similar to the usage of `AssistantAgent` and `UserProxyAgent` in other notebooks (e.g., [Automated Task Solving with Code Generation, Execution & Debugging](https://github.com/microsoft/autogen/blob/main/notebook/agentchat_auto_feedback_from_code_execution.ipynb)). Essentially, `RetrieveAssistantAgent` and `RetrieveUserProxyAgent` implement a different auto-reply mechanism corresponding to the RetrieveChat prompts.\n",23 "\n",24 "## Table of Contents\n",25 "We'll demonstrates six examples of using RetrieveChat for code generation and question answering:\n",26 "\n",27 "[Example 1: Generate code based off docstrings w/o human feedback](#example-1)\n",28 "\n",29 "[Example 2: Answer a question based off docstrings w/o human feedback](#example-2)\n",30 "\n",31 "[Example 3: Generate code based off docstrings w/ human feedback](#example-3)\n",32 "\n",33 "[Example 4: Answer a question based off docstrings w/ human feedback](#example-4)\n",34 "\n",35 "[Example 5: Solve comprehensive QA problems with RetrieveChat's unique feature `Update Context`](#example-5)\n",36 "\n",37 "[Example 6: Solve comprehensive QA problems with customized prompt and few-shot learning](#example-6)\n",38 "\n",39 "\n",40 "\n",41 "## Requirements\n",42 "\n",43 "AutoGen requires `Python>=3.8`. To run this notebook example, please install the [retrievechat] option.\n",44 "```bash\n",45 "pip install \"pyautogen[retrievechat]\" \"flaml[automl]\"\n",46 "```"47 ]48 },49 {50 "cell_type": "code",51 "execution_count": 1,52 "metadata": {},53 "outputs": [],54 "source": [55 "# %pip install \"pyautogen[retrievechat]~=0.1.2\" \"flaml[automl]\""56 ]57 },58 {59 "attachments": {},60 "cell_type": "markdown",61 "metadata": {},62 "source": [63 "## Set your API Endpoint\n",64 "\n",65 "The [`config_list_from_json`](https://microsoft.github.io/autogen/docs/reference/oai/openai_utils#config_list_from_json) function loads a list of configurations from an environment variable or a json file.\n"66 ]67 },68 {69 "cell_type": "code",70 "execution_count": 1,71 "metadata": {},72 "outputs": [73 {74 "name": "stdout",75 "output_type": "stream",76 "text": [77 "models to use: ['gpt-4']\n"78 ]79 }80 ],81 "source": [82 "import autogen\n",83 "\n",84 "config_list = autogen.config_list_from_json(\n",85 " env_or_file=\"OAI_CONFIG_LIST\",\n",86 " file_location=\".\",\n",87 " filter_dict={\n",88 " \"model\": {\n",89 " \"gpt-4\",\n",90 " \"gpt4\",\n",91 " \"gpt-4-32k\",\n",92 " \"gpt-4-32k-0314\",\n",93 " \"gpt-35-turbo\",\n",94 " \"gpt-3.5-turbo\",\n",95 " }\n",96 " },\n",97 ")\n",98 "\n",99 "assert len(config_list) > 0\n",100 "print(\"models to use: \", [config_list[i][\"model\"] for i in range(len(config_list))])"101 ]102 },103 {104 "attachments": {},105 "cell_type": "markdown",106 "metadata": {},107 "source": [108 "It first looks for environment variable \"OAI_CONFIG_LIST\" which needs to be a valid json string. If that variable is not found, it then looks for a json file named \"OAI_CONFIG_LIST\". It filters the configs by models (you can filter by other keys as well). Only the gpt-4 and gpt-3.5-turbo models are kept in the list based on the filter condition.\n",109 "\n",110 "The config list looks like the following:\n",111 "```python\n",112 "config_list = [\n",113 " {\n",114 " 'model': 'gpt-4',\n",115 " 'api_key': '<your OpenAI API key here>',\n",116 " },\n",117 " {\n",118 " 'model': 'gpt-4',\n",119 " 'api_key': '<your Azure OpenAI API key here>',\n",120 " 'api_base': '<your Azure OpenAI API base here>',\n",121 " 'api_type': 'azure',\n",122 " 'api_version': '2023-06-01-preview',\n",123 " },\n",124 " {\n",125 " 'model': 'gpt-3.5-turbo',\n",126 " 'api_key': '<your Azure OpenAI API key here>',\n",127 " 'api_base': '<your Azure OpenAI API base here>',\n",128 " 'api_type': 'azure',\n",129 " 'api_version': '2023-06-01-preview',\n",130 " },\n",131 "]\n",132 "```\n",133 "\n",134 "If you open this notebook in colab, you can upload your files by clicking the file icon on the left panel and then choose \"upload file\" icon.\n",135 "\n",136 "You can set the value of config_list in other ways you prefer, e.g., loading from a YAML file."137 ]138 },139 {140 "attachments": {},141 "cell_type": "markdown",142 "metadata": {},143 "source": [144 "## Construct agents for RetrieveChat\n",145 "\n",146 "We start by initialzing the `RetrieveAssistantAgent` and `RetrieveUserProxyAgent`. The system message needs to be set to \"You are a helpful assistant.\" for RetrieveAssistantAgent. The detailed instructions are given in the user message. Later we will use the `RetrieveUserProxyAgent.generate_init_prompt` to combine the instructions and a retrieval augmented generation task for an initial prompt to be sent to the LLM assistant."147 ]148 },149 {150 "cell_type": "code",151 "execution_count": 2,152 "metadata": {},153 "outputs": [],154 "source": [155 "from autogen.agentchat.contrib.retrieve_assistant_agent import RetrieveAssistantAgent\n",156 "from autogen.agentchat.contrib.retrieve_user_proxy_agent import RetrieveUserProxyAgent\n",157 "import chromadb\n",158 "\n",159 "autogen.ChatCompletion.start_logging()\n",160 "\n",161 "# 1. create an RetrieveAssistantAgent instance named \"assistant\"\n",162 "assistant = RetrieveAssistantAgent(\n",163 " name=\"assistant\", \n",164 " system_message=\"You are a helpful assistant.\",\n",165 " llm_config={\n",166 " \"request_timeout\": 600,\n",167 " \"seed\": 42,\n",168 " \"config_list\": config_list,\n",169 " },\n",170 ")\n",171 "\n",172 "# 2. create the RetrieveUserProxyAgent instance named \"ragproxyagent\"\n",173 "# By default, the human_input_mode is \"ALWAYS\", which means the agent will ask for human input at every step. We set it to \"NEVER\" here.\n",174 "# `docs_path` is the path to the docs directory. By default, it is set to \"./docs\". Here we generated the documentations from FLAML's docstrings.\n",175 "# Navigate to the website folder and run `pydoc-markdown` and it will generate folder `reference` under `website/docs`.\n",176 "# `task` indicates the kind of task we're working on. In this example, it's a `code` task.\n",177 "# `chunk_token_size` is the chunk token size for the retrieve chat. By default, it is set to `max_tokens * 0.6`, here we set it to 2000.\n",178 "ragproxyagent = RetrieveUserProxyAgent(\n",179 " name=\"ragproxyagent\",\n",180 " human_input_mode=\"NEVER\",\n",181 " max_consecutive_auto_reply=10,\n",182 " retrieve_config={\n",183 " \"task\": \"code\",\n",184 " \"docs_path\": \"../website/docs/reference\",\n",185 " \"chunk_token_size\": 2000,\n",186 " \"model\": config_list[0][\"model\"],\n",187 " \"client\": chromadb.PersistentClient(path=\"/tmp/chromadb\"),\n",188 " \"embedding_model\": \"all-mpnet-base-v2\",\n",189 " },\n",190 ")"191 ]192 },193 {194 "attachments": {},195 "cell_type": "markdown",196 "metadata": {},197 "source": [198 "<a id=\"example-1\"></a>\n",199 "### Example 1\n",200 "\n",201 "[back to top](#toc)\n",202 "\n",203 "Use RetrieveChat to help generate sample code and automatically run the code and fix errors if there is any.\n",204 "\n",205 "Problem: Which API should I use if I want to use FLAML for a classification task and I want to train the model in 30 seconds. Use spark to parallel the training. Force cancel jobs if time limit is reached."206 ]207 },208 {209 "cell_type": "code",210 "execution_count": 6,211 "metadata": {},212 "outputs": [213 {214 "name": "stdout",215 "output_type": "stream",216 "text": [217 "doc_ids: [['doc_36', 'doc_40', 'doc_15', 'doc_22', 'doc_16', 'doc_51', 'doc_44', 'doc_41', 'doc_45', 'doc_14', 'doc_0', 'doc_37', 'doc_38', 'doc_9']]\n",218 "\u001b[32mAdding doc_id doc_36 to context.\u001b[0m\n",219 "\u001b[32mAdding doc_id doc_40 to context.\u001b[0m\n",220 "\u001b[32mAdding doc_id doc_15 to context.\u001b[0m\n",221 "\u001b[33mragproxyagent\u001b[0m (to assistant):\n",222 "\n",223 "You're a retrieve augmented coding assistant. You answer user's questions based on your own knowledge and the\n",224 "context provided by the user.\n",225 "If you can't answer the question with or without the current context, you should reply exactly `UPDATE CONTEXT`.\n",226 "For code generation, you must obey the following rules:\n",227 "Rule 1. You MUST NOT install any packages because all the packages needed are already installed.\n",228 "Rule 2. You must follow the formats below to write your code:\n",229 "```language\n",230 "# your code\n",231 "```\n",232 "\n",233 "User's question is: How can I use FLAML to perform a classification task and use spark to do parallel training. Train 30 seconds and force cancel jobs if time limit is reached.\n",234 "\n",235 "Context is: \n",236 "- `seed` - int or None, default=None | The random seed for hpo.\n",237 "- `n_concurrent_trials` - [Experimental] int, default=1 | The number of\n",238 " concurrent trials. When n_concurrent_trials > 1, flaml performes\n",239 " [parallel tuning](../../Use-Cases/Task-Oriented-AutoML#parallel-tuning)\n",240 " and installation of ray or spark is required: `pip install flaml[ray]`\n",241 " or `pip install flaml[spark]`. Please check\n",242 " [here](https://spark.apache.org/docs/latest/api/python/getting_started/install.html)\n",243 " for more details about installing Spark.\n",244 "- `keep_search_state` - boolean, default=False | Whether to keep data needed\n",245 " for model search after fit(). By default the state is deleted for\n",246 " space saving.\n",247 "- `preserve_checkpoint` - boolean, default=True | Whether to preserve the saved checkpoint\n",248 " on disk when deleting automl. By default the checkpoint is preserved.\n",249 "- `early_stop` - boolean, default=False | Whether to stop early if the\n",250 " search is considered to converge.\n",251 "- `force_cancel` - boolean, default=False | Whether to forcely cancel Spark jobs if the\n",252 " search time exceeded the time budget.\n",253 "- `append_log` - boolean, default=False | Whetehr to directly append the log\n",254 " records to the input log file if it exists.\n",255 "- `auto_augment` - boolean, default=True | Whether to automatically\n",256 " augment rare classes.\n",257 "- `min_sample_size` - int, default=MIN_SAMPLE_TRAIN | the minimal sample\n",258 " size when sample=True.\n",259 "- `use_ray` - boolean or dict.\n",260 " If boolean: default=False | Whether to use ray to run the training\n",261 " in separate processes. This can be used to prevent OOM for large\n",262 " datasets, but will incur more overhead in time.\n",263 " If dict: the dict contains the keywords arguments to be passed to\n",264 " [ray.tune.run](https://docs.ray.io/en/latest/tune/api_docs/execution.html).\n",265 "- `use_spark` - boolean, default=False | Whether to use spark to run the training\n",266 " in parallel spark jobs. This can be used to accelerate training on large models\n",267 " and large datasets, but will incur more overhead in time and thus slow down\n",268 " training in some cases. GPU training is not supported yet when use_spark is True.\n",269 " For Spark clusters, by default, we will launch one trial per executor. However,\n",270 " sometimes we want to launch more trials than the number of executors (e.g., local mode).\n",271 " In this case, we can set the environment variable `FLAML_MAX_CONCURRENT` to override\n",272 " the detected `num_executors`. The final number of concurrent trials will be the minimum\n",273 " of `n_concurrent_trials` and `num_executors`.\n",274 "- `free_mem_ratio` - float between 0 and 1, default=0. The free memory ratio to keep during training.\n",275 "- `metric_constraints` - list, default=[] | The list of metric constraints.\n",276 " Each element in this list is a 3-tuple, which shall be expressed\n",277 " in the following format: the first element of the 3-tuple is the name of the\n",278 " metric, the second element is the inequality sign chosen from \">=\" and \"<=\",\n",279 " and the third element is the constraint value. E.g., `('val_loss', '<=', 0.1)`.\n",280 " Note that all the metric names in metric_constraints need to be reported via\n",281 " the metrics_to_log dictionary returned by a customized metric function.\n",282 " The customized metric function shall be provided via the `metric` key word\n",283 " argument of the fit() function or the automl constructor.\n",284 " Find an example in the 4th constraint type in this [doc](../../Use-Cases/Task-Oriented-AutoML#constraint).\n",285 " If `pred_time_limit` is provided as one of keyword arguments to fit() function or\n",286 " the automl constructor, flaml will automatically (and under the hood)\n",287 " add it as an additional element in the metric_constraints. Essentially 'pred_time_limit'\n",288 " specifies a constraint about the prediction latency constraint in seconds.\n",289 "- `custom_hp` - dict, default=None | The custom search space specified by user.\n",290 " It is a nested dict with keys being the estimator names, and values being dicts\n",291 " per estimator search space. In the per estimator search space dict,\n",292 " the keys are the hyperparameter names, and values are dicts of info (\"domain\",\n",293 " \"init_value\", and \"low_cost_init_value\") about the search space associated with\n",294 " the hyperparameter (i.e., per hyperparameter search space dict). When custom_hp\n",295 " is provided, the built-in search space which is also a nested dict of per estimator\n",296 " search space dict, will be updated with custom_hp. Note that during this nested dict update,\n",297 " the per hyperparameter search space dicts will be replaced (instead of updated) by the ones\n",298 " provided in custom_hp. Note that the value for \"domain\" can either be a constant\n",299 " or a sample.Domain object.\n",300 " e.g.,\n",301 " \n",302 "```python\n",303 "custom_hp = {\n",304 " \"transformer_ms\": {\n",305 " \"model_path\": {\n",306 " \"domain\": \"albert-base-v2\",\n",307 " },\n",308 " \"learning_rate\": {\n",309 " \"domain\": tune.choice([1e-4, 1e-5]),\n",310 " }\n",311 " }\n",312 " }\n",313 "```\n",314 "- `skip_transform` - boolean, default=False | Whether to pre-process data prior to modeling.\n",315 "- `fit_kwargs_by_estimator` - dict, default=None | The user specified keywords arguments, grouped by estimator name.\n",316 " e.g.,\n",317 " \n",318 "```python\n",319 "fit_kwargs_by_estimator = {\n",320 " \"transformer\": {\n",321 " \"output_dir\": \"test/data/output/\",\n",322 " \"fp16\": False,\n",323 " }\n",324 "}\n",325 "```\n",326 "- `mlflow_logging` - boolean, default=True | Whether to log the training results to mlflow.\n",327 " This requires mlflow to be installed and to have an active mlflow run.\n",328 " FLAML will create nested runs.\n",329 "\n",330 "#### config\\_history\n",331 "\n",332 "```python\n",333 "@property\n",334 "def config_history() -> dict\n",335 "```\n",336 "\n",337 "A dictionary of iter->(estimator, config, time),\n",338 "storing the best estimator, config, and the time when the best\n",339 "model is updated each time.\n",340 "\n",341 "#### model\n",342 "\n",343 "```python\n",344 "@property\n",345 "def model()\n",346 "```\n",347 "\n",348 "An object with `predict()` and `predict_proba()` method (for\n",349 "classification), storing the best trained model.\n",350 "\n",351 "#### best\\_model\\_for\\_estimator\n",352 "\n",353 "```python\n",354 "def best_model_for_estimator(estimator_name: str)\n",355 "```\n",356 "\n",357 "Return the best model found for a particular estimator.\n",358 "\n",359 "**Arguments**:\n",360 "\n",361 "- `estimator_name` - a str of the estimator's name.\n",362 " \n",363 "\n",364 "**Returns**:\n",365 "\n",366 " An object storing the best model for estimator_name.\n",367 " If `model_history` was set to False during fit(), then the returned model\n",368 " is untrained unless estimator_name is the best estimator.\n",369 " If `model_history` was set to True, then the returned model is trained.\n",370 "\n",371 "#### best\\_estimator\n",372 "\n",373 "```python\n",374 "@property\n",375 "def best_estimator()\n",376 "```\n",377 "\n",378 "A string indicating the best estimator found.\n",379 "\n",380 "#### best\\_iteration\n",381 "\n",382 "```python\n",383 "@property\n",384 "def best_iteration()\n",385 "```\n",386 "\n",387 "An integer of the iteration number where the best\n",388 "config is found.\n",389 "\n",390 "#### best\\_config\n",391 "\n",392 "```python\n",393 "@property\n",394 "def best_config()\n",395 "```\n",396 "\n",397 "A dictionary of the best configuration.\n",398 "\n",399 "#### best\\_config\\_per\\_estimator\n",400 "\n",401 "```python\n",402 "@property\n",403 "def best_config_per_estimator()\n",404 "```\n",405 "\n",406 "A dictionary of all estimators' best configuration.\n",407 "\n",408 "#### best\\_loss\\_per\\_estimator\n",409 "\n",410 "```python\n",411 "@property\n",412 "def best_loss_per_estimator()\n",413 "```\n",414 "\n",415 "A dictionary of all estimators' best loss.\n",416 "\n",417 "#### best\\_loss\n",418 "\n",419 "```python\n",420 "@property\n",421 "def best_loss()\n",422 "```\n",423 "\n",424 "A float of the best loss found.\n",425 "\n",426 "#### best\\_result\n",427 "\n",428 "```python\n",429 "@property\n",430 "def best_result()\n",431 "```\n",432 "\n",433 "Result dictionary for model trained with the best config.\n",434 "\n",435 "#### metrics\\_for\\_best\\_config\n",436 "\n",437 "```python\n",438 "@property\n",439 "def metrics_for_best_config()\n",440 "```\n",441 "\n",442 "Returns a float of the best loss, and a dictionary of the auxiliary metrics to log\n",443 "associated with the best config. These two objects correspond to the returned\n",444 "objects by the customized metric function for the config with the best loss.\n",445 "\n",446 "#### best\\_config\\_train\\_time\n",447 " \n",448 "- `seed` - int or None, default=None | The random seed for hpo.\n",449 "- `n_concurrent_trials` - [Experimental] int, default=1 | The number of\n",450 " concurrent trials. When n_concurrent_trials > 1, flaml performes\n",451 " [parallel tuning](../../Use-Cases/Task-Oriented-AutoML#parallel-tuning)\n",452 " and installation of ray or spark is required: `pip install flaml[ray]`\n",453 " or `pip install flaml[spark]`. Please check\n",454 " [here](https://spark.apache.org/docs/latest/api/python/getting_started/install.html)\n",455 " for more details about installing Spark.\n",456 "- `keep_search_state` - boolean, default=False | Whether to keep data needed\n",457 " for model search after fit(). By default the state is deleted for\n",458 " space saving.\n",459 "- `preserve_checkpoint` - boolean, default=True | Whether to preserve the saved checkpoint\n",460 " on disk when deleting automl. By default the checkpoint is preserved.\n",461 "- `early_stop` - boolean, default=False | Whether to stop early if the\n",462 " search is considered to converge.\n",463 "- `force_cancel` - boolean, default=False | Whether to forcely cancel the PySpark job if overtime.\n",464 "- `append_log` - boolean, default=False | Whetehr to directly append the log\n",465 " records to the input log file if it exists.\n",466 "- `auto_augment` - boolean, default=True | Whether to automatically\n",467 " augment rare classes.\n",468 "- `min_sample_size` - int, default=MIN_SAMPLE_TRAIN | the minimal sample\n",469 " size when sample=True.\n",470 "- `use_ray` - boolean or dict.\n",471 " If boolean: default=False | Whether to use ray to run the training\n",472 " in separate processes. This can be used to prevent OOM for large\n",473 " datasets, but will incur more overhead in time.\n",474 " If dict: the dict contains the keywords arguments to be passed to\n",475 " [ray.tune.run](https://docs.ray.io/en/latest/tune/api_docs/execution.html).\n",476 "- `use_spark` - boolean, default=False | Whether to use spark to run the training\n",477 " in parallel spark jobs. This can be used to accelerate training on large models\n",478 " and large datasets, but will incur more overhead in time and thus slow down\n",479 " training in some cases.\n",480 "- `free_mem_ratio` - float between 0 and 1, default=0. The free memory ratio to keep during training.\n",481 "- `metric_constraints` - list, default=[] | The list of metric constraints.\n",482 " Each element in this list is a 3-tuple, which shall be expressed\n",483 " in the following format: the first element of the 3-tuple is the name of the\n",484 " metric, the second element is the inequality sign chosen from \">=\" and \"<=\",\n",485 " and the third element is the constraint value. E.g., `('precision', '>=', 0.9)`.\n",486 " Note that all the metric names in metric_constraints need to be reported via\n",487 " the metrics_to_log dictionary returned by a customized metric function.\n",488 " The customized metric function shall be provided via the `metric` key word argument\n",489 " of the fit() function or the automl constructor.\n",490 " Find examples in this [test](https://github.com/microsoft/FLAML/tree/main/test/automl/test_constraints.py).\n",491 " If `pred_time_limit` is provided as one of keyword arguments to fit() function or\n",492 " the automl constructor, flaml will automatically (and under the hood)\n",493 " add it as an additional element in the metric_constraints. Essentially 'pred_time_limit'\n",494 " specifies a constraint about the prediction latency constraint in seconds.\n",495 "- `custom_hp` - dict, default=None | The custom search space specified by user\n",496 " Each key is the estimator name, each value is a dict of the custom search space for that estimator. Notice the\n",497 " domain of the custom search space can either be a value of a sample.Domain object.\n",498 " \n",499 " \n",500 " \n",501 "```python\n",502 "custom_hp = {\n",503 " \"transformer_ms\": {\n",504 " \"model_path\": {\n",505 " \"domain\": \"albert-base-v2\",\n",506 " },\n",507 " \"learning_rate\": {\n",508 " \"domain\": tune.choice([1e-4, 1e-5]),\n",509 " }\n",510 " }\n",511 "}\n",512 "```\n",513 "- `time_col` - for a time series task, name of the column containing the timestamps. If not\n",514 " provided, defaults to the first column of X_train/X_val\n",515 " \n",516 "- `cv_score_agg_func` - customized cross-validation scores aggregate function. Default to average metrics across folds. If specificed, this function needs to\n",517 " have the following input arguments:\n",518 " \n",519 " * val_loss_folds: list of floats, the loss scores of each fold;\n",520 " * log_metrics_folds: list of dicts/floats, the metrics of each fold to log.\n",521 " \n",522 " This function should return the final aggregate result of all folds. A float number of the minimization objective, and a dictionary as the metrics to log or None.\n",523 " E.g.,\n",524 " \n",525 "```python\n",526 "def cv_score_agg_func(val_loss_folds, log_metrics_folds):\n",527 " metric_to_minimize = sum(val_loss_folds)/len(val_loss_folds)\n",528 " metrics_to_log = None\n",529 " for single_fold in log_metrics_folds:\n",530 " if metrics_to_log is None:\n",531 " metrics_to_log = single_fold\n",532 " elif isinstance(metrics_to_log, dict):\n",533 " metrics_to_log = {k: metrics_to_log[k] + v for k, v in single_fold.items()}\n",534 " else:\n",535 " metrics_to_log += single_fold\n",536 " if metrics_to_log:\n",537 " n = len(val_loss_folds)\n",538 " metrics_to_log = (\n",539 " {k: v / n for k, v in metrics_to_log.items()}\n",540 " if isinstance(metrics_to_log, dict)\n",541 " else metrics_to_log / n\n",542 " )\n",543 " return metric_to_minimize, metrics_to_log\n",544 "```\n",545 " \n",546 "- `skip_transform` - boolean, default=False | Whether to pre-process data prior to modeling.\n",547 "- `mlflow_logging` - boolean, default=None | Whether to log the training results to mlflow.\n",548 " Default value is None, which means the logging decision is made based on\n",549 " AutoML.__init__'s mlflow_logging argument.\n",550 " This requires mlflow to be installed and to have an active mlflow run.\n",551 " FLAML will create nested runs.\n",552 "- `fit_kwargs_by_estimator` - dict, default=None | The user specified keywords arguments, grouped by estimator name.\n",553 " For TransformersEstimator, available fit_kwargs can be found from\n",554 " [TrainingArgumentsForAuto](nlp/huggingface/training_args).\n",555 " e.g.,\n",556 " \n",557 "```python\n",558 "fit_kwargs_by_estimator = {\n",559 " \"transformer\": {\n",560 " \"output_dir\": \"test/data/output/\",\n",561 " \"fp16\": False,\n",562 " },\n",563 " \"tft\": {\n",564 " \"max_encoder_length\": 1,\n",565 " \"min_encoder_length\": 1,\n",566 " \"static_categoricals\": [],\n",567 " \"static_reals\": [],\n",568 " \"time_varying_known_categoricals\": [],\n",569 " \"time_varying_known_reals\": [],\n",570 " \"time_varying_unknown_categoricals\": [],\n",571 " \"time_varying_unknown_reals\": [],\n",572 " \"variable_groups\": {},\n",573 " \"lags\": {},\n",574 " }\n",575 "}\n",576 "```\n",577 " \n",578 "- `**fit_kwargs` - Other key word arguments to pass to fit() function of\n",579 " the searched learners, such as sample_weight. Below are a few examples of\n",580 " estimator-specific parameters:\n",581 "- `period` - int | forecast horizon for all time series forecast tasks.\n",582 "- `gpu_per_trial` - float, default = 0 | A float of the number of gpus per trial,\n",583 " only used by TransformersEstimator, XGBoostSklearnEstimator, and\n",584 " TemporalFusionTransformerEstimator.\n",585 "- `group_ids` - list of strings of column names identifying a time series, only\n",586 " used by TemporalFusionTransformerEstimator, required for\n",587 " 'ts_forecast_panel' task. `group_ids` is a parameter for TimeSeriesDataSet object\n",588 " from PyTorchForecasting.\n",589 " For other parameters to describe your dataset, refer to\n",590 " [TimeSeriesDataSet PyTorchForecasting](https://pytorch-forecasting.readthedocs.io/en/stable/api/pytorch_forecasting.data.timeseries.TimeSeriesDataSet.html).\n",591 " To specify your variables, use `static_categoricals`, `static_reals`,\n",592 " `time_varying_known_categoricals`, `time_varying_known_reals`,\n",593 " `time_varying_unknown_categoricals`, `time_varying_unknown_reals`,\n",594 " `variable_groups`. To provide more information on your data, use\n",595 " `max_encoder_length`, `min_encoder_length`, `lags`.\n",596 "- `log_dir` - str, default = \"lightning_logs\" | Folder into which to log results\n",597 " for tensorboard, only used by TemporalFusionTransformerEstimator.\n",598 "- `max_epochs` - int, default = 20 | Maximum number of epochs to run training,\n",599 " only used by TemporalFusionTransformerEstimator.\n",600 "- `batch_size` - int, default = 64 | Batch size for training model, only\n",601 " used by TemporalFusionTransformerEstimator.\n",602 "\n",603 "\n",604 " \n",605 "```python\n",606 "from flaml import BlendSearch\n",607 "algo = BlendSearch(metric='val_loss', mode='min',\n",608 " space=search_space,\n",609 " low_cost_partial_config=low_cost_partial_config)\n",610 "for i in range(10):\n",611 " analysis = tune.run(compute_with_config,\n",612 " search_alg=algo, use_ray=False)\n",613 " print(analysis.trials[-1].last_result)\n",614 "```\n",615 " \n",616 "- `verbose` - 0, 1, 2, or 3. If ray or spark backend is used, their verbosity will be\n",617 " affected by this argument. 0 = silent, 1 = only status updates,\n",618 " 2 = status and brief trial results, 3 = status and detailed trial results.\n",619 " Defaults to 2.\n",620 "- `local_dir` - A string of the local dir to save ray logs if ray backend is\n",621 " used; or a local dir to save the tuning log.\n",622 "- `num_samples` - An integer of the number of configs to try. Defaults to 1.\n",623 "- `resources_per_trial` - A dictionary of the hardware resources to allocate\n",624 " per trial, e.g., `{'cpu': 1}`. It is only valid when using ray backend\n",625 " (by setting 'use_ray = True'). It shall be used when you need to do\n",626 " [parallel tuning](../../Use-Cases/Tune-User-Defined-Function#parallel-tuning).\n",627 "- `config_constraints` - A list of config constraints to be satisfied.\n",628 " e.g., ```config_constraints = [(mem_size, '<=', 1024**3)]```\n",629 " \n",630 " mem_size is a function which produces a float number for the bytes\n",631 " needed for a config.\n",632 " It is used to skip configs which do not fit in memory.\n",633 "- `metric_constraints` - A list of metric constraints to be satisfied.\n",634 " e.g., `['precision', '>=', 0.9]`. The sign can be \">=\" or \"<=\".\n",635 "- `max_failure` - int | the maximal consecutive number of failures to sample\n",636 " a trial before the tuning is terminated.\n",637 "- `use_ray` - A boolean of whether to use ray as the backend.\n",638 "- `use_spark` - A boolean of whether to use spark as the backend.\n",639 "- `log_file_name` - A string of the log file name. Default to None.\n",640 " When set to None:\n",641 " if local_dir is not given, no log file is created;\n",642 " if local_dir is given, the log file name will be autogenerated under local_dir.\n",643 " Only valid when verbose > 0 or use_ray is True.\n",644 "- `lexico_objectives` - dict, default=None | It specifics information needed to perform multi-objective\n",645 " optimization with lexicographic preferences. When lexico_objectives is not None, the arguments metric,\n",646 " mode, will be invalid, and flaml's tune uses CFO\n",647 " as the `search_alg`, which makes the input (if provided) `search_alg' invalid.\n",648 " This dictionary shall contain the following fields of key-value pairs:\n",649 " - \"metrics\": a list of optimization objectives with the orders reflecting the priorities/preferences of the\n",650 " objectives.\n",651 " - \"modes\" (optional): a list of optimization modes (each mode either \"min\" or \"max\") corresponding to the\n",652 " objectives in the metric list. If not provided, we use \"min\" as the default mode for all the objectives.\n",653 " - \"targets\" (optional): a dictionary to specify the optimization targets on the objectives. The keys are the\n",654 " metric names (provided in \"metric\"), and the values are the numerical target values.\n",655 " - \"tolerances\" (optional): a dictionary to specify the optimality tolerances on objectives. The keys are the metric names (provided in \"metrics\"), and the values are the absolute/percentage tolerance in the form of numeric/string.\n",656 " E.g.,\n",657 "```python\n",658 "lexico_objectives = {\n",659 " \"metrics\": [\"error_rate\", \"pred_time\"],\n",660 " \"modes\": [\"min\", \"min\"],\n",661 " \"tolerances\": {\"error_rate\": 0.01, \"pred_time\": 0.0},\n",662 " \"targets\": {\"error_rate\": 0.0},\n",663 "}\n",664 "```\n",665 " We also support percentage tolerance.\n",666 " E.g.,\n",667 "```python\n",668 "lexico_objectives = {\n",669 " \"metrics\": [\"error_rate\", \"pred_time\"],\n",670 " \"modes\": [\"min\", \"min\"],\n",671 " \"tolerances\": {\"error_rate\": \"5%\", \"pred_time\": \"0%\"},\n",672 " \"targets\": {\"error_rate\": 0.0},\n",673 "}\n",674 "```\n",675 "- `force_cancel` - boolean, default=False | Whether to forcely cancel the PySpark job if overtime.\n",676 "- `n_concurrent_trials` - int, default=0 | The number of concurrent trials when perform hyperparameter\n",677 " tuning with Spark. Only valid when use_spark=True and spark is required:\n",678 " `pip install flaml[spark]`. Please check\n",679 " [here](https://spark.apache.org/docs/latest/api/python/getting_started/install.html)\n",680 " for more details about installing Spark. When tune.run() is called from AutoML, it will be\n",681 " overwritten by the value of `n_concurrent_trials` in AutoML. When <= 0, the concurrent trials\n",682 " will be set to the number of executors.\n",683 "- `**ray_args` - keyword arguments to pass to ray.tune.run().\n",684 " Only valid when use_ray=True.\n",685 "\n",686 "## Tuner Objects\n",687 "\n",688 "```python\n",689 "class Tuner()\n",690 "```\n",691 "\n",692 "Tuner is the class-based way of launching hyperparameter tuning jobs compatible with Ray Tune 2.\n",693 "\n",694 "**Arguments**:\n",695 "\n",696 "- `trainable` - A user-defined evaluation function.\n",697 " It takes a configuration as input, outputs a evaluation\n",698 " result (can be a numerical value or a dictionary of string\n",699 " and numerical value pairs) for the input configuration.\n",700 " For machine learning tasks, it usually involves training and\n",701 " scoring a machine learning model, e.g., through validation loss.\n",702 "- `param_space` - Search space of the tuning job.\n",703 " One thing to note is that both preprocessor and dataset can be tuned here.\n",704 "- `tune_config` - Tuning algorithm specific configs.\n",705 " Refer to ray.tune.tune_config.TuneConfig for more info.\n",706 "- `run_config` - Runtime configuration that is specific to individual trials.\n",707 " If passed, this will overwrite the run config passed to the Trainer,\n",708 " if applicable. Refer to ray.air.config.RunConfig for more info.\n",709 " \n",710 " Usage pattern:\n",711 " \n",712 " .. code-block:: python\n",713 " \n",714 " from sklearn.datasets import load_breast_cancer\n",715 " \n",716 " from ray import tune\n",717 " from ray.data import from_pandas\n",718 " from ray.air.config import RunConfig, ScalingConfig\n",719 " from ray.train.xgboost import XGBoostTrainer\n",720 " from ray.tune.tuner import Tuner\n",721 " \n",722 " def get_dataset():\n",723 " data_raw = load_breast_cancer(as_frame=True)\n",724 " dataset_df = data_raw[\"data\"]\n",725 " dataset_df[\"target\"] = data_raw[\"target\"]\n",726 " dataset = from_pandas(dataset_df)\n",727 " return dataset\n",728 " \n",729 " trainer = XGBoostTrainer(\n",730 " label_column=\"target\",\n",731 " params={},\n",732 "- `datasets={\"train\"` - get_dataset()},\n",733 " )\n",734 " \n",735 " param_space = {\n",736 "- `\"scaling_config\"` - ScalingConfig(\n",737 " num_workers=tune.grid_search([2, 4]),\n",738 " resources_per_worker={\n",739 "- `\"CPU\"` - tune.grid_search([1, 2]),\n",740 " },\n",741 " ),\n",742 " # You can even grid search various datasets in Tune.\n",743 " # \"datasets\": {\n",744 " # \"train\": tune.grid_search(\n",745 " # [ds1, ds2]\n",746 " # ),\n",747 " # },\n",748 "- `\"params\"` - {\n",749 "- `\"objective\"` - \"binary:logistic\",\n",750 "- `\"tree_method\"` - \"approx\",\n",751 "- `\"eval_metric\"` - [\"logloss\", \"error\"],\n",752 "- `\"eta\"` - tune.loguniform(1e-4, 1e-1),\n",753 "- `\"subsample\"` - tune.uniform(0.5, 1.0),\n",754 "- `\"max_depth\"` - tune.randint(1, 9),\n",755 " },\n",756 " }\n",757 " tuner = Tuner(trainable=trainer, param_space=param_space,\n",758 " run_config=RunConfig(name=\"my_tune_run\"))\n",759 " analysis = tuner.fit()\n",760 " \n",761 " To retry a failed tune run, you can then do\n",762 " \n",763 " .. code-block:: python\n",764 " \n",765 " tuner = Tuner.restore(experiment_checkpoint_dir)\n",766 " tuner.fit()\n",767 " \n",768 " ``experiment_checkpoint_dir`` can be easily located near the end of the\n",769 " console output of your first failed run.\n",770 "\n",771 "\n",772 "\n",773 "\n",774 "\n",775 "--------------------------------------------------------------------------------\n",776 "\u001b[33massistant\u001b[0m (to ragproxyagent):\n",777 "\n",778 "To perform a classification task using FLAML and parallel training with Spark, you need to install FLAML with Spark support first, if you haven't done it yet:\n",779 "\n",780 "```\n",781 "pip install flaml[spark]\n",782 "```\n",783 "\n",784 "And then, you can use the following code example:\n",785 "\n",786 "```python\n",787 "from flaml import AutoML\n",788 "from flaml.data import load_openml_dataset\n",789 "from sklearn.metrics import accuracy_score\n",790 "\n",791 "# Load the dataset\n",792 "X_train, X_test, y_train, y_test = load_openml_dataset(dataset_id=21, data_dir='./')\n",793 "\n",794 "# Initialize the AutoML instance\n",795 "automl = AutoML()\n",796 "\n",797 "# Configure AutoML settings for classification\n",798 "settings = {\n",799 " \"time_budget\": 30, # Train for 30 seconds\n",800 " \"n_concurrent_trials\": 4, # Parallel training using Spark\n",801 " \"force_cancel\": True, # Force cancel jobs if time limit is reached\n",802 " \"use_spark\": True, # Use spark for parallel training\n",803 " \"metric\": \"accuracy\",\n",804 " \"task\": \"classification\",\n",805 " \"log_file_name\": \"flaml.log\",\n",806 "}\n",807 "\n",808 "# Train the model\n",809 "automl.fit(X_train, y_train, **settings)\n",810 "\n",811 "# Make predictions and calculate accuracy\n",812 "y_pred = automl.predict(X_test)\n",813 "accuracy = accuracy_score(y_test, y_pred)\n",814 "print(\"Test accuracy:\", accuracy)\n",815 "```\n",816 "\n",817 "This code will perform a classification task using FLAML AutoML with parallel training on Spark. FLAML will try different models and hyperparameters, and it will automatically stop after 30 seconds. Jobs will be force-cancelled if the time limit is reached.\n",818 "\n",819 "--------------------------------------------------------------------------------\n",820 "\u001b[31m\n",821 ">>>>>>>> EXECUTING CODE BLOCK 0 (inferred language is sh)...\u001b[0m\n",822 "\u001b[31m\n",823 ">>>>>>>> EXECUTING CODE BLOCK 1 (inferred language is python)...\u001b[0m\n",824 "load dataset from ./openml_ds21.pkl\n",825 "Dataset name: car\n",826 "X_train.shape: (1296, 6), y_train.shape: (1296,);\n",827 "X_test.shape: (432, 6), y_test.shape: (432,)\n",828 "[flaml.automl.logger: 08-11 17:25:31] {1679} INFO - task = classification\n",829 "[flaml.automl.logger: 08-11 17:25:31] {1690} INFO - Evaluation method: cv\n",830 "[flaml.automl.logger: 08-11 17:25:31] {1788} INFO - Minimizing error metric: 1-accuracy\n",831 "[flaml.automl.logger: 08-11 17:25:31] {1900} INFO - List of ML learners in AutoML Run: ['lgbm', 'rf', 'catboost', 'xgboost', 'extra_tree', 'xgb_limitdepth', 'lrl1']\n"832 ]833 },834 {835 "name": "stderr",836 "output_type": "stream",837 "text": [838 "\u001b[32m[I 2023-08-11 17:25:31,670]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n",839 "\u001b[32m[I 2023-08-11 17:25:31,701]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n"840 ]841 },842 {843 "name": "stdout",844 "output_type": "stream",845 "text": [846 "[flaml.tune.tune: 08-11 17:25:31] {729} INFO - Number of trials: 1/1000000, 1 RUNNING, 0 TERMINATED\n"847 ]848 },849 {850 "name": "stderr",851 "output_type": "stream",852 "text": [853 "2023-08-11 17:25:37.042724: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n",854 "2023-08-11 17:25:37.108934: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",855 "To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",856 "2023-08-11 17:25:38.540404: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n"857 ]858 },859 {860 "name": "stdout",861 "output_type": "stream",862 "text": [863 "[flaml.tune.tune: 08-11 17:25:42] {749} INFO - Brief result: {'pred_time': 2.349200360598676e-05, 'wall_clock_time': 10.836093425750732, 'metric_for_logging': {'pred_time': 2.349200360598676e-05}, 'val_loss': 0.29475200475200475, 'trained_estimator': <flaml.automl.model.LGBMEstimator object at 0x7fb43c642b20>}\n",864 "[flaml.tune.tune: 08-11 17:25:42] {729} INFO - Number of trials: 2/1000000, 1 RUNNING, 1 TERMINATED\n"865 ]866 },867 {868 "name": "stderr",869 "output_type": "stream",870 "text": [871 " \r"872 ]873 },874 {875 "name": "stdout",876 "output_type": "stream",877 "text": [878 "[flaml.tune.tune: 08-11 17:25:42] {749} INFO - Brief result: {'pred_time': 1.638828344999381e-05, 'wall_clock_time': 11.25049901008606, 'metric_for_logging': {'pred_time': 1.638828344999381e-05}, 'val_loss': 0.20062964062964062, 'trained_estimator': <flaml.automl.model.RandomForestEstimator object at 0x7fb43c648a00>}\n",879 "[flaml.tune.tune: 08-11 17:25:42] {729} INFO - Number of trials: 3/1000000, 1 RUNNING, 2 TERMINATED\n"880 ]881 },882 {883 "name": "stderr",884 "output_type": "stream",885 "text": [886 "[Stage 3:> (0 + 1) / 1]\r"887 ]888 },889 {890 "name": "stdout",891 "output_type": "stream",892 "text": [893 "[flaml.tune.tune: 08-11 17:25:50] {749} INFO - Brief result: {'pred_time': 3.0794482150416296e-05, 'wall_clock_time': 18.99154567718506, 'metric_for_logging': {'pred_time': 3.0794482150416296e-05}, 'val_loss': 0.0663855063855064, 'trained_estimator': <flaml.automl.model.CatBoostEstimator object at 0x7fb43c648dc0>}\n",894 "[flaml.tune.tune: 08-11 17:25:50] {729} INFO - Number of trials: 4/1000000, 1 RUNNING, 3 TERMINATED\n"895 ]896 },897 {898 "name": "stdout",899 "output_type": "stream",900 "text": [901 "[flaml.tune.tune: 08-11 17:25:51] {749} INFO - Brief result: {'pred_time': 2.8759363960150548e-05, 'wall_clock_time': 19.68805766105652, 'metric_for_logging': {'pred_time': 2.8759363960150548e-05}, 'val_loss': 0.152019602019602, 'trained_estimator': <flaml.automl.model.XGBoostSklearnEstimator object at 0x7fb43c654340>}\n",902 "[flaml.tune.tune: 08-11 17:25:51] {729} INFO - Number of trials: 5/1000000, 1 RUNNING, 4 TERMINATED\n"903 ]904 },905 {906 "name": "stdout",907 "output_type": "stream",908 "text": [909 "[flaml.tune.tune: 08-11 17:25:51] {749} INFO - Brief result: {'pred_time': 3.691017574608273e-05, 'wall_clock_time': 20.165640115737915, 'metric_for_logging': {'pred_time': 3.691017574608273e-05}, 'val_loss': 0.2608167508167508, 'trained_estimator': <flaml.automl.model.ExtraTreesEstimator object at 0x7fb43c654dc0>}\n",910 "[flaml.tune.tune: 08-11 17:25:51] {729} INFO - Number of trials: 6/1000000, 1 RUNNING, 5 TERMINATED\n"911 ]912 },913 {914 "name": "stdout",915 "output_type": "stream",916 "text": [917 "[flaml.tune.tune: 08-11 17:25:52] {749} INFO - Brief result: {'pred_time': 1.7430177597394853e-05, 'wall_clock_time': 20.693061351776123, 'metric_for_logging': {'pred_time': 1.7430177597394853e-05}, 'val_loss': 0.03318978318978323, 'trained_estimator': <flaml.automl.model.XGBoostLimitDepthEstimator object at 0x7fb43c654d90>}\n",918 "[flaml.tune.tune: 08-11 17:25:52] {729} INFO - Number of trials: 7/1000000, 1 RUNNING, 6 TERMINATED\n",919 "[flaml.tune.tune: 08-11 17:25:53] {749} INFO - Brief result: {'pred_time': 3.5216659617275313e-05, 'wall_clock_time': 21.475266218185425, 'metric_for_logging': {'pred_time': 3.5216659617275313e-05}, 'val_loss': 0.16745173745173744, 'trained_estimator': <flaml.automl.model.LRL1Classifier object at 0x7fb43c648e50>}\n",920 "[flaml.tune.tune: 08-11 17:25:53] {729} INFO - Number of trials: 8/1000000, 1 RUNNING, 7 TERMINATED\n"921 ]922 },923 {924 "name": "stdout",925 "output_type": "stream",926 "text": [927 "[flaml.tune.tune: 08-11 17:25:54] {749} INFO - Brief result: {'pred_time': 4.353435378702026e-05, 'wall_clock_time': 22.360871076583862, 'metric_for_logging': {'pred_time': 4.353435378702026e-05}, 'val_loss': 0.034725274725274737, 'trained_estimator': <flaml.automl.model.XGBoostLimitDepthEstimator object at 0x7fb43c667820>}\n",928 "[flaml.tune.tune: 08-11 17:25:54] {729} INFO - Number of trials: 9/1000000, 1 RUNNING, 8 TERMINATED\n"929 ]930 },931 {932 "name": "stdout",933 "output_type": "stream",934 "text": [935 "[flaml.tune.tune: 08-11 17:25:54] {749} INFO - Brief result: {'pred_time': 2.568628159906236e-05, 'wall_clock_time': 23.031129837036133, 'metric_for_logging': {'pred_time': 2.568628159906236e-05}, 'val_loss': 0.07177012177012176, 'trained_estimator': <flaml.automl.model.XGBoostLimitDepthEstimator object at 0x7fb43c980160>}\n",936 "[flaml.tune.tune: 08-11 17:25:54] {729} INFO - Number of trials: 10/1000000, 1 RUNNING, 9 TERMINATED\n"937 ]938 },939 {940 "name": "stdout",941 "output_type": "stream",942 "text": [943 "[flaml.tune.tune: 08-11 17:25:55] {749} INFO - Brief result: {'pred_time': 3.6701016019634797e-05, 'wall_clock_time': 23.525509119033813, 'metric_for_logging': {'pred_time': 3.6701016019634797e-05}, 'val_loss': 0.78009207009207, 'trained_estimator': <flaml.automl.model.XGBoostLimitDepthEstimator object at 0x7fb43cb5c4c0>}\n",944 "[flaml.tune.tune: 08-11 17:25:55] {729} INFO - Number of trials: 11/1000000, 1 RUNNING, 10 TERMINATED\n"945 ]946 },947 {948 "name": "stdout",949 "output_type": "stream",950 "text": [951 "[flaml.tune.tune: 08-11 17:25:55] {749} INFO - Brief result: {'pred_time': 3.9799592953107814e-05, 'wall_clock_time': 24.326939582824707, 'metric_for_logging': {'pred_time': 3.9799592953107814e-05}, 'val_loss': 0.011577071577071552, 'trained_estimator': <flaml.automl.model.XGBoostLimitDepthEstimator object at 0x7fb43c99b880>}\n",952 "[flaml.tune.tune: 08-11 17:25:55] {729} INFO - Number of trials: 12/1000000, 1 RUNNING, 11 TERMINATED\n"953 ]954 },955 {956 "name": "stdout",957 "output_type": "stream",958 "text": [959 "[flaml.tune.tune: 08-11 17:25:56] {749} INFO - Brief result: {'pred_time': 1.9423383118527775e-05, 'wall_clock_time': 24.820234775543213, 'metric_for_logging': {'pred_time': 1.9423383118527775e-05}, 'val_loss': 0.037817047817047825, 'trained_estimator': <flaml.automl.model.XGBoostLimitDepthEstimator object at 0x7fb43c9a78e0>}\n",960 "[flaml.tune.tune: 08-11 17:25:56] {729} INFO - Number of trials: 13/1000000, 1 RUNNING, 12 TERMINATED\n"961 ]962 },963 {964 "name": "stdout",965 "output_type": "stream",966 "text": [967 "[flaml.tune.tune: 08-11 17:25:57] {749} INFO - Brief result: {'pred_time': 2.987599351620653e-05, 'wall_clock_time': 25.54983139038086, 'metric_for_logging': {'pred_time': 2.987599351620653e-05}, 'val_loss': 0.030873180873180896, 'trained_estimator': <flaml.automl.model.XGBoostLimitDepthEstimator object at 0x7fb43c98b850>}\n",968 "[flaml.tune.tune: 08-11 17:25:57] {729} INFO - Number of trials: 14/1000000, 1 RUNNING, 13 TERMINATED\n"969 ]970 },971 {972 "name": "stdout",973 "output_type": "stream",974 "text": [975 "[flaml.tune.tune: 08-11 17:25:57] {749} INFO - Brief result: {'pred_time': 2.351036190738797e-05, 'wall_clock_time': 26.08720564842224, 'metric_for_logging': {'pred_time': 2.351036190738797e-05}, 'val_loss': 0.020065340065340043, 'trained_estimator': <flaml.automl.model.XGBoostLimitDepthEstimator object at 0x7fb43c98bd60>}\n",976 "[flaml.tune.tune: 08-11 17:25:57] {729} INFO - Number of trials: 15/1000000, 1 RUNNING, 14 TERMINATED\n"977 ]978 },979 {980 "name": "stdout",981 "output_type": "stream",982 "text": [983 "[flaml.tune.tune: 08-11 17:25:58] {749} INFO - Brief result: {'pred_time': 2.2003395747883512e-05, 'wall_clock_time': 26.587312698364258, 'metric_for_logging': {'pred_time': 2.2003395747883512e-05}, 'val_loss': 0.03936144936144936, 'trained_estimator': <flaml.automl.model.XGBoostLimitDepthEstimator object at 0x7fb43c9a7190>}\n",984 "[flaml.tune.tune: 08-11 17:25:58] {729} INFO - Number of trials: 16/1000000, 1 RUNNING, 15 TERMINATED\n"985 ]986 },987 {988 "name": "stdout",989 "output_type": "stream",990 "text": [991 "[flaml.tune.tune: 08-11 17:25:58] {749} INFO - Brief result: {'pred_time': 2.1086723400146556e-05, 'wall_clock_time': 27.126797914505005, 'metric_for_logging': {'pred_time': 2.1086723400146556e-05}, 'val_loss': 0.015444015444015413, 'trained_estimator': <flaml.automl.model.XGBoostLimitDepthEstimator object at 0x7fb43c99b8b0>}\n",992 "[flaml.tune.tune: 08-11 17:25:58] {729} INFO - Number of trials: 17/1000000, 1 RUNNING, 16 TERMINATED\n"993 ]994 },995 {996 "name": "stdout",997 "output_type": "stream",998 "text": [999 "[flaml.tune.tune: 08-11 17:25:59] {749} INFO - Brief result: {'pred_time': 1.6717643811435773e-05, 'wall_clock_time': 27.661753177642822, 'metric_for_logging': {'pred_time': 1.6717643811435773e-05}, 'val_loss': 0.07254232254232254, 'trained_estimator': <flaml.automl.model.XGBoostLimitDepthEstimator object at 0x7fb43c642a90>}\n",1000 "[flaml.tune.tune: 08-11 17:25:59] {729} INFO - Number of trials: 18/1000000, 1 RUNNING, 17 TERMINATED\n"1001 ]1002 },1003 {1004 "name": "stdout",1005 "output_type": "stream",1006 "text": [1007 "[flaml.tune.tune: 08-11 17:26:00] {749} INFO - Brief result: {'pred_time': 3.0297818083348173e-05, 'wall_clock_time': 28.433676958084106, 'metric_for_logging': {'pred_time': 3.0297818083348173e-05}, 'val_loss': 0.020068310068310048, 'trained_estimator': <flaml.automl.model.XGBoostLimitDepthEstimator object at 0x7fb43cb5cdf0>}\n",1008 "[flaml.tune.tune: 08-11 17:26:00] {729} INFO - Number of trials: 19/1000000, 1 RUNNING, 18 TERMINATED\n"1009 ]1010 },1011 {1012 "name": "stdout",1013 "output_type": "stream",1014 "text": [1015 "[flaml.tune.tune: 08-11 17:26:00] {749} INFO - Brief result: {'pred_time': 2.0136982600838343e-05, 'wall_clock_time': 28.9714093208313, 'metric_for_logging': {'pred_time': 2.0136982600838343e-05}, 'val_loss': 0.010807840807840785, 'trained_estimator': <flaml.automl.model.XGBoostLimitDepthEstimator object at 0x7fb43c98baf0>}\n",1016 "[flaml.tune.tune: 08-11 17:26:00] {729} INFO - Number of trials: 20/1000000, 1 RUNNING, 19 TERMINATED\n"1017 ]1018 },1019 {1020 "name": "stdout",1021 "output_type": "stream",1022 "text": [1023 "[flaml.tune.tune: 08-11 17:26:01] {749} INFO - Brief result: {'pred_time': 2.0759203400709594e-05, 'wall_clock_time': 29.460874795913696, 'metric_for_logging': {'pred_time': 2.0759203400709594e-05}, 'val_loss': 0.017751707751707736, 'trained_estimator': <flaml.automl.model.XGBoostLimitDepthEstimator object at 0x7fb43c6486d0>}\n",1024 "[flaml.tune.tune: 08-11 17:26:01] {729} INFO - Number of trials: 21/1000000, 1 RUNNING, 20 TERMINATED\n"1025 ]1026 },1027 {1028 "name": "stdout",1029 "output_type": "stream",1030 "text": [1031 "\n",1032 "[flaml.automl.logger: 08-11 17:26:01] {2493} INFO - selected model: None\n"1033 ]1034 },1035 {1036 "name": "stdout",1037 "output_type": "stream",1038 "text": [1039 "[flaml.automl.logger: 08-11 17:26:02] {2627} INFO - retrain xgb_limitdepth for 0.7s\n",1040 "[flaml.automl.logger: 08-11 17:26:02] {2630} INFO - retrained model: XGBClassifier(base_score=None, booster=None, callbacks=[],\n",1041 " colsample_bylevel=1.0, colsample_bynode=None,\n",1042 " colsample_bytree=1.0, early_stopping_rounds=None,\n",1043 " enable_categorical=False, eval_metric=None, feature_types=None,\n",1044 " gamma=None, gpu_id=None, grow_policy=None, importance_type=None,\n",1045 " interaction_constraints=None, learning_rate=1.0, max_bin=None,\n",1046 " max_cat_threshold=None, max_cat_to_onehot=None,\n",1047 " max_delta_step=None, max_depth=5, max_leaves=None,\n",1048 " min_child_weight=0.4411564712550587, missing=nan,\n",1049 " monotone_constraints=None, n_estimators=12, n_jobs=-1,\n",1050 " num_parallel_tree=None, objective='multi:softprob',\n",1051 " predictor=None, ...)\n",1052 "[flaml.automl.logger: 08-11 17:26:02] {2630} INFO - retrained model: XGBClassifier(base_score=None, booster=None, callbacks=[],\n",1053 " colsample_bylevel=1.0, colsample_bynode=None,\n",1054 " colsample_bytree=1.0, early_stopping_rounds=None,\n",1055 " enable_categorical=False, eval_metric=None, feature_types=None,\n",1056 " gamma=None, gpu_id=None, grow_policy=None, importance_type=None,\n",1057 " interaction_constraints=None, learning_rate=1.0, max_bin=None,\n",1058 " max_cat_threshold=None, max_cat_to_onehot=None,\n",1059 " max_delta_step=None, max_depth=5, max_leaves=None,\n",1060 " min_child_weight=0.4411564712550587, missing=nan,\n",1061 " monotone_constraints=None, n_estimators=12, n_jobs=-1,\n",1062 " num_parallel_tree=None, objective='multi:softprob',\n",1063 " predictor=None, ...)\n",1064 "[flaml.automl.logger: 08-11 17:26:02] {1930} INFO - fit succeeded\n",1065 "[flaml.automl.logger: 08-11 17:26:02] {1931} INFO - Time taken to find the best model: 28.9714093208313\n",1066 "Test accuracy: 0.9837962962962963\n",1067 "\u001b[33mragproxyagent\u001b[0m (to assistant):\n",1068 "\n",1069 "exitcode: 0 (execution succeeded)\n",1070 "Code output: \n",1071 "You MUST NOT install any packages because all the packages needed are already installed.\n",1072 "None\n",1073 "\n",1074 "--------------------------------------------------------------------------------\n",1075 "\u001b[33massistant\u001b[0m (to ragproxyagent):\n",1076 "\n",1077 "TERMINATE\n",1078 "\n",1079 "--------------------------------------------------------------------------------\n"1080 ]1081 }1082 ],1083 "source": [1084 "# reset the assistant. Always reset the assistant before starting a new conversation.\n",1085 "assistant.reset()\n",1086 "\n",1087 "# given a problem, we use the ragproxyagent to generate a prompt to be sent to the assistant as the initial message.\n",1088 "# the assistant receives the message and generates a response. The response will be sent back to the ragproxyagent for processing.\n",1089 "# The conversation continues until the termination condition is met, in RetrieveChat, the termination condition when no human-in-loop is no code block detected.\n",1090 "# With human-in-loop, the conversation will continue until the user says \"exit\".\n",1091 "code_problem = \"How can I use FLAML to perform a classification task and use spark to do parallel training. Train 30 seconds and force cancel jobs if time limit is reached.\"\n",1092 "ragproxyagent.initiate_chat(assistant, problem=code_problem, search_string=\"spark\") # search_string is used as an extra filter for the embeddings search, in this case, we only want to search documents that contain \"spark\"."1093 ]1094 },1095 {1096 "attachments": {},1097 "cell_type": "markdown",1098 "metadata": {},1099 "source": [1100 "<a id=\"example-2\"></a>\n",1101 "### Example 2\n",1102 "\n",1103 "[back to top](#toc)\n",1104 "\n",1105 "Use RetrieveChat to answer a question that is not related to code generation.\n",1106 "\n",1107 "Problem: Who is the author of FLAML?"1108 ]1109 },1110 {1111 "cell_type": "code",1112 "execution_count": null,1113 "metadata": {},1114 "outputs": [],1115 "source": [1116 "# reset the assistant. Always reset the assistant before starting a new conversation.\n",1117 "assistant.reset()\n",1118 "\n",1119 "qa_problem = \"Who is the author of FLAML?\"\n",1120 "ragproxyagent.initiate_chat(assistant, problem=qa_problem)"1121 ]1122 },1123 {1124 "attachments": {},1125 "cell_type": "markdown",1126 "metadata": {},1127 "source": [1128 "<a id=\"example-3\"></a>\n",1129 "### Example 3\n",1130 "\n",1131 "[back to top](#toc)\n",1132 "\n",1133 "Use RetrieveChat to help generate sample code and ask for human-in-loop feedbacks.\n",1134 "\n",1135 "Problem: how to build a time series forecasting model for stock price using FLAML?"1136 ]1137 },1138 {1139 "cell_type": "code",1140 "execution_count": 8,1141 "metadata": {},1142 "outputs": [1143 {1144 "name": "stdout",1145 "output_type": "stream",1146 "text": [1147 "doc_ids: [['doc_39', 'doc_46', 'doc_49', 'doc_36', 'doc_38', 'doc_51', 'doc_37', 'doc_58', 'doc_48', 'doc_40', 'doc_47', 'doc_41', 'doc_15', 'doc_52', 'doc_14', 'doc_60', 'doc_59', 'doc_43', 'doc_11', 'doc_35']]\n",1148 "\u001b[32mAdding doc_id doc_39 to context.\u001b[0m\n",1149 "\u001b[32mAdding doc_id doc_46 to context.\u001b[0m\n",1150 "\u001b[32mAdding doc_id doc_49 to context.\u001b[0m\n",1151 "\u001b[32mAdding doc_id doc_36 to context.\u001b[0m\n",1152 "\u001b[32mAdding doc_id doc_38 to context.\u001b[0m\n",1153 "\u001b[32mAdding doc_id doc_46 to context.\u001b[0m\n",1154 "\u001b[32mAdding doc_id doc_49 to context.\u001b[0m\n",1155 "\u001b[32mAdding doc_id doc_36 to context.\u001b[0m\n",1156 "\u001b[32mAdding doc_id doc_38 to context.\u001b[0m\n",1157 "\u001b[33mragproxyagent\u001b[0m (to assistant):\n",1158 "\n",1159 "You're a retrieve augmented coding assistant. You answer user's questions based on your own knowledge and the\n",1160 "context provided by the user.\n",1161 "If you can't answer the question with or without the current context, you should reply exactly `UPDATE CONTEXT`.\n",1162 "For code generation, you must obey the following rules:\n",1163 "Rule 1. You MUST NOT install any packages because all the packages needed are already installed.\n",1164 "Rule 2. You must follow the formats below to write your code:\n",1165 "```language\n",1166 "# your code\n",1167 "```\n",1168 "\n",1169 "User's question is: how to build a time series forecasting model for stock price using FLAML?\n",1170 "\n",1171 "Context is: \n",1172 "- `X_train` - A numpy array or a pandas dataframe of training data in\n",1173 " shape (n, m). For time series forecsat tasks, the first column of X_train\n",1174 " must be the timestamp column (datetime type). Other columns in\n",1175 " the dataframe are assumed to be exogenous variables (categorical or numeric).\n",1176 " When using ray, X_train can be a ray.ObjectRef.\n",1177 "- `y_train` - A numpy array or a pandas series of labels in shape (n, ).\n",1178 "- `dataframe` - A dataframe of training data including label column.\n",1179 " For time series forecast tasks, dataframe must be specified and must have\n",1180 " at least two columns, timestamp and label, where the first\n",1181 " column is the timestamp column (datetime type). Other columns in\n",1182 " the dataframe are assumed to be exogenous variables (categorical or numeric).\n",1183 " When using ray, dataframe can be a ray.ObjectRef.\n",1184 "- `label` - A str of the label column name for, e.g., 'label';\n",1185 "- `Note` - If X_train and y_train are provided,\n",1186 " dataframe and label are ignored;\n",1187 " If not, dataframe and label must be provided.\n",1188 "- `metric` - A string of the metric name or a function,\n",1189 " e.g., 'accuracy', 'roc_auc', 'roc_auc_ovr', 'roc_auc_ovo', 'roc_auc_weighted',\n",1190 " 'roc_auc_ovo_weighted', 'roc_auc_ovr_weighted', 'f1', 'micro_f1', 'macro_f1',\n",1191 " 'log_loss', 'mae', 'mse', 'r2', 'mape'. Default is 'auto'.\n",1192 " If passing a customized metric function, the function needs to\n",1193 " have the following input arguments:\n",1194 " \n",1195 "```python\n",1196 "def custom_metric(\n",1197 " X_test, y_test, estimator, labels,\n",1198 " X_train, y_train, weight_test=None, weight_train=None,\n",1199 " config=None, groups_test=None, groups_train=None,\n",1200 "):\n",