google/code_x_glue_ct_code_to_text

1---2annotations_creators:3- found4language_creators:5- found6language:7- code8- en9license:10- c-uda11multilinguality:12- other-programming-languages13size_categories:14- 100K<n<1M15- 10K<n<100K16source_datasets:17- original18task_categories:19- translation20task_ids: []21pretty_name: CodeXGlueCtCodeToText22config_names:23- go24- java25- javascript26- php27- python28- ruby29tags:30- code-to-text31dataset_info:32- config_name: go33  features:34  - name: id35    dtype: int3236  - name: repo37    dtype: string38  - name: path39    dtype: string40  - name: func_name41    dtype: string42  - name: original_string43    dtype: string44  - name: language45    dtype: string46  - name: code47    dtype: string48  - name: code_tokens49    sequence: string50  - name: docstring51    dtype: string52  - name: docstring_tokens53    sequence: string54  - name: sha55    dtype: string56  - name: url57    dtype: string58  splits:59  - name: train60    num_bytes: 34224314361    num_examples: 16728862  - name: validation63    num_bytes: 1372186064    num_examples: 732565  - name: test66    num_bytes: 1632840667    num_examples: 812268  download_size: 12134169869  dataset_size: 37229340970- config_name: java71  features:72  - name: id73    dtype: int3274  - name: repo75    dtype: string76  - name: path77    dtype: string78  - name: func_name79    dtype: string80  - name: original_string81    dtype: string82  - name: language83    dtype: string84  - name: code85    dtype: string86  - name: code_tokens87    sequence: string88  - name: docstring89    dtype: string90  - name: docstring_tokens91    sequence: string92  - name: sha93    dtype: string94  - name: url95    dtype: string96  splits:97  - name: train98    num_bytes: 45255383599    num_examples: 164923100  - name: validation101    num_bytes: 13366344102    num_examples: 5183103  - name: test104    num_bytes: 29080753105    num_examples: 10955106  download_size: 154701399107  dataset_size: 495000932108- config_name: javascript109  features:110  - name: id111    dtype: int32112  - name: repo113    dtype: string114  - name: path115    dtype: string116  - name: func_name117    dtype: string118  - name: original_string119    dtype: string120  - name: language121    dtype: string122  - name: code123    dtype: string124  - name: code_tokens125    sequence: string126  - name: docstring127    dtype: string128  - name: docstring_tokens129    sequence: string130  - name: sha131    dtype: string132  - name: url133    dtype: string134  splits:135  - name: train136    num_bytes: 160860431137    num_examples: 58025138  - name: validation139    num_bytes: 10337344140    num_examples: 3885141  - name: test142    num_bytes: 10190713143    num_examples: 3291144  download_size: 65788314145  dataset_size: 181388488146- config_name: php147  features:148  - name: id149    dtype: int32150  - name: repo151    dtype: string152  - name: path153    dtype: string154  - name: func_name155    dtype: string156  - name: original_string157    dtype: string158  - name: language159    dtype: string160  - name: code161    dtype: string162  - name: code_tokens163    sequence: string164  - name: docstring165    dtype: string166  - name: docstring_tokens167    sequence: string168  - name: sha169    dtype: string170  - name: url171    dtype: string172  splits:173  - name: train174    num_bytes: 614654499175    num_examples: 241241176  - name: validation177    num_bytes: 33283045178    num_examples: 12982179  - name: test180    num_bytes: 35374993181    num_examples: 14014182  download_size: 219692158183  dataset_size: 683312537184- config_name: python185  features:186  - name: id187    dtype: int32188  - name: repo189    dtype: string190  - name: path191    dtype: string192  - name: func_name193    dtype: string194  - name: original_string195    dtype: string196  - name: language197    dtype: string198  - name: code199    dtype: string200  - name: code_tokens201    sequence: string202  - name: docstring203    dtype: string204  - name: docstring_tokens205    sequence: string206  - name: sha207    dtype: string208  - name: url209    dtype: string210  splits:211  - name: train212    num_bytes: 813663148213    num_examples: 251820214  - name: validation215    num_bytes: 46888564216    num_examples: 13914217  - name: test218    num_bytes: 50659688219    num_examples: 14918220  download_size: 325551862221  dataset_size: 911211400222- config_name: ruby223  features:224  - name: id225    dtype: int32226  - name: repo227    dtype: string228  - name: path229    dtype: string230  - name: func_name231    dtype: string232  - name: original_string233    dtype: string234  - name: language235    dtype: string236  - name: code237    dtype: string238  - name: code_tokens239    sequence: string240  - name: docstring241    dtype: string242  - name: docstring_tokens243    sequence: string244  - name: sha245    dtype: string246  - name: url247    dtype: string248  splits:249  - name: train250    num_bytes: 51956439251    num_examples: 24927252  - name: validation253    num_bytes: 2821037254    num_examples: 1400255  - name: test256    num_bytes: 2671551257    num_examples: 1261258  download_size: 21921316259  dataset_size: 57449027260configs:261- config_name: go262  data_files:263  - split: train264    path: go/train-*265  - split: validation266    path: go/validation-*267  - split: test268    path: go/test-*269- config_name: java270  data_files:271  - split: train272    path: java/train-*273  - split: validation274    path: java/validation-*275  - split: test276    path: java/test-*277- config_name: javascript278  data_files:279  - split: train280    path: javascript/train-*281  - split: validation282    path: javascript/validation-*283  - split: test284    path: javascript/test-*285- config_name: php286  data_files:287  - split: train288    path: php/train-*289  - split: validation290    path: php/validation-*291  - split: test292    path: php/test-*293- config_name: python294  data_files:295  - split: train296    path: python/train-*297  - split: validation298    path: python/validation-*299  - split: test300    path: python/test-*301- config_name: ruby302  data_files:303  - split: train304    path: ruby/train-*305  - split: validation306    path: ruby/validation-*307  - split: test308    path: ruby/test-*309---310# Dataset Card for "code_x_glue_ct_code_to_text"311 312## Table of Contents313- [Dataset Description](#dataset-description)314  - [Dataset Summary](#dataset-summary)315  - [Supported Tasks and Leaderboards](#supported-tasks)316  - [Languages](#languages)317- [Dataset Structure](#dataset-structure)318  - [Data Instances](#data-instances)319  - [Data Fields](#data-fields)320  - [Data Splits](#data-splits-sample-size)321- [Dataset Creation](#dataset-creation)322  - [Curation Rationale](#curation-rationale)323  - [Source Data](#source-data)324  - [Annotations](#annotations)325  - [Personal and Sensitive Information](#personal-and-sensitive-information)326- [Considerations for Using the Data](#considerations-for-using-the-data)327  - [Social Impact of Dataset](#social-impact-of-dataset)328  - [Discussion of Biases](#discussion-of-biases)329  - [Other Known Limitations](#other-known-limitations)330- [Additional Information](#additional-information)331  - [Dataset Curators](#dataset-curators)332  - [Licensing Information](#licensing-information)333  - [Citation Information](#citation-information)334  - [Contributions](#contributions)335 336## Dataset Description337 338- **Homepage:** https://github.com/microsoft/CodeXGLUE/tree/main/Code-Text/code-to-text339 340### Dataset Summary341 342CodeXGLUE code-to-text dataset, available at https://github.com/microsoft/CodeXGLUE/tree/main/Code-Text/code-to-text343 344The dataset we use comes from CodeSearchNet and we filter the dataset as the following:345- Remove examples that codes cannot be parsed into an abstract syntax tree.346- Remove examples that #tokens of documents is < 3 or >256347- Remove examples that documents contain special tokens (e.g. <img ...> or https:...)348- Remove examples that documents are not English.349 350### Supported Tasks and Leaderboards351 352- `machine-translation`: The dataset can be used to train a model for automatically generating **English** docstrings for code.353 354### Languages355 356- Go **programming** language357- Java **programming** language358- Javascript **programming** language359- PHP **programming** language360- Python **programming** language361- Ruby **programming** language362- English **natural** language363 364## Dataset Structure365 366### Data Instances367 368#### go369 370An example of 'test' looks as follows.371```372{373    "code": "func NewSTM(c *v3.Client, apply func(STM) error, so ...stmOption) (*v3.TxnResponse, error) {\n\topts := &stmOptions{ctx: c.Ctx()}\n\tfor _, f := range so {\n\t\tf(opts)\n\t}\n\tif len(opts.prefetch) != 0 {\n\t\tf := apply\n\t\tapply = func(s STM) error {\n\t\t\ts.Get(opts.prefetch...)\n\t\t\treturn f(s)\n\t\t}\n\t}\n\treturn runSTM(mkSTM(c, opts), apply)\n}", 374    "code_tokens": ["func", "NewSTM", "(", "c", "*", "v3", ".", "Client", ",", "apply", "func", "(", "STM", ")", "error", ",", "so", "...", "stmOption", ")", "(", "*", "v3", ".", "TxnResponse", ",", "error", ")", "{", "opts", ":=", "&", "stmOptions", "{", "ctx", ":", "c", ".", "Ctx", "(", ")", "}", "\n", "for", "_", ",", "f", ":=", "range", "so", "{", "f", "(", "opts", ")", "\n", "}", "\n", "if", "len", "(", "opts", ".", "prefetch", ")", "!=", "0", "{", "f", ":=", "apply", "\n", "apply", "=", "func", "(", "s", "STM", ")", "error", "{", "s", ".", "Get", "(", "opts", ".", "prefetch", "...", ")", "\n", "return", "f", "(", "s", ")", "\n", "}", "\n", "}", "\n", "return", "runSTM", "(", "mkSTM", "(", "c", ",", "opts", ")", ",", "apply", ")", "\n", "}"], 375    "docstring": "// NewSTM initiates a new STM instance, using serializable snapshot isolation by default.", 376    "docstring_tokens": ["NewSTM", "initiates", "a", "new", "STM", "instance", "using", "serializable", "snapshot", "isolation", "by", "default", "."], 377    "func_name": "NewSTM", 378    "id": 0, 379    "language": "go", 380    "original_string": "func NewSTM(c *v3.Client, apply func(STM) error, so ...stmOption) (*v3.TxnResponse, error) {\n\topts := &stmOptions{ctx: c.Ctx()}\n\tfor _, f := range so {\n\t\tf(opts)\n\t}\n\tif len(opts.prefetch) != 0 {\n\t\tf := apply\n\t\tapply = func(s STM) error {\n\t\t\ts.Get(opts.prefetch...)\n\t\t\treturn f(s)\n\t\t}\n\t}\n\treturn runSTM(mkSTM(c, opts), apply)\n}", 381    "path": "clientv3/concurrency/stm.go", 382    "repo": "etcd-io/etcd", 383    "sha": "616592d9ba993e3fe9798eef581316016df98906", 384    "url": "https://github.com/etcd-io/etcd/blob/616592d9ba993e3fe9798eef581316016df98906/clientv3/concurrency/stm.go#L89-L102"385}386```387 388#### java389 390An example of 'test' looks as follows.391```392{393    "code": "protected final void fastPathOrderedEmit(U value, boolean delayError, Disposable disposable) {\n        final Observer<? super V> observer = downstream;\n        final SimplePlainQueue<U> q = queue;\n\n        if (wip.get() == 0 && wip.compareAndSet(0, 1)) {\n            if (q.isEmpty()) {\n                accept(observer, value);\n                if (leave(-1) == 0) {\n                    return;\n                }\n            } else {\n                q.offer(value);\n            }\n        } else {\n            q.offer(value);\n            if (!enter()) {\n                return;\n            }\n        }\n        QueueDrainHelper.drainLoop(q, observer, delayError, disposable, this);\n    }", 394    "code_tokens": ["protected", "final", "void", "fastPathOrderedEmit", "(", "U", "value", ",", "boolean", "delayError", ",", "Disposable", "disposable", ")", "{", "final", "Observer", "<", "?", "super", "V", ">", "observer", "=", "downstream", ";", "final", "SimplePlainQueue", "<", "U", ">", "q", "=", "queue", ";", "if", "(", "wip", ".", "get", "(", ")", "==", "0", "&&", "wip", ".", "compareAndSet", "(", "0", ",", "1", ")", ")", "{", "if", "(", "q", ".", "isEmpty", "(", ")", ")", "{", "accept", "(", "observer", ",", "value", ")", ";", "if", "(", "leave", "(", "-", "1", ")", "==", "0", ")", "{", "return", ";", "}", "}", "else", "{", "q", ".", "offer", "(", "value", ")", ";", "}", "}", "else", "{", "q", ".", "offer", "(", "value", ")", ";", "if", "(", "!", "enter", "(", ")", ")", "{", "return", ";", "}", "}", "QueueDrainHelper", ".", "drainLoop", "(", "q", ",", "observer", ",", "delayError", ",", "disposable", ",", "this", ")", ";", "}"], 395    "docstring": "Makes sure the fast-path emits in order.\n@param value the value to emit or queue up\n@param delayError if true, errors are delayed until the source has terminated\n@param disposable the resource to dispose if the drain terminates", 396    "docstring_tokens": ["Makes", "sure", "the", "fast", "-", "path", "emits", "in", "order", "."], 397    "func_name": "QueueDrainObserver.fastPathOrderedEmit", 398    "id": 0, 399    "language": "java", 400    "original_string": "protected final void fastPathOrderedEmit(U value, boolean delayError, Disposable disposable) {\n        final Observer<? super V> observer = downstream;\n        final SimplePlainQueue<U> q = queue;\n\n        if (wip.get() == 0 && wip.compareAndSet(0, 1)) {\n            if (q.isEmpty()) {\n                accept(observer, value);\n                if (leave(-1) == 0) {\n                    return;\n                }\n            } else {\n                q.offer(value);\n            }\n        } else {\n            q.offer(value);\n            if (!enter()) {\n                return;\n            }\n        }\n        QueueDrainHelper.drainLoop(q, observer, delayError, disposable, this);\n    }", 401    "path": "src/main/java/io/reactivex/internal/observers/QueueDrainObserver.java", 402    "repo": "ReactiveX/RxJava", 403    "sha": "ac84182aa2bd866b53e01c8e3fe99683b882c60e", 404    "url": "https://github.com/ReactiveX/RxJava/blob/ac84182aa2bd866b53e01c8e3fe99683b882c60e/src/main/java/io/reactivex/internal/observers/QueueDrainObserver.java#L88-L108"405}406```407 408#### javascript409 410An example of 'test' looks as follows.411```412{413    "code": "function createInstance(defaultConfig) {\n  var context = new Axios(defaultConfig);\n  var instance = bind(Axios.prototype.request, context);\n\n  // Copy axios.prototype to instance\n  utils.extend(instance, Axios.prototype, context);\n\n  // Copy context to instance\n  utils.extend(instance, context);\n\n  return instance;\n}", 414    "code_tokens": ["function", "createInstance", "(", "defaultConfig", ")", "{", "var", "context", "=", "new", "Axios", "(", "defaultConfig", ")", ";", "var", "instance", "=", "bind", "(", "Axios", ".", "prototype", ".", "request", ",", "context", ")", ";", "// Copy axios.prototype to instance", "utils", ".", "extend", "(", "instance", ",", "Axios", ".", "prototype", ",", "context", ")", ";", "// Copy context to instance", "utils", ".", "extend", "(", "instance", ",", "context", ")", ";", "return", "instance", ";", "}"], 415    "docstring": "Create an instance of Axios\n\n@param {Object} defaultConfig The default config for the instance\n@return {Axios} A new instance of Axios", 416    "docstring_tokens": ["Create", "an", "instance", "of", "Axios"], 417    "func_name": "createInstance", 418    "id": 0, 419    "language": "javascript", 420    "original_string": "function createInstance(defaultConfig) {\n  var context = new Axios(defaultConfig);\n  var instance = bind(Axios.prototype.request, context);\n\n  // Copy axios.prototype to instance\n  utils.extend(instance, Axios.prototype, context);\n\n  // Copy context to instance\n  utils.extend(instance, context);\n\n  return instance;\n}", 421    "path": "lib/axios.js", 422    "repo": "axios/axios", 423    "sha": "92d231387fe2092f8736bc1746d4caa766b675f5", 424    "url": "https://github.com/axios/axios/blob/92d231387fe2092f8736bc1746d4caa766b675f5/lib/axios.js#L15-L26"425}426```427 428#### php429 430An example of 'train' looks as follows.431```432{433    "code": "public static function build($serviceAddress, $restConfigPath, array $config = [])\n    {\n        $config += [\n            'httpHandler'  => null,\n        ];\n        list($baseUri, $port) = self::normalizeServiceAddress($serviceAddress);\n        $requestBuilder = new RequestBuilder(\"$baseUri:$port\", $restConfigPath);\n        $httpHandler = $config['httpHandler'] ?: self::buildHttpHandlerAsync();\n        return new RestTransport($requestBuilder, $httpHandler);\n    }", 434    "code_tokens": ["public", "static", "function", "build", "(", "$", "serviceAddress", ",", "$", "restConfigPath", ",", "array", "$", "config", "=", "[", "]", ")", "{", "$", "config", "+=", "[", "'httpHandler'", "=>", "null", ",", "]", ";", "list", "(", "$", "baseUri", ",", "$", "port", ")", "=", "self", "::", "normalizeServiceAddress", "(", "$", "serviceAddress", ")", ";", "$", "requestBuilder", "=", "new", "RequestBuilder", "(", "\"$baseUri:$port\"", ",", "$", "restConfigPath", ")", ";", "$", "httpHandler", "=", "$", "config", "[", "'httpHandler'", "]", "?", ":", "self", "::", "buildHttpHandlerAsync", "(", ")", ";", "return", "new", "RestTransport", "(", "$", "requestBuilder", ",", "$", "httpHandler", ")", ";", "}"], 435    "docstring": "Builds a RestTransport.\n\n@param string $serviceAddress\nThe address of the API remote host, for example \"example.googleapis.com\".\n@param string $restConfigPath\nPath to rest config file.\n@param array $config {\nConfig options used to construct the gRPC transport.\n\n@type callable $httpHandler A handler used to deliver PSR-7 requests.\n}\n@return RestTransport\n@throws ValidationException", 436    "docstring_tokens": ["Builds", "a", "RestTransport", "."], 437    "func_name": "RestTransport.build", 438    "id": 0, 439    "language": "php", 440    "original_string": "public static function build($serviceAddress, $restConfigPath, array $config = [])\n    {\n        $config += [\n            'httpHandler'  => null,\n        ];\n        list($baseUri, $port) = self::normalizeServiceAddress($serviceAddress);\n        $requestBuilder = new RequestBuilder(\"$baseUri:$port\", $restConfigPath);\n        $httpHandler = $config['httpHandler'] ?: self::buildHttpHandlerAsync();\n        return new RestTransport($requestBuilder, $httpHandler);\n    }", 441    "path": "src/Transport/RestTransport.php", 442    "repo": "googleapis/gax-php", 443    "sha": "48387fb818c6882296710a2302a0aa973b99afb2", 444    "url": "https://github.com/googleapis/gax-php/blob/48387fb818c6882296710a2302a0aa973b99afb2/src/Transport/RestTransport.php#L85-L94"445}446```447 448#### python449 450An example of 'validation' looks as follows.451```452{453    "code": "def save_act(self, path=None):\n        \"\"\"Save model to a pickle located at `path`\"\"\"\n        if path is None:\n            path = os.path.join(logger.get_dir(), \"model.pkl\")\n\n        with tempfile.TemporaryDirectory() as td:\n            save_variables(os.path.join(td, \"model\"))\n            arc_name = os.path.join(td, \"packed.zip\")\n            with zipfile.ZipFile(arc_name, 'w') as zipf:\n                for root, dirs, files in os.walk(td):\n                    for fname in files:\n                        file_path = os.path.join(root, fname)\n                        if file_path != arc_name:\n                            zipf.write(file_path, os.path.relpath(file_path, td))\n            with open(arc_name, \"rb\") as f:\n                model_data = f.read()\n        with open(path, \"wb\") as f:\n            cloudpickle.dump((model_data, self._act_params), f)", 454    "code_tokens": ["def", "save_act", "(", "self", ",", "path", "=", "None", ")", ":", "if", "path", "is", "None", ":", "path", "=", "os", ".", "path", ".", "join", "(", "logger", ".", "get_dir", "(", ")", ",", "\"model.pkl\"", ")", "with", "tempfile", ".", "TemporaryDirectory", "(", ")", "as", "td", ":", "save_variables", "(", "os", ".", "path", ".", "join", "(", "td", ",", "\"model\"", ")", ")", "arc_name", "=", "os", ".", "path", ".", "join", "(", "td", ",", "\"packed.zip\"", ")", "with", "zipfile", ".", "ZipFile", "(", "arc_name", ",", "'w'", ")", "as", "zipf", ":", "for", "root", ",", "dirs", ",", "files", "in", "os", ".", "walk", "(", "td", ")", ":", "for", "fname", "in", "files", ":", "file_path", "=", "os", ".", "path", ".", "join", "(", "root", ",", "fname", ")", "if", "file_path", "!=", "arc_name", ":", "zipf", ".", "write", "(", "file_path", ",", "os", ".", "path", ".", "relpath", "(", "file_path", ",", "td", ")", ")", "with", "open", "(", "arc_name", ",", "\"rb\"", ")", "as", "f", ":", "model_data", "=", "f", ".", "read", "(", ")", "with", "open", "(", "path", ",", "\"wb\"", ")", "as", "f", ":", "cloudpickle", ".", "dump", "(", "(", "model_data", ",", "self", ".", "_act_params", ")", ",", "f", ")"], 455    "docstring": "Save model to a pickle located at `path`", 456    "docstring_tokens": ["Save", "model", "to", "a", "pickle", "located", "at", "path"], 457    "func_name": "ActWrapper.save_act", 458    "id": 0, 459    "language": "python", 460    "original_string": "def save_act(self, path=None):\n        \"\"\"Save model to a pickle located at `path`\"\"\"\n        if path is None:\n            path = os.path.join(logger.get_dir(), \"model.pkl\")\n\n        with tempfile.TemporaryDirectory() as td:\n            save_variables(os.path.join(td, \"model\"))\n            arc_name = os.path.join(td, \"packed.zip\")\n            with zipfile.ZipFile(arc_name, 'w') as zipf:\n                for root, dirs, files in os.walk(td):\n                    for fname in files:\n                        file_path = os.path.join(root, fname)\n                        if file_path != arc_name:\n                            zipf.write(file_path, os.path.relpath(file_path, td))\n            with open(arc_name, \"rb\") as f:\n                model_data = f.read()\n        with open(path, \"wb\") as f:\n            cloudpickle.dump((model_data, self._act_params), f)", 461    "path": "baselines/deepq/deepq.py", 462    "repo": "openai/baselines", 463    "sha": "3301089b48c42b87b396e246ea3f56fa4bfc9678", 464    "url": "https://github.com/openai/baselines/blob/3301089b48c42b87b396e246ea3f56fa4bfc9678/baselines/deepq/deepq.py#L55-L72"465}466```467 468#### ruby469 470An example of 'train' looks as follows.471```472{473    "code": "def render_body(context, options)\n      if options.key?(:partial)\n        [render_partial(context, options)]\n      else\n        StreamingTemplateRenderer.new(@lookup_context).render(context, options)\n      end\n    end", 474    "code_tokens": ["def", "render_body", "(", "context", ",", "options", ")", "if", "options", ".", "key?", "(", ":partial", ")", "[", "render_partial", "(", "context", ",", "options", ")", "]", "else", "StreamingTemplateRenderer", ".", "new", "(", "@lookup_context", ")", ".", "render", "(", "context", ",", "options", ")", "end", "end"], 475    "docstring": "Render but returns a valid Rack body. If fibers are defined, we return\n a streaming body that renders the template piece by piece.\n\n Note that partials are not supported to be rendered with streaming,\n so in such cases, we just wrap them in an array.", 476    "docstring_tokens": ["Render", "but", "returns", "a", "valid", "Rack", "body", ".", "If", "fibers", "are", "defined", "we", "return", "a", "streaming", "body", "that", "renders", "the", "template", "piece", "by", "piece", "."], 477    "func_name": "ActionView.Renderer.render_body", 478    "id": 0, 479    "language": "ruby", 480    "original_string": "def render_body(context, options)\n      if options.key?(:partial)\n        [render_partial(context, options)]\n      else\n        StreamingTemplateRenderer.new(@lookup_context).render(context, options)\n      end\n    end", 481    "path": "actionview/lib/action_view/renderer/renderer.rb", 482    "repo": "rails/rails", 483    "sha": "85a8bc644be69908f05740a5886ec19cd3679df5", 484    "url": "https://github.com/rails/rails/blob/85a8bc644be69908f05740a5886ec19cd3679df5/actionview/lib/action_view/renderer/renderer.rb#L38-L44"485}486```487 488### Data Fields489 490In the following each data field in go is explained for each config. The data fields are the same among all splits.491 492#### go, java, javascript, php, python, ruby493 494|   field name   |      type      |                                    description                                    |495|----------------|----------------|-----------------------------------------------------------------------------------|496|id              |int32           | Index of the sample                                                               |497|repo            |string          | repo: the owner/repo                                                              |498|path            |string          | path: the full path to the original file                                          |499|func_name       |string          | func_name: the function or method name                                            |500|original_string |string          | original_string: the raw string before tokenization or parsing                    |501|language        |string          | language: the programming language name                                           |502|code            |string          | code/function: the part of the original_string that is code                       |503|code_tokens     |Sequence[string]| code_tokens/function_tokens: tokenized version of code                            |504|docstring       |string          | docstring: the top-level comment or docstring, if it exists in the original string|505|docstring_tokens|Sequence[string]| docstring_tokens: tokenized version of docstring                                  |506|sha             |string          | sha of the file                                                                   |507|url             |string          | url of the file                                                                   |508 509### Data Splits510 511|   name   |train |validation|test |512|----------|-----:|---------:|----:|513|go        |167288|      7325| 8122|514|java      |164923|      5183|10955|515|javascript| 58025|      3885| 3291|516|php       |241241|     12982|14014|517|python    |251820|     13914|14918|518|ruby      | 24927|      1400| 1261|519 520## Dataset Creation521 522### Curation Rationale523 524[More Information Needed]525 526### Source Data527 528#### Initial Data Collection and Normalization529 530Data from CodeSearchNet Challenge dataset.531[More Information Needed]532 533#### Who are the source language producers?534 535Software Engineering developers.536 537### Annotations538 539#### Annotation process540 541[More Information Needed]542 543#### Who are the annotators?544 545[More Information Needed]546 547### Personal and Sensitive Information548 549[More Information Needed]550 551## Considerations for Using the Data552 553### Social Impact of Dataset554 555[More Information Needed]556 557### Discussion of Biases558 559[More Information Needed]560 561### Other Known Limitations562 563[More Information Needed]564 565## Additional Information566 567### Dataset Curators568 569https://github.com/microsoft, https://github.com/madlag570 571### Licensing Information572 573Computational Use of Data Agreement (C-UDA) License.574 575### Citation Information576 577```578@article{husain2019codesearchnet,579  title={Codesearchnet challenge: Evaluating the state of semantic code search},580  author={Husain, Hamel and Wu, Ho-Hsiang and Gazit, Tiferet and Allamanis, Miltiadis and Brockschmidt, Marc},581  journal={arXiv preprint arXiv:1909.09436},582  year={2019}583}584```585 586### Contributions587 588Thanks to @madlag (and partly also @ncoop57) for adding this dataset.