Datasetpublic

NTU-NLP-sg/xCodeEval

The ability to solve problems is a hallmark of intelligence and has been an enduring goal in AI. AI systems that can create programs as solutions to problems or assist developers in writing programs can increase productivity and make programming more accessible. Recently, pre-trained large language models have shown impressive abilities in generating new codes from natural language descriptions, repairing buggy codes, translating codes between languages, and retrieving relevant code segments. However, the evaluation of these models has often been performed in a scattered way on only one or two specific tasks, in a few languages, at a partial granularity (e.g., function) level and in many cases without proper training data. Even more concerning is that in most cases the evaluation of generated codes has been done in terms of mere lexical overlap rather than actual execution whereas semantic similarity (or equivalence) of two code segments depends only on their ``execution similarity'', i.e., being able to get the same output for a given input.

sourceHugging Facecc-by-4.0updated 1y agoView on Hugging Face

83likes42kdownloads

50 commits on main

aa093861y ago

Update README.md

sbmaruf

e99541e2y ago

update split naming

root

467d25a3y ago

Update README.md

jackal1586

4ddc7353y ago

update readme

sbmaruf

a83a46a3y ago

typo

sbmaruf

27431c83y ago

update data loader

sbmaruf

a0c555e3y ago

update code translation

sbmaruf

8b9a67a3y ago

update apr

sbmaruf

43563073y ago

update apr

sbmaruf

7df949c3y ago

Update xCodeEval.py

sbmaruf

594695b3y ago

Update xCodeEval.py

sbmaruf

49161aa3y ago

Update loader.py

sbmaruf

a2114163y ago

Update README.md

sbmaruf

f09b7783y ago

Update README.md

sbmaruf

8dbdf4b3y ago

Update README.md

sbmaruf

1d6da1f3y ago

Update README.md

sbmaruf

b42b91e3y ago

Update README.md

sbmaruf

f1dcb5a3y ago

Update README.md

sbmaruf

19263b13y ago

update hf source

sbmaruf

cc74a7c3y ago

data loader updated

sbmaruf

58b891d3y ago

validation small updated

sbmaruf

c3712963y ago

Fix unittest_db structure

sbmaruf

2b3edeb3y ago

Add file_name feature

sbmaruf

e992da13y ago

sample loader

sbmaruf

8667a0f3y ago

update builder_script

sbmaruf

67709bd3y ago

Fix ./code_translation/train/train_096.jsonl

sbmaruf

e98d22a3y ago

Fix ./code_translation/train/train_082.jsonl

sbmaruf

b6d84b93y ago

Fix ./code_translation/train/train_076.jsonl

sbmaruf

701305e3y ago

Fix ./code_translation/train/train_087.jsonl

sbmaruf

2fa803e3y ago

Fix ./code_translation/train/train_100.jsonl

sbmaruf

c96a5b33y ago

Fix ./code_translation/train/train_083.jsonl

sbmaruf

14db49a3y ago

Fix ./code_translation/train/train_075.jsonl

sbmaruf

153011c3y ago

Fix ./code_translation/train/train_000.jsonl

sbmaruf

f1164563y ago

Fix ./code_translation/train/train_064.jsonl

sbmaruf

4c0b4583y ago

Fix ./code_translation/train/train_109.jsonl

sbmaruf

79817473y ago

Fix ./code_translation/train/train_092.jsonl

sbmaruf

4b39c9a3y ago

Fix ./code_translation/train/train_085.jsonl

sbmaruf

fdbfa513y ago

Fix ./code_translation/train/train_077.jsonl

sbmaruf

4e840bf3y ago

Fix ./code_translation/train/train_066.jsonl

sbmaruf

7d100463y ago

Fix ./code_translation/train/train_028.jsonl

sbmaruf

e9267573y ago

Fix ./code_translation/train/train_006.jsonl

sbmaruf

b1156603y ago

Fix ./code_translation/train/train_095.jsonl

sbmaruf

901216b3y ago

Fix ./code_translation/train/train_046.jsonl

sbmaruf

b3181353y ago

Fix ./code_translation/train/train_102.jsonl

sbmaruf

3eabf2f3y ago

Fix ./code_translation/train/train_002.jsonl

sbmaruf

d2056a13y ago

Fix ./code_translation/train/train_023.jsonl

sbmaruf

ee35c2e3y ago

Fix ./code_translation/train/train_069.jsonl

sbmaruf

6da74e23y ago

Fix ./code_translation/train/train_084.jsonl

sbmaruf

9eae47e3y ago

Fix ./code_translation/train/train_024.jsonl

sbmaruf

dd57d2e3y ago

Fix ./code_translation/train/train_013.jsonl

sbmaruf