rombodawg/data_processing_code

These are the scripts I used to clean the rombodawg/code_bagel and rombodawg/code_bagel_hermes-2.5 datasets. In order for these scripts to work your datasets need to be in the format bellow, with the condition that each line is its own .json object. {"instruction": "", "input": "", "output": ""} {"instruction": "", "input": "", "output": ""} {"instruction": "", "input": "", "output": ""} {"instruction": "", "input": "", "output": ""} {"instruction": "", "input": "", "output": ""}