jed351/Traditional-Chinese-Common-Crawl-Filtered
Traditional Chinese C4 Dataset Summary Data obtained from 2013～2025 Common Crawl. Downloaded and processed using code based on another project attempting to recreate the C4 dataset. The resultant dataset contains both simplified and traditional Chinese, which could be found here. It was then filtered using a modified list of simplified Chinese characters to obtain this traditional Chinese dataset. Unfortunately, I don't have enough funding to run a deduplication… See the full description on the dataset page: https://huggingface.co/datasets/jed351/Traditional-Chinese-Common-Crawl-Filtered.
Traditional Chinese C4
Dataset Summary
Data obtained from 2013～2025 Common Crawl.
Downloaded and processed using code based on another project attempting to recreate the C4 dataset.
The resultant dataset contains both simplified and traditional Chinese, which could be found here. It was then filtered using a modified list of simplified Chinese characters to obtain this traditional Chinese dataset.
Unfortunately, I don't have enough funding to run a deduplication across all dumps at the moment. I will do it in the future (hopefully).
Acknowoledgement
Special thanks to Eons Data Communications Limited and Votee AI for the compute resources to download and process the Common Crawl Dumps.
Deduplication carried out using compute resource offered under the category of General Projects by Research Insituite for Information Technology, Kyushu University. This work was also partly achieved through the use of SQUID at D3 Center, The University of Osaka.