CintraAI/code-chunker

1---2title: CintraAI Code Chunker3emoji: 🧩4colorFrom: blue5colorTo: indigo6sdk: streamlit7sdk_version: 1.33.08app_file: app.py9pinned: false10license: mit11---12 13 14# CintraAI Code Chunker15 16Cintra's Code Chunker is a novel open-source tool designed to enhance code readability and maintainability by intelligently chunking code files based on key points of interest. This tool leverages advanced parsing techniques to identify significant elements in your code, such as functions, classes, and comments, to organize your codebase into manageable, easily understandable chunks. It's an invaluable resource for applications such as RAG, code patching, and other use cases.17 18## Features19 20- **Intelligent Chunking:** Break down your code files into chunks around key points of interest like function definitions, class declarations, and crucial comments.21- **Customizable Token Limits:** Control the size of each chunk with customizable token limits, ensuring that chunks remain manageable and focused.22- **Support for Multiple Languages:** Initially supporting Python, JavaScript, and CSS, with plans to expand to more programming languages.23 24## Try Out Code Chunker!25 26Interested in seeing how it works? Check out our interactive demo on **Hugging Face Spaces**.27 28[**Click here to try it out!**](https://huggingface.co/spaces/CintraAI/code-chunker)29  30## Getting Started31 32### Prerequisites33 34- Python 3.8+35 36### Installation37 381. Clone the repository:39```sh40git clone https://github.com/yourgithubusername/code-chunker.git41```42 432. Navigate to the project directory44```sh45cd code-chunker46```474. Install the required dependencies48```sh49pip install -r requirements.txt50```51## Usage521. Chunking a Code File:53Use the CodeChunker class to chunk a specific code file. You can specify the file extension and token limit for chunking.54Example:55```py56chunker = CodeChunker(file_extension='py', encoding_name='gpt-4')57chunks = chunker.chunk(your_code_here, token_limit=1000)58CodeChunker.print_chunks(chunks)59```602. Parsing Code for Points of Interest:61 62The CodeParser class allows you to parse code to identify points of interest and comments, which can then be used for chunking or other analysis.63Example:64```65parser = CodeParser(['py'])66tree = parser.parse_code(your_code_here, 'py')67points_of_interest = parser.extract_points_of_interest(tree, 'py')68```69 703. Understanding the Token Limit in Chunking:71 72In the `chunk` method of the `Chunker` class, a `token_limit` parameter is used to control the size of each chunk of code. A 'token' can be thought of as the smallest unit of processing. In the context of text processing, a token could be a word, a sentence, or a similar unit.73 74The `token_limit` parameter limits the number of these tokens for each chunk. If the limit is, for instance, 100 tokens, that means each chunk of content produced by the `chunk` method should contain no more than 100 tokens.75 76It is worth noting that the way content is tokenized and how a token is defined depends on the specific implementation and the type of content being processed.77 78## Contributing79We welcome contributions from the community, whether it's through reporting bugs, submitting feature requests, or sending pull requests. Please check the CONTRIBUTING.md file for more details on how to contribute to the project.80 81## License82This project is licensed under the MIT license. See the License file for details83 84## Acknowledgments85- This project utilizes the tree-sitter project for parsing code.86- This also uses tiktoken to count tokens for determining chunk sizes.87 88