Apppublic

Underground-Digital/Workflow-Engine

sourceHugging Faceupdated 2y agoView on Hugging Face

0likes

rootapicoreragcleanerunstructuredunstructured_replace_unicode_quotes_cleaner.py

unstructured_replace_unicode_quotes_cleaner.py13 linesDownload Raw Back to unstructured

1"""Abstract interface for document clean implementations."""2 3from core.rag.cleaner.cleaner_base import BaseCleaner4 5 6class UnstructuredNonAsciiCharsCleaner(BaseCleaner):7    def clean(self, content) -> str:8        """Replaces unicode quote characters, such as the \x91 character in a string."""9 10        from unstructured.cleaners.core import replace_unicode_quotes11 12        return replace_unicode_quotes(content)13