Team Ai
Modelpublic

nlpai-lab/KoE5

sourceHugging Facemitupdated 2mo agoView on Hugging Face
52likes53kdownloads
Model Card

πŸ”Ž KoE5

Introducing KoE5, a model with advanced retrieval abilities. It has shown remarkable performance in Korean text retrieval.

For details, visit the KURE repository


Model Versions

Model NameDimensionSequence LengthIntroduction
KURE-v110248192Fine-tuned BAAI/bge-m3 with Korean data via CachedGISTEmbedLoss
KoE51024512Fine-tuned intfloat/multilingual-e5-large with ko-triplet-v1.0 via CachedMultipleNegativesRankingLoss

Model Description

This is the model card of a πŸ€— transformers model that has been pushed on the Hub.

Example code

Install Dependencies

First install the Sentence Transformers library:

bash
pip install -U sentence-transformers

Python code

Then you can load this model and run inference.

python
from sentence_transformers import SentenceTransformer

# Download from the πŸ€— Hub
model = SentenceTransformer("nlpai-lab/KoE5")

# Run inference
sentences = [
    'query: ν—Œλ²•κ³Ό 법원쑰직법은 μ–΄λ–€ 방식을 톡해 기본ꢌ 보μž₯ λ“±μ˜ λ‹€μ–‘ν•œ 법적 λͺ¨μƒ‰μ„ κ°€λŠ₯ν•˜κ²Œ ν–ˆμ–΄',
    'passage: 4. μ‹œμ‚¬μ κ³Ό κ°œμ„ λ°©ν–₯ μ•žμ„œ μ‚΄νŽ΄λ³Έ 바와 같이 우리 ν—Œλ²•κ³Ό r법원쑰직 법」은 λŒ€λ²•μ› ꡬ성을 λ‹€μ–‘ν™”ν•˜μ—¬ 기본ꢌ 보μž₯κ³Ό 민주주의 확립에 μžˆμ–΄ 닀각적인 법적 λͺ¨μƒ‰μ„ κ°€λŠ₯ν•˜κ²Œ ν•˜λŠ” 것을 κ·Όλ³Έ κ·œλ²”μœΌλ‘œ ν•˜κ³  μžˆλ‹€. λ”μš±μ΄ ν•©μ˜μ²΄λ‘œμ„œμ˜ λŒ€λ²•μ› 원리λ₯Ό μ±„νƒν•˜κ³  μžˆλŠ” 것 μ—­μ‹œ κ·Έ κ΅¬μ„±μ˜ 닀양성을 μš”μ²­ν•˜λŠ” κ²ƒμœΌλ‘œ ν•΄μ„λœλ‹€. 이와 같은 κ΄€μ μ—μ„œ λ³Ό λ•Œ ν˜„μ§ 법원μž₯κΈ‰ κ³ μœ„λ²•κ΄€μ„ μ€‘μ‹¬μœΌλ‘œ λŒ€λ²•μ›μ„ κ΅¬μ„±ν•˜λŠ” 관행은 κ°œμ„ ν•  ν•„μš”κ°€ μžˆλŠ” κ²ƒμœΌλ‘œ 보인닀.',
    'passage: β–‘ μ—°λ°©ν—Œλ²•μž¬νŒμ†ŒλŠ” 2001λ…„ 1μ›” 24일 5:3의 λ‹€μˆ˜κ²¬ν•΄λ‘œ γ€Œλ²•μ›μ‘°μ§λ²•γ€ 제169μ‘° 제2문이 ν—Œλ²•μ— ν•©μΉ˜λœλ‹€λŠ” νŒκ²°μ„ λ‚΄λ ΈμŒ β—‹ 5인의 λ‹€μˆ˜ μž¬νŒκ΄€μ€ μ†Œμ†‘κ΄€κ³„μΈμ˜ 인격ꢌ 보호, κ³΅μ •ν•œ 절차의 보μž₯κ³Ό λ°©ν•΄λ°›μ§€ μ•ŠλŠ” 법과 μ§„μ‹€ 발견 등을 근거둜 ν•˜μ—¬ ν…”λ ˆλΉ„μ „ μ΄¬μ˜μ— λŒ€ν•œ μ ˆλŒ€μ μΈ κΈˆμ§€λ₯Ό ν—Œλ²•μ— ν•©μΉ˜ν•˜λŠ” κ²ƒμœΌλ‘œ λ³΄μ•˜μŒ β—‹ κ·ΈλŸ¬λ‚˜ λ‚˜λ¨Έμ§€ 3인의 μž¬νŒκ΄€μ€ ν–‰μ •λ²•μ›μ˜ μ†Œμ†‘μ ˆμ°¨λŠ” νŠΉλ³„ν•œ 인격ꢌ 보호의 이읡도 μ—†μœΌλ©°, ν…”λ ˆλΉ„μ „ 곡개주의둜 인해 법과 μ§„μ‹€ 발견의 과정이 μ–Έμ œλ‚˜ μœ„νƒœλ‘­κ²Œ λ˜λŠ” 것은 μ•„λ‹ˆλΌλ©΄μ„œ λ°˜λŒ€μ˜κ²¬μ„ μ œμ‹œν•¨ β—‹ μ™œλƒν•˜λ©΄ ν–‰μ •λ²•μ›μ˜ μ†Œμ†‘μ ˆμ°¨μ—μ„œλŠ” μ†Œμ†‘λ‹Ήμ‚¬μžκ°€ 개인적으둜 직접 심리에 μ°Έμ„ν•˜κΈ°λ³΄λ‹€λŠ” λ³€ν˜Έμ‚¬κ°€ μ°Έμ„ν•˜λŠ” κ²½μš°κ°€ 많으며, μ‹¬λ¦¬λŒ€μƒλ„ μ‚¬μ‹€λ¬Έμ œκ°€ μ•„λ‹Œ 법λ₯ λ¬Έμ œκ°€ λŒ€λΆ€λΆ„μ΄κΈ° λ•Œλ¬Έμ΄λΌλŠ” κ²ƒμž„ β–‘ ν•œνŽΈ, μ—°λ°©ν—Œλ²•μž¬νŒμ†ŒλŠ” γ€Œμ—°λ°©ν—Œλ²•μž¬νŒμ†Œλ²•γ€(Bundesverfassungsgerichtsgesetz: BVerfGG) 제17a쑰에 따라 μ œν•œμ μ΄λ‚˜λ§ˆ μž¬νŒμ— λŒ€ν•œ 방솑을 ν—ˆμš©ν•˜κ³  있음 β—‹ γ€Œμ—°λ°©ν—Œλ²•μž¬νŒμ†Œλ²•γ€ 제17μ‘°μ—μ„œ γ€Œλ²•μ›μ‘°μ§λ²•γ€ 제14절 λ‚΄μ§€ 제16절의 κ·œμ •μ„ μ€€μš©ν•˜λ„λ‘ ν•˜κ³  μžˆμ§€λ§Œ, λ…ΉμŒμ΄λ‚˜ μ΄¬μ˜μ„ ν†΅ν•œ μž¬νŒκ³΅κ°œμ™€ κ΄€λ ¨ν•˜μ—¬μ„œλŠ” γ€Œλ²•μ›μ‘°μ§λ²•γ€κ³Ό λ‹€λ₯Έ λ‚΄μš©μ„ κ·œμ •ν•˜κ³  있음',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.6721, 0.3897],
#        [0.6721, 1.0000, 0.3740],
#        [0.3897, 0.3740, 1.0000]])

Training Details

Training Data

  • β€”ko-triplet-v1.0
  • β€”Korean query-document-hard_negative data pair (open data)
  • β€”About 700000+ examples used totally

Training Procedure

  • β€”loss: Used [CachedMultipleNegativesRankingLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedmultiplenegativesrankingloss) by sentence-transformers
  • β€”batch size: 512
  • β€”learning rate: 1e-05
  • β€”epochs: 1

Evaluation

Metrics

  • β€”Recall, Precision, NDCG, F1

Benchmark Datasets

  • β€”Ko-StrategyQA: ν•œκ΅­μ–΄ ODQA multi-hop 검색 데이터셋 (StrategyQA λ²ˆμ—­)
  • β€”AutoRAGRetrieval: 금육, 곡곡, 의료, 법λ₯ , 컀머슀 5개 뢄야에 λŒ€ν•΄, pdfλ₯Ό νŒŒμ‹±ν•˜μ—¬ κ΅¬μ„±ν•œ ν•œκ΅­μ–΄ λ¬Έμ„œ 검색 데이터셋
  • β€”MIRACLRetrieval): Wikipedia 기반의 ν•œκ΅­μ–΄ λ¬Έμ„œ 검색 데이터셋
  • β€”PublicHealthQA): 의료 및 곡쀑보건 도메인에 λŒ€ν•œ ν•œκ΅­μ–΄ λ¬Έμ„œ 검색 데이터셋
  • β€”BelebeleRetrieval): FLORES-200 기반의 ν•œκ΅­μ–΄ λ¬Έμ„œ 검색 데이터셋
  • β€”MrTidyRetrieval: Wikipedia 기반의 ν•œκ΅­μ–΄ λ¬Έμ„œ 검색 데이터셋
  • β€”MultiLongDocRetrieval: λ‹€μ–‘ν•œ λ„λ©”μΈμ˜ ν•œκ΅­μ–΄ μž₯λ¬Έ 검색 데이터셋
  • β€”XPQARetrieval: λ‹€μ–‘ν•œ λ„λ©”μΈμ˜ ν•œκ΅­μ–΄ λ¬Έμ„œ 검색 데이터셋

Results

μ•„λž˜λŠ” λͺ¨λ“  λͺ¨λΈμ˜, λͺ¨λ“  벀치마크 데이터셋에 λŒ€ν•œ 평균 κ²°κ³Όμž…λ‹ˆλ‹€. μžμ„Έν•œ κ²°κ³ΌλŠ” KURE Githubμ—μ„œ ν™•μΈν•˜μ‹€ 수 μžˆμŠ΅λ‹ˆλ‹€.

Top-k 1

ModelAverage Recall_top1Average Precision_top1Average NDCG_top1Average F1_top1
nlpai-lab/KURE-v10.526400.605510.605510.55784
dragonkue/BGE-m3-ko0.523610.603940.603940.55535
BAAI/bge-m30.517780.598460.598460.54998
Snowflake/snowflake-arctic-embed-l-v2.00.512460.593840.593840.54489
nlpai-lab/KoE50.501570.577900.577900.53178
intfloat/multilingual-e5-large0.500520.577270.577270.53122
jinaai/jina-embeddings-v30.482870.560680.560680.51361
BAAI/bge-multilingual-gemma20.479040.554720.554720.50916
intfloat/multilingual-e5-large-instruct0.478420.554350.554350.50826
intfloat/multilingual-e5-base0.469500.544900.544900.49947
intfloat/e5-mistral-7b-instruct0.467720.543940.543940.49781
Alibaba-NLP/gte-multilingual-base0.464690.537440.537440.49353
Alibaba-NLP/gte-Qwen2-7B-instruct0.466330.536250.536250.49429
openai/text-embedding-3-large0.448840.516880.516880.47572
Salesforce/SFR-Embedding-2_R0.437480.508150.508150.46504
upskyy/bge-m3-korean0.431250.502450.502450.45945
jhgan/ko-sroberta-multitask0.337880.384970.384970.35678

Top-k 3

ModelAverage Recall_top1Average Precision_top1Average NDCG_top1Average F1_top1
nlpai-lab/KURE-v10.686780.287110.655380.39835
dragonkue/BGE-m3-ko0.678340.283850.649500.39378
BAAI/bge-m30.675260.283740.645560.39291
Snowflake/snowflake-arctic-embed-l-v2.00.671280.281930.640420.39072
intfloat/multilingual-e5-large0.658070.277770.628220.38423
nlpai-lab/KoE50.651740.273290.623690.37882
BAAI/bge-multilingual-gemma20.644150.274160.611050.37782
jinaai/jina-embeddings-v30.641160.271650.609540.37511
intfloat/multilingual-e5-large-instruct0.643530.270400.607900.37453
Alibaba-NLP/gte-multilingual-base0.637440.264040.596950.36764
Alibaba-NLP/gte-Qwen2-7B-instruct0.631630.259370.592370.36263
intfloat/multilingual-e5-base0.620990.261440.591790.36203
intfloat/e5-mistral-7b-instruct0.620870.261440.589170.36188
openai/text-embedding-3-large0.610350.253560.573290.35270
Salesforce/SFR-Embedding-2_R0.600010.252530.563460.34952
upskyy/bge-m3-korean0.592150.250760.557220.34623
jhgan/ko-sroberta-multitask0.469300.189940.432930.26696

Top-k 5

ModelAverage Recall_top1Average Precision_top1Average NDCG_top1Average F1_top1
nlpai-lab/KURE-v10.738510.191300.674790.29903
dragonkue/BGE-m3-ko0.725170.187990.666920.29401
BAAI/bge-m30.729540.189750.666150.29632
Snowflake/snowflake-arctic-embed-l-v2.00.729620.188750.662360.29542
nlpai-lab/KoE50.708200.182870.644990.28628
intfloat/multilingual-e5-large0.701240.183160.644020.28588
BAAI/bge-multilingual-gemma20.702580.185560.633380.28851
jinaai/jina-embeddings-v30.699330.182560.631330.28505
intfloat/multilingual-e5-large-instruct0.690180.178380.624860.27933
Alibaba-NLP/gte-multilingual-base0.693650.177890.618960.27879
intfloat/multilingual-e5-base0.672500.174060.611190.27247
Alibaba-NLP/gte-Qwen2-7B-instruct0.674470.171140.609520.26943
intfloat/e5-mistral-7b-instruct0.674490.174840.609350.27349
openai/text-embedding-3-large0.663650.170040.593890.26677
Salesforce/SFR-Embedding-2_R0.656220.170180.584940.26612
upskyy/bge-m3-korean0.654770.170150.580730.26589
jhgan/ko-sroberta-multitask0.531360.132640.458790.20976

Top-k 10

ModelAverage Recall_top1Average Precision_top1Average NDCG_top1Average F1_top1
nlpai-lab/KURE-v10.796820.106240.694730.18524
dragonkue/BGE-m3-ko0.784500.104920.687480.18288
BAAI/bge-m30.791950.105920.687230.18456
Snowflake/snowflake-arctic-embed-l-v2.00.786690.104620.681890.18260
intfloat/multilingual-e5-large0.759020.101470.663700.17693
nlpai-lab/KoE50.752960.099370.660120.17369
BAAI/bge-multilingual-gemma20.761530.103640.653300.18003
jinaai/jina-embeddings-v30.762770.102400.652900.17843
intfloat/multilingual-e5-large-instruct0.748510.098880.644510.17283
Alibaba-NLP/gte-multilingual-base0.756310.099380.640250.17363
Alibaba-NLP/gte-Qwen2-7B-instruct0.740920.096070.632580.16847
intfloat/multilingual-e5-base0.735120.097170.632160.16977
intfloat/e5-mistral-7b-instruct0.737950.097770.630760.17078
openai/text-embedding-3-large0.729460.095710.616700.16739
Salesforce/SFR-Embedding-2_R0.716620.095460.605890.16651
upskyy/bge-m3-korean0.718950.095830.602580.16712
jhgan/ko-sroberta-multitask0.612250.078260.486870.13757

<br/>

FAQ

- Do I need to add the prefix "query: " and "passage: " to input texts?

Yes, this is how the model is trained, otherwise you will see a performance degradation.

Here are some rules of thumb:

  • β€”Use "query: " and "passage: " correspondingly for asymmetric tasks such as passage retrieval in open QA, ad-hoc information retrieval.
  • β€”Use "query: " prefix for symmetric tasks such as semantic similarity, bitext mining, paraphrase retrieval.
  • β€”Use "query: " prefix if you want to use embeddings as features, such as linear probing classification, clustering.

Citation

If you find our paper or models helpful, please consider cite as follows:

text
@inproceedings{jang2025kure,
  title={KURE: Embedding Model for Korean-Specific Retrieval},
  author={Jang, Youngjoon and Son, Junyoung and Lee, Taemin and Hong, Seongtae and Park, JeongBae and Lim, Heuiseok},
  booktitle={Annual Conference on Human and Language Technology},
  pages={129--134},
  year={2025},
  organization={Human and Language Technology}
},

@inproceedings{jang2024koe5,
  title={KoE5: A New Dataset and Model for Improving Korean Embedding Performance},
  author={Jang, Youngjoon and Son, Junyoung and Park, Chanjun and Choi, Soonwoo and Lee, Byeonggoo and Lee, Taemin and Lim, Heuiseok},
  booktitle={Annual Conference on Human and Language Technology},
  pages={239--244},
  year={2024},
  organization={Human and Language Technology}
}

Limitations

Long texts will be truncated to at most 512 tokens.