CaraJ/MMSearch

MMSearch 🔥: Benchmarking the Potential of Large Models as Multi-modal Search Engines

Official repository for the paper "MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines".

🌟 For more details, please refer to the project page with dataset exploration and visualization tools: https://mmsearch.github.io/.

[🌐 Webpage] [📖 Paper] [🤗 Huggingface Dataset] [🏆 Leaderboard] [🔍 Visualization]

💥 News

• —[2024.09.25] 🌟 The evaluation code now supports directly use models implemented in VLMEvalKit!
• —[2024.09.22] 🔥 We release the evaluation code, which you only need to add an inference API of your LMM!
• —[2024.09.20] 🚀 We release the arXiv paper and all MMSearch data samples in huggingface dataset.

📌 ToDo

• —Coming soon: MMSearch-Engine (for new query)

👀 About MMSearch

The capabilities of Large Multi-modal Models (LMMs) in multimodal search remain insufficiently explored and evaluated. To fill the blank of a framework for LMM to conduct multimodal AI search engine, we first design a delicate pipeline MMSearch-Engine to facilitate any LMM to function as a multimodal AI search engine

<p align="center"> <img src="https://github.com/CaraJ7/MMSearch/raw/main/figs/fig1.png" width="75%"> <br> The overview of <b>MMSearch-Engine</b>. </p>

To further evaluate the potential of LMMs in the multimodal search domain, we introduce MMSearch, an all-around multimodal search benchmark designed for assessing the multimodal search performance. The benchmark contains 300 manually collected instances spanning 14 subfields, which involves no overlap with the current LMMs' training data, ensuring the correct answer can only be obtained within searching.

<p align="center"> <img src="https://raw.githubusercontent.com/CaraJ7/MMSearch/main/figs/fig2.png" width="60%"> <br> The overview of <b>MMSearch</b>. </p>

In addition, we propose a step-wise evaluation strategy to better understand the LMMs' searching capability. The models are evaluated by performing three individual tasks (requery, rerank, and summarization), and one challenging end-to-end task with a complete searching process. The final score is weighted by the four tasks.

<p align="center"> <img src="https://raw.githubusercontent.com/CaraJ7/MMSearch/main/figs/fig3.png" width="90%"> <br> Outline of Evaluation Tasks, Inputs, and Outputs. </p>

An example of LMM input, output, and ground truth for four evaluation tasks is shown here.

🏆 Leaderboard

Contributing to the Leaderboard

🚨 The Leaderboard is continuously being updated, welcoming the contribution of your excellent LMMs!

:whitecheckmark: Citation

If you find MMSearch useful for your research and applications, please kindly cite using this BibTeX:

@article{jiang2024mmsearch,
  title={MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines},
  author={Jiang, Dongzhi and Zhang, Renrui and Guo, Ziyu and Wu, Yanmin and Lei, Jiayi and Qiu, Pengshuo and Lu, Pan and Chen, Zehui and Song, Guanglu and Gao, Peng and others},
  journal={arXiv preprint arXiv:2409.12959},
  year={2024}
}

🧠 Related Work

Explore our additional research on Vision-Language Large Models:

• —[MathVerse] MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?
• —[MathVista] MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
• —[LLaMA-Adapter] LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention
• —[LLaMA-Adapter V2] LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model
• —[ImageBind-LLM] Imagebind-LLM: Multi-modality Instruction Tuning
• —[SPHINX] The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal LLMs
• —[SPHINX-X] Scaling Data and Parameters for a Family of Multi-modal Large Language Models
• —[Point-Bind & Point-LLM] Multi-modality 3D Understanding, Generation, and Instruction Following
• —[PerSAM] Personalize segment anything model with one shot
• —[CoMat] CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching