“I’ve been part of this community for a while and have gained a lot from your insights and discussons. Today, I’m excited to share a project I’ve been working on…”, so I am making a guess that OP’s intention is to do as he received.
The project is RAG on 1B embedded vectors from sources like WP, ArXiV, and common crawl. They release the search engine, the code for the search engine, and the model that is trained to provide accurate responses. The contribution has three parts.
They also commit to this: “AgentSearch and Sensei will be valuable for the open source community, especially in scenarios where you need to perform a large number of search queries. The dataset is big and we plan to keep expanding it, adding more key sources relevant to LLM agents. If you have any suggestions for what sources to include, feel free to reach out.”
People ask whether it supports getting real-time data, and the dev points to where the commenter should go to implement that.
One commenter potentially inspires the dev in exploring how something similar could be done but with local files.
To one comment that was asking for benchmarks, they answer that since RAG benchmarks aren’t at a place that they want it to be, so they are writing an academic paper with the technology to push the RAG benchmark space.