I’ve been digging into <https://github.com/deepset...
# 05-ai-newsj
Jim Park
05/17/2023, 2:32 PMI’ve been digging into https://github.com/deepset-ai/haystack it feels more mature than langchain.
👀 1
d
Daniel Hsu
05/17/2023, 4:23 PMI've also been pretty curious how it performs in comparison, especially for the targeted use cases it has like semantic retrieval
Daniel Hsu
05/17/2023, 4:23 PMHave you built anything with it yet?
h
Han
05/17/2023, 4:34 PMNeither one scales
Han
05/17/2023, 4:35 PMHaystack is for information retreieval
Langchain is for chaining models together
Han
05/17/2023, 4:37 PMIf you need to parse/store a whole lot of data into a knowledge base / search engine / etc, langchain is absolutely the wrong tool to use and is light years behind.
d
Daniel Hsu
05/17/2023, 7:19 PM@Han any recommendations for better tools for knowledge retrieval?
h
Han
05/17/2023, 7:20 PMBoth indexing and retrieval are in the realm of search engine / information retrieval.
Depending on the need, e.g., long term vs short term memory, there are different tools suitable for the use cases
Han
05/17/2023, 7:24 PMLong term memory, e.g., a lot of data, you will be better off using existing data systems if it's already there. Or use modern search engines like Elasticsearch, Opensearch, Weaviate, Vespa, etc.
Short term memory, e.g., something that's short term, in memory, for sessions, e.g., baybAGI/autoGPT type of stuff, duckDB, ChromaDB (basically duckdb + hnswlib), Redis, or just directly use an Numpy array.
Han
05/17/2023, 7:26 PMThere's also CozoDB that works in Python using either in-memory or sqlite backend, but it's not as "popular" outside of the embedded db people
j
Jim Park
05/19/2023, 6:06 PM@Daniel Hsu, I've only just begun building https://github.com/CogniQ/CogniQ/tree/jimp/issue/17.
To @Han’s point, I think this project, and the frameworks I'm using all warrant performance rewrites at sufficient scale or pain.
I'm open for critical feedback if you have any 🙏
👾 1
5 Views