Folks my Vercel Edge-> LangChainStream -> ChatOpenAI chatbot setup, streaming is bugging out and losing conversation mid-stream after I add a system message with 8k tokens mid chat.
Has anyone seen this streaming issue with LangChainStream -> ChatOpenAI packages before?
My current hypotheses are:
1. There might be ChatOpenAI issues with heavily loaded context?
2. Is it possible that free tier Vercel is causing this weird behavior?
3. Is adding context via system message mid-chat the issue? Would it be better to start a new chat altogether?
Any suggestions/intuitions would be helpful. cc: @Hao Sun
08/17/2023, 8:26 PM
@Jeevan Gyawali Just wondering if you made progress on this. Purely curiosity. My initial gut check was that:
• you exhausted the “initial response” time from Edge or
• you blew past the context window
but you should be catching errors to that effect