So, got an agent in Unreal Engine doing a call to a multi-modal vision text foundation model every 20 seconds or so. The model is being asked "is this interesting or not" if it's interesting it sends instructions to the gameAI (to investigate the object). If not, it executes a turn, and explores another area. I'm doing this for work, but I'm wondering if anyone else is doing this kind of embodied agent work in the game domain? Less LLMs for dialogue, more LLMs for behavior.

There are some of us! I'm CTO at Jam & Tea and we're exploring novel multiplayer gameplay via AI, primarily LLMs. I have my own AI agents, interaction model, ML procgen, etc. all in Unreal 5, so definitely similar to what you're doing. There are some AI x Games communities around that I'm happy to share too.

interesting work. i have some exposure (one hackathon) in unity and self taught in unreal. Can you share the ai & game communities info. thanks. i am happy to try / help what you and Ari are building. Thanks.

@Aaron Farr DM'd you