Update: 3 actual human MIT students just posted a very deep reanalysis of that GPT-4 paper (including finding what appears to be an incorrectly redacted copy of the test questions used in the paper). tl;dr The actual MIT students seem to have pretty well savaged the contents of this paper, including finding that something like 4% of the questions were not answerable because of missing information, so there is no way GPT-4 could have answered them at all, much less correctly. Waiting to read the authors' response when it comes out. https://flower-nutria-41d.notion.site/No-GPT4-can-t-ace-MIT-b27e6796ab5a48368127a98216c76864