These prompt injection attacks on Bing Chat and Chat GPT are absolutely fascinating. Shows that there’s more to the large language models than just “the most likely sequence of paragraphs following this prompt”. Reminds me of the concept of “emergence” where what you get is more than the sum of the individual parts.
What I find most fascinating about this isn’t so much the prompt injection attacks themselves, but rather that these systems are configured (programmed?) using a bullet list of sentences.