prompt hacking
English
Etymology
Noun
- (artificial intelligence, Internet, neologism) Synonym of prompt injection.
- Hypernyms: prompting, prompt engineering, (used loosely) hacking
- Coordinate term: jailbreaking
- 2023 October 19, Aayush Mittal, “Prompt Hacking and Misuse of LLMs”, in Unite.AI[1]:
- At its core, prompt hacking involves manipulating the input to a model to obtain a desired, and sometimes unintended, output. Given the right prompts, even a well-trained model can produce misleading or malicious results. […] Three primary techniques for manipulating prompts are: prompt injections, prompt leaking, and jailbreaking.
- 2024 March 3, “Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition”, in arXiv[3]:
- This paper focuses on prompt hacking in an application-grounded setting (Figure 1): a LLM is instructed to perform a downstream task (e.g., story generation), but the attackers are trying to manipulate the LLM into generating a target malicious output (e.g., a key phrase).
Usage notes
Sometimes (see the first quotation) considered a hypernym of prompt injection, in the strict sense of definition 1 of the latter, and/or of jailbreaking. However, there is often not a clear line between the different strategies and these terms tend to be used interchangeably.
Alternative forms
- prompt hack (countable)
Related terms
- prompt hacker