prompt hacking

English

Etymology

    From prompt + hacking.

    Noun

    prompt hacking (uncountable)

    1. (artificial intelligence, Internet, neologism) Synonym of prompt injection.
      Hypernyms: prompting, prompt engineering, (used loosely) hacking
      Coordinate term: jailbreaking
      • 2023 October 19, Aayush Mittal, “Prompt Hacking and Misuse of LLMs”, in Unite.AI‎[1]:
        At its core, prompt hacking involves manipulating the input to a model to obtain a desired, and sometimes unintended, output. Given the right prompts, even a well-trained model can produce misleading or malicious results. […] Three primary techniques for manipulating prompts are: prompt injections, prompt leaking, and jailbreaking.
      • 2023 November 11, Marc Rodriguez Sanz, “Prompt Hacking: The Trojan Horse of the AI Age. How to Protect Your Organization”, in Medium‎[2]:
        In essence, prompt hacking is the crafty manipulation of prompts to skew LLM responses.
      • 2024 March 3, “Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition”, in arXiv‎[3]:
        This paper focuses on prompt hacking in an application-grounded setting (Figure 1): a LLM is instructed to perform a downstream task (e.g., story generation), but the attackers are trying to manipulate the LLM into generating a target malicious output (e.g., a key phrase).

    Usage notes

    Sometimes (see the first quotation) considered a hypernym of prompt injection, in the strict sense of definition 1 of the latter, and/or of jailbreaking. However, there is often not a clear line between the different strategies and these terms tend to be used interchangeably.

    Alternative forms

    • prompt hack (countable)
    • prompt hacker