Skip to content
tnsaiDocs
GitHub

Detect prompt injection

Prompt injection is text that tries to make the model ignore its instructions, take on another role or leak data. PromptInjectionDetector in tnsai-quality scores text against regex patterns. It is a heuristic filter: it catches known phrasings, not every attack.

The framework does not run the detector for you. @InputGuardrail(detectInjection) is not read at runtime, so call the detector before text reaches the model.

Check user input

static String screen(String userInput) {
    PromptInjectionDetector detector = PromptInjectionDetector.standard();

    InjectionDetectionResult result = detector.detect(userInput);
    if (result.shouldBlock()) {
        return "Refused: " + result.getDetectedTypes() + " (" + result.getConfidenceLevel() + ")";
    }
    return userInput;
}

Each pattern carries a confidence. The result's score is the highest confidence among the patterns that matched, and getMatchedPatterns() describes each match with a short excerpt.

LevelScore
LOWbelow 0.3
MEDIUM0.3 to below 0.6
HIGH0.6 to below 0.8
CRITICAL0.8 and above

result.shouldBlock() uses a fixed threshold of 0.6, and result.shouldBlock(threshold) takes your own. On the detector, shouldBlock(input) uses the detector's threshold, and isSafe(input) is false as soon as any pattern matches.

Tune the detector

static PromptInjectionDetector tuned() {
    return PromptInjectionDetector.builder()
        .addPattern(InjectionType.DATA_EXFILTRATION, "(?i)send .* to https?://", 0.8)
        .confidenceThreshold(0.5)
        .build();
}

standard() uses a threshold of 0.6 and strict() uses 0.5. Added patterns run alongside the built-in ones. The threshold is clamped to the range 0 to 1.

To stop processing on a hit, use throwing() or throwOnDetection(true). detect then throws a SecurityException of type PROMPT_INJECTION when the score reaches the threshold, with the matched patterns as details.

static String guard(String userInput) {
    PromptInjectionDetector detector = PromptInjectionDetector.throwing();
    try {
        detector.detect(userInput);
        return userInput;
    } catch (SecurityException e) {
        return "Refused: " + e.getDetails();
    }
}

Injection types

TypeLooks for
SYSTEM_PROMPT_OVERRIDE"Ignore" or "disregard previous instructions", "forget everything", new instructions, overriding the system prompt
JAILBREAKDAN and "do anything now", developer, jailbreak or unrestricted modes, removing restrictions
ROLE_MANIPULATIONRole reassignment, "pretend", "act as", roleplay, assuming a role
INSTRUCTION_BYPASSSkipping safety filters, bypassing restrictions, "without restrictions", no ethical guidelines
CONTEXT_MANIPULATIONFake system blocks, system delimiters, Markdown or divider lines that open a new system section
DATA_EXFILTRATIONRequests to reveal the prompt, the instructions or the system prompt, memory dumps
HARMFUL_CONTENTWeapon or malware creation, illegal content
CREDENTIAL_EXFILTRATIONReading SSH keys, .env, system or AWS credential files, dumping cloud credential variables, posting or base64-encoding secret files (project files only)
HIDDEN_INSTRUCTIONHidden or invisible HTML text, off-screen blocks, zero-width character clusters, HTML comments carrying an override (project files only)
UNKNOWNNot used by the built-in patterns

Check project files

Instruction files such as CLAUDE.md or AGENTS.md go straight into a model's context, so a planted line there works like a hostile prompt. detectInProjectContext runs the user-input patterns plus the two project-file types above.

static boolean safeToLoad(Path claudeMd) throws Exception {
    String content = Files.readString(claudeMd);
    InjectionDetectionResult result = PromptInjectionDetector.standard()
        .detectInProjectContext(content, ContextFileSource.CLAUDE_MD);
    result.getMatchedPatterns().forEach(System.out::println);   // "[CLAUDE_MD] ..."
    return !result.shouldBlock();
}

ContextFileSource is TNSAI_MD, CLAUDE_MD, AGENTS_MD, README_MD or OTHER; it only labels the matches. A null source throws IllegalArgumentException. Patterns you add with the builder do not apply to the project-file types. The framework does not call this method when it loads files; call it yourself.

Layer the defenses

Pattern matching misses paraphrased and encoded attacks. Combine it with:

  • Moderation, which also has a PROMPT_INJECTION category.
  • Guardrails on input and output.
  • @Security and approvals, so an injected instruction cannot run a sensitive action on its own.
  • A sandbox for anything that runs model-generated code.