JLine: ReDoS in Nano Editor Regex Search Mode

Summary

When regex search mode is enabled in the JLine3 nano editor, the user-supplied
search term is compiled directly as a Java regular expression with no timeout or
backtracking bound. A crafted pattern such as (a+)+b can hang the editor session
thread at high CPU, causing a denial of service for that session.

Details

In builtins/src/main/java/org/jline/builtins/Nano.java, the search implementation
uses Pattern.LITERAL only when regex mode is disabled. When regex mode is enabled,
the search term is compiled as a raw Java regex:

Pattern pat = Pattern.compile(
        searchTerm,
        (searchCaseSensitive ? 0 : Pattern.CASE_INSENSITIVE | Pattern.UNICODE_CASE)
                | (searchRegexp ? 0 : Pattern.LITERAL));

This regex is then applied to buffer content. Because Java's regex engine is
backtracking-based, nested-quantifier patterns can take exponential time on
non-matching input.

Affected source location:

• builtins/src/main/java/org/jline/builtins/Nano.java
• doSearch(String text)

PoC

1. Create a file containing a long run of a characters and open it in the JLine3
   nano editor.
2. Enable regex search mode with the editor's regex toggle.
3. Start a search and enter the pattern (a+)+b.

Expected result:

• The editor stops responding.
• The session thread consumes high CPU.

Reproduction environment:

• JLine3 on x86_64 Linux
• OpenJDK 25.0.2

Impact

This is a denial-of-service vulnerability caused by catastrophic regex backtracking.
Applications embedding org.jline:jline-builtins and exposing the nano editor are
impacted. In local use, the user can hang their own session. In remote multi-user
deployments, an attacker can occupy a server worker thread indefinitely.

Suggested Fix

The preferred fix for the current git head is to use a linear-time regex engine for
regex search mode while preserving literal matching behavior when regex mode is off.

Suggested patch:

diff --git a/builtins/pom.xml b/builtins/pom.xml
--- a/builtins/pom.xml
+++ b/builtins/pom.xml
@@
         <dependency>
+            <groupId>com.google.re2j</groupId>
+            <artifactId>re2j</artifactId>
+            <version>1.8</version>
+        </dependency>
+        <dependency>
             <groupId>org.jline</groupId>
             <artifactId>jline-reader</artifactId>
         </dependency>

diff --git a/builtins/src/main/java/org/jline/builtins/Nano.java b/builtins/src/main/java/org/jline/builtins/Nano.java
--- a/builtins/src/main/java/org/jline/builtins/Nano.java
+++ b/builtins/src/main/java/org/jline/builtins/Nano.java
@@
-import java.util.regex.Pattern;
+import com.google.re2j.Pattern;

If a dependency change is not acceptable, a fallback mitigation is to reject dangerous
regex constructs or execute regex matching with a strict timeout, but that is weaker
than replacing the backtracking engine.

Credits

This issue was identified by Michał Majchrowicz and Marcin Wyczechowski, members of the AFINE Team.