Skip to content

Fix memory leak with FastThrow optimization - #12254

Open
jpbempel wants to merge 2 commits into
masterfrom
jpbempel/fix-er-fastthrow
Open

Fix memory leak with FastThrow optimization#12254
jpbempel wants to merge 2 commits into
masterfrom
jpbempel/fix-er-fastthrow

Conversation

@jpbempel

@jpbempel jpbempel commented Aug 21, 2026

Copy link
Copy Markdown
Member

What Does This Do

The fix is to clear the snapshot list after sending the snapshot and tagging the span. We also add a cap to the number of snapshots (256) added to the ThrowableState object.

Motivation

HotSpot JVM has a special optimization when some special exceptions are (implicitly) thrown like NullPointerException. The JIT will replace a throw with a newly created Exception that walk the stack to produce the stacktrace by a throw with a singleton instance of the exception without stacktrace.
This singleton exception is kept forever by the JVm contrary to usual way to throw exception to create the object at location of the exception.
Therefore the map snapshotsByThrowable in ExceptionProbeManager which maps Exception instance to ThrowableState object keep an instance forever with the along snapshots collected.
At some point the number of snapshots kept can lead to OOME.

Additional Notes

Contributor Checklist

Jira ticket: DEBUG-6056

HotSpot JVM has a special optimization when some special exceptions
are (implicitly) thrown like NullPointerException.
The JIT will replace a throw with a newly created Exception that
walk the stack to produce the stacktrace by a throw with a singleton
instance of the exception without stacktrace.
This singleton exception is kept forever by the JVm contrary to usual
way to throw exception to create the object at location of the
exception.
Therefore the map snapshotsByThrowable in ExceptionProbeManager which
maps Exception instance to ThrowableState object keep an instance
forever with the along snapshots collected.
At some point the number of snapshots kept can lead to OOME.

The fix is to clear the snapshot list after sending the snapshot and
tagging the span. We also add a cap to the number of snapshots (256)
added to the ThrowableState object.
@jpbempel
jpbempel requested a review from a team as a code owner August 21, 2026 11:53
@jpbempel
jpbempel requested review from dudikeleti and removed request for a team August 21, 2026 11:53
@dd-octo-sts

dd-octo-sts Bot commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

  • Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

@jpbempel jpbempel added comp: debugger Dynamic Instrumentation type: bug fix Bug fix labels Aug 21, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 15c924f41d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: FAIL

Clearing the shared list leaves a reused FastThrow instance with an empty state, so later error spans lose snapshot and replay tags. The same list also has unsafe concurrent access, and its new check permits more than 256 entries.

Open Bits AI session

🤖 Datadog Autotest · Commit 15c924f · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

@datadog-prod-us1-4

datadog-prod-us1-4 Bot commented Aug 21, 2026

Copy link
Copy Markdown

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 58.41% (-0.21%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a882216 | Docs | View more details | Give us feedback!

@pr-commenter

pr-commenter Bot commented Aug 21, 2026

Copy link
Copy Markdown

Debugger benchmarks

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
ci_job_date 1787326681 1787327007
end_time 2026-08-21T15:39:29 2026-08-21T15:44:54
git_branch master jpbempel/fix-er-fastthrow
git_commit_sha 926e82f a882216
start_time 2026-08-21T15:38:02 2026-08-21T15:43:28
See matching parameters
Baseline Candidate
ci_job_id 1973014570 1973014570
ci_pipeline_id 132425097 132425097
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
git_commit_date 1787325958 1787325958

Summary

Found 5 performance improvements and 0 performance regressions! Performance is the same for 4 metrics, 6 unstable metrics.

scenario Δ mean agg_http_req_duration_min Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p75 Δ mean agg_http_req_duration_p99 Δ mean throughput
scenario:loop better
[-567.983µs; -472.190µs] or [-7.075%; -5.882%]
better
[-578.084µs; -481.663µs] or [-7.141%; -5.950%]
better
[-576.373µs; -476.092µs] or [-7.090%; -5.856%]
better
[-636.482µs; -462.786µs] or [-7.713%; -5.608%]
better
[+6.290op/s; +11.026op/s] or [+5.190%; +9.096%]
See unchanged results
scenario Δ mean agg_http_req_duration_min Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p75 Δ mean agg_http_req_duration_p99 Δ mean throughput
scenario:noprobe unstable
[-46.367µs; +42.771µs] or [-14.350%; +13.237%]
unstable
[-58858.021ns; +57100.704ns] or [-16.453%; +15.962%]
unstable
[-66702.833ns; +66144.731ns] or [-17.907%; +17.757%]
unstable
[-202.054µs; +362.897µs] or [-16.582%; +29.782%]
same
scenario:basic unsure
[-15.938µs; -2.624µs] or [-5.301%; -0.873%]
unsure
[-15.419µs; -1.839µs] or [-4.735%; -0.565%]
unsure
[-14.896µs; -0.797µs] or [-4.448%; -0.238%]
unstable
[-151.690µs; +125.260µs] or [-14.045%; +11.598%]
unstable
[-82.506op/s; +193.248op/s] or [-3.548%; +8.310%]
Request duration reports for reports
gantt
    title reports - request duration [CI 0.99] : candidate=None, baseline=None
    dateFormat X
    axisFormat %s
section baseline
noprobe (357.74 µs) : 305, 411
.   : milestone, 358,
basic (325.65 µs) : 320, 332
.   : milestone, 326,
loop (8.095 ms) : 8032, 8158
.   : milestone, 8095,
section candidate
noprobe (356.861 µs) : 300, 414
.   : milestone, 357,
basic (317.02 µs) : 310, 324
.   : milestone, 317,
loop (7.565 ms) : 7558, 7573
.   : milestone, 7565,
Loading
  • baseline results
Scenario Request median duration [CI 0.99]
noprobe 357.74 µs [304.891 µs, 410.588 µs]
basic 325.65 µs [319.755 µs, 331.544 µs]
loop 8.095 ms [8.032 ms, 8.158 ms]
  • candidate results
Scenario Request median duration [CI 0.99]
noprobe 356.861 µs [299.608 µs, 414.114 µs]
basic 317.02 µs [309.758 µs, 324.283 µs]
loop 7.565 ms [7.558 ms, 7.573 ms]

@dd-octo-sts

dd-octo-sts Bot commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.99 s 13.97 s [-0.8%; +1.0%] (no difference)
startup:insecure-bank:tracing:Agent 12.88 s 12.95 s [-1.1%; +0.1%] (no difference)
startup:petclinic:appsec:Agent 17.49 s 17.22 s [+0.6%; +2.4%] (maybe worse)
startup:petclinic:iast:Agent 17.36 s 17.49 s [-1.6%; +0.0%] (no difference)
startup:petclinic:profiling:Agent 17.40 s 17.22 s [-0.0%; +2.2%] (no difference)
startup:petclinic:sca:Agent 17.45 s 17.33 s [-0.3%; +1.7%] (no difference)
startup:petclinic:tracing:Agent 16.66 s 16.54 s [-0.3%; +1.8%] (no difference)

Commit: a8822169 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@dudikeleti dudikeleti left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It’s worth looking at the bot comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: debugger Dynamic Instrumentation type: bug fix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants