39

Detected across 4 services

0

0 critical · 0 high

Isolation Forest + EWMA

Dual-Signal Consensus Gate

Mistral AI

Automated Root Cause

A memory leak in db-proxy v1.23.0 is causing NullPointerExceptions in OrderProcessor.process(), leading to elevated error rates (500s) and high latencies. The abnormal pool utilization (626858%) and connection acquisition times (867ms+) indicate resource exhaustion consistent with past incidents in this service.

The anomaly appears to be caused by an unintentional DEBUG log level change during the v1.20.0 deploy of auth-svc, resulting in excessive DEBUG logs for token validation and OAuth token refreshes without actual service degradation. The presence of 200 status codes, low latency for successful operations (27-60ms), and clustering of DEBUG logs during the deploy window support this diagnosis.

A memory leak or resource exhaustion in db-proxy v1.23.0 is causing elevated error rates (500/502/503), high latencies (54ms to 772ms), and abnormal pool utilization shortly after deployment. This mirrors historical incidents linked to memory leaks in query processing or connection pooling logic.

The anomaly shows successful payment processing with normal latencies (305-367ms) but occurs during a period of reported external payment provider delays (similar to past incidents). The surrounding context shows elevated upstream latencies (3694ms, 1435ms) just before the anomaly window, suggesting upstream provider issues may be affecting the payments-worker despite normal response handling.

The anomaly is likely caused by a memory leak or resource exhaustion in db-proxy v1.23.0, evidenced by elevated error rates (500/502/503), high latencies (76ms to 695ms), and abnormal pool utilization (860316%) shortly after deployment. Similar past incidents (v1.14.0, v1.17.0) show identical patterns linked to memory leaks in query processing or connection pooling logic.

A regression in the newly deployed checkout-api v1.19.0 is causing unhandled exceptions during request handling, resulting in 5xx errors (500, 502, 503) and elevated latency. The pattern matches past incidents where downstream dependencies (payment processor, database, or external APIs) failed post-deployment, causing cascading failures in the checkout flow.

The anomaly shows elevated request latencies (222ms-326ms) with timeout warnings and abnormally high pool utilization percentages (443215%) after a recent deploy of db-proxy v1.17.0. This mirrors past memory leak incidents in query processing or connection pooling logic.

The anomaly is caused by an unintentional DEBUG log level change during the v1.18.0 deploy of auth-svc, resulting in excessive DEBUG logs for token validation, OAuth token refreshes, and login events without actual service degradation. The lack of functional errors (200 status codes, low latency) and clustering of DEBUG logs during deploy windows support this diagnosis.

The payments-worker is experiencing elevated latencies and 504 timeouts due to upstream delays from the external payment provider, as evidenced by consistent 'upstream slow' messages and high latency values (3241ms–4310ms) during a period of reported external API delays.

A regression in v1.15.0 of checkout-api is causing NullPointerExceptions in OrderProcessor.process(), leading to 500 errors and high latency during payment processing. The timing coincides with a recent deploy and matches patterns from past incidents where downstream dependencies failed post-deployment.

The anomaly shows 503 errors and 400 warnings during payment processing in the newly deployed v1.15.0, similar to past incidents where downstream dependencies (payment processor) failed or throttled after deployments. The high latency (3064ms) suggests retry attempts or timeouts.

The anomaly is caused by an unintentional DEBUG log level change during the v1.9.0 deploy of auth-svc, resulting in excessive DEBUG logs for token validation and session creation without actual service degradation. The lack of functional errors (200 status codes, low latency for successful operations) and clustering of DEBUG logs during deploy windows support this diagnosis.

The anomaly is caused by an unintentional DEBUG log level change during the v1.9.0 deploy of auth-svc, resulting in excessive DEBUG logs for token validation and credential stuffing attempts without actual service degradation. The lack of functional errors (200 status codes, low latency) and clustering of DEBUG logs during deploy windows support this diagnosis.

The anomaly is caused by an unintentional DEBUG log level change during the v1.9.0 deploy of auth-svc, resulting in excessive DEBUG logs for token validation and credential stuffing attempts without actual service degradation. The lack of functional errors (200 status codes, low latency) and clustering of DEBUG logs during deploy windows in past incidents support this diagnosis.

The anomaly is likely caused by a memory leak or resource exhaustion in db-proxy v1.14.0, evidenced by elevated retry attempts with 503 errors, high latencies (1675ms to 4344ms), and abnormal pool utilization percentages (696490% to 927029%). This mirrors past incidents linked to memory leaks in query processing or connection pooling logic.

The anomaly is likely caused by an unintentional DEBUG log level change during the v1.9.0 deploy, as evidenced by the clustering of DEBUG logs and lack of functional errors despite high latency. Past incidents show similar patterns during deployments, suggesting a recurring logging configuration issue.

The anomaly shows repeated 503 errors with high latency during retry attempts for downstream calls in the newly deployed v1.12.0, suggesting a regression or dependency issue introduced by the recent deploy. The surrounding context shows normal operations before the anomaly window, and past incidents point to dependency failures or resource exhaustion as likely causes.

The anomaly shows intermittent failures (502, 400) during cart validation in the newly deployed v1.12.0, suggesting a regression or dependency issue introduced by the recent deploy. The surrounding context shows normal operations before the anomaly window, and past incidents point to dependency failures or resource exhaustion as likely causes.

The anomaly is likely caused by a memory leak or resource exhaustion in db-proxy v1.13.0, evidenced by elevated retry attempts with 503 errors, high latencies (1693ms to 4655ms), and connection acquisition delays (4226ms observed in surrounding context). This mirrors past incidents linked to memory leaks in query processing or connection pooling logic.

The anomaly shows normal auth-svc operations (200 status codes, low latency) with an unusual clustering of DEBUG logs during the deploy window, suggesting a logging level change rather than a functional issue. The lack of errors or warnings indicates no actual service degradation.

The anomaly is likely caused by a memory leak or resource exhaustion in db-proxy v1.11.0, evidenced by rapidly increasing connection acquisition times (2264ms to 4611ms), elevated query latencies (2746ms to 4992ms), and a 503 retry warning, similar to a past memory leak incident in v1.8.0.

The payments-worker is experiencing intermittent upstream failures from the external payment provider, as evidenced by 5xx errors (500, 502, 503) and elevated latencies (84ms to 648ms) during a period of reported external API delays (up to 4962ms). The errors are not consistent with internal service degradation.

The anomaly is likely caused by a memory leak in db-proxy v1.8.0, evidenced by rapidly increasing heap usage (85% to 91%) within a minute, combined with elevated query latencies and slow connection acquisition times. This suggests the recent deploy introduced a regression that is consuming memory uncontrollably, leading to performance degradation.

The checkout-api service experienced a memory leak leading to OOM termination, as evidenced by progressively increasing heap usage (85% to 97%) and latency spikes (262ms to 453ms) before the fatal OOM error. The issue correlates with recent deployments and similar past incidents involving upstream dependency failures or resource exhaustion.

The anomaly shows consistent 200ms+ latencies in payment processing during a window where surrounding logs indicate external payment API delays (up to 4962ms) and successful settlements. This suggests upstream payment provider latency spikes rather than service degradation.

The anomaly shows normal auth-svc operations (200 status, low latency) but with an unusual clustering of DEBUG logs during the deploy window, suggesting a logging level change rather than a functional issue. The lack of errors or warnings indicates no actual service degradation.

The anomaly is caused by NullPointerExceptions in OrderProcessor.process() introduced by the recent v1.3.0 deploy of db-proxy, resulting in 500 errors and elevated latencies (214ms-1197ms). Pre-existing database performance issues (high connection acquisition times, slow read queries) likely exacerbated the impact of this regression.

The anomaly shows a single high-latency request (204ms) with an 'upstream slow' warning during a period of otherwise normal auth operations, suggesting a transient upstream dependency issue (e.g., database, cache, or external auth provider) rather than a service-level problem.

The anomaly shows elevated query latencies (up to 3211ms) and slow connection acquisition (836ms) during the anomaly window, with a downstream 503 error indicating service degradation. The surrounding context reveals consistently high connection acquisition times (4017ms-4798ms) and slow read queries (1892ms-2557ms) prior to the anomaly, suggesting pre-existing database performance issues or resource contention exacerbated by the recent v1.3.0 deploy.

The anomaly shows multiple 5xx errors with varying latencies (55ms to 765ms) in the payments-worker service, indicating intermittent upstream failures rather than a consistent service degradation. The presence of both 502 and 503 errors suggests gateway timeouts or upstream service unavailability, while the 500 error with low latency (55ms) indicates a possible race condition or immediate failure in request handling.

Elevated but not extreme payment processing latencies (301-316ms) during a window where most payments settled in <214ms, suggesting potential upstream payment provider delays or database query performance degradation rather than a complete failure.

A downstream service failure caused a 503 retry with extreme latency (4044ms), while other requests completed normally. This suggests a transient dependency issue (payment processor, database, or external API) rather than a code regression.

Elevated but not extreme payment processing latencies (313-443ms) during a window where most payments settled in <214ms, suggesting potential upstream payment provider delays or database query performance degradation rather than a complete failure.

The anomaly shows elevated request latencies (72ms-101ms) with upstream slow warnings in db-proxy, indicating database response time degradation. The surrounding context reveals consistently high connection acquisition times (568ms-3557ms) and slow read queries (2557ms) prior to the anomaly, suggesting pre-existing database performance issues or resource contention.

A sudden burst of high-latency auth requests (400-600ms) with 'upstream slow' warnings suggests a transient upstream dependency issue (e.g., database, cache, or external auth provider) during the v1.2.0 deploy. The OAuth token refreshes and password verifications were likely affected by this slowdown.

The anomaly shows three successful but clustered auth operations (OAuth refreshes and password verification) within an 8-second window, which is unusual compared to the sparse normal traffic pattern. This could indicate either a legitimate burst of user activity or an automated process (e.g., health checks, load testing) triggering auth operations.

The anomaly shows elevated query latencies (240ms, 4422ms) and slow connection acquisition (3557ms) during the anomaly window, suggesting database performance degradation or resource contention. The surrounding context shows consistently high query latencies (1888ms-4922ms) even before the anomaly, indicating a pre-existing issue with the database or its replicas.

The anomaly shows a sudden spike of unhandled exceptions in the checkout-api service during a 4-second window, with mixed 5xx errors (500, 502, 503) and high latency. The absence of recent deployments and lack of similar past incidents suggests a potential transient dependency failure or resource exhaustion rather than a code regression.

The anomaly shows a cluster of payment processing logs with elevated but not extreme latencies (318-418ms) compared to surrounding context where most payments settled in <214ms. The DBSCAN outlier detection suggests these are statistically unusual but not necessarily failures.