To protect the health of your system, USM Anywhere monitors the rate of events being sent to your sensor. If that rate, measured in events per second (EPS), threatens to impact your sensor's capacity USM Anywhere will engage EPS Adaptive Response. EPS Adaptive Response enables your system to take more time to process events coming in by throttling your EPS, keeping your system running without risking event loss.
Note: See Understanding Your Data Consumption Status to read more about sensor capacity and USM Anywhere tier limits.
EPS Adaptive Response may cause delays in the correlation of alarms. While activated, events will enter an adaptive processing queue on sensors or the agent pipeline to assist product stability. When deactivated, the queue will be processed at normal speeds. USM Anywhere views events in the user interface (UI) based on the time the actual event occurred and not the time the event was received by the sensor. When EPS Adaptive Response is ended and queued events get forwarded, they will be backfilled into the appropriate timeslots.
Important: After EPS Adaptive Response has ended, alarms may be delayed while throttled events are being forwarded.
EPS Adaptive Response Scenarios
There are two scenarios in which EPS Adaptive Response may be engaged:
-
Your sensor's disk space is almost full: When your sensor's disk space approaches full, throttling engages to preserve remaining disk space. This is enabled for all customers.
-
Your USM Anywhere is projected over tier: When your USM Anywhere is projected to be over tier, throttling slows down the event ingestion in both sensors and agents until you are back within your tier limits. This is only enabled for heavy usage customers.
Note: Every time EPS Adaptive Response is engaged or disengaged, your USM Anywhere sensor will create a system event. In addition, a system event is created when throttling rates change.
You may also create custom events around throttling to best suit your environment's needs.
Sensor Disk Space
If your sensor's disk volume ever filled up completely, the sensor would stop being able to process events. To prevent this, EPS Adaptive Response slows down your sensor's EPS, giving your system time to process events coming in. As your disk partition continues to fill, the rate of EPS throttling will increase to preserve what remains of your disk space and your sensor's operations. The following table summarizes the throttling rates based on your sensor usage.
| Disk Use (%) | Sensor Throttling (ms) |
|---|---|
| 88 | 0.25 |
| 90 |
0.5 |
| 92 |
2 |
| 95 |
5 |
| 98 |
10 |
| 99 | 100 |
Over Tier Projection
When your USM Anywhere is projected to go over tier, meaning either 5% over your allotment or over 50GB in total, it will analyze the rate of traffic coming through the sensor and agent data pipeline. Then it will engage EPS Adaptive Response to slow down your data rate and keep your USM Anywhere operational until data ingestion is decreased, or your tier is upgraded.
If you have more than one sensor or agent, USM Anywhere will begin by throttling only the sensor with the highest EPS. This is determined by retrieving your system's EPS per minute for every sensor and maintaining a rolling EPS average. Every hour, your system determines if throttling is necessary, and EPS Adaptive Response will be engaged on any sensor sending more EPS than 75% of this average.
When your USM Anywhere is projected to go over tier, EPS throttling is progressive, starting at 1 ms and increasing up to 250 ms as necessary until data ingestion decreases or your tier changes. Once throttling has been engaged, the projection will be sampled regularly. If the tier decrement is smaller than 2%, then the throttling factor is doubled. Otherwise, it remains the same until throttling is no longer necessary.
EPS Adaptive Response System Events
Every time the event throttling value changes in a sensor, a new system event is generated.
There are two system event types:
-
EPS throttling has been engaged: Your sensor's EPS is being throttled.
-
EPS throttling has ended: Your sensor's EPS is no longer being throttled.
Each throttling system event type has a number of possible event keys, specifying which type of event has been triggered.
| Event Type | Event Key | Event Value |
|---|---|---|
| Sensor is being throttled | event_action | SENSOR_THROTTLING |
| event_name |
Sensor is being throttled |
|
| sensor_uuid |
Sensor ID |
|
| customheader_0 |
Throttling value |
|
| customfield_0 |
Throttling value in milliseconds |
|
| Sensor throttling is over | event_action | SENSOR_THROTTLING |
| event_name | Sensor throttling is over | |
| sensor_uuid | Sensor ID | |
| customheader_0 | Throttling value | |
| customfield_0 | 0 |
Feedback