System Throttling
- Nitin Beri (Unlicensed)
Many activities of a process flow may load complete data in memory, and when the process flow executes these activities then the system may move out of the memory and process flow gets aborted. Adeptia provides a solution to this problem by running a resource availability check and pausing the process flow until the system gets ready with required memory. This would help the system from being choked and process flow getting aborted. This feature is available only with Enterprise license. Contact Adeptia Support for upgrade if your current version does not support Intelligent System Throttling.
Intelligent system throttling once enabled helps pausing specific type of activity of a process flow. Following activities can be paused in a PF if throttling is enable.
- JSON Schema
- Excel Schema
- Data Mapping
Exceptions
System throttling will not work in the following cases:
- If your licence does not support System Throttling. In this case, even if you enable System Throttling, the PF will not pause or stop but run like it does in a normal scenario.
- If Data Mapping is in streaming mode, then data mapping activity cannot be paused.
- If XLSX schema is using streaming, then this activity will not pause.
- If the process flow is set as Optimized for Real Time, system Throttling will not work.
Enabling System Throttling
By default, System Throttling is disabled in Adeptia Suite. To enable intelligent system throttling:
- Login to Adeptia Suite.
- Go to Administer > Setup > Application Settings > Update System Properties.
- Click Load Management to display the System Throttling option.
- Click System Throttling.
- Go to abpm.node.pfpause.feature.enabledProperty and set the value to true to enable System Throttling.
- Restart Kernel and WebRunner.
- You can set other properties that will be used in effectively using System Throttling. You can read the Description of each property to set its values.
Points to Note
- System throttling works on FIFO principle, that is, it would start the Process Flow that was paused first irrespective of the memory available for queued process flows.
- In case, there is no repository disk space, then the system instantly fails the process flow instead of pausing it.
- The system would fail the process flow if the Mode is in multiplier and the calculative memory size is greater than the provided memory by JVM.
Properties
Following is the list of properties where you can set the values to effectively use system throttling.
| Property Name | Description with Possible Values |
|---|---|
| abpm.node.resource.cpu.free | Set the threshold value (1-100) for CPU usage (free CPU in percentage). This property works only when resource-based round robin algorithm (or PF-Pause feature) is used. Write -1 in the property, if you want system to skip checking of CPU. Possible values: Any positive integer (e.g. 10) or -1 |
| abpm.node.resource.memory.free | Set the threshold value for the free memory (in MB). This property works only when resource-based-round-robin algorithm is used. Write -1 in the property, if you want system to skip checking of memory. Possible values: Any positive integer (e.g. 10) or -1 |
| abpm.node.resource.repositorydiskspace.free | Set the threshold value for free Repository Disk Space (in MB). This property works only when resource-based-round-robin algorithm is used. Write -1 in the property, if you want system to skip checking of repository disk space. Possible values: Any positive integer (e.g. 10) or -1 |
| abpm.node.resourceavailability.check.waitingperiod | Set the time interval (in seconds) here. System will for wait for this given time (when a job could not be allocated in the first try) while it checks again for the available resources at the nodes of a cluster. This property works only when resource-based round robin algorithm is used. Possible value: Any positive integer (e.g. 3) |
| abpm.node.resourceavailability.check.waitingperiod.multiplier | Provide the value for waiting period multiplier here. Given value exponentially multiplies with the time interval set for checking system resource availability. For example, if you set the value as 2 here and 10 seconds in the waiting period, then system will wait for 20 seconds before it checks for the resource again. In the following re-tries it would wait for 40, then 80, and 160 seconds until it finds the resource. This property works only when resource-based round robin algorithm is used. Possible values: Any positive integer (e.g. 2) |
| abpm.node.pfpause.feature.enabled | Write true to enable the process flow pause feature. Possible values: true or false. |
| abpm.node.pfpause.restrictParallelPFExecution | Write true to enable this property. Enabling this property restricts the execution of one or more parallel process flows while PF Pause feature is enabled. Possible values: true or false. |
| abpm.node.pfpause.count.parallelPFExecution | Set the process flow count that you want to run in parallel while restriction is enabled. For example, if you set the value to 2, system will allow execute two process flows in parallel. Possible values: Any positive integer (e.g. 2) |
| abpm.node.pfpause.resourceavailability.check.waitingperiod.multiplier | Set the multiplier number for time interval set for checking system resource availability, it multiplies the waiting interval exponentially. For example, if you set the value as 2 here and 10 seconds in the waiting period, then system will wait for 20 (10*2) seconds before it checks for the resource again. In the following re-tries it would wait for 40, then 80, and 160 seconds until it finds the resource. This property works only when process flow pause feature is enabled. Possible values: Any positive integer (e.g. 2) |
| abpm.node.pfpause.resourceavailability.check.waitingperiod | Set the time interval (in seconds) here. System will for wait for the given time (when an activity could not be executed due to lack for resources) while it checks again for the available resources. This property works only when process flow pause feature is enabled. Possible value: Any positive integer (e.g. 3) |
| abpm.node.pfpause.retries.count | Total count of retries for checking the system resources of nodes when an activity could not be executed (due to lack of system resources). Possible value: Any positive integer (e.g. 120) This property works only when process flow pause feature is enabled. |
| abpm.node.pfpause.count.threshold | Set the number for total counts of paused process flow. Once system has reached the given threshold of pause process flow count, no new job will be executed on the node till the paused process flow count doesn't come down. Possible value: Any positive integer (e.g. 120) |
| abpm.node.pfpause.mode | This value determines how the required data size will be calculated. If MULTIPLIER, the data size will be input data size * value and if FIXED, the data size will be the fixed value. This value will work with the values set in the following three properties (explained later in the table).
For example, if you have set the value as FIXED and gave 1024 MB as the value in the properties above, then system will execute the activity only when memory of 1024 or greater is available. Else it will wait for 1024 MB memory to be available. If you set the value as MULTIPLIER and gave 1024 MB as the value in the properties above and file received is 2 MB, then system will execute the activity only when memory of 2048 MB (1024*2) or greater is available. Else it will wait for 2048 MB memory to be available. Possible values: MULTIPLIER or FIXED |
| abpm.node.pfpause.mappingThreshold | Set the threshold value for data size upto which no resource (memory) check is required. For example, if you have set the value as 2 then the system will not pause the PF for a file with size 2MB or less. Possible values: Any positive integer (e.g. 2) |
| abpm.node.pfpause.excelSchemaThreshold | Set the threshold value for data size upto which no resource (memory) check is required. For example, if you have set the value as 2 then the system will not pause the PF for a file with size 2MB or less. Possible values: Any positive integer (e.g. 2) |
| abpm.node.pfpause.jsonSchemaThreshold | Set the threshold value for data size upto which no resource (memory) check is required. For example, if you have set the value as 2 then the system will not pause the PF for a file with size 2MB or less. Possible values: Any positive integer (e.g. 2) |
| abpm.node.pfpause.mappingFixedOrMultiplierValue | Set the value for memory (in MB) required to process 1 MB. Refer to abpm.node.pfpause.mode property description above in this table. If you want to skip this check than provide -1 as value. Possible value: Any positive integer (e.g. 1024) or -1 This value will be used based on PF pause mode configuration. |
| abpm.node.pfpause.excelSchemaFixedOrMultiplierValue | Set the value for memory (in MB) required to process 1 MB. Refer to abpm.node.pfpause.mode property description above in this table. If you want to skip this check than provide -1 as value. Possible value: Any positive integer (e.g. 1024) or -1 This value will be used based on PF pause mode configuration. |
| abpm.node.pfpause.jsonSchemaFixedOrMultiplierValue | Set the value for memory (in MB) required to process 1 MB. Refer to abpm.node.pfpause.mode property description above in this table. If you want to skip this check than provide -1 as value. Possible value: Any positive integer (e.g. 1024) or -1 This value will be used based on PF pause mode configuration. |
See Also