Comment on page
How do decisions work, exactly?
Bulwark uses a powerful technique called Dempster-Shafer theory, which enables the system to automatically handle uncertainty and make informed choices even when there is incomplete or conflicting information.
Bulwark's decision structure is tailored specifically for web application security, and it simplifies the concepts of Dempster-Shafer theory to be more easily understood by users who may not have prior knowledge of it.
When Bulwark makes decisions, it considers three main values:
unknown. These values represent different possibilities or outcomes based on the evidence available. The
allowvalue indicates the degree to which evidence supports accepting a request, while the
restrictvalue represents the degree to which evidence supports restricting or blocking a request. The
unknownvalue reflects uncertainty or the lack of evidence for either outcome.
It's important to note that uncertainty is not an actionable outcome, even though it is quantitatively represented. Also, while these component values share notable similarities with probabilities, they are not probabilities themselves. They may be treated as rough estimates and plugin authors may usefully assign them values that represent a well-informed guess.
In the example above, the
acceptvalue is 0.0, the
restrictvalue is 0.4, and the
unknownor power set value is 0.6. In this example, the decision's components indicate a higher confidence for a
restrictor blocking outcome than an
acceptoutcome, but also indicate substantial uncertainty.
To combine decisions effectively, Bulwark uses the Murphy combination rule. This rule was chosen for its ability to handle scenarios where plugin decisions may have significant conflict or disagreement. It also has good performance characteristics and is straightforward to implement. Importantly, it doesn't assume that its plugin decision inputs are independent, as independence cannot be guaranteed. It also ensures that not-a-number (NaN) values are never produced. The specific rule used for combination is opaque to the plugin API.
Bulwark provides a "score" value to report the decisions made by individual plugins and the combined ensemble. This score is calculated using a technique called a pignistic transformation. The algorithm evenly distributes the
unknowncomponent value by dividing it in half and redistributing to the
restrictvalues. Then, for the purposes of reporting a single number, the
allowvalue is disregarded, and the
restrictvalue becomes the reported score.
Scores in Bulwark are reported in terms of the
restrictvalue, following the common practice in security and fraud tools where higher scores indicate a greater level of risk.
Score values offer a better way to apply a decision against a threshold because they provide a single value with a clear meaning associated with the midpoint of the range. The range midpoint, 0.5, represents maximum uncertainty, where there is equal evidence on both sides of the decision. Values above the midpoint indicate more evidence in favor of a
restrictoutcome, while values below it indicate more evidence in favor of an
Considering the previous example decision values, to convert to a score we would divide the
unknownvalue of 0.6 by 2 and then add the result, 0.3, to both the
restrictvalues. The score would then be the new
restrictvalue of 0.7.
End-users of Bulwark have the flexibility to choose appropriate thresholds that align with their risk-tolerance for false positives and false negatives. These thresholds can be set independently of the scores reported by the ensemble.
Individual decision scores can be re-weighted to tune the results from the ensemble. The weighting algorithm involves discarding the
unknownvalue, multiplying the
restrictvalues by the weighting factor, and then scaling the result to ensure it falls within the valid range. Any remainder between 1.0 and the sum of the other two components is assigned back to the
unknownvalue. This approach maintains the relative relationship between
restrictvalues and guarantees a valid result. Weighting by a number less than 1.0 will increase that decision's uncertainty, and may be used to discount a decision value. Weighting by a number greater than 1.0 will reduce a decision's uncertainty, and may be used to boost a decision value, up to the cap, where it will generally be clamped.
acceptvalue of 0.3, a
restrictvalue of 0.2, and an
unknownor power set value of 0.5, weighting this by a multiple of 0.5 will result in a new
acceptvalue of 0.15, a new
restrictvalue of 0.1, and a new
unknownvalue of 0.75.