Review: wtt OWASP CRS 3.0 bypass

A while ago I had to make comparison of different Web Application Firewalls based on their security level protection. And as result made WAF Testing Tool (wtt). It is python cli script that sends various attacks to protected application and checks if request is blocked or hits application.
Its work is simple:

parse JSON-files with attack patterns
form HTTP requests
sends requests to application

I have gathered attack samples from different sources like github repositories of different testing tools or articles describing WAF bypass techniques. For now there are 270 SQL Injections, 4810 Cross-Site Scriptings and 115 rest of payloads like Local File Inclusions, Shellshock, XML External Entities etc.

Features

Basic, Form-based or Cookie authentication
Block action detection by HTTP status codes, page content pattern (string, regex)
CSRF-tokens that could be parsed from page content and added to POST-requests
Health-check to detect if client was blocked by IP
Multi-threading
Nice looking HTML-reports

Also it uses keep-alive connections, which adds performance on applications with SSL, since there is no need to establish new HTTPS connection for each request.

Deployment

You need to install python with its dependencies: jinja2 to generate reports and optionally tqdm for nice looking progressbar.
For tests I used ModSecurity with full set of OWASP Core Rules Set 3.0 with Paranoia Level (PL) 1.

Tests

Usage examples

# python wtt.py --help
usage: wtt.py [-h] -f PATH -u URL [-a UA] [-c COOKIE] [-o OUTPUT]
              [-t {status_code,pattern,regex}] [-p PATTERN] [--all]
              [--auth {basic,digest,http}] [--auth-url AUTH_URL]
              [--auth-params AUTH_PARAMS] [--csrf] [--csrf-name CSRF_NAME]
              [--csrf-sendname CSRF_SENDNAME] [--health HEALTH]
              [--report-template REPORT_TEMPLATE] [--threads THREADS]

Waf Testing Tool

optional arguments:
  -h, --help            show this help message and exit
  -f PATH, --folder PATH
                        Path to folder containing attacks samples
  -u URL, --url URL     Set WAF address, e.g.http://127.0.0.1/
  -a UA, --useragent UA
                        Set User-Agent header value
  --referer             Set Referer header value with request hostname
  -c COOKIE, --cookie COOKIE
                        Set Cookies informat key1=value1,key2=value2
  -o OUTPUT, --output OUTPUT
                        Output file (default: report.html)
  -t {status_code,pattern,regex}, --type {status_code,pattern,regex}
                        Detect blocked page based on HTTP response
                        code,pattern or regular expression (default:
                        status_code)
  --codes CODES         List of status codes for blocked pages
                        e.g. 403 or 403,500
  -p PATTERN, --pattern PATTERN
                        Detect blocked page based on status code or pattern
  --all                 Include passed attack to report
  --auth {basic,digest,http}
                        Enable authentication
  --auth-url AUTH_URL   Set url for authentication
  --auth-params AUTH_PARAMS
                        Set authentication parameters in format
                        key1=value1,key2=value2. E.g. for basic and digest
                        auth username should be set as login, password as
                        password: login=john,password=pwd123
  --auth-success AUTH_SUCCESS
                        Regex pattern to find in response and detect if HTTP
                        auth was successful.
  --csrf                Get CSRF-token from response (body, header or cookie)
  --csrf-name CSRF_NAME
                        Param name, that should be found in response
  --csrf-sendname CSRF_SENDNAME
                        Set param name, that is placed in POST-requests if it
                        deffers with csrf-name
  --health HEALTH       Check server availability frequency (default: 10,
                        disable: 0)
  --report-template REPORT_TEMPLATE
                        Set jinja2 report template
  --threads THREADS     Set threads number (default: 1)

Launch string sample

python wtt.py -u http://protected.app.local/ -f owasp --auth http --auth-params login=bee,password=bug,security_level=0,form=submit --auth-url http://172.16.9.34/bWAPP/login.php --auth-success Welcome.Bee --csrf --csrf-name X-Csrf-Token --csrf-sendname csrftoken -o beebox_report.html --all --threads 5

Set blocked attacks criteria

python wtt.py -u http://protected.app.local/ -f owasp -t status_code 403,500
python wtt.py -u http://protected.app.local/ -f owasp -t pattern -p Forbidden
python wtt.py -u http://protected.app.local/ -f owasp -t regex -p ID:.*

Authentication by setting Cookie

python wtt.py -u http://protected.app.local/ -f owasp -c PHPSESSID=1234abcd5678efgh,some_value=1

Basic/Digest authentication

python wtt.py -u http://protected.app.local/ -f owasp --auth basic --auth-params login=John,password=123
python wtt.py -u http://protected.app.local/ -f owasp --auth digest --auth-params login=John,password=123

HTTP authentication

python wtt.py -u http://protected.app.local/ -f owasp --auth http --auth-url http://protected.app.local/bWAPP/login.php --auth-params login=John,password=123,form=Submit --auth-success Welcome.Bee

Get CSRF token for POST-requests

python wtt.py -u http://protected.app.local/ -f owasp --csrf --csrf-name csrftoken

Attack template example

As I said, attacks are stored in JSON-format

{
   "body":"title=adsf'%20or%201%3D1%20--%201&action=search",
   "description":"SQL Injection (POST/Search)",
   "url":"/bWAPP/sqli_1.php",
   "status_code":403,
   "block":true,
   "id":"OWASP_A1_27",
   "headers":{
      "Connection":"keep-alive",
      "Content-Type":"application/x-www-form-urlencoded",
      "User-Agent":"Mozilla/5.0 Windows NT 6.3; Win64; x64 AppleWebKit/537.36 KHTML, like Gecko Chrome/44.0.2403.107 Safari/537.36"
   },
   "method":"POST",
   "payload":"adsf' or 1=1 -- 1"
}

Here you can specify:

body: request body, should be Null for GET requests
description: this field is added to report to understand what attack type was passed
url: additional url path to -u parameter in cli
status_code: expected status code
block: says if attacks should be blocked or not (in case of false positives check)
id: attack id, also added to report, and with this ID it'd be easier to find this attack in file with patterns
headers: additional headers like User-Agent
method: request type (GET, POST, PUT etc.)
payload: urldecoded string, that is added to report

If you have payload samples that should be used with this tool, you can convert them to JSON format with conversion script payload2attacks.py

python payload2attacks.py -i folder_with_payloads -o folder_for_ready_attacks -m get,post
python payload2attacks.py -i folder_with_payloads -o folder_for_ready_attacks -m all

It takes payload from file line by line and adds them to JSON-formatted file for attacks GET-, POST-request or/and Header.

ModSecurity testing results

I was checking only GET and POST requests, since apparently ModSecurity does not perform urldecode of headers and that's why doesn't detect lot of attacks there. Actually, headers are usually not urlencoded but that fits RFC5987, so it would depend on application how to work with headers and in theory may lead to vulnerability.

The script was launched with following parameters:

python wtt.py -f attacks2 -u http://test.app/login.php -t status_code --codes 403

With total of 10390 attacks only 14 were not blocked. There is a list of unique payloads that were met in GET and POST request

././.htaccess
C:/inetpub/wwwroot/global.asa
C:\inetpub\wwwroot\global.asa
%27+%0A%09%09%09%09%09%09};+%0A%09%09%09%09%09%09alert%280%29;+%0A%09%09%09%09%09%09a%20=%20{+%0A%09%09%09%09%09%09%09%27a%27+:+%27
'+alert('XSS')+'&st=1&email=lol@lol.com
'+alert(1)&&null=='

Resume

Even with up-to-date Core Rules Set 3.0, we have found couple of attacks that may bypass ModSecurity. In real life for user it is easier to disable some of the rules, than changing regex patterns to get rid of false positives. So, usually, this amount of valid payloads is higher. However, payloads work only with PL 1, and increasing it will reduce working attacks count, but will also increase false positives.
This WAF Testing Tool is a good way to make quick comparison of possible WAF solutions or check that security level was not decreased during protection mechanisms optimization.