here

deadhorsecapableInternet και Εφαρμογές Web

8 Δεκ 2013 (πριν από 3 χρόνια και 7 μήνες)

128 εμφανίσεις

Who am
I?

Past:


Commercial WAF developer since 2007


ModSecurity

maintainer 2007


2010


IDS/IPS Developer (OISF
Suricata
)


Present:


Lead WAF development @
Qualys

in Madison


IronBee

architect and developer


Lockdown 2013

2

What am I covering…


Briefly: The what and why of WAF


IronBee
, modules and rules


Overview of
Qualys

commercial WAF (beta)


How we use
IronBee


How we have simplified the process


Beta features

Lockdown 2013

3

WAF: What
is
it?


Web Application Firewall


To many this means:


“Block web based attacks.”



But, WAFs are known to be a pain


There must be more

Lockdown 2013

4

WAF: Why does it exist?


View inside your web applications


Log (and potentially block) suspicious activity


Block known and obvious attacks and tools


Limit attack surface


Buy time to fix problems


Feed your developers with more details


Deal with legacy products

Lockdown 2013

5

WAF: What gets in the way?


D
ifferent
interpretations of HTTP


D
ocument
types (HTML, XML,
JSON)


Encodings
(URL, Base64,
entities)


Different Vectors
(server, browser, DB,
DoS
)


Evasion
techniques


Application logic (
auth
, sessions, BI)


Encryption, compression
, obfuscation

Lockdown 2013

6

WAF: What can go wrong?


False Positives (oops)


False Negatives (didn't see it)


Performance cannot suffer (too much)


Device failure (site is down)

Lockdown 2013

7

WAF: How can we make it better?


Easier
to
setup and manage


Separate server/security
configs

and management


Low False Positives and low tuning costs


Flexible deployments with automated updates


Manage it all centrally


Extensible engine


Solid framework for writing security logic


Integrate with other products


Combine many advanced techniques with correlation


Acceptable performance


Intelligent application of security logic with fast algorithms

Lockdown 2013

8

IronBee

: What is it?


Open Source (Apache Software License v2
)


github.com
/
ironbee



Framework
to
inspect, block, modify and log


Extremely
flexible


Highly extensible


Tries
not to
get in your way

Lockdown 2013

9

IronBee
: Who is involved?

Christopher
Alfeld
, PhD
Mathematics and UW alumni

Experimental
projects, performance, algorithms, C+
+ API

Sam
Baskinger

Data
structures, Configuration,
Lua API

Nick Kew
, Apache Foundation

Server plugins: Apache
Trafficserver
, Apache
httpd
,
nginx
,
tserver
, …

Nick
LeRoy

Core engine, Testing

Brian Rectanus

Initial
IronBee

author, now architect and manager

Ivan
Ristić

Security Research (SSL Labs


ssllabs.com
,
LibHTP
,
ModSecurity
)


Many other supporting players at
Qualys



too many to name here.



Lockdown 2013

10

IronBee
: What's the basic concept?


Server provides HTTP data


Web server, proxy, IDS, …


Parsers break data into fields/streams


Headers, URI, POST body, cookies, …


Modules/Rules inspect these fields/streams


Sigs, scoring, tracking, learning, correlation, …


Actions performed:


L
og, block, modify, track, …

Lockdown 2013

11

IronBee
: What's a server?


P
rovide HTTP data to
IronBee


Implement blocking, modification (if possible)


Current:


Apache
Trafficserver

plugin


Apache Webserver module


Nginx

plugin


Tserver

(
nginx

fork) plugin


Clipp

(command line with PCAP support)

Lockdown 2013

12

IronBee
: What's the engine do?


Notification of events


Core HTTP fields to inspect


Rule execution


Configuration


Logging


Very minimalistic, and becoming more so.

Lockdown 2013

13

IronBee
: What are modules?


Dynamically loadable shared libraries in C, C++


Minimal modules in Lua, but reloadable with
config


Hook into
IronBee

events


Extend functionality (C/C++ only), such as:


Parsers, normalizers, operators and actions


Rule
languages (and extensions)


Embed
scripting languages (Lua)


Enable technologies
(
libinjection

-

SQLi

detection library)


Correlation (combine sigs, scoring, tracking, learning, …)


Logging




Lockdown 2013

14

IronBee
: What are rules?


Inspect data and perform actions


Simple signature language


Complex DSL (Lua @
config

time)


Full scripting language (Lua @ runtime)


Extendible via modules

Lockdown 2013

15

Module: Simple Rule Language

Specify
fields,
inspect and perform an
action:


Rule <fields> <op> <meta/actions>

Rule
REQUEST_HEADERS

\



@
rx

"
attack
|
pattern
"

\



id:ex
/1 rev:
1
\



phase:REQUEST_HEADER

\



event



Lockdown 2013

16

Module: Simple Rule Language

Transformations and meta data:


Rule
REQUEST_HEADERS.
count
()

\



@
gt

15

\



id:ex
/2 rev:1
\



phase:REQUEST_HEADER

\



severity:75 confidence:80

\



tag:http
/limits

\



event




Lockdown 2013

17

Module: Simple Rule Language

C
apture potential CC#s, blocking more than 10:


StreamInspect

RESPONSE_BODY_STREAM
\



@
dfa

"
\
d{15,16}"

\



id:ex
/3 rev:1
\



capture:CC


Rule
CC.count
()

\



@
gt

10

\



id:ex
/4 rev:1
\



phase:RESPONSE_BODY

\



event
block:immediate



Lockdown 2013

18

Module: Simple Rule Language


These are just signature rules


Simple

and come with limitations


Config

file syntax (single line)


Somewhat verbose (requires id/phase)


No real flow control other than phase/file order


Other types of rules eliminate these limits



Lockdown 2013

19

Module: Lua


Embedded scripting language


As a configuration DSL (
config

time)


As a basic module (core engine runtime)


As a rule (rule engine runtime)

Lockdown 2013

20

Lua: As a DSL

DSL is named "waggle" (we like Bee themes here)


Rule REQUEST_HEADERS
\



@
rx

"
attack
|
pattern
"
\



id:ex
/1 rev:1
\



phase:REQUEST_HEADER

\



event


Sig
("ex
/
1w",
1):



fields
("REQUEST_HEADERS
"
)
:



op
("
rx
"
, "
attack
|
pattern
"
)
:



phase
("REQUEST_HEADER")
:



action(
"
event")

Lockdown 2013

21

Lua: Programmatic Rules
C
onfig

Lua @
config

time means full support for functions, loops, etc.


--

Parameterized rule with id/regex

local
function
RequestRegex
(id, regex)



return Sig("test/
lua
/" .. id, 1):


fields("REQUEST_HEADERS”):


op("
rx
", regex):


phase("REQUEST"):


actions("event”)

end


--

Simplify management and readability

RequestRegex
(1, [[
attack|pattern
]])

RequestRegex
(2, [[attack2|pattern2]])


Lockdown 2013

22

Lua: Basic Modules

Lua executed at runtime to handle core engine events.


--

Get the
IronBee

Module object.

local
ibmod

= ...


--

Define a function to handle an event.

local
function
log_event
(
ib
)


ib:logInfo
("
Handling event=%s”,
ib.event_name
)


return
0

end


--

Register
to
be called with the event.

ibmod
:request_header_finished_event
(
log_event
)

Lockdown 2013

23

Lua: Rules


Similar to Lua module, but less complex


Lua executed by the rule execution engine


Entire script runs vs. using event callbacks

Lockdown 2013

24

Rules: Scaling to the non
-
trivial


Simple linear execution with basic rules


Executes a list of rules per phase


All rules are executed


What about 1000s or 100,000s of rules?


Need a way to limit execution


Need a way to specify dependencies/order


Need a way to cache results


Need a higher level of logic and correlation

Lockdown 2013

25

Rules: Made to be extended


Rule injection


M
odules can take ownership of rules


Modules can decide if/when rules execute


Currently two modules use this facility


Fast rules module


Predicate rules module

Lockdown 2013

26

Module: Fast Rules


Adds a fast pattern (prequalification) to rules


Rules are executed only if prequalified


All fast rules utilize modified
Aho
-
Corasick


E
xtensions to utilize fixed width patterns


Speed is independent of number of patterns


Works best with large
rulesets


Some limitations

Lockdown 2013

27

Fast Rules: An example

Utility suggests fast patterns for existing
rules by
adding comments to rules


#
FAST RE: ^(.+),
\
s*
max
-
age[^,]+
,?(.*)$

#
FAST Suggest: "
fast:max
-
age
[^,]"

Rule
RESPONSE_HEADERS:Cache
-
Control

\


@
rx

"^(.+),
\
s*
max
-
age[^,]+
,?(.*)$
"





Rule
RESPONSE_HEADERS:Cache
-
Control

\


@
rx

"^(.+),
\
s*
max
-
age[^,]+
,?(.*)
$
"


"
fast:max
-
age
[^,]"





Lockdown 2013

28

Module: Predicate Rules


Uses Lua DSL to produce predicate expressions

(and (
gt

(
atoi

(field 'Content
-
Length')) 0
)



(
streq

'GET' (field 'Request
-
Method')
) )


Complex rules are built from simple rules


Rules form an knowledge graph


Graph optimizations performed at configuration time


Common sub
-
expression merging & caching


O
nly required rules execute, and only once


Combines Lua DSL and runtime optimizations


Full Lua support enhances configuration


Graph optimizations enhance runtime

Lockdown 2013

29

Predicate Rules: Named predicates

--

Parameterized named predicate

local
function header(name)


return
P.Field
('REQUEST_HEADERS'):sub(name)

end

--

Named predicates

local
range_header_too_long

=
P.Gt
(
header('Range')
:length(), 1000
)

local
host_header_too_long

=
P.Gt
(
header('Host')
:length(), 100
)


--

Combine named predicates into a rule/signature

--

NOTE: A "/" operator is overloaded for predicates to
P.Or
(…)

Sig(”ex/p/1", 1):


predicate
(
range_header_too_long

/
host_header_too_long

)
:


phase([[REQUEST_HEADER]]):


action([[event]]):


message([[Invalid HTTP header: too long.]])


Lockdown 2013

30

Predicate Rules: Lua DSL in action

local
sensitive_file_patterns

= {


unix

= [[(?:/
etc
/
passwd
|/
etc
/hosts|/
etc
/shadow|/bin/
id)
$]],


java = [[(?:WEB
-
INF/
web.xml
|/
conf
/
server.xml
)$]],


apache = [[(?:.
htaccess
|.
htpasswd
|.
meta|.web
)$]]

}

local
function
contains_sensitive_files
(pattern)


local r =
P.false


for
i,v

in
ipairs
(
{"REQUEST_URI_PATH", "REQUEST_HEADERS", "ARGS"}
)


r =
P.Or
(r,
P.rx
(pattern,
P.Field
(v):
remove_whitespace
()
))


end


return r

end

for
name,pattern

in pairs(
sensitive_file_patterns
) do


Sig("
qrs
/
LFi
/" .. name, "1"):


predicate(
contains_sensitive_files
(pattern)
):


phase([[REQUEST_HEADER]]):


action([[event]]):


message("
LFi
: request for sensitive " .. name .. " files.")

end

Lockdown 2013

31

Framework: Automata


Iron Automata (we also like Iron themes here)


Framework and
utils

for building automata


Splits generation, optimization, execution


Generic execution environment, Eudoxus


Example Automata: Enhanced
Aho
-
Corasick


Caseless

matches


Fixed width patterns/sets (char sets, negation


Can be tuned for space
vs

time through Eudoxus

Lockdown 2013

32

IronAutomata
:
Aho
-
Corasick

Example1

Lockdown 2013

33


Aho
-
Corasick


Unoptimized


Patterns:


he


she


his


hers

IronAutomata
:
Aho
-
Corasick

Example2

Lockdown 2013

34


Aho
-
Corasick


Speed
Optimized


Patterns:


he


she


his


hers

IronAutomata
: Optimization

Lockdown 2013

35


Aho
-
Corasick


Patterns:

~250k English Dictionary


Data:

Text of "Pride and
Predjudice
" novel 10x

Module: Eudoxus Executor

Execute compiled, eudoxus automata.


Large signature database


Spam keywords


Known attack patterns


Link reputation


Custom, auto generated automata


B
ased on research


Based on website traffic profiling

Lockdown 2013

36

Utility:
Clipp


Command line utility


Testing and rule development


HTTP data via: Raw files, PCAP,
protobuf
, …


Modify HTTP data via filters


Convert between formats


Highly extendible


Ruby wrapper for unit/regression testing

Lockdown 2013

37

IronBee
:
B
atteries not included


Management is not dictated, so…


No
Config

Management


No Rule Management


No Log Management


M
ust do these yourself


You should already be doing this


T
he point is to stay out of your way


Allow you to use your own management tools

Lockdown 2013

38

Qualys

WAF: What will it add?


Managed WAF appliances via cloud


Automated updates


Software


Modules


Rules


Integration with other
Qualys

products


Web Application Scanning


Asset Management

Lockdown 2013

39

Qualys

WAF Beta: What's offered?


Initially Amazon Web Services Platform


EC2 Classic and VPC


Clustering via ELB


Auto
-
scaling


You decide how much power you need


We are expanding to other platforms

Lockdown 2013

40

Qualys

Beta WAF: What's it do?


Manage AWS based WAF Appliances


Generic attack detection


Declarative security (
fixup

cookies/headers)


Data leakage detection


Reduce attack surface
(
HTTP limitations)


ACLs (IP and geo)


Lockdown 2013

41

Qualys

WAF Beta: What's it look like?

Lockdown 2013

42


Manage AWS
Appliances


Manage events


Generic attack detection


Declarative
security


Data leakage detection


Reduce attack
surface


Access Control


Qualys

WAF Beta:
AppSec

Lockdown 2013

43

Qualys

WAF Beta:
InfoLeak

Lockdown 2013

44

Qualys

WAF Beta:
Fixups

Lockdown 2013

45

Qualys

WAF Beta: HTTP

Lockdown 2013

46

Qualys

WAF Beta: ACLs

Lockdown 2013

47

Qualys

WAF: What's coming?


QualysGuard

integration


WAS scan result feedback


Shared assets


False positive mitigation


Exception handling


Website and session profiling


Reporting



Lockdown 2013

48

We are Hiring in the Madison!


Product Management


Application Security Researchers


Developers


QA


Contact me if you are interested.

Lockdown 2013

49

Thanks!


github.com
/
ironbee

q
ualys.com
/
waf

qualys.com
/careers


Feel free to contact
me for
more info.


Brian Rectanus

brectanus@qualys.com

Lockdown 2013

50