GoLang in GoAgent

klipitiklopwarrenΛογισμικό & κατασκευή λογ/κού

7 Νοε 2013 (πριν από 3 χρόνια και 11 μήνες)

523 εμφανίσεις

GoLang in GoAgent

@phuslu

2012/11/10

GoAgent is a


Google AppEngine Application


HTTP/HTTPS Proxy


Written by Python(2011)



GoAgent ABC


GoAgent Tricks



HOSTS


DNS resolve


CRLF Injection


HTTPS multiple
-
connection

GoAgent Tricks
-

Hosts

telent 203.208.46.161 80


GET /fetch.py HTTP/1.1

Host: goagentd.appspot.com


HTTP/1.1 200 OK

Content
-
Length: XXXX


Hello GoAgent


D:
\
>ping www.google.cn


正在

偩P朠gww.goo杬e⹣渠嬲〳⸲〸⸴㘮ㄶㅝ[
具有

㌲3
字节的数据
:

来自

㈰㌮㈰㠮㐶⸱㘱2
的回复

字节
㴳㈠
时间
㴳㉭猠T呌㴴=

来自

203.208.46.161
的回复

字节
㴳㈠
时间
㴳㍭猠=呌㴴=

来自

203.208.46.161
的回复

字节
㴳㈠
时间
㴳㉭猠T呌㴴=

来自

㈰㌮㈰㠮㐶⸱㘱2
的回复

字节
㴳㈠
时间
㴳㍭猠T呌㴴=

GoAgent Tricks
-

Hosts

>>> socket.gethostbyname('www.google.com')

'74.125.128.99‘


>>> print urllib2.urlopen(urllib2.Request('https://74.125.128.99/fetch.py',
headers={'Host':'goagent
-
dd.appspot.com'})).read().decode('utf8')

GoAgent Python Server 2.0.8
已经在工作了,部署时间

㈰ㄲ
-

-
㈹‰㈺〳0ㄹ


GoAgent Tricks


TCP DNS resolve




D:
\
>nslookup
-
vc www.facebook.com 8.8.8.8

服务器
㨠 g潯杬g
-
pub汩c
-
dns
-
愮goo杬e⹣om

䅤dre獳㨠 㠮㠮㠮8


非权威应答
:

名称
㨠†⁷ww.f慣aboo欮com

䅤dre獳e猺s′愰㌺㈸㠰2㄰1ㅦ〲0f慣a㩢〰挺:

†† ††
㘹⸱㜱⸲㌴⸲6

GoAgent Tricks


UDP DNS resolve


1.
Send UDP DNS query to 8.8.8.8/8.8.4.4/168.95.1.1

2.
Socket recvform 512 bytes DNS answer

3.
If DNS answer in blacklist, goto 2

1.
blacklist set(['4.36.66.178', '8.7.198.45', '37.61.54.158',
'46.82.174.68', '59.24.3.173', '64.33.88.161',
'64.33.99.47', '64.66.163.251‘, ‘…’]

4.
If DNS answer not in blacklist, return



GoAgent Tricks


CRLF Inection

telent www.google.com.hk 80


\
r
\
n

GET /fetch.py HTTP/1.1

Host: goagentd.appspot.com


HTTP/1.1 200 OK

Content
-
Length: XXXX


Hello GoAgent


GoAgent Tricks


new CRLF Inection

telent www.google.com.hk 80


GET / HTTP/1.1

\
r
\
n

\
r
\
n

\
r
\
r

\
r
\
n

GET /fetch.py HTTP/1.1

Host: goagentd.appspot.com


HTTP/1.1 304 Move

Location:
http://www.google.com.hk/webhp?hl=zh
-
CN

\
r
\
n

\
r
\
n


HTTP/1.1 200 OK

Content
-
Length: XXXX


Hello GoAgent


GoAgent Tricks


HTTPS Connect





Go is a


New



Experimental


Concurrent


Garbage
-
Collected


Systems Language.



Why a new language?



Go fast!


Make programming fun again.


No new major systems language in a decade. But
much has changed:






-

sprawling libraries & dependency chains


-

dominance of networking




-

client/server focus





-

massive clusters






-

the rise of multi
-
core CPUs


Major systems languages were



not designed with all these factors
in mind.

Why move to GoLang?



Fast


Native.


More Standard Libs.


GoRoutine


Multi
-
Core support?

Why move to GoLang
-

Fast



Fast Event Loop


Epoll/IOCP backend in Linux/Windows


Flat net/http package


Static Type


Find out error in compiling time


Run faster in runtime

Why move to GoLang
-

Native



Standalone


No Runtime


No Dependency


Small


1 M bytes in windows

Why move to GoLang
-

Standard
Libs



crypto/x509 VS pyOpenSSL


Same points:


Issue key/certs


Difference:


crypto/x509 is a standard and pure golang lib, no
dependency


pyOpenSSL is a 3
rd

python lib and based on openssl
-
devel


Conclusion:


pyOpenSSL is hard to integrate


pyOpenSSL must be built in on every platform



Why move to GoLang
-

Standard
Libs



index/suffixarray VS str.endswith


Same points:


Judge a domain is a google domain family or not


Python Example:



google_domains = [‘.google.com’, ‘.google.com.hk’, ‘.googleapis.com’, ‘.g.cn’]

domain = ‘www.facebook.com’

for pattern in google_domains:


if domain.endswith(pattern):


# direct forward

Else:


# fetch content from GAE

Why move to GoLang
-

Standard
Libs



index/suffixarray VS str.endswith


Same points:


Judge a domain is a google domain family or not


GoLang Example:



index := suffixarray.New(google_domains)

domain = ‘www.facebook.com’

if index.LookUp(domain) {


// direct forward

} else {


// fetch content from GAE

}

Why move to GoLang
-

Standard
Libs



index/suffixarray VS str.endswith


Same points:


Judge a domain is a google domain family or not


Conclusion:


index/suffixarray is a killer feature for goagent


Why move to GoLang
-

Standard
Libs



What’s more


net/http vs SimpleHTTPServer


log/syslog vs

python
-
logging


crypto/rc4 vs python
-
rc4





Conclusion:


golang standard libs are effective for networking
programing




Why move to GoLang
-

GoRoutine



Python Example

Why move to GoLang
-

GoRoutine


GoLang Example

Why move to GoLang


GoRoutine



Conclusion:


GoAgent could connect GAE more faster by using
goroutine

Why move to GoLang


Multi
-
core



Python threading vs GoLang goroutine


Python threading is limited by GIL


GoLang runtime.GOMAXPROCS(NCPU) could enable
multi
-
core support on the fly.


Scenarios?


https/openssl


rc4/lzw


CONCLUSION


GoLang is a fast, native and effective language


Golang makes networking programming
easier.


Goroutine is a killer language feature.


Want to try it?Want to help?Want to build
libraries or tools?


http://golang.org

Any Queries??