Async networking in Python, Uriel Katz

Get Started. It's Free
or sign up with your email address
Rocket clouds
Async networking in Python, Uriel Katz by Mind Map: Async networking in Python, Uriel Katz

1. Greenlets in pure python

2. Uses libev for the event loop, libeio for file IO

3. Gevent

3.1. used by

3.2. has some problems, but Uriel says it's good for production

3.3. Uriel's favorite

3.3.1. will be used in their next release

4. gives you sync API on top of async networks

5. pluggable event loop

5.1. written in Python

6. thread pool built0in

7. monkey patching on steroids

7.1. DB API using threads

8. has a DBAPI 2.0 compatible MySQL driver

9. Doesn't seem active

10. Omegle.com in production

10.1. replaced Twisted

10.1.1. twisted

10.1.1.1. commonly used for building high-performance server in Python

10.1.1.2. reactor pattern & deferred (aka Future)

10.1.1.3. very mature & adopted

10.1.1.4. hard to learn

10.1.1.5. uses callbacks with defferes which results in terrible code

11. About

11.1. R&D manager at Binfire

11.1.1. Python for infra

12. title

12.1. with

12.1.1. gevent, concurrence, twisted, eventlet (& node.js)

13. Async networking

13.1. handling many concurrent users

13.2. basic idea

13.2.1. get a notif when a socket is ready for some IO operation & continue with other stuff until IO is available

13.2.2. like callbacks in javascript

13.3. using 1 thread like this: blocking calls block all requests

13.4. in operating systems

13.4.1. file descriptors

13.4.1.1. in Linux

13.4.1.1.1. epoll

13.5. only good for very heavy computation

14. Frameworks

14.1. gevent

14.1.1. abstracts the different OS implementations

14.1.2. fork of eventlet

14.2. twisted

14.2.1. reactor based with callbacks

14.2.2. thinks it's very difficult to work with

14.3. Concurrence

14.3.1. libevent based with Greenlets/stackless

14.3.2. like threads, but not primitive

14.3.3. from Hyves (the dutch facebook)

14.4. Eventlet

14.4.1. libevent/epoll based with greenlets from Linden Labs (SecondLife)

15. Concurrence

15.1. libevent based with greenlets

15.1.1. like Erlang's actors

15.2. use channels to communicate between tasklets

16. Eventlet

16.1. thread safe

16.2. quite slow

16.3. Websockets support

16.4. slightly slower than Gevent

16.5. Bug & messy compared to Gevent

17. Something extra - Node.js

17.1. Vitaly: "Erlang for poors"

17.2. Evented IO on top of Google's V8 JS engine

17.2.1. compiles JS to assembler

17.2.1.1. JIT

17.2.2. though main time is on IO not processing

17.3. has async file IO

17.3.1. very hard to do

17.3.2. in linux, no real async file IO

17.4. let's you write high-performance servers in JavaScirpt

17.5. parser written in Rebel

17.5.1. state-machine

18. Comet-like chat examples (no-polling)

18.1. API

18.1.1. /

18.1.1.1. just serve index

18.1.2. Untitled

18.1.3. /wait

18.1.3.1. returns when new data exists

18.2. all users call the wait method, but the server doesn't need threads serving all waiting clients

18.3. See examples the slides http://urielkatz.com

18.3.1. Untitled

18.3.2. Twisted

18.3.3. Concurrence

18.3.4. Gevent

18.3.5. Eventlet

18.3.6. Node.js

19. Largest pitfall

19.1. Each greenlet must give control to next greenlets

19.1.1. If 1 is blocking, all will be blocked

19.2. Can't escape this - a price we need to pay

20. Uriel's experience with Gevent

20.1. patches to pywsgi

20.1.1. Untitled

20.2. waitany function

20.3. stuff done in work

20.3.1. streaming proxy to Rackkspace cloud files

20.3.1.1. libevent doesn't support streaming

20.3.1.2. streaming zip

20.3.1.2.1. parts

20.3.1.2.2. 500 concurrent connections with 45MB of memory

20.3.1.3. like the url's in S3 specific for user

20.3.2. comet message bus

20.3.3. real-time drawing like Google drawing

21. what is this???

21.1. Doesn't have a standard socke implementation & no monkey-patchng support

22. New node

23. Новый узел

24. Новый узел