Linux Per-Process Syscall Hooking, by Pluf.

This document describes a new syscall hooking technique for Linux systems and exposes how it can be implemented as part of a virus or a backdoor in order to take full control over an userland application.

Although there are some well-known methods for hooking functions, they are mostly based on the ELF format itself.

This technique is focused on thoses pieces of code that are externally called by the main program and invoke a system call or system service.

A simple implementation of this hooking mechanism has been developed as a result of the research and it is included with the article.

This code provided does not have all the features you wish but includes the required ones, is not a real backdoor but a simple proof of concept, perfect to write your own one.

Popularity: 17% [?]

Recently, I’m surveying how to deploy Mozilla Firefox/Thunderbird with it’s update channel, but it’s very hard to use for un-official builders like me.

Generally speaking, Firefox will try to send HTTP request to mozilla official site with some client information. For example, a nightly BonEcho will try to get document from this URL.

AS you see, Firefox try to send some information like Firefox version, host architecture, host OS version and so on, and to get server response through SSL (in fact, it’s TLSv1). Of course, you can use any browser to see what the corresponding response is by this URL.

Then Firefox can get type “complete” of patch tag, fetching the .mar file at mozilla official FTP site, and performing update procedure.

Everything seems great, doesn’t it? Not exactly. It means that the Firefox updater program only support .mar file format, not gz, bzip2 format — the common compression we’ve known. Consequently I must have enough knowledge to know how to pack .mar archive as well.

Hence, I create a project called mar hosting on Google Code (Brief introduction on ijliao’s blog). I hope these simple tool s could create, extract, and view the mozilla archive more easy.

Popularity: 11% [?]

Effective Python Programming, from OSCON 2005.

Python’s growing popularity has brought many new developers to the language, often coming from other languages that have very different idioms and restrictions. This can lead to less-than-optimal coding styles.

This tutorial focuses on writing Python in a “Pythonic” way, working through a series of examples.

This is a concentrated exercise in helping people write code in a way that makes the best use of language features. It is of course possible to write bad Python–just as it’s possible to write bad code in any language. However, it is suprisingly easy to write good Python and hopefully this tutorial will help you bootstrap yourself into a better Python programmer.

This tutorial is aimed at people who already know Python, but don’t consider themselves experts. (Python experts will probably know most of the things discussed in this tutorial.)

You can download the presentation file (PDF) (159 slides) here.

Popularity: 9% [?]

Buffering in HTTP.SYS, from Windows Core Networking Team’s blog.

故事內容是這樣的. IIS 5.0 之前是用 WinSock 來 implement, 這樣預設會打開 buffering 機制, 到了 Windows 2003 的 IIS 6.0 開始, 則改由 HTTP.SYS 來負責. 這樣反而會遇到 performance 下降的問題.

主要的理由我猜是 Nagle’s Algorithm. 現在的 socket 實作預設都會打開 TCP 的 Nagle’s Algorithm. 這個演算法簡單的說, 就是利用 delay ack 來減少網路上所傳輸 packet 量, 進而增加 TCP 的 performance. 不幸的是, Nagle’s Algorithm 對於小 packet 的幫助比較明顯, 大 packet 反而會有 latency 拉長的反效果. 換句話說, 對於一次傳輸大量資料的網路程式, 應該要 disable Nagle’s Algorithm 會有比較好的 performance.

目前在 Windows 2003 SP1 之後有一個 workaround 的辦法, 就是利用 HTTP_SEND_RESPONSE_FLAG_BUFFER_DATA 這個 flag. 雖然說還有另一個 flag 是 HTTP_SEND_RESPONSE_FLAG_ENABLE_NAGLING, 表示預設應該是 disable Nagle’s Algorithm, 不過我還是很好奇, 為甚麼預設明明 disable Nagle’s Algorithm, performance 沒上去的原因, 和一般網路程式 enable Nagle’s Algorithm 一樣呢?

Popularity: 28% [?]

How To Create a Self-Restartable Application, from CodeProject.

This article describes the way to add restarting support to your Win32 applications. Key pointer is as followings:

Old Instance Create a Mutex and create a new process with command retrieved by GetModuleFileName().

g_RA_hMutexOtherRestarting = ::CreateMutex( NULL, TRUE, RA_MUTEX_OTHER_RESTARTING);
TCHAR szAppPath[MAX_PATH] = {0};
::GetModuleFileName(NULL, szAppPath, MAX_PATH);
CreateProcess(NULL, szAppPath, NULL, NULL, FALSE, 0, NULL, NULL, &si, &pi);

New Instance wait for this mutex for release which means old instance has been dead.

// Releasing mutex signal that process finished
DWORD dwWaitResult = WaitForSingleObject(g_RA_hMutexOtherRestarting, 0);
if (dwWaitResult == WAIT_TIMEOUT)
::ReleaseMutex(g_RA_hMutexOtherRestarting);
::CloseHandle(g_RA_hMutexOtherRestarting);

Popularity: 21% [?]

Van Jacobson’s network channels, from LWN.net.

Van Jacobson 在今年一月底的 linux.conf.au 上展示了他的 network channels 的 idea, 然後引發了一系列的討論. 雖然有不少障礙要克服, 不過設計上應該有蠻多學習的地方 ;-)

從 Van 的 slides [PDF] 上講到, 傳統的 Networking stack, 到現在的實作已經變成了 “Standard Model”. 在 Linux kernel 實作上大概如下 (From Van’s slides):

當封包被網路卡收到, kernel 會收到 interrupt, 然後呼叫 ISR, 或是有註冊 NET_RX_SOFTIRQ 的 softirq handler (一般應該是 driver; 另, 這裡也可能是 tasklet), 會根據自己的硬體運作方法把 packer 收下來組成 skb, 然後呼叫 net/core/dev.c:netif_receive_skb(). netif_receive_skb() 這裡會檢查 payload 然後解多工. 舉個例子, IP 應該會送到 net/ipv4/ip_input.c:ip_recv() 去. 當然, 後面的 Socket (更高的像是 UDP/TCP Layer) 也是會參考這個 skb, 當然就大家所知, TCP 甚至還要組成 Byte-Stream.

這樣的設計當可能有一些缺點, 為了不失原意, 我摘錄原文如下:

• Passing network packets through multiple layers of the kernel.
  When a packet arrives, the network card’s interrupt handler begins the task of feeding the packet to the kernel. The remainder of the work may well be performed at software interrupt level within the driver (in a tasklet, perhaps). The core network processing happens in another software interrupt. Copying the data (an expensive operation in itself) to the application happens in kernel context. Finally the application itself does something interesting with the data. The context changes are expensive, and if any of these changes causes the work to move from one CPU to another, a big cache penalty results. Much work has been done to improve CPU locality in the networking subsystem, but much remains to be done.
• Locking is expensive.
  Taking a lock requires a cross-system atomic operation and moves a cache line between processors. Locking costs have led to the development of lock-free techniques like seqlocks and read-copy-update (RCU), but the the networking stack (like the rest of the kernel) remains full of locks.
• The networking code makes extensive use of queues implemented with doubly-linked lists.
  These lists have poor cache behavior since they require each user to make changes (and thus move cache lines) in multiple places.

因此, 為了要增加 networking scalability, 首要就是要消除 locking 和 shared data. Van 利用 end-to-end principle 來達成這個目的. 也就是說, 盡可能的讓資料交給 application, 而不要在 kernel 任何地方等待. 於是他設計了 net channel — 一個 circular buffer (應該是 Circular FIFO queue implemented by Array) 用來取代 skb 和目前用在 networking stack 的 queue. 比方說, 原先需要用 softirq 的地方 (driver -> socket), 改用 netchannel, locking 數都明顯的下降, 進而提高 scalability.

但是, 這個方法遇到的第一個問題就是, 讓資料從 packet 一條鞭的到 application, 會把 netfilter 的原先 hook 的點變相的消除, 為了加回來 netfilter 的 support, 利用得到的優勢便蕩然無存了.

不過我有一點不懂的是, 一個還算簡單的 circular buffer, 為何是 “Cache aware, cache friendly queue” 呢? 是因為用 Array implement 這樣嗎? :p

Popularity: 11% [?]

Untwisting Python Network Programming, from O’ReillyNet.

This article introduces basic client-side networking using both core Python modules and the Twisted framework. For its example, I will show how to send, receive, and delete emails, and conduct Telnet sessions.

I have written two functionally equivalent examples, one using the core modules (mail-core.py) and another using Twisted (mail-twisted.py), with both start, stop, and interact with a server to process emails.

Here is outline:

core Python modules

• Sending Mails with smtplib
• Retrieving Emails with poplib
• Conducting Telnet with telnetlib

Twisted Way

• Sending Mails the Twisted Way
• Retrieving Mails with Twisted
• Doing Telnet with Twisted

When to Be Twisted?

The two functionally equivalent programs, one using Python core modules and the other using the Twisted framework, significantly differ from each other in terms of programming style and the amount of code. Then when should you use either of the two options?

For basic programs such as the command-line client of this example, the Python core networking modules are more desirable due to the simplicity and performance advantages. However, most real-world networking programs are very complex, and Twisted’s asynchronous programming model is more effective.

For example, BitTorrent, the popular peer-to-peer file sharing client that performs massive parallel downloading of data chunks from different sources, uses Twisted. Twisted also works well in programs with graphical user interface (GUI), because its asynchronous nature fits more seamlessly with the event-driven programming models of modern GUI frameworks. In fact, Twisted has integration with popular GUI frameworks including PyGTK, Qt, Tkinter, WxPython, and Win32.

The other area where Twisted shines is in server programming. A typical network server uses multithreading so that it can handle multiple clients concurrently. The asynchronous mechanism of Twisted alleviates the creation and handling of threads by server programs. In addition, Twisted provides several protocols on which to build new networking services, enabling rapid development of complex servers. One such project is Quotient, which adopts Twisted to build a multiprotocol messaging server that supports a variety of protocols and services including SMTP, POP3, IMAP, webmail, and SIP.

Popularity: 9% [?]