Plurk.com is hiring

Are you good at Python programming? Are you a Javascript master?

As a developer of Plurk.com, here I want to say it loudly – We’re hiring!

For more details, please reference to:

Help Spark a Social Revolution – Plurk Opens Taiwan Office and is Hiring Worldwide!

And here are the list of jobs on inside jobboard:

Python/Javascript Ninja Extraordinaire

Mobile Engineer

Schmoozy Plurkinista/Plurkster & Community Evangelist

L33T Systems H@x0R, Optimizer & Log Lord

Plurk Product Manager, Chief Motivation Officer and Delegator

Posted in English Articles, Python | Tagged , , , , | 2 Comments

Zero-downtime service migration

When I am running my website, now.in, I did encounter a trouble.  That is, when there is a bug in the production server, then I need to restart them.  Sounds fine, right? Just to restart a server.  Yes, for most of web HTTP servers, they are stateless,  it is fine to restart them whenever you want, but not true for now.in, they are audio streaming servers.  Here s a server stats diagram shows the problem:

You can see there are some gaps in the plot, that’s caused by server restarting.  Of course, for users, that would definitely be bad experience.  Thus, I’m thinking how to solve this problem recently.  Before we go into the design, let’s look the reasons for restarting server first.

• To deploy new version of program
• To fix bugs
• The process is using to much memory
• To reload environment, ulimit -n for example (the limit count of file descriptor under unix-like environment)
• To migrate from host A to host B

For simply employe new version of program, we can use reload function of Python to reload modules.  But there is some problems, reload function only rerun the module, those created instances are still there, it might work if the change is minor.  On the other hand,  reloading can’t solve memory usage problem, process environment change problem.  And here comes the final reason, to migrate service from host A to B.  Indeed, it is difficult not to make any down time for such migration, and there is little chance to do such migration, therefore, we’ll only focus on migration in same host.

The idea

The biggest challenge is, how to migrate those working connections?  My idea is simple, create a new process, and transfer those connection to the new process, and shut the old one down.  Following diagrams show my method.

The Master is a process which is charge for managing migration and receiving command from other process.  And the process A is for running service.

Before we perform the migration, the Manager startup process B, and wait it says it’s readly.

When process B said “Hey! I’m ready”, then manager tell process A to send the connection state descriptor to process B.  Process B receive the state, and take over the service.

Finally, process B took over the service, then master tells process A “You are fired.” and the process A rolls itself out.

That’s it, the service was migrated, and there is no down time.

The problem – socket transfer

The idea sounds good, right? But still, we have some technical problem to solve.  It is “How to transfer socket (file descriptor) from one process to another?”.  To solve this problem, I have some study, and eventually I know two methods to achieve the goal.

Child process

For most of unix-like OS, child processes inherit file descriptors from their parent.  Of course we can use this feature to migrate our service, but however, it got its limitation.  You can only transfer file descriptors to child process.

Sendmsg

Another way to achieve same goal is, to use sendmsg through a unix domain socket to send the file descriptors.  With sendmsg, you can transfer file descriptors to almost any processes you like, that’s much flexible.

A simple implementation

To simplify the example, we only implement process A and process B here, it is quite enough for two process to complete the migration.  Before we go into the details, there is another problem to solve, which is sendmsg is not a standard function in Python.  Fortunately, there is a third party library named sendmsg provides this function.  To install sendmsg, it is easy, just type

easy_install sendmsg

And here you are.  Okay, following are the two programs.

a.py

import os
import socket
import sendmsg
import struct

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('', 5566))
s.listen(1)
conn.send('Hello, process %d is servingn' % os.getpid())
print 'Accept inet connection', conn

us = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
us.bind(('mig.sock'))
us.listen(1)
print 'Accept unix connection', uconn

sendmsg.sendmsg(
uconn.fileno(), '', 0, (socket.SOL_SOCKET, sendmsg.SCM_RIGHTS, payload))
print 'Sent socket', conn.fileno()
print 'Done.'


b.py

import os
import socket
import sendmsg
import struct

us = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
us.connect(('mig.sock'))
print 'Make unix connection', us

result = sendmsg.recvmsg(us.fileno())
identifier, flags, [(level, type, data)] = result
print identifier, flags, [(level, type, data)]
fd = struct.unpack('i', data)[0]
print 'Get fd', fd

conn = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)
os.close(fd)
conn.send('Hello, process %d is servingn' % os.getpid())
raw_input()


The flow is simple, the a.py accept an inet socket and open an unix domain socket, wait b.py to take over the service.  And here we runs b.py, it connects to a.py, receives the fd of socket and takes over providing service.

The result

As the result shows, there is not down time between the service migration.

It is very useful to employ this technique in server programs.  You can even migrate service from Python server to a C/C++ server, or vice versa.  Also, to keep the memory usage low, you can also migrate the service to same program periodically.  I will try to employ this technique in my servers to achieve zero down time migration.  If you are interesting in this technique, have a try, it is fun and useful 🙂

Posted in English Articles, Python, Unix-Like, 分享 | | 1 Comment

無停機服務遷移

• 部署新版的程式
• 為了修正BUG
• 記憶體用量過高
• 重新讀取執行環境，例如ulimit -n，也就是檔案數量的限制大小
• 從主機A移到主機B

簡易的實作

easy_install sendmsg

a.py:

import os
import socket
import sendmsg
import struct

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('', 5566))
s.listen(1)
conn.send('Hello, process %d is servingn' % os.getpid())
print 'Accept inet connection', conn

us = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
us.bind(('mig.sock'))
us.listen(1)
print 'Accept unix connection', uconn

sendmsg.sendmsg(
uconn.fileno(), '', 0, (socket.SOL_SOCKET, sendmsg.SCM_RIGHTS, payload))
print 'Sent socket', conn.fileno()
print 'Done.'

b.py

import os
import socket
import sendmsg
import struct

us = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
us.connect(('mig.sock'))
print 'Make unix connection', us

result = sendmsg.recvmsg(us.fileno())
identifier, flags, [(level, type, data)] = result
print identifier, flags, [(level, type, data)]
fd = struct.unpack('i', data)[0]
print 'Get fd', fd

conn = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)
os.close(fd)
conn.send('Hello, process %d is servingn' % os.getpid())
raw_input()

執行結果

Posted in Python, 中文文章, 分享 | Tagged , , , , , , , | 7 Comments

Mr.DJ 2.0.3 Mac OS X版發佈

Mr.DJ 2.0.3 Mac OS X版正式發佈

Posted in Uncategorized, 中文文章 | Tagged , , , , | 3 Comments

Logo作者

Logo作者為ptt的cfi697，而原始版本因為字型寬度的問題我修正了一下

Posted in 中文文章, 無用 | Tagged , , , | 3 Comments

總結

Posted in 中文文章, 作品, 嘴砲 | Tagged , , , , | 1 Comment

Logy – a central logging system for Python

I’m glad to announce my new open source project:Logy. Logy is a simple lightweight central logging system for Python. When do you need a central logging system?  Well, when you have many servers online, and they are running different python application, you will want to have a central logging system for making sure everything is going fine.  And you might ask, why not just use email for error reports? Well, following screenshot explains:

I use email as error logging report, but however, as you can see if there is something wrong with my servers in same time, network outage for instance, there will be so many error report mails come in my mailbox like crazy.  Therefore think a central log might be a good idea, then I decide to develop a simple central logging system for my own servers.  Here is a screenshot of the Logy web page.

It categorizes your logging records by IP address automatically, and also, you can set a application name for different applications which are running in same box.  And one important feature is, it is base on Flask 🙂

It is super easy to write records to Logy, here is an example:

import logging

import ex_loghandlers

rootLogger = logging.getLogger('')
rootLogger.setLevel(logging.DEBUG)

sream_handler = logging.StreamHandler()
# set up the http handler which writes records to Logy server
http_handler = ex_loghandlers.ExHTTPHandler(host='localhost:5000',
url='/sink/TEST/myapp',
method='POST')

logger1 = logging.getLogger('myapp.area1')
logger2 = logging.getLogger('myapp.area2')

logger1.debug('Quick zephyrs blow, vexing daft Jim.')
logger1.info('How quickly daft jumping zebras vex.')
logger2.warning('Jail zesty vixen who grabbed pay from quack.')
logger2.error('The five boxing wizards jump quickly.')

try:
raise Exception("Boom")
except:
logger1.fatal("Huston, we've got a problem", exc_info=True)

logger1.info('Everything goes fine, now.')

You need to install my module named ex_loghandlers before you can run this script, it contains an ExHTTPHandler which provides extra information for Logy.

PyPi page: http://pypi.python.org/pypi/logy

Bitbucket: https://bitbucket.org/victorlin/logy/src

Posted in English Articles, 分享 | Tagged , , | 1 Comment

數學筆記: Euler’s theorem 練習前置證明

$w^{m-1} \equiv 1 \pmod{m}$

$(1+1+1+ \cdots +1)^m$

$(a+b)^n$

$(a+b)*(a+b)* \cdots *(a+b)$

$(a+b) \newline (a+b) \newline \centering \cdots \newline (a+b)$

$2^n$

$ABC$

$\Box \Box \Box$

$3*2*1$

$3!$

$ABC \newline ACB \newline BAC \newline BCA \newline CAB \newline CBA$

$A\underline{BC} \newline A\underline{CB} \newline \underline{B}A\underline{C} \newline \underline{BC}A \newline \underline{C}A\underline{B} \newline \underline{CB}A$

$AXX \newline AXX \newline XAX \newline XXA \newline XAX \newline XXA$

$N!$

$\frac {N!} {a!}$

$\frac {N!} {a!b!}$

$(a+b)^k$

$\frac {k!} {k!} a^k+ \frac {k!} {(k-1)!1!} a^{k-1} b+ \frac {k!} {(k-2)!2!} a^{k-2} b^2+ \cdots$

$\frac {k!} {(k-2)!2!} a^{k-2} b^2$

$w^m=(1+1+1+ \cdots +1)^m$

$1^m+1^m+1^m+ \cdots +1^m$

$\frac {m!} {(m-1)!1!}1^{m-1}1+\frac {m!} {(m-2)!2!}1^{m-2}1^2+ \cdots$

$\frac {m!} {(m-2)!2!}1^{m-2}1^2+ \cdots$

m!的很多項都會被消掉，像這個會被消成

$\frac {m(m-1)} {2!}$

$w^m \equiv w \pmod m$

$w^{m-1} \equiv 1 \pmod m$

Posted in 中文文章, 數學筆記 | Tagged , | 3 Comments

Po translate – automatic translation tool for .PO files

It is kinda hateful to translate .PO files from Traditional-Chinese to Simplified-Chinese, most of the meanings are identical but different in simplified word or traditional word.  It is not a very difficult to translate.  But however, it is still an hateful job to do that manually.  For this reason, I wrote a tool which uses Google translate API to translate .PO files automatically.  It is quite easy to install

easy_install po_translate

It is also very easy to use, too.

po_translate input.po output.po zh-TW zh-CN

Of course, you can replace zh-TW and zh-CN to the source and destination language you like, e.g. English to Chinese, if the result is acceptable to you 🙂

Posted in English Articles, Python, 分享 | | 1 Comment

Po translate 自動化翻譯工具

easy_install po_translate

po_translate input.po output.po zh-TW zh-CN

PyPi的連結

Bitbucket的Repo

Posted in Python, 中文文章, 分享 | | 1 Comment