Python 多进程

3/8/2017来源:ASP.NET技巧人气:1237

1.os.fork()

普通的函数调用,调用一次,返回一次,但是fork()调用一次,返回两次,因为操作系统自动把当前进程(称为父进程)复制了一份(称为子进程),然后,分别在父进程和子进程内返回。

子进程永远返回0,而父进程返回子进程的ID。这样做的理由是,一个父进程可以fork出很多子进程,所以,父进程要记下每个子进程的ID,而子进程只需要调用getppid()就可以拿到父进程的ID。 Python的os模块封装了常见的系统调用,其中就包括fork,可以在Python程序中轻松创建子进程:

import os PRint 'Process (%s) start...' % os.getpid() pid = os.fork() if pid==0: print 'I am child process (%s) and my parent is %s.' % (os.getpid(), os.getppid()) else: print 'I (%s) just created a child process (%s).' % (os.getpid(), pid)

输出结果:

Process (3898) start… I (3898) just created a child process (3899). I am child process (3899) and my parent is 3898.

在Windows平台中没有fork调用,上面的代码在Windows上无法运行。

2.multiprocessing

from multiprocessing import Process import os def run_proc(name): print 'Run child process %s (%s)' %(name,os.getpid()) if __name__=='__main__': print 'Parent process %s.' % os.getpid() p = Process(target = run_proc, args=('test',)) print 'Process will start!' p.start() p.join() print 'Process end.'

输出结果如下:

Parent process 3640. Process will start! Run child process test (3961) Process end.

3.Pool

如果要启动大量的子进程,可以用进程池的方式批量创建子进程:

from multiprocessing import Pool import os, time, random def long_time_task(name): print 'Run task %s (%s)...' % (name, os.getpid()) start = time.time() time.sleep(random.random() * 3) end = time.time() print 'Task %s runs %0.2f seconds.' % (name, (end - start)) if __name__=='__main__': print 'Parent process %s.' % os.getpid() p = Pool(8) for i in range(9): p.apply_async(long_time_task, args=(i,)) print 'Waiting for all subprocesses done...' p.close() p.join() print 'All subprocesses done.'

输出结果如下:

Parent process 3640. Run task 0 (3912)… Run task 1 (3913)… Run task 3 (3915)… Run task 2 (3914)… Run task 4 (3916)… Run task 6 (3918)… Run task 5 (3917)… Run task 7 (3919)… Task 4 runs 0.06 seconds. Run task 8 (3916)… Waiting for all subprocesses done… Task 5 runs 0.21 seconds. Task 0 runs 0.99 seconds. Task 6 runs 1.07 seconds. Task 3 runs 1.17 seconds. Task 7 runs 2.37 seconds. Task 1 runs 2.48 seconds. Task 8 runs 2.50 seconds. Task 2 runs 2.77 seconds. All subprocesses done.

4.Queue

Process之间肯定是需要通信的,操作系统提供了很多机制来实现进程间的通信。Python的multiprocessing模块包装了底层的机制,提供了Queue、Pipes等多种方式来交换数据。 以Queue为例,在父进程中创建两个子进程,一个往Queue里写数据,一个从Queue里读数据:

from multiprocessing import Process,Queue import os,time,random def write(q): for value in ['A','B','C']: print 'Put %s to queue...' %value q.put(value) time.sleep(random.random()) def read(q): while True: value = q.get(True) print 'Get %s from queue.' %value if __name__=='__main__': q = Queue() pw = Process(target = write,args=(q,)) pr = Process(target = read,args=(q,)) pw.start() pr.start() pw.join() pr.terminate()

输出结果:

Put A to queue… Get A from queue. Put B to queue… Get B from queue. Put C to queue… Get C from queue.

参考: http://www.liaoxuefeng.com/wiki/001374738125095c955c1e6d8bb493182103fac9270762a000/0013868323401155ceb3db1e2044f80b974b469eb06cb43000 《python核心编程》