Follow

Look I really like programming in Python but seriously, screw the GIL. If all that effort re-inventing MPI was devoted instead into yanking out the GIL, maybe we'd almost be there and kludges like using multiprocessing instead of threading like every other language wouldn't be necessary.

Can Python 4 please ditch the big fucking lock?

@mjog let me guess, you're probably doing some data processing as opposed to writing an HTTP service?

To my memory, it's a relatively recent phenomenon with people describing using multiple processing as a kludge. During the decade when everyone was writing Web in #Python the question simply didn't come up.

@isagalaev Sure, because Python web apps spend most of their time in native code waiting for requests or responses, and because SMP and multi-core processors was so rare. However back then even with things like mod_python it was still a PITA.

In any case, in today's environment where multi-core is the default, the GIL is farcical.

@mjog uhm… No, it's mostly because Web services are shared-nothing short request-response workload, running them in multiple pre-forked processes is the most natural thing to do. gunicorn or uwsgi has made it easy, and the application code doesn't have to do anything to run in parallel. You just provide a WSGI app and it's being run for you by an app server. And they of course use all cores.

In other words, GIL wasn't a problem for the majority of Python code in the wild.

@isagalaev Heh heh, good one! I guess if all you develop is web apps then the problem space looks like a web app.

@mjog that's a given :-)

But in all earnestness, what I'm saying is that the presence of GIL for so much time is not really surprising. It's only recently (relatively) when people making ETL pipeline came to Python and started to write parallel code *by hand* it became a prominent issue. At least that was my impression.

@isagalaev I guess we disagree then, in that I find it quite surprising: It was a poor design decision in the first place, and the fact that it's survived though a non-backwards compatible major release is exceptionally lame.

I know why it's there, but that doesn't change the fact that it shouldn't be.

@mjog historically, the design decision was actually pretty sound. Heavily paraphrasing and simplifying Guido: just because threads were popularized by Windows programming and Java doesn't mean they're essentially better than forked processes. On Unixes the latter is quite a natural choice, and it doesn't have shared state problems.

Removing it turned out really hard, they've been trying to do it for ages. I won't try to explain it here, but here's a link I found: youtube.com/watch?v=pLqv11ScGs

@isagalaev Disagree with that as well.

If threads aren't necessary, then don't add them. Of course, threads are exceptionally useful in a multi-core pervasive environment, so it has turned out to be an even worse decision today than it was back then.

@isagalaev In any case, fork+exec on UNIX actually sucks: It's hard to get right, even more so portably. True, it doesn't have shared state problems, but it has a it still has the same co-ordination problems and also whole host of other problems (shared memory anyone?). But if that's the model you want, just do that and don't add a broken threads impl.

Sign in to participate in the conversation
Octodon

Octodon is a nice general purpose instance. more