You must log in or register to comment.
!lemmySilver
I’ll be honest, this only matters when running single services that are very expensive. it’s fine if your program can’t be pararlelized if the OS does its job and spreads the love around the cpus
all programs are single threaded unless otherwise specified.
It’s safe to assume that any non-trivial program written in Go is multithreaded
I absolutely love how easy multi threading and communication between threads is made in Go. Easily one of the biggest selling points.
Key point: they’re not threads, at least not in the traditional sense. That makes a huge difference under the hood.
Well, they’re userspace threads. That’s still concurrency just like kernel threads.
Also, it still uses kernel threads, just not for every single goroutine.
What I mean is, from the perspective of performance they are very different. In a language like C where (p)threads are kernel threads, creating a new thread is only marginally less expensive than creating a new process (in Linux, not sure about Windows). In comparison creating a new ‘user thread’ in Go is exceedingly cheap. Creating 10s of thousands of goroutines is feasible. Creating 10s of thousands of threads is a problem.
Also, it still uses kernel threads, just not for every single goroutine.
This touches on the other major difference. There is zero connection between the number of goroutines a program spawns and the number of kernel threads it spawns. A program using kernel threads is relying on the kernel’s scheduler which adds a lot of complexity and non-determinism. But a Go program uses the same number of kernel threads (assuming the same hardware and you don’t mess with GOMAXPROCS) regardless of the number of goroutines it uses, and the goroutines are cooperatively scheduled by the runtime instead of preemptively scheduled by the kernel.
But it’s still not a guarantee
And yet: You’ll still be limited to two simultaneous calls to your REST API because the default HTTP client was built in the dumbest way possible.
I initially read this as “all programmers are single-threaded” and thought to myself, “yeah, that tracks”
I think OP is making a joke about python’s GIL, which makes it so even if you are explicitly multi threading, only one thread is ever running at a time, which can defeat the point in some circumstances.
no, they’re just saying python is slow. even without the GIL python is not multithreaded. the
threadlibrary doesn’t use OS threads so even a free-threaded runtime running “parallel” code is limited to one thread.If what you said were true, wouldn’t it make a lot more sense for OP to be making a joke about how even if the source includes multi threading, all his extra cores are wasted? And make your original comment suggesting a coding issue instead of a language issue pretty misleading?
But what you said is not correct. I just did a dumb little test
import threading import time def task(name): time.sleep(600) t1 = threading.Thread(target=task, args=("1",)) t2 = threading.Thread(target=task, args=("2",)) t3 = threading.Thread(target=task, args=("3",)) t1.start() t2.start() t3.start()And then
ps -efT | grep pythonand sure enough that python process has 4 threads. If you want to be even more certain of it you canstrace -e clone,clone3 python ./threadtest.pyand see that it is makingclone3syscalls.Now do computation in those threads and realize that they all wait on the GIL giving you single core performance on computation and multi threaded performance on io.
Correct, which is why before I had said
I think OP is making a joke about python’s GIL, which makes it so even if you are explicitly multi threading, only one thread is ever running at a time, which can defeat the point in some circumstances.
Ups, my attention got trapped by the code and I didn’t properly read the comment.
Isn’t that what threading is? Concurrency always happens on single core. Parallelism is when separate threads are running on different cores. Either way, while the post is meant to be humorous, understanding the difference is what prevents people from picking up the topic. It’s really not difficult. Most reasons to bypass the GIL are IO bound, meaning using threading is perfectly fine. If things ran on multiple cores by default it would be a nightmare with race conditions.
is this stackless?
anyway, that’s interesting! i was under the impression that they eschewed os threads because of the gil. i’ve learned something.
Does Python have the ability to specify loops that should be executed in parallel, as e.g. Matlab uses
parforinstead offor?Experimentally, yes
Cool.
Are you still using matlab? Why? Seriously
I was telling a colleague about how my department started using Rust for some parts of our projects lately. (normally Python was good enough for almost everything but we wanted to try it out)
They asked me why we’re not using MATLAB. They were not joking. So, I can at least tell you their reasoning. It was their first programming language in university, it’s safer and faster than Python, and it’s quite challenging to use.
“Just use MATLAB” - Someone with a kind heart who has never deployed anything to anything
No, I’m not at university anymore.
Good for you
Poor prof
We weren’t doing any ressource extensive computations with Matlab, mainly just for teaching FEM, as we’ve had an extensive collection of scripts for that purpose, and pre- and some post processing.
python has way too many ways to do that.
asyncio,future,thread,multiprocessing…I’ve always hated object oriented multi threading. Goroutines (green threads) are just the best way 90% of the time. If I need to control where threads go I’ll write it in rust.
nothing about any of those libraries dictates an OO approach.
Unless it’s java.
Meh, even Java has decent FP paradigm support these days. Just because you can do everything in an OO way in Java doesn’t mean you need to.
If I have to put a thread object in a variable and call a method on it to start it then it’s OO multi threading. I don’t want to know when the thread spawns, I don’t want to know what code it’s running, and I don’t want to know when it’s done. I just want shit to happen at the same time (90% of the time)
the thread library is aping the posix thread interface with python semantics.
Of the ways you listed the only one that will actually take advantage of a multi core CPU is
multiprocessingyup, that’s true. most meaningful tasks are io-bound so “parallel” basically qualifies as “whatever allows multiple threads of execution to keep going”. if you’re doing numbercrunching in pythen without a proper library like pandas, that can parallelize your calculations, you’re doing it wrong.
I’ve used multiprocessing to squeeze more performance out of numpy and scipy. But yeah, resorting to multiprocessing is a sign that you should be dropping into something like Rust or a C variant.
Most numpy array functions already utilize multiple cores, because they’re optimized and written in C
I prefer this default. Im sick of having to rein in Numba cores or OpenBlas threads or other out of control software that immediately tries to bottleneck my stack.
CGroups (Docker/LXC) is the obvious solution, but it shouldn’t have to be
Do you mean Synapse the Matrix server? In my experience, Conduit is much more efficient.
I may have something to read up on.
The documentation is kinda lacking, but if you could figure out how to set up Synapse then you can probably figure out Conduit too. https://conduit.rs/
Yep, I mean as in matrix. There is currently no was to migrate to conduit/conduwuit. Btw from what I’ve seen conduwuit is more full-featured.
i wish they would switch the reference implementation to conduit
there is core components on the client side in rust so maybe that’s the way for the future
I tough this was about excel and was like yeah haha!
But is about Python, so I’m officially offended.
Oh wow, a programming language that is not supposed to be used for every single software in the world. Unlike Javascript for example which should absolutely be used for making everything (horrible). Nodejs was a mistake.
Nodejs was a mistake.
More choice is always better
Citations Needed: Episode 95: The Hollow Vanity of Libertarian “Choice” Rhetoric
Episode webpage: https://dts.podtrac.com/redirect.mp3/traffic.libsyn.com/secure/citationsneeded/CN95_20191205_choice_Stites_v2.mp3
Fucking Citations Needed, every time I finish an episode, someone comment something related to it.
And some of those choices are mistakes.
I like Typescript >:3
I appreciate Typescript for addressing the sins of its predecessor.
It only took us how many years?
Python
…so… so you made it single threaded?
let’s be honest here, he actually means 0.01 core performance
Yes, 0.99 performance being consumed by the interpreter.
Oooooh this is really cool, thanks for sharing. How could I install it on Linux (Ubuntu)? I assume I would have to compile CPython. Also, would the source of the programs I run need any modifications?
From memory I can only answer one of those: The way I understand it (and I could be wrong), your programs theoretically should only need modifications if they have a concurrency related bug. The global interlock is designed to take a sledgehammer at “fixing” a concurrency data race. If you have a bug that the GIL fixed, you’ll need to solve that data race using a different control structure once free threading is enabled.
I know it’s kind of a vague answer, but every program that supports true concurrency will do it slightly differently. Your average script with just a few libraries may not benefit, unless a library itself uses threads. Some libraries that use native compiled components may already be able to utilize the full power of you computer even on standard Python builds because threads spawned directly in the native code are less beholden to the GIL (depending on how often they’d need to communicate with native python code)
Thanks for the answer, I really hope Synapse will be able to work with concurrency enabled.
In this case, it’s a feature of the language that enables developers to implement greater amounts of parallelism. So, the developers of the Python-based application will need to refactor to take advantage of it.
don’t worry it’ll use all the RAM anyway
I paid for all the memory. I’ll use all the memory.
JG Memoryworth
No RAM gets wasted!
