Location via proxy:   [ UP ]   [Report a bug]   [Manage cookies]    No cookies    No scripts    No ads    No referrer    Show this form

https://lwn.net/Articles/961884/ This is a special feed containing comments posted to the individual LWN article titled "Windows NT synchronization primitives for Linux". en-us Tue, 17 Sep 2024 13:05:21 +0000 Tue, 17 Sep 2024 13:05:21 +0000 https://www.rssboard.org/rss-specification lwn@lwn.net https://lwn.net/Articles/975593/ https://lwn.net/Articles/975593/ riking <div class="FormattedComment"> "Wait for all with timeout and alert" feels like the core operation here that is currently impossible to perform on Linux without an arbitrator, and when you don't know if a program is going to use that operation you need to send all operations on all of these objects over the costly arbitrator connection.<br> </div> Tue, 28 May 2024 16:54:59 +0000 https://lwn.net/Articles/964675/ https://lwn.net/Articles/964675/ farnz <p>Yep. Now that they're confident that they don't want to deliberately break user-space, they're going for mainline, just like everything else, because there's good reasons to be in mainline. It's just that when you know you want to break mainline's rules (unstable userspace interfaces like OpenVPN DCO, wrong licence like NVidia's proprietary driver), you need to stay out until you're ready to keep to mainline's rules. Thu, 07 Mar 2024 11:36:33 +0000 https://lwn.net/Articles/964573/ https://lwn.net/Articles/964573/ florianfainelli <div class="FormattedComment"> And yet this driver is now attempting to enter mainline: <a href="https://lore.kernel.org/netdev/20240106215740.14770-1-antonio@openvpn.net/">https://lore.kernel.org/netdev/20240106215740.14770-1-ant...</a><br> <p> The technical reasons for attempting to upstream should be simple and clear: it facilitates the distribution of your module and it gives you some amount of maintenance "for free".<br> </div> Wed, 06 Mar 2024 19:18:31 +0000 https://lwn.net/Articles/964100/ https://lwn.net/Articles/964100/ farnz <p>That would work, too, but then needs the people proposing a Linux-only accelerator for wait-for-any behaviour to create the compat options + list, so that people can use it. <p>This is not an unsoveable problem; it's "just" that Wine isn't happy with a gain in performance for some applications at the expense of others now failing to work at all, and people will have to do the work so that this becomes a gain in performance for some applications (those that only use wait-for-any) but not others. Thu, 29 Feb 2024 19:20:09 +0000 https://lwn.net/Articles/964095/ https://lwn.net/Articles/964095/ raven667 <div class="FormattedComment"> <span class="QuotedText">&gt; the challenge is that if the kernel driver only implements wait-for-any, and neither implements a way to "return" those waits to userspace early nor a wait-for-all option, then Wine has a regression and won't use this driver for wait-for-any, even though it works for that purpose</span><br> <p> I'm catching up but isn't the way that MS solves these kinds of compat issues in the Windows world is tedius baked-in lists of specific applications which need the differential behavior? A WINE config/cli flag that indicates when emulated wait-for-all is needed which uses the existing slow implementation with a default which uses the fast kernel implementation, that aborts with a sensible error if the app uses features which aren't implemented, so the user/admin can restart the app with the right compat option saved. Maybe an option to upload those compat lists somewhere so the config could be distributed, if the existence of the app isn't sensitive data.<br> <p> I know having things work correctly automatically is more awesome but sometimes just doing the dumb brute force thing is more effective and efficient than working through all the details, coordination and judgement to automate something. Technical debt isn't always bad as long as you are picky when you create it.<br> </div> Thu, 29 Feb 2024 18:55:08 +0000 https://lwn.net/Articles/963542/ https://lwn.net/Articles/963542/ itsmycpu <div class="FormattedComment"> <span class="QuotedText">&gt; I am actually more familiar with Windows events than futexes so excuse my heresy :) One problem I see is auto-reset events, where pulse specifically only wakes one *and only one* of the waiting threads. But I don't think futexes don't have an easy way of guaranteeing the wakes happen atomically in that fashion: it seems like this would wake *at least* one waiting thread, absent a very clever workaround or modifying the kernel directly. Of course I could be missing something. But I suspect Windows kernel being written to atomically wake up a single waiting thread is probably expensive to emulate on Linux.</span><br> <p> Futexes do have an easy way to wake exactly one thread (of course, otherwise how would you efficiently implement a lock using them).<br> </div> Sat, 24 Feb 2024 23:37:36 +0000 https://lwn.net/Articles/963407/ https://lwn.net/Articles/963407/ farnz <p>Note that the thing that warrants a kernel driver is the wait-for-any functionality of the call; the challenge is that if the kernel driver <em>only</em> implements wait-for-any, and neither implements a way to "return" those waits to userspace early nor a wait-for-all option, then Wine has a regression and won't use this driver for wait-for-any, even though it works for that purpose. <p>The exact implementation of this is up to the people doing the work; do you put a wait-for-all mode in the kernel, even though it's rarely needed, or do you come up with a way for Wine to reclaim all the wait-for-anys sitting in the kernel and push all wait-for-anys (including the reclaimed ones) through the slow path, not using the kernel driver? <p>Both work, since we only have evidence that there are applications that do not use wait-for-all, but would benefit from a faster wait-for-any than can be implemented in userspace alone. Fri, 23 Feb 2024 18:35:58 +0000 https://lwn.net/Articles/963405/ https://lwn.net/Articles/963405/ mb <div class="FormattedComment"> Please see the root of this thread.<br> This is going way off topic to what I originally commented on.<br> <p> The original assumption was that there were *zero* use cases. (which was *not* claimed by me)<br> Which has been clarified as being wrong.<br> <p> The corrected assumption is that there are proprietary apps hidden somewhere that wine does not want to break.<br> Whether that warrants a kernel driver or not is a completely different question. IMO it doesn't, but feel free to have a different opinion. I'm Ok with that.<br> </div> Fri, 23 Feb 2024 17:47:20 +0000 https://lwn.net/Articles/963322/ https://lwn.net/Articles/963322/ nicklecompte <div class="FormattedComment"> <span class="QuotedText">&gt; (Also - why isn't the "pulse" operation just FUTEX_WAKE without actually writing to the futex location? I'm certainly missing something....)</span><br> <p> I am actually more familiar with Windows events than futexes so excuse my heresy :) One problem I see is auto-reset events, where pulse specifically only wakes one *and only one* of the waiting threads. But I don't think futexes don't have an easy way of guaranteeing the wakes happen atomically in that fashion: it seems like this would wake *at least* one waiting thread, absent a very clever workaround or modifying the kernel directly. Of course I could be missing something. But I suspect Windows kernel being written to atomically wake up a single waiting thread is probably expensive to emulate on Linux.<br> </div> Fri, 23 Feb 2024 12:41:49 +0000 https://lwn.net/Articles/963318/ https://lwn.net/Articles/963318/ timon <div class="FormattedComment"> But why then do you think that the ntsync driver only serves a single application? Do you count anything running under Wine as the same application?<br> <p> From what I understand from the article and its comments, there might be numerous (legacy?) applications that use those NT features and that one may want to run on a Linux kernel with Wine, or there might be not a single application ever using this NT feature with Wine.<br> <p> That Wine wants to be able to offer the Windows APIs as complete, conformant and performant as possible seems like a laudable goal and a win for FOSS -- and I don't see why one would want to hinder their efforts by relegating their work to out-of-tree modules via DKMS, as long as there are no legal or severe technical hurdles.<br> </div> Fri, 23 Feb 2024 11:24:17 +0000 https://lwn.net/Articles/963317/ https://lwn.net/Articles/963317/ farnz <p>The OpenVPN DCO driver is out of tree not because it has a single use, but because the kernel has a guideline of "don't break userspace applications", and the OpenVPN DCO driver currently wants the freedom to make changes that will deliberately break userspace applications. This is a very good reason to not enter mainline - you don't want to be under the strictures that Linus applies. Fri, 23 Feb 2024 10:57:27 +0000 https://lwn.net/Articles/963307/ https://lwn.net/Articles/963307/ mb <div class="FormattedComment"> Yes, but this is not what I meant.<br> <p> With "single-purpose" I did _not_ mean that a driver would only drive a single type of hardware. That is _obvious_. That's the case for most drivers.<br> And I also was not referring to the number of users of this driver.<br> There are even drivers in the mainline where probably only a single instance of the hardware exists. Does it make sense? No, it does't. But that's a completely different thing. Doesn't have anything to do with how we discuss ntsync.<br> <p> I was referring to the number of _applications_ that would use the driver. I'm sorry that I didn't make that clearer to begin with. That was my fault.<br> ntsync only serves a single _application_.<br> <p> It is like the DCO driver for OpenVPN. Which is a DKMS. And I don't see anything wrong with that. Except for maybe that it breaks module signing.<br> <p> And yes, I do know that we have more features in the kernel that only serve a single application. You don't need to bring that up. ;-)<br> </div> Fri, 23 Feb 2024 06:26:59 +0000 https://lwn.net/Articles/963287/ https://lwn.net/Articles/963287/ farnz <p>Because you said "So what's wrong with dkms? Why does this single-purpose driver have to be in mainline?" - but that applies to every last one of those things that are also single-purpose drivers in mainline. <p>I'd like you to explain what criteria makes a single-purpose driver sensible to include in mainline (rather than being things that should be in dkms), such that all the single-purpose drivers in mainline that I'm never likely to use meet that criteria, but this doesn't. Thu, 22 Feb 2024 22:25:04 +0000 https://lwn.net/Articles/963285/ https://lwn.net/Articles/963285/ mb <div class="FormattedComment"> Exactly. What about all those other unrelated things over here?<br> <p> As I said: Thanks for explaining some technical details. But now you keep on saying things that are off topic.<br> </div> Thu, 22 Feb 2024 22:09:18 +0000 https://lwn.net/Articles/963283/ https://lwn.net/Articles/963283/ farnz <p>Wine needs to take over <em>before</em> the ref count drops to zero, else you've regressed. To match Windows kernel behaviour, you either need the Linux kernel to support wait-for-all somehow, or you need the Linux kernel to hand over the existing wait-for-any waits to Wine, so that Wine can correctly emulate the corner cases of wait-for-all. Thu, 22 Feb 2024 21:36:24 +0000 https://lwn.net/Articles/963284/ https://lwn.net/Articles/963284/ farnz <p>But all of those other stuff are "single use drivers" for things that I'm never likely to use; I have more chance of benefiting from ntsync than I do from all of those other things. Thu, 22 Feb 2024 21:35:55 +0000 https://lwn.net/Articles/963268/ https://lwn.net/Articles/963268/ mb <div class="FormattedComment"> That's what-about-ism. ntsync has nothing to do with analogue TV tuners and all the other stuff you mentioned.<br> <p> But thanks to all for explaining the technical details and the background behind ntsync.<br> </div> Thu, 22 Feb 2024 19:04:53 +0000 https://lwn.net/Articles/963257/ https://lwn.net/Articles/963257/ Wol <div class="FormattedComment"> That was the point of my ref-counting - wine takes over, but cannot proceed until the ref count drops to zero. Almost certainly not nice ...<br> <p> Cheers,<br> Wol<br> </div> Thu, 22 Feb 2024 17:04:59 +0000 https://lwn.net/Articles/963226/ https://lwn.net/Articles/963226/ farnz <p>You also need some mechanism to tell all the existing calls in the kernel that <tt>wineserver</tt> is taking over wait calls now, so they need to return to userspace and be retried (inside Wine) by the IPC mechanism instead of the kernel mechanism, otherwise you can't correctly implement the corner-cases of Win32 when you have both wait-for-any and wait-for-all referencing the same objects. Thu, 22 Feb 2024 15:21:15 +0000 https://lwn.net/Articles/963211/ https://lwn.net/Articles/963211/ Wol <div class="FormattedComment"> And a wait-for-any call will go direct from the app bypassing wine? <br> <p> Otherwise you could presumably ref-count wait-for-any, and if the count was non-zero you'd have to divert new calls and wait for existing calls to go to zero. Messy, assuming it's even possible ...<br> <p> Cheers,<br> Wol<br> </div> Thu, 22 Feb 2024 15:05:33 +0000 https://lwn.net/Articles/963144/ https://lwn.net/Articles/963144/ farnz <p>And just looking at the kernel, there's a huge number of single-purpose drivers that most users are never going to need - 100G Ethernet, Infiniband, FireWire, UIO, parallel SCSI, PCMCIA, amateur radio, SLIP, FPGA drivers, SGI system drivers, analogue TV tuners, multipath block I/O, old PC-style gameports, Industrial I/O (iio), obscure HID devices, CXL, ATM, PATA and probably more. The kernel's full of stuff that most people don't need - either because it's legacy (analogue TV tuners, ATM, PATA), or special case (IIO, UIO, amateur radio), or because it's too expensive for most of us (100G Ethernet, Infiniband, FPGA drivers). Thu, 22 Feb 2024 09:59:32 +0000 https://lwn.net/Articles/963141/ https://lwn.net/Articles/963141/ farnz <p>You can't do it on the first call, because to implement it properly, you need to prevent previous wait-for-any calls from using the kernel mechanism; time-travelling backwards like that is a technical challenge. And Windows handles it by having its kernel equivalent of <tt>/dev/ntsync</tt> handle wait-for-all as well as wait-for-any; it may have fast paths in there for wait-for-any, but it's all in-kernel. Thu, 22 Feb 2024 09:52:55 +0000 https://lwn.net/Articles/963133/ https://lwn.net/Articles/963133/ Wol <div class="FormattedComment"> Even better, if wait_for_all_calls is part of wine (I get the impression it might be), on the first call it could itself disable optimisation.<br> <p> Cheers,<br> Wol<br> </div> Thu, 22 Feb 2024 07:41:20 +0000 https://lwn.net/Articles/963132/ https://lwn.net/Articles/963132/ calumapplepie <div class="FormattedComment"> Can we do what windows itself does, then?<br> Grep through the binary of whatever wine is executing for wait_for_all calls, then disable the optimization if any are spotted?<br> </div> Thu, 22 Feb 2024 06:54:52 +0000 https://lwn.net/Articles/963069/ https://lwn.net/Articles/963069/ Cyberax <div class="FormattedComment"> <span class="QuotedText">&gt; So what's wrong with dkms? Why does this single-purpose driver have to be in mainline?</span><br> <p> Why not move out drivers for all of the one-off devices out of the kernel, then? DKMS significantly complicates the OS updates and secure boot.<br> </div> Wed, 21 Feb 2024 17:54:39 +0000 https://lwn.net/Articles/963049/ https://lwn.net/Articles/963049/ pizza <div class="FormattedComment"> <span class="QuotedText">&gt; So what's wrong with dkms? Why does this single-purpose driver have to be in mainline?</span><br> <p> Because there's a non-trivial amount of users that would see a substantial improvement? As I mentioned earlier in this thread, all sorts of insanity is merged into Linux every cycle that only yields a low-single-percentage improvement on narrow (and more often than not, proprietary/internal) use cases; this would seem to be a no-brainer by those same general principles.<br> <p> </div> Wed, 21 Feb 2024 14:55:03 +0000 https://lwn.net/Articles/962983/ https://lwn.net/Articles/962983/ johill <div class="FormattedComment"> Perhaps hyperbolic, but taken to the extreme that argument could also mean no drivers, filesystems, etc. really need to be in the kernel since you could always compile extra modules out of tree, put them into the initramfs, and be done with it.<br> <p> But yeah, that's not how Linux works? There might be whole architectures with fewer users than this feature would have ...<br> <p> It's also tremendously impractical with modules signing, having to have compilers everywhere, etc.<br> </div> Wed, 21 Feb 2024 13:13:16 +0000 https://lwn.net/Articles/962982/ https://lwn.net/Articles/962982/ farnz <p>The kernel maintainers want everything in mainline, not in DKMS. DKMS is meant for backports from a later mainline kernel, or for cases where legal issues prevent something being merged into mainline (e.g. licensing conditions). <p>You'd have to ask Greg K-H and others why they don't want a stable API or ABI for modules so that things can stay outside mainline forever. Wed, 21 Feb 2024 13:01:33 +0000 https://lwn.net/Articles/962981/ https://lwn.net/Articles/962981/ mb <div class="FormattedComment"> So what's wrong with dkms? Why does this single-purpose driver have to be in mainline?<br> </div> Wed, 21 Feb 2024 12:37:53 +0000 https://lwn.net/Articles/962979/ https://lwn.net/Articles/962979/ farnz <p>The justification for merging is that it's a significant performance improvement for applications running under Wine. And the justification for not putting the full functionality into <tt>futex</tt> is that the wait-for-all operation is only needed under Wine, and even then, only when you have applications using edges of the Windows API. Most of the time, pulse and wait-for-any are all that you need. Wed, 21 Feb 2024 12:24:05 +0000 https://lwn.net/Articles/962976/ https://lwn.net/Articles/962976/ mb <div class="FormattedComment"> My proposal was to not merge ntsync at all.<br> I have not yet seen a good justification for having it in mainline.<br> </div> Wed, 21 Feb 2024 12:09:47 +0000 https://lwn.net/Articles/962973/ https://lwn.net/Articles/962973/ farnz <p>And therein lies the problem: if you merge this driver without resolving the "wait for all" problem, it won't get used by Wine, and you're at risk of needing two mechanisms that do the same thing with different UAPI (this one for older Wine forks, and one with the tweaks needed to resolve the "wait for all" problem for newer Wine versions). <p>So, a solution has to be found to the "wait for all" problem; can you make it possible for Wine to transparently fall back to the old methods if an application uses "wait for all", for example? Is there a trivial way for userspace to "reclaim" all waits that are in-kernel using this mechanism, and fall back to <tt>wineserver</tt> IPC? Can you come up with a simple way to implement "wait for all" that's low performance (after all, the existing method is low performance anyway)? Is there an easy way to detect that an application uses "wait for all" waits before it uses a "wait for some" wait, and thus disable this optimization? <p>And these are questions that need answering before <tt>/dev/ntsync</tt> merges, not after. Wed, 21 Feb 2024 11:50:13 +0000 https://lwn.net/Articles/962974/ https://lwn.net/Articles/962974/ farnz <p>That's not Wine's way of working (although a Wine fork like Proton may do just that). Wine would like to remove configuration options that depend on you knowing details of what APIs your application uses, not add more, since you don't have to set options on Windows to say "this is a perfectly well-behaved Win32 app that should function on Windows NT 3.1 to Windows 11 without modification, but that gets a performance boost if you set this modification"; instead, Windows itself detects applications that can't have the performance boost, and disables it appropriately. Wed, 21 Feb 2024 11:46:47 +0000 https://lwn.net/Articles/962972/ https://lwn.net/Articles/962972/ Wol <div class="FormattedComment"> Can they add an option (default off) to enable this performance gain, in the knowledge that it may trigger a regression for in-house apps?<br> <p> After all, if they are in-house, there's always the option (provided they haven't lost the source) to work around the regression ...<br> <p> Cheers,<br> Wol<br> </div> Wed, 21 Feb 2024 11:36:48 +0000 https://lwn.net/Articles/962971/ https://lwn.net/Articles/962971/ mb <div class="FormattedComment"> Improving the performance of some old and crusty proprietary application that might exist somewhere in private is not really a good justification for getting a new locking mechanism merged into the kernel.<br> Just live with the old performance then or change it to use faster locking.<br> Or use your own special purpose wine dkms driver.<br> <p> I don't see why the mainline Linux kernel should care.<br> </div> Wed, 21 Feb 2024 11:35:43 +0000 https://lwn.net/Articles/962940/ https://lwn.net/Articles/962940/ farnz <p>Wine already has a solution to this problem; it's just excessively slow because it depends on relatively complex user-space emulation of a kernel primitive, where the <tt>wineserver</tt> process does a lot of the work to emulate the Windows kernel behaviours atop the host OS. The goal of this work is to allow you to completely bypass <tt>wineserver</tt> and rely on the kernel doing all of the synchronization work for the Windows WaitForMultipleObjects family of API calls. <p>However, if you bypass <tt>wineserver</tt>, it can't then correctly emulate the wait for all behaviour, since it no longer has all the information it needs (it needs to know about all waiters on a given object to correctly emulate Windows behaviour, since Windows has some fairness between waiters that you need to emulate to get it right). Telling <tt>wineserver</tt> before and after each wait destroys the performance gain from not doing <tt>wineserver</tt> IPC, so that's off the table. And Wine doesn't want to regress on API support; while commercial applications don't use wait for all, Wine also wants to be able to run all in-house applications perfectly, and I know that such applications have existed (since my employer 20 years ago had one, written for NT 4.0). <p>So, you're asking Wine to make a choice: <ul> <li>Regress on API support, knowing that there are probably applications that the Wine developers don't have access to that depend on the old behaviour, and thus that they're quite likely to be faced with bug reports about the regression. <li>Refuse to accept a speed-up change (not a correctness change) until it correctly handles the same API surface that the existing Wine implementation handles. </ul> <p>Wine is choosing to not knowingly regress purely for the sake of performance, and asking the people who are trying to push a performance-only change to avoid regression. How they do that is up to them; they may find a way for wait for all operations to be handled by <tt>wineserver</tt> collaborating with the <tt>/dev/ntsync</tt> mechanism, or add it to <tt>/dev/ntsync</tt> fully. Wed, 21 Feb 2024 09:52:43 +0000 https://lwn.net/Articles/962959/ https://lwn.net/Articles/962959/ itsmycpu <div class="FormattedComment"> <span class="QuotedText">&gt; Ah, so this mythical other approach is just that; no implementation much less any benchmarks showing it to be just as good or better than the kernel-based approach that exists _today_.</span><br> <p> There are several existing approaches that have the same performance, just the existing ones also have shortcomings which that approach would not have. <br> <p> <span class="QuotedText">&gt; ...Um, you do realize that Wine needs to synchronize between multiple independent heayweight *processes* ?</span><br> <p> Yes, as indicated I do, however I wonder what you mean with a "heavyweight" process?<br> <p> Shared memory is as fast between processes (I measured it), and can be read-protected or write-protected for specific processes.<br> </div> Wed, 21 Feb 2024 03:23:39 +0000 https://lwn.net/Articles/962958/ https://lwn.net/Articles/962958/ pizza <div class="FormattedComment"> <span class="QuotedText">&gt; In my view the best slide of that presentation is the one mentioning "fast user-space RPC" as a "half-baked idea", called an "interesting idea".</span><br> <p> Ah, so this mythical other approach is just that; no implementation much less any benchmarks showing it to be just as good or better than the kernel-based approach that exists _today_.<br> <p> <span class="QuotedText">&gt; I'm using something that could go by that label (though so far within a single process, between threads) for many years, and it works very well, using lock-free queues</span><br> <p> ...Um, you do realize that Wine needs to synchronize between multiple independent heayweight *processes* ?<br> </div> Wed, 21 Feb 2024 03:04:50 +0000 https://lwn.net/Articles/962955/ https://lwn.net/Articles/962955/ itsmycpu <div class="FormattedComment"> <a href="https://www.youtube.com/watch?v=NjU4nyWyhU8">https://www.youtube.com/watch?v=NjU4nyWyhU8</a><br> At about 14:06.<br> <p> In my view the best slide of that presentation is the one mentioning "fast user-space RPC" as a "half-baked idea", called an "interesting idea".<br> <p> (Such concepts exist for a long time in all kind of variations. Some people associate the concept with "actors" in a multi-threaded context, or "asynchronous message queues". I'm using something that could go by that label (though so far within a single process, between threads) for many years, and it works very well, using lock-free queues. As a low-level implementation, execution time in a loop is a small single-digit number of nanoseconds.)<br> </div> Wed, 21 Feb 2024 02:00:11 +0000 https://lwn.net/Articles/962953/ https://lwn.net/Articles/962953/ itsmycpu <div class="FormattedComment"> Maybe that was just meant to say that implementing a mutex requires similar programming techniques (atomic operations, avoiding issues like race conditions) as lock-free programming.<br> <p> Of course, there can be (and there are already some) higher level abstractions for lock-free programming as well.<br> </div> Wed, 21 Feb 2024 01:26:04 +0000