By late 2024, however, Rust enthusiasts were frustrated with stalls and blocks on their efforts, with the Rust for Linux lead quitting over “nontechnical nonsense.” Torvalds said at the time that he understood it was slow, but that “old-time kernel developers are used to C” and “not exactly excited about having to learn a new language.” Still, this could be considered a normal amount of open source debate.
But over the last two months, things in one section of the Linux Kernel Mailing List have gotten tense and may now be heading toward resolution—albeit one that Torvalds does not think “needs to be all that black-and-white.” Greg Kroah-Hartman, another long-time leader, largely agrees: Rust can and should enter the kernel, but nobody will be forced to deal with it if they want to keep working on more than 20 years of C code.
Previously, on Rust of Our Lives
Earlier this month, Hector Martin, the lead of the Asahi Linux project, resigned from the list of Linux maintainers while also departing the Asahi project, citing burnout and frustration with roadblocks to implementing Rust in the kernel. Rust, Martin maintained, was essential to doing the kind of driver work necessary to crafting efficient and secure drivers for Apple’s newest chipsets. Christoph Hellwig, maintainer of the Direct Memory Access (DMA) API, was opposed to Rust code in his section on the grounds that a cross-language codebase was painful to maintain.
Torvalds, considered the “benevolent dictator for life” of the Linux kernel he launched in 1991, at first critiqued Martin for taking his issues to social media and not being tolerant enough of the kernel process. “How about you accept that maybe the problem is you,” Torvalds wrote.
Working at the intersection of Apple’s newest hardware and Linux kernel development, for the benefit of a free distribution, was never going to be easy. But it’s been an especially hard couple of weeks for Hector Martin, project lead for Asahi Linux, capping off years of what he describes as burnout, user entitlement, and political battles within the Linux kernel community about Rust code.
In a post on his site, “Resigning as Asahi Linux project lead,” Martin summarizes his history with hardware hacking projects, including his time with the Wii homebrew scene (Team Twiizers/fail0verflow), which had its share of insistent users desperate to play pirated games. Martin shifted his focus, and when Apple unveiled its own silicon with the M1 series, Martin writes, “I realized that making it run Linux was my dream project.” This time, there was no jailbreaking and a relatively open, if tricky, platform.
Support and donations came quickly. The first two years saw rapid advancement of a platform built “from scratch, with zero vendor support or documentation.” Upstreaming code to the Linux kernel, across “practically every Linux subsystem,” was an “incredibly frustrating experience” (emphasis Martin’s).
Then came the users demanding to know when Thunderbolt, monitors over USB-C, M3/M4 support, and even CPU temperature checking would appear. Donations and pledges slowly decreased while demands increased. “It seemed the more things we accomplished, the less support we had,” Martin writes.
Martin cites personal complications, along with stalking and harassment, as slowing down work through 2024, while Vulkan drivers and an emulation stack still shipped. Simultaneously, issues with pushing Rust code into the Linux kernel were brewing. Rust was “the entire reason our GPU driver was able to succeed in the time it did,” Martin writes. Citing the Nova driver for Nvidia GPUs as an example, Martin writes that “More modern programming languages are better suited to writing drivers for more modern hardware with more complexity and novel challenges, unsurprisingly.”
Linux and its code are made by people, and people are not with us forever. Over the weekend, a brief message on the Linux kernel mailing list reminded people of just how much one person can mean to a seemingly gargantuan project like Linux, and how quickly they can disappear:
Denise Finger, wife of the deceased, wrote to the Linux Wireless list on Friday evening:
This is to notify you that Larry Finger, one of your developers, passed away on June 21st.
LWN.net reckons that Finger, 84, contributed to 94 Linux kernel releases, or 1,464 commits total, at least since kernel 2.6.16 in 2006 (and when the kernel started using git to track changes). Given the sometimes precarious nature of contributing to the kernel, this is on its own an impressive achievement—especially for someone with no formal computer training, and who considered himself a scientist.
The deepest of trenches: Linux Wi-Fi in the 2000s
That kind of effort is worth celebrating, regardless. But it’s the space that Finger devoted himself to that makes him a notably patient, productive contributor.
Getting Wi-Fi to work on a device running Linux back when Finger started contributing was awful. The chances of your hardware being recognized, activated, and working properly right after install was akin to getting a straight flush in poker. If nobody had gotten around to your wireless chipset yet, you used NDISwrapper, a Windows-interfacing kludge tool that simultaneously made your Linux install less open and yet still painful to install and maintain.
Finger started fixing this with work on Broadcom’s BCM43XX drivers. Broadcom provided no code for its gear, so Finger helped reverse-engineer the necessary specs by manually dumping and reading hardware registers. Along with Broadcom drivers, Finger also provided Realtek drivers. Many commenters across blogs and message boards are noting that their systems are still using pieces of Finger’s code today.
Fixing mainframes, science gear, and RV resorts
Larry Finger, and fish, from his Quora profile.
Quora
Finger doesn’t have a large footprint on the web, outside of his hundreds of kernel commits. He has a page for DRAWxtl, for producing crystal-structure drawings, on his personal domain, but not a general personal page. He sometimes answered Quora questions. He had a GitHub profile, showing more than 100 contributions to projects in 2024.
Perhaps the biggest insight into Finger found in one place is a three-part series for Linux Journal, “Linux in a Windows Workstation Environment,” written in 2005, when he was roughly 65. He summarizes his background: Fortran programmer in 1963, PDP-11 interfaces to scientific instruments in the 1970s, VAX-11/780 work in the early 1980s, and then Unix/Linux systems, until retiring from the Carnegie Institution for Science in Washington, DC, in 1999. The mineral Fingerite is named for Finger, whose work in crystallography took him on a fellowship to northern Bavaria, as noted in one Quora answer about the Autobahn.
“At that time, I became a full-time RV resident, dedicated to the avoidance of cold weather,” Finger writes. He and his wife Denise arrived that year at a 55-plus RV community in Mesa, Arizona. He joined the computer club, which had a growing number of Windows PCs sharing a DSL connection through one of the systems running WinGate. A new RV resort owner wanted to expand to 22 workstations, but WinGate licenses for that many would have been expensive for the club. Finger, who was “highly distrustful of using Windows 98 in a mission-critical role,” set to work.
Finger goes on across the series to describe the various ways he upgraded the routing and server capacity of the network, which grew to 38 user stations, Samba shares, a membership database, VPN tunnels, several free RJ-45 ports, and “free Wi-Fi access… throughout the park.”
Lots of people have commented on the broad work Finger did to make Linux usable for more people. A few mention that Finger also mentored people, the kind of work that has exponential effects. “MB” wrote on LWN.net that Finger “mentored other people to get the Broadcom Open Source code into kernel. And I think it was a huge success. And that was only a small part of Larry’s success story.”
In a 2023 Quora response to someone asking if someone without “any formal training in computer science” can “contribute something substantial” to Linux, Finger writes, “I think that I have.” Finger links to the stats for the 6.4 kernel, showing 172,346 lines of his code in it, roughly 0.5% of the total.
I have never taken any courses in Computer Science; however, I have considerable experience in coding, much of which happened when computers were a lot less powerful than today, and it was critical to write code that ran efficiently.
Finger suggests in his response small patches, deep reading of the guidelines, and always using git’s send-email to send patches: “Nothing will get shot down more quickly than a patch submitted from a mailer such as Thunderbird.” Finding typos and errors in comments and text strings can help, especially after translation. Finger advises being patient, expecting criticism about following rules and formats, and to keep plugging away at it.
In another Quora response about kernel driver development, Finger says, “This activity can be highly rewarding, and also equally frustrating!” You should learn C, Finger suggested, and maybe start with analyzing USB drivers, and take your time learning about DMA.
“Do not lose hope,” Finger wrote. “It took me about 2 years before I could do anything more than tell the experts where my system was generating a fault.”
Enlarge/ Cans of Tab diet soda on display in 2011. Tab was discontinued in 2020. There has never been a soda named “Spaces” that had a cult following.
Getty Images
Anybody can contribute to the Linux kernel, but any person’s commit suggestion can become the subject of the kernel’s master and namesake, Linus Torvalds. Torvalds is famously not overly committed to niceness, though he has been working on it since 2018. You can see glimpses of this newer, less curse-laden approach in how Torvalds recently addressed a commit with which he vehemently disagreed. It involves tabs.
The commit last week changed exactly one thing on one line, replacing a tab character with a space: “It helps Kconfig parsers to read file without error.” Torvalds responded with a commit of his own, as spotted by The Register, which would “add some hidden tabs on purpose.” Trying to smooth over a tabs-versus-spaces matter seemed to awaken Torvalds to the need to have tab-detecting failures be “more obvious.” Torvalds would have added more, he wrote, but didn’t “want to make things uglier than necessary. But it *mightbe necessary if it turns out we see more of this kind of silly tooling.”
If you’ve read this far and don’t understand what’s happening, please allow me, a failed CS minor, to offer a quick explanation: Tabs Versus Spaces will never be truly resolved, codified, or set right by standards, and the energy spent on the issue over time could, if harnessed, likely power one or more small nations. Still, the Linux kernel has its own coding style, and it directly cites “K&R,” or Kernighan & Ritchie, the authors of the coding bible The C Programming Language, which is a tabs book. If you are submitting kernel code, it had better use tabs (eight-character tabs, ideally, though that is tied in part to teletype and line-printer history).
By attempting to smooth over one tiny part of the kernel so that a parsing tool could see a space character as a delineating whitespace, Prasad Pandit inadvertently spurred a robust rebuttal:
It wasn’t clear what tool it was, but let’s make sure it gets fixed. Because if you can’t parse tabs as whitespace, you should not be parsing the kernel Kconfig files.
In fact, let’s make such breakage more obvious than some esoteric ftrace record size option. If you can’t parse tabs, you can’t have page sizes.
Yes, tab-vs-space confusion is sadly a traditional Unix thing, and ‘make’ is famous for being broken in this regard. But no, that does not mean that it’s ok.
Torvalds’ hidden tabs appear in the fourth release candidate for Linux kernel 6.9, which Torvlads wrote had “nothing particularly unusual going on” the week of its release.
Disclosure: The author is a tab person insofar as he has any idea what he’s doing.
This post was updated at 6: 33 pm Eastern to fix some line-break issues in the Torvalds blockquote. The irony was duly noted. A better link regarding the Tabs Vs. Spaces debate was also swapped in.
Reiser, 59, serving a potential life sentence in a California prison for the 2006 murder of his estranged wife, Nina Reiser, wrote back with more than 6,500 words, which Brennan then forwarded to the LKML. It’s not often you see somebody apologize for killing their wife, explain their coding decisions around balanced trees versus extensible hashing, and suggest that elementary schools offer the same kinds of emotional intelligence curriculum that they’ve worked through in prison, in a software mailing list. It’s quite a document.
What follows is a relative summary of Reiser’s letter, dated November 26, 2023, which we first saw on the Phoronix blog, and which, by all appearances, is authentic (or would otherwise be an epic bit of minutely detailed fraud for no particular reason). It covers, broadly, why Reiser believes his system failed to gain mindshare among Linux users, beyond the most obvious reason. This leads Reiser to detail the technical possibilities, his interpersonal and leadership failings and development, some lingering regrets about dealings with SUSE and Oracle and the Linux community at large, and other topics, including modern Russian geopolitics.
“LKML and Slashdot.org seem like reasonable places to send it (as of 2006)”
In a cover letter, Reiser tells Brennan that he hopes he can use OCR to import his lengthy letter and asks him to use his best judgment in where to send his reply. He also asks, if he has time, Brennan might send him information on “Reiser5, or any interesting papers on other Filesystems, compression (especially Deep Learning based compression), etc.”
Then Reiser addresses the kernel mailing list directly—very directly:
I was asked by a kind Fredrick Brennan for my comments that I might offer on the discussion of removing ReiserFS V3 from the kernel. I don’t post directly because I am in prison for killing my wife Nina in 2006.
I am very sorry for my crime–a proper apology would be off topic for this forum, but available to any who ask.
A detailed apology for how I interacted with the Linux kernel community, and some history of V3 and V4, are included, along with descriptions of what the technical issues were. I have been attending prison workshops, and working hard on improving my social skills to aid my becoming less of a danger to society. The man I am now would do things very differently from how I did things then.
ReiserFS V3 was “our first filesystem, and in doing it we made mistakes, because we didn’t know what we were doing,” Reiser writes. He worked through “years of dark depression” to get V3 up to the performance speeds of ext2, but regrets how he celebrated that milestone. “The man I was then presented papers with benchmarks showing that ReiserFS was faster than ext2. The man I am now would stat his papers … crediting them for being faster than the filesystems of other operating systems, and thanking them for the years we used their filesystem to write ours.” It was “my first serious social mistake in the Linux community, and it was completely unnecessary.”
Reiser asks that a number of people who worked on ReiserFS be included in “one last release” of the README, and to “delete anything in there I might have said about why they were not credited.” He says prison has changed him in conflict resolution and with his “tendency to see people in extremes.”
Reiser extensively praises Mikhail Gilula, the “brightest mind in his generation of computer scientists,” for his work on ReiserFS from Russia and for his ideas on rewriting everything the field knew about data structures. With their ideas on filesystems and namespaces combined, it would be “the most important refactoring of code ever.” His analogy at the time, Reiser wrote, was Adam Smith’s ideas of how roads, waterways, and free trade affected civilization development; ReiserFS’ ideas could similarly change “the expressive power of the operating system.”
