So, how do I handle repairing a RAID-5 in a server I can’t touch?

Two drives failed in a 5-disk RAID-5 array at a client who dropped our services; fortunately, I’d put in a backup system, so when they brought us back on, I had a full backup from midnight to restore from. Unfortunately, only one drive was on order for various reasons and they needed it back up as soon as possible…now, here’s the million dollar question:

How do you reconstruct a RAID-5 array with 3 good drives remaining, which once was a 5-drive array, ? Consider the following:

• The original array was 5x 500GB drives, of which only 3 remain
• The backup data is about 500GB (so just RAID-1 for the data isn’t a choice)
• The Linux boot/root filesystem sits on a RAID-1 across the first partitions of all five hard drives (resilient against up to all but the last single drive failing)

Since the software is still resilient against three drive failures, concerns over that are pointless. The client doesn’t have time to wait on a spare drive or two, but we want to order up a spare because of possible future capacity needs so we can pull off the next step. What was the best compromise? Simple!

mdadm --create /dev/md1 -l 5 -n 4 /dev/sd[abc]2 missing

This creates a 4-drive RAID-5 from the existing old 5-drive RAID’s partitions that is intentionally degraded for when we add the fourth disk in later. From there, I wrote a new XFS filesystem, mounted it, and restored the data. This is what /proc/mdstat looks like now on that system (note that no reboot was needed for these repairs at all!):

# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md1 : active raid5 sdc2[2] sdb2[1] sda2[0]
      1406519040 blocks super 1.2 level 5, 128k chunk, algorithm 2 [4/3] [UUU_]

md0 : active raid1 sda1[0] sdb1[1] sda1[4] sdd1[5](F)
      19542976 blocks [5/3] [UU__U]

unused devices: <none>

Manually copying a RAID-0 striped array to a single drive for data recovery

This question was posed on a forum:

I have a customer who has a computer, 2 SATA disk (striped in RAID config. Windows won’t load. Diag reports bad hard drive. When I disconnect one, it kills the stripe and the computer appears to not have a hard drive at all. Seems kind of silly to have it set this way as it increases the risk of failure. Other than putting each hard drive in another computer, I’d like to determine which of the disk are bad.

Also, not quite sure how to attack data recovery as they are a stripe set and plugging in to a SATA to USB does not appear to be a valid method. If I put a third hard drive in as a boot drive, do i have to reconfig the stripe set and if i do, will it kill the data.

I have reassembled two RAID-0 “striped” drives to a single larger drive by hand before. It’s actually a programmatically simple operation, but you require a lot of low-level knowledge and some guesswork to do it. The specific pair I had appeared to store the metadata somewhere other than the start of the disk, and I was able to discover through a hex editor that the drive was on a 64KB stripe size. I also spotted which drive had a partition table and which didn’t, because that’s only on the first drive which contains the first stripe.

At a Linux command prompt, with the two RAID-0 disks (that were NOT being detected properly by the Linux LVM2 “FakeRAID” algorithms, by the way) and a disk of twice their size connected, I wrote a very simple script that looked something like this (sda/sdb as RAID-0, sdc as the destination disk, and this might work under ash or similar as well).

—- cut here —-

#/bin/bash

; X=sda position, Y=sdb position, Z=sdc position, STRIPE=stripe size
X=0; Y=0; Z=0; STRIPE=65536

; Retrieve the size of a RAID-0 disk so we can terminate at the end
SIZE=$(cat /proc/partitions | grep ‘sda$’ | awk ‘{print $3}’)
; Divide size by stripe, including any tail blocks (expr truncates)
SIZE=$(( SIZE + STRIPE – 1 ))
SIZE=$(expr $SIZE / $STRIPE ))
while [ "$Z" -lt "$SIZE" ]
do
dd if=/dev/sda of=/dev/sdc seek=$Z; skip=$X bs=$STRIPE count=1
Z=$(( Z + 1 ))
dd if=/dev/sda of=/dev/sdc seek=$Z; skip=$Y bs=$STRIPE count=1
Z=$(( Z + 1 ))
X=$(( X + 1 ))
Y=$(( Y + 1 ))
done

—- cut here —-

Note that all it does is load 64K at a time from each disk and save it to the third disk in sequential order. This is untested, and requires modification to suit your scenario, and is only being written here as an example. It does not fail if a ‘dd’ command fails, so it will work okay for data recovery; you will lose any stripe that contains a bad block, though, and the algorithm could be improved to use dd_rescue (if you have it) or to copy smaller units in a stripe so that only a partial stripe loss occurs on bad data.

My take on Windows 8, Metro, touchscreens, and other desktop disasters

Windows 8 has this shiny new user interface that’s known as “Metro.” I hate Metro. LOTS of people hate Metro. Metro is supposed to be easier for touchscreen usage, but Windows is a desktop operating system. I don’t want to re-hash everything that other people have written about why Metro is garbage, so I’ll just drop a few points to get my ideas across.

• Metro is designed specifically with touchscreens in mind. Some all-in-one desktop computers are now touch-capable, and Windows 8 is supposed to become available for ARM architectures so that Windows 8 can be used on new tablets. However, there are two major problems: MOST desktop and laptop computers DO NOT HAVE TOUCH CAPABILITY AT ALL (that’s the vast majority of what it runs on) and TOUCH IS NOT PRACTICAL FOR DESKTOP USE.
• Touchscreens require holding your arm up to manipulate what we traditionally would use a mouse and pointer to work with. That’s fine for a minute, but if you think your arm is NOT going to get tired ten minutes into touchscreen-centric hell, you’re fooling yourself.
• Have you seen the Explorer windows? They brought that awful, terrible “ribbon UI” from Office 2010 into Windows 8. Not only is it annoying as hell to use, it’s counterintuitive: with monitors trending towards widescreen displays, vertical screen space is in much shorter supply, while ironically still being the most needed type of screen space for office applications and for seeing more files at once in Explorer’s “details” file view. Yet somehow, Microsoft’s logic is to replace one toolbar with something that’s three toolbars in height. Way to go, you idiots. (If a ribbon popped out of the left or right, it’d make more sense, but ribbon organization is actually less efficient than toolbars, in my opinion.)
• Start button in desktop mode: GONE. WHY?! The Start button paradigm was revolutionary. There’s a reason that it’s persisted since the introduction of Windows 95, and is often imitated in many Linux desktops: it gets the job done, and does so pretty well, as long as you didn’t have 100 folders inside it (and Vista fixed that with the introduction of a scrolling Start menu programs list that ACTUALLY HAS A FREAKING SCROLL BAR…what took so long to come up with that?!)

If I was to advocate for a radical UI change, I’d want to see something more like Fluxbox on Linux systems. I can right-click anywhere on the “desktop” to get a program menu, with no Start button required. If I use a Fluxbox theme with rounded top corners on the windows, I can launch my mouse to the upper-left or upper-right corners of the screen (two of the most prominent “hotspots” as any skilled UI designer will tell you) and right-click to get said menu as well. Right-clicking on the title bar brings up all of the window management functions I could ever need. Fluxbox isn’t the prettiest thing in the world, and it’s a little weird to someone who is used to choosing between “Start menu” and “Mac dock” ways of working with programs, but being able to call up a Start menu of sorts without even needing the button in the first place isn’t hard to get used to, and is much faster than having to aim for a button.

Honestly, I’ve gotten spoiled by Fluxbox and Linux. I can’t believe how fast a huge application like Firefox starts up under Fluxbox. Ubuntu and other distributions with heavy full-blown desktop environments are on par with Windows, but with a minimalist one like Fluxbox, the world just seems so  much faster, even with an unaccelerated VESA video driver.

I digressed a bit, but the moral of the story is this: simple is beautiful, fast, and functional. All this metro/ribbon/touchy crap wastes screen space, slows things down, and frustrates users. I knew things were going sour when Windows had keyboard shortcut accelerator underlining disabled by default, but I didn’t know we would end up with this Metro disaster. I’m making a call out to everyone to advocate for a simpler desktop that doesn’t need to change for the sake of change because it’s functionally sound and easy to work with, without the eye candy and bells and whistles and massive tool ribbons.

Linux PowerPC yaboot + initramfs/initrd woes solved; no more “unable to mount root fs” problem!

This one had me ripping my hair out for two days straight. Anyone who has tried to create a Linux bootable CD for a PowerPC system has either run into this problem, followed some kind of magic set of directions that don’t explain the details that could cause this problem, or do something crummy like using the CD as the root filesystem.

PowerPC systems are very different from i386/i686/x86_64 systems in how they boot, and because they are much less common, they garner less interest and also have less available documentation and Internet forum assistance. The specific problem that I ran into is this: using the Tritech Service System’s construction for x86 as a template, and gleaning information from other PPC bootable CD images, I was able to create a CD that would properly boot the iBook G3 I used for testing into yaboot, the PowerPC Linux loader. The process of figuring out how to pull this off took many hours of reading and dissection, and I could easily chalk a full day’s work up as wasted on this process due to the fact that it’s not well-documented. From there, yaboot was configured to load my kernel and initrd (in this case, an initramfs, not an initrd, but the loading process is the same.) However, I was greeted every single time with kernel output that showed no indication of any initrd/initramfs being loaded and handed off to the kernel. I was stumped. It seemed as if I had done everything that the others do, yet it didn’t work. I tried these things to resolve the problem, to no avail:

• Copying the map.hfs file from another Linux distribution that seemed more complete
• Editing yaboot.conf to add and remove things like ramdisk_size=16384 or device=cd: to the options
• Recompiling the PPC32 kernel with initrd turned on (shouldn’t be needed for initramfs, but I was quite annoyed and desperate)
• Playing with the ofboot.b text file to see if anything inside could make a difference (CHRP is becoming a dirty word in my book)
• Booting the G3 to Open Firmware and typing excessively cryptic and obnoxious commands that make learning “sed” look like a cakewalk
• Pondering the consumption of potent alcoholic beverages while at work to defer blame for not figuring this nonsense out

So, after two days of trying to go from a collection of packages and a kernel to a real-world bootable Tritech Service System 2.7.6 ISO for PowerPC Macs, and nearly losing my sanity in the process, I finally hit upon an obscure, nasty, rarely discussed, extremely STUPID, yet horribly important fact:

yaboot doesn’t load initrd or initramfs if the kernel image is compressed.

Yes, that’s it. That’s the source of my ills. The godforsaken bootloader will detect a compressed kernel and simply and quietly ignore the “initrd=/boot/initrd1.gz” parameter. The even simpler solution? Instead of using the compiled kernel at arch/powerpc/boot/zImage.pmac, one must use the compiled kernel at…well, you might not believe this…just plain vmlinux. The uncompressed raw kernel image produced immediately under the Linux kernel folder you build in. That’s all that I had to do, and I have never been so pissed off over such a small detail in my life.

All too often in the computer world, I see the “user” aspects of things documented repeatedly and done to death; entire volumes have been written just to explain how to perform basic functions or configure a program to the liking of the user. Even the process of compiling a Linux kernel is so thoroughly documented and explained that it’s fairly hard to fail to do it if you use a decent guide. Why is it, though, that these crucial points involving low-level details and bootloader quirks are overlooked and go largely undocumented? If I type “yaboot initramfs” into Google or Yahoo or Bing, why doesn’t the very first page that appears scream at me in bold text “YABOOT WILL NOT LOAD INITRD IMAGES IF THE KERNEL IMAGE IS COMPRESSED!” I know of approximately ZERO bootloaders that have this obnoxiously non-standard behavior. I’ve messed with LILO, SILO, GRUB, SYSLINUX, ISOLINUX, PXELINUX, BootX, and U-Boot on a $99 WM8650 ARM-based netbook, and not once have I run into this problem with ANY of those bootloaders AT ALL.

I hope that this information helps anyone trying to master a Linux on PowerPC bootable ISO to not waste two days and use their CD burner to create ten useless shiny silver coasters in the process. Also, could someone explain to me WHY the yaboot bootloader can’t load both images as compressed images?

Kernel panic – not syncing: Attempted to kill init! (glibc problem)

While working on the Tritech Service System, I made the mistake of using a glibc package compile for an i686 in the initramfs for an i586 kernel. I happened to do some searching to figure out the source of the problem, since all of my kernels would crash with this message in the exact same place, and thought I’d share it with everyone. This could frustrate custom Linux distro attempts easily.

In short: if the kernel panics because something “attempted to kill init!” then make sure your C library (glibc, eglibc, uclibc, dietlibc, whatever) is not compiled for a CPU higher than the CPU you’re trying to run on.

What’s happening is the system is attempting to execute “init” which immediately terminates due to the fact that the library uses invalid CPU instructions (the older processor doesn’t know about the newer instructions compiled into the library). The message “attempted to kill init!” is technically correct: init was killed because it tried to do something bad, but init is required to run anything else, so once init immediately crashes out, there’s nothing left for the system to do, and the kernel hangs itself up.

It’s 2012, and the Embeddable Linux Kernel Subset (ELKS) is back!

A long time ago, some Linux kernel developers forked the Linux kernel (in truth, it was more “rewrite” than “fork”) into a version that would run on 8086/8088-class processors, like in the original IBM PC and PC/XT. That kernel was coined the Embeddable Linux Kernel Subset, or ELKS for short. The project has seen very rocky times in the 2000s as developers gradually moved away from the project, interest slowly faded, and perhaps the near-death blow for ELKS was the tragic loss of Riley H. Williams, whose contributions of time and skill kept ELKS going for years until his death. (Rest in peace, Riley; we miss you.) After Riley died, no one who was involved had the time or energy to steer the ELKS project anywhere. I released a broken tarball of all of the source code contained in the ELKS CVS database on SourceForge, dubbed it release version elks-0.1.3, and hoped that someone else with more time and knowledge than myself would be inspired and take the initiative to get the ball rolling again.

Other than an occasional blip on the ELKS mailing list (linux-8086 at vger.kernel.org), nothing really happened. I tried to spark some conversation about where ELKS should go, but almost nothing came of that effort either. ELKS was doomed to obscurity, and even I mostly forgot about it as I was distracted by other things in my life. And then, something changed!

Juan Perez-Sanchez posted a patch to linux-8086 to fix compilation errors on 2012-01-10. I starred it but didn’t really think much of it, because I was very busy that day, and I forgot about it. Juan, though, wasn’t kidding around. Over the remainder of January of 2012, Juan posted four more patches to the mailing list to fix a variety of small issues. Hans posted that “it is good to know that ELKS is not completely abandoned,” and after seeing that message in my inbox on top of all these patches, I couldn’t keep myself away any longer.

I transitioned the code on SourceForge from CVS to Git on Feb 5, and the next day I imported the CVS commit history into Git as well. I’ve decided to fully take the reins, suck it up, and become the leader of ELKS, particularly since I am the only administrator left on SourceForge who is active. So far, there has not been a single day since 2012-02-05 that the list hasn’t had a post. I’ve begun digging up some old hardware to test on, and I can safely say that ELKS isn’t dead.

ELKS is back. Anyone who wants to help should subscribe to the linux-8086 mailing list at vger.kernel.org and contribute!

Tritech Service System 2.1 progress

So far, only a few bugs remain in TSS 2.1, which is currently at  version 2.1-alpha5.  There are some problems currently being worked out with KMS support, which is the biggest issue so far.  The entire “init” system has been rewritten to replace traditional “init” with the runit-init tools provided by BusyBox.  The move to a partially modular kernel has been done, and we’re testing that out on machines to make sure it behaves as expected.  (Modular support is necessary for drivers that must load firmware, like most wireless network adapters and some of the KMS video drivers in the kernel).

We haven’t set up the “beacon server” features yet, mainly because some security issues need to be addressed.  Persistent home support has been thrown out for now, since we will be replacing it with something more robust in the future.  We’re not too far from the 2.1 release.  It’s going to be pretty sweet!  Stay tuned!

Update (2011-04-08): Up to -alpha7b2, trying to help the Linux kernel developers with some serious framebuffer issues.  It seems that Intel i915 framebuffers (inteldrmfb) like to cause a completely black screen at boot time on plenty of computers (and that’s even using today’s most recent git pull of the kernel code).  Radeon and nVidia once in a while has the same issue.  In the meantime I’ve built a second kernel that has all the graphics stuff stripped out so that we can use console-only mode.

Helping Linux applications play sounds on remote Windows and Linux machines

(If you just want the good stuff and don’t care about the story, visit the snd2remote web page. Bottom line: run a listener on the LAN, then on a Linux box type “snd2remote C:/WINDOWS/Media/ding.wav”)

Problem: You’re running applications on a remote Linux system across a LAN, possibly using an X server such as Xming or x11vnc to view them on a Windows machine.  The programs run fine, displaying on the Windows machine, but there is no way to hear notifications on events such as receiving an email or IM. Every program has an option that lets you “run a custom command” on notification events, or has an available add-on that enables this functionality. There is no custom command that makes sounds play on the Windows machine you’re actually using.

Solution: Roll your own solution using free tools that are already available!

I use Mozilla Thunderbird for email and Pidgin for all instant messaging services. For two years, I have sat directly in front of the server at the shop and used these applications. Now, however, I use an old Windows laptop that we can’t sell for technical reasons along with Xming to run these Linux apps on my sever and show them as apps on my Windows laptop.  Unfortunately, as I have been getting busier and busier, I have been forgetting to periodically check my mail window.  Thunderbird in Xming has no tray mail notification and no sound on the Windows machine at all; likewise with Pidgin, which (at best) can put a star in the chat window title.  However, Pidgin on Linux comes with an option to “use a custom command” for playing sounds.  Thunderbird doesn’t, but an add-on easily adds “during new mail notification, run this command” capabilities to it.

So, I know that I CAN do this if I can run a command which plays the sound on the Windows machine, but a brief Google hunt turns up nothing of interest.  Oh, I’m sure somehow I can use JACK or some other complicated audio system, but I could care less about setting that up and dealing with the extra admin overhead if there’s a simpler way!

It takes some real Linux geek thought processes to do, but I figured out a way that, regardless of which machine in my shop I sit at, I can get audio notifications for my Xming apps (or any app that can run a command to play a sound for that matter).  The answer: UDP broadcast the sound you want to play to all the machines on the network.

I created a bash shell script and a Windows NT/2000/XP command prompt  batch file (it uses SET /P which I don’t think DOS/Win9x/Me have) that pair together to trigger sounds on my Windows machine from my Linux machine. An additional constraint which I artificially imposed on myself to minimize the number of required downloaded files was to only use Windows batch commands and not “cheat” by pulling a UNIX-style shell, sed, grep, and other UNIX commands.  This also forced me to learn some ways to do things in a batch file that I had never done before, meaning that not only do I make this easier to duplicate for others, but I’ll be a better Windows admin too.

I had to download the following free tools and Linux packages to make it work (links provided):

• socat on Linux (netcat doesn’t let you broadcast UDP)
• A native win32 port of netcat (as nc.exe) from Jesse Morgan’s website
• PlayWav from Mu Consulting (to play WAV files, I originally used Media Player Classic but it was very slow)
• Mailbox Alert add-on for Mozilla Thunderbird (for running a custom command as a notification)

Here’s how we do it. The computer which is generating the notification event needs to order the computer at which we’re actually sitting to play some sort of sound. Ideally, we need to be able to play lots of different sounds, too. So, the quick and easy solution was to use socat (and netcat for Windows since there’s no native socat port) to broadcast the names of sound files to play and receive those broadcasted names, then try to play them.  Most of the time I spent was actually on file scanning and filtering code rather than making the *cat tools behave nicely.

“snd2remote” on Linux can both listen for events AND broadcast them, and does some fairly clever search tricks to try very hard to play a WAV file that is not valid for the Linux machine as-is.  The Windows version is restricted to batch file commands, which means it’s much “dumber” and therefore needs more hand-holding; most of the clever logic is in the Linux side so the Windows side can afford to be dumbed down.

snd2remote taught me the following lessons and/or has the following nice features:

1. You can make netcat exit immediately on receipt of a single UDP packet in listen mode by piping an empty “echo” into it. This drastically decreases latency because the 1-second timeout is unnecessary.
2. The way to remove timeout latency in socat is to specify a “udp4-recvfrom” endpoint instead of “udp4-listen” and also specify the “unidirectional” switch.
3. Translating to backslashes for dumb Windows batch listeners and letting Linux listeners translate them back to normal forward slashes makes life a thousand times easier.
4. Making the Linux version assume UNC paths such as \\server\share\path\file.wav equals /home/share/path/file.wav on the Linux side allows the use of a default Samba home directory share as a source for both machines to fetch the same WAV files from.
5. Letting the Linux script perform quick scans in a couple of obvious locations for alternate copies of the specified file makes it extremely easy to feed Windows-specific paths and have them still play on the Linux listeners without replicating the path to the same file on Windows.
6. snd2remote in listen mode checks the ROOT of the executing user’s home directory, does full subdirectory scanning of a custom directory (defaults to $HOME/media), and performs the previously mentioned UNC path to home directory translation, in an exhaustive effort to find a playable copy of the requested WAV file.
7. It’s better to use the shell construct $(commands) than to use (backticked) `commands` because the latter interprets special characters in a way that makes life very painful.

By running “snd2remote -l” on a Linux box or “snd2remote-listener.bat” on a Windows box, any other Linux box on the LAN can broadcast a sound event to all listening computers with a simple command:

snd2remote [-q] /path/to/file.wav

For example, to play the classic Windows “ding” on all listeners:

snd2remote C:/WINDOWS/Media/ding.wav

If you are interested in the internal workings of these scripts, they are heavily commented to explain in detail what is going on.  Lots of shell/command interpreter constructs and command combinations are used in them which I had not needed, or had rarely used, before; therefore, these scripts would be a good starting point to nudge your way into some interesting scripting.

The code has grown too long and complex to document everything here, but it is freely available at the c02ware snd2remote page.

Sort compressed tar archives to make them smaller… 20% smaller!

How would you like for your file archives to be 20% smaller, with just the tools every Linux distribution already provides and a little ingenuity?  Read on and see how I did it!

There was a folder “NESRen” that I wanted to pack up for archival, and I knew it contained many files that share the same data, minus a few changes here and there.  When packing them up and compressing them into a tarball archive, I knew that I will achieve better compression if these largely similar files are put side-by-side in the archive so that the repeated blocks of data “compress themselves away” and take up almost no space.  Unfortunately, the GNU “tar” command in nearly every Linux distribution packs up files and folders in the order that the underlying filesystem chooses, which is almost always unordered and not optimal for compression.

How do we make tar put files in an order which will compress better?  The answer is to use the tar -T option, which lets you feed tar a list of files to pack up.  The list is processed line-by-line, and each file is packed up in the order provided.  You can, for example, create a list of files with the “find” command, then hand-edit that list to be optimal, and pass the list to tar (you must use the –no-recursion option when creating the archive from this list since the “find” makes a recursive list already):

find folder_of_files/ > list.txt
vi list.txt
tar -c --no-recursion -T list.txt | xz > archive.tar.xz

In my case, however, the folder structure and naming conventions allowed for creative use of the “sort” command to arrange the files. Since there is one folder “NESRen” followed by a series of categorizations, followed by the file names themselves (i.e. “NESRen/World/Pinball (JU).nes”) I can do something like this to make all files with the same name sort beside each other, regardless of the name of the category directory (as “sort” with no options would do):

find NESRen | sort -t / --key=3 | \
  tar -cv -T - --no-recursion | xz -e > NESRen.tar.xz

The “-t /” tells sort to use a slash as a field delimiter, and –key=3 tells it to sort by the third field (NESRen is field 1, the folder is 2, the file is 3).  What kind of difference did that make for the size of my .tar.xz archive?  Take a look (-nosort archive created with “tar -c NESRen | xz -e > NESren-nosort.tar.xz”):

Size of each file in bytes:

212958664 NESRen-nosort.tar.xz
170021312 NESRen.tar.xz

Size of the original folder and each file in megabytes:

705M    NESRen
204M    NESRen-nosort.tar.xz
163M    NESRen.tar.xz

By sorting the files, I saw a 20.1% drop in archive size using the exact same compression method, with a total compression ratio of 23.1% versus the unsorted 28.9%.  That’s a huge difference!  If this were 70.5GB instead of 705MB and the data exhibited identical performance, the final archive would be 4.1GB smaller–nearly the entire capacity of a single-layer DVD-R in space savings, just by sorting the file names before compression.

Applying a similar sort-then-compress process to the packing of the “ext” version of the Tritech Service System, a 700KB reduction in the total size of the archive containing “ext” was seen.  Of course, this doesn’t help as much because the archive itself was already 32.7MB in size (700KB is only a 2.1% reduction) but it still means shorter load and boot times due to less overall data to handle.

Next time you’re packing a lot of stuff up, see if you can use these tricks to improve your compression ratio.

Work toward Tritech Service System 2.1

I’ve officially started the ground work on version 2.1 of the Tritech Service System.  Major changes and bug fixes that are already in the works:

• Changing startup scripts and Busybox “init” to Busybox “runit” style: The biggest benefit of this change is that system services will be able to start in parallel, which will lower startup time drastically.
• Partially modular kernel: Some drivers in the kernel work better as modules, and some don’t work at all unless they’re modules because of missing firmware files (specifically many wireless cards). While most of the kernel will remain monolithic, select drivers will be modularized to increase usability of the system.
• Wireless support that actually functions: Previous versions didn’t support wireless adapters in any way that could be considered usable for most configurations. TSS 2.1 is going to include working wireless support in the kernel and supporting software.  Actual “easy” wireless configuration tools are not planned until a 3.0 release.
• Basic support for packages: In our private TSS 2.0.7, we’ve included early support for loading packages from local media and our internal network server. What we use currently is not good enough for public release, but it does lay the foundation for that type of support.  TSS 2.1 will have a working implementation of boot-time package loading support, which allows extending the system without completely rebuilding initrd packages.
• Native KMS driver/X.org support: Fixes the long-standing issues with most nVidia and some Intel graphics controllers failing to work with TSS out-of-the-box. X.org auto-configuration is also being implemented.  At least one test system with two video cards automatically set itself up with dual displays after adding Intel KMS and Nouveau KMS to the kernel!
• udev automatic mount point handling: No more /mnt folder with tons of mount points for nonexistent devices! Mount points will be created more like mainstream Linux distributions, with volume label mount points in /media and device name mount points  in /mnt, all automatically handled by udev.
• Software updates across the board: New versions of important tools such as the NTFS-3G driver from Tuxera, the latest Xine media player, and more are included with Tritech Service System 2.1. Additionally, old unnecessary packages and general junk files have been removed.
• Improved boot times: Parallel service startup, a smaller initrd file to boot, and use of faster compression technologies all contribute to a much quicker startup time than previous versions.

Visit http://c02ware.com/tss.php periodically so you don’t miss the release!