<< Previous | Home | Next >>

mdadm: replacing a disk

It's the first time for me that a HDD in a raid1/5 array fails on me. I guess there has to be a first time for everything.

Anyways, I have a 6-2TB disk raid5 array for a total capacity of 10TB. I've had that array for now about 5 years. It's filled with 2TB hard disk drives and has been working pretty smoothly for me. 450MB/s read throughput is how I like it. It's even faster than my GB network, so this has never been the culprit in any of my operations.

For the record, I used the tool idle3-tools from cbothamy to reset my sleep time to something worthy. I have WDC green drives (see the picture, which is actually the drive that failed), and their default settings are really nuts. Really nuts.

Also for the record, my boot drive is a single old Maxtor 320GB drive and has nothing to do with mdadm. So if you have issues with booting onto your mdadm array, look no further. This page is not for you.

Now, I've had a failure this morning and I've had to change one of the drive. I have a "regular" box (See here for pics and french text) so there was no hot-swapping involved. But just a 5 minute downtime was good enough for me.

How did I find out about the faulty drive? Well, I have an "Error" section in my conky configuration and it outputs the diff of the regular /proc/mdstat with the initial one. Just logging into my desktop alerted me right away with a big red section that is usually empty right on my desktop. This is important, because if you don't monitor your raid array, it'll fail eventually on more than one drive and you'll lose everything. Note that from time to time, a red section appears in my conky as mdadm runs a routine check of the raid array.

Now, how do you get down to it. First, I needed to figure out which drive in my raid array was faulty. Here was the result of my mdstat file:

pieroxy@ubuntu-home-server:~$ cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid5 sdd1[3] sdc1[2] sdb1[1] sda1[0] sdf1[6](F) sde1[4] 9767552000 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/5] [UUUUU_] unused devices: pieroxy@ubuntu-home-server:~$

First a little explanation. The [UUUUU_] indicates the number of good disks (U) vs. the number of faulty disks (_). Similarly, the 6/5 indicates the number of total disks in the array (6) vs. the number of working disks (5). At last, the (F) next to sdf1 indicates the partifion that isn't available anymore.

Well... clearly one of the drives has made it to heaven, and it was /dev/sdf. Or has it? Could it just be a software glitch? In doubt, I reboot the server to see if things will get better. After a fight against the BIOS to make it boot with a faulty drive, I notice the same thing. I kinda hoped it would be a defect in the disk driver. Alas...

Well, the HDD seems to be dead. Let's remove it.

First, let's remove the drive from the array in mdadm:

pieroxy@ubuntu-home-server:~$ sudo mdadm --manage /dev/md0 --remove /dev/sdf1 pieroxy@ubuntu-home-server:~$ cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid5 sdd1[3] sdc1[2] sdb1[1] sda1[0] sde1[4] 9767552000 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/5] [UUUUU_] unused devices: pieroxy@ubuntu-home-server:~$

This is done and mdadm is now aware that there is no disk /dev/sdf1 anymore. Next, I need to open the box and locate the good drive. Now... which one is it? I have 6 WDC green in my box... Let's list all the drives with their serial number:

pieroxy@ubuntu-home-server:~$ sudo lshw -class disk | egrep "(logical|serial|disk)" *-disk logical name: /dev/sdg serial: V60QMY5G *-disk:0 logical name: /dev/sda serial: WD-WCAZA6106876 *-disk:1 logical name: /dev/sdb serial: WD-WCAZA5914364 *-disk:2 logical name: /dev/sdc serial: WD-WCAZA5891419 *-disk:3 logical name: /dev/sdd serial: WD-WCAZA5880785 *-disk:0 logical name: /dev/sde serial: WD-WCAZA5885124 *-disk:1 logical name: /dev/sdf pieroxy@ubuntu-home-server:~$

Well... The faulty drive doesn't show its serial number anymore. The good news is that I have the serial numbers of the other drives and I'll be able to locate the one that I don't have.

I just opened the box, looked for a drive with a serial number not on my list, swapped it with a new one and put everything back together. Note that I did take care of making sure I did not swap SATA plugs. I wanted to make sure the new drive was on the same SATA plug the faulty one was on. I don't know if it matters.

After booting up, my raid array was in the same state as before shutting down. First, I had to partition the drive the same way the others were partitioned. I ran (as root):

root@ubuntu-home-server:~$ sfdisk -d /dev/sde | sfdisk /dev/sdf root@ubuntu-home-server:~$

Now, don't get mixed up with your drives. This will effectively erase the partition table of your drive, copying from /dev/sde to /dev/sdf. Depending on your setup, you can lose everything on the target drive. Here it was /dev/sdf, my newly installed drive.

The last thing I needed was to add the new drive to the array:

pieroxy@ubuntu-home-server:~$ sudo mdadm --manage /dev/md0 --add /dev/sdb1 pieroxy@ubuntu-home-server:~$

And that's all... almost. Now mdadm will rebuild the new drive with the data it should contain. It took about 500 minutes for me, a little more than 8 hours, so be patient. This will take time.

A few hours later, and my raid array is slowly recovering from the disaster:

pieroxy@ubuntu-home-server:~$ cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid5 sdf1[6] sdb1[1] sda1[0] sdd1[3] sdc1[2] sde1[4] 9767552000 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/5] [UUUUU_] [================>....] recovery = 83.5% (1631248716/1953510400) finish=102.6min speed=52327K/sec unused devices: pieroxy@ubuntu-home-server:~$

As you can see, I still have 102.6 minutes left.

When everything was done:

pieroxy@ubuntu-home-server:~$ cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid5 sdf1[6] sdb1[1] sda1[0] sdd1[3] sdc1[2] sde1[4] 9767552000 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6] [UUUUUU] unused devices: pieroxy@ubuntu-home-server:~$

I have to say I'm surprised at how smooth everything went. I haven't lost a byte and I still have a spare drive, although I'll need a few others if/when this happens again.

My first experience with mdadm was kind of a disaster, but it was because I used it as my / partition. Debugging it in busybox wasn't a lot of fun. But this time around, everything went smooth.


Bootstraping my mdadm CLI skills: http://tldp.org/HOWTO/Software-RAID-HOWTO-6.html Where is that damn disk again? http://unix.stackexchange.com/questions/121757/harddisk-serial-number-from-terminal mdadm recovery, removing, adding a drive. https://www.howtoforge.com/replacing_hard_disks_in_a_raid1_array
Categories : General rambling

Embrace boredom

I know, I'm a programmer. And as such, everyday or so of my life for the last 20 years I've written code. Most of it has been thrown away by now, but a good chunk is still alive and kicking. And what happens to this code now? Well, from time to time someone take a look at it and try to think "how am I going to make this code do what I want it to be doing". And if this person finds an answer quickly, the code I wrote was good.

As Dan McKinley puts it very well, the best way for this code to make things easy for other people is to be understandable at first glance. Smart and clever code doesn't meet this standard. Boring code does.

Boring code means code that meets long-established standards. It's boring because those standards have been known for a long time and you're smart! You can do better! Yes, you can, but no, you shouldn't. Because the next person modifying your code won't have a clue about your smart idea of the day.

In the same way, when choosing a library to answer a need you have, before jumping into the latest shiny bandwagon, ask yourself if the old and rock-stable lib everyone knows can do the job. If it can, there is probably no better choice. If and when it fails, you're more likely to have at least a few people onboard that know about that failure. When you push it to the limits, you also have experienced people that can better predict what will happen.

The only exception to this rule is areas in which you innovate, and there should be very few of them. You can't innovate on every front. Because innovation is draining your resources: it is longer to setup, harder to pickup for newcomers, harder to fine tune, more prone to bugs which are harder to debug. Just because it's new and no one has any experience with it. So choose those areas very carefully.

This is where the GTD (Get Things Done) should kick in. Use rock-solid and proven libs. Use rock-solid and proven patterns. You will be more likely to build a rock-solid platform that works without a glitch. And it will be faster and less expensive to build.

Categories : General rambling

User-Agent detection in Java

java-user-agent-detection is a small lib that gives back information when given a user-agent string. It is fast (under 0.1ms) and optimized. Where you need to go from there:

If you want to leave feedback, the comments space below is a good place.

Categories : Java

Target your CSS fonts on different platforms

Continuing on my journey to find the proper set of fonts for my website, I was a little surprised to see the way the different vendors handle native fonts in CSS. In this realm, there are three players that handle both the OS and the browser: Apple (Safari on Mac OS and iOS), Microsoft (IE on Windows) and Google (Chrome and their default browser on Android). Their handling of the native fonts is — of course — different. Let's see the different approaches.


Let's start out with the simplest one: Android. There is one font, Roboto. You can target it with the following values: sans-serif, sans-serif-light and sans-serif-condensed for the different variants. There are a few aliases such as Arial (alias to sans-serif) or Georgia (alias to serif) but if you want to target Android you should add the serif-liked syntax for better support. Then you can use font-weight and font-style to target different weight and the italic version. However, you cannot target the sans-serif-light or sans-serif-condensed with CSS attributes. You have to select them with the font-family selector.

So, to target a narrow font, you have to use sans-serif-condensed you cannot use the font-stretch:condensed property. Less options is not good. One font is not good. This is not very rich.


  • This holds true for both Chrome and the abomination installed by default on Android.
  • I haven't tested anything on Chrome OS. Common sense from a vendor perspective would say that it should be similar to Android. Common sense from a product placement perspective dictates that I find it doubtful that a desktop OS would ship with so little fonts installed. All bets are open.


IE allows also the font-stretch to be used, along with the name of the font — if at all available, which is rare. For example, you can target font-stretch:condensed;font-family:Arial or directly font-family:"Arial Narrow", both work. Unfortunately:
  • there are not even a handful of fonts with condensed versions
  • none are installed by default, they're installed with MS Office
  • no other variation than condensed can be targeted (save the Arial Black oddity which quite doesn't work as expected — see below)
  • most fonts are crap anyways

font-weight and font-style work as expected but for the fact that all fonts have only two weight available. Again, Arial Black is the exception, giving one more weight to the Arial family. But if you target it with font-weight:100 (trying to get a thinner version) well, you still end up with Arial Black.

So despite its interesting support for different types of font targeting, IE falls short because of the lack of unity and the lack of nice fonts. While testing, be also aware that not all windows users have MS Office installed, so if you do try to find a machine that doesn't to test your fonts on. MS Office comes along with a bunch of fonts and not everyone has them. This is unfortunate because those are nice fonts...


Safari doesn't support the font-stretch CSS property so far. To target a narrow/condensed font, you have to target it with its family name. The surprise on both iOS and MacOS is that you can target all variations of most fonts by their family name. Italic, light, black, bold, condensed, every variation has a family name you can target right in the font-family CSS attribute. Now, apart from the stretch, you can also target those with font-weight and font-style. For example, if you specify font-weight:600;font-family:Helvetica-Light, you'll get the regular Helvetica (the bold version of the thin font). With font-weight:700 you'll get the bold version. This is the best of both world, really. Whenever Apple starts supporting font-stretch, Safari will be the sweet spot for font selection. But as it is, it's already the best.

Moreover, Apple's (modern) fonts are declined in a variety of styles (like 5 different thickness; light, thin, regular, medium and bold) allowing better control, and they are pretty nice. So — somewhat unsurprisingly — most sites having paid attention to their fonts and relying on native font stacks will be rendered best on Apple devices.

Mixed platforms

For Firefox, well, things look very much like Chrome. It also depends on the OS. For example, trying to target Arial Narrow on Windows has to be done with font-stretch:condensed;font-family:Arial, but on Linux for example, you have to use font-family:"Arial Narrow". font-stretch:condensed; doesn't do anything on Linux.

How to test?

Testing, as always, is the right thing to do. I've set up a small page that list the fonts available on your browser. You can access this page from an iOS, Android, MacOS and Windows machine to see the fonts available. Then you can build your font-family stack with reasonable confidence. If you don't have all of those machines available, hop on to saucelabs.com to access some of those machines for free. Test the fonts on your browser

All in all, I have found that the worst platform to find fonts for is Windows. But in every case, I've found fonts that I liked for every system without much work. @font-face is not yet something I will consider for any of my websites.

Categories : CSS, iPhone and iOS, Android

Chrome and Arial Narrow

As you discovered with my previous entry, I was looking for nice condensed web fonts (read: native) for my headings. Well, I found plenty for everyone but for Chrome on Windows. Chrome issue is troublesome for me because this combination represent 30% of my traffic right now, by far outweighing other combinations of browser/OS.

How come I couldn't do it? I could do it with IE, Firefox on all platforms and Safari on all its platforms. Even Chrome on non-Windows platforms. It turns out that Chrome does not support the font-stretch: condensed CSS attribute. On other platforms, there are fonts that are condensed and can be targeted by using their name, but alas, Arial Narrow is not a font you can target with Chrome (nor with Firefox), even though IE can target it. Note that Franklin Gothic is in the same situation.

It's the first time Chrome is lagging in terms of features behind IE and Firefox (at least on something I use).

Let's hope Google addresses this problem as Arial Narrow is pretty much everywhere. And it's a pretty nice font, even though Helvetica people have probably left my website by now.

Edit on October 16: It is an issue with Chrome 37. So hopefully it'll get patched soon.

Edit on October 21: Chrome 38 is out. This is fixed.

Categories : CSS, Web Design, html

The quest for a condensed web font

I was looking to freshen up my blog's CSS. It's that time of the year. After looking here and there, I decided I wanted a condensed font for my headings. Sans Serif. This seemed like the right thing to do.

A condensed font is a font that is a little narrower than a regular font. For example:

As you can see, the letters are stretched and narrower than the regular version of the font.

I want a web safe and native font-stack because it is much lighter than a font-face (download-wise). You now see everywhere on the web that there is no point in targeting web safe fonts, because there are too few. So just use the CSS3 @font-face and be done with it. Well, I don't agree with this. I still care about my mobile users and imposing them a 100kB payload just to view the headers in the font I chose is not something I think is reasonable. Plus, I like the idea that my website looks a little different on every device. Hopefully it even matches the UI of the device since its using one of the device's font. I mean, using the Ubuntu font on an Ubuntu system is more likely to trigger a familiar feeling to the reader. This font is displayed in quite a few places in the OS such as the splash screen or the menus. And it seems fun.

Of course, I want my font-stack to be cross-browser. So I looked for default fonts on all my target systems. Fortunately, in my home there are plenty of different devices: iOS, Android, Windows, MacOSX and Ubuntu (Linux). This is my target right there.

The Quest

On Ubuntu, there is a nice font pre-installed called Ubuntu, and there is a condensed variation. In bold it looks fine. I also found a Liberation Sans which also has a condensed version (called Narrow). There's a better chance of having these fonts on non-Ubuntu Linux boxes. Since I couldn't find any resource on the web telling me which fonts are preinstalled on an Ubuntu system, I made it with the fonts installed on mine. Also, testing with saucelabs.com I found that their bare Linux install has Liberation Sans installed. I'll call it a day.

Then I went to iOS, where a very nice website lists all the fonts on all the versions of iOS: iosfonts.com (best viewed on an iOS device). I found two fonts that I liked in there: AvenirNext (since iOS6) and Futura (since iOS3). So there I am, iOS is now covered. The extra bonus is that for old devices I will fallback to Futura and all iOS devices can install iOS3, so I've pretty much covered 100% of iOS devices. Sweet.

Then I went to Android, where there is a new font called Roboto (as you can see there). This is unfortunately only true for recent Android devices (how recent I don't know), but it seems to me that there was no condensed font installed before that point. Too bad. There is also something peculiar about Android font families: You cannot target font-family: Roboto. This would be too simple I guess. You instead have to target a generic condensed font: font-family: sans-serif-condensed. Oh well, after all what I'm looking for is a sans serif condensed font and if/when other browsers on other platforms start picking up this peculiar syntax, well, it'll give me what I want. I guess. I hope.

I went quickly to my wife's MacBookAir, just long enough to make sure that one of my iOS fonts was installed on her Mac: Futura. They also make available a list of fonts for their OS: ht1642. The wikipedia page is much more helpful in that it has images of what the fonts look like. So I found Helvetica Neue which has a nice CondensedBold version. There I was. Done for MacOSX.

Then I went to Windows, my least favorite platform but let's face it: The one used by the vast majority of my users. I was firmly decided to get it done quickly. More than 90% of my users come from Windows 7 or 8, so I went to the Win7 fonts page. And there is the real disappointment: no condensed or narrow fonts. The narrowest font I could find is Trebuchet MS, the font I was already using for my headings. Disappointed. Very much so. So I thought that most users having Windows have Microsoft Office. I then stumbled on this page: Fonts supplied with Office 2010. There I got plenty of interesting fonts. Unfortunately, I tested this on my son's Windows 8 laptop with Office 2013, but most of the fonts listed weren't even installed. Too bad.

Then, wandering on the fringes of the interwebs, I stumbled on cssfontstack.com where I discovered a few things:

  • There is a font called Arial Narrow. It works on my ubuntu even though it's not even on the installed fonts list!! Let me try that on windows... Well, things are complicated. On IE, you can target it fine either by putting Arial Narrow or by targeting Arial and specifying font-stretch: condensed. Firefox only goes with the font-stretch: condensed and Arial. Chrome as well (doesn't work on Chrome 37 though). So I'll put it with both forms as a fallback.
  • Franklin Gothic Demi Cond seems to work on Windows and looks a lot nicer than Arial Narrow. Firefox and Chrome let me use the font-stretch: condensed modifier which does the job on Franklin Gothic. With IE, it works both ways. Unfortunately, I've found a few systems with Arial Narrow which don't have Franklin Gothic Demi Cond, so I cannot target it for Firefox/Chrome or else Firefox will just display Franklin Gothic, not the condensed version on those systems.
  • Both these fonts are installed with Office, none are on a bare Windows install. Furthermore, to target Arial Narrow for Firefox and Chrome, you need to target Arial with a font-stretch: condensed. Hence, whether Arial Narrow is installed or not, Arial will be targeted and your font stack stops right there, because this is such an ubiquitous font. Since most Windows users have MS Office, I will still try to target it. And the rest of them will see Arial.

The Results

Here are my results so far:

h1 { font-family: /*iOS*/"AvenirNextCondensed-Bold", "Futura-CondensedExtraBold", /*MacOSX*/HelveticaNeue-CondensedBold, /*Ubuntu*/ "Ubuntu Condensed", "Liberation Sans Narrow", /*Windows*/"Franklin Gothic Demi Cond", "Arial Narrow", /*Android*/sans-serif-condensed, /*Fallback*/Arial, "Trebuchet MS", "Lucida Grande", Tahoma, Verdana, sans-serif; font-stretch: condensed; }

Since most of the fonts I found do please me, I listed them in no particular order, except for

  1. My old fonts are at the last position as a last resort fallback.
  2. Android's declaration seems a little too generic for me, so I put is just before the fallback fonts in case some future version of browsers start interpreting it giving me unexpected results.

Final thoughts

Finally, I've got a few surprises in my quest.
  • iOS and MacOS are much more polished environments than the rest and finding nice fonts was just plain trivial. There are lots of documentation and examples, and devices follow them nicely.
  • I thought a modern OS like Android would be more loaded with fonts and would have a better documentation.
  • Ubuntu is an undocumented pile of heterogeneous stuff. Linux. No surprises there.
  • I did not expect Windows to fare particularly well in here, but I certainly didn't expect it to fare that bad! Not only is the documentation just plain fuzzy and wrong, but the font choice is the worst of all OSes. To put it bluntly, it doesn't have a narrow font, not even in Windows 8. And to think this is the major desktop OS by far... I'm not wondering why MacOS shares are increasing.

And, as expected, no need for a @font-face declaration and a 70k download just to get a font I like. Most Windows users have Office installed (I hope) so this isn't a really big issue. Plus, Arial isn't that bad as a fallback. My font stack works well enough in all platform I've tested so far. That's more than enough for me.

At last, check out flippingtypical.com. It will list all the fonts your browser knows about with a nice preview. Very helpful.

The Demo Area

Here are all the fonts selected. You will see of course only the ones you have installed on your system. This is more of a test area for me than for you, but I threw it there so that you can see it. Note: I built a little tool to help you detect all the fonts you can target. Test the fonts on your browser
  • No style: All examples looking like this below means they don't exist on your system.
    This is the test area
  • iOS6: Avenir Next Condensed Bold
    This is the test area
  • iOS3: Futura Condensed Extra Bold
    This is the test area
  • MacOSX: Helvetica Neue Condensed Bold
    This is the test area
  • Ubuntu: Ubuntu Condensed
    This is the test area
  • Ubuntu: Liberation Sans Narrow
    This is the test area
  • Windows: Franklin Gothic
    This is the test area
  • Android: Generic Sans Condensed
    This is the test area
  • Fallback: Arial Narrow
    This is the test area
  • Fallback: Trebuchet MS
    This is the test area
  • Fallback: Lucida Grande
    This is the test area
  • Fallback: Tahoma
    This is the test area
  • Fallback: Verdana
    This is the test area
  • Fallback: sans-serif (This is the last instruction - I want a sans serif font if nothing else is available)
    This is the test area
<< Previous | Home | Next >>