RTFM

[Read This Fine Material] from Joshua Hoblitt

Pickled Chicken Boobs

2008-02-22 by jhoblitt | 0 comments

A couple of nights ago, as an experiment Jeyhan and I put a couple of chicken boobs in a vacuum marinater with nothing but the juice out of a pickle jar. We let it stand for about a hour and a half then fried them up in a skillet with olive oil and a lid over the pan. The meat had an interesting tangy/pickle flavor to it (sounds strange but it was good) and we used most of it in quesadillas. I recovered the left overs from the fridge tonight and it was surprisingly still moist. Pickle juice, who’d have thunk it?

Replacing a failed mdadm mirror disk

2008-02-20 by jhoblitt | 0 comments

The scenario is that we have a system with only two SATA disks setup as a RAID1/mirror set. Both disks are partitioned as follows:


Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1          13       97656   fd  Linux raid autodetect
Partition 1 does not end on cylinder boundary.
/dev/sda2              13        2005    16000000   82  Linux swap / Solaris
/dev/sda3            2005        5988    32000000   fd  Linux raid autodetect
/dev/sda4            5988       60802   440288927+  fd  Linux raid autodetect

Where sda[134] are mirrored to partitions sdb[134] (named /dev/md[123]), while sda2 & sdb2 are used directly as swap space (there is no reason to mirror swap space and using each partition separately doubles the size of usable swap). Also worth noting is that since the system is booting from one or the other of this disks (in the event of a disk failure), they both need to have identical MBRs. Now lets say that /dev/sda has failed (as actually happened to a system here) and that the bad disk has been pulled and replaced. We end up with a system that looks like this:


# cat /proc/partitions 
major minor  #blocks  name

   8    16  488386584 sdb
   8    17      97656 sdb1
   8    18   16000000 sdb2
   8    19   32000000 sdb3
   8    20  440288927 sdb4
   9     2   31999936 md2
   9     3  440288832 md3
   9     1      97536 md1
   8    32  488386584 sdc

What’s happened here is that Linux has recognized that even thou there is a device attached to same SATA channel as /dev/sda was on, that this device is different and should be given a different name. This is also a brand new disk that does not yet contain a partition table. In order to recover a mdadm managed mirror set we need to have a partition of equal or great size to add into the raid set. We could parition /dev/sdc by hand but then we’d still have to deal with running grub and setting up the MBR. Fortunately there is an easier way:


# dd if=/dev/sdb of=/dev/sdc bs=512 count=1

Which copies both the MBR and the partition table. Next we need to the kernel to attempt to re-read the parition table for this disk.


# partprobe /dev/sdc
# cat /proc/partitions 
major minor  #blocks  name

   8    16  488386584 sdb
   8    17      97656 sdb1
   8    18   16000000 sdb2
   8    19   32000000 sdb3
   8    20  440288927 sdb4
   9     2   31999936 md2
   9     3  440288832 md3
   9     1      97536 md1
   8    32  488386584 sdc
   8    33      97656 sdc1
   8    34   16000000 sdc2
   8    35   32000000 sdc3
   8    36  440288927 sdc4

At this point were finally read to start recovering the RAID individual RAID sets. This is what the status of the RAID sets currently is. Note the “(F)”s from where I had alrady tried to rebuild the RAID sets onto the faultly /dev/sda.


# cat /proc/mdstat     
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] 
md1 : active raid1 sda1[2](F) sdb1[1]
      97536 blocks [2/1] [_U]
      
md3 : active raid1 sda4[2](F) sdb4[1]
      440288832 blocks [2/1] [_U]
      
md2 : active raid1 sdb3[1]
      31999936 blocks [2/1] [_U]
      
unused devices:

From here is just a simple matter of adding back in the /dev/sdcX equivilants to /dev/sdaX.


mops11 ~ # mdadm --manage /dev/md1 --add /dev/sdc1
mdadm: added /dev/sdc1
mops11 ~ # mdadm --manage /dev/md3 --add /dev/sdc4
mdadm: added /dev/sdc4
mops11 ~ # mdadm --manage /dev/md2 --add /dev/sdc3
mdadm: added /dev/sdc3
mops11 ~ # cat /proc/mdstat 
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] 
md1 : active raid1 sdc1[0] sda1[2](F) sdb1[1]
      97536 blocks [2/2] [UU]
      
md3 : active raid1 sdc4[2] sda4[3](F) sdb4[1]
      440288832 blocks [2/1] [_U]
      [>....................]  recovery =  0.0% (378368/440288832) finish=4708.3min speed=1557K/sec
      
md2 : active raid1 sdc3[2](F) sdb3[1]
      31999936 blocks [2/1] [_U]
      
unused devices:

Drat! It looks like /dev/sdc has died too. This probably means that either this SATA channel is bad or there is a cabling problem. Looking at the system’s dmesg we see:


sd 0:0:0:0: [sdc] Synchronizing SCSI cache
sd 0:0:0:0: [sdc] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
sd 0:0:0:0: [sdc] Stopping disk
sd 0:0:0:0: [sdc] START_STOP FAILED
sd 0:0:0:0: [sdc] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
...
scsi 0:0:0:0: Direct-Access     ATA      WDC WD5000YS-01M 09.0 PQ: 0 ANSI: 5
sd 0:0:0:0: [sdd] 976773168 512-byte hardware sectors (500108 MB)
sd 0:0:0:0: [sdd] Write Protect is off
sd 0:0:0:0: [sdd] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 0:0:0:0: [sdd] 976773168 512-byte hardware sectors (500108 MB)
sd 0:0:0:0: [sdd] Write Protect is off
sd 0:0:0:0: [sdd] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sdd: sdd1 sdd2 sdd3 sdd4
sd 0:0:0:0: [sdd] Attached SCSI disk
sd 0:0:0:0: Attached scsi generic sg0 type 0
scsi 0:0:0:0: rejecting I/O to dead device

Had this not happened we would have only had to resetup the swap partition:


# mkwap /dev/sdc2

and reboot the system.

UPDATE: 17:36

In this case, it turned out to be dust in the SATA connectors (or at least, blowing the ends of the cables out seems to have fiex the issue). Here is what the system should look like with a properly rebuilding RAID set:


Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] 
md1 : active raid1 sdb1[0] sda1[1]
      97536 blocks [2/2] [UU]
      
md3 : active raid1 sda4[2] sdb4[1]
      440288832 blocks [2/1] [_U]
      [=====>...............]  recovery = 25.8% (113689216/440288832) finish=81.2min speed=67012K/sec
      
md2 : active raid1 sdb3[0] sda3[1]
      31999936 blocks [2/2] [UU]
      
unused devices:

Democratic Chaos ’08

2008-02-20 by jhoblitt | 0 comments

I went to my first caucus meeting ever this evening. It was the Democratic caucuses for Hawaii Congressional District: 1, District 22, Precinct: 02. Although I have mixed feelings about associating myself with a political party, I was forced to register with the Democratic party at the door. And that door took about 50 minutes in line just to reach. On the other side was total chaos. Nobody knew who was in charge or who was supposed to be doing what. Someone announced on the PA that in 2004 only 40 people showed up to this caucus. My guestimate is that there were on the order of 400 people in the room and yet more outside. I’m guessing that 100’s more left because of the long lines and disorganization. Speaking of disorganization, they ran out of ballots, then tried to apparently photocopy more… After a failed attempt to heard us into Clinton/Obama/other campus within our precints, they ended up resorting to cutting up legal pads and having us write our district/precinct on them and our candidate of choice. No name, no checking of id, no authentication what so ever. I was given a big pile of paper scraps too. I predict that pull box stuffing was rampant. Or perhaps *is* rampant as they are supposedly going to hold the caucus open until everyone has a chance to vote (presumably that means vote at least once, yarr). Geoff and I had good enough sense to bail before the local precinct seats and delegate elections started.

Blogger SFTP errors

2008-02-18 by jhoblitt | 0 comments

After being unable to update my blog with blogger for ~4 months, which errors like:

[sshd] Did not receive identification string from x.x.x.x

I finally sat down and decided to move to functional blogging software. Unfortunately, the software I decided on, Movable Type, doesn’t have a handy autoimport from Blogger feature nor does Blogger have an export format. To covert a blog from Blogger to Movable means setting up your own custom templates and turing on on Blogger’s “archive” feature. Well that doesn’t sound to bad but you have to actual be able to republish your blog for this to work!!!

One way or the other, I needed to figure out how to make Blogger work with SFTP again (and by again, I truely mean again; over the last year Blogger has started and stopped working with sftp several times). The Blogger forums and FAQ were no help but I finally found this blog post: Fixing Blogger SFTP errors. Which explains that it’s merely just a problem with sshd rejecting plaintext passwords (note to Google: WTF guys? Can’t you implement a rational version of ssh?). The fix is to set this option in sshd_config:

PasswordAuthentication yes

At some point in the near future I’m am still going to dump Blogger. There are people screaming all over Google’s support forums about this issue and there has been absolutely no response from the support staff. Apparently “Blogger” is unsupported software. There is a wonderful presentation on how to move a blog to Movable Type here: b2mt.

Who are the Wizards that design airports?

2008-01-08 by jhoblitt | 0 comments

Ever notice that most airports are lacking something that almost all shopping malls have? No clue what I’m talking about? Well it’s a map! I don’t mean just a small wall mounted map or two hidden away in some dark corner but big light up things the size of a 2001esk monolith placed in central locations so you can’t miss them.