Ohmster’s LVM Disaster

Scenario:

I had a Fedora Core 3 system for a long time, long enough to be able to install Fedora Core 5 and that is what I wanted to do. But, I had a lot of work put into my old disto. 3 FQDNs on an apache server, vsftpd setup just the way I want it, all of my stuff for my family in the public_html directories, scripts, and system tweaks that I had learned for years. I for sure did not want to lose this stuff so I decided to do a clean install of Fedora Core 5 on a brand new 200Gb IDE hard drive and pull the existing 200Gb hard drive out for safekeeping. Cool, I install Fedora Core 5, all went well, but for the fact that FC5 won’t config my ISA Soundblaster AWE64 card.

I then install the old FC3 drive as /dev/hdb. Great. Time to mount that sucker. Not happening, it appears that the drive is an LVM drive, and to make it worse, it has the same name as the LVM drive that Fedora Core 5 has created for the system. What to do? I fiddle around, ask questions in Usenet and out of desperation, decide to experiment with webmin on it. I decide to rename the drive from VolGroup00 to VolGroup01. It did not appear to be successful but oh well. At least I could not tell if it was successful. I then, like an idiot, decide to create a new physical volume on the drive, not at all seeing the warning, ALL DATA WILL BE LOST. I commit to the change, see the warning, try to stop it. Oh oh…

I shut down the system, reconnect the drive as /dev/hda again and boot to the drive as I have always been able to do. Kernel panic, no / filesystem. Oh crap, the warning was not kidding. I cry to Usenet, alt.os.linux again, everyone tries to help but little by little, they all peter out, but for noi. He hangs in there with me. Giving me ideas, URLs (Thank you so much for the URLS noi!), and mostly, inspiration, and confidence. A week goes by and no luck. I finally stumble on the answer, thanks to noi’s URLs.

The Fix:

A good URL that got me started was this one:
http://www.linuxjournal.com/article/8874

It showed me how to grab a backup file from the toasted drive by doing this:
dd if=/dev/md2 bs=512 count=255 skip=1 of=/tmp/md2-raw-start

Modified for my system as such:
dd if=/dev/hdb bs=512 count=255 skip=1 of=/tmp/md2-raw-start

I then follow the instructions and try to create a backup file from it as described in the above URL and also this URL, (Big help):
http://blog.herlo.org/2006/09/15/lvm-from-failure-to-recovery/

Now my original backup file was not worth a darn because the 2nd URL showed me that I did indeed need the original logical volume IDs as well as the physical volume IDs. I recreated this backup by pulling up two term windows, one with a backup of /etc/lvm/backup/VolGroup00 in a vim editor, and the other one of my crap from the dd output file. I just copied and pasted all the relevant numbers, IDs, etc., into the existing backup file of VolGroup00’s file.

Now since both drives did contain the same name, VolGroup00, the existing backup was useless. I did end up with a backup of VolGroup01 but it was after the disaster (Remember, I did rename it with webmin.) and so it was useless.

So now I saved the very much edited backup of /etc/lvm/backup/VolGroup00 as /root/recoverybox/test/test-no-lv2. I named it that because what I was able to salvage from the dd output file was enough to recreate the physical volume and the first logical volume, the 2nd logical volume was lost in binary gibberish. This might be okay because the first logical volume was the big one and that is the one I wanted, with all my stuff in it. Here is what it looks like:
———————————————————————
# Generated by LVM2: Sun Oct 1 16:33:20 2006

contents = “Text Format Volume Group”
version = 1

description = “Created *after* executing ‘vgscan’”

creation_host = “ohmster.com” # Linux ohmster.com 2.6.17-1.2187_FC5 #1 Mon Sep 11 01:17:06 EDT 2006 i686
creation_time = 1159734800 # Sun Oct 1 16:33:20 2006

VolGroup01 {
id = “3agBFX-D3N0-Bp3c-I7se-JVLY-ZIIa-fqMcIF”
seqno = 2
status = ["RESIZEABLE", "READ", "WRITE"]
extent_size = 65536 # 32 Megabytes
max_lv = 0
max_pv = 0

physical_volumes {

pv0 {
id = “v40FVM-IWMU-T26b-I9eD-UprZ-WNqu-a9MTGq”
device = “/dev/hdb2″ # Hint only

status = ["ALLOCATABLE"]
pe_start = 384
pe_count = 6074 # 186.188 Gigabytes
}
}

logical_volumes {

LogVol00 {
id = “aaOmGX-y3gI-zucm-xEkQ-IxnV-w7Od-Xauy6E”
status = ["READ", "WRITE", "VISIBLE"]
segment_count = 1

segment1 {
start_extent = 0
extent_count = 6011 # 184.219 Gigabytes

type = “striped”
stripe_count = 1 # linear

stripes = [
"pv0", 0
]
}
}
}
}
———————————————————————
By the way, all this lvscan, pvscan, and all the mucking around with the drive removed what was still in the first sector of the disk and all I had now was binary gibberish. Thank God that I did not delete my original dd output file in the /tmp directory!

Okay, tried to restore it with the command:
vgcfgrestore -f /root/recoverybox/test/test-no-lv2 VolGroup01

No such luck, said it could not find the physical drive as named in the recovery file. Bummer. I would have to create the physical drive, and use the physical volume ID number of what the original drive used to contain, never mind that one already existed. This is what I did:

pvcreate -u 3agBFX-D3N0-Bp3c-I7se-JVLY-ZIIa-fqMcIF /dev/hdb2

That would not work because it would overwrite an existing physical group, I would have to force it with the -ff flag, here goes.

pvcreate -ff -u 3agBFX-D3N0-Bp3c-I7se-JVLY-ZIIa-fqMcIF /dev/hdb2

Hmmm, that seemed to go all too well, no output is good output. Does the Volume Group exist?

[root@ohmster recoverybox]# lvscan
ACTIVE ‘/dev/VolGroup01/LogVol00′ [187.84 GB] inherit
ACTIVE ‘/dev/VolGroup00/LogVol00′ [184.22 GB] inherit
ACTIVE ‘/dev/VolGroup00/LogVol01′ [1.94 GB] inherit
[root@ohmster recoverybox]#

Indeed it does! Will it mount? Maybe, but I am sure I will have to fsck the hell out of it, lose a lot of data in the process too, no doubt. I try to mount it.

mount /dev/VolGroup00/LogVol01 /mnt/old_sys

“/dev/VolGroup00/LogVol01 is not a block device” WTF?!

I use the mount command to see just how these kind of drives are mounted on my system…

[root@ohmster recoverybox]# mount
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)

Ohh, I see, let’s try again.

mount dev/mapper/VolGroup00-LogVol00 /mnt/old_sys

Huh, no output, the command worked! But what is left of all my stuff?

I go see.

[root@ohmster recoverybox]# cd /mnt/old_sys
[root@ohmster old_sys]# ls -la
total 304
drwxr-xr-x 23 root root 4096 Oct 1 13:11 .
drwxr-xr-x 3 root root 4096 Sep 30 20:57 ..
-rw——- 1 root root 154 Dec 10 2005 .bash_history
drwxr-xr-x 2 root root 4096 Jun 29 04:42 bin
drwxr-xr-x 2 root root 4096 May 7 2005 boot
drwxr-xr-x 4 root root 4096 May 7 2005 dev
-rw-r–r– 1 root root 644 Oct 1 13:05 env.reboot
drwxr-xr-x 104 root root 12288 Oct 1 13:11 etc
-rw-r–r– 1 root root 27599 May 7 2005 .fonts.cache-1
-rw-r–r– 1 root root 0 Oct 1 13:11 halt
drwxr-xr-x 8 root root 4096 Dec 8 2005 home
drwxr-xr-x 2 root root 4096 Aug 12 2004 initrd
drwxr-xr-x 10 root root 4096 Jun 29 04:14 lib
drwx—— 2 root root 16384 May 7 2005 lost+found
drwxr-xr-x 6 root root 4096 Jan 29 2006 media
drwxr-xr-x 2 root root 4096 Mar 23 2005 misc
drwxr-xr-x 2 root root 4096 Oct 8 2005 mnt
drwxr-xr-x 2 root root 4096 Aug 12 2004 opt
drwxr-xr-x 2 root root 4096 May 7 2005 proc
drwxr-x— 38 root root 4096 Sep 27 19:45 root
drwxr-xr-x 2 root root 12288 Jun 29 04:43 sbin
drwxr-xr-x 2 root root 4096 May 7 2005 selinux
drwxr-xr-x 2 root root 4096 Aug 12 2004 srv
drwxr-xr-x 2 root root 4096 May 7 2005 sys
drwxrwxrwt 12 root root 61440 Oct 1 13:07 tmp
drwxr-xr-x 18 root root 4096 Oct 19 2005 usr
drwxr-xr-x 25 root root 4096 Jun 23 19:12 var
[root@ohmster old_sys]#

OMFG, it worked!!! I go and browse my stuff, it is all there, ALL of it!

“Joy to the world, all the boys and girls. Joy to the fishes in the deep blue see. Joy to you and me!”

I quickly copy my stuff home where it belongs with the cp - a command. All is well again.

The Moral:

Don’t give up. If you don’t know what you are doing right now, leave it alone. Keep whatever data files you have found or created. Don’t discard them. Make backups! (Oh I had better do that right now!). Go back and try again as you learn more about this project.

Very Helpful URLS:
http://www.linuxjournal.com/article/8874
http://blog.herlo.org/2006/09/15/lvm-from-failure-to-recovery/
http://codeworks.gnomedia.com/archives/2005/general/lvm_recovery/

Thanks:
noi