Recovery OS -- System Rescue CD v 2.8.0
Production OS - Debian Squeeze (6) 2.6.32 stock kernel, using mdadm raid
/dev/md0 in raid level RAID10 - members /dev/sdb1, /dev/sdc1, /dev/sdd1, /dev/sde1 all with partion id=fd
HDD /dev/sdb went bad, replaced it with another disk with same size partion (id=fd) using System Rescue CD v2.8.0
1. System Rescue CD recognized the md devices but it comes up as 'inactive'
Searched for possible solutions and I have tried several things including zeroing the super block and adding them back to the array.
Still unable to bring back /dev/md0 with all 4 partions in active mode.
I have included below, the entire transcript of the commands I have tried to recover /dev/md0
I have data on /dev/md0 that I need. I do have back ups of critical files (but not all).
I prefer solving the problem v/s recreating the /dev/md0 from scratch.
Any help in solving this problem would be highly appreciated.
TIA, -- Arun Khan
--------------- transcript of mdadm activity with System Rescue CD v2.8.0 ----------------
# mdadm -V mdadm - v3.1.4 - 31st August 2010
# cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : inactive sdd1[2] sde1[3] 312574512 blocks super 1.0
# mdadm -S /dev/md0 mdadm: stopped /dev/md0
# cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] unused devices: <none>
# mdadm -v -v -A /dev/md0 -R /dev/sd[bcde]1 mdadm: looking for devices for /dev/md0 mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 0. mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 1. mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 2. mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 3. mdadm: added /dev/sdb1 to /dev/md0 as 0 mdadm: added /dev/sdc1 to /dev/md0 as 1 mdadm: added /dev/sde1 to /dev/md0 as 3 mdadm: added /dev/sdd1 to /dev/md0 as 2 mdadm: failed to RUN_ARRAY /dev/md0: Input/output error mdadm: Not enough devices to start the array.
# cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : inactive sdd1[2] sde1[3] 312574512 blocks super 1.0
unused devices: <none>
from /var/log/messages Feb 12 06:13:59 sysresccd kernel: [ 7593.339015] md: md0 stopped. Feb 12 06:13:59 sysresccd kernel: [ 7593.374016] md: bind<sdb1> Feb 12 06:13:59 sysresccd kernel: [ 7593.374417] md: bind<sdc1> Feb 12 06:13:59 sysresccd kernel: [ 7593.374604] md: bind<sde1> Feb 12 06:13:59 sysresccd kernel: [ 7593.374869] md: bind<sdd1> Feb 12 06:13:59 sysresccd kernel: [ 7593.374899] md: kicking non-fresh sdc1 from array! Feb 12 06:13:59 sysresccd kernel: [ 7593.374903] md: unbind<sdc1> Feb 12 06:13:59 sysresccd kernel: [ 7593.379016] md: export_rdev(sdc1) Feb 12 06:13:59 sysresccd kernel: [ 7593.379041] md: kicking non-fresh sdb1 from array! Feb 12 06:13:59 sysresccd kernel: [ 7593.379044] md: unbind<sdb1> Feb 12 06:13:59 sysresccd kernel: [ 7593.386010] md: export_rdev(sdb1) Feb 12 06:13:59 sysresccd kernel: [ 7593.387382] md/raid10:md0: not enough operational mirrors. Feb 12 06:13:59 sysresccd kernel: [ 7593.387410] md: pers->run() failed ...
# mdadm -D /dev/md0 /dev/md0: Version : 1.0 Creation Time : Fri Apr 29 04:27:04 2011 Raid Level : raid10 Used Dev Size : 156287232 (149.05 GiB 160.04 GB) Raid Devices : 4 Total Devices : 2 Persistence : Superblock is persistent
Update Time : Mon Feb 11 13:43:52 2013 State : active, FAILED, Not Started Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0
Layout : near=2 Chunk Size : 32K
Name : brahmaputra:0 UUID : f2d4e898:2e026f85:244a7e9c:908e1af7 Events : 783527
Number Major Minor RaidDevice State 0 0 0 0 removed 1 0 0 1 removed 2 8 49 2 active sync /dev/sdd1 3 8 65 3 active sync /dev/sde1
# mdadm -E /dev/sdc1 /dev/sdc1: Magic : a92b4efc Version : 1.0 Feature Map : 0x1 Array UUID : f2d4e898:2e026f85:244a7e9c:908e1af7 Name : brahmaputra:0 Creation Time : Fri Apr 29 04:27:04 2011 Raid Level : raid10 Raid Devices : 4
Avail Dev Size : 312574512 (149.05 GiB 160.04 GB) Array Size : 625148928 (298.09 GiB 320.08 GB) Used Dev Size : 312574464 (149.05 GiB 160.04 GB) Super Offset : 312574640 sectors State : clean Device UUID : 2fbc103e:ca40e0c2:b8e4d64f:0fbc7b94
Internal Bitmap : -8 sectors from superblock Update Time : Mon Feb 11 13:43:52 2013 Checksum : 2e8e9fad - correct Events : 0
Layout : near=2 Chunk Size : 32K
Device Role : spare Array State : ..AA ('A' == active, '.' == missing)
# mdadm -E /dev/sdb1 /dev/sdb1: Magic : a92b4efc Version : 1.0 Feature Map : 0x1 Array UUID : f2d4e898:2e026f85:244a7e9c:908e1af7 Name : brahmaputra:0 Creation Time : Fri Apr 29 04:27:04 2011 Raid Level : raid10 Raid Devices : 4
Avail Dev Size : 312574512 (149.05 GiB 160.04 GB) Array Size : 625148928 (298.09 GiB 320.08 GB) Used Dev Size : 312574464 (149.05 GiB 160.04 GB) Super Offset : 312574640 sectors State : clean Device UUID : e27c187b:9004cb93:5bb05639:164822cd
Internal Bitmap : -8 sectors from superblock Update Time : Mon Feb 11 13:43:52 2013 Checksum : 5ea77bd0 - correct Events : 0
Layout : near=2 Chunk Size : 32K
Device Role : spare Array State : ..AA ('A' == active, '.' == missing)
# mdadm -E /dev/sde1 /dev/sde1: Magic : a92b4efc Version : 1.0 Feature Map : 0x1 Array UUID : f2d4e898:2e026f85:244a7e9c:908e1af7 Name : brahmaputra:0 Creation Time : Fri Apr 29 04:27:04 2011 Raid Level : raid10 Raid Devices : 4
Avail Dev Size : 312574512 (149.05 GiB 160.04 GB) Array Size : 625148928 (298.09 GiB 320.08 GB) Used Dev Size : 312574464 (149.05 GiB 160.04 GB) Super Offset : 312574640 sectors State : clean Device UUID : 78fc82bc:2eb18f07:56c98922:7639269e
Internal Bitmap : -8 sectors from superblock Update Time : Mon Feb 11 13:43:52 2013 Checksum : ce19a703 - correct Events : 783527
Layout : near=2 Chunk Size : 32K
Device Role : Active device 3 Array State : ..AA ('A' == active, '.' == missing)
# mdadm -E /dev/sdd1 /dev/sdd1: Magic : a92b4efc Version : 1.0 Feature Map : 0x1 Array UUID : f2d4e898:2e026f85:244a7e9c:908e1af7 Name : brahmaputra:0 Creation Time : Fri Apr 29 04:27:04 2011 Raid Level : raid10 Raid Devices : 4
Avail Dev Size : 312574512 (149.05 GiB 160.04 GB) Array Size : 625148928 (298.09 GiB 320.08 GB) Used Dev Size : 312574464 (149.05 GiB 160.04 GB) Super Offset : 312574640 sectors State : clean Device UUID : 1dce2f63:fef488cb:1a362c57:7ed908ac
Internal Bitmap : -8 sectors from superblock Update Time : Mon Feb 11 13:43:52 2013 Checksum : 7b44c944 - correct Events : 783527
Layout : near=2 Chunk Size : 32K
Device Role : Active device 2 Array State : ..AA ('A' == active, '.' == missing)
# mdadm --zero-superblock /dev/sdb1
# mdadm -E /dev/sdb1 mdadm: No md superblock detected on /dev/sdb1.
# mdadm /dev/md0 --add /dev/sdb1 mdadm: add new device failed for /dev/sdb1 as 4: Invalid argument
from /var/log/messages Feb 12 06:52:26 sysresccd kernel: [ 9900.298880] md0: ADD_NEW_DISK not supported