HPE Storage Users Group https://www.3parug.com/ |
|
HP3par 7200 18 disks degraded how to replace https://www.3parug.com/viewtopic.php?f=18&t=2939 |
Page 1 of 2 |
Author: | coolirc [ Sat Jul 28, 2018 12:24 pm ] |
Post subject: | HP3par 7200 18 disks degraded how to replace |
Hello all i'm new to this community , we have an HP3par 7200 which we bought in 2014 now we have alarms for 18 degraded disks and i am searching for an advice i've found on the net how to replace procedure but they don't talk about data on the disks or something. so i'm asking how can i replace the 18 degraded disks , one by one or all a time , what commands should i perform before proceeding should i consider a down time or a better time for replacing the disks during weekend etc ? what will happen to the data on the disks , should i consider a backup or something ? will the actual data on the disks be lost? should i consider putting the esxi into maintenance mode and shutting down the vm's ? thanks for help. here's the issued command Code: HP3PAR_7200 cli% showpd -failed -degraded
--Size(MB)--- ----Ports---- Id CagePos Type RPM State Total Free A B Capacity(GB) 18 1:0:0 FC 15 degraded 278528 4096 0:0:1 1:0:1* 300 19 1:1:0 FC 15 degraded 278528 2048 0:0:1* 1:0:1 300 20 1:2:0 FC 15 degraded 278528 3072 0:0:1 1:0:1* 300 21 1:3:0 FC 15 degraded 278528 3072 0:0:1* 1:0:1 300 22 1:4:0 FC 15 degraded 278528 4096 0:0:1 1:0:1* 300 23 1:5:0 FC 15 degraded 278528 3072 0:0:1* 1:0:1 300 24 1:6:0 FC 15 degraded 278528 4096 0:0:1 1:0:1* 300 25 1:7:0 FC 15 degraded 278528 4096 0:0:1* 1:0:1 300 26 1:8:0 FC 15 degraded 278528 5120 0:0:1 1:0:1* 300 27 1:9:0 FC 15 degraded 278528 4096 0:0:1* 1:0:1 300 28 1:10:0 FC 15 degraded 278528 4096 0:0:1 1:0:1* 300 29 1:11:0 FC 15 degraded 278528 4096 0:0:1* 1:0:1 300 30 1:12:0 NL 7 degraded 923648 0 0:0:1 1:0:1* 1000 31 1:13:0 NL 7 degraded 923648 0 0:0:1* 1:0:1 1000 32 1:14:0 NL 7 degraded 923648 0 0:0:1 1:0:1* 1000 33 1:15:0 NL 7 degraded 923648 0 0:0:1* 1:0:1 1000 34 1:16:0 NL 7 degraded 923648 0 0:0:1 1:0:1* 1000 35 1:17:0 NL 7 degraded 923648 0 0:0:1* 1:0:1 1000 --------------------------------------------------------------------- 18 total 8884224 45056 HP3PAR_7200 cli% |
Author: | MammaGutt [ Sun Jul 29, 2018 12:50 am ] |
Post subject: | Re: HP3par 7200 18 disks degraded how to replace |
What does checkhealth -svc -detail say? Looks to me like all drives in cage1 has a problem so I'm guessing the problem is with the cage and not the 18 drives. |
Author: | coolirc [ Sun Jul 29, 2018 7:44 am ] | |||||
Post subject: | Re: HP3par 7200 18 disks degraded how to replace | |||||
MammaGutt wrote: What does checkhealth -svc -detail say? Looks to me like all drives in cage1 has a problem so I'm guessing the problem is with the cage and not the 18 drives. hmm i think you are right Code: HP3PAR_7200 cli% checkhealth -svc -detail Checking alert Checking ao Checking cabling Checking cage Checking dar Checking date Checking file Checking ld Checking host Checking license Checking network Checking node Checking pd Checking pdch Checking port Checking qos Checking rc Checking snmp Checking task Checking vlun Checking vv Checking sp Component -------------------Description-------------------- Qty Alert New alerts 4 Cabling Wrong I/O module or port 2 host Hosts not seen by multiple nodes 9 Host Host ports not configured for virtual port support 4 Network Too few working admin network connections 1 PD PDs that are degraded 18 QoS Unable to check QoS 1 Component --Identifier-- ----------------------------------------------------------------------------------------Description---------------------------------------------------------------------------------------- Alert sw_cp:1:FC_r5 CPG 1 (FC_r5) could not grow with its normal grow parameters.-The following parameters were used: createald -wait 0 -cpsd FC_r5 -ssz 6 -ha mag -t r5 -p -devtype NL -n tp-1-sd-2 -sz 8192 Alert sw_sysmgr Total FC raw space usage at 6449G (above 95% of total 6528G) Alert sw_sysmgr Total NL raw space usage at 21228G (above 95% of total 21648G) Alert sw_os An Update is Available Cabling cage1 Cable in (cage1, I/O 0, DP-1) should be in (cage1, I/O 1, DP-1) Cabling cage1 Cable in (cage1, I/O 1, DP-1) should be in (cage1, I/O 0, DP-1) host SRV_ESXi01_112 Host is not seen by multiple nodes, only seen from node 1 host SRV_ESX06_92 Host is not seen by multiple nodes, only seen from node 1 host SRV_ESX3_68 Host is not seen by multiple nodes, only seen from node 1 host SRV_ESX2_107 Host is not seen by multiple nodes, only seen from node 1 host SRV_ESX04_106 Host is not seen by multiple nodes, only seen from node 1 host SRV_ESXi11_67 Host is not seen by multiple nodes, only seen from node 1 host SRV_ESXi12_66 Host is not seen by multiple nodes, only seen from node 1 host SRV_ESXi09_70 Host is not seen by multiple nodes, only seen from node 1 host SRV_ESXi09_74 Host is not seen by multiple nodes, only seen from node 1 Host Port:1:1:1 Port WWN not found on FC Fabric attached to Port:0:1:1 Host Port:1:1:2 Port WWN not found on FC Fabric attached to Port:0:1:2 Host Port:0:1:1 Port WWN not found on FC Fabric attached to Port:1:1:1 Host Port:0:1:2 Port WWN not found on FC Fabric attached to Port:1:1:2 Network -- Node 1 has no admin network link detected PD disk:18 Degraded States: Invalid_connection PD disk:19 Degraded States: Invalid_connection PD disk:20 Degraded States: Invalid_connection PD disk:21 Degraded States: Invalid_connection PD disk:22 Degraded States: Invalid_connection PD disk:23 Degraded States: Invalid_connection PD disk:24 Degraded States: Invalid_connection PD disk:25 Degraded States: Invalid_connection PD disk:26 Degraded States: Invalid_connection PD disk:27 Degraded States: Invalid_connection PD disk:28 Degraded States: Invalid_connection PD disk:29 Degraded States: Invalid_connection PD disk:30 Degraded States: Invalid_connection PD disk:31 Degraded States: Invalid_connection PD disk:32 Degraded States: Invalid_connection PD disk:33 Degraded States: Invalid_connection PD disk:34 Degraded States: Invalid_connection PD disk:35 Degraded States: Invalid_connection QoS -- Unable to check QoS - This system is not licensed for System Reporter features HP3PAR_7200 cli% i think the cables connections are incorrect , Cabling cage1 Cable in (cage1, I/O 0, DP-1) should be in (cage1, I/O 1, DP-1) Cabling cage1 Cable in (cage1, I/O 1, DP-1) should be in (cage1, I/O 0, DP-1)
|
Author: | MammaGutt [ Sun Jul 29, 2018 10:39 am ] |
Post subject: | Re: HP3par 7200 18 disks degraded how to replace |
With no guarantee I think you can do the following: Move cable from cage1, I/O 1, DP-1 to cage1, I/O 1, DP-2 Move cable from cage1, I/O 0, DP-1 to cage1, I/O 1, DP-1 Move cable from cage1, I/O 1, DP-2 to cage1, I/O 1, DP-1 Between each step do showpd -p (I think this is showpd -path) to verify that all PDs have two paths before moving the next cable. Finish with admithw and a new checkhealth. |
Author: | coolirc [ Sun Jul 29, 2018 11:50 am ] |
Post subject: | Re: HP3par 7200 18 disks degraded how to replace |
MammaGutt wrote: With no guarantee I think you can do the following: Move cable from cage1, I/O 1, DP-1 to cage1, I/O 1, DP-2 Move cable from cage1, I/O 0, DP-1 to cage1, I/O 1, DP-1 Move cable from cage1, I/O 1, DP-2 to cage1, I/O 1, DP-1 Between each step do showpd -p (I think this is showpd -path) to verify that all PDs have two paths before moving the next cable. Finish with admithw and a new checkhealth. Hello thanks for your answer but i still did not figure out what is what . where is cage 1 and where is I/O 0 and where is I/O 1 . DP1 and DP2 is already mentionned on the cage but still . thanks |
Author: | MammaGutt [ Sun Jul 29, 2018 1:44 pm ] |
Post subject: | Re: HP3par 7200 18 disks degraded how to replace |
coolirc wrote: MammaGutt wrote: With no guarantee I think you can do the following: Move cable from cage1, I/O 1, DP-1 to cage1, I/O 1, DP-2 Move cable from cage1, I/O 0, DP-1 to cage1, I/O 1, DP-1 Move cable from cage1, I/O 1, DP-2 to cage1, I/O 1, DP-1 Between each step do showpd -p (I think this is showpd -path) to verify that all PDs have two paths before moving the next cable. Finish with admithw and a new checkhealth. Hello thanks for your answer but i still did not figure out what is what . where is cage 1 and where is I/O 0 and where is I/O 1 . DP1 and DP2 is already mentionned on the cage but still . thanks Cage number should be visable thru a LED in the front. Node cage is always cage0. To find I/O 0 and I/O 1, look all the way to the left or right on the back on the disk cages (between PSUs and I/O modules). You will see a red tab with 0 and green tab with 1. That is I/O 0 and I/O 1. Same with the nodes. When I look at your pictures now, you have labeled your cables correctly (red on both ends for one cable and green on both cables for the other) but you have connected the green cable to the red node and the red cable to the green node..... So the best would be to change the ports on the node end, but I've never done that so I can't say anything as to what types or errors you might run into.... If you have a spare 3PAR backend cable you could use that so you get the "right color coded cable" in the right node and cage .... But I would just suggest to do what I've suggested above and re-label the cables with spare labels. |
Author: | coolirc [ Sun Jul 29, 2018 2:45 pm ] |
Post subject: | Re: HP3par 7200 18 disks degraded how to replace |
hello i re-pluged my cables according to this figure now the number of degraded disks decreased to 6 would you like me to proceed with your figure or stay as i am ? Code: HP3PAR_7200 cli% showpd -path ---------Paths--------- Id CagePos Type -State-- A B Order 0 0:0:0 FC normal 1:0:1 0:0:1 1/0 1 0:1:0 FC normal 1:0:1 0:0:1 0/1 2 0:2:0 FC normal 1:0:1 0:0:1 1/0 3 0:3:0 FC normal 1:0:1 0:0:1 0/1 4 0:4:0 FC normal 1:0:1 0:0:1 1/0 5 0:5:0 FC normal 1:0:1 0:0:1 0/1 6 0:6:0 FC normal 1:0:1 0:0:1 1/0 7 0:7:0 FC normal 1:0:1 0:0:1 0/1 8 0:8:0 FC normal 1:0:1 0:0:1 1/0 9 0:9:0 FC normal 1:0:1 0:0:1 0/1 10 0:10:0 FC normal 1:0:1 0:0:1 1/0 11 0:11:0 FC normal 1:0:1 0:0:1 0/1 12 0:12:0 NL normal 1:0:1 0:0:1 1/0 13 0:13:0 NL normal 1:0:1 0:0:1 0/1 14 0:14:0 NL normal 1:0:1 0:0:1 1/0 15 0:15:0 NL normal 1:0:1 0:0:1 0/1 16 0:16:0 NL normal 1:0:1 0:0:1 1/0 17 0:17:0 NL normal 1:0:1 0:0:1 0/1 18 1:0:0 FC normal 1:0:2 0:0:2 1/0 19 1:1:0 FC normal 1:0:2 0:0:2 0/1 20 1:2:0 FC normal 1:0:2 0:0:2 1/0 21 1:3:0 FC normal 1:0:2 0:0:2 0/1 22 1:4:0 FC normal 1:0:2 0:0:2 1/0 23 1:5:0 FC normal 1:0:2 0:0:2 0/1 24 1:6:0 FC normal 1:0:2 0:0:2 1/0 25 1:7:0 FC normal 1:0:2 0:0:2 0/1 26 1:8:0 FC normal 1:0:2 0:0:2 1/0 27 1:9:0 FC normal 1:0:2 0:0:2 0/1 28 1:10:0 FC normal 1:0:2 0:0:2 1/0 29 1:11:0 FC normal 1:0:2 0:0:2 0/1 30 1:12:0 NL normal 1:0:2 0:0:2 1/0 31 1:13:0 NL normal 1:0:2 0:0:2 0/1 32 1:14:0 NL normal 1:0:2 0:0:2 1/0 33 1:15:0 NL normal 1:0:2 0:0:2 0/1 34 1:16:0 NL normal 1:0:2 0:0:2 1/0 35 1:17:0 NL normal 1:0:2 0:0:2 0/1 36 0:18:0 NL normal 1:0:1 0:0:1 1/0 37 0:19:0 NL normal 1:0:1 0:0:1 0/1 38 0:20:0 NL normal 1:0:1 0:0:1 1/0 39 0:21:0 NL normal 1:0:1 0:0:1 0/1 40 0:22:0 NL normal 1:0:1 0:0:1 1/0 41 0:23:0 NL normal 1:0:1 0:0:1 0/1 42 1:18:0 NL degraded 1:0:2\0:0:1 0:0:2\1:0:1 0/1 43 1:19:0 NL degraded 1:0:2\0:0:1 0:0:2\1:0:1 1/0 44 1:20:0 NL degraded 1:0:2\0:0:1 0:0:2\1:0:1 0/1 45 1:21:0 NL degraded 1:0:2\0:0:1 0:0:2\1:0:1 1/0 46 1:22:0 NL degraded 1:0:2\0:0:1 0:0:2\1:0:1 0/1 47 1:23:0 NL degraded 1:0:2\0:0:1 0:0:2\1:0:1 1/0 ------------------------------------------------------ 48 total Code: HP3PAR_7200 cli% admithw Checking for drive table upgrade packages Checking nodes... Checking volumes... Checking system LDs... Checking ports... Checking state of disks... The following disks are NOT in an acceptable state: Id CagePos Type -State-- --Detailed_State-- 42 1:18:0 NL degraded Invalid_connection 43 1:19:0 NL degraded Invalid_connection 44 1:20:0 NL degraded Invalid_connection 45 1:21:0 NL degraded Invalid_connection 46 1:22:0 NL degraded Invalid_connection 47 1:23:0 NL degraded Invalid_connection ------------------------------------------- 6 total Enter c to continue despite this issue or q to quit and fix the issue manually: c Checking cabling... Checking cage firmware... Checking if this is an upgrade that added new types of drives... Checking for disks to admit... 0 disks admitted Checking admin volume... Admin volume exists. Checking if logging LDs need to be created... No new logging LDs need to be created Checking if preserved data LDs need to be created... No new preserved data LDs need to be created Checking if system scheduled tasks need to be created... Checking if the rights assigned to extended roles need to be updated... No need to update extended roles rights. Rebalancing and adding FC spares... FC spare chunklets rebalanced; number of FC spare chunklets increased by 0 for a total of 544. Rebalancing and adding NL spares... NL spare chunklets rebalanced; number of NL spare chunklets increased by 0 for a total of 1260. Rebalancing and adding SSD spares... No SSD PDs present System Reporter data volume exists. Checking system health... Checking alert Checking cabling Checking cage Checking dar Checking date Checking host Checking ld Checking license Checking network Checking node Checking pd Checking port Checking rc Checking snmp Checking task Checking vlun Checking vv Component -------------------Description-------------------- Qty Alert New alerts 4 host Hosts not seen by multiple nodes 9 Host Host ports not configured for virtual port support 4 LD LDs with reduced availability 2 Network Too few working admin network connections 1 PD PDs that are degraded 6 admithw has completed. Code: HP3PAR_7200 cli% checkhealth
Checking alert Checking cabling Checking cage Checking dar Checking date Checking host Checking ld Checking license Checking network Checking node Checking pd Checking port Checking rc Checking snmp Checking task Checking vlun Checking vv Component -------------------Description-------------------- Qty Alert New alerts 4 host Hosts not seen by multiple nodes 9 Host Host ports not configured for virtual port support 4 LD LDs with reduced availability 2 Network Too few working admin network connections 1 PD PDs that are degraded 6 HP3PAR_7200 cli% |
Author: | MammaGutt [ Mon Jul 30, 2018 2:14 am ] |
Post subject: | Re: HP3par 7200 18 disks degraded how to replace |
Oh lord.... Those 8 PDs wasn't in your first print screens. I assumed all PDs in cage1 was degraded..... I'm going to take a wild guess on this one. At one point in time everything was good and great, and cage1 had 16 PDs. At some later point in time the 3PAR was probably moved and recabled, resulting in the incorrect cabling and the 16 PDs complaining about the cabling being incorrect/changed. Nothing was done and at a later time, 8 additional PDs was added to cage1 when the cabling was incorrect, and those 8 PDs assumed that everything was good (considering every upgrade procedure should include a step where you check that you have a health system prior to doing any changes). So now have 16 PDs expecting the cabling to be the correct one and complaining about it being wrong, and 8 PDs assuming everything is good. So then you recable the 3PAR to make it correct so the 16 PDs go "all OK" while the last 8 is now complaining that the correct cabling is not the cabling it is expecting. So .... the only way I know would fix this, is to empty one PD at a time, completely remove it with the correct set of commands and re-admit it. The bad news is that you have 0 chunklets free on your NL drives so you can't empty one out..... It could be that this could also be fixed by using servicemag and trick the system into thinking one drive has failed and replacing (and re-admitting) it with the same drive, but I wouldn't even know where to start on that ... so others need to shed some light on that if it is possible..... Might be a good time to get in touch with HPE Support. |
Author: | coolirc [ Mon Jul 30, 2018 4:13 am ] |
Post subject: | Re: HP3par 7200 18 disks degraded how to replace |
Hello Thanks for your reply , i think the cause of the 0 free chunklets would be that i created a raw volumes and exported to vmware vcenter so i think i should i can move the recently created NL volume and then move the vms and data from that volume and replace the disks one by one and see. |
Author: | MammaGutt [ Mon Jul 30, 2018 6:07 am ] |
Post subject: | Re: HP3par 7200 18 disks degraded how to replace |
If you can free up space, than you should try that. You don't have to replace them as they are not broken, but you need to remove them and re-add them... setpd ldalloc off <DiskID> movepdtospare -f -vacate -nowait <DiskID> showpdch -mov (too see status/what is left, if it doesn't complete, you can just just tunesys cpg <NL CPG>) removespare PDID:a dismisspd <DiskID> Remove PD Reinstall PD admithw But as mentioned there might be a smoother way to do this with servicemag so others might give some advise here as well. |
Page 1 of 2 | All times are UTC - 5 hours |
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group http://www.phpbb.com/ |