[8200 3.3.1] SetSize with 8 SSD

Chris212 · **Joined:** Thu Jun 01, 2017 1:55 am **Posts:** 38

Hi everyone,

We had an issue with our new 3par 8200 (3.3.1 P02).
An SSD has failed during the night and after a few hours (10), a LUN crashed because it could not allocate space from the CPG...

We asked HPE why we had this kind of issue in a RAID5 configuration where we should be running without issue until the disk is replaced and here is their answer :

Issue: There are only 8 drives and CPG configured with set size 8. One disk failure caused VV allocation failures.
Replacing the disk solved the VV allocation failure.

Cause: The CPG has a set size of 8, so 8 PDs are required. If the customer only has 8 PDs, then losing 1 PD for any reason will prevent CPG grows.This is a minimal configuration and will have these types of issues.

Recommendation: Use set size 7 instead of 8 to have an additional disk to avoid such issues or use different setsize or RAID type

BUT, here is our CPG which is configured as follow :
Device type SSD 100K
Raid type RAID 5
Availability
Requested Magazine (Lower)
Current Magazine (Lower)
Set size 7 data, 1 parity
Preferred chunklets Fast
Disk filter -t r5 -ha mag -ssz 8 -ss 64 -p -devtype SSD -rpm 100

And in the HPE best practice documentation they also recommend a 7+1 configuration with SSD.

Then what, do you agree that we should go with a 6+1 configuration with 8 SSD ? That sounds quite weird to me...

Thank you for your answers and for helping us on this.

aberus · **Posted:** Mon Aug 07, 2017 6:09 am

On page 15 of the HPE 3PAR StoreServ Storage best practices guide it states it states:

"Solid-state drive CPGs
Best practice: Solid-state drive (SSD) CPGs should be of the RAID 6 type with a “set size” of 3+1 by default. This will bring superior performance/capacity ratio and 7+1 on systems that have enough SSDs to support 7+1. If maximum performance is required, use RAID 1."

In your configuration I would recommend either a RAID5 3+1 or a RAID1 because you have so few disks.

I will point out that it appears that HPE misstated when they said RAID6 3+1 since that is the minimum for RAID5, RAID6 would be 4+2.

You can find the BP guide here: https://www.hpe.com/h20195/v2/GetPDF.aspx/4AA4-4524ENW.pdf

Chris212 · **Joined:** Thu Jun 01, 2017 1:55 am **Posts:** 38

We will not go for a raid 6 / raid 1 configuration (30K $ the 7.6TB SSD...)
We have 2 8200 nodes with peer persistant.
If we loose two SSD at the same time in the same array, we still have the other node where it will switchover.

But here is not the question. In the document they advise a Raid6 3+1 OR a Raid5 7+1 for better perf/capacity which is the one we have chosen.

Why would we have such an issue with our configuration ?

With 8 ssd's with a set size of 7 data and 1 parity we normally can loose a disk.

Is there something we are missing here ?

aberus · **Posted:** Mon Aug 07, 2017 12:10 pm

The issue is that you only have 8 drives total, I would recommend creating a new CPG using RAID 5 3+1 and then tune your VV's to that new CPG., this can be done online.

I was in a somewhat similar situation recently but a different use case. I am using mine for AFC and AO and I wanted to maintain cage level redundancy, at the time the 1.92TB SSD's were ungodly expensive so I opted to go with 24 400GB SSD's.

I only mention this because sometimes it can be cheaper and provide a more salable design to not fall into the capacity/density trap that the sales folks like to push and instead plan for resiliency and growth. So in your case you "probably" could have purchased two additional enclosures and either 1.92TB or 3.84TB drives for around the same price, or slightly more, which would have provided you with redundancy and growth.

Only you know your situation but if HPE sold this solution to you to be deployed as you have then you may want to talk with your vendor and local HPE sales guy (referencing the case) and look into swapping out the drives for lower capacity disks. I have had success taking this approach in the past with FilePersona. Don't get me started on that subject...lol

Chris212 · **Joined:** Thu Jun 01, 2017 1:55 am **Posts:** 38

Thank you for your advice about 3+1 but still i do not understand why it does NOT work with 7+1 with 8 drives...

Is there any technical reason why a CPG RAID5 7+1 with 8 disks start to run out of space when ONE disk fails ?

Sorry but a raid 5 system do provide protection against a one drive failure... unless HPE reinvent the Raid 5 :?:

Noss · **Joined:** Tue Oct 21, 2014 3:26 am **Posts:** 55

Maybe its to do with that, while your in a degraded state, it will continue to serve data from spare chunklets but when it comes to try and grow the CPG (create new LDs) using the template 7+1, it can't meet that requirement because you are one drive short at that time.

aberus · **Posted:** Tue Aug 08, 2017 8:53 am

I believe you are correct Noss and is also what what HPE is referring to, although you are meeting you RAID requirements you may not be meeting them for the CPG so resetting this to 3+1 should resolve that for you.

I don't use peer persistence here but I do use remote copy and have tuned VV's into different CPG's while online so I would "expect" you should be able to do the same. I would recommend creating a 1GB test VV and run through the process to be sure.

markinnz · **Joined:** Tue Mar 29, 2016 9:32 pm **Posts:** 136

Well for the moment ignoring the god awful configuration you have been sold :-)

What HPE tried to describe to you is correct, they should maybe have been more verbose.

You have a CPG with a policy to make sure that logical disks making up your volumes span 8 physical disks.
How will follow that policy with one disk missing ? :-)

(next bit is simplified to save fingers typing ...)

You know how when a volume is created 2 logical disks get created (one per node and each probably 16GB in size) from chunklets, then your volume basically gets striped across those volumes and starts using capacity, when the LD capacity is used up the system tries to grow them and follows the rules of the CPG to make sure the growth spans 8 physical disks .. which if a disk is failed (leaving you with only 7) it can no longer do so, it can't grow, then of course the volume itself that prompted the LD growth has but none made available to it and fails.

Verdict ... change your CPG to 6+1 and tune your volumes to the new policy if you want things to still work after a single disk failure.

Better verdict ... use smaller capacity disks but more of them.
- those big SSDs look tempting but are only any good if you have lots of them for your system to work with.
- honestly your array config is dreadful and should never have been sold to you like that it's just plain wrong (OK rant over :-)

Mark

ailean · **Joined:** Wed Nov 09, 2011 12:01 pm **Posts:** 392

This is also a 'fault' with the way 3PAR configures small systems by default. We have a couple of 8400 with 6 SDDs per node shelf, the default OS created R5 and R6 CPGs are 5+1 and 4+2. :shock:

So the out of the box setup for this system would have the user creating all there VVs in CPGs which would hit this allocation issue should a single disk fail. :oops:

Admittedly these were created automatically during a 3.2.2 eMU4 upgrade and the originally OS version only created a R1 CPG as I think 8 SSDs per node shelf was the minimum at that time for R5/6 so all my VVs are R1 atm but I was intending to move them to R5 for some more space.

Luckily after spotting this thread I went back and tweaked the CPGs for smaller set sizes, this comes from 3PAR being designed around 100s of disks originally but they could do with tweaking the defaults at least for smaller systems and adding some warnings when creating CPGs with these limits.

Chris212 · **Joined:** Thu Jun 01, 2017 1:55 am **Posts:** 38

markinnz wrote:

Well for the moment ignoring the god awful configuration you have been sold :-)

What HPE tried to describe to you is correct, they should maybe have been more verbose.

You have a CPG with a policy to make sure that logical disks making up your volumes span 8 physical disks.
How will follow that policy with one disk missing ? :-)

(next bit is simplified to save fingers typing ...)

You know how when a volume is created 2 logical disks get created (one per node and each probably 16GB in size) from chunklets, then your volume basically gets striped across those volumes and starts using capacity, when the LD capacity is used up the system tries to grow them and follows the rules of the CPG to make sure the growth spans 8 physical disks .. which if a disk is failed (leaving you with only 7) it can no longer do so, it can't grow, then of course the volume itself that prompted the LD growth has but none made available to it and fails.

Verdict ... change your CPG to 6+1 and tune your volumes to the new policy if you want things to still work after a single disk failure.

Better verdict ... use smaller capacity disks but more of them.
- those big SSDs look tempting but are only any good if you have lots of them for your system to work with.
- honestly your array config is dreadful and should never have been sold to you like that it's just plain wrong (OK rant over

Mark

Do you advise to go with a 6+1 CPG or 3+1 ? What is the better option here ?

If only it was only that... :evil:

HPE sold us two 8200 nodes and told us that dedup+compression would do a great job on our array...
Well... it does (total compaction 4.8) but now we have CPU running at 75%... (a case is open for that) and service time at 2.2 ms.... (they sold us the array telling us it would be under 1ms...)

We are now very close to demand at least they change us the array for 8450...

HPE Storage Users Group

[8200 3.3.1] SetSize with 8 SSD

Who is online