HPE Storage Users Group

A Storage Administrator Community




Post new topic Reply to topic  [ 7 posts ] 
Author Message
 Post subject: 3PAR 8200 - Reclaim Used Space Step by Step
PostPosted: Wed Apr 27, 2016 5:35 am 

Joined: Wed Apr 27, 2016 5:16 am
Posts: 9
Hi Folks,

I struggle currently with a 3PAR that seems to be running out of space.
A HP guy in the past told me I always should take the first look into the physical disk capacity and if there are no space left, the Black Card is taken and the systems will stop any IO to this disks.

The Environment:
2 3PAR 8200 (Peer Persistence config)
Inform OS 3.2.2.290 P03,P04
3 Tiers of Disks / one CPG per Tier
all Licenses
vSphere 5.5 (3 Hosts with FC access to the SAN)

The system was running fine the lasts months.
Some cust changes prevent the systemn from send Notifications.
Yesterday I take a look at the system.

Now I take a look and the VVs are used at 55% to 65% from the vSphere (5.5) view. But at 3PAR MMC I was able to see a usage of 95% on some VVs.

The physical disk capacity of the FC and NL class disks was at 99% at the FC Tier and 93% at the NL Tier.

Ok, I thougth I zero out the unused space with esxcli unmap command for nearly the half of the Volumes.
I performed compactcpg for the FC and NL CPG.

Result: Reserverd User size at the VV stays at 95%.
CPG Space at 99%.

The physical disks still at 99-100%, but the system and VMs are up and running?!
How is this possible?!

From my understanding, the CPG Space should be more now and
much more important the physical space should be increased.

Where I'm wrong?!
Do I forgot anything?!

Thanks for any suggestions!


Top
 Profile  
Reply with quote  
 Post subject: Re: 3PAR 8200 - Reclaim Used Space Step by Step
PostPosted: Wed Apr 27, 2016 6:49 am 

Joined: Wed May 07, 2014 1:51 am
Posts: 267
After "unmap", you nee to wait some time until the garbage collection changed the free space reserved for the single VVs to freespace reserved for the CPG. Only then compactcpg can return this space to the system wide freespace. Garbage Collection can take a long time, especially on NL drives, and you can not force/trigger it as 3PAR-admin.

As long as there is freespace reserved for a vv, it will continue to work until this reserved space is used up (changed to used space). Then the vv needs more space from the cpg. Only when this is used up as well (used space = reserved space on cpg level), the cpg needs to grow from the systemwide freespace. And THEN you run into problems if there is no space left.

We don't use AO here, so I don't know if there are any special things to take into account with AO.

_________________
When all else fails, read the instructions.


Top
 Profile  
Reply with quote  
 Post subject: Re: 3PAR 8200 - Reclaim Used Space Step by Step
PostPosted: Wed Apr 27, 2016 7:17 am 

Joined: Wed Apr 27, 2016 5:16 am
Posts: 9
Thanks apol for the short and clear clarification!
You saved my day and so I can work on it without the pressure of a storage outage.

So it seems to be I'm on the right way and only have to
wait some time after unmap to restart compactcpg job.

I will recreate the Peer Persistence Volumes (copies) at the second system, it seems to be the best solution.
Here I was running in a further issue with unexporting a copy. If I unexport the copy, the primary Volume was disconnected also from the vSphere Cluster and the VMs stop working.
So I have to schedule a downtime to troubleshoot this issue, to prevent further LUN outages...

Has anybody integrated an automated unmap solution for Thin Provisioned Volumes at vSphere 5.5?
I'm sure but the question is, how can I unmap unused space at the copies? The Host byself only "sees" the primary Volume. Maybe it's required to perfrom a scheduled switchover...? Ok, maybe it's a different topic...

The compactcpg task can be scheduled with 3PAR task planner, it's not a big deal.


Top
 Profile  
Reply with quote  
 Post subject: Re: 3PAR 8200 - Reclaim Used Space Step by Step
PostPosted: Thu Apr 28, 2016 1:00 am 

Joined: Wed May 07, 2014 1:51 am
Posts: 267
Glad I could help, but honestly: Your day is saved when you're no longer in danger of running out of space on some production vvs...

One caveat with vv-reserved freespace is: you always have it on a vv that doesn't need it. You can have tbs of freespace reserved for some vvs and still get really deep in trouble when that one very important vv needs more space, and needs to get back to the cpg that's run out of space as well... That's why compactcpg and garbage collection make a lot of sense: The further down the chain your freespace is, the less flexible it can be used!

_________________
When all else fails, read the instructions.


Top
 Profile  
Reply with quote  
 Post subject: Re: 3PAR 8200 - Reclaim Used Space Step by Step
PostPosted: Thu Apr 28, 2016 4:17 pm 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1570
Location: Europe
I would recommend reading the concepts guide. This will tell you how space is allocated from raw space on PD to usable space in VV.

Quick summary. Whenever a TPVV/TDVV wants to grow it will take available space from the VV. When the VV is "full" (ie, User Used = User Reserved), it will allocate new space from the CPG. When the CPG is "full" it will allocate new space from RAW space (PDs). When the RAW space for that tier is full, you basically have a full array. So a growing VV in a growing CPG will most likely be 99% "full".

3PAR will "borrow" space from a lower level to prevent an outage due to a full array if a tier is completely full (ie, if you write to FC layer and there is no more available RAW space there, it will use NL space to keep the system running). A lower level is of course slower, but I guess most people rather want to have a slow write, rather than no write. Be advised that clean-up after something like this might be time consuming depending on the amount of space written to a lower level, the number of VVs/CPGs affected and how full your array is when you've cleaned up/installed more hardware.

Reclaiming available space from VV is a background task. That space will be made available to the CPG. Reclaiming available space from the CPG is done by using compact cpg. If you only have one CPG, there is no harm in having a low amount of RAW space available as long as you have a lot of space available in the CPG, however it might be confusing having to look multiple places to get the "combined" available free space and if you have multiple CPGs on the same tier, the "combined" availble free space might differ from VV to VV as some space is most likely tied up in a CPG.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: 3PAR 8200 - Reclaim Used Space Step by Step
PostPosted: Mon Oct 15, 2018 5:41 am 

Joined: Sat Nov 15, 2014 7:30 am
Posts: 5
MammaGutt wrote:
I would recommend reading the concepts guide. This will tell you how space is allocated from raw space on PD to usable space in VV.

Quick summary. Whenever a TPVV/TDVV wants to grow it will take available space from the VV. When the VV is "full" (ie, User Used = User Reserved), it will allocate new space from the CPG. When the CPG is "full" it will allocate new space from RAW space (PDs). When the RAW space for that tier is full, you basically have a full array. So a growing VV in a growing CPG will most likely be 99% "full".

3PAR will "borrow" space from a lower level to prevent an outage due to a full array if a tier is completely full (ie, if you write to FC layer and there is no more available RAW space there, it will use NL space to keep the system running). A lower level is of course slower, but I guess most people rather want to have a slow write, rather than no write. Be advised that clean-up after something like this might be time consuming depending on the amount of space written to a lower level, the number of VVs/CPGs affected and how full your array is when you've cleaned up/installed more hardware.

Reclaiming available space from VV is a background task. That space will be made available to the CPG. Reclaiming available space from the CPG is done by using compact cpg. If you only have one CPG, there is no harm in having a low amount of RAW space available as long as you have a lot of space available in the CPG, however it might be confusing having to look multiple places to get the "combined" available free space and if you have multiple CPGs on the same tier, the "combined" availble free space might differ from VV to VV as some space is most likely tied up in a CPG.


Hi All

What happens if the VV is already on the lowest tier NL?

What is meant to be the process in this situation which tier does it borrow from?

or does everything come to a grinding halt?

""Cheers


Top
 Profile  
Reply with quote  
 Post subject: Re: 3PAR 8200 - Reclaim Used Space Step by Step
PostPosted: Mon Oct 15, 2018 8:29 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1570
Location: Europe
It will only borrow from lower tier. For NL there is no lower tier to borrow from. Hence full = full = no new writes.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 7 posts ] 


Who is online

Users browsing this forum: No registered users and 36 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group | DVGFX2 by: Matt