HPE Storage Users Group
https://www.3parug.com/

Missing space 7200
https://www.3parug.com/viewtopic.php?f=18&t=953		 Page 1 of 2
Author:  M1kkel [ Thu Oct 16, 2014 8:45 am ]
Post subject:  Missing space 7200
Hi there.

First of all, I am new to 3par.

I have a vv called 3par_EXCH where i have placed all my exchange servers. 2 DAG servers, and 2 CASHubtransport servers.

the vv total size is 1525 GB. In my Management console it shows that i have used 85% of the space, 1383,00 GB.
However, if i take all the disks on my servers, i can see the following:

Total allocated disk space: 1.421 GB
On my windows disks, i have totally consumed: 1.161 GB, so it seems like 222 GB have dissapered into a black hole.
I have enabled THIN provisioning on the vv ofcourse.

The question is, what i can do now.
Running sdelete is one option, but since this is exchange servers, i would expect the exchange servers to always overwrite space, because there are so many writes all the time, but maybe i am not right.

I have read about a command called compactcpg, but i don't think that that command is meant for my situation?

Suggestions?

Thanks and best regards from Denmark.
Author:  Cleanur [ Thu Oct 16, 2014 1:45 pm ]
Post subject:  Re: Missing space 7200
You have NTFS underneath Exchange so it decides where the data actually resides in the file system and it doesn't overwrite old blocks unless it's getting low on space. There are a couple numbers you need to look at, one is "reserved" and the other "used" easiest way to look at this is at the CLI

showvv -s "vvname" and it will show you what 3PAR believes is used vs what it has reserved for a given volume. Once you have output you'll have a better idea.

You may as well try a compact CPG anyway it'll let you know if there's space that can be reclaimed from the CPG.

Latest version of the thin prov doc here, you could use the fsutil for the reclaim as it's safer than sdelete page 27/28.
http://h20195.www2.hp.com/V2/GetDocumen ... c=us&lc=en

I'd also add the "used user size" to the IMC, should be the default for 4.6 if not use reset to defaults or take a look at the screenshot.

Attachments:
Space.png
Space.png [ 44.24 KiB | Viewed 22284 times ]
used user.png
used user.png [ 45.17 KiB | Viewed 22284 times ]
Author:  M1kkel [ Fri Oct 17, 2014 4:03 pm ]
Post subject:  Re: Missing space 7200
Hi there.

Thanks for your reply.

So i ran a dry run of compact cpg, and found out there was nothing to be done on my exchange CPG, however on my exchange CPG for AO, there where some which could be freed.

Do you recommend to run compactcpg once each week or month to free up space?

next step will be to use the fsutil to write a balloon file on the empty space.

In the 3par console it shows 1398 GB consumed, in the IMC the reserved user size is 1398, and the used user size is the same. That's great.

So i assume the only way there is to go, is to use fsutil. I have 50 or more vm's on this system - it feels kind of lame to schedule fsutil on all my servers. What do other people do?

compactcpg output

Initial trim would reclaim 0 MB SA space, 3072 MB SD space, and 0 MB USR space.
3072 MB from LD tp-2-sd-0.1
Would move 0 MB SA space, 13312 MB SD space, and 0 MB USR space.
Final trim would reclaim 0 MB SA space, 36864 MB SD space, and 0 MB USR space.
15360 MB from LD tp-2-sd-0.1
21504 MB from LD tp-2-sd-0.0

Attachments:
3parIMC.PNG
3parIMC.PNG [ 2.89 KiB | Viewed 22255 times ]
3parusedspace.PNG
3parusedspace.PNG [ 6.33 KiB | Viewed 22255 times ]
3parCPGspace.PNG
3parCPGspace.PNG [ 2.62 KiB | Viewed 22255 times ]
Author:  Cleanur [ Mon Oct 20, 2014 11:50 am ]
Post subject:  Re: Missing space 7200
Yes run a scheduled compact CPG weekly, preferably out of hours, AO has an option also to compact the CPG's after each AO move completes.

Since your reserved and used are pretty close it looks like 3PAR is seeing your deleted data as valid, in which case you need to do a zero reclaim with the balloon file or similar, this will attempt to overwrite the deleted data with zero's and allow 3PAR to de-allocate these blocks.

Which tool you use will depend on the Hypervisor / guest OS and level of granularity required, most of the newer stuff supports unmap so it's getting easier. But if your environment doesn't support this then a company call Raxco sell a product. perfect disk which supports centralized deployment, scheduling etc. Not used it myself but looks a decent option for larger environments.

Note the 16KB blocks are initially given back to the volume, If they are contiguous 128MB of blocks they will be given back to the owning CPG, from where a scheduled Compact CPG wil return them to the system. If however the returned blocks are heavily fragmented the system will need to first defrag them before being able to return them to the CPG which can take a while as it's a background process, in this case you'll see reserved space drop slowly over time to ~match user space.
Author:  M1kkel [ Wed Jan 14, 2015 9:49 am ]
Post subject:  Re: Missing space 7200
Hi there.

Should i also run the vmware commands to reclaim free space on the datastores. Say that we have removed 200 gb of servers from inside the vmware vsphere console?

cd /vmfs/volumes/<volume-name>
# vmkfstools -c <size of space to reclaim> -d eagerzeroedthick zerofile
# rm zerofile-flat

Or does vmware automatically unmap the free space? We are using 5.1 ..
Author:  Namlehse [ Wed Jan 14, 2015 10:02 am ]
Post subject:  Re: Missing space 7200
5.1 Doesn't do a very good job at cleaning up after itself. 5.5 is better, but it's still lacking. You do want to run the VMWare commands, though how often depends on you and your space issues. We went a year without any issues or cleanup, however recently we opened up hosting services to 17 or so business under us, giving way to.. explosive growth..

You really need tools outside of 3PAR to help with that (Depending on your size). We use SolarWinds VMware tools to keep track of VM's that have a habbit of growing and shrinking. I keep those on a VVol that has a chron job to free up space once a week, the majority of application servers however don't grow much, let alone shrink.

If you use snapshots (We use them extensively for backing up), you can use the "freespace" command from the CLI as well to reclaim copy space as well as admin space that's been freed up. I was able to reclaim around 10% of my space using freespace + compactcpg after using the VMWare commands you posted above.

I will also add that HP is very little help in this matter, If you have a good sales engineer to go to that's your best bet, they'll generally know more about what you're doing and how to fix it.
Author:  JohnMH [ Wed Jan 14, 2015 11:51 am ]
Post subject:  Re: Missing space 7200
New thin technologies whitepaper.

http://www8.hp.com/h20195/v2/GetDocumen ... A3-8987ENW

Note the guest file system and then the hypervisor file system are both layers of abstraction from each other and also the 3PAR array so it's not a simple one to one mapping each is hiding free space from the other. To get the max back you should reclaim at the guest, then the datastore, this will return the space to the VV, if you allow the internal defrag to complete this will return to the CPG as a whole and then you can optionally compact the CPG to return the space to the system. Don't get obsessed with trying to reclaim the last few GB's of data in the CPG. It'll only get overwritten in the near future anyway, so it's probably best to just leave it where it is otherwise the system will have to do more work growing to accommodate new writes.
Author:  M1kkel [ Thu Jan 15, 2015 7:31 am ]
Post subject:  Re: Missing space 7200
JohnMH wrote:
New thin technologies whitepaper.

http://www8.hp.com/h20195/v2/GetDocumen ... A3-8987ENW

Note the guest file system and then the hypervisor file system are both layers of abstraction from each other and also the 3PAR array so it's not a simple one to one mapping each is hiding free space from the other. To get the max back you should reclaim at the guest, then the datastore, this will return the space to the VV, if you allow the internal defrag to complete this will return to the CPG as a whole and then you can optionally compact the CPG to return the space to the system. Don't get obsessed with trying to reclaim the last few GB's of data in the CPG. It'll only get overwritten in the near future anyway, so it's probably best to just leave it where it is otherwise the system will have to do more work growing to accommodate new writes.


Awesome, thanks to both of you. You say "f you allow the internal defrag to complete this will return to the CPG as a whole"
How do i allow internal defrag?
Author:  M1kkel [ Thu Jan 15, 2015 7:44 am ]
Post subject:  Re: Missing space 7200
Ahh, so defragt is what compact cpg does? I have scheduled them to run each night.

my backup solution is VEEAM, which makes a snapshot of each vm each day, and removes it afterwards. That, i assume, will also take up space after deletion?
Author:  M1kkel [ Thu Jan 15, 2015 9:28 am ]
Post subject:  Re: Missing space 7200
Namlehse wrote:
5.1 Doesn't do a very good job at cleaning up after itself. 5.5 is better, but it's still lacking. You do want to run the VMWare commands, though how often depends on you and your space issues. We went a year without any issues or cleanup, however recently we opened up hosting services to 17 or so business under us, giving way to.. explosive growth..

You really need tools outside of 3PAR to help with that (Depending on your size). We use SolarWinds VMware tools to keep track of VM's that have a habbit of growing and shrinking. I keep those on a VVol that has a chron job to free up space once a week, the majority of application servers however don't grow much, let alone shrink.

If you use snapshots (We use them extensively for backing up), you can use the "freespace" command from the CLI as well to reclaim copy space as well as admin space that's been freed up. I was able to reclaim around 10% of my space using freespace + compactcpg after using the VMWare commands you posted above.

I will also add that HP is very little help in this matter, If you have a good sales engineer to go to that's your best bet, they'll generally know more about what you're doing and how to fix it.


Maybe you can show me your script to run the command - first the script must calculate free space, and afterwards run the command - correct?
Page 1 of 2 All times are UTC - 5 hours
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/