HPE Storage Users Group

A Storage Administrator Community




Post new topic Reply to topic  [ 18 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: High IOPS, high bandwith poor performance
PostPosted: Tue Nov 07, 2017 9:36 am 

Joined: Fri Feb 24, 2017 10:44 am
Posts: 10
Hello all
I'm currently trying to understand a performance problem I encountered this morning on my 7200C

the 7200C has 42FC disks , and should be able to run around 5500IOPS
This morning, service time was not good at all, and I had 10.000iops , max 12.000

This increase of IOPS was caused by a high total bandwith : around 200MBPS

Image

Then, I'm trying to find which VV was responsible for a high bandwith and I tought the report "Exported Volumes Compare by Performance" could help :

Image[/url]

unfortunately , the sum of the bandwith of each exported volume is around 100MB, when I expect 200MB
Am I doing something wrong ?

My goal is to find which host, or which exported volume can be the responsible of the high total BW
with CLI, statvlun -hostsum could help me to find the BW for each host. But now the problem is over, and I suppose we cannot check in the past with CLI


Top
 Profile  
Reply with quote  
 Post subject: Re: High IOPS, high bandwith poor performance
PostPosted: Tue Nov 07, 2017 10:42 am 

Joined: Fri Jun 27, 2014 2:01 am
Posts: 390
You could try "help sr" and see the whole bunch of System Reporter command available to you through SSMC and obviously through SSH/CLI.

srstatport -port_type host -btsecs -3h -etsecs -2h
srstatvlun -groupby VV_NAME -btsecs -3h -etsecs -2h

These commands could give you some details on VV performance between 3 and 2 hours ago.


Top
 Profile  
Reply with quote  
 Post subject: Re: High IOPS, high bandwith poor performance
PostPosted: Tue Nov 07, 2017 11:26 am 

Joined: Fri Feb 24, 2017 10:44 am
Posts: 10
nice !
I'm going to use these srxxx commands to find out the responsible :)


Top
 Profile  
Reply with quote  
 Post subject: Re: High IOPS, high bandwith poor performance
PostPosted: Wed Nov 08, 2017 1:36 am 

Joined: Fri Jun 27, 2014 2:01 am
Posts: 390
Enjoy the hunt :D


Top
 Profile  
Reply with quote  
 Post subject: Re: High IOPS, high bandwith poor performance
PostPosted: Wed Nov 08, 2017 4:42 am 

Joined: Wed May 07, 2014 1:51 am
Posts: 267
To your first question "unfortunately , the sum of the bandwith of each exported volume is around 100MB, when I expect 200MB. Am I doing something wrong ?"

Do you have more than 10 VVs? I think the "compare by"-report templates per default just comapre the top 10 vvs and ignore the other ones. This could explain the missing bandwith if you did not change the report settings.

_________________
When all else fails, read the instructions.


Top
 Profile  
Reply with quote  
 Post subject: Re: High IOPS, high bandwith poor performance
PostPosted: Wed Nov 08, 2017 5:17 am 

Joined: Fri Feb 24, 2017 10:44 am
Posts: 10
Result of the hunt :
with your commands I successed to find out wich host / vv was consuming bandwith, but it resulst that the activity is the same than the days before .. I think my performance problem comes from my 3Par which is overloaded

1 CPG / 42 FC . Theory 5400iops
Nowadays the 3par is running around 5400iops (yesterday, the total was around 10.000iops during some minutes, and all the disks were 0% idle)

I think I just need more performance
Any advice for this point ?

my 3par needs to assume more iops , but does not need more capacity ..
SSD could be a good point to get more IOPS. Do you think I can mix SSD and FC disks in a same CPG ?

I also tried the adaptive flash cache simulator . and hits on the AFC is only around 7% : if I understand well, this solution will not really improve the situation


Last edited by korny on Wed Nov 08, 2017 5:58 am, edited 1 time in total.

Top
 Profile  
Reply with quote  
 Post subject: Re: High IOPS, high bandwith poor performance
PostPosted: Wed Nov 08, 2017 5:27 am 

Joined: Fri Feb 24, 2017 10:44 am
Posts: 10
apol wrote:
To your first question "unfortunately , the sum of the bandwith of each exported volume is around 100MB, when I expect 200MB. Am I doing something wrong ?"

Do you have more than 10 VVs? I think the "compare by"-report templates per default just comapre the top 10 vvs and ignore the other ones. This could explain the missing bandwith if you did not change the report settings.


I just realize that is the point, you're right ! I have around 30VV
in the view "Exported Volumes Compare by Performance.." only 10vv are displayed

But, in the view "exported volume performance" the bandwith diplays the total of all the VV.
and if I compare this total to the "pd performance bandwitch view", the bandwiths are not the same
ie, : at the same time

BW total exported VV : 150MBPS
BW PD : 236 MBPS

So I have still the same question :)
I read that this difference could come from the RAID configuration ..


Top
 Profile  
Reply with quote  
 Post subject: Re: High IOPS, high bandwith poor performance
PostPosted: Wed Nov 08, 2017 5:32 am 

Joined: Wed May 07, 2014 1:51 am
Posts: 267
"I read that this difference could come from the RAID configuration .."

Yes, I think so as well. And it could be that cache prefetching, work done by vaai-commands and so on are involved as well. On the vv-level, that's the reason why there are statvv (contains array-internal stuff) and statvlun (contains stuff from hosts).

_________________
When all else fails, read the instructions.


Top
 Profile  
Reply with quote  
 Post subject: Re: High IOPS, high bandwith poor performance
PostPosted: Wed Nov 08, 2017 6:48 am 

Joined: Wed Nov 09, 2011 12:01 pm
Posts: 392
I suspect your extra load is write based, this always hits the disks a lot harder, you may have seen similar levels before but were more read based so easily absorbed by cache and lower disk load.

I think AFC is only read based so why you have a low hit rate on the simulation. Try to nail down the write intensive VVs and if they are re-writing the same blocks a lot or writing lots of new blocks.

If same blocks then adding SSD and Adaptive Optimization might help, if lots of new blocks then you'll probably have to move the VVs to SSD (needing more space).

Hopefully it'll be a small number of aggressive VVs and once they are dealt with things will be happy. Of course you could always poke the server/application owners are ask what they are doing, it might turn out (as often does here) that they didn't know it was doing something and magically the IO falls when they kill some rogue process. :roll:


Top
 Profile  
Reply with quote  
 Post subject: Re: High IOPS, high bandwith poor performance
PostPosted: Mon Nov 13, 2017 7:50 am 
User avatar

Joined: Tue Oct 07, 2014 8:34 am
Posts: 214
Location: Germany
Look at: statpd

On right you see if Disks are Overloaded.
Idle > 20% and Qlen < 5 is OK

If not add Disks or SSD and enable Tiering.

FC Disks deliver about 150 IOPS.


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 18 posts ]  Go to page 1, 2  Next


Who is online

Users browsing this forum: Bing [Bot] and 135 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group | DVGFX2 by: Matt