HPE Storage Users Group

A Storage Administrator Community




Post new topic Reply to topic  [ 18 posts ]  Go to page Previous  1, 2
Author Message
 Post subject: Re: High IOPS, high bandwith poor performance
PostPosted: Mon Nov 13, 2017 8:56 am 

Joined: Fri Feb 24, 2017 10:44 am
Posts: 10
Finally I find out 3 virtual machines which were making IOPS because of an application problem.

I earn an average of 1500 iops for one exported volume ( a csv for hyperV)

This is the evolution of iops before and after I solved the virtual machine problem
Image

BTW, to help me finding the problem I used the local installed 3parmanagement console, which permits making reports on a specific VV, when the web-base management console doesnt offers this possiblity

result :

on exported VV side : the 3par was running with an average of 5500 iops a day , now : 3000 iops
on physical disks side : before : average of 5600 uiops and now : 5500iops

I wonder why the disk average iops doest really change
5400iops is the max theorical iops my 3par can handle (42 FC disks 1 cpg. 130iops x 42 = 5460iops )

Anyway, situation is now better regarding the 1500-2000 iops I just earned (on exported vv side). Thanks for your help !

Anyone could explain why a FC disk (max 130iops) could be displayed with 250iops in the view statpd ?


Top
 Profile  
Reply with quote  
 Post subject: Re: High IOPS, high bandwith poor performance
PostPosted: Mon Nov 13, 2017 9:06 am 
User avatar

Joined: Tue Oct 07, 2014 8:34 am
Posts: 214
Location: Germany
In the Web GUI it is a bit hidden, but even better. :D

Select the "Exported Volumes Performance"
Left Click on the Graph.
This Open a new Window "Specific Time Data - Exported Volumes Performance"
Select: Top 10 + Total IO/s + LUN ID & Hostname + Daliy + Apply

This Show you the LUN & Server with most IOPS.

To see a Graph per Volume use "Exported Volumes Compare by Performance"


Top
 Profile  
Reply with quote  
 Post subject: Re: High IOPS, high bandwith poor performance
PostPosted: Mon Nov 13, 2017 10:05 am 

Joined: Fri Feb 24, 2017 10:44 am
Posts: 10
hidden, and interesting !
I tried, and for a specfic time, I can see quicly the host / VV with more IO. I will re use it for sure.
the old management console does not have this feature

But, I 'm still not able to make a report showing a graphic of the iops evolution for a specific host or a specific VV in a specific time window. Or maybe this option is hidden also :)


Top
 Profile  
Reply with quote  
 Post subject: Re: High IOPS, high bandwith poor performance
PostPosted: Thu Nov 30, 2017 5:20 am 

Joined: Fri Feb 24, 2017 10:44 am
Posts: 10
me again :)

since I've solved my iops problem on the front end (5000iops => 2500iops), I see the exported VVs servicetime double up. I don't know why !
Less activity, Less bandwith, less iops, .. but worse service time.
I can't understand why .. Any idea ?

Image


Top
 Profile  
Reply with quote  
 Post subject: Re: High IOPS, high bandwith poor performance
PostPosted: Thu Nov 30, 2017 5:30 am 

Joined: Mon Sep 21, 2015 2:11 pm
Posts: 1570
Location: Europe
Probably the reduced load was good (ie low service time), while you now have one or more hosts with issues affecting the service time.

I would look at (I think this will sort of current service time)
statvlun -hostsum -ni -rw -sortcol 8
to see if there are some hosts that has a lot higher service time than others. If that is the case I would start looking at those hosts... Do they have an old 4Gbit HBA, very large IOs or CPU going thru the roof ... etc.

To be honest, >5msec latency on read isn't that bad for FC drives. You will usually see around 10msec latency for FC drives for random reads so this is probably random reads and 50% or the IO is served by cache and doesn't need to be fetched by the spinning media.

_________________
The views and opinions expressed are my own and do not necessarily reflect those of my current or previous employers.


Top
 Profile  
Reply with quote  
 Post subject: Re: High IOPS, high bandwith poor performance
PostPosted: Tue Dec 05, 2017 4:52 am 

Joined: Fri Feb 24, 2017 10:44 am
Posts: 10
finally I didnt sucess to find out which host(s) could be the responsible for this
Actually the affected service time is located on 3 hosts, which are hyperV clustered servers.. then , around 100vm, which moves between hosts. Very difficult to solve. Anyway, I will keep in mind that my actual frontend service time is not so bad :D

For information I also find another problem, and 3par support brought me the answer

I upgraded my storeserv 7200c to v3.2.2. Since this time, the physical disks service time has double up

we can see this here : (3.2.2 installed on 24-oct), backend service time was average 11ms and after patching, st has an average of 18 ms

Image

according to what the support says : It is normal, the 3.2.2 firmware consumes more 3Par resouces due to new features added to this firmware. This is strange, but normal


Top
 Profile  
Reply with quote  
 Post subject: Re: High IOPS, high bandwith poor performance
PostPosted: Tue Dec 05, 2017 6:18 am 

Joined: Wed Nov 09, 2011 12:01 pm
Posts: 392
korny wrote:
finally I didnt sucess to find out which host(s) could be the responsible for this
Actually the affected service time is located on 3 hosts, which are hyperV clustered servers.. then , around 100vm, which moves between hosts. Very difficult to solve. Anyway, I will keep in mind that my actual frontend service time is not so bad :D

For information I also find another problem, and 3par support brought me the answer

I upgraded my storeserv 7200c to v3.2.2. Since this time, the physical disks service time has double up

we can see this here : (3.2.2 installed on 24-oct), backend service time was average 11ms and after patching, st has an average of 18 ms

Image

according to what the support says : It is normal, the 3.2.2 firmware consumes more 3Par resouces due to new features added to this firmware. This is strange, but normal


Yeah I spotted that too, I'm sure in a previous upgrade it went the other way. :roll:


Top
 Profile  
Reply with quote  
 Post subject: Re: High IOPS, high bandwith poor performance
PostPosted: Tue Dec 05, 2017 7:49 am 

Joined: Fri Feb 24, 2017 10:44 am
Posts: 10
Well, that's good to hear this :)
On my side, the update was 3.2.1 => 3.2.2.612
same one your side ?


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 18 posts ]  Go to page Previous  1, 2


Who is online

Users browsing this forum: No registered users and 41 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group | DVGFX2 by: Matt