HPE Storage Users Group
https://www.3parug.com/

High IOPS, high bandwith poor performance
https://www.3parug.com/viewtopic.php?f=18&t=2707
Page 2 of 2

Author:  korny [ Mon Nov 13, 2017 8:56 am ]
Post subject:  Re: High IOPS, high bandwith poor performance

Finally I find out 3 virtual machines which were making IOPS because of an application problem.

I earn an average of 1500 iops for one exported volume ( a csv for hyperV)

This is the evolution of iops before and after I solved the virtual machine problem
Image

BTW, to help me finding the problem I used the local installed 3parmanagement console, which permits making reports on a specific VV, when the web-base management console doesnt offers this possiblity

result :

on exported VV side : the 3par was running with an average of 5500 iops a day , now : 3000 iops
on physical disks side : before : average of 5600 uiops and now : 5500iops

I wonder why the disk average iops doest really change
5400iops is the max theorical iops my 3par can handle (42 FC disks 1 cpg. 130iops x 42 = 5460iops )

Anyway, situation is now better regarding the 1500-2000 iops I just earned (on exported vv side). Thanks for your help !

Anyone could explain why a FC disk (max 130iops) could be displayed with 250iops in the view statpd ?

Author:  cali [ Mon Nov 13, 2017 9:06 am ]
Post subject:  Re: High IOPS, high bandwith poor performance

In the Web GUI it is a bit hidden, but even better. :D

Select the "Exported Volumes Performance"
Left Click on the Graph.
This Open a new Window "Specific Time Data - Exported Volumes Performance"
Select: Top 10 + Total IO/s + LUN ID & Hostname + Daliy + Apply

This Show you the LUN & Server with most IOPS.

To see a Graph per Volume use "Exported Volumes Compare by Performance"

Author:  korny [ Mon Nov 13, 2017 10:05 am ]
Post subject:  Re: High IOPS, high bandwith poor performance

hidden, and interesting !
I tried, and for a specfic time, I can see quicly the host / VV with more IO. I will re use it for sure.
the old management console does not have this feature

But, I 'm still not able to make a report showing a graphic of the iops evolution for a specific host or a specific VV in a specific time window. Or maybe this option is hidden also :)

Author:  korny [ Thu Nov 30, 2017 5:20 am ]
Post subject:  Re: High IOPS, high bandwith poor performance

me again :)

since I've solved my iops problem on the front end (5000iops => 2500iops), I see the exported VVs servicetime double up. I don't know why !
Less activity, Less bandwith, less iops, .. but worse service time.
I can't understand why .. Any idea ?

Image

Author:  MammaGutt [ Thu Nov 30, 2017 5:30 am ]
Post subject:  Re: High IOPS, high bandwith poor performance

Probably the reduced load was good (ie low service time), while you now have one or more hosts with issues affecting the service time.

I would look at (I think this will sort of current service time)
statvlun -hostsum -ni -rw -sortcol 8
to see if there are some hosts that has a lot higher service time than others. If that is the case I would start looking at those hosts... Do they have an old 4Gbit HBA, very large IOs or CPU going thru the roof ... etc.

To be honest, >5msec latency on read isn't that bad for FC drives. You will usually see around 10msec latency for FC drives for random reads so this is probably random reads and 50% or the IO is served by cache and doesn't need to be fetched by the spinning media.

Author:  korny [ Tue Dec 05, 2017 4:52 am ]
Post subject:  Re: High IOPS, high bandwith poor performance

finally I didnt sucess to find out which host(s) could be the responsible for this
Actually the affected service time is located on 3 hosts, which are hyperV clustered servers.. then , around 100vm, which moves between hosts. Very difficult to solve. Anyway, I will keep in mind that my actual frontend service time is not so bad :D

For information I also find another problem, and 3par support brought me the answer

I upgraded my storeserv 7200c to v3.2.2. Since this time, the physical disks service time has double up

we can see this here : (3.2.2 installed on 24-oct), backend service time was average 11ms and after patching, st has an average of 18 ms

Image

according to what the support says : It is normal, the 3.2.2 firmware consumes more 3Par resouces due to new features added to this firmware. This is strange, but normal

Author:  ailean [ Tue Dec 05, 2017 6:18 am ]
Post subject:  Re: High IOPS, high bandwith poor performance

korny wrote:
finally I didnt sucess to find out which host(s) could be the responsible for this
Actually the affected service time is located on 3 hosts, which are hyperV clustered servers.. then , around 100vm, which moves between hosts. Very difficult to solve. Anyway, I will keep in mind that my actual frontend service time is not so bad :D

For information I also find another problem, and 3par support brought me the answer

I upgraded my storeserv 7200c to v3.2.2. Since this time, the physical disks service time has double up

we can see this here : (3.2.2 installed on 24-oct), backend service time was average 11ms and after patching, st has an average of 18 ms

Image

according to what the support says : It is normal, the 3.2.2 firmware consumes more 3Par resouces due to new features added to this firmware. This is strange, but normal


Yeah I spotted that too, I'm sure in a previous upgrade it went the other way. :roll:

Author:  korny [ Tue Dec 05, 2017 7:49 am ]
Post subject:  Re: High IOPS, high bandwith poor performance

Well, that's good to hear this :)
On my side, the update was 3.2.1 => 3.2.2.612
same one your side ?

Page 2 of 2 All times are UTC - 5 hours
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/