So I am assuming this question in focused on host ports only, Service time is a good one to watch, however there is a caveat to that. Some hosts (linux) have a feature called "write barriers" used enabled by default on journaled file systems that effectively disables write cache by flushing cache before and after every write, causing write service time to (needlessly?) skyrocket. If that is the case in your environment, you may be forced to abandon Svc Time as a functional metric.
This leaves Bandwidth, IOPS, and Queue Length.
You have bandwidth nailed down, you may want to set up SR alerts to let you know when ports are 80% maxed, or so, per your preference. If you discover this triggers alot, you may need to review your multipathing setup, ensure round robin is being used on the host(s), or split your host HBA port to two 3PAR storage hostports, so a single host running at "full speed" will only push the 3PAR hostports at max 50%. Since SR does not have "time window exclusions" you can use outlook rules to filter out unwanted SR alerts during blocks of time... for that I had to use the "message header contains" to delete emails where the header contains "04:" for the 4:am hour. Outlook 2016 does not appear to support a "time between" function.
So the hostports have a published max queue depth you can lookup in the "How to Calculate Suitable Queue Depths" guide found here:
http://h20195.www2.hp.com/v2/getpdf.asp ... 094ENW.pdfI recently started to use 1/2 the published queue depth as a System Reporter alert for hostports, so far we don't come close, our legacy 15k RPM PDs bottleneck before the hostport queue depth does, however I would imagine with an all flash array this will be important to monitor.
The mystery metric that I have the most issues with is how many IOPS can a hostport do. The ASICs on the HBA have a physical limit to how many IOPS they can process per second, but if you are lucky enough to find a published specification for this, it's usually referencing a micro block size that is not real world.
So how many disks, and disk types do you have? If you are using spinning disk, chances are your bottleneck will be best seen via PD performance.