HPE Storage Users Group

We have two 3PAR 7200 in a 1-to-1 FC synchronous remote copy setup with Vsphere Peer Persistence, that were functioning correctly with 3.1.3.
I upgraded to 3.2.1 about two weeks ago, and since then both 3PAR's loose replication links a few times per day at random.

A case is open at HP. It is determined that the FC ports on the nodes are sometimes being reset. This causes for a +20 second outage of a FC replication link, and each time a lot of alerts are triggered and a case is automatically generated at HP. 30 seconds later, the link comes back on-line and status is OK for the whole system.

HP asked to install patch P01 for OS 3.2.1, but this has not yet resolved the problem.
The case is escalated to level 3, and we've been waiting for a couple of days now.

Just wanted to let people know who are thinking about upgrading.
I will post back when the issue is resolved.

Today came 3.2.1MU1 out with many fixed Bugs/Problems.

http://h20565.www2.hp.com/portal/site/h ... cachetoken

As far as I understand, 3.2.1 MU1 = 3.2.1 + P01 + dedupe feature.

We don't need dedupe, and P01 is already installed.
Patch P01 fixes a bunch of remote copy problems indeed. But apparently not our issue.

I'm still waiting for L3 feedback, where the case is still open.

I finally got feedback. Not so good. There is a confirmed problem with 3.2.1 remote copy under certain circumstances, and engineering has to find out what's the root cause:

Hi Han,
I am really sorry that its taking so long to investigate the issue you raised with your 3Par.
As you already know, the investigation has been sent further up to HP 3par engineering, as we could not figure why the “Automatic, failover single RC link active (degraded)”.

We got the below updates from ERT

“The alert the array is getting for the RCFC port is not really alarming . It is an issue that appears to be popping up since 3.2.1. ERT needs some time to dive into the code to figure out what they actually changed...”

The lab has determined that due to an issue in 3.2.1.MU1, online upgrades from 3.2.1.GA to 3.2.1.MU1 with remote copy enabled cannot be supported. A release vehicle to support online upgraded with RC enabled will be forthcoming, along with official communication.

Hi

I also had the same issue with dropping RCFC links on one of our two 7400 arrays after upgrade to 3.2.1GA.

Today I had HP engineer onsite and we performed an online upgrade to 3.2.1MU1 but we did that with all RCgroups and Quorum Witness stopped. No issues with uppgrade, no issues to restart QW and all RCgroups and have them successfully resync.

However, verifying firmware revisions as an post upgrade task we found out that on our second array and the second node (which had the dropping RCFC links) we were not running latest NEMOE firmware (shownode –mcu).

I not sure what NEMOE firmware does and if it's at all related to RCFC issues, but we have sent this to L2 for review.

Regards,
Peter

HP just released P04 for 3.2.1 (MU1) today, which implements additional fixes for Remote Copy and also issues with thin deduplication.

Release notes are here:
http://h20565.www2.hp.com/hpsc/doc/public/display?sp4ts.oid=5044394&docId=emr_na-c04511879&docLocale=en_US

Update:
HP Support has implemented 3.2.1 MU2, which solved all of our replication problems/errors, and some additional (false) alerts about corrupt files on the controller node flash disks.

At this moment, 3.2.1 MU2 is not user-installable. A bunch of scripts were run by HP support engineer during the upgrade, in some kind of 'root mode' only accessible by HP employee.

Unlike other upgrades I did before, it was not as simple as just downloading and staging an ISO file, and let the system upgrade itself.

Anyway, 3.2.1 MU2 seems to include a bunch of fixes for 3.2.1 GA.

Just for reference, here's a list of all the errors we received (multiple times per day) and which were all resolved after upgrade to 3.2.1 MU2:

Abstract:
RCLINK15(3PAR01_1_1_2) COMP_STATE failed. Remote Copy Link Down Due To Send Error

Abstract:
1639484 'CompType' 'comp_sw_atf' not found

Abstract:
comp_sw_atf0 COMP_STATE degraded. Automatic failover Single RC link active(Degraded)

Abstract:
NODE1 IDE_FILEINTEGRITY invalid files found on the internal drive(Major)

Abstract:
Pattern search failed

Abstract:
1 CLI CMD_ERR {3parsvc super all {{0 8}} -1 192.168.x.x 26114} {Command:
get(Minor)