July 2017 PSU, Patch Bugs and How to Avoid Getting Bit

Most DBAs spend significant time patching databases or find ways to automate patching databases. Oracle has introduced new patch utilities, Patch Plans and such to mitigate the demands, but still many of us depend on the quarterly PSU patch from Oracle for many of their products to address bugs, security issues and add enhancements.

The July 2017 PSU took on a record 308 fixes, which is a pretty awesome venture, but the idea that we’re introducing issues when we patch can be frustrating. It now appears that the July 2017 PSU is experiencing an issue for some customers, surrounding delayed restarts, one-off outages and an inability to rollback the patch. Oracle did notify customers with patch notification turned on and with a separate reference ID 2264643.1:

*Dear Oracle Customer, You are receiving this email because our
records indicate you downloaded the following patch: Patch number:
26550381 Release: Database Proactive Bundle Patch 12.1.0.2.170730
<12102170730> Platform: Linux x86-64 Please note that this patch has been
put on hold and is no longer available for download. Oracle recommends that
you not install the downloaded patch.*

Delphix is making the recommendation, understanding that impact to uptime is always a concern, to hold off on applying the July 2017 PSU until the new patch that supersedes the initial one is fully tested and available for any issues recently discovered.

If you’d like to review the discussions on the topic, you can view the following links:

• Oracle-L ORA-600 and Production Down

• OTN- DBCA Issue

• OTN- Delayed Startup

• OTN- Grid Infrastructure 11.2.0.4

Although the risk is to a small percentage of patched instances, some customers are taking the issue seriously and it is always important to review the risk matrix from Oracle from the patch and make the decision for yourself.

If you have Delphix in place and experience an outage, there are some opportunities to recover far quicker than if you didn’t.

Development or Test Virtual Database Impact

1. Spin up a new virtual database off of an unpatched (virtual or physical) Oracle home and the user will be back up in a matter of minutes. Wait until the updated patch is released.

2. Spin up a new virtual database from the patched Oracle Home and see if it experiences the same slowness in startup.

3. Spin up a new virtual database and Oracle Home from the source and run the patch again to see if you bypass the bug, (experience has shown its very inconsistent in occurrence.)

Production is impacted and there’s an outage, (four main steps to process)

1. Disable the sync to Delphix from Production

2. Create a new virtual database, (VDB) from the latest version before the patch was applied.

3. Point production to this new VDB to continue return access for production

4. Begin a Virtual to Physical, (V2P) from the VDB being used for production to be the production if a recovery is required.

There’s a couple different ways to recover quickly via the Delphix Data Platform, but this gives you an idea of how flexible the virtualization component really is. The user can simply wait for Oracle to release the updated patch for production and then switch back, synchronizing the data between the databases post the event.