Wednesday, March 13, 2024

Interesting Issue on Oracle Forms 12C due to Digicert Code Signing ceriticate-related Revocation Checks

Forms 12C customers should know this. The jar files/the standard code delivered by Oracle is signed with the Digicert certificates and these certificates, which are there for the code signing, are used ensure the security on the client side. I won't go into the details of how code signing works, and I suppose most of my followers already know this.

Here, I want to share something interesting.. This is based on a true story, on an issue escalated to me.

The environment was a clustered Forms 12C (12.2.1.3). It was highly critical and very crowded.(3000 active users)

The issue appeared suddenly and clients started to complain about the slowness of client-side forms.  Actually forms java client startup was taking time and java console gave us the clues.

We had read timeouts recorded in the java console output both for ocsp and crl3 urls of the Digicert..

Following is an example of ocsp related output taken from the java console:

Our java output: network: Connecting socket://ocsp.digicert.com:80 with proxy=DIRECT security: Revocation Status Unknown com.sun.deploy.security.RevocationChecker$StatusUnknownException: java.net.SocketTimeoutException: Read timed out

Clients had internet connection, no issues were found on firewall but we had read timeouts, and we started analyzing it.

We had the following 3 workarounds:

1)Use FSAL mode of running forms.

2)Modify the "Certificate revocation check" to "Publisher's certificate only." Follow Start -> Control Panel -> Java to access the Java Control Panel then click on the Advanced tab. Under "Perform signed code certificate revocation checks on" check the "Publisher's certificate only" or do not check option using radio button. -- note that.

3)Disable ocsp and crl based revocaction checks on the java client using java control panel.

But! our customer didn't accept these.. (due to the difficulty of delivering these configurations to 3000 clients)

Then, we analyzed deeper, and found that the issue is caused by Digicert itself.

Digicert was having troubles giving these ocsp and crl3 services and due to that, forms clients were facing these issues. (openssl can be used to test the revocation checks manually..)

We also found the following web pages in the digicert website and these let us see the current status of digicert services and follow the incidents and planned maintenance tasks (noted to be used for the future)

https://status.digicert.com

https://status.digicert.com/history

Anyways, problem disappeared when the Digicert's crl3 and ocsp services started working again.

But this showed us Oracle's dependency to Digicert. (Oracle Forms in this case), and we had no possible solutions for getting rid of this dependency. 

That dependency cannot be destroyed as the forms jar files are signed using the Digicert certificates. Again, workaround comes into picture.

We thought we can resign the jar files with --let's say-- a Verisign certificate and get rid of the dependency to Digicert, but it is not supported.. The default forms jar file should not be signed with any custom certificates.

Anyways.. It was an interesting issue but we learned something.. You can't easily find client-server architecture in the enterprise environments nowadays, but if you have one, this might help you one day.

Again, these tiny little details gives you a lot of information.

Of course information is contextual and it is never absolute! But, this might save your day one day :) --especially if you are in the same context --> using Oracle Forms..

By the way, clients having Oracle Forms 12.2.1.3 should consider 12.2.1.4 upgrade.

Actually it was expected that customers will move away from 12.2.1.3.0 in 2020, one year after 12.2.1.4.0 was released.

Upgrading to 12.2.1.4.0 is not so hard and you can even do it with zero down time. You can also use the same domain files. Also keep in mind that 12.2.1.4.0 will be out of support in September this year, so I encourage you to upgrade to 12.2.1.19.0 and receive the full support you deserve for the same money you are paying for an obsolete version.

Tuesday, March 12, 2024

Oracle SuperCluster M8 - End Of Life (EOL) & End Of Support (EOS)

I want to remind you something important regarding the Engineered systems. That is, as you may already know, the Oracle Supercluster M8 reached its End of Life (EOL) on June 30th, 2019. End of Life (EOL) signifies that Oracle no longer sells the hardware for the Supercluster M8.

In addition to that, end of Support (EOS) status is also important.. EOS indicates that Oracle no longer provides guaranteed support for the system, including hardware replacement, bug fixes, and security patches. Well, last ship day of Supercluster M8 is Jun 2019. So, the premier support will end in June 2024.


After June 2024, you may go with the extended support. But! It won't be like premier support and may have cons. (delays, lack of premier support-like concentration on the Oracle side and stuff like that..)

For example, when you need to replace a faulty disk in an environment with extended support, you may wait longer than in an environment with premier support. (That disk type you need may not be present in your region, and getting it from another region may take time.).

Delays, guarantees... You may have disadvantages in these.

Ref: List of Oracle Supported Hardware with Last Ship Dates Announced (Doc ID 1450710.1)

Note that, after five years from last ship date, replacement parts may not be available and/or the response times for sending replacement parts may be delayed.”

Ref: https://www.oracle.com/us/hardware-systems-support-policies-069182.pdf

Anyways.. While you can technically continue using the Supercluster M8 after June 2024, the effective end of support falls on that date. Here are some potential risks of using an Oracle Supercluster M8 after EOS:

  • Increased security vulnerabilities: Without security patches, your system becomes more susceptible to attacks.
  • Limited hardware support: If a hardware component fails, you might have difficulty finding replacements or repairs.
  • Compatibility issues: Newer software versions may not be compatible with the EOS system, hindering future upgrades.
  • No guaranteed bug fixes: You'll be on your own for troubleshooting any software bugs encountered.

Considering these risks, it's recommended to migrate from Oracle Supercluster M8 to a supported system (Like EXADATA or PCA or EXADATA + PCA, or ODA + EXADATA or Oracle/Sun Traditional Servers + ZFS.. and KVM virtualization.. Choice depends on the context actually.. These are some on-prem options... ) before June 2024.

Thursday, February 22, 2024

RDBMS 19C -- Tablespace and File I/O Stats missing in AWR reports / real fix -> dbms_workload_repository.modify_table_settings , flush_level

You may be hitting this bug even if your Oracle database version is 19C (even if it is 19.14..)

You have a workaround (setting statistics_level to ALL), but it is not recommended.. (due to the fact that, you may end up collecting too much data for the  AWR snapshots) Note that, your AWR snapshot related data collection tasks may also take too much time to complete, if you set statistics_level to ALL.. especially if you have a large buffer cache...

At the first glance, this problem seems addressed with the patch 22048821 TABLESPACE IO STATISTICS MISSING FROM AWR REPORT (Patch).. But it is actually not so :)

Also, this seems already fixed (the fix is already delivered) in versions like "19.8.0.0.200714 (Jul 2020) Database Release Update (DB RU)", but it is practically not so :)

Note that, this problem is also seen in 19.14, but when I checked my 19.20 DB, the issue is not there.. It seems the Enhancement request which is opened for 19.14 received a good response...

Related ER -> Bug 35122334 : REQUEST TO INCLUDE TYPICAL AS A DEFAULT VALUE TO BYPASS MANUAL EXECUTION OF DBMS_WORKLOAD_REPOSITORY.MODIFY_TABLE_SETTINGS AFTER APPLYING FIX 25416731

Related MOS Doc. -> Missing Tbspace I/O Stats AWR Report need to change the default dbms_workload_repository.modify_table_settings for WRH$_FILESTATXS WRH$_DATAFILE Tempfile WRH$_TEMPFILE and WRH$_TEMPSTATXS (Doc ID 2954137.1)

Probably, after 19.14, the issue is fixed. That is, after the Release 19.14, Oracle included the fix for bypassing the need of manual execution of the DBMS_WORKLOAD_REPOSITORY.MODIFY_TABLE_SETTINGS.

Anyways, for Oracle 19.x (<=19.14), the fix of that bug and fix of related the Enhancement Request 27871293 doesn't really fix this problem.  

To fix this and get the TBS and File I/O Stats data back in AWR reports in 19.X versions, we should enable flushing for the WRH$_FILESTATXS, WRH$_DATAFILE, WRH$_TEMPFILE and WRH$_TEMPSTATXS tables by taking the statistics level of the Tempfile Group into consideration.

Well.. We have to run the following in order to do that;

     $ sqlplus / as sysdba

        exec dbms_workload_repository.modify_table_settings(table_name  => 'WRH$_FILESTATXS', flush_level => 'TYPICAL');

        exec dbms_workload_repository.modify_table_settings(table_name  => 'WRH$_DATAFILE', flush_level => 'TYPICAL');

        exec dbms_workload_repository.modify_table_settings(table_name  => 'Tempfile Group', flush_level => 'TYPICAL');

        exec dbms_workload_repository.modify_table_settings(table_name  => 'WRH$_TEMPFILE', flush_level => 'TYPICAL');

        exec dbms_workload_repository.modify_table_settings(table_name  => 'WRH$_TEMPSTATXS', flush_level => 'TYPICAL');

After these actions, we will have those missing stats back.. but! as you may imagine -> we will have those stats in the new snapshots and in the new AWR reports...

This is a very important thing and should be taken care off.. You may not be aware of this... But without having these stats in place, believe me, one day your AWR reports will leave you alone in dealing with a real performance problem 

It is important for being able to check historical IO performance of Oracle Database and that's why I shared it with you.

Tuesday, February 20, 2024

Let's quantize it! :) -- Basics of Quantum Information

I received my first certificate in IBM quantum and I am sharing it here. More to come :)

This was for being able to describe quantum information at a foundational level. While preparing, you learn how to describe fundamental concepts, such as quantum teleportation, superdense coding, and the Clauser, Horne, Shimony, and Holt (CHSH) game — all of which are connected to the phenomenon of entanglement. 

By studying these, you gain the core skills required to explore deeper fundamental quantum algorithms and start using Qiskit Runtime.

Saturday, February 10, 2024

Erman Arslan's Oracle Forum / NOV 24 2023 - FEB 10, 2024 - "Q & A Series"

Remember, you can ask questions and get remote support using my forum. Just click on the link named "Erman Arslan's Oracle Forum is available now". Click here to ask a question", which is available on the main page of Erman Arslan's Oracle Blog. More than 2000 questions were asked, almost 10000 comments have been made and here is the links to the latest questions from the last period;

-- or just use the direct link:


Supporting the Oracle users around the world. Let's check what we have in the last few weeks..

Bursting Emails R12.2 by satish

Need advice on Configuration - R12.2 by satish

Concurrent user connections with RAC by satish

perl adcfgclone.pl appsTier fails by big

How to temporarily stop sending emails - PROD by satish

EBS Support by satish

Query Regarding Patching of Inactive Products by VinodN

Hide APPS password in OS script by VinodN

EBS 12.2 New Installation with RAC DB by veysel

Autoconfig on patch file system failed. by raiq1

SSO login Error 404--Not Found by big

Strat managers by big

change Weblogic admin passeword fails by big

APPS DBA Course by VinodNN

Cutover failed after jdk upgrade by karthikmca.bhc

IREC setup withing DMZ by akhan

Adop phase=prepare failed on External nodes. by akhan

 While trying to lookup didn't find subcontext 'jdbc' Resolved ''; remaining name 'jdbc/SSSTCCRP' by satish

Friday, November 24, 2023

Erman Arslan's Oracle Forum / OCT 12 - NOV 24, 2023 - "Q & A Series"

Remember, you can ask questions and get remote support using my forum. Just click on the link named "Erman Arslan's Oracle Forum is available now". Click here to ask a question", which is available on the main page of Erman Arslan's Oracle Blog. More than 2000 questions were asked, almost 10000 comments have been made and here is the links to the latest questions from the last period;



-- or just use the direct link:


Supporting the Oracle users around the world. Let's check what we have in the last few weeks..

Monday, November 20, 2023

-Revisiting- --> Problem installing Oracle Linux 7 and 8 on Virtual Box 7.0.19 -- Unsupported command : unsupported_hardware

Found a quicker way to workaround the issue, that's why revisiting this  ->(https://ermanarslan.blogspot.com/2023/11/problem-installing-oracle-linux-7-and-8.html).

Some background info:

This problem is fixed in the Virtual Box Release version 7.0.12.

In the problem-free Virtual box release 7.0.12, they commented that unsupported_hardware line in the default ks.cfg, that is delivered with the Virtual Box. That ks.cfg is automatically used in unattended installation, as you may guess. Therefore, the unattended installation works properly in 7.0.12.

The Quicker Way -> So,  If you are still using  in 7.0.10, and if you just comment the unsupported_hardware line in the ks.cfg in the relevant directory, which is created on your PC with the Virtual Box installation, then the unattended installation will also work properly. No need for other extra efforts..

Also, it seems inconsistent to both provide ks.cfg in Linux boot and select unattended installation in Virtual Box. One of these should be done.

To boot with the kickstart in the problematic Virtual Box environments, the ks.cfg that comes with the virtual box should be modified (as suggested above) and unattended installation should be selected for the Linux boot. This is the simplest method to kick-start in Virtual Box. Actually, there is "no need" to go into the Linux ISO, repackage it and do any work there. If we select the unattended installation, then the Virtual Box uses its own ks.cfg to kickstart.. Probably Virtual Box has its own boot loader and that's why it is able to do it. I mean, without touching anything Virtual Box starts a kick start installation with its own ks.cfg and we don't need to  touch anything in the Linux ISO for this..  (This is just my opinion, I didn't dive too deep about it.. Just saying...)

Friday, November 3, 2023

Another fun departure from the norm : Another TV show! This time It was a LIVE Brodcast!

As an Oracle ACE PRO + System and Database Senior Director, I was the guest of University TV's Professional Agenda Program. We had a nice conversation about my areas of expertise with the host of the program. We had some chats on AI and Quantum Computing, too. We discussed proper career planning for university students. We talked about curiosity and passion in this context. This was a live broadcast on Wednesday, October 18, 2023.

Nowadays, offers are coming one after another for me to participate in the TV programs and I am starting to enjoy it :)

Here I 'm leaving some photos from the last show. Sharing the poster of the TV Program as well.. Just for the memories..




Problem installing Oracle Linux 7 and 8 on Virtual Box 7.0.19 -- Unsupported command : unsupported_hardware

Let's do some Linux & virtualization gymnastics by going through a real life problem.

Getting unsupported command: unsupported_hardware error during Oracle Linux 7 (and 8) installation on Virtual Box 7.0.10 (a very up-to-date release of Virtual Box release at the moment). 

The installation couldn't start actually.. It was terminated due to the unknown command error. (note that, this is not related with hardware.) That error message may be a bit misleading in the first glance.. Having unsupported_hardware there make it a bit counter-intuitive.. The real error is the unknown command error.. The installer just doesn't know the command which is "unsupported_hardware".

Source Specs: Windows 11 (or Windows 10) -- actually there is no causal connection between this error and the client OS.  The error seems purely related with Virtual Box. But! still noted the Windows version here.

The reason behind this error is using an deprecated command in the kickstart file..

unsupported_hardware is a deprecated command and that's why it was causing the error. (note that using these type of commands in the kickstart "may" cause an error and here it did!)

Brief info about kickstart:

Kickstart helps us to automate the installation with a simple text file.. 

It is an automated process that let us install the operating system by specifying all our requirements and configurations in a file.  (time zone, drive partitions, packages etc..) 

No user intervention is needed. 

It is mostly used when deploying Linux on a large number of systems at once.

Workaround 1)

In order to workaround the problem and continue the installation, open a different TTY when the issue appeared.. (using Ctrl + Alt + F3 : (tty3) or Ctrl + Alt + F4 : (tty4)). 
When opened the TTY, you 'll find yourselves in installation filesystem.. Modify the kickstart file (delete the unsupported_hardware line), save it and then run the anaconda from there, in that shell.. As simple is that.

Workaround 2)

A better workaround is to edit the vmlinuz command (by pressing TAB when the installation menu is displayed as seen in the screenshot below)..  Just delete the word quiet and the words related with the kickstart installation. Press Enter and you will be good.. The installation will start and it will be with GUI.

Note that, vmlinuz /Virtual Memory LINUx gZip = Compressed Linux kernel is the compressed image of the linux kernel. Uncompressed, loaded into memory, and executed during the process.


Workaround 3)

An even better workaround is to check the Skip Unattended Installation checkbox which is present in Virtual Box 7.0.1..  Skip unattended installation is enabled by default as far as I see, but we disable it..
Not all the Operating Systems are tested with this option.. (according to Oracle, following OEL versions are tested: Oracle Linux 6.1,Oracle Linux 7.0,Oracle Linux 7.7,Oracle Linux 8.0,Oracle Linux 8.2.. Note that we are trying with Oracle Linux 8.6....)


Workaround 4 - can be considered as the fix..

Regenerate Oracle Linux 8 iso with a modified ks.cfg / kick-start config file which doesn't contain that deprecated command. This is a stylish move, but a little complex -- especially if you are not that familiar with Linux..


Workaround 5 - "Just use Virtualbox 7.0.12" :) - this can also be considered as the fix.

Thursday, October 12, 2023

Erman Arslan's Oracle Forum / AUG 25 - OCT 12, 2023 - "Q & A Series"

Remember, you can ask questions and get remote support using my forum. Just click on the link named "Erman Arslan's Oracle Forum is available now". Click here to ask a question", which is available on the main page of Erman Arslan's Oracle Blog. More than 2000 questions were asked, almost 10000 comments have been made and here is the links to the latest questions from the last period.


-- or just use the direct link:


Supporting the Oracle users around the world. Let's check what we have in the last few weeks..

Need Advice on job queue processes for upgrading to 12.2.10 by prabhunoule

autostart grid by Roshan

Pending OPP issue with custom program by VinodN

Unable to Login EBS R12.2.10 in Windows 7 32bit by prabhunoule

Workflow Java Mailer Analyzer by big

enterprise manager wallet by Roshan

Login Issue with users by prabhunoule

restore data pump by Roshan

mariadb locks by Roshan

undo_retention in 19c database by raiq1

Query on interoperability patches from 11.2.0.4 to 19C by prabhunoule

restart one node by Roshan

Discoverer 10g by Samia

restore data pump by Roshan

Oracle Database 12.1.0.2 to 19c non-CDB upgrade by Hassan141

refresh DB by Roshan

need ebs online instance by motoky

XLAACCPB: Create Accounting issue by Samia

Clone failed 19c database. by satish

Prepare phase failed ORA-00904: "TABTYPE": invalid identifier by satish