Two Clusters with Same Name. How to Fix It?

Hi everyone, hope you’re doing well!

I was working on a RAC migration project for a client. I did the Cluster install for the first environment. Cluster and DB versions are 19 on Linux. As a DBA working on Linux, I always prefer to install the software using the silent option with a response file. No GUI!

The client DBA team then did the installation for the rest of the environments. They were basically copying the control file that I built to the other environments and were replacing some variables like: SCAN name, cluster nodes, cluster name, etc.

Well, when they were doing the installation in the environments, the DBA forgot to change the parameter for the cluster name in one of the clusters. So, what happened? Two clusters configured with the same name.

The official documentation is very clear about this:

You can find this info here:

Cluster Name and SCAN Requirements

If you have two or more clusters with same name on your network, you must fix it.

If you are running your cluster below version 19c, you will need to use the -deconfig and re(-config) your cluster as listed on Support Note: How to Change the Cluster Name in RAC (Requires -deconfig and (re)-config of Grid Infrastructure) (Doc ID 1967916.1)

Luckily, our client was running on 19c, and now the procedure is easier and faster. You can follow the new procedure as per my blog post, if you prefer, you can read directly from the Support Note: Steps to rename the cluster name (Doc ID 2725377.1)

I will explain here about the steps, however, it’s important to understand the pre-requirements:

• Supported only for “Standalone Cluster”
• Not supported for SIHA
• Not supported if RHP Server or RHP Client is configured
• Grid Infrastructure stack should be up on all nodes when the “Cluster Rename Procedure” is performed
• Requires unique new cluster name

Update, thank you Maicon Carneiro for catching up this!

I also recommend to check if rhpclient resource is enabled and active.
If so, the DBA will need to unregister all workingcopies and the client metadata related to this cluster in the RHP Server before removing the rhpclient in the Cluster that is being renamed.

After the renaming is complete, the RHP Client can be configured again with the new name, and the workingcopies can be registered manually as well. This is not a trivial task, but can be done.

As on my environment, the DB server is not working as a RHP Client, I did not performed any deletion on workingcopies, etc.

You can check if RHP Client is configured running this:

srvctl status rhpclient

To list and delete workingcopies, and delete rhpclient, you need to use rhpctl command. You can check the reference for rhpctl here.

OK, we met all the requirements listed above.

Let’s start our procedure doing some checks.

Let’s check if cluster is running on all nodes. Both for grid and root users, we have the environment variables set pointing to GRID_HOME:

[root@node01 ~]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@node01 ~]#


[root@node02 ~]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@node02 ~]#

On any node, as root, check the current cluster name:

[root@node01 ~]# cemutlo -n
wrongcluster

The client is not using RHP here, if you are using, keep in mind you will need to disable and remove the resource from the cluster to proceed using the rename cluster feature without deconfig. Let’s check if RHP is running. As grid user on any node:

[grid@node01 ~]$ srvctl status rhpserver
Rapid Home Provisioning Server is enabled
Rapid Home Provisioning Server is not running

OK, we can see that RHP is enable but not running, let’s disable it:

[grid@node01 ~]$ srvctl disable rhpserver

[grid@node01 ~]$ srvctl status rhpserver
Rapid Home Provisioning Server is disabled
Rapid Home Provisioning Server is not running

Now that we disabled RHP, let’s remove it from the cluster:

[root@node01 ~]# srvctl remove rhpserver

Perfect! We are good to proceed! You’ll see that cluster rename operation is pretty straightforward!

On the first node, as root user, perform the cluster rename operation:

[root@node01 ~]# crsrename cluster rightcluster
CRS-42004: successfully set the cluster name; restart Oracle High Availability Services on all nodes for new cluster name to take effect

Perfect! Cluster name changed! Let’s restart the cluster services on all nodes. As root on all nodes:

[root@node01 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'node01'
CRS-2673: Attempting to stop 'ora.crf' on 'node01'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'node01'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'node01'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'node01'
CRS-2677: Stop of 'ora.crf' on 'node01' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'node01'
CRS-2677: Stop of 'ora.drivers.acfs' on 'node01' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'node01' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'node01' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'node01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'node01' has completed
CRS-4133: Oracle High Availability Services has been stopped.

[root@node02 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'node02'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'node02'
CRS-2673: Attempting to stop 'ora.crf' on 'node02'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'node02'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'node02'
CRS-2677: Stop of 'ora.drivers.acfs' on 'node02' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'node02' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'node02' succeeded
CRS-2677: Stop of 'ora.crf' on 'node02' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'node02'
CRS-2677: Stop of 'ora.gipcd' on 'node02' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'node02' has completed

Now that cluster services are stopped in all nodes. Let’s start it:

[root@node01 ~]# crsctl start crs -wait
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.evmd' on 'node01'
CRS-2672: Attempting to start 'ora.mdnsd' on 'node01'
CRS-2676: Start of 'ora.mdnsd' on 'node01' succeeded
CRS-2676: Start of 'ora.evmd' on 'node01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'node01'
CRS-2676: Start of 'ora.gpnpd' on 'node01' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'node01'
CRS-2676: Start of 'ora.gipcd' on 'node01' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'node01'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'node01'
CRS-2676: Start of 'ora.cssdmonitor' on 'node01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'node01'
CRS-2672: Attempting to start 'ora.diskmon' on 'node01'
CRS-2676: Start of 'ora.diskmon' on 'node01' succeeded
CRS-2676: Start of 'ora.crf' on 'node01' succeeded
CRS-2676: Start of 'ora.cssd' on 'node01' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'node01'
CRS-2672: Attempting to start 'ora.ctssd' on 'node01'
CRS-2676: Start of 'ora.ctssd' on 'node01' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'node01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'node01'
CRS-2676: Start of 'ora.asm' on 'node01' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'node01'
CRS-2676: Start of 'ora.storage' on 'node01' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'node01'
CRS-2676: Start of 'ora.crsd' on 'node01' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'node01'
CRS-6017: Processing resource auto-start for servers: node01
CRS-2672: Attempting to start 'ora.scan3.vip' on 'node01'
CRS-2672: Attempting to start 'ora.scan2.vip' on 'node01'
CRS-2672: Attempting to start 'ora.scan1.vip' on 'node01'
CRS-2672: Attempting to start 'ora.node01.vip' on 'node01'
CRS-2672: Attempting to start 'ora.ons' on 'node01'
CRS-2672: Attempting to start 'ora.node02.vip' on 'node01'
CRS-2672: Attempting to start 'ora.qosmserver' on 'node01'
CRS-2672: Attempting to start 'ora.cvu' on 'node01'
CRS-2676: Start of 'ora.cvu' on 'node01' succeeded
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'node01' succeeded
CRS-2676: Start of 'ora.node01.vip' on 'node01' succeeded
CRS-2672: Attempting to start 'ora.MGMTLSNR' on 'node01'
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'node01'
CRS-2676: Start of 'ora.node02.vip' on 'node01' succeeded
CRS-2676: Start of 'ora.scan3.vip' on 'node01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'node01'
CRS-2676: Start of 'ora.scan2.vip' on 'node01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'node01'
CRS-2676: Start of 'ora.scan1.vip' on 'node01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'node01'
CRS-2676: Start of 'ora.qosmserver' on 'node01' succeeded
CRS-2676: Start of 'ora.MGMTLSNR' on 'node01' succeeded
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'node01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'node01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'node01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'node01' succeeded
CRS-2676: Start of 'ora.ons' on 'node01' succeeded
CRS-2672: Attempting to start 'ora.mgmtdb' on 'node01'
CRS-2672: Attempting to start 'ora.RACTST.db' on 'node01'
CRS-2676: Start of 'ora.mgmtdb' on 'node01' succeeded
CRS-2672: Attempting to start 'ora.chad' on 'node01'
CRS-2676: Start of 'ora.chad' on 'node01' succeeded
CRS-2676: Start of 'ora.RACTST.db' on 'node01' succeeded
CRS-2672: Attempting to start 'ora.RACTST.preprod_batch_srvc.svc' on 'node01'
CRS-2672: Attempting to start 'ora.RACTST.DBRAC_svc.svc' on 'node01'
CRS-2676: Start of 'ora.RACTST.preprod_batch_srvc.svc' on 'node01' succeeded
CRS-2676: Start of 'ora.RACTST.DBRAC_svc.svc' on 'node01' succeeded
CRS-6016: Resource auto-start has completed for server node01
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.


[root@node02 ~]# crsctl start crs -wait
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'node02'
CRS-2672: Attempting to start 'ora.evmd' on 'node02'
CRS-2676: Start of 'ora.mdnsd' on 'node02' succeeded
CRS-2676: Start of 'ora.evmd' on 'node02' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'node02'
CRS-2676: Start of 'ora.gpnpd' on 'node02' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'node02'
CRS-2676: Start of 'ora.gipcd' on 'node02' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'node02'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'node02'
CRS-2676: Start of 'ora.cssdmonitor' on 'node02' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'node02'
CRS-2672: Attempting to start 'ora.diskmon' on 'node02'
CRS-2676: Start of 'ora.diskmon' on 'node02' succeeded
CRS-2676: Start of 'ora.crf' on 'node02' succeeded
CRS-2676: Start of 'ora.cssd' on 'node02' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'node02'
CRS-2672: Attempting to start 'ora.ctssd' on 'node02'
CRS-2676: Start of 'ora.ctssd' on 'node02' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'node02' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'node02'
CRS-2676: Start of 'ora.asm' on 'node02' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'node02'
CRS-2676: Start of 'ora.storage' on 'node02' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'node02'
CRS-2676: Start of 'ora.crsd' on 'node02' succeeded
CRS-6017: Processing resource auto-start for servers: node02
CRS-2673: Attempting to stop 'ora.node02.vip' on 'node01'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'node01'
CRS-2672: Attempting to start 'ora.ons' on 'node02'
CRS-2672: Attempting to start 'ora.chad' on 'node02'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'node01' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'node01'
CRS-2677: Stop of 'ora.node02.vip' on 'node01' succeeded
CRS-2672: Attempting to start 'ora.node02.vip' on 'node02'
CRS-2677: Stop of 'ora.scan1.vip' on 'node01' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'node02'
CRS-2676: Start of 'ora.chad' on 'node02' succeeded
CRS-2676: Start of 'ora.node02.vip' on 'node02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'node02'
CRS-2676: Start of 'ora.scan1.vip' on 'node02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'node02'
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'node02' succeeded
CRS-33672: Attempting to start resource group 'ora.asmgroup' on server 'node02'
CRS-2672: Attempting to start 'ora.asmnet1.asmnetwork' on 'node02'
CRS-2676: Start of 'ora.asmnet1.asmnetwork' on 'node02' succeeded
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'node02'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'node02' succeeded
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'node02' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'node02'
CRS-2676: Start of 'ora.ons' on 'node02' succeeded
CRS-2676: Start of 'ora.asm' on 'node02' succeeded
CRS-33676: Start of resource group 'ora.asmgroup' on server 'node02' succeeded.
CRS-2672: Attempting to start 'ora.RECOV_VP.dg' on 'node02'
CRS-2672: Attempting to start 'ora.DATAV_VP.dg' on 'node02'
CRS-2672: Attempting to start 'ora.REDO2_VP.dg' on 'node02'
CRS-2672: Attempting to start 'ora.REDO1_VP.dg' on 'node02'
CRS-2676: Start of 'ora.RECOV_VP.dg' on 'node02' succeeded
CRS-2676: Start of 'ora.DATAV_VP.dg' on 'node02' succeeded
CRS-2676: Start of 'ora.REDO2_VP.dg' on 'node02' succeeded
CRS-2676: Start of 'ora.REDO1_VP.dg' on 'node02' succeeded
CRS-2672: Attempting to start 'ora.RACTST.db' on 'node02'
CRS-2676: Start of 'ora.RACTST.db' on 'node02' succeeded
CRS-2672: Attempting to start 'ora.RACTST.DBRAC_svc.svc' on 'node02'
CRS-2676: Start of 'ora.RACTST.DBRAC_svc.svc' on 'node02' succeeded
CRS-6016: Resource auto-start has completed for server node02
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.

Well, we have reconfigured RHP in case client would like to start using it. As root on first node:

[root@node01 ~]# srvctl add rhpserver -storage /opt/oracle/rhp_images -diskgroup GIMR_VP -pl_port 8896 -enableHTTPS YES

Now let’s check the config for the RHP:

[root@node01 ~]# srvctl config rhpserver
Storage base path: /opt/oracle/rhp_images
Disk Groups: GIMR_VP
Port number: 8896
Temporary Location:
Transfer port range:
Rapid Home Provisioning Server is enabled
Rapid Home Provisioning Server is individually enabled on nodes:
Rapid Home Provisioning Server is individually disabled on nodes:
Email address:
Mail server address:
Mail server port:
Transport Level Security disabled
HTTP Secure is enabled
Endpoint : gns

Let’s check if RHP is configured, enabled, but not running (same status we had before the cluster rename operation):

[grid@node01 ~]$ . oraenv
ORACLE_SID = [oracle] ? +ASM1
The Oracle base has been set to /u01/app/grid
[grid@node01 ~]$ srvctl status rhpserver
Rapid Home Provisioning Server is enabled
Rapid Home Provisioning Server is not running

As the last step, let’s check if the cluster name now is the right one:

[root@node01 ~]# cemutlo -n
rightcluster

You can also check the cluster name using the following command:

[grid@node01 ~]$ olsnodes -c
rightcluster

Perfect! Our cluster rename operation worked like a charm!

I hope it helps.

Peace,

Vinicius