While trying to start Shareplex instance, it is failing with following errors.
##--- ##--- starting Shareplex instance ---## ##--- [oracle@mylab-01 log]$ $SP_BIN/sp_cop -u$SP_COP_TPORT & [1] 406 [oracle@mylab-01 log]$ ******************************************************* * SharePlex for Oracle Startup * Copyright 2014 Dell, Inc. * ALL RIGHTS RESERVED. * Protected by U.S. Patents: 7,461,103 and 7,065,538 * Version: 8.6.2.43-m64-oracle120 * VarDir : /shareplex/pdb131_2103 * Port : 2103 ******************************************************* can't setup shared memory statistics capability - exiting
Let’s check, if we can find anything from the event log.
##--- ##--- errors logged in event_log file ---## ##--- Error 2015-12-05 23:10:18.627346 406 3377944400 Cop: Error cleaning up previous shared memory segment 134381585 [module shs] Error 2015-12-05 23:10:18.627468 406 3377944400 Cop: Cannot delete because there are users attached [module shs] Notice 2015-12-05 23:10:18.627571 406 3377944400 Cop: Check if SharePlex processes are running and kill them if necessary [module shs] Notice 2015-12-05 23:10:18.627592 406 3377944400 Cop: Remove shared-memory segments, owned by the SharePlex user, with a key value ending in '8cbf' [module shs] Notice 2015-12-05 23:10:18.627616 406 3377944400 Cop: Remove semaphores, owned by the SharePlex user, with key values of 0x00008cbf or 0x00108cbf [module shs] Error 2015-12-05 23:10:18.627632 406 3377944400 Cop: can't setup shared memory statistics capability - exiting [module cop]
As per the event_log, Shareplex is not able to clean up the shared memory segments and looks like there are some stale processes attached to those shared memory segments. It is asking us to check if there are any Shareplex process running against the same port and kill those processes if required. Further, it is asking us to remove specific shared memory segments (key: 8cbf) and semaphores (Keys: 0x00008cbf and 0x00108cbf) owned by the Shareplex user.
Let’s first check, if there are any stale Shareplex processes running on the system.
##--- ##--- Shareplex processes running on the system ---## ##--- [oracle@mylab-01 log]$ ps -ef | grep sp_ oracle 12143 1 0 16:25 pts/2 00:00:00 /shareplex/splex_8.6.2_12/.app-modules/sp_xport -u2103 oracle 12144 1 0 16:25 pts/2 00:00:00 /shareplex/splex_8.6.2_12/.app-modules/sp_opst_mt -u2103 oracle 12145 1 0 16:25 pts/2 00:00:00 /shareplex/splex_8.6.2_12/.app-modules/sp_ordr -u2103 oracle 12252 1 0 16:29 pts/2 00:00:00 /shareplex/splex_8.6.2_12/.app-modules/sp_mport 0xc0a8e60c+PI+mylab-02+sp_mport+0xc0a8e60d R -u2103 oracle 12256 1 0 16:29 pts/2 00:00:03 /shareplex/splex_8.6.2_12/.app-modules/sp_ocap -u2103 oracle 14152 12087 0 16:40 pts/2 00:00:00 grep sp_
As we can see there are Shareplex processes running on this system. However, all of these processes are in stale state as the associated parent process (sp_cop) is not running. There is a possibility that Shareplex was terminated abruptly which left the child processes in a stale state. These stale processes are now preventing Shareplex to startup the instance against the same port. We need need to terminate these stale processes to be able to startup the Shareplex instance on the same port.
Let’s terminate the stale Shareplex processes.
##--- ##--- Terminating stale Shareplex processes ---## ##--- [oracle@mylab-01 log]$ kill -9 12143 12144 12145 12252 12256 ##--- ##--- Validating no stale processes exist ---## ##--- [oracle@mylab-01 log]$ ps -ef | grep sp_ oracle 14252 12087 0 16:41 pts/2 00:00:00 grep sp_
In most cases terminating the stale processes will also release the shared memory segments and semaphores associated with the stale processes. However, if the shared memory segments and semaphores are not released automatically, we need to do that manually.
We can use the ipcs command to identify the shared memory segments and semaphores associated with the stale processes. To find the shared memory segments and semaphores, we will match the memory segments and semaphores with the key found in the event_log. In our case the key is 8cbf for shared memory segments and 0x00008cbf, 0x00108cbf for semaphores.
Let’s check the status of shared memory segments and semaphores to find out if there are any segments/semaphores matching the given keys.
##--- ##--- checking shared memory segments and semaphores ---## ##--- [oracle@mylab-01 log]$ ipcs -smaa ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x00000000 3538944 root 644 80 2 0x00000000 3571713 root 644 16384 2 0x00000000 3604482 root 644 280 2 0x00000000 131694605 oracle 640 2932736 94 0x00000000 131727374 oracle 640 796917760 47 0x00000000 131760143 oracle 640 5455872 47 0x8abc60fc 131792912 oracle 640 16384 47 0x00008cbf 134381585 oracle 644 33554432 3 #--> matching segment 0x00108cbf 134414354 oracle 644 139264 3 #--> matching segment 0x00308cbf 134447123 oracle 644 8388608 2 #--> matching segment 0x00408cbf 134479892 oracle 644 33554432 1 #--> matching segment 0x00508cbf 134512661 oracle 644 8388608 0 #--> matching segment ------ Semaphore Arrays -------- key semid owner perms nsems 0xc7a6f808 2162689 oracle 640 126 0xc7a6f809 2195458 oracle 640 126 0xc7a6f80a 2228227 oracle 640 126 0xc7a6f80b 2260996 oracle 640 126 0xc7a6f80c 2293765 oracle 640 126 0x00008cbf 2326534 oracle 666 12 #--> matching semaphore 0x00108cbf 2359303 oracle 666 26 #--> matching semaphore ------ Message Queues -------- key msqid owner perms used-bytes messages
As we can see there are shared memory segments (shmid: 134381585,134414354,134447123,134479892,134512661) matching with the given key which needs to be released. Additionally, we can also find the semaphores (semid: 2326534,2359303) matching with the given key which also needs to be released.
Let’s release the identified shared memory segments and semaphores associated with the stale processes. We can use the command ipcrm with a flag -m shmid to release shared memory segments and with a flag -s semid to release the semaphores as shown below.
##--- ##--- releasing shared memory segments with 'ipcrm -m' ---## ##--- [oracle@mylab-01 log]$ ipcrm -m 134381585 [oracle@mylab-01 log]$ ipcrm -m 134414354 [oracle@mylab-01 log]$ ipcrm -m 134447123 [oracle@mylab-01 log]$ ipcrm -m 134479892 [oracle@mylab-01 log]$ ipcrm -m 134512661 ##--- ##--- releasing semaphores with 'ipcrm -s' ---## ##--- [oracle@mylab-01 log]$ ipcrm -s 2326534 [oracle@mylab-01 log]$ ipcrm -s 2359303
Lets validate if the memory segments and semaphores are released.
##--- ##--- checking shared memory segments and semaphores ---## ##--- [oracle@mylab-01 log]$ ipcs -smaa ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x00000000 3538944 root 644 80 2 0x00000000 3571713 root 644 16384 2 0x00000000 3604482 root 644 280 2 0x00000000 131694605 oracle 640 2932736 88 0x00000000 131727374 oracle 640 796917760 44 0x00000000 131760143 oracle 640 5455872 44 0x8abc60fc 131792912 oracle 640 16384 44 0x00000000 134381585 oracle 644 33554432 1 dest 0x00000000 134414354 oracle 644 139264 1 dest 0x00000000 134447123 oracle 644 8388608 1 dest ------ Semaphore Arrays -------- key semid owner perms nsems 0xc7a6f808 2162689 oracle 640 126 0xc7a6f809 2195458 oracle 640 126 0xc7a6f80a 2228227 oracle 640 126 0xc7a6f80b 2260996 oracle 640 126 0xc7a6f80c 2293765 oracle 640 126 ------ Message Queues -------- key msqid owner perms used-bytes messages
As we can observe, there are no more memory segments or semaphores matching with the key found in the event_log, which means those segments and semaphores are now released. Now, we should be able to start up the Shareplex instance without any issues.
##--- ##--- starting up Shareplex instance ---## ##--- [oracle@mylab-01 log]$ $SP_BIN/sp_cop -u$SP_COP_TPORT & [1] 1399 [oracle@mylab-01 log]$ ******************************************************* * SharePlex for Oracle Startup * Copyright 2014 Dell, Inc. * ALL RIGHTS RESERVED. * Protected by U.S. Patents: 7,461,103 and 7,065,538 * Version: 8.6.2.43-m64-oracle120 * VarDir : /shareplex/pdb131_2103 * Port : 2103 ******************************************************* ##--- ##--- validate Shareplex is up and running fine ---## ##--- [oracle@mylab-01 log]$ $SP_BIN/sp_ctrl ******************************************************* * SharePlex for Oracle Command Utility * Copyright 2014 Dell, Inc. * ALL RIGHTS RESERVED. * Protected by U.S. Patents: 7,461,103 and 7,065,538 ******************************************************* sp_ctrl (mylab-01:2103)> show Process Source Target State PID ---------- ------------------------------------ ---------------------- -------------------- ------ Capture o.orppdb13_1 Running 1400 Read o.orppdb13_1 Running 1403 Import mylab-02 mylab-01 Running 1405 Post o.mypdb_01-mylab-02 o.orppdb13_1 Running 1402 Export mylab-01 mylab-02 Running 1404
As we can see, we are now able to start up the Shareplex instance without any issue and all the Shareplex processes are up and running fine.