Problem Summary
When a target OEM agent is put under blackout from OEM console (GUI), the only valid way to remove the blackout is to use OEM console. However, there might be situations when we might not have access to the GUI or the GUI is not working for us or we simply want to be able to remove any blackout from command line. In that case, we have the option to remove the target blackout using the command line tool EMCTL
However, when a blackout is set from Console (GUI), the command utility (emctl) would not allow it to be removed and we can see a error message similar to the following when we try to remove the blackout using emctl.
Blackout stop Error : Blackout "Blackout-Aug 13 2015 2:25:18 AM" exists, but was created by the console.
The error message clearly indicates that, the blackout does exist. However, it can not be removed as that was created by the console (GUI).
In this post, I will demonstrate few simple tricks that can be used to remove this kind of blackout (the one’s set from the GUI console) using the command line tool (emctl).
Demonstration
Lets say, we have the following blackout (“Blackout-Aug 13 2015 2:25:18 AM”) exist on the target server and we want to remove it using command line tool emctl.
oracle@mylab01:~> /app/oracle/product/Agent12c/core/12.1.0.4.0/bin/emctl status blackout
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Blackoutname = Blackout-Aug 13 2015 2:25:18 AM
Targets = (mylab01.oraclebuffer.com:host,orplab01.oraclebuffer.com:oracle_database,mylab01.oraclebuffer.com:3872:oracle_emd,OraDb11g_home1_1_mylab01:oracle_home,sbin12c1_4_mylab01:oracle_home,agent10g_2_mylab01:oracle_home,agent12c1_3_mylab01.oraclebuffer.com:oracle_home,LISTENER_mylab01_mylab01.oraclebuffer.com:oracle_listener,)
Time = ({2015-08-13|2:26:37|3600 Sec,|} )
Expired = False
Now, lets try to remove that blackout using emctl command line tool.
oracle@mylab01:~> /app/oracle/product/Agent12c/core/12.1.0.4.0/bin/emctl stop blackout "Blackout-Aug 13 2015 2:25:18 AM" Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Blackout stop Error : Blackout "Blackout-Aug 13 2015 2:25:18 AM" exists, but was created by the console.
We are not allowed to remove the blackout using emctl as that blackout was set from console (GUI).
Okay, Here comes the trick in action. Let me give you the foundation on which the trick is based upon.
Whenever, a OEM blackout is set either from OEM console or from command line (emctl), an entry for that blackout is placed in a file called blackouts.xml located under $AGENT_BASE_DIR/sysman/emd directory on the target server.
oracle@mylab01:~> cd /app/oracle/product/Agent12c/sysman/emd/ oracle@mylab01:/app/oracle/product/Agent12c/sysman/emd> ls -lrt blackouts.xml -rw-r----- 1 oracle oinstall 914 2015-08-13 02:26 blackouts.xml
If we take a look at the content of the blackouts.xml file, we could see it has entry for the blackout that we have seen earlier.
The simplest trick would be to just delete all the contents of this blackouts.xml file and reload the agent as shown below.
---- ---- Removed all contents from blackouts.xml ---- We can use "touch blackouts.xml" as a shorthand to get rid of all contents ---- oracle@mylab01:/app/oracle/product/Agent12c/sysman/emd> cat blackouts.xml
Lets check the status of blackout after removing the contents of blackouts.xml file
---- ---- Reload agent ---- oracle@mylab01:~> /app/oracle/product/Agent12c/core/12.1.0.4.0/bin/emctl reload agent Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. --------------------------------------------------------------- EMD reload completed successfully ---- ---- Check blackout status ---- oracle@mylab01:~> /app/oracle/product/Agent12c/core/12.1.0.4.0/bin/emctl status blackout Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. No Blackout registered.
The blackout no longer exists. Simple isn’t it?
But wait… What if we have multiple blackouts and we want to just remove one of them which was set from console (GUI)? We can’t just remove all the contents from the blackouts.xml file as that would also remove the blackout that we do not intend to remove.
Well, we have another alternative here.
Lets say, we have two blackouts namely “Blackout-Aug 13 2015 2:25:18 AM” and “Blackout-Aug 13 2015 3:51:10 AM” as shown in the following blackouts.xml file.
Lets say we just want to remove the blackout with name “Blackout-Aug 13 2015 2:25:18 AM”. In this case, we can not delete all the contents from blackouts.xml file as we did in the previous case. Here is another trick that we can apply here.
If we observe the contents in this file, we can see the clause CREATED_USING=”CONSOLE”. This is the check, which is preventing the command line tool (emctl) from removing the blackout.
The valid values for CREATED_USING clause are:
- CONSOLE: Indicates, blackout was set from console (GUI)
- COMMAND_LINE: Indicates, blackout was set from command line tool (emctl)
When the command is issued to remove the blackout, it checks this flag to identify from where the blackout was set. If, it finds the blackout was set from console (GUI) and the “stop blackout” command was initiated from command line tool (emctl), we would not be allowed to remove that blackout from command line.
What we can do here is, just edit the CREATED_USING flag in blackouts.xml file to change the value from CONSOLE to COMMAND_LINE and then initiate the “stop blackout” command.
Here is, how the blackouts.xml file would look like after editing the CREATED_USING for the specific blackout.
Lets try to remove the blackout again using the command line tool (emctl)
oracle@mylab01:~> /app/oracle/product/Agent12c/core/12.1.0.4.0/bin/emctl stop blackout "Blackout-Aug 13 2015 2:25:18 AM" Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Blackout Blackout-Aug 13 2015 2:25:18 AM stopped successfully EMD reload completed successfully
We are now allowed to remove the blackout using command line tool (emctl) as we had changed the CREATED_USING flag to indicate the blackout was set from COMMAND_LINE and not from CONSOLE.
Lets verify, what is the status of the current blackout settings for the target agent
oracle@mylab01:~> /app/oracle/product/Agent12c/core/12.1.0.4.0/bin/emctl status blackout
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Blackoutname = Blackout-Aug 13 2015 3:51:10 AM
Targets = (mylab01.oraclebuffer.com:host,orplab01.oraclebuffer.com:oracle_database,)
Time = ({2015-08-13|3:53:51|3600 Sec,|} )
Expired = False
As expected, the blackout “Blackout-Aug 13 2015 2:25:18 AM” no longer exists and we are left with the blackout “Blackout-Aug 13 2015 3:51:10 AM” which we intended to keep. Here is how the blackouts.xml file looks after removing the blackout.
Note: We could have also deleted the entries related to the blackout that we wanted to remove (like the way, we did in the first case) from blackouts.xml file. However, my intention was to demonstrate the second trick and hence I haven’t used a similar approach.
Conclusion
We have seen two different methods (tricks) that we can use to remove a blackout from command line utility (emctl) when the blackout was set through console (GUI).