UPLOAD_SYSTEM Threshold (UploadMaxNumberXML: 5000) exceeded with 5698 files) EM 12c Agent

I was facing  Issues on Oracle Enterprise Manager cloud control agent with Error "UPLOAD_SYSTEM Threshold (UploadMaxNumberXML: 5000) exceeded with 5698 files)" for command "emctl status agent"

I tried to stop/start agent but failed to communicate with OMS Server.

This Issue occurred because of incorrect configuration of open files. You need to properly configure the value for open files. I encountered this Issue on Oracle Solaris 10 OS. This Issue can also be encountered on other unix/linux flavour .

Output for command "emctl status agent":

[oemagent@oradb01][/oem/OEMAGENT/agent/agent_inst/bin]# ./emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.2.0
OMS Version       : (unknown)
Protocol Version  : 12.1.0.1.0
Agent Home        : /oem/OEMAGENT/agent/agent_inst
Agent Binaries    : /oem/OEMAGENT/agent/core/12.1.0.2.0
Agent Process ID  : 23697
Parent Process ID : 23413
Agent URL         : https://oradb01:3872/emd/main/
Repository URL    : https://oemserver1.orasol.com:4901/empbs/upload
Started at        : 2021-10-12 08:49:40
Started by user   : oemagent
Last Reload       : (none)
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload           : 5,719
Size of XML files pending upload(MB)         : 5.48
Available disk space on upload filesystem    : 50.79%
Collection Status                            : [COLLECTIONS_HALTED(
  UPLOAD_SYSTEM Threshold (UploadMaxNumberXML: 5000) exceeded with 5698 files)]
Heartbeat Status                             : Agent is blocked
Last attempted heartbeat to OMS              : 2021-10-12 09:40:33
Last successful heartbeat to OMS             : (none)
Next scheduled heartbeat to OMS              : 2021-10-12 09:43:33

---------------------------------------------------------------
Agent is Running and Ready

Solution:

1) Stop agent
2) Increase ulimit value
3) unblock agent
4) start agent
5) upload to OMS
6) check agent status


1) Stop agent

[oemagent@oradb01][/oem/OEMAGENT/agent/agent_inst/bin]# ./emctl stop agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Stopping agent .......... stopped.

2) Increase ulimit value

- Current Value of open files:

[oemagent@oradb01][/oem/OEMAGENT/agent/agent_inst/bin]# ulimit -a
core file size        (blocks, -c) unlimited
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
open files                    (-n) 256
pipe size          (512 bytes, -p) 10
stack size            (kbytes, -s) 8192
cpu time             (seconds, -t) unlimited
max user processes            (-u) 29995
virtual memory        (kbytes, -v) unlimited 

- Increase Value of open files:

[oemagent@oradb01][/oem/OEMAGENT/agent/agent_inst/bin]# ulimit -n 65536
[oemagent@oradb01][/oem/OEMAGENT/agent/agent_inst/bin]# ulimit -a
core file size        (blocks, -c) unlimited
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
open files                    (-n) 65536
pipe size          (512 bytes, -p) 10
stack size            (kbytes, -s) 8192
cpu time             (seconds, -t) unlimited
max user processes            (-u) 29995
virtual memory        (kbytes, -v) unlimited
[oemagent@oradb01][/oem/OEMAGENT/agent/agent_inst/bin]#

3) unblock agent

Navigate to Setup >> Manage clound control >> agent >> 

- Select block agent and unblock target

4) start agent

[oemagent@oradb01][/oem/OEMAGENT/agent/agent_inst/bin]# ./emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Starting agent ........................................................................................................................... started but not ready.

5) upload to OMS

[oemagent@oradb01][/oem/OEMAGENT/agent/agent_inst/bin]# ./emctl upload

6) check agent status 

[oemagent@oradb01][/oem/OEMAGENT/agent/agent_inst/bin]# ./emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.2.0
OMS Version       : 12.1.0.2.0
Protocol Version  : 12.1.0.1.0
Agent Home        : /oem/OEMAGENT/agent/agent_inst
Agent Binaries    : /oem/OEMAGENT/agent/core/12.1.0.2.0
Agent Process ID  : 12790
Parent Process ID : 12532
Agent URL         : https://oradb01:3872/emd/main/
Repository URL    : https://oemserver1.orasol.com:4901/empbs/upload
Started at        : 2021-10-12 11:24:50
Started by user   : oemagent
Last Reload       : (none)
Last successful upload                       : 2021-10-12 11:35:14
Last attempted upload                        : 2021-10-12 11:35:14
Total Megabytes of XML files uploaded so far : 0.03
Number of XML files pending upload           : 146
Size of XML files pending upload(MB)         : 0.17
Available disk space on upload filesystem    : 50.83%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2021-10-12 11:35:06
Last successful heartbeat to OMS             : 2021-10-12 11:35:06
Next scheduled heartbeat to OMS              : 2021-10-12 11:36:06

---------------------------------------------------------------
Agent is Running and Ready
[oemagent@oradb01][/oem/OEMAGENT/agent/agent_inst/bin]#

Now the agent is successfully able to communicate with OMS and upload XML.

Thanks for reading.

regards,

Zaheer