IP Office Runaway Java Issue on R12

IP Office Runaway Java Issue on R12

We experienced some kind of bug that caused a Java process to spiral out of control. It was first noticed on a Select Server Edition upgrade from R11.0 to R11.1 and happened again going to R12.0. The R12 iteration was a completely clean AWS installation with a CSV upload of the config; a Voicemail Pro backup was also restored.

Noticeable symptoms while Java was slamming the overprovisioned CPU included:

• Web Manager was not accessible
• Hundreds of J1xx phones were slow to register – many temporarily ended up in the IP blacklist
• Dialing from J1xx and 9508 phones was laggy

Screenshot of the ‘top’ command from CLI

Ticket 1-22267262412 was opened with Avaya support and they ran a number of commands to clear the Web Manager cache such as:

1. Open a Linux console, login with “root” user
2. Type command “service WebManager stop”
3. Type command “service apache-tomcat stop”
4. Go to “/opt/WebManager/diskcache” and delete the contents.
5. Type command “service apache-tomcat start”
6. Type command “service WebManager start”

They also logged into port 7071 and performed Updates>Clear Local Cache.

Return of the IP Office Runaway Java Issue

The problems went away for a little while until we made a change in hard client Manager. We added a User Rights group common to both systems but immediately noticed the new User Rights Group was not showing up in Web Manager. A while later, Web Manager was down again and Java was running away. As a stab in the dark, we deleted the newly created User Rights Group using hard client Manager and behold: the Java process calmed down instantly!

Make of this what you will…

Screenshot of Web Manager while issue occurs

On a somewhat related front, we also had major issues trying to restore a ~2,000 user manager config to a clean R12 installation as an upgrade path. The typical process would be to take the R11.1 config and restore it to a new R12 instance. However, when we tried to do so there were a few cryptic errors and the whole thing crashed & burned per screenshots below. These problems may have been due to something we found later – identical names for a 8-10 users.