These are minor updates, and it's common to have hundreds of patches available after only a few months.   These updates are typically applied via the "zypper up" or "zypper up -t patch" command (although there are other options).

• Perform the updates during a defined maintenance window.  Shut off the GMS services and take a snapshot before applying, just in case GMS services don't load correctly after the update.
• It's generally frowned on to leave GMS services running while doing updates during full production. It's very likely the system will stop syncing mail. Yes, it does depend on which modules are being patched, but you don't really ever know 100% for sure what modules will cause GMS to stop working.
• When applying patches from the channel, you need to reboot the server immediately after the update. Do not try to keep the system online until later. It is likely that GMS won't be operational until you reboot.
• Generally speaking, I recommend NOT configuring automatic updates in SLES on a GMS server. This is because the updates could cause GMS to stop functioning and you won't know why.

The Problem with Leaving GMS running while running Channel Updates
You can see from the image/screen shot below that  I have left the gms services running while doing a SLES channel update.   At the end of the process, you can see that it's suggested to reboot the server, but you can also see that even though the gms services are running, the GroupWise sync agent in the dashboard is stopped.  At this point, services are down and phones will not sync.  That is why I generally recommend that you stop the gms services before channel patching, or you at least plan on rebooting the server or restarting the gms services immediately after the updates.

GroupWise Mobility Service Updates and Patches (From GMS 18.3.1 and higher)

GMS 24.1, 24.2, and 24.3 require SLES 15 SP4 or higher due to Python and Django library changes, and I strongly recommend going a step further. Please ensure that ALL SLES patches are installed from the SUSE channel prior to upgrading GMS to version 24.1, 24.2, or 24.3. 

The process to update GMS is the same for all GMS versions 18.3.1 and higher. Using the full media, the installation process updates all of the code and performs all necessary updates. The process is relatively painless in most cases. The process goes like this:

• Save the GMS media file (something like groupwise-mobility-service-xx.x-x86-64.iso) to a location on the Linux file system where GMS is running.
• Mount the ISO image to the file system.
  • mhcfs03 #  mount -o loop groupwise-mobility-service-xx.x-x86-64.iso /mnt
  • This command mounts the ISO CD image to the /mnt folder. That is where the installation will run from.
• Run the installer from the /mnt folder.
  • mhcfs03:/mnt #  ./install.sh
  • Accept the license agreement.
  • Generally, step through the various options until finished. Note the following:
    • Perform air gap install?  (yes/no)   [no]   (Generally unless you have very specific reason to do so, you do not need this)
    • You'll be prompted to shut down GroupWise Mobility and update the service. Go ahead and allow that.
      "The update process may take some time. During this process, the GroupWise Mobility Service will be shut down. Are you sure you want to update the GroupWise Mobility Service now? (yes/no):"
    • You'll be prompted to reset the log levels to INFO. I generally leave the logs alone but it's very situational.
      Reset log level for all GroupWise Mobility Services to Info? (yes/no) [yes]:no
    • You'll be prompted to enable anonymous info to be sent to Open Text. Pick your preference here.
      Enable anonymous information to be automatically sent to Open Text to improve your GroupWise mobile experience? (yes/no) [yes]:
• GMS will restart after the update.
• If you have any problems during the update and it aborts, you will have to resolve the issue and start over.

Major SLES Service Packs (ie SLES 15 SP5)

These can be done offline using an ISO image, but I generally use the "zypper migration" command to perform an online upgrade. This process works and it's my go-to method for service packs. Note the following considerations:

• Perform the updates during a defined maintenance window.  Shut off the GMS services and take a snapshot before applying, just in case GMS services don't load correctly after the update.
• Before applying a major service pack, I always apply all available updates from the channel. Follow the process as outlined above for how to navigate that process with GMS.
• If part of your plan is to upgrade GroupWise Mobility (in place upgrade), do that BEFORE applying the SLES Service Pack. There are many different reasons why I suggest this, the reasoning doesn't make a lot of sense, nor is it easy to explain. The main reason is that the newer GMS build will patch some bugs that could be problematic when doing a major SLES OS patch. If you are not planning on updating GMS, I would strongly suggest reconsidering that if you are not already on the latest GMS version (meaning GMS v23.4 or newer).

Functionality Test after Patches/Upgrades

Regardless of what type of patch or update was applied, these are the general checks you should perform afterwards to confirm that the system is functioning correctly.

• After the system boots up again, confirm that all GMS services are running at the command line. Do this via the "gms status" command:
  • mhcfs03#  gms status
• Login to the GMS dashboard (https://xx.xx.xx.xx:8120) and confirm that everything is functional as expected:
  • Ensure that the "GroupWise Sync Agent" and the "Device Sync Agent" both show as "running" on the Home page of the dashboard.
  • Make sure you can see statistics on the Dashboard. You should see numbers and status on each of the indicators. If it's blank, GMS has a problem.
  • Look at your user list and confirm that all users are listed as expected.
  • Review any Agent Alerts, especially RED ones and take any appropriate action.
  • Run an MCHECK General Health Check and address any issues reported.
• Check mail flow from devices. Confirm that you are able to send and receive mail correctly.
• Watch for any critical GMS alerts that show up in the GMS dashboard and handle them accordingly.  Note the date/time stamp of the alert, sometimes the alerts are old and unrelated to what you're doing at the moment.
• Specifically keep an eye on the GMS dashboard for this alert below. You may not see it immediately. Sometimes you won't see it for a couple hours. It just depends on what people are doing on their mobile devices.  The reason for this alert is that a SLES update overwrote one of the Python modules that GMS needs. GMS detected that this happened and corrects it by copying in the correct module. However, you must restart GMS for it to take effect.  With GMS 18.4,1 and higher this is very common to see after any SLES patches or updates are applied. Older versions of GMS did not generate this alert.

Contingency

If GMS is in a state of failure and it does not appear to be an easy or quick fix after any update, consider reverting to the snapshot unless you can easily identify the issue. It's likely a python related issue and there are some TIDs that discuss them. However, it's not always cut and dry.

Version Upgrades from older GMS to GMS 18.3.1 and higher

If you're running an older version of GMS such as GMS 14.x, 18.0, 18.1, 18.2, or 18.3.0, you cannot upgrade.  To get to GMS version 18.3.1 or higher, you must build a new server. There is no upgrade path and there is no migration path. Use the rest of this guide to help deploy a new server.

Server OS Selection

I generally use the latest version of SLES 15 available when installing GroupWise Mobility.  The only supported OS's for GroupWise Mobility at the moment are:

• SLES 15 SP4 (Worked well with GMS 18.3.x and 18.4.x.  Sometimes problematic with GMS 18.5)
• SLES 15 SP5 (Released on June 20, 2023. Works well with GMS 18.5 and higher. )

SLES 15 SP5 was released on June 20, 2023. I use SLES 15 SP5 for ALL GMS implementations at the moment. There is no reason to use any earlier version of SLES 15.

** UPGRADE NOTE **

• Can you upgrade an older version of GMS (14.x, 18.0, 18.1, 18.2, 18.3.0) to version 18.3.1?  No. Build a new server.
• Can you upgrade an older version of GMS (14.x, 18.0, 18.1, 18.2, 18.3.0) to any newer GMS version such as GMS 18.5 or GMS 23.4/24.1/24.2? No. Build a new server.
  
• Can you upgrade GMS version 18.3.1, GMS 18.4.x, or GMS 18.5.x to GMS 23.4, GMS 24.1, or GMS 24.2? Yes you should be able to as long as your SLES server is running a supported SLES version.

File System Selection and Partitioning

SLES 15 Defaults to a BTRFS.  Do not use this.  Instead, change the partitioning to EXT4. This is required per the documentation.

• Specify only EXT4 as the GMS server’s File System: SLES 15 SP2 and later default to BTRFS and you must manually change this to EXT4.

Additionally, note the following:

• The SLES 15 installation tries to partition the /home partition using half of the available disk space.  It's important to NOT use the defaults for anything. Otherwise, you will waste a massive amount of space since nothing on Mobility uses the /home folder.  Start the partitioning from scratch and define your own partitions.

Note About GMS 18.4 and BTRFS.

With GMS 18.4 there is now some support for the BTRFS file system.  Reference this statement in the GMS 18.4 release notes below.  However, I personally prefer to NOT use BTRFS and do not trust that it is stable for use in a production GMS system.

Separate Disk/Partition for GMS Data

If you utilize a 2nd partition for GMS data, it is important to understand the storage structure of GMS and where the data is kept. Per the documentation:

The largest consumers of disk space are the Mobility database (/var/lib/pgsql) and Mobility Service log files (/var/log/datasync). You might want to configure the Mobility server so that /var is on a separate partition to allow for convenient expansion. Another large consumer of disk space is attachment storage in the /var/lib/datasync/mobility/attachments directory.

Typically, with a 2nd disk, I mount it to the /var directory. This will put /var and all subfolders on the 2nd disk, including non-gms files located in /var.  This is typically fine.

Possible 3rd Disk for Logging

The one "possible" downside of mounting disk2 to /var is that both gms data and all server logging (including GMS logging) will be on the same disk. There is potential for the partition to fill up due to the amount of logging that could be done by GMS (especially if you're in diagnostic logging mode).  On very large systems I sometimes create a 3rd disk and mount it specifically to /var/log.  That will prevent any excessive logging from affecting the GMS services.

All about SSL Certificates

SSL Certificates Name and Location

SSL Certificates for GroupWise Mobility have been renamed and moved to a new location with GroupWise Mobility 18.3.1 and beyond:

This information is from the developers about each file and what it's purpose is:

3rd Party Trusted SSL Certificates

It's nearly impossible to use self-signed certificates anymore. Most phones require trusted certs in order to communicate. I am not going to go into detail about how to generate the certificates, but this shows you what you need to do to take your existing 3rd party certificates and configure them on a GMS server. This is what you'll need:

• The private key file (That you generated when you created the CSR to submit to your SSL vendor)
  • Also of importance is to note that GMS requires that the Private Key file does NOT have a password on it. This is the exact opposite of what is required by the GroupWise Agents.
• The server certificate (Generally .crt file) from your SSL vendor
• The SSL Vendor's intermediate SSL Certificate.

Combining All Certificate and Key Files into a single PEM file

GMS requires that all of your certificate files are combined into a single PEM file. You basically concatenate all files together using the sequence below. Please note that you will need to use the name of your specific files, not the samples I have used.

# cat passwordless.key > gms_mobility.pem   (Ensure that you are using the Key File that does NOT have a password on it)
# cat server.crt >> gms_mobility.pem   (This is the certificate provided by the SSL vendor)
# cat intermediate.crt >> gms_mobility.pem   (This is the Intermediate certificate provided by the SSL vendor)
# cp gms_mobility.pem /var/lib/datasync/
# gms restart

Reference TID from Micro Focus

This Micro Focus TID is not completely accurate for GMS 18.3.1+, but the general process is the same and it does go into more detail about the entire process. Note the file name and location changes that I have outlined.  How to configure Certificates from Trusted CA for Mobility (microfocus.com)

Starting / Stopping / Enabling / Disabling GroupWise Mobility

Everything below this point relates to the general troubleshooting of a GroupWise Mobility system.  Many issues you may experience are related to performance issues and resource exhaustion. The tips below are things I do to squeeze more performance out of my systems.

GMS Troubleshooting Tool: MCHECK

• Choose "4. Checks & Queries" from the menu
• This will bring you to the Checks & Queries Submenu where you will see the "General Heatlh Check Option"

MCheck (Version: 24.1) - Running as root
--------------------------------------
Checks & Queries
1. General Health Check
2. GW pending events by User (consumerevents)
3. Mobility pending events by User (syncevents)
4. Generate csv list of inactive users
0. Back

Select option:  

• Choose "1. General Health Check". This will start the process and generate the results as shown below:

===Running General Health Check===
--------------------------------------
Checking Mobility Services...      Passed
Checking Trusted Application...    Passed
Checking Required XMLs...          Passed
Checking XMLs...                   Passed
Checking PSQL Configuration...     Passed
Checking Proxy Configuration...    Passed
Checking Disk Space...             Passed
Checking Memory...                 Warning
Checking VMware-tools...           Passed
Checking Automatic Startup...      Passed
Checking Database Schema...        Passed
Checking Database Maintenance...   Passed
Checking Reference Count...        Failed
Checking Databases Integrity...    Failed
Checking Targets Table...          Passed
Checking Certificates...           Warning

The remaining checks may take a while.
Please be patient...

Checking Disk Read Speed...        Passed
Checking Server Date...            Passed
Checking RPMs...                   Passed
Checking Nightly Maintenance...    Passed

Press Enter to continue

MCHECK Log Files & Viewing Problems Found

The log files for MCHECK provide more detail about the issues that are reported. Since the checks cover a lot of different things, it's difficult to outline every possible scenario here. The best thing to do is look at the logs, view the detailed information about any specific issue, and determine how to proceed from there. If the issue is over your head or beyond your comfort level, don't hesitate to reach out.  The MCHECK log files are located here:

/opt/novell/datasync/tools/mcheck/logs

For the General Health Check log files, look for the file(s)  "generalHealthCheck_" + Date/Timestamp + ".log".   The file will be timestamped with the date and time the check was run.  After you run a general health check, just check the most recent log file.  Below is an example of the file listing of the log folder.

-rw-r--r-- 1 root root     0 Jul 18  2022 generalHealthCheck_2022-07-18T23:18:20.log
-rw-r--r-- 1 root root 27817 Nov  2  2022 generalHealthCheck_2022-11-02T05:40:05.log
-rw-r--r-- 1 root root 27516 Jan  4 22:13 generalHealthCheck_2024-01-04T22:12:23.log
-rw-r--r-- 1 root root 27694 Jan 14 19:45 generalHealthCheck_2024-01-14T19:44:44.log
-rw-r--r-- 1 root root 27671 Jan 14 19:48 generalHealthCheck_2024-01-14T19:47:59.log
-rw-r--r-- 1 root root 55336 Jan 14 20:21 generalHealthCheck_2024-01-14T20:19:53.log
-rw-r--r-- 1 root root     0 Jan 14 21:05 generalHealthCheck_2024-01-14T21:05:55.log
-rw-r--r-- 1 root root 27668 Jan 14 21:09 generalHealthCheck_2024-01-14T21:08:51.log

Memory Considerations

Symptoms of Low RAM in a GroupWise Mobility Server

When the RAM on a GMS server is depleted, you will typically experience the following symptoms:

• Extreme latency with receiving messages on your phone.
• Phones stop syncing system wide.
• A reboot of the server immediately resolves the issue because it frees up all the RAM and starts over with its usage.
• Your SWAP space will be consumed, which can cause even more problems, especially if your server is virtualized.

Diagnosing Low Memory

A good way to determine the memory status on your GMS server is by using the "free -m" linux command as shown below:

Analysis

You can see that this server has 8GB Ram (7682MB).  For reference, this system is a very small system with only 15 users on GroupWise Mobility.  If the customer were to tell me that they were having problems syncing phones, I would look at this and most likely increase the RAM to 10GB. Here are a few items I look at:

• Free:  This shows 124MB free out of 7682MB. That is pretty low free space.
• Swap: Out of a 2GB swap space, 14MB has been used.  The fact that swap space has been used at all tells me that it needs more RAM. In a perfect world, I want to avoid using swap space (That is a discussion for another day).
• Uptime:  Uptime stats for this server:  19:36:20 up 73 days 9:13, 1 user, load average: 0.36, 0.59, 0.61
  Because the server has been up for 73 days, I can be confident that the Memory usage has normalized and I'm not being influenced by a server reboot. Having a server that's been up for a while gives me more confidence in my memory stats and analysis.

CPU Considerations

Symptoms of an overburdened CPU in a GroupWise Mobility Server

Ensure that your RAM allocations are adequate before troubleshooting the processor. RAM accounts for most performance problems.  After you are confident you have enough RAM, note the following symptoms that could indicate the CPU allocations are too small.

• Very high CPU utilization at the console.
• Latency sending and receiving email on phones, system-wide.
• If virtualized, CPU alerts in VMWare.

The Linux "top" Command

The "top" command shows the most heavily utilized processes in real-time on a Linux server. Let's take a look at this screenshot below. It outlines the heaviest processes that are hitting the CPU.

As a point of reference, this server is a very busy server. It currently has 36GB Ram assigned, 6 CPUs, and 400 users provisioned.  I am troubleshooting high utilization and complaints from the customer.

CPU Count / "top" analysis

You can see from the results of this 'top' command that the majority of the processes are either "python3" or "postgres".   GroupWise Mobility is built using Python3, and it uses the Postgres database.  Therefore, you can safely say that GroupWise Mobility is hitting the CPU hard on this server. Increasing the CPU count from 6 CPUs to 8 CPUs could help with performance.

I recently experienced 2 very large GMS systems with chronic CPU utilization issues. They both had 8 Cores assigned in VMware.  Each server hosts approximately 500 users.  Begrudgingly, I increased the CPU's from 8 to 12 and noted an immediate and significant reduction in CPU usage. I do approach this carefully, but in this case the host servers had CPU's with 36 cores each and are very high end servers. So despite other considerations, I felt like it could handle it and it was the right decision.

NOTE: There are numerous articles available online related to the number of CPUs in a server, and research shows that more CPUs does not always mean better performance. In fact, it is argued that sometimes it could reduce performance.  So when it comes to increasing the number of CPU's in a GroupWise Mobility server, I approach this cautiously. It is usually something I would change after I have exhausted all other recommendations in this article.   It's critical to understand how CPU allocation works with VMWare before randomly adding CPU's to any VM.

Diagnostic Logging and Finding Problems

Helpful Tips about Logging

• If you're troubleshooting GMS, also ensure that your POA's are in Verbose logging mode so you can troubleshoot that side if needed.
• If restarting GMS clears up the problem temporarily, it's better to NOT restart it because that may interfere with a diagnosis. Instead, get the logs set the logs to debug, restart GMS, and then let it run a while.  Then when the problem comes back, you should have a lot of good diagnostic data to work with.
• Unless you are looking for specific historical data or trends, old log files are not that useful in most cases. Sometimes it is helpful to shut down GMS, remove (or move) all the log files in /var/log/datasync (and subfolders) and get a clean start for simplicity's sake. The amount of logging is overwhelming, so a clean start makes it easier to know you're looking at the right data.
• When troubleshooting specific sync issues, it's important to have some information handy relevant to the situation:
  • Date/Time of the issue or message that was sent but not synced or received.
  • Subject Line of messages that might not be syncing.
  • Email recipient for problematic message.
• One additional strategy you could use on a very busy system is to ensure that the gms log files are on a different disk partition than the gms data. This would balance the disk writes across different devices and theoretically minimize the hit on the primary gms data partition.  The way I have outlined above in the system build recommendations, you will accomplish this by having 2 or even 3 disks. If you've built a system with a different partitioning strategy, you may have to improvise and adjust based on how it is configured. If the system is already running, completely changing the partitioning strategy is not practical or possible.  But you could add another VMDK file (assuming VMware) to the server and partition it and mount it to the gms log location which is /var/log/datasync.   If you already have the main GMS data partitioned separately, and /var/log is set to the system partition, then this suggestion would be less applicable.

Note About Changing LOG Settings

• Changing the log level requires a GMS restart.  However, restarting GMS may temporarily fix any persistent problem you are having. Having to restart GMS to set diagnostic mode is problematic. I generally keep diagnostics logging on if I am having a problematic server, but then turn it off once the issue is resolved. Once you have changed to Diagnostic mode and restarted GMS, you may have to wait a while for the problem to come back depending on the situation.

This will display a current view of your relevant log file and show you how many threads are idle.  Here are outputs from two different servers for analysis.  In the first one, you can see that there is consistently only 1 of 21 threads available.  In the second one, there are consistently 12 of 20 threads available.

Server Example 1

2024-02-02 14:00:17.751 INFO [CP Server Thread-51] [DeviceInterface:302] [userID:] [eventID:] [objectID:] [] Idle threads = 1 / 21.
2024-02-02 14:00:18.396 INFO [CP Server Thread-40] [DeviceInterface:302] [userID:] [eventID:] [objectID:] [] Idle threads = 1 / 21.
2024-02-02 14:00:18.466 INFO [CP Server Thread-37] [DeviceInterface:302] [userID:] [eventID:] [objectID:] [] Idle threads = 1 / 21.
2024-02-02 14:00:34.627 INFO [CP Server Thread-66] [DeviceInterface:302] [userID:] [eventID:] [objectID:] [] Idle threads = 1 / 21.

Server Example 2

2024-02-02 14:04:02.361 INFO [CP Server Thread-56] [DeviceInterface:302] [userID:] [eventID:] [objectID:] [] Idle threads = 13 / 20.
2024-02-02 14:04:02.669 INFO [CP Server Thread-47] [DeviceInterface:302] [userID:] [eventID:] [objectID:] [] Idle threads = 12 / 20.
2024-02-02 14:04:03.198 INFO [CP Server Thread-43] [DeviceInterface:302] [userID:] [eventID:] [objectID:] [] Idle threads = 12 / 20.

There is a TID from Opentext that discusses this a little bit, and suggests that if you consistently have low threads, you may want to increase the threads. However they also recommend not exceeding 20 threads, which seems to be the predefined default.   You can view that TID here: https://support.microfocus.com/kb/doc.php?id=7014654

If you are experiencing sync issues, and you see that your threads are exhausted, here is what I would suggest:

• Monitor the threads for a while and see if they clear up or the situation improves on its own. Just because they're low does not mean there is an issue. But it is something to be aware of.
• Restart GMS and immediately monitor the threads again to see what kind of impact that makes.  Note the differences with the thread usage before and after the restart. If a situation existed where threads malfunctioned, a restart of GMS would immediately clear that up and allow the system to utilize all threads again, which will help performance.
• Note that thread info is not logged unless you have Diagnostic logging enabled in the GMS dashboard.

Persistent Device Sync Issues & Rebuilding GMS server

Preparation