GroupWise 18.5 Released (Potential Issues)

GroupWise 18.5 officially released

GroupWise 18.5 was released 5/23/2023. I have run into the following issues as noted below:

Starting and Stopping the Agents on Linux

The standard "rcgrpwise stop/status/start" method has changed slightly and it appears that they're moving the control mechanism more towards the standard systemctl service structure.  In doing so I have observed that sometimes when you want an agent to stop, such as the post office agent, it doesn't actually stop the process.  For example, if you are trying to stop your post office agent, and the command you type is "rcgrpwise stop phoenixpo.mydomain", it's possible that even though it shows that it has stopped, you may find that the process is still actually running.  Because of this you may want to confirm that no GroupWise processes are running by using the "ps" command. You can do that like this:

• ps aux|grep gwpoa (This will show you any instances of a GroupWise Post Office agent, aka POA, that is running)
• ps aux|grep gwmta (This will show you any instances of a GroupWise Message Transfer Agent, aka MTA, that is running)
• ps aux|grep gwia (This will show you any instances of a GroupWise Internet Agent, aka GWIA that is running).

If you do find that any processes are still running even after stopping them with the rcgrpwise stop command, issue the command "kill -9" followed by the process ID identified in one of the commands above.

Potential SSL Private Key File Issue

I have run into an issue that could affect GroupWise systems on a widespread basis due to the nature of the issue.  It has to do with the SSL Certificate and Key Pair, specifically 3rd party certificates from any SSL vendor like Godaddy, Digicert, or Verisign. And it's not an issue with the actual Certificate file. The issue is with the Private Key file and how it encodes the password.

This issue does not occur with the Self Signed certificates created by the GroupWise Admin Console.

In working very closely with the GroupWise developers, we have been able to confirm the exact scenario that causes the problem, and know the exact steps required to fix it.  It is not technically a bug or defect in the GroupWise code. The issue is actually that a standard encoding method used with the "openssl" tool to encode the password into the Private Key file has been deprecated. Any private key files encoded using that deprecated method will no longer work with GroupWise 18.5.

If your GroupWise system is configured with end-to-end SSL (It should be, that is the best practice), and if you encoded the password into the private key file yourself, it's very possible that you will run into this problem depending on the syntax used to create the Private Key file.

How This Will Affect You

If you are affected by this Private Key file issue, anything that uses the Private Key will fail because the loading of the certificate/private key will be unsuccessful. Anything that depends on the SSL connection will no longer be able to connect. Essentially you will experience a system wide outage. The following is a list of connection points that could be affected, depending on your own specific GroupWise configurations:

• Agent to Agent communication failure. MTA to POA, MTA to GWIA, MTA to MTA.  Essentially a complete mail-flow failure.
• Client to Post Office communication failure. Inability to login with the GroupWise client..
• GroupWise Mobility connection failure to the POA via SOAP.
• GWWEB connection failure to the POA via SOAP.
• Inability to access HTTP agent pages including POA, MTA, GWIA, and DVA.
• IMAP or POP3 connections to GWIA or POA.
• Any other app or service that connects to one of the GroupWise agents securely.
• During an upgrade to groupWise 18.5, these breaks in communication will prevent necessary changes from being propogated throughout the system.

Symptoms of this problem as follows:

Other than the specific items listed above, you can confirm that you are likely having this issue from the following indicators:

• a red Dot is displayed (instead of Green) on all affected GroupWise Agents in the GroupWise Admin Page.
• Log file of the POA/MTA agents show the following "SSL Configuration has been disabled because of failure in setting up SSL rc= [891D]
• HTTPS agent page won't load at all and results in connection error.
• HTTP Agent page shows "Web Console Access has been disabled due to error in loading SSL Certificate: [8209].
• The error 8209 error is typically a "file not found" error, however in this case it appears that it's just not able to read the file format correctly, even though it exists.
• The 891D error is documented as a "Bad SSL private key or password".

Background

Understanding the problem requires that you have a decent understanding of how you got there in the first place. Note the following considerations:

1. GroupWise agents configured for SSL require that you have a certificate file and associated private key file, and that your key file is encoded with a password.
2. While you obtain the actual certificate from the 3rd party vendor, the private key is controlled entirely by you. The vendor never sees or touches the key file.
3. The Private Key file is typically created at the same time you create the Certificate Signing Request (CSR) that you send to the 3rd party vendor. The commands to create the CSR and Key File are typically done at the Linux command line with the openssl command.
4. In many cases, and depending on the syntax used, your original Private Key file will not have a password encoded into it. You are then required to encode a password into the Private Key in order for it to work with the GroupWise agents.
5. The process/syntax used to encode the password into the Private Key file is where the problem occurs.
6. A standard syntax that I would have typically used to take a Private Key file with no password, and then encode a password into the Private Key was as follows (this is a widely standard and documented syntax):

   • openssl rsa -in nopassword.key -des -out newkeywithpassword.key    (This syntax no longer works with GroupWise 18.5)

7. The resulting file, in this case "newkeywithpassword.key", is a file in PKCS#5 format.  The password is encoded with the "des"  encryption algorithm.  This works fine with GroupWise 18.4 and older. However, with GroupWise 18.5, the "des" algorithm is depricated and no longer works.

What you need to know:

The "des" encryption algorithm used to encode the password into the Private Key file is deprecated and will no longer work with GroupWise 18.5.  When generating the password encoded Private Key file, you must use a stronger method. A method that will work is "des3" (Triple DES).  When using "des3", I am able to load SSL correctly on all of my agents correctly.

The following syntax will encode a password into the Private Key file using des3 encryption:

• openssl rsa -in nopassword.key -des3 -out withdes3pw.key

If the 'nopassword" key file is not available, strip the password from the existing key file via this command first:

• openssl rsa -in withpassword.key -out nopassword.key

Note the following:

• Once you have the new Private Key file, you must then assign the correct Certificate and Private Key to your GroupWise Agents.
• The GroupWise Agents need to be restarted. The agents don't just reload the SSL certs automatically like they do with other administrative tasks. It requires that they be restarted manually.

Important Note:

There is also a "note" in the admin console help file that states the following, which I have found to be very important here as well:

NOTE: If you are replacing existing certificate and key files, delete the entries in both the SSL Certificate File and SSL Key File fields before generating the new certificates.

Further Analysis:

If you look at your existing Private Key file, you may be able to determine if it is encoded in a format that will work or not.  The Encoding method is listed in the header at the top of the file.  I have provided examples of a Private Key file encoded with "des" and one encoded with "des3".

Non-Working Private Key File (DES, Will not work with GroupWise 18.5)

In the header below, you can see the text "DEK-Info: DES-CBC".   I have found that DES-CBC indicates standard DES encoding. This Private Key File will not work with GroupWise 18.5.

-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-CBC,26B3F7AF4CCF8A30

f+I5pNiaFyeS3o4jkBS+ZOJXxhiuVP76miKZxTZ32eLk9tF4ErbKMN3cKEX2VsUh
......truncated.....
uSDrziBL2OnfbXEpPrlkJiisqQco+Xa5raJ8CZPnabYTWsqviJ9zoJQX1hDONGr6
-----END RSA PRIVATE KEY-----

Working Private Key File (DES3 or Triple DES, Required for use with GroupWise 18.5)

In the header below, you can see the text "DEK-Info: DES-EDE3-CBC".   I have found that DES-EDE3-CBC indicates Triple DES encoding, and this will work perfectly with GroupWise 18.5.

-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,AAC421BB1E9F5F71

q39a5sxkcWjZSE3ugj5gYe6tn0uGgarAxhdFyV5IxEpzK06ORW3AT3E1yLxdCQNj
..........truncated................
uY7MbBxtiToVdy2gne/6uda5nQJnt76HxAjjZvJmrl1aUVlDvNZEPpsS5W/mmMTR
-----END RSA PRIVATE KEY-----