In the late 80's, the word "Open" was applied to many different parts of the computing world to mean different things.
The most familiar to us today is "Open Source." The concept of open source existed quite a bit before the late '80's with the BSD license (one form of what is known as open source today). This source of "open" though is late '90s:
The label "open source" was adopted by a group of people in the free software movement at a strategy session held at Palo Alto, California, in reaction to Netscape's January 1998 announcement of a source code release for Navigator.
In February 1998, Raymond made the first public call to the free software community to adopt the new term.
(from Wikipedia on Open Source)
Thus, this is not the source of the Open in Open Server as it predates it by a decade. Which brings us to...
Lesser known and largely not important anymore (the Unix Wars are largely over) is The Open Group which was previously known as Open Software Foundation.
The OSF (and there's a name that might stick in some people's memory - OSF/1) was founded "to create an open standard for an implementation of the UNIX operating system."
Prior to OSF, each major company had a very different flavor of Unix. There was AIX, and AUX, and Ultrix, and IRIX, and Minix, and SunOS, and SCO, and Xenix, and BSD, and...
Each of these was tied to a specific hardware platform. One couldn't run the OS of one on another hardware platform. The definition of what was Unix and what standards one needed to compile code from Ultrix on an AIX machine were still very much in flux (Posix was set down in 1988).
With the late 80's, this lock in raised in prominence and some companies renamed their software to indicate that they were not locked in to particular hardware. This is about open standards and open systems - not open software.
OpenVMS (from Wikipedia) is one such naming:
In 1991 it was renamed again to OpenVMS to indicate its support for industry standards such as POSIX and Unix compatibility, and to drop the hardware connection as the port to DIGITAL's 64-bit Alpha RISC processor was in process.
This naming would mirror the naming of OpenServer by SCO, being released in 1989.