I have seen that term mostly used in an Oracle context. From the Oracle docs:
A database instance is a set of memory structures that manage database
files. A database is a set of physical files on disk created by the
CREATE DATABASE statement. The instance manages its associated data
and serves the users of the database.
So in that context, the term database means simply the files holding the schema and the actual data, while database instance refers to the server processes and the related memory structures which actually manage those files.
Microsoft, for example, uses the term "instance" for "a copy of the sqlservr.exe executable that runs as an operating system service", so the meaning is very similar, though in that context one instance can manage multiple databases.
I am not sure if other vendors use the terms exactly in one of the above meanings, but I would not be too surprised.