Is it called "Low Level" in what is it designed for (a compiler infrastructure) or because it is works on a "lower level" than other tools?
LLVM is lower level than typical VMs like JVM and CLR. For example, while it has hooks for a garbage collector, it does not provide a garbage collector itself.
Likewise, the JVM has a built-in JIT compiler (except in really ancient versions). LLVM has some JIT compilers for LLVM IR, but it's still up to the developer to hook things together and actually JIT the code.
When the JVM encounters an unresolved external, it goes out and finds the right class to satisfy it, and knows how to look for .class files directly in the file system and in .jar files1. LLVM's JIT compilers have hooks where you can decide how things like that are handled. As you'd expect, some people have written some default versions, so it can do things roughly on the same order as the JVM can -- but you're also free to ignore those and do things differently if you choose.
Simply put, if you're developing a compiler (or something on that order) it has a lot of tools to make your life easy. Instead of worrying much about optimization, you can do roughly the simplest translation you can manage from your source code to LLVM IR, and then use the LLVM libraries to manage optimizing, JITing, linking, etc. Nonetheless, they are libraries -- it provides some really useful functions so you don't have to deal with all the details, but they're still functions and you're still writing code to invoke them. It's not a finished product, just useful tools for building products relatively quickly and easily.
1 Technically, not all of this is built into the JVM proper. It specifies what's usually called the primordial class loader as part of the JVM proper, and then there are user class loaders specified in java.util.ClassLoader that handle other things. Some class loaders are included by default, and if you want to badly enough, you can supplement those by defining your own.
It is low level because it is designed so existing or future virtual machines (the JVM) can use it as the core of their implementation.
See it this way: The virtual machines of Java and Python are very portable because they are written in standard C, and rely a lot on their own libraries. The intent of the LLVM is to provide an infrastructure so that it is easy to port any virtual machine to a platform that already supports LLVM.
LLVM is offering support for static, and JIT compilation, and in is designed in a way that it could run in trusted operating system space. That means that, in theory, running a virtual machine on top of LLVM should mean much less work, and produce something much more fast and efficient. In theory.
Then LLVM, being lower level, should be easier to port to different OS and hardware architectures.
There are large savings to be had if language implementors could write to a single, easy-to-port low-level platform. The best known VMs are midway between the language they serve and the operating system, and they have to implement their own intermediate representations, and JITC.
The "low level" part of the LLVM name refers to the level of the virtual instruction set used. The intermediate language (IR) that LLVM uses is close to the level of machine code although it is architecture agnostic and general.
The bytecode of the JVM and CLR on the other hand is fairly high level in that it has instructions that are on a higher abstraction layer. They are both object-oriented stack-based assembly languages. For example the JVM has the instruction invokevirtual, which is an instruction that has to know about the specific object model of the Java language.
It is certainly lower level than any of the language-specific VMs and the unified VMs like JVM and CLR. Its design is close to the low level intermediate representation in GCC (GIMPLE) and similar compilers. There is no default GC, no specific high level type system is enforced, no alignment is assumed (it must be specified explicitly), integer and floating point data types are explicit (and platform-dependant), and the lowest level of all - a decent pointer arithmetics is available.