The use of the term signature in the RFC is analogous to a digital signature in asymmetric cryptography. In asymmetric cryptography if the sender encrypts a message with their private key, anyone who has the message can decrypt it with the sender's public key. So the goal with the term signature isn't to keep a message secret, but to verify the integrity/sender of the message, that is hasn't been altered.
In the case of JWTs the sending system is both the creator and consumer of the message (see diagram below), and the goal is to make sure the token passed to the user wasn't tampered with (e.g. given elevated privileges).
And as @Robert mentioned, the JWTs can/should still be encrypted with TLS.
Here is a good explanation of JWTs and signatures from which the below image is sourced. 5 Easy Steps to Understanding JSON Web Tokens (JWT)