The answer for such question cannot be short, so let's start with the terms:
- frontend - software subsystem directly interacting with user. In consumer applications term "frontend" usually refers to an application running in a web browser or to a mobile application.
- backend - software subsystems that do not interact with user directly. As a backend software developers usually consider the application server or microservice plus a database and related services (e.g. message queue or distributed cache).
- JavaEE - is no longer known as such. Modern name for this technology stack is JakartaEE (https://www.jakarta.ee) This is a group of specifications of various APIs in Java, which greatly vary in popularity and use: pure JEE project today is not a common thing - according to many surveys, only Servlet and JPA APIs are widely used as part of Spring/Hibernate combo.
- JSF - more often hated than loved UI framework that is in steady decline for last 5 years due to limitations of its architecture. I would not recommend to start a new project on JSF and look for a more mainstream solution.
- A layer in software architecture - logically separated part of the application that performs a specific role. In Java applications layers are typically represented by specific packages or even jar modules. One of the examples of layers is persistence layer, which represents an object-oriented abstraction of a database. By definition layer is not a separate application, so your option 1 does not make sense as a solution to interact with database. Hibernate as a database layer however does make sense. Please note, that Hibernate is a popular implementation of JPA API from JEE, so if you are already on JEE stack, you either already use it or don't need it, because you use some other implementation (e.g. EclipseLink).
Now, let's talk about the architecture in general. Your option 1 is a modern and the most popular solution for the whole application. Spring is a framework of choice for majority of Java developers according to recent surveys (including SO), with more and more of them choosing Spring Boot as a complete technology stack and alternative to JEE. Hibernate is one of the popular options for persistence there, but there are others - check the Spring Boot reference guide (https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/).
Building a REST API on the backend (Spring Boot/Hibernate), that can be consumed by your frontend is one of the easiest ways to develop interactive web applications today, but of course not the only one - choice of the particular solution should always be based on business requirements. For web applications displayed in browser the frontend is usually written on JavaScript or related languages (e.g. TypeScript) with one of the many frameworks that exist there (check Angular and ReactJS).
The considerations for option 1 that you mention:
- JSON mapping is pretty straightforward. In the frontend it's automatic, in the backend easily implemented with the libraries like Jackson - comes out of the box with Spring Boot.
- You will have just one server, either providing REST API to JS-based frontend or implementing server-side rendering (in JEE via JSF/JSTL, in Spring Boot - with Thymeleaf).
- Security is a business requirement. If you need to protect some data, implementation of it must be present. With Spring you can use Spring Security with various options.