I'm developing a REST API using Nodejs along with MySQL as the database.Is it okay to keep open the MySQL database connection open through out the Application or open and close connection query-wise ? Is there a best practice or a standard methodology for handling situations like this.
For fast response time and high throughput, it's actually best to keep database connections open and reuse them for subsequent requests. Most database frameworks offer some kind of connection pool mechanism where a request handler can get a database connection for its work and return it to the pool afterwards.
Note that the answer referenced by @RomanBoegli mentions connection pooling, too, so we're not in contradiction here. You should just mentally replace opening and closing of the database connection with getting one and returning it to the pool, and should keep it for as short as possible.
Consider the database as your data treasury. Hence, it is in your interest to keep traffic through the open door to a minimum. Therefore I suggest you to open the DB connection as late as possible, ideally just right before the query execution, and closing it right after it again. See this answer for concrete implementation.