This might be a partial answer but I think there's some info that might be useful in it, so...
First, I'll try to answer the questions:
- "Would you all recommend sticking with the above plan or if you suggest doing something different, what would it be?"
I would actually try to convince the stakeholders (money payers) that using an enterprise software would be in their best interest 'cause you're describing a very hard problem and you'll likely spend more money developing it than something like New Relic costs.
That said, the implementation totally changes based on what you're currently doing for releases and if you control the application code you're trying to monitor or if it's some 3rd party plugin you want to get data about too. I'm going to guess that the app is your company's and that your might have some CICD but it's not 100%.
If that's close enough, I think it would be easy enough to stand up a Kafka cluster or three and log everything you can get your hands on into them. Maybe one is for RESTFUL calls and one is for app-level errors or whatever you decide. This could stand as your centralized logging system.
Since you control the application code, you can log pretty much anything you like by asking the developers to add more logging, where you want. That's not usually a favorite request but if it makes sense and especially if higher ups back the idea, you can probably get all the logging you want to get. That will tell you about RESTful calls, app errors and things like that but you'll have to go deeper to get latency reporting and info on slow SQL queries and all that. I believe you can get some good info out of some free Nagios logging too. At a minimum, you should be able to get stats about how the hosts are doing; e.g. Memory available, CPU consumed, Disk space, etc.
You can then have something consuming all these messages and tie in some pretty useful automations, based on the state of your system and the actions coming in. "Oh look, everyone is buying all the X-product! Let's make a promotion to capitolize on it!" or "Oh look, all of our apps are throwing 500s! Let's go hide somewhere until this all blows over!"
I'd say you should use something like AWS's SQS because you don't have to worry about managing the Kafka/Zookeeper cluster on top of managing your application(s) code. SQS is pennies for millions of messages but if you're going 100% open source, Kafka is a solid choice, in my humble opinion.
Choosing NoSQL...that one's tough here. I think it's probably the right choice but maybe, if you can, hold off on deciding about that level in your system until you find out how you want to use that data, in other words, make sure the use case fits NoSQL before you commit. Switching and keeping any history would be tricky.
- "What I lack knowledge in is "BigData". What exactly is BigData in a nutshell? Can we apply it for our needs? Are there open source solutions for BigData?"
I'm no PHD but I dabble and I work with those teams, daily. With that history, I'll try to explain in my words. Big Data is a blanket term you can use to talk about problems and solutions with, when the problem you're trying to solve has a grip of data tied to it by nature. It's usually not helpful to work from this mentality when your data is still in the GB range but I'm sure there's plenty of use cases where it makes sense.
Anyways, it works like this; Say there's this thing that we all do, like use the internet. Pretend, if you will, that there was some way to collect data about our internet usage, like what sites we visit and what search term we used to get to that site. That's a looooot of data and when you think of it and use it as kind of a resource, you get a lot of insights and benefits out of it but it's also really hard to sift through all that data and reason about it. In this example, the data is all our internet usage. The problem is how to sale crap to us, using google adwords or whatever, and the "Big Data" in this is that we're considering all this data and how we can use it. A big part of "Big Data" is also the infrastructure used to run it. It's not trivial so it usually gets a lot of attention.
There's lots of open source tools you can use for using your data.
If your apps were to use any tools, Spark comes to mind. You could set up streaming from your logging message system to modify all the incoming data so that it's easier to use for humans and machines. I guess if you get really ambitious, you could try to implement some machine learning based on all your data. You'll have to come up with the problems you want to solve first, though.
- "Would using BigData or any other solution help with faster analysis/querying of data for our analytics dashboard?"
Yes. That's the whole point of it all. The problem is, you have to hire a bunch of expensive people to implement it. Your best bet is to tackle that hurdle when you get to it. If you can create your logging system and the dashboard, you'll be worlds better than you are now. No need to build an F1 car when you just need a tractor, ya know? And if I were to continue the F1 car and tractor metaphor, you can think of it like this -> The tractor that you're going to build now is creating the paved surface for your BigData F1 car to do impressive things with. You'll need all that data your logging system is going to collect before you can use the data to reason about, if that makes sense.
- "Is there an out of the box open source solution for dashboards(UI) so that we don't have to build it from scratch?"
I don't know of one but that doesn't mean it doesn't exist. Building the dashboard can be as big or small a project as you make it. Honestly though, I think this is the easiest part of the work ahead of you. Enjoy, if you build it yourself! It should be fun.
- "If sticking with NoSQL is a good plan, what NoSQL DB would you recommend? MongoDB? Cassandra?"
My opinion here is just to use a cloud tool and let them decide the engine. I doubt, from what I'm hearing, that you'll be taxing any graph database on tough queries. Your issue is most likely going to be from reads and writes, so maybe figure out how much you think you'll be pushing into the DB and how you'll be querying that data. It should lead you to whether or not NoSQL is a better fit and during your reading, I bet you'll stumble on some article that makes up your mind on which engine you chose.
It's an interesting problem and it sounds really fun to work on.
Good luck and I hope that helps, at least a little!