I'm interested in machine learning in relations to the stock market (predicting future values of stocks etc).
What topics would I need to learn - e.g. what branch of AI to look into etc? and what libraries/tools do I need?
Software Engineering Stack Exchange is a question and answer site for professionals, academics, and students working within the systems development life cycle. It only takes a minute to sign up.Sign up to join this community
Machine learning is a whole field in Computer Science, which is quite different from Artificial Intelligence.
Stanford provides a free online class for machine learning which will get you started.
You need to understand that machine learning alone will most probably not be sufficient to create good trading strategies; you need to understand well how the markets work and most systematic strategies require large enough capital to minimize the transaction costs (otherwise the strategy is not profitable, see implementation shortfall).
To be able to understand and apply machine learning, you'll also need a good mathematical background (the better it is, the deeper you can go with the algorithms). I would recommend Information Theory, Inference and Learning Algorithms (MacKay) as a reference book for the field.
I'm working closely with machine learning experts, and I see that it requires a lot of expertise and math knowledge to apply ML correctly, You have not stated what your current background is. If this is mathematics, phyics, or aome other math heavy study, you could probably learn a lot online and from books. If not, to get up to the level to use ML as more than a black box, consider doing a masters degree which incorporates ML. People in finance not knowing statistics/ML, but still applying it because it gets them something that impresses their boss/client, but also could lead them to providing advive which results in the loss of millions of dollars.
I realize that doing a masters degree of two years is quite an investment. However, I think to really use ML sensibly, this could be a requirement.