A/B tests can be a powerful tool to test a hypothesis about your user interface. A hypothesis is a falsifiable proposition, e.g. “including a rating in each book tile increases engagement”. You can then implement an experiment and put your hypothesis to the test. As this is a statistical experiment, you need to decide on certain parameters such as confidence intervals for the rejection/acceptance of the hypothesis, sampling method, and sampling size. Once the experiment has completed, you can perform an analysis and get one of three results: the hypothesis was accepted, rejected, or is inconclusive.
You suggested a sampling method that runs each variant consecutively for a week. This is not a very good sampling method, because you have added another parameter “time” to the experiment. Maybe the engagement rate is time-dependent? With such an experiment design you cannot tell. Also, some users may revisit the site during the experiment and experience both variants. The order in which the variants are presented may be relevant, and adds another dimension to the experiment which you did not consider in the design – you might get different results depending on whether you run A or B first.
To avoid such sampling problems: first calculate the sample size that you need for your desired statistical power. If your expected effects are small, you will need larger samples to show a conclusive result. When you compare the necessary sample size with your expected number of users, you can find out how long you have to run your experiment for. Ideally, run the experiment for an integer number of weeks so that the effects of weekdays can be eliminated (especially weekdays vs. weekends). During the experiment you can then randomly sort your users 50/50 into the A/B groups, and keep them in that group for the duration of the experiment.
For some changes you don't want to expose half your users to a new idea, just to take it away later. If you have enough users, it might make sense to run the experiment on a fraction of your users, or to run a pilot study before the actual experiment with a tiny sample of users.
Even when you get a conclusive result, your user base may not be uniform. A change that improves your user interface on average may still worsen it for some. So check for that first – a confirmed hypothesis is not a “go” decision!
All of that is a lot of effort. Designing a good experiment is hard. Getting a good sample might take a long time. The experiment may have ethical or legal implications that must be cleared first. The statistics might get non-trivial.
This effort is often not worth it. Once you have a steady stream of users and have implemented the necessary A/B test infrastructure, being able to have this data is nice! But before that, they might be a massive waste of your time.
There are many simpler ways to get feedback on your user interface.
Trust your guts. A lot of design is common sense and/or doesn't matter. This is how you'll handle most decisions anyway.
Ask other people for quick feedback, but ask a useful question. E.g. “Do you think showing a rating like this will make fiction readers more curios about the book?” (Though that borders on a leading question.)
Create personas. A persona is a fictional model user with a set of goals and desires. Validate your design by checking whether this design helps the personas achieve their goal. Tip: include personas with limitations, e.g. colour blindness or slow internet connections.
Make feedback easy, then just make the change. If it was a bad change, users will complain. Although users will always complain about any change. Openly communicating about your intentions (e.g. blogging) can give you early feedback from power users.
Ask questionnaires. Such surveys are bad at producing statistically reliable data because of sampling problems, but can still produce valuable insights from your users. E.g. you could ask which factors make them interested in a book, and offer various suggestions (cover, author, blurb, average user rating, critic ratings, …). Designing good surveys is more difficult than designing an A/B experiment, but you can run a survey before having implemented the feature. E.g. you might find that few users are interested in user ratings, but some would really like to see a content description.
Use focus groups and interview people. This won't give you quantitative results like an A/B test, but can unearth valuable observations. One big advantage is that you can do interviews before you have a user base or even a functional product, which allows you to validate ideas early. You can also use interviews to refine your personas.
So yes, A/B tests are great! Just like Porsche sports cars are great. But sometimes all you need is a bicycle.