I started working on a website,for tracking and rating watched anime/manga/etc. and recommendations, and it should also have an API, for providing the info about series and other things.

On similar sites, I have noticed that, to use an API, one typically needs a token/auth of sorts, and there are certain usage limits, even if it's for reading info publicly available on the site.

But the problem is, you could circumvent all those limits by crawling the site directly. Even if the format is less convenient, once you have a parser in place there's no problem. Actually, if it uses clientside rendering, the info will already be sent in a convenient format. And on the other hand, this would also put more strain on the server, because the info may be spread out on multiple pages, needing more requests, and it would also send info not required by the client app.

In the end, is there a point in restricting the API used for info that's available publicly on the site? Should there be an unrestricted, unauthed API for reading public info, in order to avoid needless blunder for both sides? Or should, instead, the site itself have request limits, like an API?

  • 2
    Any high-volume, high-reliability website will rate-limit web UI access in addition to API access. Since the web UI intent is to provide access for humans, that rate is often at least one order of magnitude lower. Jul 27, 2015 at 12:25
  • @RossPatterson Thanks, I was wondering about the web rate limits, but hadn't found any clear info. That does make a lot of sense. Jul 27, 2015 at 13:12

2 Answers 2


Having a public API for data access from your site is about making the data available in a convenient, supported, well-defined and always-up-to-date manner. It is a way for a site owner to say 'here is data I collect and own, but I want you to be able to use it so I'm making it available. Oh, and I promise not to change the structure or do anything that might break your applications without communicating about it clearly'.

Crawling has some technical limitations, some very important legal considerations AND is prone to breaking without any sort of notification from the owner of the data. Personally I would not hesitate to consume a public JSON API if that has data I need, but I'd be hard pressed to start writing a crawler/parser to get it off a website...

  • 1
    The question is about the merits of having a restricted API, though.
    – Celos
    Jul 27, 2015 at 11:07
  • I see, I haven't thought about structure changes, that's a good argument. Jul 27, 2015 at 11:08
  • @Celos It's more about if it's worth restricting access to the "public info" API (through authing, and usage limits), if the info is available through crawling. He did provide arguments for why people would not just crawl instead, so it's fine. Jul 27, 2015 at 11:17

Token-authorized APIs are used for lots of reasons.

  • Controlling access to restricted information. Since a token usually maps somehow to an authorized identity, the site can restrict programmatic access to data exactly as it would to interactive access via the web UI. Tokens also prevent passwords from being disclosed in requests, and are usually revocable by both the user and the site.
  • Rate-limiting individual consumers. Since programs can easily present load many times higher and faster than people, being able to prevent overloading by an overly-aggressive consumer can make the difference between a site being up or down.
  • Controlling errant consumers. Some sites issue tokens to the program accessing the site, in addition to, or instead of, the user. This allows the site to make decisions such as blocking programs that abuse the API, that produce too many invalid requests, etc.
  • Improving usage statistics. Interactive usage is usually discussed both in terms of page views and sessions. API requests are the equivalent of page views, but there is nothing like an API session (unless the site allows persistent idle HTTP sessions, but that's uncommon). Tokens allow the site to group requests for the same user, and with some time-boxing, that can be a reasonable proxy for a session.
  • I understand their purpose for restricted info, I was merely wondering if tokens are necessary for info that's already accessible publicly and anonymously. In that case individual rate limits and controlling aren't practical (a user can have a dynamic IP and remove their cookies). And furthermore, there's no practical way to distinguish between a program and a human, both can use your site. Jul 27, 2015 at 13:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.