I wasn't sure how to ask the question. But basically, it's a textbook scenario. I'm working on a site that's article based, but the article information is stored in a database. Then the page is rendered with the information in the database based on the requested article id:

For example: http://www.mysite.com/articles/9851

I'm new to SEO, so I'm wondering how engines are able to crawl the contents of pages like this and/or what I need to do in order to ensure that it will be crawled.

So for instance, this site. All of the articles/posts on this site appear to live in a database somewhere. The URL has an ID which looks like it is used to tell the server which data to use to generate the page -- so the page doesn't actually exist somewhere, but it's template does. When I search google, I might find one of these posts based on the content of the post.

I understand that crawlers normally just find a page and follow it's links and follow its links' links and so on, but how does that work when the site is search based like this? Do have to create a page that randomly picks articles out of the database so that the crawler can see it or something?

  • Sharing your research helps everyone. Tell us what you've tried and why it didn’t meet your needs. This demonstrates that you’ve taken the time to try to help yourself, it saves us from reiterating obvious answers, and most of all it helps you get a more specific and relevant answer. Also see How to Ask
    – gnat
    Aug 1, 2014 at 5:47
  • 2
    Basically search engines will start at the top of your page and from there follow your link structure. They will repeat this once in a while (depending on various factors maybe about once a month, or more often if the decide your site is important). In addition you can provide a site map that helps them to understand your sites structures (or reveal links that are not reachable directly). Aug 1, 2014 at 7:02
  • @gnat, the first thing I mention is the post is that I am not sure how to ask the question because I'm not even sure what I'm looking for. If I can't ask a question in a human read forum using plain english, how am I supposed to ask a search engine? But to try an address your comment, every article about SEO I came across regarding dynamically generated pages only returned tips for SEO for AJAX. This isn't ajax. Aug 1, 2014 at 16:08
  • 1
    As far as I know it's roughly about once per month for 'average' sites. But you will see that sites like Stack Exchange or Reddit are crawled much more. Links on these sites appear more or less instantly on Google. It depends on a lot of things, mostly how important Google thinks you are, how often they actually find something new, meta data in your headers, how fast your server responds to requests and and and... Aug 1, 2014 at 18:08
  • 1
    You can use a sitemap (others already describes some details) or some kind of page with "posting history" where you keep a paginated list of everything posted. Even if it's not very helpful to your users it would ensure that everything can be reached by simple links. Aug 1, 2014 at 18:11

2 Answers 2


At its very simplest, search engines are just reading HTML. The web page at a certain URL is just an HTML file to a search engine, so it doesn't know a database query is involved. It only knows that the HTML file contains text and links -- which is an overly simplified explanation, but accurate enough for this scenario.

@RomanMik has the right idea. You need to review Google's documentation, but regardless of the search engine, the process is the same:

  1. Submit your site to the search engine
  2. Your site needs links -- lots of links, and links with descriptive text -- to the rest of the pages on your site
  3. A robots.txt file in the root of your domain can help tweak the search indexers, usually by telling them to ignore certain directories or file types
  4. Create fresh content and update the home page and section pages of your site to link to the new content

A search indexer will start at your home page, index it, then it will move on to child pages in your site based on the links it finds in the document it is currently indexing.

Forget about thinking of search engines and your site in terms of your database. It's all HTML to a search engine, and they need fresh content, descriptive content, and links to other pages. Do that and you've cleared the first major hurdle.

  • Thanks! Between this and other comments, I think I'm seeing how it works a bit better. Aug 1, 2014 at 18:09

Since Google is the largest Search Engine, and ALL sites want to be crawled by google, I suggest you start with google's webmaster tools page. (https://www.google.com/webmasters/). It is full of helpful resources.

To answer your question, you provide a robot.txt or a sitemap with all links that you want search engines to crawl and index. Here's a detailed answer about how to submit a sitemap to google. https://support.google.com/webmasters/answer/183669?hl=en

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.