In general, you are correct about the anticipation of problems. However, everything is good in moderation. Too much and unnecessary checking makes the code hard to write, read and test.
During application testing, you should catch many errors that you could analyze and determine if they fit into your plan of error management (see below for the details).
If you are working in a team, the team should have this plan communicated so that code is built in a consistent manner.
I classify execution results of a piece of code according to their nature using the phrase: "the Good, the Bad and the Ugly".
The good is your normal processing. That is when the method executes the sweet case.
The Bad category is when the method can't do its job because the data passed is bad or not enough. Catching this kind of errors is important since these would arise from your intentionally built checks.
For example, you can verify the validity of passed arguments to a method before processing them, as another example is when you request a method to Add Customer but you don't provide a name for that customer.
Another type of rules that falls under this category is regarding arithmetic overflow and null values. These are slightly different than regular user-data validation. For example, a complex equation may result in a large number that you are not sure whether it would fit in your target data type. Such problems should worth coding for only when in-doubt. By the end of your testing, such checks may not be necessary.
The ugly category is when the problem is outside of your code. This is an error in either the system, the logic, or whatever. For example, in the Add Customer, if during the insert, the database gets full. This is not your program's or your logic fault. You must identify, log and catch such errors but only in a generic way. A common approach for this is the Catch clause (I am not sure how PHP provides for that).
If you follow the above classification of types of errors, you will not have to check every statement.
Also, you should build your code with the above in mind. So, don't validate business rules in the middle of processing I/O. Structured programming concepts help here. To get cleaner code, you may want to separate methods for data validation from I/O processing.
In languages such as Java and .Net, the languages allow exceptions to bubble up, if PHP allows that, then this will greatly reduce the need for detailed checks since your actions can be protected from errors when you place your code within the Try block's scope.
Another important rule is you should avoid repeating coding the rules at different levels when you can. For example, leave the Ref. Integrity (RI) checks and database constraints to the database and don't repeat them in your code. I have seen developers coding the delete manually even though the Database is doing that for them.
The way you define your method interfaces affect your ability to check results greatly. Sometimes people don't even return primitive types and use classes as a way to communicate detailed feedback from method calls.
Is there some standard or rule of thumb to go by when deciding when and where to do data validation?Apart from cleaning up user input (anything that comes from the client), there isn't some other rule of thumb, you should validate everything that makes sense.I realize that I could go crazy trying to verify each and every line of code but at the same time I want the code as resilient as possible.It would help to put your code up for code review, at Code Review Stack Exchange