Suppose I have created an application that makes remote connections and does stuff, how does an anti-virus/anti-malware program decide if my app is harmful?

I know there is a signature they check for to identify a program as either good or bad but it seems to me that would result in a lot of false positives because a lot of code will be common practice.

I mean, how is Team Viewer different from my hypothetical backdoor application?


It doesn't know, it guesses based on a set of rules.

Some rules, based on similarity to known malware, give reasonably reliable positives. Malware mutates and hides, so negatives are less reliable.

Some rules are based on behavior, such as opening connection on certain ports.

Some rules are based on cryptographic signatures; programs signed by good guys can't be bad, and vice versa.

There are lots of other kinds of rules, but none are foolproof or terribly effective against previously unknown threats.

  • 1
    Signing: As an example, a MacOS application can be unsigned, or it could be signed by a company with some known URL (say www.microsoft.com, but if it's signed by www.rnicrosoft.com then this means nothing), or it can be signed by some company, with Apple kind of guaranteeing that they know who that company is. – gnasher729 Apr 21 '18 at 16:15
  • 1
    Actually, all that signing guarantees is that someone paid someone else for a certificate. The actual verification of "good guy" status is little more than cashing a check. Signatures do give reasonable assurance that the code is as the signer delivered it, and hasn't been modified since. – ddyer Apr 21 '18 at 16:21
  • 1
    @senaps: For digital signatures; publisher creates a hash of the file, encrypts the hash with their private key and attaches the encrypted hash to the file; then checker creates a hash and uses publisher's public key to decrypt the attached encrypted hash, and compares them. If hash and decrypted hash match then the file wasn't modified and publisher must be the owner of the private/public key pair (authentication). – Brendan Apr 21 '18 at 18:18
  • 1
    @senaps: "proof of file not been changed or messed with" can't really work without authentication (e.g. attacker could modify and then sign the modified copy themselves). That creates a second problem (knowing who should've signed it) - you can check the signature is correct (without caring who signed it); and you can check that whoever signed it is who you think it should be (without caring if the signature is correct). What I described earlier is the fundamentals for the first part, and is mostly always the same (just implementation differences). I haven't described the second part at all. – Brendan Apr 22 '18 at 9:19
  • 1
    @senaps: For that second part (checking if whoever signed it is who you think should have signed it); I only know how I think it should work (e.g. publisher generates their own key pair and registers their public key with the OS vendor for free with no third-party involved), and I don't know how it actually works (with "certificates" and fees to third-party "men in the middle"). – Brendan Apr 22 '18 at 9:22

Mostly, it knows that Team Viewer and similar well-known programs aren't malware because the vendors have a database of cryptographic hashes of programs that have been manually checked, and Team Viewer will be on that database.

The fact that the heuristics used by this kind of software often has false positives is hidden from view by the use of such whitelists, but can be a serious problem for people using less well-known systems.


It depends, custom code will probably not get tagged unless it’s accessing resources that requires admin privileges. Customized code is the reason why Advanced Persisitent Threats are so hard to detect. Using anomaly based detection can result in false positives like you mentioned. You’re more likely to get a warning prompt.


As you have correctly guessed, yes legitimate Remote Administration Tool (RAT) like TeamViewer or sshd is technically indistinguishable from malware. In fact, malware often would ship and configure a regular, unmodified RAT correctly signed by the original RAT author, instead of custom RAT, to make it difficult for anti malware to distinguish it from legitimate use of the RAT.

How antimalware normally detect this case is to detect the configuration rather than the program. For example, if a legitimate RAT has been configured to accept the public key of known attacker or connect to a known botnet, then we can be pretty sure that this RAT is not configured by the legitimate user.

  • so there is no way they can predict what this RAT app is doing right? if the attacker has used their botnet or public key before or there is some way to know them from previous attacks, anti-malware catches them, other than that, if no good policies are available on firewall, we are exposed to the attacker. – senaps Apr 23 '18 at 11:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.