Vulnerability Assessment – Automated v/s Manual Testing

We all know that there are 3 distinct pillars of software testing – functional testing, security testing and performance testing. There are multiple tools out there, to make tester’s life easy. While these automations are a important, there are few lacuna that a tester should be aware of, and this is especially true in case cyber security vulnerability assessment (VA) and penetration testing (PT).

Difference between VA and PT

Vulnerability assessment is a process by which an ethical hacker scans a website or a network and maps loopholes. The next step is to find vulnerabilities which are nothing but the inherent security loopholes within the software program or the network component. Penetration testing is a process in which the ethical hacker actually performs actions to exploit those loopholes and create a proof of the test. It is also checked if there are more underlying vulnerabilities which are exposed as a side effect of the exploitation, and if those could further be exploited.

Why automation fails?

If you look at the process mentioned above, it becomes very clear that VA can be automated but the PT cannot. The subtle reason behind this, is based on how human mind works in each case. Exploiting a vulnerability needs cascaded intelligence wherein one needs to perform an action, and take next steps based on the results of the first action. Since each and every application or website or network is different it is almost impossible for a tool to understand the results and make an analytical judgement on it. There are tools which does perform these tests to some extent but those are still based on pattern based testing and not really possess cascaded intelligence.

Example of Net-banking Portal Security Testing

Consider a net-banking website with a page that lets user transfer money to another account. This page can be susceptible to CSRF (Cross Site Request Forgery) attack whereby an attacker can create a bogus request and submit the form on behalf of the user. One of the ways to fix this problem, is to have a CSRF token as a hidden parameter on the page. Now here is the problem. A VA tool can check presence of this token and if present, it can pass that test saying everything is fine. Unfortunately this is not true. The real test is not to check the presence of token but to perform series of intelligent tests to ascertain that the token is being validated properly on the server side, thus preventing bogus submissions.

In my firm, while performing penetration testing for business critical applications or networks, we have witnessed numerous cases where the customer was relying upon tools for periodic security testing. When we performed manual testing, it opened up the Pandora’s box and to the dismay of our customer, at least 5 times more vulnerabilities were found which they were completely unaware of.

There are tons of examples such as stray ports, vulnerable SMTP services, mis-configured web servers, SQL Injection attacks, cross site scripting (XSS) attacks etc., where it is proved that automated VA is like swimming and being afloat, while manual VAPT is analogues to scuba diving, where you see a different world altogether. This article soon follows another article providing statistics on manual testing methods which should demonstrate how effective it is in cyber security testing.

More about penetration testing at http://www.valencynetworks.com/penetration-testing-services/steps-of-penetration-testing.html