Scanner++: Enhanced Vulnerability Detection of Web Applications with Attack Intent Synchronization

Vulnerabilities in web applications are prevalent nowadays, accounting for the vast majority of security issues in the Common Vulnerabilities and Exposures database [10]. Some of them can lead to severe consequences once exploited by attackers. Today, many different methods have been applied to detect and discover security issues in web applications. Among them, web vulnerability scanners are one of the most commonly used tools.

In general, web vulnerability scanners can be divided into two categories, white-box, and black-box. White-box scanners are static application software testing tools (SAST). They can analyze the source code of web applications and trace their program logic to detect security issues. For example, phpSAFE [25] uses static analysis to identify vulnerabilities in PHP-based applications developed with OOP (Object-oriented Programming). RIPS [33] performs semantic analysis based on source code to build a model for the program and detects vulnerabilities using taint analysis. In practice, however, the source code of web applications is sometimes not available. Besides, many vulnerabilities are caused by the overlapping effects between application security issues and inappropriate server configurations, which are difficult for white-box scanners to deal with. These tools can only detect vulnerabilities from the perspective of the application source code, ignoring server factors. It will inevitably lead to more false positives and miss security issues.

In contrast, black-box scanners are dynamic application software testing tools (DAST), only requiring access to the target website. Because of their simplicity of use and low false alarm rate, they have attracted much attention. Some black-box scanners even become a necessary part of several standards like Payment Card Industry Data Security Standard [29]. Black-box scanners work by simulating the hacking process, constructing attack intents against the target website, and analyzing response packets to detect vulnerabilities. A valid attack intent consists of a vulnerable attack surface (i.e., website path) and appropriate attack vectors (i.e., payloads). The work cycle of a typical black-box scanner can be divided into three stages, discovering attack surfaces, generating attack vectors, and analyzing response packets. The first stage mainly performs a content discovery procedure. Scanners will collect the scanning scope, web pages, and input points to determine possible attack surfaces of the target website. Subsequently, in the second stage, the attack module of the scanner will generate attack vectors for each attack surface to construct attack intents and send them to the target site. Finally, when the corresponding response packet is returned, it will be handled by the analysis module for vulnerability confirmation. This module will estimate whether the intent counts as a valid attack or not.

However, when we apply these black-box scanners in practice, their performance varies accordingly in different applications. As part of our preliminary evaluation, we used four scanners, BurpSuite, AWVS, Arachni, and ZAP, to test against several websites for a side-by-side comparison, as shown in Table 1.

SEACMS (SEA for short) is a web application used for content management. It provides commonly used functions like article publishing, user comments, software download, user management, etc. In this application, BurpSuite [30] detected 37 vulnerabilities, which has the best bug detection capability among the four tools we tested. However, it performed poorly in the MyBloggie application, or shortly MYB (a personal blog application), detecting seven fewer security issues than other tools like AWVS. Previous studies like [4, 5] have also shown the existence of such problem when comparing the performance of scanners, which can seriously affect the effectiveness of vulnerability detection.

When we further collected and analyzed the coverage and the number of attack requests, we found that such inconsistency mainly arises from a single scanner’s insufficient strategy when constructing attack intents. The existing intent construction mechanism of each scanner is usually designed with preferences. It can be efficient in some applications but may be less effective in others. More specifically, the problem lies in two aspects.

(1) In the first stage of the working cycle, many scanners’ detection performance is limited due to their inability to discover more comprehensive website content. The current content discovery mechanism relies on crawling and dictionary-based path enumeration. The crawling component of different scanners may be implemented with different configurations (e.g., depth of tracking, support for different content types). The embedded enumeration dictionaries also vary among scanners. For different sites, the optimal content discovery strategies to achieve better performance are not the same. Using only one strategy for all different test targets is not enough to obtain comprehensive attack surfaces. If the scanner does not cover a relatively complete scope of pages initially, the subsequent process and the detection of vulnerabilities will be limited significantly. However, if we can synchronize attack intents among scanners, a single tool can acquire the attack surfaces explored by multiple strategies. Take the scanner Arachni [34] as an example. When we scan the SEA website directly, it can only cover 32.76% of the website, with only five vulnerabilities detected. If we can give Arachni assistance by synchronizing website structure information explored by the other three scanners, its coverage can reach up to 75.86%, while 42 security issues can be identified.

(2) In the second stage, many scanners have a limited and narrow strategy for generating attack vectors. For various test targets, applying only one type of attack vector generation mechanism is not sufficient. If the attack vectors produced by the scanner can be more diverse, there is a greater probability of detecting more vulnerabilities successfully. Synchronizing attack intents allows a single tool to fuse attack vector generation strategies used by multiple scanners. For example, when scanning the site SCH, ZAP [18] was able to generate only 198 attack requests and detected two vulnerabilities. However, when we assisted ZAP in synchronizing attack vectors with the other three scanners, it can generate 64 times the number of attack requests and detect 11 vulnerabilities in the site.

Based on these observations, we propose Scanner++, an enhanced web vulnerability detection framework with attack intent synchronization. Firstly, by extracting and refining contents from request packets sent by base scanners, we can aggregate attack surfaces and attack vectors effectively, forming a synchronized attack intent library. Secondly, we design a run-time intent synchronization mechanism. Through analyzing the detection spot of the target, we synchronize related attack intents to the base scanner, thus augmenting its detection process. In this way, we can consolidate attack surfaces explored by different scanners, and share attack vectors produced by them. By synchronizing attack intents constructed with various mechanisms, the overall scanning process can be more robust and better applicable for diverse real-world targets.

For evaluation, we implemented Scanner++ and chose four high-performance base scanners working under it. We employed benchmark applications that have been widely used in previous research, and three real-world applications in CCDC, one of the largest security depository companies in the world as test-beds. The experiment results demonstrate that the base scanners perform differently on various applications, while Scanner++ consistently and effectively improves their vulnerability detection performance. Specifically, on ten open-source web applications, working under the Scanner++ framework makes BurpSuite [30], AWVS [2], Arachni [34], and ZAP [18] cover 15.26%, 37.14%, 59.21%, 68.54% more pages, construct 12.95×, 1.13×, 15.03×, 52.66× more unique request packets, and discover 77, 55, 77, 176 more bugs, respectively. Moreover, using Scanner++ can help base scanners detect 205 more security issues, even comparing with their combined results. Furthermore, when we applied Scanner++ on the well-tested real-world web applications in CCDC, eight serious previously unknown vulnerabilities were detected and fixed.

Overall, this paper makes the following contributions:

The remainder of the paper is organized as follows: Section 2 presents a background of web application vulnerabilities and scanners. In Section 3, we present an example to illustrate the motivation of intent synchronization. Section 4 demonstrates the architecture and methodology of Scanner++. Section 5 presents the evaluation of Scanner++. Section 6 discusses the potential threats to the validity of Scanner++. Section 7 introduces related work, and we conclude in Section 8.

When a user is interacting with a web application, the browser will send a request packet to a specific path of the server. The request packet usually contains several parameters to put the information that the user wishes to transmit. Then the browser will wait for the response packet from the server and parse the web page contained in it. There are two most commonly used request methods, the GET method and the POST method. One of the main differences lies in the position of parameters in the request packets. The request packets using the GET method will have parameters placed at the link position, while the POST method will put parameters in the body section of the request packets.

Some web applications contain severe security vulnerabilities, resulting in sensitive information leakage, user identity theft, or even leaving the entire server under the control of attackers. The process of exploiting a vulnerability is similar to the normal interaction mentioned above. An attacker will construct request packets with parameters filled with malicious data and send them to a path of the vulnerable host. The attacker then estimates whether they have successfully exploited the vulnerability and gathers the information needed from the response packets. The host of the target server and the corresponding path make up an attack surface, which is the starting point if an invasion happens. The malicious data sent by the hacker is described as attack vector. The attack surface and the corresponding attack vectors constitute of an attack intent. Thus, a complete attack against the target application means to construct a series of valid attack intents. Specifically, it implies that the hacker should find a vulnerable attack surface, generate effective attack vectors, assemble them into request packets, and then send out.

To prevent malicious requests, developers often apply some sanitization measures to user inputs. Sanitization is a set of instruments that estimate whether the input contains attack vectors. Ideally, each user input that may cause an attack will be properly sanitized, but based on previous research [21], reaching such a high standard is challenging. Many attackers will try various methods of mutating attack vectors to bypass such sanitization and achieve successful attacks.

Black-box scanners work essentially by mimicking the process described above. They detect vulnerabilities by automatically constructing attack intents on the target application. First, the scanner gathers as much information about the target’s attack surfaces through the content discovery process. The gathered attack surfaces essentially determine the scope of pages that scanners will attempt to attack. The more comprehensive the detection of the attack surfaces, the more effective the scanner will be. Then, for each attack surface, the scanner will generate attack vectors to construct the request packet. The attack vector generation process basically determines the attack capability of the scanners. The more diverse the constructed attack vectors, the more effective the scanner will be. After sending the request, the scanner will analyze the corresponding response to determine whether it constitutes a valid attack. If necessary, it will further mutate the attack vectors to bypass some existing sanitization mechanisms. When the scanner succeeds in composing a valid attack, this indicates that it has found a vulnerability in the website.

We use a simplified example to illustrate the motivation of enhanced scanning with intent synchronization, as shown in Figure 1. This example is extracted from real-world web applications and the vulnerability detection process of scanners.

The website’s root path has three pages (edit.php, user.php, search.php) and an admin path, which contains three more subpages (login.php, manage.php, view.php). Among them, search.php contains a remote file inclusion vulnerability [36]. manage.php has an arbitrary file download vulnerability with a sanitization mechanism to check the extension in input to protect from attack. It will verify whether the requested file is a picture with jpg extension. However, the designed mechanism is inadequate, so it is still vulnerable. An attacker can insert NULL character to truncate the path, and harvest the sensitive file content while keeping the extension as jpg at the same time. Such insufficient sanitization situation is prevalent in today’s web applications.

In each stage of a black-box scanner’s work cycle, the strategies often vary in different scanners, leading to inconsistent results. In the first step, the content discovery process, each scanner can detect a portion of the site’s structure. Scanner A can discover search.php, but cannot generate attack vectors against the type of vulnerability hidden on that page. This makes Scanner A impossible to discover this vulnerability, as shown in the sixth and seventh columns of the first row in Table 2. Although Scanner B supports constructing that kind of attack vectors, it fails to discover the page search.php in the first step. The incomplete attack surfaces of Scanner B make it unattainable to report the vulnerability as well, as shown in the second row of Table 2. In the second step, attack vector generation, although Scanner C generates the attack vector path=./././etc/passwd against the page manage.php, but this attack vector is too naive and simple to bypass the website’s sanitization mechanism. Thus, Scanner C cannot successfully exploit and discover the issue, as shown in the third and fourth columns of the third row in Table 2.

In a word, if we use these three scanners separately, each one can only use its own content discovery approach, thus exploring a part of the attack surfaces in the first stage, as shown in the first row of Figure 2. Subsequently, targeted at that discovered part of attack surfaces, they can only generate attack vectors with their own strategies. Even if we combine the results produced by these three scanners, the overall coverage can be summed up, but in the end, the two security issues still cannot be detected, as shown in the fourth row of Table 2 and the upper part of Figure 2.

Based on this motivation, we design the framework Scanner++, enabling multiple tools to synchronize attack intents during their scanning process. However, unlike many other security testing tools, web vulnerability scanners often operate in a complete closed-loop workflow. Direct interference with its internal state using intrusive methods would severely impair the universality. For open-source scanners, this method will require significant manual labor when integrating new ones. While for proprietary scanners, such a method will be completely unadaptable. How to implement a fully non-intrusive attack intent synchronization framework is the first challenge we need to tackle. Therefore, as shown in Figure 2, we implement a proxy-based architecture and a package-based intent synchronization approach. The former ensures that applying Scanner++ does not require any modifications to base scanners, allowing rapid integration of new ones. The latter ensures that Scanner++’s intent-synchronization approach can be applied to all base scanners, as Web applications’ interaction is based on the request-response packet exchange process. In this example, Scanner++ will work as a proxy between Scanner A, B, C, and the target site. All the attack requests sent by base scanners can be intercepted, and all the responses can be modified by Scanner++.

Originated from this methodology, request and response packets are the appropriate entry points to enhance scanners. We need to make full use of such constrained information to achieve efficient attack intent synchronization. That is the second challenge we need to tackle. Therefore, a purification mechanism and a run-time intent synchronization approach are designed. The former process aggregates and refines the attack intents extracted from request packets sent by base scanners to construct a synchronized attack intent library. It contains consolidated attack surfaces explored by three scanners during their content discovery process and valuable attack vectors generated based on various strategies. In this way, all the five website pages discovered and the attack vector generated by Scanner C on page manage.php will be collected. Based on this library, the latter process achieves run-time synchronization by converting and supplementing needed attack intents into corresponding response packets transmitted to base scanners.

In such a manner, Scanner B is able to share Scanner A’s content discovery results, B can reach the page search.php. Then B can generate an attack vector against it and successfully detect the vulnerability. Meanwhile, Scanner A can obtain the attack vector produced by Scanner C against manage.php, it can further mutate it to produce a more complex attack vector, “path=./././etc/passwd%001.jpg”. Such an attack vector has an image extension (i.e., “.jpg”) that can pass the checks of the sanitization module. However, the file download module reads the file contained in the truncated path (i.e., the path before the NULL character), which is the content of “/etc/passwd”. In this way, this attack vector can harvest the sensitive file while keeping the extension remains “jpg” at the same time. Such a fusing strategy can help them bypass the sanitization mechanism, and detect the vulnerability hidden in manage.php.

With Scanner++, three scanners can synchronize their attack intents, and these two vulnerabilities can be detected. By synchronizing the content discovery results, each scanner will be able to obtain the website pages discovered by the others. This can make every scanner form a set of more comprehensive attack surfaces, as shown in the third row of Figure 2. Moreover, by sharing generated attack vectors accordingly, each scanner can reuse, further mutate, and generate more complex and diverse ones. Overall, scanners’ test coverage and attacking capability will be significantly enhanced, as shown in the lower part of Figure 2.

In this section, we will introduce the methodology of Scanner++ to achieve enhanced web vulnerability detection. The framework of purifying and synchronizing attack intents of base scanners is presented in Figure 3.

First, an Attack Intent Purification process is conducted. Scanner++ can intercept the original requests from base scanners and extract attack intents. Then it utilizes the intent refinement algorithm to deduplicate and refine the collected intents to construct the Synchronized Intent Library. Consisting of a comprehensive Attack Surface Set and Attack Vector Pool, this library serves as a synchronized summary of the attack intents constructed by each scanner.

To share information effectively between scanners, Scanner++ implements a Run-time Intent Synchronization mechanism. In the first step, it selects the relevant attack intents from the synchronized library based on the current scanning position of the target. Then, Scanner++ converts and injects the obtained attack intents into the response packets. Finally, Scanner++ transfers the modified response packets to the base scanners. The supplemented attack surfaces can help the scanner’s content discovery process identify a more comprehensive website structure. At the same time, each scanner can use and further mutate the obtained attack vectors to improve the scanner’s attack capability.

To present the effectiveness of enhanced scanning, we conduct thorough evaluations on ten benchmark web applications that have been used as test-beds by previous research like [3, 13, 35]. We also applied Scanner++ to detect vulnerabilities in three real-world web applications of CCDC (Central Depository & Clearing Co., Ltd), an important securities depository company, to verify its performance in industrial practice. We answer the following five questions:

RQ1 focuses on verifying the performance improvement of Scanner++ for the base scanners. We use the coverage rate, number of attack requests, and numbers of detected vulnerabilities as metrics to compare the performance of scanners working with and without Scanner++. RQ2 is used to evaluate the effectiveness and performance of the intent refinement algorithm. Experiments are conducted to compare attack intents and the performance of scanners with and without intent refinement algorithm. RQ3 is mainly used to assess the effectiveness of the synchronized intent library of Scanner++. In addition to the four base scanners, we adapt a new scanner, Wapiti, with Scanner++ by leveraging the existing attack intent library. RQ4 is used to evaluate the performance of Scanner++ in real-world web applications. We also give a case study to illustrate the process of the framework in assisting vulnerability detection. RQ5 is primarily used to evaluate the scalability of Scanner++. By gradually increasing the scanners involved in the synchronization, we evaluate the performance of Scanner++ under various base tool scenarios.

Based on the evaluation of benchmark applications and real-world applications, we demonstrate that Scanner++ helps base scanners perform better. However, some limitations still threaten the usability and performance of enhanced scanning. The main limitations are discussed below.

The first potential threat is the selection of base scanners. Diversified base scanners can facilitate the effectiveness of Scanner++. Not only is this reflected in the types of vulnerabilities supported, but the variety of content discovery and attack vector generation strategies can also make the synchronized scanning better. Subsection 5.1 describes our base scanner selection. We selected these scanners based on previous performance comparative research. Also, there is diversity in the types of vulnerabilities supported by the four base scanners. BurpSuite supports 153 different types of vulnerabilities, Arachni supports 27 types of vulnerabilities, AWVS supports 42 types of vulnerabilities, and ZAP supports 158 types of vulnerabilities. Specific type lists can be found in [1, 20, 28, 31]. However, just because scanners work well on their own does not necessarily mean that they are sufficiently diverse in their strategies. To the best of our knowledge, there has been no previous comparative research of the strategy differences between scanners at each stage. Many scanner developers are also reluctant to disclose the specific mechanisms they used. Therefore, for the time being, we can only select the scanners based on their performance. A possible solution to this threat is to try more scanners testing against the same site and use the framework to collect and observe the attack surfaces and attack vectors explored and constructed by each one. We can have a grasp on their content discovery strategies and attack vector generation mechanisms by analyzing their intermediate process. We can then further evaluate their diversity based on the result and decide which scanners have more strategy differences. These scanners are suitable to participate in the framework Scanner++.

The second potential threat is that the enhanced scanning process may still not easy enough to use. In Section 4.4, we described three facilities to improve the usability of scanning and reduce the repetitive configuration process of scanners. These facilities can handle common scenarios that require unique settings for individual scanners and improve preliminary preparation efficiency. However, the usability can still be improved for better scalability on various applications and different scan tasks. For example, we can integrate some APIs that call scanner functions into the framework. The user can then operate the framework, and the framework translates the corresponding operation into the API of each scanner to call the corresponding function. This allows the framework to provide a unified access portal to the user.

The third potential threat to validity is the overhead of enhanced scanning. Based on our experiment, Scanner++ itself does not have much impact on the base scanners’ efficiency, as the number of attack requests per unit time does not decrease significantly. The number of attack requests sent per second only changes from 10.81 to 9.87, which is only a 8.70% decrease, demonstrating Scanner++’s low overhead despite the added mechanisms. Meanwhile, it takes only 1.31 seconds to refine 8,479 attack vectors on average, demonstrating the high efficiency of the refinement algorithm. The process of intent injection is essentially converting attack intents into website metadata or page elements and appending them to the end of the response packet. The conversion process follows a predefined template and is therefore very fast. Based on our experiment, the conversion time of 5,367 GET-type attack vectors and 1,202 POST-type attack vectors (all the attack vectors collected from ten benchmark websites) is only 0.0061 seconds and 0.0512 seconds, respectively. Each intent injection only needs a part of these vectors, so it will not slow down the working process. These statistics demonstrate that intent injection will not affect the efficiency of base scanners. It is worth noting that using the total execution time of the tool to measure the additional overhead of efficiency is not an appropriate metric. Unlike some other dynamic testing tools (e.g., fuzzers) that run continuously, web application scanners automatically stop after iterating through all explored paths and predefined attack vector generation strategies. Lower coverage and fewer attack vectors of scanners working alone will result in shorter runtime, while its effectiveness will be limited. But with Scanner++, along with a wider range of attack surfaces and more vectors, we can accomplish those increasing number of valid attacks or requests with more scanning time, thus detecting more vulnerabilities. In the future, we can make better use of the distributed scanning capabilities of some base scanners, deploying Scanner++ in a distributed manner to further improve the efficiency of enhanced scanning through parallelization.

The fourth potential threat is the manual efforts needed for adapting Scanner++ to the new scanners. As a non-intrusive proxy running between base scanners and target applications, Scanner++ can be easily applied to most black-box scanners. Adapting new scanners only requires configuring their proxy server. The vast majority of scanners provide a direct interface for this. For the few scanners that do not provide an interface, adaptation can be made using a simple external proxy tool such as ProxyChains. Once configured the proxy settings, Scanner++ can monitor every exchanged packet, fetch and complement information of base scanners without any modification.

Vulnerability Detection on Web Applications. With increasing attacks on web applications, many researchers have developed web vulnerability detection tools.

White-box vulnerability detection tools use techniques like static analysis and symbolic execution to analyze the source code of web applications. Among them, RIPS [33] is one of the most widely used scanners in industrial practice. Dahse and Holz [11] used intra- and inter-procedural analysis to model application in PHP language and uses taint analysis to detect vulnerabilities. phpSAFE [25] constructed a program model with AST and supported analyzing web applications developed with OOP. Livshits et al. [22] used specifications which were provided by users to find security issues in Java web applications.

Black-box vulnerability detection tools can work without the source code of applications and check security issues by simulating the attacking process. Commercial scanners like BurpSuite [30], AWVS [2], and Nessus [38] are the most common testing tools used by companies. A few open-source scanners, like Arachni [34], ZAP [18], and Wapiti [39] also perform quite well in vulnerability detection. In addition to these tools, many researchers are trying to optimize black-box scanners to detect more kinds of vulnerabilities. In [12], authors proposed a state-aware scanner, by inferring the application’s internal state to achieve higher coverage and assist the detection process. In [14], authors used genetic algorithm to generate attack requests and detect cross site scripting vulnerabilities in web applications. In [8], authors focused on detecting inconsistent input validation between the front-end and back-end, using black-box detection methodology to examine parameter tampering opportunities in web applications. In [37], authors used taint tracking to scan persistent client-side XSS in the wild. Some research also studied the hybrid black-box and white-box approaches to detect vulnerabilities. For example, Navex [3] used static analysis to construct a property graph and combines dynamic analysis to scan PHP web application vulnerabilities. Saner [6] used static analysis technique to model the sanitization process, and composed it with dynamic analysis to execute sanitization code on malicious inputs to detect improper sanitization procedures.

Unlike previous works, we are not proposing a new concrete scanner. Instead, we systematically study the framework of enhanced scanning with attack intent oriented synchronization. Black-box scanners can be integrated into our framework and performs better than working separately.

Enhance Vulnerability Detection with Multiple Tools. There are some research working on fusing multiple tools for vulnerability detection. Some works tries to combine the results of single tools directly to reduce the false negative. For example, SmartBugs [16] combines the results of multiple vulnerability detection tools to perform security checks on smart contracts. Nunes et al. [24] combine the union of results from several static analysis tools (i.e., white-box scanners) to detect security issues in PHP-based web applications. Different from those work, Scanner++ supports interaction of different scanners during the whole scanning process and can enhance multiple black-box scanners without access to the source code, thus supporting web applications developed in any languages.

In fuzzing of libraries, EnFuzz [9] integrated diverse fuzzers with global seed pool to achieve higher coverage and bug discovery. Based on EnFuzz, Cupid [19] designed a complementary selecting strategy to automatically combine fuzzers and improve the final coverage. Different from fuzzers, black-box web vulnerability scanners work in distinct ways and solve different problems. Fuzzers randomly mutate and generate invalid data as test cases to detect bugs in programs and libraries. In comparison, scanners need to interact with web applications based on the request-response model and generate highly structural test cases based on vulnerability exploitation templates to detect security issues like SQL injection. Therefore, fuzzers like AFL cannot be used to detect web application vulnerabilities.

Due to the differences of scanners and fuzzers above, the research challenges and target domain of Scanner++ and cooperating fuzzers like EnFuzz are significantly different. Scanner++ is a non-intrusive black-box intent synchronization framework targeted at web application vulnerabilities. Since black-box scanners work in a closed-loop manner, we adopted a proxy-based working architecture and a package-based intent synchronization approach to ensure scalability. Ensemble fuzzers like Enfuzz and Cupid are intrusive grey-box fuzzing integration frameworks targeted at bugs in programs or libraries. They enhance base fuzzers’ coverage through seed synchronization.

Therefore, Scanner++ is entirely incomparable with cooperating fuzzers because of the following reasons. First, integration strategies adopted by cooperating fuzzers cannot apply to scanners. Unlike fuzzers, most scanners work in a complete closed-loop manner. Any intrusive methods designed in cooperating fuzzers will severely change base scanners’ original workflow. Second, web application vulnerabilities cannot be detected by cooperating fuzzers like EnFuzz and Cupid. They can only be detected through highly structural test cases generated based on exploitation characteristics and mutated based on semantic equivalent substitution. Meanwhile, since most vulnerabilities in web applications will not trigger crashes, they can only be identified by parsing response packets. Therefore, fuzzers, based on random mutation strategy and crash-based oracle, cannot identify web application vulnerabilities.

In this paper, we systematically investigate the idea of attack intent synchronization to enhance the vulnerability detection of web applications. Scanner++ improves the capability of existing scanners by synchronizing valuable attack intents and supplementing related ones among different base scanners according to their detection spots at run-time. First, this framework assists the scanners’ content discovery process by providing related attack surfaces identified by each other. In this way, scanners can obtain a more comprehensive site structure and achieve higher coverage. Second, it enhances scanners’ attack vector generation process by sharing related attack vectors generated by each other. Scanners can be guided to mutate and generate more complex attack vectors, thus increasing their attack capabilities. Based on our evaluation, Scanner++ helps popular base scanners perform better in terms of coverage, unique request amount and detected vulnerabilities on benchmark applications. On real-world web applications used in CCDC, we have found eight serious previously unknown vulnerabilities. Moreover, Scanner++ can be easily utilized to integrate more base scanners for industrial practice.

Our future work mainly focuses on three aspects: the first is to evaluate the diversity of different base scanners more systematically and try to choose more diverse ones for the enhanced scanning framework; the second is to design more advanced attack intent purification mechanism and run-time intent synchronization mechanism to further optimize the performance; the third is to upgrade the usability of the framework and design more auxiliary facilities to reduce time consumption of the pre-configuration process.

Vulnerabilites in the tested real-world applications can cause a great deal of damage. As an example, we illustrate in more detail how the arbitrary file download vulnerability found by Scanner++ in the application BIRS can be exploited. As shown in Figure 8, we further attempted to exploit this vulnerability in three ways. (A) First, we harvested all the source code of the web application with this vulnerability. For attackers, trying to compromise the application based on analyzing the source code will make vulnerability discovery and exploitation much easier. (B) Subsequently, we tried to obtain configuration files of the web application. Checking through the harvested files, we found that the needed credential to connect to the database is exposed. With this information, we successfully connected to the Oracle database attached to the application. A malicious attacker may tamper with sensitive data stored in the database directly or with an SSRF vulnerability. With accessing the database, we used system command execution functions (e.g., DBMS_EXPORT_EXTENSION in Oracle DBMS) to get a reverse shell and gain control of the server. (C) In addition, many web applications are hosted on servers that have multiple network interfaces. The server used by BIRS has two network interfaces, one connected to the public network to receive requests from users and the other connected to the intranet to obtain necessary information from other internal hosts. We harvested network-related files such as /proc/net/arp, /etc/network/interfaces to probe internal network structure and gather information of other internal hosts. Since the internal systems are normally protected by the network topology, they often have a weaker security posture. Considering that the authorization scope of our study is limited to this server, we did not go further trying to compromise intranet hosts. For an attacker, this server can be used as a perfect entry point to carry out attacks on other hosts in the intranet.

Since Scanner++ uses a proxy architecture, adapting new scanners only requires configuring their proxy server. Therefore, little manual efforts are needed to add new scanners to Scanner++. In the repository, we provide a detailed procedure for configuring proxies for several popular scanners. Also, we list the configuration process of the tool ProxyChains as a general proxy configuration scheme.

To further demonstrate this, we invited five software engineers from CCDC to configure five base scanners, BurpSuite, AWVS, Arachni, ZAP, and Wapiti, working under Scanner++. We recorded the time consumption of the configuring process, as shown in Table 13.

Considering the different configuration process of base scanners, the average preparation time of adapting the tool varies from 36 seconds to 223.2 seconds. Meanwhile, all software engineers can prepare base scanners for working under Scanner++ within five minutes. This fully illustrates that Scanner++ can be quickly applied to new scanners.