May-June 2020

ISSN: 0193-4120 Page No. 3469-3473

Recon-Automation using OSINT

1
Sorna Shanthi D, 2Jai Krishna B, 3Jagan RM, 4Harish M
1,2,3,4
Department of Computer Science and Engineering
Rajalakshmi Engineering College
Chennai, India

Article Info Abstract

Volume 83 In Today's technology driven world, there are many tools and sources to
Page Number: 3469-3473 gather publicly available data for the purpose of information security
Publication Issue: investigation, but the major shortcoming is that it still needs intervention
May-June 2020
of a person to infer whether the data obtained is relative and genuine. In
Open Source Intelligence (OSNIT) the name “open source” clearly
depicts that the information obtained is available to anyone and present
publicly, but OSINT mainly works on how the information is gathered.
OSINT involves information that is not only gathered from search
engines but also from various other sources that search engines cannot
deliver. Main application of OSINT in cyber-security is threat, predictive
intelligence. The system that is proposed not only has the ability to
perform link analysis where relationships between endpoints are
Article History evaluated but will also try to find correlation between the data fetched
Article Received: 19 August 2019
from different open source endpoints.
Revised: 27 November 2019
Accepted: 29 January 2020
Publication: 12 May 2020 Keywords: OSNIT , cyber-security, Recon-Automation

1. Introduction available on any particular organization, a person that is

The more we are connected to the internet, the quantum made available publicly by intention or unintentionally.
of information about each and every user keeps scaling This gathered data is processed and provided as
up. This information can be put to good and bad use by intelligence to the user.
either penetration testers or blackhats respectively. From
an organisation‟s perspective this actually means, 2. Related Works
substantial information has been exposed in public data Focussing on OSINT, there are a number of closed source
sources. This is an area that is often flouted by tools and open source endpoints where focus is mainly on
information security consultants and auditors when particular types of data. There exists a tool called
assessing the security posture of an organisation. Open „Harvester‟ which would take an email address or domain
Source Intelligence (OSINT) is an intrinsic part of name as the target and try to gather information about the
penetration testing and anticipating any security threats specific given email address the result of which can be
beforehand. OSINT can be simply described as usernames of social sites or IP address of the domain
information that can be gathered on any specific topic, specified. But the limitation with this approach is that
trend or even a person from the web and publicly after getting the usernames or IP out of the Harvester,
available resources in it.OSINT searches can be pivotal in there is a need to rely on another tool to gather
terms of conducting investigations when time and cost information from social sites according to the resultant
parameters are taken into consideration. Network attacks usernames. Shodan is a search engine unlike any search
on firms are mostly instantiated from such leaked engines like google and bing, Shodan gives IT
information. OSINT searches not only help in unravelling infrastructure related information like IP addresses,
one‟s own vulnerabilities but can also be used to gain network devices, webcam and other internet related
valuable information about their competitors. devices as a result. Maltego is a proprietary software
This paper presents an efficient way for analysing whose main purpose is to mine data from various open
publicly available information on a specific target by sources and map the obtained data thereby gathering
crawling upon various sources of data using OSINT information on the target. This information obtained is
gathering. The main goal is to retrieve information depicted to the user henceforth presenting, exploring

Published by: The Mattingley Publishing Co., Inc. 3469

 May-June 2020
ISSN: 0193-4120 Page No. 3469-3473

relationships amongst the information gathered on the 4. Methodology

target. Handler:
Every tool is useful in gathering information on Handler does the job of mapping each data point that is to
specifics but there is a need to find correlation between be crawled based on the user input and the type of data
the data obtained and to recursively continue processing extracted from each thread. Handler holds the
which is not focussed by much of the tool at present. responsibility of invoking each and every data point
Considering Harvester, information on usernames can be module based on the data that is extracted upon execution
obtained but one individual has to provide usernames as of each individual thread.
input and this process is repetitive. There is a need to
automate the processing of data after gathering it using Modules:
OSNIT. There‟s a need for bringing this correlation in, 1.Fetching the target data from Social Media:
since this eradicates the necessity of repeating the same The main goal of this module is to find if any
process for different data. Social Media profiles exist with a particular username or
its combination. The username can be specified as the
3. Framework target by the user or upon crawling a website, individual
names can be extracted. By using the combinations of
such extracted names can also be passed to this module to
check the existence of Social media profiles. The profile
returned may not be entirely accurate but the ratio of
getting legitimate results is greater than that of failures.
Working:
The username that is to be used by this module
can be given by the user as input or it may be obtained
from processing of other modules. This module analyses
if any matching profile exists by appending this username
to various social media URLs and by analysing the
response, outcome can be determined. For example, upon
receiving a username it can be determined if any
matching profile in Instagram exists by using
{www.instagram.com/username}. Response from this
request helps us determine whether such profile exists or
not
This goal can also be achieved by passing this username
Figure 1: Framework of the Proposed System to the search engine module which determines an account
exists by using dork.
The current norms of an individual doing all the
tedious investigation of finding valuable data from crucial 2. Fetching the target data from Search Engines:
endpoints and checking for their authenticity can be There are a variety of search engines out there to
automated by creating the framework as shown in the provide specific search responses. Using these search
figure 1.Since it's impossible for a human individual to go engines would result in vast volumes of valuable
through all the endless data that's available on public information that can be gathered about a particular target,
resources, optimisation is done by presenting the endless which may assist us in better understanding our target.
available data to the framework, and try to get more Content that has been retrieved from these search queries
efficient and valuable information. The proposed system may expose different types of valuable data like some
has the ability to perform link analysis and find website address, or phone numbers, or any user related
correlation between the data fetched from different source information. These different types of results can be
or platform. further processed and put into other modules to find even
The terminologies used in the Framework such as the more valuable findings.
handler which acts as the control centre and the modules Working:
which represent modularly programmed micro services Search Engine based data gathering involves
that interact with the handler, extract data from their using advanced search techniques which can help to
respective data point, further process the data and report uncover data that may be gathered with usual web search
back to the handler are further explained in the with keywords,
methodology section. Dorking involves using search engines to their full
potential to unearth results that are not visible with a
regular search. It allows to refine the searches and dive
deeper, and with greater precision, into web pages and

Published by: The Mattingley Publishing Co., Inc. 3470

 May-June 2020
ISSN: 0193-4120 Page No. 3469-3473

documents that are available online. freegeoIP API. Upon passing the IP address, this API
Example: returns the physical location of the host pertaining to the
Dork: intext:(password | passcode) intext:(username| IP address in json format.
userid| user) insite:www.target.com This dork Physical location can also be obtained by using
provides results with pages that contain keywords GeoCoding API. The response/output provided by this
password username etc within the target domain indexed API is a set of latitude and longitude coordinates for the
by the search Engine. address provided.
For example:
3. Email Reconnaissance: Request:
This module does the job of detecting if an account exists "formatted_address" : "56/69,BR Gardens, Gandhi Nagar,
with this Email-Id across various forums. It also verifies Delhi-110019".
if the specified Email-ID has been part of any data
breach, if so it also notifies the user of the source site of Response:
the breach. User names can also be extracted from email- "geometry" : {
id‟s by using regular expressions to split data based on "location" : {
certain special characters. Domain names within which "lat" : 37.4224764,
the email falls in can also be determined. Domain names "lng" : -122.0842499
can be used in who is lookup to extract valuable }}
information about it.
Working: 6. Identifying Interesting Artifacts:
1.Email-verification:Initially this modules verifies if the Interesting artifacts refers to all readable files that may be
mail address exists or if it is invalid, this operation is available on websites, servers. These files hold sensitive
performed in stages by first connecting to the target mail data and may or may not be leftover on purpose. In some
server and then verifying if the target mailbox exists in negligent cases, this has resulted in serious security issues
the mail server. to organisations since the data on the file may be of high
2.Breach data Analysis : Breach data analysis involves significance.
checking if the particular email address has been present Interesting information can also be obtained by
in data breaches,this is performed by querying the Breach using files like robots.txt, sitemap.xml files. Robots.txt
data monitoring websites such as have i been pwned file contains a list of directories that should not be
,Dehashed etc. indexed by web crawlers. This helps in extracting
interesting locations on the website.
4. Fetching information on SSL Certificates:
Since most of the domains which are hosted on the 7. DNS:
internet nowadays have a higher probability of having DNS holds major information about a website.
SSL certificate to enable secure data sharing between the This can be used to extract the IP address of the server. In
server hosted and client programs. This module focuses turn, this IP address can be passed to various modules for
on these SSL certificates to gather valuable information processing based on the IP address.
associated with owners of these certificates.
Working: Working:
1.CT-logs- Google's Certificate Transparency Program DNSSEC Zone walking:
Provides organizations an option to verify their Zonewalking is a technique where it unveils
subdomain and certificates further since it is open source Internal records if the zone is not configured properly.The
it can be leveraged to gather the subdomains of an information that can be obtained can help us to map
organization. network hosts by enumerating the contents of a zone.
2.crt.sh - Certificate Search based on crt.sh provides on
the organization's certificates issued along with the Configuration Enumeration:
respective subdomain Gather and analyse NS, MX, AXFR
and A records, as well as remote BIND version from the
5. Fetching Location Information: DNS server which may provide data about the target if
Location based reconnaissance can be done just the target has a dedicated dns server either in On-prem or
by making use of an IP address. This IP address can be cloud.
specified by the user or maybe obtained upon resolving
DNS of the host. This IP address can be used to extract its 8. Tor Based Recon:
current physical location. Tor uses a relay network to mask the original point of
data. When the data or packet reaches its destination
Working: server, it appears to have originated from the Tor exit
Upon resolving the IP address of the host, its exact point i.e. the last node in the relay. The main focus of this
physical location can be obtained by passing it to the module is to check whether the target IP has been listed

Published by: The Mattingley Publishing Co., Inc. 3471

 May-June 2020
ISSN: 0193-4120 Page No. 3469-3473

as a Tor exit point, this provides valuable Intel on

whether the target is behind a tor or relay network.

Working:
In order to detect whether the IP is using Tor, we
provide this IP as a parameter to Exonerator-Tor Metrics.
Upon passing, it checks it against a database of IP‟s
enlisted that have been a part of Tor network. If it
matches an IP in the list, it is affirmative that the IP is a
Tor node.
Example:
“https://metrics.torproject.org/exonerator.html?ip=”IP”&t
imestamp=”Timestamp”&lang=en”

9. Malicious Aggregate Index:

Malicious Aggregate Index performs the task of
analysing if the target is malicious across multiple
malicious artifact databases and further calculates and
presents a number in a scale of 1 to 10 where 1 denotes
that the target is least likely to be malicious and 10
denotes that the target is highly likely malicious
according to multiple artifacts database.

Process:
On execution of the handler which interacts with the UI
components, the input (target) from the user is obtained,
based on the nature of the target the handler invokes the
model in a certain fashion based on the interoperability
and relationship between these modules. Further the data
obtained from the execution of either single or multiple
interoperated modules is pushed into an elastic search
which is a database cum search engine that allows further
correlating data between the data points and also allows
custom search and querying of the data.

5. Sample Screenshot

6. Conclusion and Future Scope

The application can be extended by implementing
continuous monitoring and periodic assessment of the
target domain to keep track of the variation of the data
from various data points, this assists the soc and threat
hunting/ intelligence teams in large organizations to
continuously monitor for any alarming events. Further
providing a dashboard with customizable visualization
patterns and graphs provides much more insights of the
data gathered than browsing through huge dumps of data.

Published by: The Mattingley Publishing Co., Inc. 3472

 May-June 2020
ISSN: 0193-4120 Page No. 3469-3473

To conclude, reconnaissance and osint which are the

crucial parts not only in penetration testing or security
assessments but also they act in helping the defensive
security teams (Blue ) to safeguard your company's
information, henceforth it indispensable to have an
automation tool to execute all the tedious manual efforts
involved in gathering data and further providing higher
accuracy rate.

References
[1] https://www.hackerone.com/blog/how-to-recon-
and-content-discovery
[2] Open Source Intelligence Methods and Tools: A
Practical Guide to Online Intelligence by
NihadA.Hassan, Rami Hijazi
[3] Open Source Intelligence Techniques: Resources
for Searching and Analyzing Online Information
by Michael Bazzell
[4] Web Mining for Open Source Intelligence
[5] Open source intelligence base cyber threat
inspection framework for critical infrastructures
[6] https://medium.com/bugbountywriteup/whats-
tools-i-use-for-my-recon-during-bugbounty-
ec25f7f12e6d
[7] https://www.hackerone.com/blog/how-to-recon-
and-content-discovery

Published by: The Mattingley Publishing Co., Inc. 3473