Part I: An Introduction To OSINT Research For Protective Intelligence Professionals

Our two part series is divided into the following segments: Part I, in which we highlight the context in which protective intelligence professionals use open source intelligence (OSINT) and Part II, a brief overview of OSINT collection methods and how they relate to our organizations’ protection strategies.

What makes OSINT research so important to protective intelligence professionals is that it’s not only a tool for us to use for good, in collecting information to support decision makers, but also a potentially malicious tool to be used against us (and our organization’s assets). Having a basic understanding of OSINT supports our proactive assessment of threats, and it also supports our assessment of OPSEC/information related vulnerabilities of our own organizations. In understanding OSINT collection methods, we not only better understand how to protect our organization’s sensitive information, but we also stay up to date with potential access points that an adversary may infiltrate and exploit.

We will begin our introduction with an overview of the intelligence cycle, then move on to important background information about the internet.

OSINT & The Intelligence Cycle

I will let one of the authorities in this area define OSINT for us. In “Open Source Intelligence Techniques” 5th Edition by Michael Bazzell, he cites the following definition:

“Open Source Intelligence, often referred to as OSINT, can mean many things to many people. Officially, it is defined as any intelligence produced from publicly available information that is collected, exploited, and disseminated in a timely manner to an appropriate audience for the purpose of addressing a specific intelligence requirement. For the CIA, it may mean information obtained from foreign broadcasts. For an attorney, it may mean data obtained from official government documents that are available to the public. For most people, it is publicly available content obtained from the internet.”

You may find it helpful to consider that definition of OSINT in conjunction with the following definition of intelligence:

“Intelligence deals with all the things which should be known in advance of initiating a course of action.” [1]

These definitions are easily understood in the context of the intelligence cycle (pictured below).

Here’s one simple explanation to describe the intelligence cycle:

The intelligence cycle begins with direction from a decision maker. Analysts then define the problem on a granular level and develop a specific plan. Planning is an essential step because the scope of the research needs to be identified, the objectives of the research need to be evaluated in terms of feasibility (resources/time), and information to be collected (and its sources) needs to be identified.

While collection, processing, and analysis are self-explanatory – gathering the information and using inductive/deductive reasoning to conclude what the information means for decision makers, production on the other hand is very much dependent on the consumer of the intelligence product. For informal intelligence products that stay within the department, it could be as simple as a 300 word email summarizing the analyst’s conclusions, or for a Fortune 100 client’s HR/Legal team it may be a 30 page document with methodologies, supporting documentation, and more. Lastly, analysts need to seek feedback from the consumer (end user) to learn how to better serve their goals and needs going forward.

Facts About OSINT

What makes OSINT so relevant is the proliferation of mediums and users across the web, giving analysts evermore useful information to collect from an increasing number of sources. This can be demonstrated with a review of the historical growth of the largest current social media platform, Facebook. The number of active Facebook users in 2004 was approximately 1 million, 8 years later it had grown to 1 billion, and 5 years after reaching 1 billion, they hit 2 billion in 2017. And to provide you with even more context, we would like to cite a statement attributed to Eric Schmidt in 2010 (former CEO of Google):

“Every two days now we create as much information as we did from the dawn of civilization up until  2003… That’s something like five exabytes of data,” he says.
Source: TechCrunch

The content that Eric Schmidt is referring to is the same content (Facebook posts, Tweets, Instagram updates, etc.) that analysts and investigators are collecting to support their security intelligence programs.

Now that we have seen two examples of the proliferation of social media use, let’s examine some quick facts about the most popular social media platforms and key judgements from the Pew Research Center’s recent report: “Social Media Use in 2018.”

Monthly active users per social media platform (approximation):

 

• Facebook: 2 Billion
• YouTube: 1.5 Billion
• Instagram: 700 Million
• Twitter: 328 Million
• Snap Chat: 255 Million

*Source: Tech Crunch

Conclusions from Pew Research Center: (United States) “Social Media Use in 2018”

• “Roughly three-quarters of the public (73%) uses more than one of the eight platforms measured in this survey, and the typical (median) American uses three of these sites. As might be expected, younger adults tend to use a greater variety of social media platforms.”
• “Roughly two-thirds of U.S. adults (68%) now report that they are Facebook users, and roughly three-quarters of those users access Facebook on a daily basis.”

Context: These excerpts come from “Social Media Use in 2018” by the Pew Research Center. In this document, Pew Research included the following social platforms in their study: Facebook, YouTube, Twitter, Instagram, Pinterest, Snap Chat, LinkedIn, and WhatsApp. The report was published on March 1, 2018.

Defining Collection Sources: Surface Web, Deep Web, And Dark Web

The term surface web describes those parts of the internet that have been indexed/crawled by search engines such as Google, and are thus searchable via various search engines. The deep web is the remainder of the internet that has not been indexed by search engines. Generally, it is accepted that about 95% of the internet is the deep web, while about 5% of the internet is the surface web (see iceberg image below).

So, what parts of the internet are not indexed by Google and the other major search engines? Generally, most online databases, communities, networks, etc. are not indexed by Google. Example – The most popular photo sharing website on the internet, Flickr, has over 1 Million public photos uploaded to it PER DAY. Repositories of information (in this case photos) such as these are generally not indexed by search engines. In addition, website owners can add code into their webpages that instructs search engines to specifically NOT index their pages (Google the phrase “Robots.txt” to learn more).

Lastly, the dark web is just a subset of the deep web, with two general characteristics: (1) they are websites that are not indexed by the major search engines and (2) they have to be accessed by using special software, methodologies, and related permission layers.