Black Box Testing is a method of examining the functionality of a system or software without prior knowledge of its internal workings. Metaphorically speaking, the test is a black box hiding something inside that a tester doesn’t know and can’t see. To determine what’s in the black box, the tester must fumble in the dark, so to speak, to discern as much information as possible about the environment. The testers must discover for themselves information they leverage and see how deep in the system they can get.
Black box testing is notable as it removes testing bias that can stem from focusing
solely on the scope that is designated within a testing contract. It aims to ensure that discoverable issues aren’t overlooked. That is why it is critical to invest the appropriate amount of your cybersecurity budget on comprehensive black box testing.
But black box testing isn’t a cure-all. It’s effectiveness depends greatly on the skill and experience of the testers, especially in reconnaissance and discovery, not to mention the time allocated to the testing. In effect, black box testing tests the tester’s ability to discover your environment! Which can be very effective, but perhaps not as cut and dry as you may think. If the testers come back with minimal findings, is it because they found no risks or did they just not dig deep or well enough?
Even within a larger testing scope, there is simply no way to come even remotely close to the almost infinite effort attackers collectively leverage to exploit weaknesses and pathways to compromise. The combined effort of all criminal hacking activity that daily attacks your public facing systems and applications is astronomical. It is comprised of spray and pray attacks, port scanning, ransomware scanning, worms and targeted attacks by highly skilled cybercriminals whose objective may be monetary, political or activism. Regardless of their motivation, their combined nefarious effort is astronomical, continuous and skillful. It never stops, day or night, week in and week out. And sooner or later, a technique or method will find its matching weakness whether it is automated or there is a hooded, sun-glassed, dark roomed criminal hunched behind the keyboard. In contrast to this endless siege, the pen testing team is limited to a glimpse of the environment and current state for a period of days or at most a couple of weeks.
Lastly, attackers are compensated handsomely either by the victims of ransomware or by the organizations and nation-states that fund them. Cybercriminals level of payment or reward usually far outweighs the earnings of even the best ethical hacker or pen tester.
What does an organization realistically achieve with a black box pen test? Maybe some low hanging fruit are discovered. Perhaps some stub networks and services are identified that weren’t included in the scope. But do those discoveries truly bring you the peace of mind and sense of cybersecurity you were seeking? Should you blindly trust the report and believe that the team discovered that one critical issue that will make or break your career? Maybe not.
The fact is, there are better ways to spend your budget – approaches that might not find every issue, but will create a greater awareness of the vulnerabilities and entry points your adversaries are discovering and leveraging to craft attacks.
Zero-Knowledge testing is an assessment aimed at discovering what information is readily available about an organization. This data can include public IP address space, system information and applications, email addresses and email format (for phishing). It may also include information on systems that are causing errors and showing up in a database, past breach information such as account and password dumps, DNS information about the organization that may span the core DNS and ASN name space and beyond. It can also include other open source intelligence (OSINT) information that may be of value to cybercriminals.
What sets zero-knowledge testing apart from black box testing is that while it is bundled with penetration testing as an add-on service, it is not the only source used in further testing. Instead, it is part of the test and includes a report and discussion, because some pieces of information can be just as much of a weakness as a missing patch or non-secure configuration.
This is a relatively recent term and perhaps more of a buzz word, but essentially expands on the concept of zero-knowledge and is also a continuous service. In addition to OSINT, dark web scanning may include monitoring your digital footprint and looking for “squatters” who take your content, images, names and the like to set up shop pretending to be you.
One of the best approaches to defending against the ongoing onslaught of criminal hacking is more frequent penetration testing and vulnerability scanning, if you are not already doing so. A bit like brushing your teeth, frequent testing and scanning reduces the risk of issues cropping up and getting out of control like a cavity. Good hygiene is important!
Vulnerabilities are both created and discovered. Software vulnerabilities are created during the development of the code. Usually they are not known to the developer or vendor and don’t get caught in the QA process. Sometimes they are released as a bug, but the risk is underestimated. However, most lie dormant within the code until they are discovered by users, researchers and criminals alike. Whether they become known depends on the discovery source. Criminals and some researchers keep these discoveries to themselves to use or sell to the highest bidder. But those software vulnerability discoveries that make it to the manufacturer or into public domain will typically be addressed, given a name and eventually a patch or method to address the weakness is created. Weaknesses also stem from configurations and use of protocols.
Combining black box or zero-knowledge discovery with traditional penetration testing through a process of phased testing is called Grey Box. Testers conduct discovery and OSINT research, document their findings and how they were discovered, discuss any potential attack pathways based on that information, and review with the company what was missed.
In the next phase, those issues that were missed, like IP ranges and systems, are included in further testing. Additionally, the testers and the client have a conversation about this part of the scope to understand why these things were not discovered.
In Addition to methods discussed above, taking a proactive approach to testing can be very helpful and is part of the pathway to greater maturity in your penetration testing and risk management practices. If you have conducted several general or network penetration tests over the past years, it might be time to try something different and perhaps change the objectives and testing scope.
For example, instead of a network penetration test, pick a few of your most critical applications or systems and focus the testing on gaining access to those or their related data. We call this application testing or application penetration testing and it typically takes longer than a general network penetration test even if it is focused on a single IP address or URL. The effort is dependent on the complexity of the application or system as well as the environment that it lives in.
Sometimes testing is exclusively focused on the application front end such as user login page, user levels, administrator login etc. The aim is to discover application and code vulnerabilities within the pages of the application and that are embedded in the underlying code. Sometimes the testing scope includes the backend of servers, databases and access systems that may be hosted in the cloud. Sometimes those are accessible through the application front end and become part of the scope or even the target or flag to capture. If your Penetration Testing and Risk management program is fairly mature, it may be a good idea to explore the Mitre ATT@CK! Framework. This framework maps pathways and techniques for attacks as well as their defenses. It is very extensive and can be used as the basis to create testing strategies and plans. It is vast and will take time to work through and become familiar with.
Click here to listen to “Cyberitaville Episode 8: The Two Faces of Artificial Intelligence.”