When I founded DecryptedTech it was with the intention of finding a better and more realistic way to represent hardware and software to the consumer. I had noticed, after many years of testing, that the industry was more than happy to run a few scripted benchmarks, give an award and publish “results” with little to know actual analysis on what anything really meant. It was maddening to see this happen time and time again from sites that are well known and considered to be experts on the topic.
The issue does not stop at motherboards and CPUs though. It exists in all verticals of the technical market. Gaming reviews are another area where the distributor has massive sway on what (and when) information is released. If a sites does not play along with those requirements, well they get cut off (We know this from actual experience). All vendors are exercising more and more control over what can and cannot be said in a review. They are more than happy to cut off a site for publishing information (factual or not) that does not fall in line with what they want. This control by the industry actually hurts the consumer in a very significant way by allowing inaccurate (or incomplete) information to be released about a product.
In the security industry the issue is even more impactful. By using scripted or canned tests a developer can plan for these tests and ensure a good score. This means that a potential buyer might make a choice that leaves them unprotected and vulnerable to attack. Although it is important to make sure known threats are protected against, security products must be able to defend against more than yesterday’s threats. The idea that simple signature and hash checks can protect an environment is no longer one that we can tolerate in our security tools.
Threat actors can be lazy, but even when they are lazy they are creative. I have observed threat actors mutate even common tools like PSExec to get around hash blocking in a target environment. The tool is the same one they wanted to use, but the hash has been changed so that it does not pop on the threat list. I have also watched a live monitor of a compromised C2 server where the malware used was mutated multiple times a day to ensure that it was not being caught. This same system was also rekeying the encryption used to transmit control data as often as twice a day. They are doing this to get around your typical consumer grade AV/AM software and it is also working against corporate AV/AM software as well.
The next generation of AV/AM software has to be smarter than what we have today. It must not only be able to recognize malware from a list, but also when normal software or tools are being abused or used in a manner they should not be. This is not an easy task, but it is a necessary one. The next generation of attacker has a very solid understanding of how anti-malware and even process/application control solutions work and can get around those limitations surprisingly quickly. The use of custom malware combined with normal debugging and network tools is becoming more and more common so the old signatures are out the window as area most heuristic systems. This also means that scripted testing based on “known” bad samples and heavy handed actions are also out of the window. Using them paints an incomplete and inaccurate picture of what a solution can do and gives the consumer a false sense of security. It also enables solution developers to continue doing the same thing without changing anything. As long as they keep cashing checks (thanks in part of inaccurate testing) there is no reason to do anything different or new. If the testing changes to expose the inherent weaknesses of their products, the money will slow down and the industry will finally make the changes needed.
Of course this change needs to take place across the entire industry and not just security, but it would be great if it changed for security tools sooner rather than later. After all buying a crappy motherboard based on an inaccurate review is not going to result in the loss of 1 million Credit Cards, buy a bad AV/AM solution could.
