Security operators need a web data collection network they can trust

The first line of defence against potential cyber-security threats is solid and reliable data. Successful security teams the world over – from teams within organisations such as banks or NGOs, to cyber-security firms themselves – use public web-scraping techniques to gain enhanced visibility over the cyber-security landscape.

Web data helps security operators better understand and prepare against any vulnerabilities present within their infrastructure, as well as those present inside the networks of other organisations. This allows them to prepare for different threat scenarios that could potentially affect them.

Data, and public web data in particular, can be used to address a variety of threats to digital security. In its simplest form it can be used to automate checks that can help discover the presence of possible malware, phishing links, fraud, information leaks and counterfeiting schemes. It can also be deployed to carry out security research, intelligence and testing on a variety of potential security threats, actors or outcomes.

If you go a step further, it can be used to test organisational infrastructure for potential vulnerabilities in networks, applications, routers, switches or appliances, and measure the threat detection and incident response capabilities of an organisation. And of course, it can be applied to address and prevent real live threats or intrusions through increased visibility and detection proficiency.

Web data providers and cyber-security

To collect data from the internet, security firms rely on collection networks to access large amounts of web data for various security purposes. Web data collection networks (IP proxy networks included) offer several benefits. The first is that the web data provider itself can help security teams overcome different blocking techniques related to multiple types of websites that may not want their content accessed.  

Second, and most important, as using web data collection platforms and proxy networks provides access to multiple public data sources, the result is increased visibility over the cyber-security landscape. You can’t overestimate the importance of having live and up-to-date reliable data to use in order to be better informed and protect against potential security threats.

The importance of compliance

As a security team or organisation it is important to choose a data provider that houses a compliance-driven platform that can be fully monitored in a transparent manner. This ensures that the security teams have a suitable environment to perform their work within, and that the ecosystem of the entire platform is not compromised. If the integrity of the web data provider is compromised, then so can the operations of the security firms, and their own networks.

Such a platform should use a multifaceted compliance process which includes a combination of internal safeguards and procedures – such as manual reviews, automated code-based prevention and technological response mechanisms – in conjunction with third-party audits to identify non-compliant activity patterns. They also need to ensure all use-case practices using the web data platform follow the overall compliance guidelines and abide by international regulations as well as by the company’s and domain best practices.

Such providers should also be certified by leading security organisations, such as Clean Software Alliance, AppEsteem, Mcafee, Avast, AVG and others which run comprehensive tests data vendors must take before actually being certified. 

Abuse.ch

One security organisation that uses such a web data network (Bright Data) is a non-profit focusing on fighting malware, botnets and exposing malicious websites using a web data collection platform. One of the platforms anti-malware initiative abuse.ch operates is URLhaus, which has so far taken down millions of websites that were used by bad actors to spread malware, using information-sharing between security researchers.

With the access help this network provides, once abuse.ch identifies a malicious website, it freely shares the information on URLhaus, which security researchers, solution vendors and law enforcement departments can access before taking the proper course of action. This includes legal action as well as using the information to protect their own networks and users from these proven threats.

The open-source information is regularly accessed by domain name system (DNS) service providers, who use it to protect millions from cyber-security threats.

Finance

Many security departments at top US banks use web data collection networks to gather information about threat actors, check for potential phishing links, and examine malware in a suitable setting.

These banks also use the platform to automatically scan the internet to identify different phishing techniques, attacks or websites related to obtaining bank assets or details, such as usernames, passwords and credit card information. This inevitably helps to safeguard its customers into unknowingly entering sensitive information on malicious websites.

Furthermore, global financial services corporations use web data products and services offered by web data collection networks to analyse e-commerce transactions and merchant info, as well as fight fraud.

Key takeaways

These are just a few of the many ways that web-scraping networks assist the security sector in fostering a more secure and open online environment for the public. Integrating with such networks enhances a security firm’s capabilities all round, from carrying out research to gathering possible threat intelligence and protecting valuable assets. Therefore, whatever a firm’s specialty may be, their capabilities are attached to the web data platform and network they hook themselves up to.

This is why it’s crucial to find a compliant partner in order to get the most reliable results. After all, these firms aim to ensure that security is top of mind, and that’s especially complex when dealing with the vast and ever-growing web ecosystem.  

By Ofir Meir, Director of Security Partnerships, Bright Data