I was called by a client to investigate a strange behaviour in their network infrastructure. It turned out to be a major cyber security breach. Of course, I cannot disclose the full details of the incident as Non Disclosure Agreement rules apply. But I wanted to share some lessons and tips learned from that incident to help network and security administrators.
How Strange Is Weird?
It all started with a strange behaviour on the core network infrastructure. Why should a routing device at the core of the network attempt to make regular outbound DNS requests to IP addresses that appear to be random home computers in Asia?
Moreover, it is not usual to have core routers attempt ro make constant HTTP requests to unknown home computers in Easter Europe and then flood their own local network with DDoS attacks to bring down critical network services. These are some of the strange symptoms that caught my attention.
Paranoid Rootkit
Further investigation using sys-admin and network tracing tools revealed that some of the core network devices had been compromised.
The attacker installed a kernel rootkit that took over the network stack of those devices allowing full control of those devices. Compromised devices were allowed to join a global botnet that was using several command and control centres spread around the globe. Further ‘Cyber Crime Scene Investigation’ (CCSI) was required to get to the bottom of what this rootkit was up to.
CCSI Miami
Having established the root cause of the strange network behaviour, the challenge was now to gather and analyse all the information in order to better understand what else could have been infected in the core network. The rootkit malware seemed to have some sort of awareness about its local network surroundings.
With full control of the infected device network stack, the malware was able to monitor local activities and decide when, how, and what to connect to in order to obtain further commands under certain conditions.
By simulating the right conditions the malware was able to trigger and replicate several of its assaults on the local network. This allowed the collection of useful information for analysis, including traffic patterns, botnet architecture, and trigger conditions that provided a definitive answer of the inner workings of the malware rootkit.
Killing The Zombie Device Dead
Following a detailed analysis of network traces and system processes behaviour on the compromised device it was clear that this device was now a fully grown ‘zombie’ and had to be given a new life. Having collected all the evidence, a fresh and clean install was required to ensure no traces remained from the earlier breach. Cleaning up the offending device was just the start. Ensuring that the rootkit had not infected any other network components was crucial. A thorough analysis and monitoring of the whole network infrastructure and locked down was required to ensure that the malware was completely removed and cleanse. Of course, the lessons learned highlighted below had to be applied in order to strengthen the security of the overall infrastructure and reduce the likelihood of such breach occurring again in the future.
Lessons Learned
From this incident I have drawn 7 lessons. Most of which are common sense guidelines to cybersecurity defence-in-depth strategies. Cybersecurity is an on-going battle between technology, people, and processes. The three must work together to ensure that network infrastructures and applications remain secure and always available. Here are some guidelines that network & security administrators can apply to strengthen their networks:
- Tightly control remote access: Never allow access to a network device from the Internet without strong authentication. Always use 2-factor authentication with a strict Access Control List (ACL) to restrict what, where, when, and who can manage critical network devices.
- Defend in many layers: A defence-in-depth architecture should be followed to strengthen the security of the overall network infrastructure. By doing so this will greatly reduce the risk of a single component infecting the entire network infrastructure. For example, at the technology layer, a combination of next generation firewalls at the edge of the network, intrusion detection systems, and advanced endpoint detection and response is a minimum requirement.
- Monitor, detect, and remediate: A robust monitoring, detection, and remediation system and processes should be in place to establish a baseline of normal network traffic behaviour. From that baseline, anomalies can be detected quickly and remediation applied promptly. Security analytics must play a key role here.
- Plan and prepare for remediation: The question is not if a network will be hacked but when. Given enough time and will any network can be hacked therefore it is necessary to have regular drills and a clear response plan to prepare for a major breach.
- The cyber battle never stops: Cyber security is an on-going battle between people, processes, and technology. Technology alone cannot guarantee cyber security but a mix of the three will provide the strongest cyber defence.
- Tightly control all inbound and outbound flows: Most companies strongly control inbound access to their network infrastructure and services, but often, outbound access to is left unchecked. By putting in place strict rules for outbound connectivity a single device breach will remain isolated and reduce the chance of a malware infection spreading throughout the network. For a malware rootkit this means starving it from connecting to its control and command centres and stopping it from downloading further commands as a stepping-stone to amplify its attacks.
- Establish a secure baseline of device build: All network devices should be hardened and added to the network with a secure baseline to ensure consistency and to avoid basic mistakes. This procedure should be much stricter for all Internet facing devices.
Conclusion
This quote from Robert Mueller illustrates the challenge that network & security administrators face to protect their digital assets.
“I am convinced that there are only two types of companies: those that have been hacked and those that will be. And even they are converging into one category: companies that have been hacked and will be hacked again.” ~ Robert S. Mueller, III Director Federal Bureau of Investigation
The question of a cyber breach for any network infrastructure connected to the internet is not if but when. Therefore, by putting in place the right controls from a technology point of view, cyber attacks can be detected swiftly and stopped before damage is done. Moreover, people and processes play a key role in strengthening the security posture of an organization, therefore, continuous training and process tuning to adapt to the constantly changing cyber threats is key.
