(Author note: This paper published here anonymously)
All security systems were built on the foundation of Confidentiality, Integrity and Availability. Operating system (OS) must be able to perform its CIA duties with the help of tools, security patches and a skilled administrator. Security should always be the first priority for all new technologies. In our report, we would highlight the general principles of hardening Linux OS and explore the ways a management tool could be used to discover its OS vulnerabilities and patch up the loopholes. Through learning from experience, we would state and explain 2 past incidents which had happened on the Linux OS so as to strengthen our security knowledge in order to protect our current operating environment more effectively.
2. What was hardening a system OS
“Appropriate security settings and removing unused code.” Bret Hartman, CTO at RSA
The goal for hardening was to maintain system in the most secured state possible while maintaining its functionality and minimizing possible threat factors. Hardening was not only about hardening an OS, but also to harden all the components on the host machine. OS could be compromised due to wrong configuration, buggy device driver, other poor design system or code running on top of OS.
2.1. Various Definitions
Harden system: (H) referred to the desired state of secure system.
Baseline OS Hardening: (Bos) referred to the baseline operating system hardening.
Application hardening: (A) referred to hardening an application that resided on the OS. Examples of applications were database, web-server, mail server, DNS, DHCP, samba server device drives and other vulnerable applications.
Base Hardening: (B) base hardening = baseline OS hardening + Application (B)=(Bos) + (A)
Custom Harding: (C) additional hardening to the system, such as DMZ setting, KIOSK, Appliance, TCPWarappers etc)
Mathematical formulation of hardening system:
H = hardened system
B = base hardening
C = custom hardening
So H = B + C 
3. Why we need to harden a system OS
Any host connected to LAN/WAN or Internet must be hardened to reduce the risk of a successful attack. Hardening involved the process of eliminating as much vulnerabilities as possible. More and more complex applications and applications were susceptible to attack e.g. buffer overflow for Wu-ftp daemon enable attacker for remote code execution.
Coding and custom applications that faced the Internet e.g. Java, PHP, various Linux services and all those security settings and configurations would be exposed to outsiders’ threats.
Brute-force attacks against remote services such as SSH, FTP and telnet were some of the more common ways to compromise Linux servers facing the Internet.
3.1. Securing physical machine
Some examples of the ways to secure a physical machine included the following:
Placing hardware (host and other related peripherals) in a secured location and restricting unauthorized persons from enter the premises.
Setting bios password, disabling and enabling hardware security feature such as external boot option, bios password.
Disallowing any external device use e.g. Storage device (USB drive or external storage drive) to protect from data theft or virus attack due to infected external hardware.
Aforementioned were some of the common procedures for physical security of servers.
3.2. Steps for hardening an OS
Hardening of the OS should be the first step to safeguard system from intruder. However, hardening an OS could be very time-consuming, confusing and frustrating. Linux, like many other OS, had been designed for a wide range of roles, applications and user levels. Most distributions default installations had been designed to present the users with many preconfigured and active application as possible. 
Securing Linux system not only required the understanding of inner workings of our system, but also required us to uncheck default options to shield ourselves from default configuration.
Step 1: Data encryption
When data transmit over network, it would be susceptible to monitoring. Therefore it should be a good practice to encrypt data transmission between machines.
Step 2: Limit concurrent connection
Limiting the number of simultaneous active connection may safeguard system from brute force attack.
Step 3: Minimize software installation
Simple way to avoid vulnerabilities in software was to avoid installing unnecessary software. Most software had been installed as default e.g.: X-windows system should not be installed for Internet facing host.
Step 4: Isolating network service
If possible, a host should be dedicated to service one network. This technique could prevent host from compromising due to flaw in other network services.
Step 5: Use least privilege where possible
Assigning a super user rights unnecessarily posed a risk. One should consider carefully the user access rights, which include who should access and what type of access based on his/her knowledge and job scope. For instance, in the case of Shell access: /bin/false should be assigned for nobody, guest and other account.
Step 6: Harden operating system boot kernel (LILO or GRUB)
Hacker could execute malicious payload during boot process by modifying boot record to spread virus or control program during boot process.
Step 7: File system security
One should secure file system from unauthorized access, monitor file temper and execute file permission.
Step 8: User account security
Next step was to rename administrator (root) account and use complex password with password aging and logout policy.
Step 9: Logging and auditing
One should also audit and monitor the log activity to trace any unauthorized access and potential risk.
Step 10: Removing redundant services
Next, we should disable or remove all unnecessary services to secure system, free up system resources, run require services with unprivileged accounts and patch up OS regularly.
Step 11: Hardened installed applications
OS could be compromised due to flaws in applications e.g. buffer overflow enabled remote code execution, poor design application could pose security risk and stability of the operating system that running on.
Step 12: Use of host base firewall or default firewall feature
Lastly, one should use commercial supported host base firewall or Linux native enhance firewall feature to secure host from external threat. e.g. Iptable for SELinux.
Below was a diagram of an illustration of enabling firewall upon installation.
4. Benefits of hardening
Hardening OS played an important part for system to be more reliable, efficient, secure thus providing us an optimize performance. Hardening also reduced security risk from vulnerable outside threats.
4.1. Examples of OS hardening benefit
- Faster recovery in case of system failure
- Detection of inaccurate system configurations
- Better-secured information systems
- Lower risk of system failure
- Improved configuration management for security related settings
- Compliance with regulatory requirements
- Ease of auditing systems
- Prevention of financial loss due to service interruption and data loss
As illustrated in the figure above marked with red boxes, many of these exploits might be detected or prevented through hardening .
4.2. Issue with hardening
In most of the cases, administrators were unable to identify what to harden. Due to insufficient in-depth knowledge of OS, hardening might lead to damage or lockdown to the system. Hardening would be a time-consuming affair and complex process. Hence, justifications of such needs as well as management support were usually necessary.
5. Hardening Trend & Tools
Securing and hardening network equipments were the main dominant factors since evolved of Internet era. All these tasks were performed by experienced professionals and it had currently reached a maturity level. In contrast, OS and application hardening had been undermined due to the fact that majority of the system administrators did not have enough knowledge to deal with it. Failing to focus on application and OS hardening had brought unnecessary high costs to many organizations. The current trend was hence moving towards OS and application hardening. 
We would go on to explore some of the tools we could use in our environment.
HIPS – A host base IPS resided on specific host IP address for a particular computer. HIPS worked in heuristic antivirus detection methods, since it did not need continuous updates to stay ahead of new malware; evil code need to modify the host or other application residing on the system to achieve evil hope. Comprehensive HIPS system would be able to detect some of the resulting changes and prevent the action by default or notify user for permission. 
Yum –Yum was Linux native complete software management system tools to install and upgrade package. Currently most of the Linux distribution offered web-based live package upgrade option.
Altiris – Altiris total management suite was designed as a policy base software management, providing automated patch management, automated application self healing and configuration settings management. 
Patchlink – Patchlink patch management solution provided discover, remediate, audit, generate reports automatically, and continuously mitigated vulnerability risks and enforcing security and compliance policies. 
Tripwire – Tripware for servers offered configuration control of system by alerting IT to improper changes to key system files, directories and registries. 
Snort – Snort was an open source network intrusion prevention and detection system (IDS/ IPS). It was the most widely deployed IDS/IPS technology worldwide. Snort had become the de facto standard for IPS. 
6. Nessus case study
Nessus was a very useful tool for corporations or individuals to scan for vulnerabilities, test network devices, discover applications and system security flaws. Firstly, we need to specify a security policy by defining what areas we wished to scan and detect. We could then specify the ports to be used for scanning.
Once the scanning process finished, it would tabulate a report listing with 4 main columns and list the different levels of security breaches as well as the open ports of the OS. It would then provide the solutions and recommendation of minimizing the risks involved.
Nessus used plugins to maintain it security database, so it was important for Nessus to update the database regularly from its server to get the latest security fixes and updates. Fixing every issue identify by Nessus ensures that Linux OS was operating at its most secured and up-to-date level. We should also make sure our OS was updated with the latest patches and fixes.
Before we updated the OS, the number of high severity cases discovered was 12. For our new install OS, we should always do an update to the latest fixes, and for our case. We were using the command yum update for our Linux OS.
We could see that after the update process, the number of high severity cases had been drop to 0, and the middle and low severity cases lowering in numbers.
7. Linux security incidents & solutions
Next, let us take a look at a few examples of attack on Linux Operating Systems. By reviewing these security incidents, we could see how OS hardening could help us to reduce the risk of intrusion.
Incident 1: Websites Mass Defacement
In Nov. 2009, some websites hosted on Daily Internet Services (UK-based web hosting company) had been massively defaced by a hacking group named TH3_H4TTAB. All pages with name like index.html, index.htm or index.php were replaced with an image of cartoon Linux penguins.  (Figure. 1.1)
Figure 1.1 Tux with 3 different poses
After noticing the occurrence of this attack, Daily Internet Services carried out the process of restoration from the backups. The company also took out several servers from their web cluster for investigation and diagnosis. Meanwhile, the company strengthened their security policy on the servers. So far, there was no official report about the precise means of this attack. However, it was believed that the outdated PHP module in the web server played an important role in this incident, as it had been upgraded to a newer version right after this attack. Obviously, if the newer version of the PHP module was available before the attack, and the server administrator had upgraded it, the hacker would not get any chance to exploit the vulnerabilities existing in the older module.
Incident 2: Fedora Project Infrastructure Intrusion
In Aug. 2008, the Fedora system administrators found that some packages on the package build and signing server had been modified. Luckily, "the modified packages were discovered before anyone accessed the system to sign any packages". Otherwise, the fraudulently signed package with some possible backdoors added by the intruder could be distributed to the whole Fedora community, it could become a terrific disaster.
Figure 1.2 had been drawn out to help understanding how intruder penetrated into Fedora’s secured infrastructure.
Figure 1.2 Fedora Project Infrastructure Intrusion
Some fedora system administrators never used passphrase to safeguard their SSH private key. The SSH keys were used to auto login Fedora servers. Intruder had stolen the SSH key from one of the administrator, and made use of it to login to Fedora’s package build and signing server. Since the intruder managed to login as system administrator, he had whatever privileges that administrator had. The intruder proceeded to build the modified version of OpenSSH and some RPM. However, the intruder could not sign those fraudulent packages that he had built, because the signing key passphrase was protected on the server. Even if the intruder had not cracked the password of the signing key, it would be a matter of time before he could crack it. So, for precautionary reason, Fedora team deployed a new signing key. After this incident, Fedora had established a new policy and required all their administrators to safeguard their private key using passphrase.
8. Summary & Conclusion
“Security is not a goal, it is a process, Security is not a product, it is a mindset. Security is a never ending task. If you think you are secure, just wait a few minutes until the next spoilt is released. Security is like breathing – If you stop, you die.”(Pezzo – May 2001) 
Securing a system would be a complex process and required very careful assessment, consideration, relevant domain knowledge and skills. Establishment and enforcement of proper security policies and procedures were also essential part of hardening our system. Security should be an iterative process and should never remain stagnant. From the two incidents aforementioned, we could conclude that to secure our information environment, we had to not only harden the baseline operating system, but also harden the applications and revamp people’s security mindset.
 ISACA Austin Chapter (2009), Platform Hardening, Retrieved on Feb 01, 2010, from http://www.isacaaustin.org/uploads/9/4/9/1/949112/10-6-2009_platform_hardening.pdf
 eTutorials.org(2009), OS Hardening Principles, Retrieved on Feb 01, 2010,
 CSI (2009), CSI Computer Crime and Security Survey, Retrieved on Jan 30, 2010, from
 Wikipedia (2010), Intrusion Prevention System, Retrieved on Feb 04, 2010,
 Symantec (2010), Altiris Total Management Suit, Retrieved on Feb 04, 2010,
 Patchlink (2010), Patch management Solution, Retrieved on Feb 04, 2010,
 Tripwire (2010), Configuration Control for Servers, Retrieved on Feb 05, 2010,
 Sonrt.org (2010), Intrusion Prevention, Retrieved on Feb 05, 2010,
 Lucian Constantin (2009), Web Host Hack Results in Mass Defacement, Retrieved
on Feb 04, 2010,
 Paul W. Frields (2009), Update and Report on Fedora August 2008 Intrusion, Retrieved on Feb 04, 2010,
 Secify Security Consulting (2010), Retrieved on Feb 05, 2010,