Go Back   Singapore Web Hosting Talk > LEARNING CENTER > 1-2-3 Linux/Unix > Monitoring

Monitoring Main steps in monitoring your server.

Thread Tools
Old 09-03-2009, 11:49   #1
The Dude
SGWHT Veteran Member
Join Date: 07-02-2003
Posts: 664
The Dude is on a distinguished road

Before the checklist my advice is:

- Dont wait to be Hacked! Use the various HOW-TOs here to secure your box from day one

- Dont think of security as something you need to do after the fact,security must be your hourly/daily mindset

Now,theres a lot of stuff on this subject,but to start off with,this section from the Red Hat Linux Guide is a good place to start.

The first thing an intruder typically does is install a "rootkit". There are many prepackaged rootkits available on the Internet. The rootkit is essentially a script,or set of scripts,that makes quick work of modifying the system so the intruder is in control,and he is well hidden. He does this by installing modified binaries of common system utilities and tampering with log files or by using special kernel modules that achieve similar results. So common commands like ls may be modified so as to not show where he has his files stored. Clever!

A well designed rootkit can be quite effective. Nothing on the system can really be trusted to provide accurate feedback. Nothing! But sometimes the modifications are not as smooth as intended and give hints that something is not right. Some things that might be warning signs:
  • Login acts weird. Maybe no one can login. Or only root can login. Any login weirdness at all should be suspicious. Similarly,any weirdness with adding or changing passwords.
    Wierdness with other system commands (e.g. top or ps) should be cause for concern as well.
  • System utilities are slower,or awkward,or show strange and unexpected results. Common utilities that might be modified are: ls,find,who,w,last,netstat,login,ps,top. This is not a definitive list!
  • Files or directories named "..." or ".. " (dot dot space). A sure bet in this case. Files with haxor looking names like "r00t-something".
  • Unexplained bandwidth usage or connections. Script kiddies have a fondness for IRC,so such connections should raise a red flag.
  • Logs that are missing completely or missing large sections. Or a sudden change in syslog behavior.
  • Mysterious open ports or processes.
  • Files that cannot be deleted or moved. Some rootkits use chattr to make files "immutable" or not changable. This kind of change will not show up with ls or rpm -V,so the files look normal at first glance. See the main pages for chattr and lsattr on how to reverse this. Then see the next section below on restoring your system as the jig is up at this point. This is becoming a more and more common script kiddie trick. In fact,one quick test to run on a suspected system (as root):
    /usr/bin/lsattr `echo $PATH | tr ':' ' '` | grep i--

This will look for any "immutable" files in roots PATH,which is almost surely a sign of trouble since no standard distributions ship files in this state. If the above command turns up anything at all,then plan on completely restoring the system (see below). A quick sanity check:
# chattr +i /bin/ps
# /usr/bin/lsattr `echo $PATH | tr ':' ' '` | grep "i--"
---i---------- /bin/ps
# chattr -i /bin/ps

This is just to verify the system is not tampered with to the point that lsattr is completely unreliable. The third line is exactly what you should see.
  • Indications of a "sniffer",such as log messages of an interface entering "promiscuous" mode.
  • Modifications to /etc/inetd.conf,rc.local,rc.sysint or /etc/passwd. Especially,any additions. Try using cat or tail to view these files. Additions will most likely be appended to the end. Remember though such changes may not be "visible" to any system tools.

Sometimes the intruder is not so smart and forgets about roots .bash_history, or cleaning up log entries,or even leaves strange leftover files in /tmp. So these should always be checked too. Just dont necessarily expect them to be accurate. Often such left behind files or log entries will have obvious script kiddie sounding names,e.g. "r00t.sh".

Interpreting sniffer output is probably beyond the grasp of the average new user.

As mentioned,a compromised system will undoubtedly have altered system binaries,and the output of system utilities is not to be trusted. Nothing on the system can be relied upon to be telling you the whole truth. Re-installing individual packages may or may not help since it could be system libraries or kernel modules that are doing the dirty work. The point here is that there is no way to know with absolute certainty exactly what components have been altered.

You can use rpm -Va |less to attempt to verify the integrity on all packages. But again there is no assurance that rpm itself has not been tampered with,or the system components that RPM relies on.

If you have pstree on your system,try this instead of the standard ps. Sometimes the script kiddies forget about this one. No guarantees though that this is accurate either.

You can also try querying the /proc filesystem,which contains everything the kernel knows about processes that are running:
# cat /proc/*/stat | awk '{print $1,$2}'

This will provide a list of all processes and PID numbers (assuming a malicious kernel module is not hiding this).

Another approach is to visit http://www.chkrootkit.org,download their rootkit checker,and see what it says.
The Dude is offline   Reply With Quote


Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT +8. The time now is 02:32.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.
Copyright (C) 2002-2012 Brought to you by Singapore Web Hosting Talk (SGWHT). All Rights Reserved.