detailed kernel bootup debug messages

Users like me prefer as much debugging output in applications as possible because this makes performance and stability issues easy to understand. This also allows one to see what all is going on behind the scenes for crash debugging. A few amendments can be added to the kernel boot parameters of a Linux kernel to allow for detailed messages and verbose output:

By modifying the 'kernel' line in grub or editing the relevant boot file under /boot/grub and adding the following[in bold]:

kernel /boot/vmlinuz26 root=/dev/disk/by-label/Arch ro debug ignore_loglevel log_buf_len=10M print_fatal_signals=1 LOGLEVEL=8 earlyprintk=vga,keep sched_debug

heavy debugging can be easily activated.

debug = activates internal debugging.
ignore_loglevel =  ignores any sort of log level and maximizes debug output.
log_buf_len = increases the log buffer length to 10 MiB
print_fatal_signals = print any fatal signals.
LOGLEVEL = enables level 8 logging.
earlyprintk = enables early printing of messages to the vga screen
,keep = keeps the messages on for longer.
sched_debug = Enables verbose scheduler debug messages.

Ref:
http://www.kernel.org/doc/Documentation/kernel-parameters.txt
https://wiki.archlinux.org/index.php/GRUB#Advanced_Debugging

examining firewall log entries...

While going through the usual iptables log, its pretty interesting to see what all lurks out on the world wide web. There are some usual and unusual entries such as:

localhost kernel: iptables DENIED: IN=ppp0 OUT= MAC= SRC=xxx.xxx.xx.x DST=yyy.yyy.yyy.yyy LEN=60 TOS=0x00 PREC=0x00 TTL=45 ID=61742 DF PROTO=TCP SPT=4993 DPT=23 SEQ=2106909432 ACK=0 WINDOW=5808 RES=0x00 SYN URGP=0 OPT (020405AC0402080A012E13D60000000001030300)

localhost kernel: iptables DENIED: IN=ppp0 OUT= MAC= SRC=xx.xxx.xxx.x DST=yyy.yyy.yyy.yyy LEN=40 TOS=0x00 PREC=0x00 TTL=95 ID=256 PROTO=TCP SPT=6000 DPT=1433 SEQ=2031616000 ACK=0 WINDOW=16384 RES=0x00 SYN URGP=0

localhost kernel: iptables DENIED: IN=ppp0 OUT= MAC= SRC=xxx.xxx.xx.xxx DST=yyy.yyy.yyy.yyy LEN=40 TOS=0x00 PREC=0x00 TTL=119 ID=256 DF PROTO=TCP SPT=12200 DPT=1080 SEQ=474681 ACK=0 WINDOW=8192 RES=0x00 SYN URGP=0

Which upon analysis reveal some pretty interesting facts...

1. Firstly I notice a lot of TCP Syn packets being denied at ports 1080, 3128, 8000, 8080 etc
2. Also a lot of attempts at ports 1433, 1434
3. And finally some random attempts at 22, 23, 25

Whats interesting is that all the source IPs are random and in some situations, there is a flag on 2 ports per IP.


These are what I think may be the causes:

1. A simple lookup would reveal the purpose of these ports as the following:
1080 : socks
3128 : squid-http
8000 : http-alt
8080 : http-proxy
With this, one can easily deduce that many systems over the web are intentionally/unintentionally checking for open proxies on random IP addresses. These may be normal boxes with scripts/programs running that generate random IPs or using old server connection logs. They may also be infected systems with malware performing the same task. This is one of the many ways lists of open proxies surface on underground websites.

2. These are the ports on which Microsoft SQL server typically listens on and may well be sought after by full blown malware looking for newer prey or automated scans running on a cracker's box.

3. These are usually ports for the typical ssh, telnet and smtp services and are most probably being scanned for vulnerabilities or vulnerable configurations by malware on rooted boxes or automated/active scans running on a cracker's box.

Its quite funny to see the intensity of these reports on the logs. Quite a few popup every hour or so. It is a reminder that a by just connecting a system to the internet without properly securing it can be pretty fatal. Its a dangerous WWW out there and web safety requires proper measures to be taken before connecting the wire.