Monday, November 27, 2006

The OpenSolaris Quiz - FOSS.IN/2006

These are OpenSolaris Quiz questions presented in FOSS.IN/2006 These answers are mine and disclaimers apply here. I am putting it here, since I found the questions good enough to make me interested toward OpenSolaris

Edit: Please consider the questions only and not the answers. Attended the BOSUG, and Ananth helped in confirming a few answers. Some went over my head though









The OpenSolaris Quiz - FOSS.IN/2006




1. Which Computer Scientist who co-founder the Apache HTTP server servers as a board member of the OpenSolaris CAB.

Roy Thomas Fielding



2. On a Quad core, dual processor machine, how many times can the DTrace probe 'profile:::tick-17hz' fire in a span of 3 seconds



17



3. What does this Dtrace one-liner do ?

proc:::signal-send { printf("%s - %s". args[2], args[1] - }



Trace all the signals sent to all the processes running on the system.



4. 256 Quadrillion Zetabytes is a significant number as regard ZFS. This is the amount of data required to fillup a ZFS filesystem. How many digits are there in that number ? (when expressed as bytes)



39

5. Expand COW in the context of ZFS.



Copy on Write.



6. FreeBSD = Ktrace, Linux = strace , Solaris = ?



truss (dtrace is way too advanced :-))



7. Which binary / libc function enables seamless execution of 64 and 32 bit binaries on a 64 bit OpenSolaris machine ?



isaexec.



8. How do you encrypt a file with AES algorithm using the OpenSolarius Cryptographic Frameword ?



encrypt -a aes -i file.txt



9. Which is the distro wholly created by the Bangalore OpenSolaris community ?



Belenix



10. How many privileges does OpenSolaris have by default ?



48



11. Which OpenSolaris project is underway to introduce virtualization in networking ?



CrossBow.



12. What is the latest Source Code Management System used to OpenSolaris development ?



Merucrial / Hg



13. Write down a minimal fool proof "C" function which takes a (void *) pointer as the argument and returns :

0 - if it is an invalid userland pointer

1 - If the pointer points to a valid userland address backed up by a page. No core dumps, No signal handling. (Elegant Solutions = ++ points)



shh...shh...Answer is something I may be able to figure out after studying C :-D



14. Write a DTrace script to print the absolute path of ALL files being opened by processes running in a system.

Note : Your script should not throw any kind of errors when being executed.



dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }'



15. Draw an approximate diagram that explains the code-flow when a system callis called from a kernel thread

Syscall ==> Kernel panic :-)

16. Give three expansions for the acronym BFU which would be valid in the OpenSolaris World.



a) Blinding Fast Upgrade

b) Big Fucking Make

c) Bonwick-Faulkner Upgrade



17. Assuming you could completely populate a zpool to its theoretical limit

Qa) Find the approximate energy required to do the same (in Electron Volt)



7.488 x 1046



Qb) Find the Mass Equivalnet for the energy



136 billion kg



18. You want to find out how many minor faults occur from the time your kernel loads till the time you get your login prompt during boot. How would you go about doing it ?



dnk again. It must be a dtrace script. I will try to write one once I install OpenSolaris.



19. SMF automatically manages dependencies betweekn services.

a. What is the ideal data structure to store the dependency information ?



Sorted Tree / Graph



b. What is the best algorithm to find the order in which the serices should be started ?



Topology Sort



20. Using any or all of the new and old technologies in OpenSolaris, write about a really wild or cool project that you would like to implement.



I hardly know of any technology of OpenSolaris in detail :-(. I am looking for clustering and building a NAS using ZFS. I got some idea, but will try to do a bit after going home only.

I missed FOSS.in for the last years

Yes that's true. Being a delegate there for last 2 days of the India's premium FOSS event conducted from Nov 24 - Nov 26th, I am now realizing what I missed in the last few years. Chances of meeting FOSS's who's who. Oh wait!!..for anyone who dont know what is FOSS it stands for "Free and Open Source Software". We had talks by Rasmus Lerdorf, founder of php (till v2), Tim Pritlove (all in all of Chaos Computer Club), Luke Kanies (man behind the PuPPeT) and Harald Welte (gpl-violations.org and the diamond sponsor of FOSS.in). On the eve of 25th, there was also a interesting panel discussion by the India's who's who of FOSS where, Karunakar (The linux localite), Kishore Bhargava (The Technological Evangelist), Atul Chitnis (one and only toolz), Frederick Noronha (Foss media guy), Arun Sharma (The FreeBSD India) , Sirtaj Singh Kang (wah taj!!, the KDE India Inc.) and K.Dakshinamurthy. And the great job of moderation was done by Sudhakar Thaths Chandra (Thaths). Talk on past and future 10 years of linux in India was so interesting and in the question session, (which was stopped by IISc guys) the pioneers of FOSS movement in India gave tit for tat replies for all of us, who speaks a lot, and works a bit in spreading FOSS.

The most lively hall was of a MNC who are pioneers in technology, but lacks the marketing. The enthusiasm I could see in each and every person in SUN's hall in spreading the knowledge and the work. The real FOSS activists :-). I missed their BoF's :-(. I believe the entire BOSUG team was there. Most of them know what they are doing. Sanjeeva, Shivakumar, Sheshadri all were of great help in explaining the concepts. I was more interested in hearing about details on ZFS. And yeah I registered to BOSUG list. Anil Gulecha, who is a third year computer science student at JSS Academy, Bangalore and the person who put a live bootable Belenix into a thumbdrive, delivered excellent talk on booting Belenix (Ingeniously Indian!!) from USB and explaining the bootup calls like livecd, liveUSB and why they prefer USB to CD and so on. Unfortunately, I couldn't bootup the liveCD they provided. Need to spend some time on it, or have to get the laptop to next BOSUG Meet. BTW AFAIK Belenix is the first OpenSolaris distribution to be able to run from a USB drive and it supports profiling too :-) They say it gets you the desktop in 60-90 sec which is awesome.

I am a guy who have been using Linux for some time, about 6 years, and if I remember right, I first installed linux RH 7.1 after 6 months when I first touched a computer. Believe it didnt fascinate me much. I installed Linux on a 2Gb partition of 20Gb disk and the 64MB RAM, and linux GUI (XF86 ??) was damn slow and I couldn't do anything there which resulted in formatting the disk space. It was only after a few months later, I learned about multiple virtual terminals at Ctrl + Alt + F1 --> F7 and from that point, I switched to Linux. I loved the b/w screen and could get HCF modem work there in linux, fixed a few issues of other's HD's where HD was not at all detected in windows and at one point or the other, I completely switched to linux unknowingly.

I believe it's the time I play with OpenSolaris and explore the latest technologies. I will surely attend the next BOSUG. I will go back to Cochin and talk with Bejoy Sir more on that. He and Prajeev Sir are my inspirations. They are the known best in Solaris, both SCNA :-) I need to read a lot about dtrace and ZFS. I will explain about them, as I understand in this blog at a latter point. Long live FOSS.IN.

NB: Next blog is the questions of OpenSolaris Quiz conducted at FOSS.in 2006 and my answers to it, as I think.

Meanwhile visit foss.in and view the snaps at,

http://www.flickr.com/photos/tags/foss.in
http://www.flickr.com/photos/tags/fossin
http://www.flickr.com/photos/tags/fossin2006

Friday, November 24, 2006

Symlinks in Unix OSes

Around a year back, this was actually tip or quite a new information to me. I was really shocked to know that I didn’t know the fact. :-( Well the information is the symbolic link (the soft links) have the same size as the number of characters in it’s target

Creating the soft link,

ln -s /hdb9/SuSe9.3/SUSE-9.3-Prof-i386-CD1.iso Suse1.iso

ls -l Suse1.iso
lrwxrwxrwx 1 guest guest 40 May 1 01:21 Suse1.iso -> /hdb9/SuSe9.3/SUSE-9.3-Prof-i386-CD1.iso

wc -
(to take input from stdin)
/hdb9/SuSe9.3/SUSE-9.3-Prof-i386-CD1.iso (Ctrl + D here to get the results)
0 1 40 -

Signing off..

Upgrading kernel in CentOS

A piece of cake for sysadmins. A old tutorial, I wrote, but pasting now, in case it helps someone in any way.

Make sure that you have enough space for your kernel in /boot before you install the new kernel. At least 15 MB!!

Why should I update the kernel ?
================================

Three reasons mainly,

1) For better driver support
2) For better performance, stability and reliability.
3) For security reasons.

The default kernels coming with OS version less than CentOS 4.3 and CentOS 3.7 are exploitable. For the details of CentOS 4 bugs, refer (CentOS = Free RHEL)

https://rhn.redhat.com/errata/RHSA-2006-0617.html
http://www.securityfocus.com/bid/18992/info
http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2006-3745

How do I know my current kernel version installed
=================================================

1) Login to server as user root and issue the command `uname -r`

OR

2) `rpm -q kernel` command issued as root, will return all the types and versions of kernels installed on your server.

Which is my boot loader - LILO or GRUB ?
=======================================

How do I know whether my bootloader is grub or lilo ?

Unless you specifically changed the bootloader, it is GRUB by default on most of newer RPM based OS, like CentOS 3.x , CentOS 4.x, RHEL. Though this question was unnecessary since then, for those who have played with installing both lilo and grub on your servers may have some doubt on the present bootloader.

Assuming you installed the bootloader to the MBR and your harddisk is /dev/hda,
you could run (as root):

dd if=/dev/hda bs=512 count=1 2>&1 | grep GRUB
if = Input File (Everything in a UNIX system is a file!!! even harddisk :-), ofcourse with a few exceptions)
bs = bytes
count = 1st block

If this returns:

Binary file (standard input) matches

then you're using GRUB. (Basically you're just searching the MBR to see if the word 'GRUB' shows up inside.) If it doesn't say anything, then you're not using GRUB and instead LILO. You may need to confirm that by 'grep LILO' instead.

You can identify the device name from fdisk -l command. Device name will be /dev/hd[a-h] for IDE disks, /dev/sd[a-p] for SCSI disks. My desktop, shows the fdisk -l output as below, which means my device name would be /dev/hdc and for knowing the bootloader, I would actually issue the command listed below.

Disk /dev/hdc: 60.0 GB, 60011642880 bytes
255 heads, 63 sectors/track, 7296 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
...o/p continued

dd if=/dev/hdc bs=512 count=1 2>&1 | grep GRUB

Where do I get the latest RedHat kernels ?
===========================================

There are a few places where you can get the later versions.

http://people.redhat.com/~jbaron/rhel4/RPMS.kernel/

The above URL is updated even before, the repositories of CentOS are updated.

http://mirror.centos.org/
http://www.centos.org/modules/tinycontent/index.php?id=13

Before proceeding further, you have to detemine the OS version by login to server via ssh and issuing the command

cat /etc/redhat-release

If the output of command is CentOS release 3.x (Final) , then you have to descend to the directory centos-3 and then the correct 3.x directory. (x can be 5, 6 or 7 depending on the version available at the time of your order)

Again, if the output is CentOS release 4.x (Final), then you have to click the centos-4 directory and once inside, proceed to appropriate versions, depending on value of x in 4.x.

Entering the updates/i386/RPMS/ directory , you will see lots of rpms alphabetically ordered. Look for kernel RPMs and you will see lots of RPMs, as below

for CentOS release 4.x (Final) OSes, inside the http://mirror.centos.org/centos-4/4.3/updates/i386/RPMS/

kernel-2.6.9-34.0.1.EL.i586.rpm 24-May-2006 17:29 10M
kernel-2.6.9-34.0.1.EL.i686.rpm 24-May-2006 17:15 10M
kernel-2.6.9-34.0.2.EL.i586.rpm 07-Jul-2006 22:26 10M
kernel-2.6.9-34.0.2.EL.i686.rpm 07-Jul-2006 22:27 10M
kernel-2.6.9-42.0.2.EL.i586.rpm 23-Aug-2006 05:01 11M
kernel-2.6.9-42.0.2.EL.i686.rpm 23-Aug-2006 05:03 11M
kernel-2.6.9-42.EL.i586.rpm 12-Aug-2006 13:05 11M
kernel-2.6.9-42.EL.i686.rpm 12-Aug-2006 13:06 11M


and it repeats for kernel-devel, kernel-hugemem, kernel-smp and kernel-doc directories.

It is the highest numbered kernel which is of interest to us. You may also look
at the timestamps, the latest kernel in the list is that uploaded on 23-Aug-2006, kernel-2.6.9-42.0.2.EL.i686.rpm.

Now the question which is the architecture, you should choose. 99.99%, it is the i686 rpm. A output of the command 'uname -a' will show the architectures supported like,


Linux cookie.cuckoos.com 2.6.9-34.0.1.EL #1 Wed May 24 07:40:56 CDT 2006 i686 i686 i386 GNU/Linux



Time to Upgrade
==============

I now know which is the RPM to be downloaded (here, kernel-2.6.9-42.0.2.EL.i686.rpm). I have downloaded it using wget using the command

cd /usr/src/redhat
wget -c http://mirror.centos.org/centos-4/4.3/updates/i386/RPMS/kernel-2.6.9-42.0.2.EL.i686.rpm
rpm -ivh kernel-2.6.9-42.0.2.EL.i686.rpm

Use the -i argument with the rpm command to keep the old kernel. If the -U option is used to upgrade the kernel package, it will overwrite the currently installed kernel.

If the system is a multi-processor system, install the kernel-smp packages as well



cd /usr/src/redhat/
wget -c http://mirror.centos.org/centos-4/4.3/updates/i386/RPMS/kernel-smp-2.6.9-42.0.2.EL.i686.rpm
rpm -ivh kernel-smp-2.6.9-42.0.2.EL.i686.rpm



Configuring the BootLoader (GRUB)
================================

On installing the kernel rpm, (using -ivh option), it configures the GRUB boot loader to boot the newly installed kernel. However, it does not configure the boot loader to boot the new kernel by default. If you have used rpm -Uvh, option instead of ivh, it will configure the bootloader config, at /boot/grub/grub.conf (a symlink/shortcut from /etc/grub.conf also exists) to boot the new kernel as well.

Now open up the file /boot/grub/grub.conf in your favorite editor and confirm that it contains a title section with the same version as the kernel package just
installed, like that of below



# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/sda3
# initrd /initrd-version.img
#boot=/dev/sda
default=1
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.9-42.0.2.EL)
root (hd0,0)
kernel /vmlinuz-2.6.9-42.0.2.EL ro root=LABEL=/
initrd /initrd-2.6.9-42.0.2.EL.img
title CentOS (2.6.9-34.0.2.EL)
root (hd0,0)
kernel /vmlinuz-2.6.9-34.0.2.EL ro root=LABEL=/
initrd /initrd-2.6.9-34.0.2.EL.img



If you don't have a separate /boot partition, the paths to the kernel and initrd image are relative to the / partition, like kernel /boot/vmlinuz-2.6.9-42.0.2.EL ro root=LABEL=/

Notice that the default is not set to the new kernel. To configure GRUB to boot
the new kernel by default, change the value of the default variable to the title section number for the title section that contains the new kernel. The count starts with 0. For example, if the new kernel is the second title section, set default to 1. However in our case, it is the first title and the default must be changed to 0.

Cross your fingers and reboot. If it's all good, it will come back.

FailSafe mode
=============

Since the kernel upgradation is via RPM, you dont have to worry about the boot failure. Still mentioning the grub's failsafe feature. Though Grub don't have a perfect failsafe boot mechanism as lilo, you can use it's 'failsafe' feature for
booting the newly installed kernel for once and editing grub.conf after successful reboot. And in case the newly installed kernel doesn't come back, we can simply remote reboot the server and it will boot back to the default kernel as in grub.conf


[root@server ~]# grub

Screen will get refreshed and you will get grub shell as below



GNU GRUB version 0.95 (640K lower / 3072K upper memory)

[ Minimal BASH-like line editing is supported. For the first word, TAB lists possible command completions. Anywhere else TAB lists the possible completions of a device/filename.]

grub>

In the grub shell, enter these
grub> savedefault --default=0 --once
grub> quit


--default=0 is the first title, default=1 is the second title and so on as explained earlier.

Once the server is successfully rebooted, login and then do a 'uname -a' to make sure that the new kernel is loaded and if it did, edit the grub.conf and make the newly installed kernel as default.

Thursday, November 23, 2006

Tables in blogger and some good VI (ViM) Commands

Someone someday said something about VI - If I remember right, it was : Either learn Vi or learn unix. You can't do both in a life time. VI itself is a unix , millions of command or shorcuts hidden inside vi. When I say VI, I meant VIM. I joined orkut community of VI and is copying many of the useful (to me) commands discussed there to my blog. Copyleft wrongs belong Orkut VI community.

BTW I must say it was VI which helped me to join all the pretty tables to a single line table, to avoid the huge space just above the table in a matter of secs. The command I used was :1,$join!

Normal/Command Mode
-------------------

cw Delete a word and put in insert mode
dw Delete a word
~ Change the case letters
guu lowercase line
gUU uppercase line
ga display hex,ascii value of character under cursor
u Undo
Ctrl + R Redo
Alt + W Move word by word
G or ]] Go to last line
gg or [[ Go to first line
$ Go to the end of line
0 Go to the beginning of line
ggdG or [[dG or [[d]] Delete the entire lines of a file
^g Display current line number on the bottom (every detail:-))
cc Cut the line, leaving a blank line there
dd Delete the line (similar to cc, but wont leave a blank line)
yy Yank/Copy a line
p Paste the line just cc'ed or dd'ed or yy'ed
. Yes, that's a dot. (Period). Repeats the last command executed in normal mode
/word Search for the word, 'word' in the document
% To match appropriate brace close for an open brace
J Join 2 lines. To join more than one line (say 10) press : 10 shift+j
D Delete from cursor to end of line
X backspace
x Delete character under cursor
ma Mark the current line as "line a"
mb Mark the current line as "line b"
'a Return to the line marked "a"
d'a or y'a delete or copy the line marked a
/\<\d\{4}\> Search for exactly 4 digit numbers
/\<\a\{4}\> Search for 4 letter words
/first\_s*second/i Search for first followed second on a new line
/bugs\(\_.\)*bunny bugs followed by bunny anywhere in file
/^\n\{3} Find 3 empty lines

Ex Mode

-----------

By Ex mode I mean the executable mode, where the command starts with a colon [ : ]

:help Vi's inbuilt help
:%s/OlD/new/g Replace 'OlD' with 'new' word all over the document / file
:%s/OlD/new/gi Same results as above, but case insensitive replacements
:s/old/new/g Replace 'old' word with 'new' word in the line at which cursor is currently pointed at
:2,5s/old/new/g Replace old with new from line number two to five
:%s/\r/\r/g Turn DOS returns ^M into real returns. I usally do a search and replace of Ctrl+v Ctrl+m.
:%s/^\(.*\)\n\1$/\1/ Delete duplicate lines, which are together
:shell Escape to shell to do anything then exit returning to vi
:q Quit :-D
:x, :wq Save and Quit
ZZ Quit equivalent to :wq!
:ma a ma is the command and a is the argument. Marks the point where cursor is present as a. (But it explicitly wont show it). Now if you want to copy somthin from some other point to the point 'a' , keep the cursor at the other point and type y'a (in escape mode) (y can be replaced by d and so on).
:r!cmd r followed by shell command, reads the command o/p and paste in a line just below cursor.
:rew! rew! is for rewind. Clear all the buffers and the files to initial state of editing.
:set nu Display line numbers
:%! nl -ba Enough display, really number the lines
:sp it splits your screen and ^W - to move between windows
:X prompts for an encryption key. After writing your key, if you save your document it will be encrypted and no one else (but you and vim) can read your documents. If you reopen the file, VIM will ask for the key. If you want to disable encryption, just type :set key=
:%!xxd view in Hex format
%!xxd -r Revert to Normal format from Hex format
:g/^\s*$/d Delete all blank lines
:v/./.,/./-1join compress empty lines (two or three lines to 1)
:s/\(.*\)\ \(.*\)/\2 \1/g Shift the last word of the line to the first, first to second and so on. %s - will do it for entire file


That's all for now. I will add more as I learn more. I need to learn the basics of advanced VI, playing with the registers.

Wednesday, November 22, 2006

Dictionary attack spamming on cpanel servers

Never again..It was bad and tough to fight against the Dictionary attack spamming. Either the server load will rise or

Dictionary attack
-----------------

From wikipedia :

Spammers may also use a form of dictionary attack in order to harvest e-mail addresses. For example, a spammer may send messages to adam@example.com, betty@example.com, carl@example.com, etc. Any addresses to which messages are delivered, as opposed to being bounced back, the spammer can then add to his or her sending list.

My issue was that, the attacker was spamming the domain in such a way that in matter of seconds, the cPanel server, which had the setting of smtp_accept_max = 150 will show

Connection refused from [xx.xx.xx.xx] : too many connections


not allowing other legit customers to send and receive emails. The spammers were targetting only one domain and I didnt have to go for some automate application.

RBLs were not working as expected and had to end up using acl_smtp_connect (Exim 4.53), It was dropping after a connection is made.

acl_smtp_connect = check_host

Touched two files, /etc/eximwhites and /etc/eximblacks and just after begin acl, added this

check_host:
accept
hosts = /etc/eximwhites

deny
log_message = match eximblacks
hosts = /etc/eximblacks
accept


I could configure ACL to use the RBL in the connect, but 50% of many spammer IPs being used was not on any RBL lists. So executed two piped commands as below to start with,

For adding server Ips to white list

ifconfig | grep 'inet addr' | cut -d ":" -f 2 | cut -d " " -f 1 >> /etc/eximwhites

To add spammer's IP to black list. Replace example.com with the domain under attack.

grep example.com /var/log/exim_mainlog | grep "rejected RCPT" | cut -d "[" -f 2 | cut -d "]" -f 1 | sort | uniq >> /etc/eximblacks

Do appropriate greps. This one worked for me.

wohoo. eximblacks file had 800+ IPs all on a sudden. And those IPs were rejected at the time of connection itself, before the SMTP banner or greeting sending the "550 administrative prohibition".

Cleared the log and finally restarted exim and made sure that there are no errors reported in the /var/log/exim_mainlog. All was fun after that, problem solved and no more connection refused errors. This is not a perfect solution, but worked for me.

Update : This solution again worked for me today, but this time, I had to execute the script every 15s to get things under control. There were 20000 IPs and it took me an hour. I need a better solution. Any suggestions ?

Sunday, November 19, 2006

ImageMagick and errors

ImageMagick always causes trouble with versions. Either we have to upgrade, downgrade or install some other packages like PerlMagick to get it work. This time it was the usage of Image::Magick perl module and was getting the errors.

Can't load '/usr/lib/perl5/site_perl/5.8.7/i686-linux/auto/Image/Magick/Magick.so' for module Image::Magick: libMagick.so.10: cannot open shared object file: No such file or directory at /usr/lib/perl5/5.8.7/i686-linux/DynaLoader.pm line 230.


I dont know of a proper solution to this. But the steps I did, solve the issue. ldconfig (ldconfig scans a running system and sets up the symbolic links that are used to load shared libraries properly. It also creates a cache (/etc/ld.so.cache) which speeds the loading of programs which use shared libraries.) was not considering /usr/local/lib where the ImageMagick was installed and once it was fixed everything was working.

edited /etc/ld.so.conf
Added /usr/local/lib as a new line and saved the file.
Ran ldconfig :-)

That did the trick.