Renaming multiple files in Linux or unix boxes

I thought it was simple to do this. But I couldn't get rename command to work. I tried several methods found in google, quoted below, but at last it was the for loop and the regexes which came to help.


To rename all the files in a folder to .bak extension

for i in `ls -l | awk '{print $9}'`; do mv $i $i.bak; done

Simpler,

for i in `ls -1A` do mv $i $i.bak ; done

rename 's/\.bak$//' *.bak

Is said to remove the .bak extensions , though it didnt work for me.

To change all upper case file name to lowercase try this



rename 'y/A-Z/a-z/' *

Below command did help me to remove the .bak extensions on a series of files in a folder.

for f in *.bak ; do mv "$f" "${f%.bak}"; done

Recovering from corrupted RPM database

When accessing the yum command, the error we got was similar to this.

error: rpmdb: damaged header #67 retrieved -- skipping.
Traceback (most recent call last):
File "/usr/bin/yum", line 29, in ?
yummain.main(sys.argv[1:])
File "/usr/share/yum-cli/yummain.py", line 82, in main
base.getOptionsConfig(args)
File "/usr/share/yum-cli/cli.py", line 206, in getOptionsConfig
errorlevel=opts.errorlevel)
File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 132, in doConfigSetup
self.conf = config.readMainConfig(startupconf)
File "/usr/lib/python2.4/site-packages/yum/config.py", line 598, in readMainConfig
yumvars['releasever'] = _getsysver(startupconf.installroot, startupconf.distroverpkg)
File "/usr/lib/python2.4/site-packages/yum/config.py", line 667, in _getsysver
hdr = idx.next()
StopIteration

Solution is straight enough, to repair the RPM database. So how do we repair the RPM database. The straight forward method is to, remove the /var/lib/rpm/__db* and the run

rpm --rebuilddb

I believe I discussed this issue earlier in this blog itself. But that didn't fix in the first go and had to reissue the command again. And that time it did fix it.

Another solution to fix the issue is using the db_dump and db_load from the db4-utils package. This process involve re-building master package metadata file /var/lib/rpm/Packages

mv Packages Packages-BAKUP
db_dump Packages-BAKUP | db_load Packages
rpm -qa
rpm --rebuilddb
/usr/lib/rpm/rpmdb_verify Packages

That should fix it. Check out http://people.redhat.com/berrange/notes/rpmrecovery.html for yet another solution. Complicated and may not need it.

Where all standard deletion fails...

Long time, since I updated the blog eh ? A good..a bit challenging one, till I remembered the command, debugfs.

Issue

Location : One of my clients' VPS.
Concern : One of his vps clients is not restarting properly. Stucks at initializing the vzquota. Error message when doing 'vzctl start 738' is below

Starting VE ...
Initializing quota ...
vzquota : (error) quota check : lstat `photos.friendster.comphotos6968532886961_494183449s.jpg': No such file or directory
vzquota init failed [1]

Diagnosis

I couldn't find any files with the name photos.friendster.comphotos6968532886961_494183449s.jpg inside the vps, or when looking at /vz/private/738. Then did a

find . -name "*494183449s.jpg" inside the /vz/private/738. It returned me location as ./.trash/lsm/photos.friendster.comphotos6968532886961_494183449s.jpg

/me so happy. Ran

rm -f ./.trash/lsm/photos.friendster.comphotos6968532886961_494183449s.jpg

worked fine. But the problem persisted . Reason was though it didnt give any errors, it actually didnt delete the file. Ran a ls -l inside the directory lsm which showed output where it was question marks everywhere except for the name. AFAIK, name of a file is also stored in the directory info in the FS.

[root@vpsit lsm]# ls -l
total 0
?--------- ? ? ? ? ? photos.friendster.comphotos6968532886961_494183449s.jpg

So how can I fix this and make the vps start ? "rm -f filename" was not working.. neither a bigger command, "find . -exec rm -rf {} \;" nor even rm -rf /.trash or rm -rf /.trash/lsm. The directory delete was not working, since it was not empty. Tried unlink also.

[root@vpsit lsm]# find . -exec rm -f {} \;
rm: cannot remove `.' or `..'
find: ./photos.friendster.comphotos6968532886961_494183449s.jpg: No such file or directory

Then tried to turn off quota for vps ..so executed the vzquota off command and got the o/p as below

vzquota off 738
vzquota : (error) Can't open quota file for id 738, maybe you need to reinitialize quota: No such file or directory

So didnt want to initialize the quota by command, since it wont work for sure, just as in case when it starts the vps.

Temporary fix: turn off disk_quota parameter in /etc/sysconfig/vz and started the vps. Turned on the parameter later.

Permanent fix: you can't use fsck to fix this, on a live server. Not when you have to run the fsck in / itself. So fsck was not an option here. So next choice went to debugfs

here is what I did with debugfs

debugfs -w /dev/hda1
cd /vz/private/738/.trash/
unlink lsm

gone..and the issue is fixed. I dont see a proper solution anywhere in web for this ???? question marked cases of file deletion and hence this post.

I am tired of freelancing. Need to find some salaried job..Till next time..cya..

Multiline search and replace using sed

I had to code in to remove a host entry from hosts.cfg file of nagios. Here is a bit of superb code to do so..

sed '/#/{:a;N;/\}/!ba;/212.217.202.112/s/#.*\}//;}' hosts.cfg

That will remove the entire block starting with "# '212.217.202.112' host definition" and ending in "}"

Obfuscated code eh ? Not so..learn about registers in sed!!

I hate MicroSoft

Nothing new..But I hate MicroSoft. Normally I don't like writing flames, but I am not in a good mood now, after the day turned out to be worst after fighting with my ISP. So have to express my anger here to get my head cool. So here it goes,

I knew that RTR Ready to Run Software, withdrew the support for MS FrontPage extensions. I never used FrontPage, but customers does and I must admit that it is hard to solve their issues without access to FP program. A customer was asking to install FP on his bare server. He was specific about the FP5 version. Ended up at RTR site to read the faq,

#Where can I get the latest Unix/Linux FrontPage Server Extensions?
The FPSE kits are no longer available for download.
 
#I really need the Unix/Linux FPSE kit, can you please send me a copy?
No, unfortunately, end-of-life really means end-of-life.

#May I distribute the FPSE kit(s) that I have previously downloaded?
No.

Searched for a few more, but all I could find is rpm based fp pack inside the plesk's rpm packages. They had everyone removed the file from every google cache.

Though I admit that FP extensions was one of the packages with only security holes, I don't understand why Microsoft has to withdraw the support from Unix completely instead of announcing a EOL ? Looks like MicroSoft was frustrated with the popularity of Unix boxes over Windows. Or did they buy RTR ? I know lots of customers uses FP extensions on Linux, for it's ease of use and support for FP components. Plesk already withdraw the FP support in fresh installation of the panel on Linux. When is cPanel going to ?

I always have a feeling that Anti Piracy cell of Microsoft is doing a great job in spreading Linux and other FOSS. Microsoft, please do more piracy raids, especially at small and medium business. Let me get more clients wanting to convert Windows boxes to Linux and convert their MS Office and Excel docs to sxw and sxi formats.

Ask the IT Managers to pay you per computer connected to the Windows server, and they will consult with us and replace windows server to be replaced with Samba, released under GPL (v2 or v3 ?:-P). Praise the Anti Piracy department of M$ for spreading Linux and other FOSS.

Inside the Linux boot process

Wow..Why did I miss this article :-( No one can explain the linux boot process better. Tim Jones did an excellent job with his article published at ibm.com developerworks, in explaining the linux boot processes. If you are a linux learner at a novice level, this is a must to read. Quoting some linux jokes here, before I give you the link to that article,

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Windows: "Where do you want to go today?"
Linux: "Where do you want to go tomorrow?"
FreeBSD: "Are you guys coming, or what?"
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Macs are for those who don't want to know why their computer works.
Linux is for those who want to know why their computer works.
DOS is for those who want to know why their computer doesn't work.
Windows is for those who don't want to know why their computer doesn't work.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


The process of booting a Linux® system consists of a number of stages. But whether you're booting a standard x86 desktop or a deeply embedded PowerPC® target, much of the flow is surprisingly similar. This article explores the Linux boot process from the initial bootstrap to the start of the first user-space application. Along the way, you'll learn about various other boot-related topics such as the boot loaders, kernel decompression, the initial RAM disk, and other elements of Linux boot.

Read more here

Limiting apache connections per IP

There are many cases, where on a shared hosting environment, one of the sites may be getting slashdotted or dugg for various reasons. That is where mod_limitipconn comes to help. There are many such modules available. This situation arised on apache 1.3 server for me, and so the patch is meant for 1.3 server only.

Installation
~~~~~~~~~~~~~~~~~

wget http://dominia.org/djao/limit/mod_limitipconn-0.04.tar.gz
wget ftp://ftp.opennet.ru/pub/web/modules/limits/mod_limitipconn-0.04-vhost.patch
tar xvzf mod_limitipconn-0.04.tar.gz
cd mod_limitipconn-0.04/
patch -p1 < ../mod_limitipconn-0.04-vhost.patch

This was a cPanel server and apache root was at /usr/local/apache. So used the following commands

/usr/local/apache/bin/apxs -c mod_limitipconn.c
/usr/local/apache/bin/apxs -i -a -n limitipconn mod_limitipconn.so

The above two commands, should have made a backup copy of your existing httpd.conf and added two new lines,

LoadModule limitipconn_module libexec/mod_limitipconn.so
AddModule mod_limitipconn.c

The apxs command need not be issued, you can simple edit the Makefile and change the first line which says,

APXS = apxs
to
APXS = /usr/local/apache/bin/apxs

and then run the normal make and make install thing.

Configuration
~~~~~~~~~~~~~

For the Vhost you have to limit, say, you want to limit access to http://www.example.com/gallery/ (which may be having lots of controversial pics) use this inside it's virtual host section, like

<Location /gallery>
        MaxConnPerUid 25
        MaxConnPerIP 2
</Location>

What it essentially means 25 connections to gallery and each can access or view two images at a time. There are more directives for limitconnip module which you can find from the code. After the addition the Vhost section may look like below,

<Virtualhost 192.168.1.102>
ServerAlias example.com
ServerAdmin webmaster@example.com
DocumentRoot /home/example/public_html
BytesLog domlogs/example.com-bytes_log
ServerName www.example.com

<IfModule mod_limitipconn.c>
        <Location /gallery>
        MaxConnPerUid 25
        MaxConnPerIP 2
        </Location>
</IfModule>

User example
Group example
CustomLog /usr/local/apache/domlogs/example.com combined
ScriptAlias /cgi-bin/ /home/example/public_html/cgi-bin/
</VirtualHost>

Other options to check are mod_vhost_limit , mod_throttle , mod_bandwidth, mod_curb and mod_cband. There may be few more such bandwidth throttling modules available. If you find better ones, do suggest. I will be interested in learning them.

Booting from your USB/Pen/Thumb Drive

I just read a post from Joe (I guess he is a manager at SUN - if so, a good one to support the developers :-)) mentioning about the visual media coverage of Anil Gulecha's and Moinak Ghosh's achievement of booting live Solaris from USB. Watch the video below

In this light, I thought of mentioning a few USB bootable live CDs in linux and mention a few links also, mainly for my own future reference. Linux is the only OS, I know atleast a bit of, even I am playing with OpenSolaris, not yet as an admin. Is there a hosting based GUI control panel available for OpenSolaris ? I could find only Zomos (though the site says only v9 is supported, but both arch SPARC and x86). If there is a good and "non-technical user" friendly one, I bet OpenSolaris is gonna compete with Linux in hosting industry in next few years.

PenDriveLinux This is one good discussion site and the Live Linux I would recommend is Slax So here goes a few URLs,

Boot Knoppix from USB
Knoppix Remastering toolkit FAQ can be read here
BootUSB from debian Wiki
Googling for a HOWTO boot from USB can fetch you this link