Long time, since I updated the blog eh ? A good..a bit challenging one, till I remembered the command, debugfs.
Issue
Location : One of my clients' VPS.
Concern : One of his vps clients is not restarting properly. Stucks at initializing the vzquota. Error message when doing 'vzctl start 738' is below
Starting VE ...
Initializing quota ...
vzquota : (error) quota check : lstat `photos.friendster.comphotos6968532886961_494183449s.jpg': No such file or directory
vzquota init failed [1]
Diagnosis
I couldn't find any files with the name photos.friendster.comphotos6968532886961_494183449s.jpg inside the vps, or when looking at /vz/private/738. Then did a
find . -name "*494183449s.jpg" inside the /vz/private/738. It returned me location as ./.trash/lsm/photos.friendster.comphotos6968532886961_494183449s.jpg
/me so happy. Ran
rm -f ./.trash/lsm/photos.friendster.comphotos6968532886961_494183449s.jpgworked fine. But the problem persisted . Reason was though it didnt give any errors, it actually didnt delete the file. Ran a ls -l inside the directory lsm which showed output where it was question marks everywhere except for the name. AFAIK, name of a file is also stored in the directory info in the FS.
[root@vpsit lsm]# ls -l
total 0
?--------- ? ? ? ? ? photos.friendster.comphotos6968532886961_494183449s.jpg
So how can I fix this and make the vps start ? "rm -f filename" was not working.. neither a bigger command, "find . -exec rm -rf {} \;" nor even rm -rf /.trash or rm -rf /.trash/lsm. The directory delete was not working, since it was not empty. Tried unlink also.
[root@vpsit lsm]# find . -exec rm -f {} \;
rm: cannot remove `.' or `..'
find: ./photos.friendster.comphotos6968532886961_494183449s.jpg: No such file or directory
Then tried to turn off quota for vps ..so executed the vzquota off command and got the o/p as below
vzquota off 738
vzquota : (error) Can't open quota file for id 738, maybe you need to reinitialize quota: No such file or directory
So didnt want to initialize the quota by command, since it wont work for sure, just as in case when it starts the vps.
Temporary fix: turn off disk_quota parameter in /etc/sysconfig/vz and started the vps. Turned on the parameter later.
Permanent fix: you can't use fsck to fix this, on a live server. Not when you have to run the fsck in / itself. So fsck was not an option here. So next choice went to debugfs
here is what I did with debugfs
debugfs -w /dev/hda1
cd /vz/private/738/.trash/
unlink lsm
gone..and the issue is fixed. I dont see a proper solution anywhere in web for this ???? question marked cases of file deletion and hence this post.
I am tired of freelancing. Need to find some salaried job..Till next time..cya..
