LinuxPlanet Blogs

I decided that on my vacation I would do some catch-up work. I have many times mentioned that I am a consummate procrastinator, and if you combine that with me being just generally whooped tired after 12 hours away from home on any average day, you understand why my computers seem to go uncared for. I think it’s the same as the whole “the mechanics car is never fixed” thing.

I mentioned a couple days ago that I installed ESXi on one of my home servers (redundant servers) to fix a strange problem I had been having with VMware Server 2.x. That was the first job I needed to so, or at least the most important, and so far it has been doing beautifully.

Next on the list was Mint 8 on the old laptop. It has been running Mint 7 since the distro was released and it was time for an upgrade. Everything was working just fine on 7, I just wanted to catch up the latest/greatest. As expected, the upgrade was a no-brainer and it’s running gorgeously, as Mint does.

Today, so far, I decided to upgrade my desktop machine to Mint 8. This machine, a P4 3Ghz with 3Gb of ram runs like absolute crap. I don’t exactly know why, but it always has. Now I have replaced the cpu fan a couple times and also the power supply at least twice. The computer is noisy, whiny, but not physically broken that I can tell. It just seems to run slower than hell and always has. The installation of Mint 8 on it did make it prettier, but sure didn’t make it seem to run any faster. I think it just dogs over the dual display and craptasticly old Nvidia card. Perhaps if I bought it a new quiet power supply, a better working and quieter cpu fan, a new better video card and a new dvdrom drive (yeah that’s pretty broken too), I could resuscitate this thing so that I could stand using it again. But then again, I could probably buy a whole new desktop computer for what I would spend on repairs to this one. Dang.

So, what’s next? Well, I should install ESXi on my redundant server now that I am satisfied with how the other one is running. I should also upgrade to Mint 8 on my Acer Aspire All In One netbook (notice a pattern here). Other than that, I am not sure.. Maybe work on some code projects I have been stringing along for months and months.

So what kinds of great computery projects are you all up to? Or what SHOULD you be up to

WHEW!

A couple days ago I relayed the story about how my VMware Server 2 infrastructure was suffering some issues. Basically it would randomly just shut down my VMs. I don’t know why. I absolutely poured over the logs for days on end while simultaneously searching google for *any* inkling or hint of an idea on how to remedy the situation or even why it was happening. Nothing….

Frustrated, I was searching around for a different solution and after passing on Virtualbox, Parallels, KVM and others for various reasons, not the least of which was the learning curve on some, I settled on ESXi. I run a lot of ESX and some ESXi at work, so the familiarity is there and it’s been my experience that it’s a rock solid and stable platform, not to mention that it’s bare metal and wickedly fast.

There were some drawbacks. ESX(i) requires a Windows management interface (or Virtual Infrastructure Controller – VIC) and I wasn’t even sure my hardware would accomidate. You see, ESXi has only a certain set of hardware that it will work with.

Well, after a bit of research, I was mostly convinced that my hardware would work, albeit with a little tweak to get the IDE drive recognised. I registered for, and downloaded the free ESXi 4.x release from http://www.vmware.com/products/esxi/, burned it to a cd and I was off to the races.

The installation was completely a no-brainer. Just put the cd in, boot it up and go. It really is an almost no-touch install. I was also pleasantly surprised that it recognised my IDE drive automatically with no tweaking whatsoever. When the install was done, there were only a couple settings to adjust like configuring the IP address and root password, and they are all accessed and changed in a very plain and simple text interface. All in all, in less than a half an hour and with 1 reboot I had an ESXi server just begging me for some VMs.

Once it was up and running I decided I would try everything possible NOT to have to resort to running Windows at home for a management interface. Luckily, other people have decided the same and there is good information available on the web on using the built in command line tools to do what you need to. And they aren’t difficult at all.

First, I needed to be able to access the command line tools on ESXi, and that required turning on SSH access. I followed the instructions here:
http://www.vm-help.com/esx/esx3i/ESXi_enable_SSH.php

After that, I needed to get my VMware Server 2.x VMs on the ESXi box. I turned to VMware Converter for that. Downloaded it (again free) from VMware and installed it on my VMware Server 2.x host machine so that the converter would have access to the local VM files.

I shut down the VMs and used vmware converter to convert them to the ESXi box. Each conversion of a 12GB VM took approximately 40 minutes (give or take). Since the converter is a GUI app, I did a “ssh -Y vmwareserver2host vmware-converter.pl” to run the converter console on my local machine because my vmwareserver2 machine is a headless server.

When the VMs were converted to the ESXi box, I took a cue from this page:
http://episteme.arstechnica.com/eve/forums/a/tpc/f/833003030931/m/150009304931
to add vncserver to each VM, which allowed me to connect to the VMs and make 1 integral change to each virtual machine when they were running.

To get the machines running I used ESXi’s “vim-cmd vmsvc/getallvms” command on the ESXi box, which listed all the VMs I copied there with their assigned vm number. “Then, I ran vim-cmd vmsvc/power.on #” where number is the vm number listed from the getallvms command.
Once they were started, I used vncviewer to connect to the VMs, log in and fix their networking. You see when you move a vm to a different host machine, the mac address gets reassigned and hoses up your VMs network config. Once that was quickly fixed, I rebooted the VMs and they were good to go!

There are a couple other things that I need to get tweaked, like adding my registration number to ESXi, which I found directions for at http://www.vm-help.com/esx40i/manage_without_VI_client_2.php. I also noticed that vmware adds some filesystem into the VMs /etc/hosts file which errors out on boot. Just comment that out and it’s fine. Lastly, since I migrated the VMs from Server 2.x, they already had the vmware tools from that loaded in and I noticed a little barking about those tools while the VMs were booting, so I disabled them by doing a “service vmware-tools stop ; chkconfig vmware-tools off” on my VMs which are CentOS, so your method of disabling those tools may vary.

My impressions so far: Although this all sounded hard, long and technical, nothing could be farther from the truth. It was extremely easy – much more than I had initially hoped. And, if my VMs *stay running* now, it will be well worth it. I also believe that these VMs ABSOLUTELY SCREAM compared to how they ran before. They are much more responsive now in every way. The change was well worth it!

Some times you just can’t win.

A while back I got some decent server hardware and decided I was going to migrate my main server to a VM, so I could make it more portable and more reliable, etc.. Well that worked out really well, for the most part, until lately.

I set up the 64 bit version of VMware Server 2.X, created a brand new VM server, got all my services running on it and all was well initially. For some reason, though, recently my VMs have just been randomly powered off. I cannot find anything in the logs, and the host machine is still running. This has happened a couple times to some other people as well. It’s very puzzling and quite frustrating to me. The only thing I have noticed about the problem is that, on my server, it seems that the VMs stay powered on longer if the management console is not running. When it *is* running, the VMs get shut down in a few hours (always when I am not around to watch them or restart them it seems) and the management interface gets stopped at the same time.

Searching the internet has been absolutely no help on this one, so if anyone has any hints, tips or information, please shoot me an email before I rip out what is left of my hair.

Background

Recently I was trying to create a Ubuntu 9.04 vmware image using Vmware Server 1.08, but I was doing it remotely from my Fedora 10 laptop which was running Vmware Server 1.06. This can normally be done by using Vmware Server’s ability to remotely connect to other Vmware Servers.

VMware Server Console

Here’s my work flow. NOTE: Ubuntu 9.04 will be running on the vmware server on CentOS 5.4.

VMware Workflow

Problem

While going through the setup I encountered a problem I hadn’t seen before where the arrow keys didn’t appear to be working within GRUB while I was installing Ubuntu 9.04.

broken arrow keys

Solution

Turns out there is an option you can enable (i.e. set to TRUE) in the $HOME/.vmware/preferences file which fixes this.

1

xkeymap.nokeycodeMap = "TRUE"

NOTE: This change was made to the Vmware Server that was running on the Fedora 10 box.

I did not need to restart the vmware service, /etc/init.d/vmware, for this change to get picked up, however I did need to restart the vmware server GUI that was running on Fedora 10.

working arrow keys

This link on the vmware communities forum proved helpful in figuring out this workaround.

One of the parts of my day job is dealing with and managing our HPC cluster. This is an 8 node Rocks cluster that was installed maybe a week after I started. Now I was a bit green still at that point and failed to get a better grasp on some things at the time, like how to maintain and upgrade the thing, and I have recently been paying for that

Apparently, the install we have doesn’t have a clear-cut way to do errata and bug fixes. It was an early version of the cluster software. Well, after some heated discussions with our Dell rep about this, I decided what I really needed to do was a bit of research to see what the deal really was and if I could get us upgraded to something a bit better and more current.

Along came my June 2009 issue of The Linux Journal which just happened to have a GREAT article in it about installing your very own Rocks Cluster (YAY!). Well, I hung on to that issue with the full intention of setting up a development/testing cluster when I had the chance. And that chance came just the other day.

Some of you probably don’t have a copy of the article, and I needed to do some things a bit different anyhow, so I am going to try and summarize here what I did to get my new dev cluster going.

Now what I needed is probably a little different that what most people will, so you will have to adjust things accordingly and I’ll try and mention the differences as I go along where I can. First off, I needed to run the cluster on RedHat proper and not CentOS, which is much easier to get going. I also am running my entire dev cluster virtually on an ESX box and most of you would be doing this with physical hardware.

To start things off I headed over to The Rocks CLuster website where I went to the download section and then to the page for Rocks 5.2 (Chimichanga) for Linux. At this point, those of you who do not need specifically RedHat should pick the appropriate version of the Jumbo DVD (either 32 or 64 bit). What I did was to grab the iso’s for the Kernel and Core Rolls. Those 2 cd images plus my dvd image for RHEL 5.4 are the equivalent to your one Jumbo DVD iso on the website that uses CentOS as the default Linux install.

Now at this point, you can follow the installation docs there (which are maybe *slightly* outdated(?), or just follow here as the install is pretty simple really. You will need a head node and one or more cluster nodes for your cluster. Your head node should have 2 interfaces and each cluster node 1 network interface. The idea here is that your head node will be the only node of your cluster that is directly accessible on your local area network and that head node will communicate on a separate private network with the cluster nodes. With 2 interfaces, plug your eth0 interface on all nodes, head and cluster into a separate switch and plug eth1 of your head node into your LAN. Turn on your head node and boot it up from the Jumbo DVD, or in the case of the RHEL people, from the Kernel cd.

The Rocks installer is really quite simple. Enter “build” at the welcome screen. Soon you will be at the configuration screen. There you will choose the “CD/DVD Based Rolls” selection where you can pick from your rolls and such. I chose everything except the Sun specific stuff (descriptions on which Rolls do what are in the download section). Since I was using RHEL instead of CentOS on the jumbo dvd, I had to push that “CD/DVD” button once per cd/dvd and select what I needed from each one.

Once the selections were made it asks you for information about the cluster. Only the FQDN and Cluster name are really necessary. After that you are given the chance to configure your public (lan) and private network settings, your root password, time zone and disk partitioning. My best advice here would be to go with default where possible although I did change my private network address settings and they worked perfectly. Letting the partitioner handle your disk partitioning is probably best too.

A quick note about disk space: If you are going to have a lot of disk space anywhere, it’s best on the head node as that space will be put in a partition that will be shared between compute nodes. Also, each node should have at least 30gb of hdd space to get the install done correctly. I tried with 16gb on one compute node and the install failed!

After all that (which really is not much at all), you just sit back and wait for your install to complete. After completion the install docs tell you to wait a few minutes for all the post install configs (behind the scenes I guess) to finish up before logging in.

Once you are at that point and logged into your head node, it is absolutely trivial to get a compute node running. First, from the command line on your head node, run “insert-ethers” and select “Compute”. Then, power on your compute node (do one at a time) and make sure it’s set to network boot (PXE). You will see the mac address and compute node name pop up on your insert-ethers screen and shortly thereafter your node will install itself from the head node, reboot and you’ll be rockin’ and rollin’!

Once your nodes are going, you can get to that shared drive space on /state/partition1. You can run commands on the hosts by doing “rocks run host uptime”, which would give you an uptime on all the hosts in the cluster. “rocks help” will help you out with more commands. You can ssh into any one of the nodes by simply doing “ssh compute-0-1″ or whichever node you want.

Now the only problem I have encountered so far is I had an issue with a compute node that didn’t want to install correctly (probably because I was impatient). I tried reinstalling it and it and somehow got a new nodename from insert-ethers. In order to delete my bad info in the node database that insert-ethers maintains I needed to do a “rocks remove host compute-0-1″ and then “rocks sync config” before I was able to make a new compute-0-1 node.

So now you and I have a functional cluster. What do you do with it? Well, you can do anything on there that requires the horsepower of multiple computers. Some things come to mind like graphics rendering and there are programs and instructions on the web on how to do those. I ran folding at home on mine. With a simple shell script I was able to setup and start folding at home on all my nodes. You could probably do most anything the same way. If any of you find something fantastic you like to run on your cluster, be sure to pass it along and let us know!

Last night I finally finished cutting all my server services to their new residence on Pukwudgie (my spectacular CentOS 5.3 based server VM). I turned off my old Thinkpad server, which has been doing the job reliably for over 2 years, rebooted pukwudgie just to make sure everything starts up correctly unattended and that was that. My first impressions are that everything seems to run faster. I really expected that, though, because there are a lot more resources available to Pukwudgie than there were to the old server. I am loving it so far and it sure is nice to have an up-to-date server. The old server was running Ubuntu 6.10, which was so old I couldn’t even get security patches for it anymore and this new CentOS server is completely current.

Hopefully this is a move for the better, and I can probably offer the old lappy/server on FreeLinuxBox.org too!

Late last ‘week I noticed that my new nagios server was not responding anymore. Well, I checked it and it was down. Not only that, it was a vm on my test server and the entire server was down as well. Arrrgh.

Usually I use this as a foray to tell you all to remember to do your backups. Well, in this case I didn’t do them either. Hey, it’s a test vm server right? Yeah, well I am kicking myself about that anyhow. I just got nagios working really well the way I wanted. Oh well, I guess I get to practice some more right

Well, as it turns out, my server had a catastrophic drive failure. I did EVERYTHING to try and resuscitate this thing. To start with, it had no partition table at all. Luckily I bought 2 of these servers and they were identically configured, so I checked out the partition table of the one and used fdisk to apply it to the broken one. After that I was able to fsck one partition, but as it would happen, that partition was only boot. Feh. The other partition had lost all it’s superblock info. I couldn’t even use a backup superblock. Nada. I noticed that mkfs had a command line switch of -S, which writes the superblock info on a artition without formatting or touching the inodes. I tried that and it appeared to be successful. At leat I could run fsck on the partition now and it was fixing the inodes. YAY! except that after a few hours of fixing, I still got nothing but a few system files in a pile under the lost-n-found directory. Shortly thereafter the drive lost it’s partition info again anyway. That’s life I guess.

So, it was off to Microcenter to get a new hdd. I brought that home and did a fresh CentOS 5.3 32 bit install and played with it a bit and thought to myself, hey, maybe I should run some kind of burn-in test on this server before I go investing a lot of time into it again.

That is where Sys_Basher comes in. Sys_Basher is a multithreaded memory and disk exerciser. That’s what the website says. It makes a pretty good burn in program by continually testing your memory and disk (which pushes on your cpu as well) for any length of time you specify. I kinda like it actually, and that is a good thing because there are woefully few burn-in or stress test type programs available to the Linux community. In fact, if you are a programmer and looking for a great project, you could generate a lot of traffic and interest by making one. Not that I don’t like Sys_Basher, mind you, but variety is the spice of life and certainly the way of open source!

Anyway, I ran Sys_Basher overnight on my new machine which passed with flying colors. Then, this morning, I decided that maybe I should run 64bit Linux on this box. Some days I am so fickle, but I decided it would be in my best interest to change up the OS before building a bunch of new test vms on there

Maybe this time I’ll even back the darn thing up too! Wish me luck and, btw, do your backups!

A little while back I picked up a couple Appro servers from geeks.com for an outstanding price and also managed to convince Dann to do the same (after making a public plea on TLLTS to raise the moolah). Shortly thereafter, the deal was a bust and there were no more of those servers available. Well, today, Tyler, a listener of TLLTS, sent me an email letting me know that there were more of them available at geeks.com and they are only $109 a piece! Well, what better blog post to break the no-blogging ice with than a great deal like that! Let me say that these servers, are referbished and work GREAT. I have been using mine as a VM server for months now and it works flawlessly. Thanks again for the heads up Tyler!

Need some great and inexpensive servers? You can pick yours up if you hurry at:
http://www.geeks.com/details.asp?invtid=APPRO-1122HI-2B&cat=SYS

CentOS

First, install CentOS, which is an enterprise class Linux distribution that is basically a rebadged/rebranded RedHat, compiled from sources with a few tweaks. I did the generic server only install. After the install, make sure to do all your updates! (yum update)

Once that is done, head on over to VMware’s site and pick up the VMware Server 2. As far as I am concerned, VMware stil has the corner on the market for machine virtualization. There are plenty of other products out there, but I have done a *lot* of virtualization with VMware and it’s rock solid, and it’s free version has a really great price point for what it provides too! Sign yourself up and pick up the free server rpm and make sure to get your registration code as well.

Once that is done, the rest is easy:
Log in as root.
yum groupinstall “Development Libraries”
yum groupinstall “Development Tools”
yum install kernel-devel xinetd
rpm -Uvh VMware-server-2.xxxxxxxxxxxxxxxxxx

Run the perl config script that the RPM install will tell you about and pick the default settings (worked for me anyway). That’s it!

Point your firefox browser to https://yourCentOSserver:8333 and login using your root account and password.

One thing to note as you start creating VM’s is you may need to have access to iso images to do installs from. Place them into your /var/lib/vmware/Virtual\ Machines/ directory and you’ll be able to find them in your vmware datastore.