Virtualizing Server Pitfalls - Part 2

After performing a V2V from a GSX server to an ESX server, the CD-Rom does not appear in ‘my computer’. This makes it difficult to install the VMWare tools required for network and performance tuning. What you have to do is to shutdown the VM, remove the CD from the menu. Power on the VM, log into the server and perform a shutdown. Then re-add the CD from the Edit menu. This way you are guaranteeing that this is a clean install of the CD-Rom drive.

Performance Tuning - Part 1

Virtual Machine Alignment

I have been asked, recently, about “how do I make my VMs run better?” This is a loading question, if I have ever heard one. There are a lot of things that you can do to increase the performance of the Virtual Machines on an ESX host.

In a SAN environment, one things that you can do to increase the performance of your VMs, is to align the partition. 64KB Track Alignment for both the physical host and the VM partitions yield I/O performance improvements, such as, reduced latency and increased throughput.

An unaligned partition results in a track crossing and an additional I/O, incurring a penalty on latency and throughput. The additional I/O (especially if small) can impact system resources significantly on some host types. An aligned partitions ensures that the single I/O is serviced by a single device, eliminating the additional I/O and resulting in overall performance improvement.

You need to align, not only the SAN LUN, but the VMFS Partition and all guests on the ESX server. Here is a link to the VMWare document that explains the “How To”. Link

VMware I/O queues, “micro-bursting”, and multipathing

I colleague of mine came across this article and I had to share. It is a very technical read, but very relevant to all SAN environments. Anyone that is using VMWare should read this and understand the performance impact to your SAN.

“There was an interesting discussion/whiteboard around the topic of storage network design around FC/FCoE (though this applies to iSCSI as well). The Cisco folks made some really interesting analogies with VoIP and TP “micro-bursting” that I thought were awesome and I wanted to share.

It’s also the reason why there is so much discussion around LUN queues and queue management in vSphere. It’s not just EMC with PowerPath/VE, but I’ve heard 3Par start to talk about “adaptive queuing”, and last week Dell/EqualLogic announced their own addition to the Pluggable Storage Architecture. It’s all about the vStorage APIs, baby :-)

It’s apropos based on the recent discussion over queuing and comparisons of NFS and block-based storage options here.

Lastly, it’s also a topic of discussion based a recent post here on the topic of whether MPPs like PowerPath/VE are there to help “legacy” arrays and whether “virtualized” arrays get any benefit from more host-side sophistication. I’ve also heard the implication that NMP + Round Robin + ALUA are good enough in all cases – if you choose the right array :-)

Vaughn and I tend to agree a LOT – and we do, much more than we disagree. We both are super into VMware, which means there’s a lot of things we share. But there are times when we disagree – and this is one of them.

But this isn’t about EMC or Netapp, and isn’t personal, it’s core technical fundamentals – and apply to all storage vendors. If you want to understand and learn more – read on!

I will say this – with all arrays – EMC’s, HP’s, HDS’s IBM’s, and NetApp’s – all of them, the back-end design DOES matter, and you should look at it closely – and they are all wildly different architecturally. Each has their advantages and disadvantages – and usually what makes them an advantage in one circumstance is a disadvantage with another. This is why I’m personally of the opinion that it’s more about knowing how to leverage what you happen to have.”

http://virtualgeek.typepad.com/virtual_geek/2009/06/vmware-io-queues-micro-bursting-and-multipathing.html

Funny.. I have had this conversation

P2V using BartPE

This seems to be very cool... I'll need to play with this and report back on how it works. I have a “couple hundred” physical servers that I need to virtualize.

“Ultimate-P2V (fix-vmscsi) is a free plug-in that allows you to clone a physical machine to virtual machine - and perform the neccessary “system reconfiguration” required to make it bootable. Without this tool or a commerical P2V tool the virtual machine would just give a “blue screen of death”

This document outlines how to configure a BartPE Boot-CD with the appropriate drivers and plug-ins to perform the P2V. We use Symantec Ghost 8 merely as an example of a disk cloning tool - but really any cloning software (with a supported plug-in) would work just as well.

The guide walks you through a completely manual creation of these plug-ins and drivers. The intention is allow you to configure a BartPE with the lastest drivers if you so wish. Alternatively, you can use the pre-configured plug-ins below.”
Article Source

Newest Addition

We have a new addition to our household, a new dog. I would like to say for the record that it was all my doing, but I can’t take full responsibility for the new acquisition. My wife sent me an email with a link and a picture of the dog. I am only Human!

As most of you are aware, I lost my “Freya” dog last year, a 14 year old chow-husky-wolf mix. For Christmas, the girls got me a 2 year old yellow Lab, Thor. He is a great dog, but he is just not my kind of dog. Don’t get me wrong, he is a sweet dog, but not so bright. He is about 100 pounds of love, who always wants to be either on you or next to you. You put your hand down… and… look, there is a dog. He is like a bull in a china shop, knocking over anything in his way, very high energy that loves chewing on things. It really does not matter what it is, shoes, plastic, metal cans, wood…etc..

The newest edition is Skadi, in keeping with the Norse naming, a 2 year old half husky – half Malamute. Because of her bread, she is very much a pack oriented. To this end, I immediately established my dominance as Alpha, Norsa as Beta, the other two can fight out who is next in the chain of command. Even though, she is about 50-60 pounds, smaller than the other two by atleast 20 pounds, Skadi holds her own and is not afraid of taking you down. She is highly intelligent and it shows. She is loves destroying things.. must be the age… Thor does this too. Well, there was a stuffed toy in a plastic box, on top of another box, in the garage. In about an hour, she managed to pull the plastic box down, take it over to the other side of the garage, open the top and take out the toy. All this without making a lot of noise or destroying the plastic lid. If it was Thor, the box and everything in the box would be in pieces, not just the toy. She loves the kids and is getting along with the other two dogs. Unlike Freya, who we had to keep separated from the kids. Freya was very tail protective and went from play to “I am going to eat you NOW” in about.. that long… There were many times where her and I went rounds, not always did I come out unskathed. But, Freya was a great protector dog was missed when she died of cancer.

Skadi will fill that void that was left with the death of Freya. She has the temperament of being that protector dog. (Yes, I have seen her angry, not pretty.)

Penguin USB drive is infinitely cute, hilarious

I saw this.. and nearly busted a gut!

"http://www.engadget.com/2009/06/10/penguin-usb-drive-is-infinitely-cute-hilarious/" It's not like we haven't seen slightly funny / interesting / unorthodox USB flash drives before, but there's just something about a headless penguin hanging from a USB port that gets us chuckling. Available in 2/4/8/16GB capacities, the simply titled Penguin drive even gives a portion of its proceeds to the World Wildlife Fund, providing you with even more reason to cave to the urge and pick one up. This little fellow's up on Amazon right now for $12.95 to $42.95.

Virtualizing Server Pitfalls - Part 1 - Updated 6/23/09

Firstly, I want to begin with a big disclaimer – this my approach to the P2V’d Virtual Machine. I accept ABSOLUTELY no responsibility for any unforeseen and unwanted side-effects.

Most of you know or have read that I have taken a new position, no longer with Comcast. In the new job, I have become “Virtual Boy”, dealing with everything Virtual. So, I thought that I would pass along some of my insight for your P2V / V2V ventures. This will probably be a multiple part message.. So, bare with me.

Windows 2000 servers are probably what most people are trying to get rid of in their environment, but can’t seem too... because of software limitations. These are also the most problematic to virtualize. Here are some items that you need to be aware of during your migration process.

Pre-Migration Process

1. Software – All servers have some kind of raid/teaming/Compaq (HP) crap software for the system. This needs to be removed, to the best you can, before attempting a P2V convert.
2. Services – If you are not able to remove the software, disable the service.
3. Hidden devices – such as network cards. (you know the ones with the IPs that you need) This has bitten me a couple of times. But, there is process to have them show up and uninstall them - KB Article.
4. Multiple Network Interfaces – This comes back to the “teaming” situation. You really should break the team and set them up as a single interface.
5. Use a block copy method rather than the file method – With the block copy method, you are able to bypass the file locks that Window places on a file that is in use.
6. Oh.. That brings up a good point – Make sure that no one is saving data to the server! Simply, it probably will not be there after the convert.
7. HALs are a big deal – but – with most name brands, this is really not an issue. Buy name brand servers, not Bob’s Server.
8. Key FABs and External Cards – It can’t take them with it. There is a workaround for the USB Keys, but not for cards. (Link)
9. Local Administrator Account - Make sure that you have the local administrator account username and password. (or make an account for you to use, just in case)

Post-Migration Process

1. Turn off the physical server before you power the VM. If you don’t… (Well, can’t fix….) Many tools out there will power down the physical server for you. However, I have found that the power down process take longer than my new VM to boot. So, power the box down and verify that it is truly off before powering on the VM.
2. Install the VM tools and reboot.
3. Adjust your RAM requirements – Here is a perfect time to down the amount of RAM that you have allocated. Truly evaluate how much RAM you really need.
4. Check connectivity – Just because it says that you are “connected”… CHECK! Do various Ping sweeps from the server. Also, you may want to check and verify that you can RDP (TS) to the VM from your workstation. (Assuming you could before)
5. Remote issues – I have not found a solution to this issue, as of yet. Various Windows 2000 servers lose the ability to be able to remote into them from your workstation. Everything else works on the server… but, you are not able to RDP to the server.
6. Domain Account - I have had several issues where the new virtual server is unable to log you into the domain. Log in using your admin account, verify your IP settings (including DNS), take it out of the domain (do not put in your credentials and do not reboot), re-add it back to the domain. Now Reboot. This way, you did not remove the computer account in AD. Basically, you changed the SID on the computer object.
7. Application / Services Check – It should go without saying… Verify that your application and services are running.
8. Finally, Check your server! Check the Event Log for errors, log into the applications on the server, check, check, and do it all over again!

I will append to this entry as I remember or find things to post about.