Shared VMFS Volumes on non-clustered hosts


There was an interesting post on the EMC Community Network “Everything VMware” forums yesterday about the nature of VMFS locking and how it is affected by multiple hosts accessing the volumes without those hosts being part of a cluster together (or even parts of different clusters).

The first really important piece to understand about this question is how VMFS implements locking.  Obviously the hardest part of any clustered filesystem is ensuring that disk writes (and to a lesser extent, reads) are done in a sane, coordinated and reliable fashion.  Generally this means that any given file can/should only be accessed by one host at a time (again, except for read requests).  There are a million different ways this can be done, and many filesystems / volume managers rely on network access and configuration files to achieve this.

VMFS is different – VMFS uses exclusively on-disk locking semantics and SCSI protocols to achieve its needs.  I will leave the discussions of how reservations/locking works to others who have done great work describing their mechanisms.  You should read those posts – udnerstanding VMFS at a low level will help you every day in your work.

So, this property of VMFS’ exclusive use of disk semantics for locks means that hosts dont need to know about each other (via network or config files) in order to effectively use a shared cluster VMFS volume.  Everything about the locking is on the disk, and if the ESX host can see it, it knows everything it needs to know.

So, the short answer is that its perfectly functional to have multiple hosts and even multiple clusters accessing the same VMFS volume, even if those hosts are in different clusters, folders, and even datacenters.  Its even OK if none of them are managed by vCenter.  Its even OK if they are all the free ESXi license.  From a technical perspective, as long as you are not exceeding the maximums (32 hosts / volume, etc) you are in a reasonable (and even supported) configuration.

Now, the relevant question is not ‘can you’, but should you? I would argue that you shouldn’t, with 3 notable exceptions.

You should stick with a given set of VMFS volumes masked/accessible only by the cluster on which their VM’s primary run.  The reason for this is really around management.  Do you want to have to keep a spreadsheet about what volumes are primary for what cluster? Do you want to have a new admin accidently put something on a non-preferred volume?  What if you want to isolate performance issues?  Sure, this method can strand a little bit of storage, but honestly modern dedupe and thin provisioning methods make that less of an issue.

What are the exceptions (in my mind)?

  1. A volume containing only templates/ISOs (very little risk here, and you dont want to duplicate all that).  I think that NFS also works very well here.
  2. A “swing” volume used just for moving virtual machines between clusters.
  3. Home / very small business clusters where you dont have any storage to spare and you aren’t using advanced features like vMotion anyways.

So there are my thoughts.  Comments?

Big Changes


Its with somewhat mixed emotions that I announce to the world some changes in my life.

As of this morning, Monday the 18th of 2011, I’m no longer a Sales Engineer / Solutions Architect for 3PAR/HP. Let me be clear – I’m not making this change because of concern over the 3PAR or HP product lines. I think that HP’s aquisition of the 3PAR company and product line was a pretty smart choice. However, I came from a true engineering role. Doing operations is a lot of fun, and I learned a ton while being in that role. I went to a Sales Engineering role hoping to parlay that experience into helping 3PAR’s customers solve huge problems. While I had that oppurtuntity, I missed the technical side of things.

I missed getting my hands dirty.

I missed finding bugs and fighting to fix them. I missed pushing the limits of the newest technology.

So, as of this morning, I start my first day as a Sr. vSpecialist for EMC. I know that most of my past colleagues just spit out their (coffee|soda|water|milk) after reading that. If there was one company they didn’t expect me to head to (besides Microsoft), it would have been EMC. They know that over the years I’ve been, historically, a critic of EMC’s storage and some of the technologies. So why would I go join the dark side, so to speak?

Its because I think things over there (here?) are changing. I think that the technology base are significantly improving. The Clariion line is starting to look significantly better with the advent of VNX (which is not to say there’s no room for improvement – there is). The EMC VSI vCenter plugins are the best integration on the market. EMC Smarts^H^H^H^H^H^H Ionix is starting to get better. But most importantly, I think that EMC, even more so than VMware has the aggressive ‘vision’ needed to make VMware the default standard in the enterprise. Its that change that I want to be a part of. So, I’ve decided to cash in and go join Chad’s Army.

I think EMC knows that I’ve been critical in the past. Even knowing that, they still hired me. I truly hope to be able to use my perspective to improve the EMC product line, help the world figure out this virtualization fad (as my Dad calls it) and get my hands back into the muck.

For a good intro into what a vSpecialist does all day, I’d recommend my new colleague Tommy Trogden’s post, or Nicholas Weaver’s post on the subject.

So here’s to a cool looking mechanic’s shirt and getting my hands dirty once again.

Why visudo for the vMA or Unix systems.


On Twitter recently, Maish Saidel-Keesing posed an interesting question: “Why bother with visudo at all?”.  The implication/suggestion here is that visudo is nothing more than alias to ‘vi /etc/sudoers’. Fortunately, its not.

visudo actually creates a copy of the sudoers file, edits that using vi (or whatever editor you have set for $VISUAL in your shell), checks the syntax and then, on success, copies it into place.

What if you didn’t use visudo and just edited the file directly, but made a mistake?  Well, here’s an example

[vi-admin@vma ~]$ sudo -l
>>> sudoers file: syntax error, line 101 <<<
sudo: parse error in /etc/sudoers near line 101

As you can see, once you have a syntax error, you dont get to use sudo at all.  You;ll have to login as root directly (might be hard if you’ve disabled the root account like in the vMA) or wait for someone with the root password to fix it for you.  However, if I make a mistake  using visudo, its very clear about telling me:

[vi-admin@vma ~]$ sudo visudo
Warning: undeclared User_Alias `CRAP' referenced near line 101
>>> sudoers file: syntax error, line 101 <<<
What now?
Options are:
(e)dit sudoers file again
e(x)it without saving changes to sudoers file
(Q)uit and save changes to sudoers file (DANGER!)

visudo very clearly saves me from a major mistake (well, at least warns me about it).

So what do you do if you want to edit the file for pushing out to a large number of machines and dont want to load it on the current system.  Well, visudo has you covered there too.  Use it in ‘check’ mode:

[vi-admin@vma ~]$ visudo -c -f sudoers.testing
sudoers.testing file parsed OK

Life is good!  Thanks to Maish for the idea.

Using SSH in ESXi (Password-Less)


Looking into this, its a little funny – you can SSH into ESXi, but not out.  Dropbear has the ability to create a valid keypair, but theres no actual ssh binary in ESXi.  However, you can make it happen.

1) “Create” an ssh symlink to dropbear:

ln -s /sbin/dropbearmulti /bin/ssh

This works because dropbearmulti is a multicall binary, which allows it to change behavior depending on how you execute it.

2) Create a keypair:

dropbearkey -t dss -f privatekeyfile -s 1024

Why 1024?  Because thats the only keylength supported by DSS keys!.  You’ll end up with a file in the current directory called ‘privatekeyfile’ and the system will output a public key in SSH format on the screen:

~ # dropbearkey -t dss -f private -s 1024

Will output 1024 bit dss secret key to 'private'

Generating key, this may take a while...

Public key portion is:

ssh-dss AAAAB3NzaC1kc3MAAACBAJbXscSKNxkxs3NYfMgMLs8tsh3iio9vFN3fzq8/5HrsgcGK3gHc+SQlLmhtP...hostname.domain

Copy all the lines of this starting from “ssh-dss” through to the end of “hostname.domain” to your clipboard.

3) Add this copied public key to your Linux host in the right location – usually ~/.ssh/authorized_keys:

linuxhost% cat .ssh/authorized_keys
ssh-dss AAAAB3NzaC1kc3MAAACBANPYWCXvqAVK95Xa0qM1rUPM7h2CWB85d2Qk3paYsRU6x....

4) Now use the private key to make sure that it works from ESXi:

~ # ssh -i privatekeyfile [email protected]

Last login: Tue Apr 12 15:01:15 2011 from domain.lan
[user@host] (Linux 2.6.18-194.26.1.el5)%

Life is good!

VCDX Process / Experience


So many people have posted about their experience with the VCDX project, and many of those posts were a huge help to me, in the words of Jason Boche (a fellow VCDX), here’s my point of view to pay it forward and help those who come after me.

I wont spend a lot time on the process, as its been covered quite clearly.  Some references would be Jason Boche’s blog ( ,Chris Kranz’s blog ( or Jason Nash’s blog (– they are all very accurate.

Instead, I’d just like to give my thoughts on specific parts in bullet format before I forget anything.

  • VCP – This one should be a breeze for you.  If its not, I’d suggest considering putting some more time under your belt before attempting this process.
  • Enterprise & Design Admin Exams.  These probably wont be a breeze, but they aren’t all that difficult if you spend all day doing VMware related stuff.  If you are a Microsoft-center person, get familiar with the command line methods of doing things, as there are plenty of questions around that, both in the Lab and multiple-choice sections.  The lab section does allow for use of the man pages and help, so don’t be afraid to look it up.
  • VCDX Application – This was the hardest part for me.  There’s really 2 parts to this, and they are fundamentally different.  In both, I’d urge you to think like a consultant.  I’m not a consultant and never have been, so this was key to getting my head around things.
  • Assume nothing – not even that your audience knows what ‘LUN’ stands for.
  • Document everything – even the things that are standard in your organization.  In my company, we have well known standards for LUN naming, datastore, naming etc, so my initial application materials simply didn’t cover that.  Same went for things we wouldn’t do internally like ‘Next Steps’ and ‘Timelines’.  What the application says about that (“If they aren’t included it’s a reject”) is totally true – the committee will reject it on that (mine did on my first 2 tries!).  If I could make a specific recommendation, see if your VMware PSE or TAM or SE or Salesrep can get you a blank copy of their PSO engagement template – it’s a very good guide for what you need to include as well as the level of detail expected.
  • Application – don’t overanalyze this.  Take your time and fill it in well.  Look at Duncan Epping’s blog entry about this, as well as John Arrasjid’s
  • Technical+
    • This is not just a technical exam – your communication and design and evaluation skills are being tested.  Most have mentioned to ‘know your design’ and this is definitely true.  Know why you chose 3 vSwitches over 2, and why you didn’t choose 4.  Know why you didn’t do Etherchannel, and why you did choose Dell over HP or Sun.  These are all more than technical decisions, and you need to be able to explain them
  • Defense
    • Presentation
      • This will go approx 2x faster than you think, no matter how fast you think you can talk.  You will be interrupted for questions and clarifications, etc.  You will need all of the time allotted, and when that clock starts blinking red, you will be astonished at where the time went.  The best that I can offer here is to have a strong outline for your presentation that you can follow to avoid going on tangents that eat time.  That, and have lots of slides with diagrams.  I had 3 or 4, and wished I had 6 or 7.
    • Design
      • This was the hardest part for me personally.  You will be given requirements to meet, but just like normal customers and clients, they may be incomplete or wrong ☺.  Be prepared to ask a lot of questions and assume nothing.  As others have said, the point is NOT to complete a design in the 30 minutes allotted (I doubt its even possible).  The point is to show that you are an Expert Designer by the kinds of questions you ask and how you think out  loud.
      • Think Out Loud.  Its my style to think silently for 5 minutes evaluating information before stating an idea, but that wont work in a defense.  Practice speaking your evaluations of information out loud, because the committee will need to hear it.
    • Troubleshooting
      • See above about thinking out loud.  Getting the right answer isn’t the primary concern here (though I’m sure it helps).  Talk out loud about how to solve the problem you are facing.  Even if you haven’t the foggiest clue, talk out loud about what you’d be looking for.  Your thought process is whats important
    • The committee isn’t out to get you.  I got the feeling (as did others) they wanted me to pass – I just needed to provide them the proof they could point to so they could pass me with clear conscience.
  • Don’t waste breaks – Your email isn’t that important.  Your twitter isn’t that important.  Your voicemail isn’t important.  Walk around.  Look at the art.  Meditate.

I might have more to add later, but for now chew on that!  I found out that morning that I passed and was granted VCDX #52, so maybe I’m not nuts with the above.


System.out.println(“Hello VMware World”);


I’ve always wondered if I should start a blog.  On one hand, it has always seemed a little odd to have something online that you expect that people might read.  Maybe a little self-serving, even.

On the other hand, I’ve also got ideas and thoughts that I think are worth sharing and discussing for which I don’t have an alternate forum.  Some of the most important things in my corner of the industry get announced or discussed on a blog somewhere.  I figure it might also improve my writing.

So, I think I’ll give this a shot – what’s one more blog to read if you’ve got a good RSS reader anyways.

What is it that I’d like to talk about?  Well, mostly things that i know about.  I be talking about VMware, virtualization more generally, high end storage engineering, automation/mass deployment – those sorts of things.

If you care, my background is mostly in Unix/Linux deployments, VMware and storage engineering.  I’ve come from startups doing large web farms handling 100M+ hits/day to working for Technical Operations at and most recently at 3PAR (now Hewlett Packard, of course).