Corrupt Snapshot Causing View Composer Service Crash

Error during provisioning:Unexpected VC fault from View Composer

The above error might show up in Horizon View, if you try to use a corrupt snapshot for the deployment of new VMs.

How To Identify If  A Corrupt Snapshot Is The Cause?

  • Look at the Microsoft System Event Log on your View Composer
    • You should see something like this:

      The VMware View Composer service terminated unexpectedly. M It has done this 2 time(s).  The following corrective action will be taken in 60000 milliseconds: Restart the service.

       

  • Search for FATAL messages in the vmware-viewcomposer.log
    • The following message will indicate a corrupt snapshot

      2015-10-06 17:52:53,956 | WFE thread 9 | FATAL | ServiceCore.WorkflowEngine.WorkflowEngine – Unexpected exception occurred.Error reading single property ‘config.hardware.device’ from managed object: snapshot-15587

       

Read More

Remove VIB – Device or resource busy

Today, I was playing around with some vSphere Installation Bundles (VIB) and ran into an issue when I tried to remove vib:

Even adding the –force attribute did not help in this situation.

The following workaround seemed to be working for me:

  1. Stophostd on theESXi host – this will be non-disruptive to yourVMs
  2. Runlocalcli to uninstallVIB

    Note: We need to run localcli, since esxcli is not available if hostd is stopped
  3. Starthostd on theESXi host
  4. Verify thatvmware-esx-KoraPlugin no longer shows up

You should no longer see the VIB installed on your ESXi host.
Localcli is not widely spread within the community and mainly used by VMware’s Technical Support. It provides more troubleshooting capabilities, even if hostd is not running.

 

 

Silicon Valley OpenStack Ops Meetup

Yesterday, I attended the Silicon Valley OpenStack Ops Meetup and held a troubleshooting session focusing on Cinder, Keystone and Nova. The event was hosted by Nimble Storage, ElasticBox and SwiftStack. The focus of this Meetup was to share tips and tricks.

The event was hosted at Nimble Storage’s campus in San Jose, CA.

Nimble_Storage_HQ211-281 River Oaks Pwky

Even though the San Francisco Giants were playing their first World Series game, roughly 100 people attended the event. I think this is a pretty good turnout for the first Silicon Valley OpenStack Ops Meetup.

20141021_182130  20141021_182138

I was lucky and gathered a slot together with Wen Yu to cover OpenStack Shared Storage and Troubleshooting Tips and Tricks. To be honest, I have never been this nervous before. This was the first time speaking in front of more than 20 people.

 

Agenda:

5:45 PM – Doors Open, Food Served, Meet and Greet
6:20 PM – Bill Borsari & Pat Darisme ( Meetup Organizers ),  Nimble Storage – Meet Up kickoff
6:30 PM – Ravi Srivatsav ( CEO ), ElasticBox – Avoiding cloud lock-in to give you total freedom to build, manage, and deploy applications faster than ever before.
6:50 PM – John Dickinson ( SwiftStack technical lead & OpenStack Swift PLM ), SwiftStack – Swift Product Line Manager talks about Object Storage and Swift in the Enterprise
7:10 PMWen Yu ( Nimble Product Manager ) & Jan Schwoebel ( Nimble Virtualization Support Lead ), Nimble Storage – OpenStack Shared Storage and TroubleShooting Tips and Tricks
7:30 PM – 9 PM – Meet the Presenters

  • Bill Bosari and Pat Darisme kicked off the event and welcomed all participants, who made it to the event even though the SF Giants had the first World Series Game.
  • Robin
    • OverviewofElasticBox
      • Mission: ElasticBox empowers business to innovate faster by making it insanely easy for IT, ops and developers to build, manage and deploy applications in the cloud
      • Architecture:
        • Build any application and host it within any, supported, cloud (Amazon, Google, VMware, OpenStack,…)
        • Seamlessly migrate applications from cloud to cloud, don’t be locked down to one cloud solution
        • Share applications and “boxes” with people
          • Boxes are a bundle of packages
  • John Dickinson – Slides can be found here 
    • What is Swift?
      • Swift is an Object Store
      • Great for unstructured data which grows and grows (Images, Videos, Documents,…)
    • What problem does Swift solve?
      • It is build for availability and durability
      • Users do no longer have to worry about where the data is located
      • Great manageability
      • Migrate data without any downtime for your users
    • HowdoesSwiftStack fit in?
      • Provides a manage and control center for Swift
      • Add two additional components, controller & gateway
      • Gateway is a SMB/CIFS and NFS server
      • SwiftStack will provide an all-day workshop in SF on October 28th. Details can be found here
  • Wen Yu
    • Value of Shared Storage
    • Nimble Cinder Features
    • ITO – Image Transfer Optimization.

Screen Shot 2014-10-21 at 3.02.37 PMScreen Shot 2014-10-21 at 3.06.26 PM Screen Shot 2014-10-21 at 3.02.54 PM Screen Shot 2014-10-21 at 3.03.01 PM Screen Shot 2014-10-21 at 3.03.13 PM Screen Shot 2014-10-21 at 3.03.24 PM Screen Shot 2014-10-21 at 3.02.44 PM

  • Jan Schwoebel
    • OpenStack Troubleshooting and Tips
    • About me
    • Troubleshooting Keystone
    • Troubleshooting Cinder
    • Troubleshooting Nova

Screen Shot 2014-10-21 at 3.10.45 PM Screen Shot 2014-10-21 at 3.10.52 PM Screen Shot 2014-10-21 at 3.11.00 PMScreen Shot 2014-10-21 at 3.11.23 PM Screen Shot 2014-10-21 at 3.11.32 PM Screen Shot 2014-10-21 at 3.11.41 PM Screen Shot 2014-10-21 at 3.11.48 PM  Screen Shot 2014-10-21 at 3.11.55 PM  Screen Shot 2014-10-21 at 3.12.03 PM Screen Shot 2014-10-21 at 3.12.10 PM

Unfortunately, I haven’t received the slides from Robin and John yet. However, as soon as I receive them, I’ll add them to this post.

Crucial Data In Your VMware ESXi 5 Log Files

As an Escalation Engineer, part of my daily work is reviewing log files of various systems and vendors. In my first blog post, I would like to show which VMware ESXi 5 log files are most relevant for troubleshooting storage and networking related problems.

All current ESXi 5 logs are located under /var/log and as they rotate, they’ll be available under /scratch/logs

 

vmware_esxi_logs

/var/log/vmkernel.log:

  • VMkernel related activities, such as:
    • Rescan and unmount of storage devices and datastores
    • Discovery of new storage like iSCSI and FCP LUNs
    • Networking (vmnic and vmks connectivity)

/var/log/vmkwarning.log:

  • Extracted warning and alert messages from the vmkernel.log

/var/log/hostd.log::

  • Logs related to the host management service
  • SDK connections
  • vCenter tasks and events
  • Connectivity to vpxa service, which is the vCenter agent on the ESXi server

/var/log/vobd.log:

  • VMkernel observations
  • Useful for network and performance issues

Also, if you have a VM which is affected in particular, it might be worth looking into the vmware.log which is stored with the Virtual Machine. You can find the log under /vmfs/volumes/datastore_name/VM_name/vmware.log.

For the location of ESXi 3.5 and 4.x log files, can be found here.