Tim Groeneveld

Random musings from the world of an Open Source geek

libguestfs: library for accessing and modifying VM disk images

/ | 1 Comment on libguestfs: library for accessing and modifying VM disk images

Imagine if there was an application that you could feed a disk image to, wether it be Microsoft Windows 2008 or Red Hat EL 6 and see a list of applications that are installed – no step back – imagine if there was an application that you could feed a disk image to and just discover what operating system was installed.

Well fear not! There is such an application. I kid you not, if you are anything like me you are going to look and see this as one of the most amazing discoveries. When I found this, I had a “oh my goodness, why didn’t I know about this sooner” moment.

libguestfs is a set of tools for accessing and modifying virtual machine (VM) disk images. You can use this for viewing and editing files inside guests, scripting changes to VMs, monitoring disk used/free statistics, P2V, V2V, performing partial backups, cloning VMs and much much more.

libguestfs basically allows you to do anything that you have ever dreamed of. Yes. It will even read your Windows Registry “hive” files and spit them out as XML files.

The virt-inspector command examines a virtual machine or disk image and tries to determine the version of the operating system and other information about the virtual machine.

libguestfs is definitely on the TODO list for installing 🙂 I am love. I could just imagine a nice web application thrown on top of this that would allow you to do funky things like that a WIndows machine offline, run a Windows Update *while the virtual machine is not running* and then start the machine up again. Doesn’t sound awesome yet? Think about it this way: a new Windows update is released. You take an LVM snapshot of the currently running box. You apply any Windows Update to the new LVM snapshot, shutdown the current running box while bringing up the new one (or edit the registry and remove any details of the IP and add a script to run on startup…)… so many ideas 🙂

Red Hat Enterprise Linux 5.6 is out!

/ | Leave a Comment

Red Hat is pleased to announce the availability of the latest update to Red Hat Enterprise Linux 5, release 5.6 (with kernel-2.6.18-238.el5)!

In virtualisation land, one of the major changes have been the addition of the virtio balloon driver. This new driver in Red Hat 5.6 allows guests to express to the hypervisor how much memory they require. The balloon driver allows the host to efficiently allocate memory to the guest and allow free memory to be allocated to other guests and processes.

Also, In Red Hat Enterprise Linux 5.6, a global synchronization point is added to pvclock, providing a more stable time source for guests, which is good because this has been the cause of so many time slips in guest clocks.

RHEL 5.6 also pushes the BIND server to the latest release number.

And guess what. Oh my goodness: PHP 5.3! I know, it seems strange, but finally Red Hat ships with PHP 5.3 as default!

There are many other noteworthy changes in Red Hat Enterprise Linux 5.6, and I would highly suggest that you read the release notes for Red Hat 5.6.

Karanbir (the lead developer of CentOS) has asked on Twitter what should be released and how: CentOS 5.6 or CentOS 6? Arrfab has stated the reasons why CentOS 5.6 should be released first:

  • The centos 5.x install base is there while there is (obviously) no centos 6 install base.
  • So those people having machines in production, faced to the net (, etc, etc, …) would prefer having their machines patched and up2date (security first !)
  • People running CentOS 5.x on servers and willing to install php53 packages, now officially included
  • On the build side, the el5 build process is clearly identified and known since 2007 : packages with branding issues are already identified and patches/artwork is already there, meaning that it will be probably (no, surely !) faster to have 5.6 out of the door than 6
  • Same rule for the QA process : people from the QA team can “blindly” focus on their previous tests, and just have a look eventually at some newer packages (a few, like php53 but not that much in comparison with el6)

and although I am excited to see CentOS 6.0 be released, I must say that I agree that CentOS 5.6 should be released first.

Reminder: root needs > 10GB

/ | Leave a Comment

*sigh*

[tim@beast ~]$ df / -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/disk/by-uuid/ebc29b81-cc7e-4fa5-9777-8953f54752ff
9.9G  9.3G   70M 100% /

Why oh why do I do this to myself?

Holding back way to much!

/ | Leave a Comment

A typical day’s worth of updating my machines usually looks a bit like this:

[tim@myhost tmp]$ sudo pacman -Syu
:: Synchronizing package databases...
core is up to date
extra is up to date
community is up to date
multilib is up to date
:: Starting full system upgrade...
warning: gnome-disk-utility: ignoring package upgrade (2.30.1-1 => 2.32.0-2)
warning: gvfs: ignoring package upgrade (1.6.3-1 => 1.6.6-1)
warning: kernel26: ignoring package upgrade (2.6.34.3-1 => 2.6.36.3-1)
warning: kernel26-headers: ignoring package upgrade (2.6.34.3-1 => 2.6.36.3-1)
warning: libbonoboui: ignoring package upgrade (2.24.3-1 => 2.24.4-1)
warning: libgnome: ignoring package upgrade (2.30.0-1 => 2.32.0-3)
warning: nvidia: ignoring package upgrade (256.53-1 => 260.19.29-1)
warning: nvidia-utils: ignoring package upgrade (256.53-1 => 260.19.29-1)
warning: proftpd: local (1.3.3-5) is newer than extra (1.3.3c-1)
warning: rdesktop: local (1.6.0_rc1-1) is newer than extra (1.6.0-5)
warning: udev: ignoring package upgrade (151-3 => 165-1)
warning: udisks: ignoring package upgrade (1.0.0-0.20100223 => 1.0.2-1)
warning: youtube-dl: local (2010.12.09-2) is newer than community (2010.12.09-1)
resolving dependencies...
looking for inter-conflicts...

You know it’s bad when you have so many custom packages, that you may as well have your own distribution :).

Internet Travels

/ | Leave a Comment

*sigh* the Internet. From updated software to the Himalayas’.

I thought that I might just describe today how I got lost in an interesting maze of Handbrakes to Schools in the Indian Himalayas.

So, I was doing my normal morning Internet browse, off discovering what new and interesting tidbits of information I could discover. On Freshmeat, I learnt that there was a new version of Handbrake, a popular (and very, very awesome tool) for converting DVDs into other formats so they can be viewed on iPods and iPhones.

Cool, I thought. Upon reading the changelog, they listed a couple of changes to ‘ghb’. What was ghb, I wondered? Well, a Google found that GHB was infact Gamma-Hydroxybutyric Acid. Indeed, Gamma-Hydroxybutyric Acid was not what I was looking for, but it did start the wild search party.

From learning all about GHB and it’s affects, Wikipedia was only more then happy to teach me more information. Xyrem, manufactured by Jazz Pharmaceuticals is a orphan drug. What’s an orphan drug? Well, it’s a pharmaceutical agent that has been developed specifically to treat a rare medical conditions.

Since the market for any drug with such a limited application scope would, by definition, be small and thus largely unprofitable, government intervention is often required to motivate a manufacturer to address the need for an orphan drug. One of the interventions that can be undertaken by a government is to create a government-run enterprise to engage in research and development, otherwise known as a Crown Corporation.

An example of a Crown Corporation is the Australian based National Broadband Sceme, which is owned and operated by the Department of Communication and the Digital Economy.

On the DBCDE site, there was a media release sent out: “A reminder for Australians affected by floods about the Satellite Phone Subsidy Scheme”. Naturally, it was time to have a look at who was offering and what the price of satellite phone are. $1,000+. One of the coolest phones being the Thuraya XT. (ZOMG! It has a color screen!)

Looking for how much it actually costs to have a satellite phone (ie, monthly charges) I found a page listing where the Thuraya phones are used.

Leading me to an isolated village situated in the Indian Himalayas, Kargyak. In September 2009 the Surya Civic Association finished construction of the specially designed “passive” school-building heated by sun, fully ecological and also compatible with local structures also in terms of appearance.

It’s pretty cool and I suggest you take a look.

Broadcom is part of the Linux Foundation

/ | Leave a Comment

When you look at the Linux landscape, there are traditionally two companies which are well known for lacking support when it comes to Linux. ATI and Broadcom. Well, good news when looking at the Broadcom landscape! Broadcom, well known for frequently holding out when it came to delivering drivers – especially Wi-Fi drivers and source code for Linux users has officially joined the Linux Foundation, with plans to extend its open development and collaboration with the Linux community.

The move announced a couple of weeks ago might just be the biggest change in the Open Source landscape* this year.

Heavily contributing to the reputation Linux had for not being as compatible with essential technologies such as Windows and OS X, Broadcom’s decision to join the Linux Foundation and release new open source drivers (brcm80211).

Since the release of that new open driver, it has been integrated into the recent Linux kernel release 2.6.37, where it can be actively improved upon by the entire Linux community.

Off topic a little bit, we need to see more Open Source organisations open up to the community their proprietary drivers.

Although the opening up of commercial code in the kernel is one of my favorite topics, to save myself typing out a whole new discussion on the topic I will simply point over to David Airlie, the maintainer of Maintainer of all the direct rendering manager code and the X.org co-maintainer for Red Hat, covering both Fedora and RHEL.

David goes on to ask one very important question: ‘So what are they [orgnaistaions providing closed source userspace drivers] actually hiding in userspace?

PHP Fog gets $1.8m in funds

/ | Leave a Comment

PHP is one of the most popular web development language in the world, with millions of active developers and tens of millions of PHP-based sites already in deployment.

A new company on the PHP scene has been created to service the hundreds of thousands of application developers who work with PHP on a daily basis.

PHP Fog is the only company offering deployment and infinite scaling of PHP applications in the cloud with little to know effort. The company offers one-click deployments for many popular PHP apps and frameworks including WordPress, Drupal, Kohana, Zend, and SugarCRM.

The company is currently in private beta but expects to launch publicly in the first half of 2011.

The technology around PHP Fog allows you to create a new site in less then one minute of your time. You do not even need to know how to code (although, it really, really is beneficial if you can :)).

Deploying changes to your site is as easy as git push, No longer do you need to work with FTP or any other 1990’s technology. PHP Fog has been built to work around the developer – which is a good thing, because it means that your developers are no longer system administrators – especially in smaller shops.

With PHP Fog, all of your servers (cache, web/http, database servers and load balancers) are on separate boxes

It will be interesting to see how this technology evolves. A similar provider for Rails, Heroku, was not too long ago acquired by Salesforce for $212 million in cash. What company will buy PHP Fog and for how much?

Yahoo are big users of PHP, so they could be in line. Facebook seem to have gone their own way with Hiphop. Interesting times ahead! I can’t wait to see what times I can get out of PHP Fog for page loads!

Phase III of PA-DSS and what it means for you

/ | Leave a Comment

A major PCI-related deadline came and went on the 1st of October that many PCI/PA DSS compliant people may not have even felt fly past, in fact most merchants were not even aware of the details, even though they’ve been given an entire year to get up to speed.

It is known as “Phase III” of the PA DSS compliance mandates, and it is the first major push to get Level 3 and 4 merchants to wake up to the importance of PCI. Like other PCI related mandates, the drive to be compliant is being driven by Visa, through the all of the card acquirers.

This new push requires that card acquirers not board (ie, sign up) any new merchants that are not EITHER PCI compliant OR running PA DSS compliant payment applications. This is designed to stop merchants from switching from “tough” to “easy” acquirers, among other objectives.

What does this mean to appication developers and merchants? Well, merchants have either two options (as stated above) and that is to either be PCI-DSS compliant or be running an application that is certified to be PA-DSS.

Complying with the PCI Data Security Standard (PCI DSS) is vital for all merchants who accept credit cards, online or offline, because nothing is more important than keeping payment card data secure. The size of your business will determine the specific compliance requirements that must be met, however all levels require the same basic set of requirements (that is, make sure that data will not escape the corporate network unless it is specifically required to. Make sure that employees are trusted and log everything).

From the world’s largest corporations to small Internet stores, everyone is required to comply. Fines are even issued to organisations that do not comply! There is a whole list of things that are being added to the requirements to pass the PCI DSS compliency, including ensuring that access is restricted by secure PIN Entry Devices (PEDs). PCI hell is only going to get bigger in the coming years! What this space.

Handy Hints: Create random MAC addresses

/ | Leave a Comment

Quick and simple MAC address generator:

dd if=/dev/urandom bs=1 count=3 2>/dev/null | od -tx1 | head -1 \
         | cut -d' ' -f2- | awk '{ print "00:50:19:"$1":"$2":"$3 }'

I normally have this set up as an alias in my profile under ‘mac’.

Google owns On2. What now?

/ | Leave a Comment

When it comes to companies that create multimedia codecs, I have followed none closer then that of On2 Technologies.

On2 came to my attention when TrueMotion VP3 was released as Open Source Software in late 2001. On2 released the source code of their VP3 codec. They irrevocably disclaimed all rights to the codec and gave all individuals and organisations working on the code a royalty-free license grant for any patent claims it might have over the software and any derivatives their patents on the video codec technology. On2 made a further donation to the Open Source community when in 2002 it re-released the source code under the LGPL and gave the source code of VP3 to the Xiph.org Foundation under a BSD license to create what is now known as ‘Theora’.

Their video codec “TrueMotion” amazed me with the quality that it could provide with such low bitrates, and the technology in VP4, VP5 and VP6 continued to just get better. Unfortunately, even after (or maybe because of) Flash 8 supporting VP6 videos natively code did not continue to flow into the open source community.

Comparison of different video formats on the web

VP6 has always been a popular format on the web, but since the arrival of H.264 on the codec scene this has most surely changed. On August 26, 2010 MPEG LA announced that H.264 encoded internet video that is free to end users will never be charged for royalties, however, the license terms are updated in 5-year blocks which does not give the MPEG Licensing Alliance that long* to change their minds.

* Yes, 5 years is a long time, but not in ‘technology’ time. A lot of revolutions occur in five years time. Five years is not a long time for a Licensing organization such as MPEG LA to change their minds on something as crucial as what format your videos will be licensed under.

On 19 February 2010, Google had completed the acquisition of On2 Technologies. By May 19, 2010 something that I long dreamed of occurred. Google finally open sourced the latest version of the On2 codec.

Finally seeing a codec such as On2’s VP8 as a fully open source standard, with everything open sourced is a fantastic sight. We can all only hope that Google will ensure that the technology only gets better from here on in.

MPEG LA has claimed that video codecs such as Theora and VP8 infringe on patents owned by its licensors, without disclosing the affected patent or patents. That does not sound like SCO all over again, does it?