Nov 12

Cluster 101: Building your first dumb cluster

Experimental, Linux No Comments »

I would like to introduce you tonight to a x-part series (I don’t know how many parts there are at the moment, we will see as time progresses) entitled “How to build a www cluster in x days“.

In this series, we will be looking at a wide variety of different techniques that can be used to create a cluster for hosting your next large project on.
When I talk about clusters, I like to categorise them into two particular sections: dumb and intellegent clusters. A dumb cluster is a group of machines that essentially know very little of their surroundings. They do not take into account how many machines are currently running, where the machines are located or how the data is to be devlivered to it’s location. Put simply: you set it up, and it works. No complex heartbeat configurations (except when necessary) and nothing fancy.
Intelligent clusters on the other hand know alot of information about where other machines on the network are. They know where the vistor to the website is coming from, and what the fastest way is to deliver content to that user.
Put simply: a “dumb cluster” is a group of machines that are set up to deliver content. If one machine goes down, the load balancer will detect this and bring that node out of the server pool until it is reported back up again. an “intelegent cluster” knows where the user is visiting. It can take into consideration what the shortest internet route is to it’s desired location, or what route should be taken to get the best availible speeds.
In this series, we will be looking more at the dumb clustering side of things. Dumb clusters are easy to set up, and well, if you really want an intellegent cluster, hire me :] So lets look at what we are going to discuss in this series.
  1. Setting up our first webserver
  2. Setting up our shared filesystem
  3. Setting up our http/https load balancer
  4. Setting up our mail server
  5. Setting up our backup system
  6. Setting up our mysql cluster
  7. Setting up our shell server
  8. Setting up our TCP/IP load balancer
For the purpose of writing this series, I will be using Xen installed on my machine, using CentOS 5.4 as the base opperating system. At any one time, I will have up to five domains running on Xen. I will be testing loads and response times using my laptop.
Stay tuned, the first part of the series comes on Tuesday.

written by Tim Groeneveld \\ tags: , , , , ,

Jun 11

Getting Synergy working propperly on Windows XP / Ubuntu

Experimental, Linux No Comments »

At Digital Pacific, where I work, I do all my work on three computers. It’s a nice life looking at three monitors all the time, however, it is not the coolest thing in the world when you have three keyboards in front of you.

When I moved here, I already knew about Synergy, it’s great software.

My setup is comprised of a Windows XP (32-bit) machine, Kubuntu [7.04] Intrepid (32-bit) machine and a Kubuntu [9.04] Jaunty (64-bit) machine. The machine that runs all my applications that I write is the new Kubuntu 9.04 machine. To allow my keyboard and mouse to be used over all three of these machines, I have the Windows XP machine run the Synergy server, and all the Linux machines run the client. The reason I have Windows XP run the server is because it’s usually the machine that’s off. That might sound strange, but I do alot of work from home, and it seems useless to have a server running that is not needed to be on the machine.

One thing that really used to annoy me with this setup however was the Windows machine would detect all keyboard repeats, but the Linux machines would not detect up, down, page up and page down repeated key presses. You could imagine as a programmer that this could get quite annoying while writing code.

This morning I got frustrated at the Linux machines and I decided it was time to go and compile my own version, with a few patches I found around the net to fix this issue.

Being the package Nazi that I am, I don’t really like to install anything, unless it’s installed with some sort of package management. This is how to rebuild Synergy, and get a perfect .deb that you can install :) 

# wget 'http://internode.dl.sourceforge.net/sourceforge/synergy2/synergy-1.3.1.tar.gz'
# tar -xvf synergy-1.3.1.tar.gz
# wget http://ftp.de.debian.org/debian/pool/main/s/synergy/synergy_1.3.1-5.diff.gz
# gunzip synergy_1.3.1-5.diff.gz
# wget http://coderazzi.net/linux/synergy.patch/synergy-coderazzi.patch
# patch -P0 <../synergy-coderazzi.patch
# sudo apt-get install patch build-essential
# cd synergy-1.3.1/
# patch -p0 <../synergy-coderazzi.patch
# chmod +x debian/rules debian/control

Edit debian/rules and after the line that says:
CFLAGS="$(CFLAGS)" CXXFLAGS="$(CXXFLAGS)"
Add the following line:
find . -type f -name Makefile | xargs perl -pi -e "s/-Werror//g"

# sudo dpkg-buildpackage
# cd ../
# dpkg -i synergy_1.3.1-5*.deb

After doing this, and restarting the Synery clients, you will have perfect key responses between all the machines.

Thanks to http://coderazzi.net/linux/synergy.patch/index.htm for providing the great patch for this post.

written by Tim Groeneveld \\ tags: , , ,

Aug 08

My Xen experience

Experimental No Comments »

No! When I Xen, I am not talking about a relationship to foreign material. Well, that is – not exactly. Xen is an open source hypervisor. Think: VMware, but much, much cooler.

I was talking to Tim Post, about a project that I was completing for ShareSource2. Basically, the conversation came to the point where I said that I was using OpenVZ for my server virtualization. Tim then came to me and said “Why don’t you just use Xen”.

Well, I was not born yesterday – I did tell him that I had tried Xen, and that when I tried it – it just refused to work.

Then Tim pointed me to Xen unstable, a mercurial repository containing a version of Xen that does not need a new CPU to actually boot up.

Xen unstable, the version I checked out yesterday, is actually tagged as being “3.2.0-rc4″, which probably means that it is coming it’s way to becoming a nice stable Xen product that people other then me have actually been able to use.

This is what I used on my Debian to be able to install a Xen-enabled kernel:

~# mkdir Xen
~# cd Xen
~# hg clone http://xenbits.xensource.com/xen-unstable.hg
[[ wait for around 3 minutes while it grabs Xen's history
~# cd xen-unstable.hg
~# apt-get install bin86 bcc
~# make world
[[ Note: this step will also check out a 2.6.18.8-xen kernel ]]
~# sudo make install

Then I had to add an entry to my /boot/grub/menu.lst which looked like this:

title Xen 3.2 / XenLinux 2.6
kernel /boot/xen-3.gz console=vga
module /boot/vmlinuz-2.6.18.8-xen root=UUID=66559e96-e405-4a1d-be28-b8caf5b5b89d ro console=tty0
module /boot/initrd-2.6.18.8-xen.img

Reboot, select my new kernel, and faster then you can say “WOW IT WORKS” it was booting me into a brand new (well, actually, my OpenVZ kernel was 2.6.20 – so a brand old) kernel

To try Xen out, just go over to http://jailtime.org and download an image of a nice operating system (I chose CentOS 5.1). When you have downloaded your image, do the following:

tar -xvf {downloaded image name}
ls -lar
xen create -c {smallest file (usually)}

and Voila! First thing you should be introduced to is a nice Grub screen welcoming you to your Xen experience.

I am still as of today yet to actually have built my own Archlinux or Frugalware Xen image. If anyone knows how to do this successfully, please drop me a line or leave a comment!

written by Tim Groeneveld \\ tags: , , , ,

Jul 27

Ruby on Rails

Experimental 3 Comments »

A good friend of mine, Jordan Bracco recently forced me to rediscover Ruby on Rails. He told me that the new version, 1.9 is “5000x faster”, so I thought, well, why not!

The last version of Ruby I tried was either version 1.7 or 1.8, and it was not really something that I would say rocked my boat. It was slow, and just plain did not do (at that time what I though) I wanted.

Me being a version freak – and a speed freak, decided that I would go to the Ruby, check out their CVS repository out (seriously, these guys need to update to mercurial, just like Mozilla has!) and compiled Ruby’s latest trunk version.

The speed speaks for it’s self. A basic “Hello World” Ruby on Rails app does 700 requests a second, while a basic “Hello World” in PHP does 800. For this use case, yes, PHP is faster, but what you have to remember is that Ruby on Rails is much, much, much more sophisticated in terms of functionality.

I recently rewrote a page that is on ShareSource.org, and it is almost twice as fast. The difference here, is that the Ruby rewrite had no cache. Every single hit with Apache benchmark was generated on demand, while the PHP version of the ShareSource.org page in question was kept in cache for five minutes.

Aaaahhh, I am loving Ruby already.

written by Tim Groeneveld \\ tags: , ,

Jul 26

Anouncing MyBanco!

Experimental, MyBanco 1 Comment »

Ever wanted to run your own bank? Well, my new [experimental] web application allows you to do exactly that. While some may say that I just have way too much time on my hands, others might just say that I should get top marks for my IPT [Information Processing Technology] assignment.

We had to build a database system, which implemented relational databases, and I thought, well, I could make my very own banking software, and fulfil one of my many dreams: to run my own bank.

The software, written in PHP was a testing ground for many new things that I have wanted to go and try, but never really had a project to try them out on.

I must say that MyBanco would have to be the first application that I have written with the mindset of handling around 700 transactions every second.

MyBanco uses a backend for it’s database which I have dubbed ‘MyInfo’. MyInfo is a JSON-RPC like protocol, which is easy to use, and can send multiple packets (I just thought then it might have been better to call these chunks… hmmmm) all at once, and get a response from all those packets. Think of it as a easy to read and faster to pharse version of XML-RPC.

I will chat at MyBanco a little more later, when the time is right, but for the moment this post will do :)

(hope you like the new blog!)

written by Tim Groeneveld \\ tags: , , , ,