Checking what processes need to be restarted after a system upgrade

Posted by & filed under Linux, RedHat.

With updates going on in the last couple of months for various packages, such as OpenSSL and GLibC which have fixed a number of important security vulnerabilities, I thought I might share a one liner that might save you one day.

sudo lsof -n | grep -v \#prelink\# | grep -e '\.so' | grep -e DEL | grep -e lib | grep -v ^init | sed -re 's|^([^0-9]*)\s*([0-9]*)[^/]*(\/.*)$|\1 (\2) \3|' | sort -u

Running lsof will list all of the currently opened files from processes running on your system. -n will stop lsof from resolving hostnames for processes that have opened network ports to different processes (such as your webserver, mail server etc)

Running grep a couple of times will ensure that we find all the processes that have loaded a shared binary that has been deleted.

Note that the “init” process has been excluded. This is done on purpose. init can not be restarted without rebooting or otherwise killing the system.

The sed magic will show a list of all the processes and their PID’s, along with the library that was deleted that triggered it being listed as an application that should be restarted.

tim@srv215.production [~]# REGEX='^([^0-9]*)\s*([0-9]*)[^/]*(\/.*)$|\1 (\2) \3'
tim@srv215.production [~]# sudo lsof -n | grep -v \#prelink\#  \
                            | grep -e '\.so' | grep -e DEL | grep -e lib \
                            | grep -v ^init \
                            | sed -re "s|$REXEG|" | sort -u
auditd      (1802) /lib64/
auditd      (1802) /lib64/
auditd      (1802) /lib64/
auditd      (1802) /lib64/
auditd      (1802) /lib64/
auditd      (1802) /lib64/
auditd      (1802) /lib64/

Note that this will not work if the application is dynamically loaded (for example using dlopen(3)) or if the application is statically linked.

Google+: Please give us a write API

Posted by & filed under Random Thoughts.

It seems that Google has been on a relentless campaign to get Google+ to work. From only allowing YouTube users to comment with a Google+ account to their desperate attempts to integrate it into Google’s own search results (which seem to be all but gone now…) it all seems to be very pointless on Google’s behalf.

Their problems have been two sided. Firstly, they have battled the massive loss of data that has happened thanks to Facebook now having much of the information that was once the forte of GMail (although they still do have YouTube) but on their own side, it seems that Google have still not really wanted to side with people trying to integrate with their own social platform.

With Facebook and Twitter, it is extremely simple to get external applications to post to the respective accounts. Seriously, you can get an API account set up literally in three minutes, and information posted on either of the respective streams within another 20 minutes.

Google+? Dream on.

Despite the fact that Google has released this functionality to a select group of partners (of which only HootSuite comes to mind…), Google after more then three years seems to feel that this functionality should be all but unavailable.

Issue #43 has been around in Google+’s platform issue tracker since September 2011 and to this day there have been nothing but excuses.

More then 400 developers ask Google for a feature inside their API, and the best that Google can come up with is that their API needs to be perfect before they release it.

People have their hopes on write access to Google+ being a well kept secret that is to be revealed at Google I/O this year, but I am unsure about that.

I don’t get what is the deal? Why does Google have a social platform that still has no documentation for writing to the Google+ streams? You look at all of the major social platforms like Twitter and Facebook and there is much growth that can be attributed to having an API.

Google’s mission is “[…] to organize the world’s information and make it universally accessible and useful”. Here are developers that are hanging to give you (Google) information. Google: please get this; we want to be able to feed you data, and your not allowing is to put it into your hands.

With Facebook dropping it’s organic reach, it’s no longer useful to disseminate information on that platform, as the cost to benefit ratio is deteriorating as Facebook tries to get us to pay more to get the organic reach of days gone past.

Google’s inability to listen to clients wants (and somewhat arguably, their needs) have been their downfall.

They had their chance with a nice simple interface and the ability to comfortably share long form texts to be the replacement for blogs. Now all Google+ feels like a really noisy version of

These days, all I use Google+ for is to complain about the fact that I can’t post to Google+ using software, and instead must use their web interface. As someone who wants to post Google+ content for five different content brands with one or two pieces of original content per brand with twelve different writers, this is almost impossible. I don’t want to hand out these login details to my writers, they should be able to share the content from within their content management platform.

But really, this ranting is all in vain. I already quit Google+ for my brands last month. I asked Google for these features, they never came.

No difference was ever felt. Even with Facebook’s decreased organic reach, one brand that has more followers on Facebook receives a higher click through rate for content then a Google+ page with more followers.

Using files or Semaphores for long running process locks with PHP

Posted by & filed under Linux, PHP.

Last week I spoke at SydPHP, which was hilariously horrible due to my lack of public speaking skills.

During my Introduction to Laravel & Composer there was a very interesting question posted asking about an issue that he came across while developing a cron that was running on his server.

The process was set to be started every minute to go and process some data. The problem was he didn’t know how long it would take for the data to processed. Sometimes it could take five seconds, other times it could take five days.

To try and solve the problem, the process would attempt to block the PHP process by running by using the sem_acquire function. This worked. That is, until the same process was launched multiple times and the request to acquire a semaphore would ultimately fail.

So, the first part of the problem is that semaphores like everything else to do with a computer have a limit to them. Semaphores are different to other methods of locking because they main purpose for existing is to control access to data when developing software that will be needing to access data from multiple threads (i.e., parallel programming).

Viewing semaphore limits can be done by the ipcs command, which is provided by the util-linux-ng package in RHEL and Arch Linux.

root@staging [~]# ipcs -ls

------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767

A call to sem_get will add one extra array into the kernel if the key has not already been used, which is fine, but the important part is that at least on RHEL, there is a limit of 250 semaphores per array. This means that after the 251th attempt, sem_acquire will fail to get a semaphore.

There is no simple way to fix this. There is essentially two options. Either the maximum number of semaphores created is increased, or you create less semaphores. You don’t really want to add more semaphores though, and the number of arrays set by default is actually a very forgiving number.

If you wanted to see what the kernel’s current settings for semaphores are without using ipcs, you could use find the information from /proc/sys/kernel/sem.

root@timgws ~ # cat /proc/sys/kernel/sem
250 32000   32  128

The numbers are separated by tabs, and are represented in this order:

  1. 250 The number of semaphores per array.
  2. 32000 The limit of semaphores system wide (should be roughly 250*128 i.e., [semaphores per array]x[max number arrays]).
  3. 32 Ops per semop call
  4. 128 Number of arrays

The configuration can be written out by using printf

printf '250\t32000\t32\t200' >/proc/sys/kernel/sem

… but you said semaphores are bad…

Well, not exactly. Semaphores are amazing for the purpose they were built for, that is, preventing processes from accessing a resources while another process is performing operations on it. However, if you are not going to need access to the resources straight away, then you don’t want to use a semaphores, and the reasons are plentiful.

Files are awesome when it comes to locking two processes from running at the same time. It’s not just me who thinks that, too!

RPM uses files to lock the application. When you attempt to install two packages from two processes at one time, the process that is launched the second time will fail, thanks to the first application creating a file saying that RPM is locked.

flock is more portable than sem_get. (Semaphores don’t work on Windows, files however do work on Windows. With caveats.).

Here is a simple lock class that I wrote. It will check if a file exists, if it doesn’t, it will be created.


class ProcessLock {

    private $lockFilePath = '/tmp/tim.lock';

    function aquire() {
        if (file_exists($this->lockFilePath))
            return false;

        // Get the PID and place into the file...
        $pid = getmypid();

        // Hopefully our write is atomic.
        @file_put_contents($this->lockFilePath, $pid);

        if (!file_exists($this->lockFilePath))
            return false; // our file magically disapeared?

        $contents = file_get_contents($this->lockFilePath);

        $lockedPID = (int)$contents
        if ($pid === $lockedPID)
            throw new Exception("Another process locked before us for somehow");

        return false;

    function release() {
        return true;

To use this class, we simply create a new instance, and attempt to acquire the lock. If successful, then we run our code.

$myLock = new ProcessLock();

$gotLock = $myLock->aquire();

if ($gotLock) {
    // ... this is where I put all of my code that only
    // one process should run at a time

    // Then we release
} else {
    echo "Can't run, sorry";

When the lock has been acquired, you might get bonus points if you check if the process is still running or if the lock is actually stale. This can be done by checking if /proc/$lockedPID exists, and if it does, if /proc/$lockedPID/exe is still symlinked (using readlink) to PHP_BINARY (though this will only work on Linux).

Nagios plugin for checking the status of supervisord processes

Posted by & filed under PHP.

If you have never used it before, supervisor is an application that runs on your server to monitor a number of different applications. With supervisord, you get all of the benefits of turning long-running applications into daemons without all the extra code required to make that happen natively.

The configuration for supervisord is extremely flexible, and one of the things that it will allow you to do is go and configure how many times a process will get restarted, the minimum amount of time an application should be running and what the action should be if your long-running process crashes (i.e., should it restart, or do nothing?).

An example configuration for a project that is monitored by supervisors:

command=/opt/nodejs/bin/node /opt/<project>/server.js


Inside the configuration files of Supervisor, you specify the process that you want to monitor, how many copies of the same program that you want to run, and then reload supervisor. If that program crashes, supervisor will detect the signal from the child process it spawned and will go and restart the process for you. No more wondering if your process is still running, and no more running your applications in screen.

Also, there is no more second guessing if your application is still running. Almost, that is.

The program will follow all the rules that you specify for restarting processes.

Luckily, Supervisor comes backed in with a web server, which is configured by setting up the [inet_http_server] section inside /etc/supervisord.conf. By giving Supervisor an IP address and port to listen to, you will get an integrated interface where all of the processes that are currently running are listed.

Which is fine for development, but when your running long-running apps in production, normally you would also want to have those processes monitored to ensure that there are no issues with the availability of your application.

Queue the nagios-supervisord-processes repository.

check_supervisor is a plugin for use with Nagios that uses supervisor’s XML-RPC API that is built into the same web server that is provided by Supervisord to check the status of any processes that are set up to be running inside supervisord.

If you have nagios checks already being performed, configuration is mind-numbingly simple to get up and running.

Install the script into your user scripts directory inside Nagios, add a command template and finally add one service for each program or service you want to monitor.

The nagios check will automatically find out if any of the processes started are not running and alert your system administrators accordingly.

Check out the project over at GitHub and open up an issue if you come across any issues or see any bugs.

The price of BitCoin

Posted by & filed under Random Thoughts.

The first pizza bought using Bitcoin was paid for using 10000 BTC in 2010. At today’s exchange rate that is about $6.7 million.

Installing all* the fonts on Red Hat EL6

Posted by & filed under Experimental, RedHat.

I am currently working on a project that takes screenshots of websites, and needed a quick way to get the most common fonts installed on the screenshot server. This worked wonders (although, it still needs a bit of tuning, I will work on that later.

yum install google-\*-fonts wine-fonts dejavu-\*-fonts

Internet Explorer 6 makes me sad.

Posted by & filed under Random Thoughts.

In this day and age, I can’t believe that people are still using Internet Explorer.

Because I am a good net citizen, I decided to help my uneducated users out.

if (preg_match('/MSIE [456]/i', $_SERVER['HTTP_USER_AGENT'])) {
require 'templates/blank_development/ie6.php';
Bye Bye IE

WordPress 3.5 is almost complete

Posted by & filed under PHP.

I am writing this blog article to you on the new WordPress 3.5.

The updates since 3.4 are really nice. I am especially enjoying the new media manager, with it’s easy to use ‘Drop files anywhere to upload’ screen. Drop files anywhere to upload

According to their Trac install, there is only six issues left to fix before they release the new update for everyone to consume.

See the remaining issues in the development version of WordPress, or try the WordPress Beta Tester plugin to test the very latest (trunk) WordPress right now.