Archive for December, 2009

lazy function loading

December 8th, 2009

Even though bash is not my favorite programming language, I end up writing a bit of code in it. It's just super practical to have something in bash if you can. I mentioned in the past how it's a good idea to avoid unnecessary cpu/io while initializing the shell by doing deferred aliasing. That solves the alias problem, but I also have a bunch of bash code in terms of functions. So I was thinking the same principle could be applied again.

Let's use findpkgs as an example here. The function is defined in a separate file and the file is source'd. But this means that every time I start a new shell session the whole function has to be read and loaded into memory. That might not be convenient if there are a number of those. networktest, for instance, defines four function and is considerably longer.

So let's "compile to bytecode" again:

findpkgs ()
{
    [ -f ~/.myshell/findpkgs.sh ] && . ~/.myshell/findpkgs.sh;
    findpkgs $@
}

When the function is defined this way, the script actually hasn't been sourced yet (and that's precisely the idea), but it will be the minute we call the function. This, naturally, will rebind the name findpkgs, and then we call the function again, with the given arguments, but this time giving them to the actual function.

Okay, so that was easy. But what if you have a bunch of functions loaded like that? It's gonna be kinda messy copy-pasting that boilerplate code over and over. So let's not write that code, let's generate it:

lazyimport() {
# generate code for lazy import of function
	function="$1";shift;
	script="$1";shift;

	dec="$function() {
		[ -f ~/.myshell/$script.sh ] && . ~/.myshell/$script.sh
		$function \$@
	}"
	eval "$dec"
}

Don't worry, it's the same thing as before, just wrapped in quotes. And now we may import all the functions we want in the namespace by calling this import function with the name of the function we want to "byte compile" and the script where it is found:

## findpkgs.sh

lazyimport findpkgs findpkgs

## networktest.sh

lazyimport havenet networktest
lazyimport haveinet networktest
lazyimport wifi networktest
lazyimport wifiscan networktest

## servalive.sh

lazyimport servalive servalive

So let's imagine a hypothetical execution. It goes like this:

  1. Start a new bash shell.
  2. Source import.sh where all this code is.
  3. On each call to lazyimport a function declaration is generated, and eval'd. The function we want is now bound to its name in the shell.
  4. On the first call to the function, the generated code for the function is executed, which actually sources the script, which rebinds the name of the function to the actual code that belongs to it.
  5. The function is executed with arguments.
  6. On subsequent executions the function is already "compiled", ie. bound to its proper code.

So what happens, you may wonder, in cases like the above with networktest, where several mock functions are generated, all of which will source the same script? Well nothing, whichever of them is called first will source the script and overwrite all the bindings, remember? It only takes one call to whichever function and all of them become rebound to the real thing. So all is well. :)

I must stop being amazed

December 4th, 2009

Amazement is something for a special occasion. It is supposed to be rare. It is supposed to be worth a story. It is not for everyday use.

I amaze so easily, and so frequently, that amazement has ceased to be special to me. It has become mundane. I need to check my standards for amazement. I need to raise the bar. I need to make amazement once again worth having.

I must stop being amazed, for example, when a man rings my doorbell because he cannot figure out the house numbers on my street. "Is this number thirty", he asks. I lean out, in a mock gesture, to gaze at the street number opposite my front door. It does not say thirty. I imagine this gesture will suffice to make him understand. Instead he reiterates his dilemma. "Is this number thirty." No. It is not. I must not be amazed, even if it is a man in his fifties. With gray hair, an elegant tie, and a fancy suit. Who proceeds to reenter his expensive automobile. How does a person like that not know how to read street numbers. I must not be amazed.

I must not be amazed, either, at the communal workers. Who must necessarily have intimate knowledge of such complicated administrative intricacies as are street numbers. Through their work of visiting various addresses everyday. Who still ring my doorbell by mistake.

One wonders how such people can accomplish complicated tasks like air travel, which requires all sorts of documents, procedures, requires remembering important facts and following a timetable. How do they manage it? It seems amaz I'm sure they pull it off somehow.

i nuotatori socievoli

December 2nd, 2009

Ero in piscina un bel giorno. Stavo nuotando quando mi sono accorto del fatto che a qualche persona piace nuotare insieme. Sono in due, addirittura in tre qualche volta. Nuotano insieme, uno accanto all’altro. Vogliono chiacchierare quando sono in piscina, il nuoto è diventato un momento sociale. Sì, sì, perché non ci incontriamo in piscina. Sicuramente questo non disturba nessuno. A parte le altre persone che in piscina vogliono semplicemente nuotare. Non avete mai visto un bar?? Ci si siede, si chiacchiera quanto si vuole. Non lo fate in piscina, dove state bloccando gli altri! Avere due persone in corsia è come avere una persona molto più grande. Non vogliono dare spazio quando devi superarli ma allo stesso tempo stanno nuotando troppo lentamente per poter stare loro dietro. Allora fai bene ad evitare completamente quel lato della piscina.

Ma non finisce qui. Visto che nuotare insieme è una bella occasione per vedersi, non basta solo nuotare. Quando raggiungono la fine vogliono prendersi una pausa. Naturalmente, stanno sempre insieme, dunque ora bloccano il muro, che serve a tutti per spingersi quando si nuota. Una volta c’erano due gruppi che facevano una pausa: uno di due persone, l’altro di tre. Volevo gridare a loro: “Questo non è un parcheggio! Via! Via!”

peek: monitor files for changes

December 1st, 2009

It seems to me that we have pretty good tools for managing files that aren't changing. We have file managers that display all the pertinent details, they'll detect the file type, they'll even show a preview if the content is an image or a video.

But what about files that are changing? Files get transfered all the time, but network connections are not always reliable. Have you ever been in the middle of a transfer wondering if it just stopped dead, wondering if it's crawling along very slowly, too slow, almost, to notice? Or how about downloading something where the host doesn't transmit the size of the file, so you're just downloading not knowing how much there is left?

These things happen, not everyday, but from time to time they do. And it's annoying. A file manager doesn't really do a great job of showing you what's happening. Of course you can stare at the directory and reload the display to see if the file size is changing. (Or the time stamp? But that's not very convenient to check against the current time to measure how long it was since the last change.) Or maybe the file manager displays those updates dynamically? But it's still somewhat lacking.

Basically, what you want to know is:

  1. Is the file being written to right now?
  2. How long since the last modification?

And you want those on a second-by-second basis, ideally. Something like this perhaps?

peek

Here you have the files in this directory sorted by modification time (mtime). One file is actively being written to, you can see the last mtime was 0 seconds ago at the time of the last sampling. Sampling happens every second, so in that interval 133kb were written to the file and the mtime was updated. The other file has not been changed for the last 7 minutes.

The nice thing about this display is that whether you run the monitor while the file is being transfered or you start it after it's already finished, you see what is happening, and if nothing is, you see when the last action took place.

#!/usr/bin/env python
#
# Author: Martin Matusiak <numerodix@gmail.com>
# Licensed under the GNU Public License, version 3.
#
# <desc> Watch directory for changes to files being written </desc>

import os
import sys
import time


class Formatter(object):
    size_units = [' b', 'kb', 'mb', 'gb', 'tb', 'pb', 'eb', 'zb', 'yb']
    time_units = ['sec', 'min', 'hou', 'day', 'mon', 'yea']

    @classmethod
    def simplify_time(cls, tm):
        unit = 0
        if tm > 59:
            unit += 1
            tm = float(tm) / 60
            if tm > 59:
                unit += 1
                tm = float(tm) / 60
                if tm > 23:
                    unit += 1
                    tm = float(tm) / 24
                    if tm > 29:
                        unit += 1
                        tm = float(tm) / 30
                        if tm > 11:
                            unit += 1
                            tm = float(tm) / 12
        return int(round(tm)), cls.time_units[unit]

    @classmethod
    def simplify_filesize(cls, size):
        unit = 0
        while size > 1023:
            unit += 1
            size = float(size) / 1024
        return int(round(size)), cls.size_units[unit]

    @classmethod
    def mtime(cls, reftime, mtime):
        delta = int(reftime - mtime)
        tm, unit = cls.simplify_time(delta)
        delta_s = "%s%s" % (tm, unit)
        return delta_s

    @classmethod
    def filesize(cls, size):
        size, unit = cls.simplify_filesize(size)
        size_s = "%s%s" % (size, unit)
        return size_s

    @classmethod
    def filesizedelta(cls, size):
        size, unit = cls.simplify_filesize(size)
        sign = size > 0 and "+" or ""
        size_s = "%s%s%s" % (sign, size, unit)
        return size_s

    @classmethod
    def bold(cls, s):
        """Display in bold"""
        term = os.environ.get("TERM")
        if term and term != "dumb":
            return "\033[1m%s\033[0m" % s
        return s

class File(object):
    sample_limit = 60  # don't hold more than x samples

    def __init__(self, file):
        self.file = file
        self.mtimes = []

    def get_name(self):
        return self.file

    def get_last_mtime(self):
        tm, sz = self.mtimes[-1]
        return tm

    def get_last_size(self):
        tm, sz = self.mtimes[-1]
        return sz

    def get_last_delta(self):
        size_last = self.get_last_size()
        try:
            mtime_beforelast, size_beforelast = self.mtimes[-2]
            return size_last - size_beforelast
        except IndexError:
            return 0

    def prune_samples(self):
        """Remove samples older than x samples back"""
        if len(self.mtimes) % self.sample_limit == 0:
            self.mtimes = self.mtimes[-self.sample_limit:]

    def sample(self, mtime, size):
        """Sample file status"""
        # Don't keep too many samples
        self.prune_samples()
        # Update time and size
        self.mtimes.append((mtime, size))

class Directory(object):
    def __init__(self, path):
        self.path = path
        self.files = {}

    def prune_files(self):
        """Remove indexed files no longer on disk (by deletion/rename)"""
        for f in self.files.values():
            name = f.get_name()
            file = os.path.join(self.path, name)
            if not os.path.exists(file):
                del(self.files[name])

    def scan_files(self):
        # remove duds first
        self.prune_files()
        # find items, grab only files
        items = os.listdir(self.path)
        items = filter(lambda f: os.path.isfile(os.path.join(self.path, f)),
                       items)
        # stat files, building/updating index
        for f in items:
            st = os.stat(os.path.join(self.path, f))
            if not self.files.get(f):
                self.files[f] = File(f)
            self.files[f].sample(st.st_mtime, st.st_size)

    def display_line(self, name, time_now, tm, size, sizedelta):
        time_fmt = Formatter.mtime(time_now, tm)
        size_fmt = Formatter.filesize(size)
        sizedelta_fmt = Formatter.filesizedelta(sizedelta)
        line = "%6.6s   %5.5s   %6.6s   %s" % (time_fmt, size_fmt,
                                               sizedelta_fmt, name)
        if time_now - tm < 6:
            line = Formatter.bold(line)
        return line

    def sort_by_name(self, files):
        return sorted(self.files.values(), key=lambda x: x.get_name())

    def sort_by_mtime(self, files):
        return sorted(self.files.values(),
                      key=lambda x: (x.get_last_mtime(),x.get_name()))

    def display(self):
        time_now = time.time()
        files = self.sort_by_mtime(self.files.values())
        print("\nmtime>   size>   delta>   name>")
        for f in files:
            line = self.display_line(f.get_name(),
                                     time_now, f.get_last_mtime(),
                                     f.get_last_size(), f.get_last_delta())
            print(line)


def main(path):
    directory = Directory(path)
    while True:
        try:
            directory.scan_files()
            directory.display()
            time.sleep(1)
        except KeyboardInterrupt:
            print("\rUser terminated")
            return


if __name__ == '__main__':
    try:
        path = sys.argv[1]
    except IndexError:
        print("Usage:  %s /path" % sys.argv[0])
        sys.exit(1)
    main(path)