Sorting Records With Emacs

I use sort-lines fairly frequently to keep my #includes, uses or variable names in order. Occasionally though, I need to sort something slightly more complicated and I end up writing a little perl script. I found myself thinking it must be equally easy in emacs.

For example, I say I have the following data which I want to sort by the second field which is in position 12 and 13.

Edward Wood 9   01/07/2005
Ed Smith    25  06/12/2004
James Brown 18  01/07/2005
Jon James   13  05/15/2007

Now, emacs does have a sort-columns function which does all sorts of fancy things including checking if we are using Unix and shelling out to sort if possible, but it doesn’t seem to be able to do a numerical comparison. However, in sort.el it does have a general purpose sorting function called sort-subr.

(defun sort-subr (reverse nextrecfun endrecfun
                          &optional startkeyfun endkeyfun predicate)

sort-lines itself is a good start for my custom function as it has all the code for handling regions, reversal and skipping between records (lines) that we need.

(defun sort-lines (reverse beg end)
  (interactive "P\nr")
  (save-excursion
    (save-restriction
      (narrow-to-region beg end)
      (goto-char (point-min))
      (let ((inhibit-field-text-motion t))
        (sort-subr reverse 'forward-line 'end-of-line)))))

Getting the startkeyfun and endkeyfun correct was fairly straight forward but I wasn’t entirely sure what information was passed to the predicate.

(defun sort-on-field (reverse beg end)
  (interactive "P\nr")
  (save-excursion
    (save-restriction
      (narrow-to-region beg end)
      (goto-char (point-min))
      (let ((inhibit-field-text-motion t))
        (sort-subr reverse 'forward-line 'end-of-line
                   (lambda () (forward-char 12))
                   (lambda () (forward-char 2))
                   (lambda (a b)
                     (message (format "[%s] [%s]" a b))))))))

[(703 . 705)] [(676 . 678)]
[(757 . 759)] [(730 . 732)]
[(757 . 759)] [(703 . 705)]
[(730 . 732)] [(703 . 705)]

The final working version:

(defun sort-on-field (reverse beg end)
  (interactive "P\nr")
  (save-excursion
    (save-restriction
      (narrow-to-region beg end)
      (goto-char (point-min))
      (let ((inhibit-field-text-motion t))
        (sort-subr reverse 'forward-line 'end-of-line
                   (lambda () (forward-char 12))
                   (lambda () (forward-char 2))
                   (lambda (a b)
                     (let ((s1 (buffer-substring (car a) (cdr a)))
                           (s2 (buffer-substring (car b) (cdr b))))
                       (< (string-to-number s1) (string-to-number s2)))))))))

Providing a nice way of asking for the start and end location of the sorting field is left as an exercise for the reader.

Emacs Lisp Keyword Parameters

I don’t create many lisp macros but it is still a huge advantage having them. I’m a big user of macros that other people have created. For example, by default emacs lisp does not support keyword parameters. Thanks to the magic of macros they have been added in the cl.el common lisp package.

(require 'cl)

(defun* example (&rest args
                 &key (a 1) (b 2))
  (format "%s %s %s" args a b))

(example) works as expected and returns "nil 1 2"

(example :a 10) is pretty good too although you may be surprised that keywords are not removed from the &rest args "(:a 10) 10 2"

However, (example 1) returns the following surprise (error "Keyword argument 1 not one of (:a :b)").

If you are using &rest and &key together, you can fix this with &allow-other-keys.

(defun* example (&rest args
                 &key (a 1) (b 2)
                 &allow-other-keys)
  (format "%s %s %s" args a b))

Okay, so much for pithy examples. How about something practical?

(defun* remote-path (&rest path
                     &key (user "jared") (server "someserver")
                     &allow-other-keys)
  (concat "ftp:" user "@" server ":"
          (mapconcat #'identity path "/")))

(remote-path :user "uat-user" "/tmp" "test-dir") throws this error (wrong-type-argument sequencep :user). This is because if you use &rest arguments and &key arguments together, the keyword arguments are also passed through into &rest.

I couldn’t find an obvious function to remove keyword parameters so I’ve written my own remove-keyword-params.

(defun remove-keyword-params (seq)
  (if (null seq) nil
    (let ((head (car seq))
          (tail (cdr seq)))
      (if (keywordp head) (remove-keyword-params (cdr tail))
        (cons head (remove-keyword-params tail))))))

(defun* remote-path (&rest path
                     &key (user "jared") (server "someserver")
                     &allow-other-keys)
  (concat "ftp:" user "@" server ":"
          (mapconcat #'identity (remove-keyword-params path) "/")))

Now it works correctly.

(remote-path :user "uat-user" "/tmp" "test-dir")
;; "ftp:uat-user@someserver:/tmp/test-dir"

So what do I want this for? For my directory aliases of course.

(defconst *dired-dirs*
  (list (cons "dev" (remote-path "~/dev"))
        (cons "dev:system"
              (remote-path :user "dev-user" "~/system"))
        (cons "uat:system"
              (remote-path :user "dev-user" "~/system"))))

And surprise surprise, because emacs is architected so well, it all works remotely through tramp completely transparently.

A Strange Tramp Mode Problem

Does anyone else use tramp? It is a mode that allows emacs to transparently edit remote files. For example, you have a file called afile.txt on a server called someserver.

/ftp:jared@someserver:~/afile.txt

Then behind the scenes emacs will ftp the the file to and from the server every time you save.

What Is Happening In Ange-ftp?

Recently tramp has been letting me down. When I try and access a remote-file it fails to login to the ftp process. Sad.

The ftp login as handled by ange-ftp. I finally tracked the problem down to this section.

(defun ange-ftp-get-process (host user)
      ;; ...
      (let ((pass (ange-ftp-quote-string
                   (ange-ftp-get-passwd host user)))

ange-ftp-get-process calls (ange-ftp-quote-string ...). Normally, this isn’t a problem. For example, on the Windows PC I am using at this precise moment (ange-ftp-quote-string "!\"£$%^&*()") expands to "!\"£$%^&*()". However, at work, I get:

"\\!\"£\\$\\%^\\&\\*()"

Could this be due to the customize EmacsW32 option? I’m not sure. Anyway, absent the underlying cause, my work-around is to (require 'ange-ftp) and then redefine the ange-ftp-get-process without the quoting of the password.

(require 'ange-ftp)

(defun ange-ftp-get-process (host user)
      ;; ...
      (let ((pass (ange-ftp-get-passwd host user))

Emacs Utility Functions

I have a number of utility functions in a file called my-utils.el. The first one I wrote was add-path.

add-path

(defun add-path (&rest path)
  (let ((full-path ""))
    (dolist (var path)
      (setq full-path (concat full-path var)))
    (add-to-list 'load-path (expand-file-name full-path))))

This means that when I add a new third-party library (in its own directory), I can just write:

(add-path *elisp-dir* "muse-3.12")

I have a function called identity that returns whatever was passed to it.

(defun identity (e) e)

This is particularly useful in functions like mapconcat.

(defun add-path (&rest path)
  (add-to-list 'load-path (mapconcat #'identity path "")))

I like this function that I found on the internet a while ago (can’t find it again for attribution unfortunately).

insert-YYMMDD

(defun insert-YYYMMDD ()
  (interactive)
  (insert (format-time-string "%Y%m%d")))

remove-dupes

remove-dupes removes duplicate items that are next to each other in a list in the same way that uniq does. (I mentioned this previously in one of the enabling your users posts.)

(defun remove-dupes (list)
  (let (tmp-list head)
    (while list
      (setq head (pop list))
      (unless (equal head (car list))
        (push head tmp-list)))
    (reverse tmp-list)))

assoc-replace

Sometimes you will have an assocation list where you just want to replace one of the values.

(defun assoc-replace (seq values)
  "Replace an element within an association list where the cars match."
  (mapcar (lambda (elem)
            (let* ((key (car elem))
                   (val (assoc key values)))
              (if val (cadr val) elem))) seq))

And often, only the cdr needs to change so kv is a shortcut for that case. (kv 'a 'b) returns (a (a . b)).

(defsubst kv (k v)
  `(,k (,k . ,v)))

(These functions were mentioned in emacs association lists)

ext-mode-map

ext-mode-map allows me to easily setup a major mode for extensions that are inconsistently capitalized.

(defun file-extensions (l)
  (concat "\\.\\("
          (mapconcat
           (lambda (s)
             (mapconcat (lambda (c)
                          (let ((c (upcase (char-to-string c))))
                            (concat "[" c (downcase c) "]")))
                        (symbol-name s) ""))
           l "\\|")
          "\\)\\'"))

(defun ext-mode-map (extensions mode)
  (cons (file-extensions extensions) mode))

Then, for example, I can setup my perl extensions like this:

(add-to-list 'auto-mode-alist (ext-mode-map '(pl perl pm) 'cperl-mode))

(These functions were previously mentioned in autoloading an emacs major mode)

duplicate-current-line

I’m still using this function to duplicate the current line. It works unless you are on the final line which is very rare and you can work around that by opening a line below.

(defun duplicate-current-line ()
  (interactive)
  (beginning-of-line nil)
  (let ((b (point)))
    (end-of-line nil)
    (copy-region-as-kill b (point)))
  (beginning-of-line 2)
  (open-line 1)
  (yank)
  (back-to-indentation))

(This function was mentioned previously in A Simple Emacs Shortcut – Duplicate Line)

set-longlines-mode

I often copy text into buffer and want to read it nicely wrapped around. I’ll change this function to use visual-line-mode when all my emacs versions are upgraded to 23.

(defun set-longlines-mode ()
  (interactive)
  (text-mode)
  (longlines-mode 1))

count-words

Other people have talked about their own count-words functions in the past. I have my own ideas about what constitutes a word.

(defun count-words ()
  (interactive)
  (let ((words (count-matches "[-A-Za-z0-9][-A-Za-z0-9.]*"
                              (point-min) (point-max))))
    (message (format "There are %d words" words))))

regex-replace and string-repeat

I sometimes have a text transform that I want to apply globally to a buffer. Exercise for the reader, fix regex-replace to only apply to a region if one is selected.

(defun regex-replace (regex string)
  (goto-char (point-min))
  (while (re-search-forward regex nil t)
    (replace-match string)))

(defun string-repeat (str n)
  (let ((retval ""))
    (dotimes (i n)
      (setq retval (concat retval str)))
    retval))

Strawberry Perl and POE

I’m (unfortunately) doing more work on Windows on these days so I’ve installed a mini Windows-dev environment at home. It is impressive what the libre software guys have done – between Emacs, Cygwin and Perl (and a rusty MinGW C compiler) I feel almost at home.

On Windows, there is an embarrassment of Perl options. I could choose from ActiveState Perl, Strawberry Perl, CygPerl or try and build it myself. I haven’t tried Strawberry Perl so I thought I’d give it a whirl. The install process was pleasant enough but after that I had some problems. I tried my usual perl -MCPAN -e shell incantation followed by installing (upgrading) the CPAN package but it didn’t work. Woe is me!

I reached for old reliable cygperl but that didn’t work either. Hmmm… fishy, I’m sure I haven’t had problems previously. Thankfully googling the exact error lead to a solution which unfortunately I didn’t make a note of. I do remember that it involved a rebase from the static shell, ash. Following that, Strawberry Perl worked too.

The next thing to do was to install POE. POE allows you to structure your code in an event-driven way. I’ve experienced some strange behaviour with POE on various Windows Perl implementations in the past so I want to test a few things.

My basic POE script.

use strict;
use warnings;

use POE;

POE::Session->create(
    inline_states => {
        _start => sub { $_[KERNEL]->yield('loop'); },
        loop   => sub {
            print "Normal loop\n";
            $_[KERNEL]->delay(loop => 1);
        },
    }
);

$poe_kernel->run();

That was a cumbersome way to implement a while 1 loop eh?

while (1) {
    print "Normal Loop\n";
    sleep 1;
}

The first thing I want to test is timeouts. I actually like the crufty way that timeouts are implemented in perl with eval / sig alarm. I use sleep here to simulate a long running process despite the warning in the documentation.

It is usually a mistake to intermix alarm and sleep calls. (sleep may be internally implemented in your system with alarm)

use strict;
use warnings;

use POE;

sub timeout_test
{
    eval {
        local $SIG{ALRM} = sub { die "alarm\n" }; # NB: \n required
        alarm 3;
        sleep 5; # This simulates a long-running process
        alarm 0;
    };
    if ($@) {
        die unless $@ eq "alarm\n"; # propagate unexpected errors
        print "Timeout\n";
        # timed out
    }
    else {
        # didn't
    }
}

POE sessions are a bit like co-operative threads. If the session doesn’t yield to the kernel, another session can’t run so if the timeout doesn’t work, we won’t see any output after the timeout loop starts.

POE::Session->create(
    inline_states => {
        _start => sub { $_[KERNEL]->yield('loop'); },
        loop   => sub {
            print "Normal Loop\n";
            $_[KERNEL]->delay(loop => 1);
        },
    }
);

POE::Session->create(
    inline_states => {
        _start => sub { $_[KERNEL]->yield('loop'); },
        loop   => sub {
            print "Timeout loop\n";
            timeout_test();
            $_[KERNEL]->delay(loop => 1);
        },
    }
);

$poe_kernel->run();

Fortunately, it works fine.

$ perl basic-poe.pl
Normal Loop
Timeout loop
Timeout
Normal Loop
Timeout loop
Timeout
Normal Loop
Terminating on signal SIGINT(2)

My Emacs Defaults

Now that I have customised my emacs extensively, the default configuration is quite uncomfortable for me to use. I have a file called my-defaults.el which is the bare minimum I need to make using emacs a pleasant experience. If I have to sit down at your emacs session, I will probably need to cut and paste these into a temp buffer and call M-x eval-region.

I’ve mentioned some of these modifications before.

I always always use ido and uniquify. Ido makes it so nice for finding files and switching buffers, I now find the default behaviour surprising and sometimes even catch myself waiting for the options to appear.

(require 'ido)
(require 'uniquify)

flex-matching is a given of course, and I don’t like being prompted unnecessarily for new buffers – I’m always creating them.

(ido-mode t)
(setq ido-enable-flex-matching t)

(setq ido-create-new-buffer 'always)

The way emacs deals with identically named files by default is poor, but it is great that it is so easy to fix.

(setq uniquify-buffer-name-style 'reverse)
(setq uniquify-separator "|")
(setq uniquify-after-kill-buffer-p t)
(setq uniquify-ignore-buffers-re "^\\*")

Since emacs23, fonts now look great. This is from my windows config. Other folks have written about beautifying emacs for other OSes. I summarised those posts here.

(set-default-font
 "-outline-Consolas-normal-r-normal-normal-14-97-96-96-c-*-iso8859-1")

It should be obvious what most of these do. The most important ones are setting yes-or-no-p to accept y or n rather than forcing me to type yes<RETURN> and removing the toolbar. Actually no, scratch that, these are all important. (Emacs23 has some of these set by default).

(global-font-lock-mode 1)
(setq inhibit-splash-screen t)
(setq font-lock-maximum-decoration 3)
(setq use-file-dialog nil)
(setq use-dialog-box nil)
(fset 'yes-or-no-p 'y-or-n-p)
(tool-bar-mode -1)
(show-paren-mode 1)
(transient-mark-mode t)
(setq case-fold-search t)
(blink-cursor-mode 0)

It took me ages to figure out how to prevent emacs converting a bunch of spaces into tabs. And of course, the scrollbar should always be on the right.

(custom-set-variables
 '(scroll-bar-mode 'right)
 '(indent-tabs-mode nil))

Where am I?

(line-number-mode 1)
(column-number-mode 1)

Keep emacs backup files in one place. This is from my windows config again.

(push '("." . "c:/home/jared/.emacs-backups") backup-directory-alist)

And make it so that when I copy a region, that gets sent to the OS clipboard.

(setq x-select-enable-clipboard t)
(setq interprogram-paste-function 'x-cut-buffer-or-selection-value)

Basic Emacs Muse Configuration

Looking back over my old posts, I found I didn’t cover basic muse setup at the time when I did my original muse series.

part 1 part 2 part 3 part 4

As I’ve mentioned previously, I generate the html (including the funky syntax highlighting) for my posts and squidoo lenses using muse with htmlize.

You can download muse here and if you are using emacs23 then you need the patched version of htmlize, available here.

---

The first thing I do with most third party modules, is I add the require lines and the paths I need.

(load "my-vars.el")

(defconst *elisp-muse* (concat *elisp-3rd* "/muse-3.12/lisp"))

(add-to-list 'load-path *elisp-muse*)

(require 'htmlize)

(require 'muse-mode)
(require 'muse-publish)
(require 'muse-html)

my-var.el contains constants pointing to the top-level directories (I still haven’t got around to putting everything under emacs.d)

(defconst *elisp-dir* (expand-file-name "~/emacs-files"))
(defconst *elisp-3rd* (concat *elisp-dir* "/third-party"))

(provide 'my-vars)

I use a couple of minor modes together with muse which are added to muse-mode-hook. (Note, I’m still using longlines-mode as at work I’m unable to use emacs23 where visual-line-mode was introduced)

(defun muse-minor-modes ()
  (longlines-mode 1)
  (flyspell-mode 1)
  (font-lock-mode 0))

(add-hook 'muse-mode-hook 'muse-minor-modes)

And I like publishing to be activated by a single key.

(defun my-muse-publish ()
  (interactive)
  (muse-publish-file (buffer-file-name) "html"))

(define-key muse-mode-map [f7] 'my-muse-publish)

Enabling Your Users Part 2

Storing Session History

A little while ago we were talking about writing a little emacs-based application to enable the users to help themselves. The beginning of this tool needs a light-weight dired using emacs buttons to use for navigating around the filesystem. Today we will look at adding functionality to remember which files and directories have been accessed previously.

First of all we need some variables to store the directories and files in.

(defvar file-editor-current-dir nil)

(defvar file-editor-save-dirs nil)
(defvar file-editor-save-dirs '(a b c))
(defvar file-editor-save-files nil)

Then we provide a customizable variable where the history will be saved between emacs sessions.

(defcustom file-editor-history-file "~/.file-editor-history"
  "File in which the file-editor history is saved between invocations.
Variables stored are: `file-editor-save-dirs', `file-editor-save-files'."
  :type 'string
  :group 'file-editor)

We will frequently be adding the same file and directory into the lists and we don’t want to get dupes. I could use a data structure that helps avoid dupes or I could just sort the lists and remove adjacent dupes. Guess which option I chose.

(defun remove-dupes (list)
  (let (tmp-list head)
    (while list
      (setq head (pop list))
      (unless (equal head (car list))
        (push head tmp-list)))
    (reverse tmp-list)))

(defun file-editor-sort-history ()
  (setq file-editor-save-dirs
        (remove-dupes (sort file-editor-save-dirs #'string<)))
  (setq file-editor-save-files
        (remove-dupes (sort file-editor-save-files #'string<))))

ido has code that stores history between sessions. I’ve stolen most of it to save the file editor history. (ido-pp ...) pretty prints the variable contents into the buffer, e.g. something like this.

;; ----- file-editor-save-dirs -----

( "dir1" "dir2" "dir3" )

(require 'ido)

(defun file-editor-save-history ()
  "Save file-editor history between sessions."
  (let ((buf (get-buffer-create " *file-editor data*"))
        (version-control 'never))
    (unwind-protect
        (with-current-buffer buf
          (erase-buffer)
          (file-editor-sort-history)
          (ido-pp 'file-editor-save-dirs)
          (ido-pp 'file-editor-save-files)
          (write-file file-editor-history-file nil))
      (kill-buffer buf))))

When it comes time to load the history back, (read (current-buffer)) loads it back into the variables. You can see the use of unwind-protect and condtion-case in the code below as I talked about in my emacs lisp error handling post.

(defun file-editor-load-history ()
  (let ((file (expand-file-name file-editor-history-file)) buf)
    (when (file-readable-p file)
      (let ((buf (get-buffer-create " *file-editor data*")))
        (unwind-protect
            (with-current-buffer buf
              (erase-buffer)
              (insert-file-contents file)
              (condition-case nil
                  (setq file-editor-save-dirs (read (current-buffer))
                        file-editor-save-files (read (current-buffer)))
                (error nil)))
          (kill-buffer buf))))))

The obvious time to save the history is when we exit emacs.

(defun file-editor-kill-emacs-hook ()
  (file-editor-save-history))

(add-hook 'kill-emacs-hook 'file-editor-kill-emacs-hook)

Modifications To The Original Code

The way we choose which files and directories will be remembered is each time a file is opened, the parent directory and the file including full path are added to the appropriate variable.

(defun file-editor-open-file-editor-file (button)
  (let ((parent (button-get button 'parent))
        (file (button-get button 'file))
        (file-complete (concat parent "/" file)))
    (push parent file-editor-save-dirs)        
    (push file-complete file-editor-save-files)
    (find-file file-complete)
    (file-editor-mode)
    (longlines-mode 1)))

We need to extend the file-editor-default-dirs function to display the previously stored directories and files.

(defun file-editor-default-dirs ()
  (let ((buffer (get-buffer-create "*file-editor-dir-list*")))
    (with-current-buffer buffer
      (let ((inhibit-read-only t))
        (erase-buffer)
        (file-editor-sort-history)

        (insert "*** Default File List ***\n\n")

        (dolist (dir file-editor-default-dirs)
          (file-editor-insert-opendir-button "" dir))

        (when file-editor-save-dirs
          (insert "\n")
          (dolist (dir file-editor-save-dirs)
            (file-editor-insert-opendir-button "" dir)))

        (when file-editor-save-files
          (insert "\n")
          (dolist (file file-editor-save-files)
            (insert-text-button file 'parent "" 'file file
                                :type 'open-file-editor)))))))

And for some future functionality I am thinking about we also store the current directory that is being visited.

(defun file-editor-dir-list (parent)
  (let ((buffer (get-buffer-create "*file-editor-dir-list*")))
    (with-current-buffer buffer
      (let ((inhibit-read-only t) files dirs)
        (setq parent (expand-file-name parent))
        (setq file-editor-current-dir parent)
        (erase-buffer)
        ;; ...
))))

Interesting Emacs Links – 2009 Week 25

Org Mode

The wonderful Org mode has deservedly been getting a lot of [word] press recently. This is a really great tutorial. There is a nice customization guide at orgmode.org. endperform talks about using it for time tracking and remembering useful tricks. Emacs-fu has an article on generating html with org-mode. ByteBaker talks about using it to organise papers he downloaded and to make a wiki.

My Emacs Posts

I’ve started a series about a light-weight alternative to dired mode. Part two, which will remember locations you have visited previously is on the way.

A quick mention of longlines-mode got a comment about visual-line-mode which is the replacement in Emacs 23 onwards. I’ve switched over and it does seem better. longlines-mode was fairly reliable, but occasionally it would forget that it was supposed to be wrapping words and I would need to disable it and enable it.

Other Emacs Posts

alieniloquent talks about using advice to disable other window is you use the universal prefix (C-u). Nice trick.

Aneesh Kumar has post on switching from vim to emacs, or actually viper. As I use vim a lot, I’ve tried viper in the past but I always found that it made accessing various emacs commands harder (or maybe just different) than it is in vanilla emacs so I always switched back.

Armin Sadeghi says that his two favourite editors are SlickEdit and Emacs.

The big difference between SlickEdit and Emacs is that SlickEdit is commercial software and Emacs is open source.

If that is the big difference, why not just use Emacs?

The Real World versus BlogWorld

In the real world

• we don’t get to choose Ruby just because we want to
• we don’t have time to implement perfect solutions
• we don’t go back and fix ugly but working code
• the second law of thermodynamics always holds
• copy/paste and singleton are not the root of all evil
• Java/C++/Perl are not dying
• we still need to write code that works in IE
• for most programmers IDEs are better than emacs or vim
• two journeyman programmers are better than one rockstar

And in blogworld bloggers often forget that the plural of anecdote is not data.