Working AppEngine environment on Ubuntu Lucid

Ubuntu Lucid ships Python 2.6. Python 2.5 has been completely removed. Google AppEngine requires Python 2.5 to work. So if you want to develop AppEngine apps on a Lucid machine, you need to setup your working environment manually. This post tells how to do it.

(I assume you install Python to /opt/python2.5 and the user can write there; I assume you create a virtual environment, a directory where Python packages are installed, in $HOME/.py25. Choose different locations if you like and adjust instructions accordingly)

1. Get the latest Python 2.5 release and build it from source.

Make sure that you have necessary development libraries installed. In particular, you probably want to install libsqlite3-dev before building Python. Otherwise you'll have a Python without SQLite3 support, and GAE will not work with it.

Go to a directory where you build software and do from the command line:

wget  http://www.python.org/ftp/python/2.5.5/Python-2.5.5.tar.bz2 -O - | tar jx
cd Python-2.5.5
./configure --prefix=/opt/python2.5
make -j 2
make install
cd ..

If there are configure errors, likely missing packages or header files for C libraries, install them and repeat.

2. Get virtualenv and setup a new Python environment

You will use a separate Python environment for AppEngine. So you will not mess with system packages. Fetch and unpack virtualenv:

wget http://bitbucket.org/ianb/virtualenv/get/tip.gz -O - | tar zx

Run it to get a new environment in ~/.py25 (I use full path to the newly installed Python 2.5 here):

/opt/python2.5/bin/python virtualenv/virtualenv.py ~/.py25

Now to enable Python 2.5 you can source activate script from this environment, and to disable deactivate it:

$ which python ; python --version
/usr/bin/python
Python 2.6.5
$ source ~/.py25/bin/activate
(.py25)$ which python ; python --version
/home/sergey/.py25/bin/python
Python 2.5.5
(.py25)$ deactivate

To install packages for Python 2.5 you can use pip. For example, to install Python Imaging Library (PIL), which is used by AppEngine, run:

(.py25)$ pip install PIL

Or you can use -E ~/.py25 option of pip without activating the environment.

3. Run the development server

Put your GAE SDK in the PATH, get the source of the application, and run the server. For example:

(.py25)$ git clone http://github.com/anotherjesse/webpy-appengine-helloworld.git gae-hello
...
(.py25)$ cd gae-hello
...
(.py25)$ dev_appserver.py .
INFO     2010-06-28 16:35:04,755 appengine_rpc.py:159] Server: appengine.google.com
INFO     2010-06-28 16:35:04,761 appcfg.py:357] Checking for updates to the SDK.
INFO     2010-06-28 16:35:05,068 appcfg.py:371] The SDK is up to date.
INFO     2010-06-28 16:35:05,106 dev_appserver_main.py:431] Running application hello-webpy on port 8080: http://localhost:8080

Network Management disabled

Sometimes (I suspect it happens after complete battery discharge and incorrect shutdown), my Ubuntu Lucid machine boots with networking disabled. This is a nasty bug, because it doesn't allow to google for solution.

The solution is

sudo sed -i 's/\(NetworkingEnabled=\).*/\1true/' /var/lib/NetworkManager/NetworkManager.state

and then

sudo service network-manager restart

https://bugs.launchpad.net/ubuntu/+bug/555571

OpenMP in the Multicore Era

Nice slides by Christian Terboven on “OpenMP in the Multicore Era”. Enough to get started.

http://www.autodiff.org/ad08/talks/ad08_terboven.pdf

By the way, to build an OpenMP program with modern GCC it’s enough to

gcc -fopenmp -o eval42 eval42.c

It should work out of the box. GCC 4.1 supports OpenMP 2.5, and GCC 4.4 supports OpenMP 3.0.

Trying to do my daily tasks with Emacs

Lots of frustration, but I slowly remember some habits from 10 years ago. The thing I miss the most is ability to move quickly around and to delete/change an object of text, such as a word, a sentence, a paragraph, part of the line, a tag, etc. All those dsomething from vim.

Some command correspondence (beyond the crash course available elsewhere): (open table in a separate page)

It seems it's time to learn Emacs Lisp. I've bound toggle-viper-mode to C-<escape>. It's my preferred mode to move the cursor and delete stuff.

Tag installed packages to delete later

One of the lesser known but very useful features of aptitude is user tags.

For example, someone wants to install a set of packages temporarily just to build some program from source. Later, when the -dev packages are not needed anymore, it is not easy to remember which packages were installed.

Fortunately, one may tag the packages during install (I use builddeps tag in the example below):

$ sudo aptitude install --add-user-tag builddeps \
      libsomething-dev libsomething-else-dev ...

Now when these packages are to be deleted, we just do:

$ sudo aptitude purge '?user-tag(builddeps)'

The search pattern ?user-tag(yourtag) can be used with any other aptitude search patterns. Also, you may tag packages during many other commands, not only install.

Tested in Debian Lenny (aptitude-0.4.11).

Прочесть по-русски.

3 ways to re-indent XML

There is a lot of data in XML formats, but often it's hardly readable: written by programs for programs, everything in one line. Indenting XML automatically helps to read such files.

1. Using XSLT

I have a file with an XSL transformation:

<xsl:stylesheet version="1.0" 
   xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml"/>
<xsl:param name="indent-increment" select="'   '" />

<xsl:template match="*">
   <xsl:param name="indent" select="'&#xA;'"/>

   <xsl:value-of select="$indent"/>
   <xsl:copy>
     <xsl:copy-of select="@*" />
     <xsl:apply-templates>
       <xsl:with-param name="indent"
            select="concat($indent, $indent-increment)"/>
     </xsl:apply-templates>
     <xsl:value-of select="$indent"/>
   </xsl:copy>
</xsl:template>

<xsl:template match="comment()|processing-instruction()">
   <xsl:copy />
</xsl:template>

<!-- WARNING: this is dangerous. Handle with care -->
<xsl:template match="text()[normalize-space(.)='']"/>

</xsl:stylesheet>

I found it here. There are also some other variants.

In addition to XSLT file, I have a one-line script which actually runs this transformation. I use xmlstarlet, which is a nice CLI utility to deal with XML.

#!/bin/sh
xmlstarlet tr ~/bin/indent-xml.xsl

Run this script as:

$ xmlindent < original.xml

Along with xmlstarlet you can use other XSL processors. For example, xsltproc should work too.

2. Using xmllint

Inside libxml2-utils package (Debian/Ubuntu) there is an XML validator tool xmllint. It can also reformat (indent) XML:

$ xmllint --format original.xml

This must be even easier.

3. xmlindent

xmlindent is a pure C utility with almost no dependencies. It is intended to do just what it is named: indent XML. I didn't try it.

Прочесть по-русски

Monitor file changes in a shell script

Problem: monitor file changes from a shell script and execute some commands when necessary. For example, rebuild LaTeX document or compile program every time when one of its source files is changed.

Solution: inotify-tools help to monitor file changes. There are two utilities. The first one, inotifywait, blocks and waits for changes, then returns. If the event it was waiting for happened, its return code is 0 (success). See an example of using inotifywait below. The second utility is inotifywatch, it monitors files' changes, collects information and prints a nice table on exit. Please visit inotify-tools' site to see examples of its use.

Example: inotifywait monitors all *.tex and *.bib files in the current directory, and when any of them changes, it runs pdfLaTeX and BibTeX to rebuild document:

while true ; do \
  inotifywait *.tex *.bib \
  && ( pdflatex -interaction=nonstopmode mypaper && \
       bibtex mypaper && \
       pdflatex -interaction=nonstopmode mypaper ) \
done

P.S. Please note that when we run LaTeX with -interaction=nonstopmode, it does not ask questions on errors but we can still see those errors.

P.P.S. inotify-tools run only on Linux. You may need to use pnotify or kqueue on *BSD.

Прочесть по-русски