Working AppEngine environment on Ubuntu Lucid

Ubuntu Lucid ships Python 2.6. Python 2.5 has been completely removed. Google AppEngine requires Python 2.5 to work. So if you want to develop AppEngine apps on a Lucid machine, you need to setup your working environment manually. This post tells how to do it.

(I assume you install Python to /opt/python2.5 and the user can write there; I assume you create a virtual environment, a directory where Python packages are installed, in $HOME/.py25. Choose different locations if you like and adjust instructions accordingly)

1. Get the latest Python 2.5 release and build it from source.

Make sure that you have necessary development libraries installed. In particular, you probably want to install libsqlite3-dev before building Python. Otherwise you'll have a Python without SQLite3 support, and GAE will not work with it.

Go to a directory where you build software and do from the command line:

wget  http://www.python.org/ftp/python/2.5.5/Python-2.5.5.tar.bz2 -O - | tar jx
cd Python-2.5.5
./configure --prefix=/opt/python2.5
make -j 2
make install
cd ..

If there are configure errors, likely missing packages or header files for C libraries, install them and repeat.

2. Get virtualenv and setup a new Python environment

You will use a separate Python environment for AppEngine. So you will not mess with system packages. Fetch and unpack virtualenv:

wget http://bitbucket.org/ianb/virtualenv/get/tip.gz -O - | tar zx

Run it to get a new environment in ~/.py25 (I use full path to the newly installed Python 2.5 here):

/opt/python2.5/bin/python virtualenv/virtualenv.py ~/.py25

Now to enable Python 2.5 you can source activate script from this environment, and to disable deactivate it:

$ which python ; python --version
/usr/bin/python
Python 2.6.5
$ source ~/.py25/bin/activate
(.py25)$ which python ; python --version
/home/sergey/.py25/bin/python
Python 2.5.5
(.py25)$ deactivate

To install packages for Python 2.5 you can use pip. For example, to install Python Imaging Library (PIL), which is used by AppEngine, run:

(.py25)$ pip install PIL

Or you can use -E ~/.py25 option of pip without activating the environment.

3. Run the development server

Put your GAE SDK in the PATH, get the source of the application, and run the server. For example:

(.py25)$ git clone http://github.com/anotherjesse/webpy-appengine-helloworld.git gae-hello
...
(.py25)$ cd gae-hello
...
(.py25)$ dev_appserver.py .
INFO     2010-06-28 16:35:04,755 appengine_rpc.py:159] Server: appengine.google.com
INFO     2010-06-28 16:35:04,761 appcfg.py:357] Checking for updates to the SDK.
INFO     2010-06-28 16:35:05,068 appcfg.py:371] The SDK is up to date.
INFO     2010-06-28 16:35:05,106 dev_appserver_main.py:431] Running application hello-webpy on port 8080: http://localhost:8080

Random Python snippets

Lists and dictionaries

Given a flat list, like [key1, value1, key2, value2] convert it to an alist or dictionary:

>>> toalist = lambda kvs: zip(kvs[0::2], kvs[1::2])
>>> toalist(range(4))
[(0, 1), (2, 3)]
>>> dict(toalist(range(4)))
{0: 1, 2: 3}

Convert a dictionary to a flat list:

>>> # dict to alist
... al = list({1:2,3:4}.iteritems())
>>> al
[(1, 2), (3, 4)]
>>> # alist to flat list
... reduce(lambda acc,t: acc + list(t), al, [])
[1, 2, 3, 4]

To tranpose a list of lists/tuples unpack as a list of function arguments and zip zip(*mylist):

>>> l = list(enumerate("abcdef"))
>>> l
[(0, 'a'), (1, 'b'), (2, 'c'), (3, 'd'), (4, 'e'), (5, 'f')]
>>> # transpose list of lists/tuples
... zip(*l)
[(0, 1, 2, 3, 4, 5), ('a', 'b', 'c', 'd', 'e', 'f')]
>>> # once again
... zip(*_)
[(0, 'a'), (1, 'b'), (2, 'c'), (3, 'd'), (4, 'e'), (5, 'f')]

Flatten a list of lists:

>>> lofl = [[1,2], [3], [4,5]]
>>> import operator
>>> reduce(operator.add, lofl)
[1, 2, 3, 4, 5]

An alternative approach is to use chain from itertools (this works also on huge lists if used wisely!):

>>> list(itertools.chain(*lofl))
[1, 2, 3, 4]

Apply a function to either an iterable (list, tuple) or a scalar:

>> def fmap(f,xs):
...   try: return map(f,xs)
...   except TypeError: return f(xs)
... 
>>> fmap(lambda x:x*x, range(5))
[0, 1, 4, 9, 16]
>>> fmap(lambda x:x*x, 5)
25

Strings and Unicode

Unicode stuff is changing in 3.0. For earlier versions, it is important to distinguish strings ("abc") and unicode strings (u"abc"). The former can be converted to the latter with unicode():

>>> "абв"
'\xd0\xb0\xd0\xb1\xd0\xb2'
>>> u"абв"
u'\u0430\u0431\u0432'
>>> unicode("абв","utf8")
u'\u0430\u0431\u0432'

Please note there are 3 unicode symbols in the original literal and there are three values in the unicode string. This is how the strings are to be represented internally. Any communication with an external world usually requires that unicode data is encoded. There are various encodings, "UTF-8" is one of the most common. Any encoded input should be decoded to be processed:

>>> "абв".decode("utf8")
u'\u0430\u0431\u0432'
>>> u"абв".encode("utf8")
'\xd0\xb0\xd0\xb1\xd0\xb2'

To live a long and happy life it is important to understand if you are working with an encoded data (practically binary data) or decoded unicode text. To test if an object is a string (either ascii string or unicode), test if it is an instance of basestring:

>>> isinstance("abc",basestring)
True
>>> isinstance(u"abc",basestring)
True
>>> isinstance(42,basestring)
False

To convert to a string and from string (depends on type):

>>> str(42)
'42'
>>> unicode(42)
u'42'
>>> int("42")
42
>>> float("42")
42.0

Backporting to Python 2.4

With Python 2.5, 2.6 and even 3.0 around, I still need to make some scripts run with Python 2.4. Just two tricks, to make sqlite3 code work:

try:
   import sqlite3
except:
   from pysqlite2 import dbapi2 as sqlite3 # cheating with py2.4

and to make ElementTree work:

try:
        import xml.etree.ElementTree as ET
except:
        import cElementTree as ET  # not xml.etree in py2.4, use celementtree

epi2fox: import Epiphany bookmarks into Firefox 3

I used Epiphany as my main browser for a long time because I find its bookmarks system much better than anything else. However, as new Firefox 3 permits tagged bookmarks too, I decided to give it a try once again. But I wanted all my bookmarks from Epiphany available in Firefox too. With the same tags.

I didn't find any ready solution, so I wrote a script, epi2fox.py. Assuming, you have an almost empty Firefox profile, run this script like this:

$ epi2fox.py ~/.mozilla/firefox/yourprofile/places.sqlite

The script is not perfect, but it did the job. One of its major shortcomings: while Epiphany permits multiple bookmarks for the same URL, Firefox does not. Probably, such bookmarks should be merged on importing, but the script just throws away duplicates (and prints error messages).

Links:

• epi2fox, script repository
• Firefox bookmarks DB schema (PDF)
• epi2fox: конвертация закладок Epiphany в Firefox 3 (this announcement in Russian)

PS. Please backup your places.sqlite before running the script.

rss2xmpp, a script to crosspost any feed to Jabber

Usage:

$ rss2xmpp.py feed-URL your-jabber-id

On the first run the script will complain that you have to put jabber settings in ~/.rss2xmpp. It writes an example for GoogleTalk there. Either RSS or Atom feeds should work.

Requirements: FeedParser, html2text, and xmpppy, and Python, of course.

The script itself is in the BitBucket: rss2xmpp.py.

BTW, I discovered, that BitBucket not only provides free OpenSource hosting for mercurial repositories, but has free SSH access and allows one private repository per account.

See also:

☙ Скрипт rss2xmpp, кросспост чего угодно в Jabber (this post in Russian)

Python inside LaTeX (and Sage too)

I discovered recently, that it is possible to run Python scripts from LaTeX documents and use the to generate document's content. This can be used to read/convert data, generate tables and figures, do on-the-fly calculations.

How to run Python code from LaTeX:

• download python.sty. Put it in the same directory as the LaTeX document
• \usepackage{python}
• Put Python code inside \begin{python} — \end{python} environment. Whatever this code prints, will become part of the LaTeX document.
• Run LaTeX with -shell-escape option (this permits running external code from LaTeX)

This is an example of the LaTeX document and the PDF produced.

And some more advanced examples are in this LaTeX file (see also the PDF). They cover symbolic calculations from inside LaTeX, plotting and variable persistence between python environments.

P.S. What's more, one can even embed the full-featured symbolic math package Sage into LaTeX. For this purpose use sagetex package. By the way, you do not need to install Sage to start using it, please check out its web-version. It is really powerful.

Links:

• Embedding Python in LaTeX
• Some examples of Python scripts inside LaTeX: plotting, symbolic calculations, variable persistence
• Sage, a Matematica alternative
• Sage Online (use online without installation, like Google Docs)

This post in Russian: Python внутри LaTeX (и математический пакет Sage тоже)