tag:blogger.com,1999:blog-30915940527087537602024-03-13T13:22:36.271+01:00*nix tipsUnknownnoreply@blogger.comBlogger34125tag:blogger.com,1999:blog-3091594052708753760.post-29108437543573187022011-06-23T12:44:00.008+02:002011-06-27T13:32:28.162+02:00Steepness of a* and b* channels in Darktable<p>As many aspiring photogrpahers, I want to adjust color curves in Lab color space. Changing steepness of a* and b* curves is a popular method to boost color contrast in the image (see Dan Margulis books or <a href="http://www.dgrin.com/showthread.php?t=18203" title="Margulis LAB Color Book - Reading group">this summary</a>). It works best for abstract and some of the landscape photography.
<p>There are few open source solutions which allow to do it. Gimp can decompose an image into three Lab layers, but it doesn't have an interactive preview, and internally Gimp represents color with only 8 bits per channel. There are also <a href="http://code.google.com/p/delaboratory/">Delaboratory</a> and <a href="http://www.mm-log.com/lab-curves-gimp">Lab curves for Gimp</a>, but they don't fit my workflow very well.
<p>These days I do almost all my photo processing in <a href="http://darktable.sourceforge.net/">Darktable</a>, and wanted to be able to adjust color contrast there without using another program (Delaboratory) everytime. I knew that Darktable uses floating point representation for color channels and works in Lab already. Then I noticed, that basic Margulis' recipes are about “straight” symmetric curves anyway. So if there are only two degrees of freedom, then even a simple plugin with two sliders can be very useful. And so wrote my first plugin for Darktable, and named it “Color contrast”.
<p>This plugin allows to control the effect with only two sliders. They have user-friendly labels “green vs magenta” and “blue vs yellow”. So this plugin can be used even by users who do not know anything about Lab. Technically, two values which the sliders control are the tangents of straight symmetric curves for the transforms in a* and b* channels respectively.
<p>An original image in Darktable:
<p><img src="http://img-fotki.yandex.ru/get/4704/s-astanin.22/0_5eec5_1d348bde_XL.jpg" alt="Darktable: before Color Contrast" >
<p>And after applying Color Contrast plugin:
<p><img src="http://img-fotki.yandex.ru/get/5806/s-astanin.22/0_5eec6_4c3bbf15_XL.jpg" alt="Darktable: after Color Contrast" >
<p>Update: The plugin has been pushed to <em>master</em> branch of the Darktable git. See <a href="http://darktable.sourceforge.net/install.shtml#git">how to install it</a>.
<strike><p>The plugin should appear in <em>post-0.9</em> branch of Darktable after the release of 0.9; and I hope it gets accepted for the main line. In the meanwhile you may apply these two patches over <em>master</em>:
<p><a href="http://sourceforge.net/mailarchive/attachment.php?list_name=darktable-devel&message_id=4DFBCCBF.9000808%40gmail.com&counter=1">0001-colorcontrast-change-steepness-of-a-and-b-curves.patch </a>
<p><a href="http://sourceforge.net/mailarchive/attachment.php?list_name=darktable-devel&message_id=BANLkTi%3DnY_j%2B_-x6Eizd9qe7jA0T%2BjA5LQ%40mail.gmail.com&counter=1">0002-colorcontrast-clip-a-and-b-channels-reload_defaults.patch</a>
<p>Or you may get the patches from the (rebasing) branch <em>custom</em> of my git repository: <a href="https://github.com/sergeyastanin/darktable-custom/">darktable-custom</a>.</strike>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3091594052708753760.post-56872197430565855242011-06-14T13:07:00.016+02:002011-08-23T08:30:22.337+02:00Jumping to correct tags in Emacs (and in Vim)<p>I use Emacs to edit Python code, and navigate the code using ETags and <code>find-tag</code> (<kbd>M</kbd>-<kbd>.</kbd>) function. However, I noticed, that if some modules use imports like <code>from x import foo</code>, then sometimes <code>(find-tag "tagname")</code> jumps to some random file which <em>imports</em> a <code>tagname</code>, rather than to a file where <code>tagname</code> is defined.
<p>I was very frustrated. Initially I thought it was a bug in <code>etags</code>. It appears that it is a feature. Fortunately, there is a solution which allows to find the symbol definition (scroll down to the end of the post if impatient).
<h4>Example</h4>
<p>This is an example which reproduces the problem. Let's suppose I have three files. <code>A.py</code>:</p>
<pre>from x import foo
from y import bar
if __name__ == "__main__":
foo()
bar()</pre>
<p><code>x.py</code>:</p>
<pre>def foo():
print "foo"</pre>
<p>and <code>y.py</code>:</p>
<pre>def bar():
print "bar"</pre>
<p>Then if I generate a <code>TAGS</code> file as
<pre>$ etags *.py</pre>
this is what it will look like:
<pre>^L
A.py,53
from x import foo^?foo^A1,0
from y import bar^?bar^A2,18
^L
x.py,19
def foo():^?foo^A1,0
^L
y.py,19
def bar():^?bar^A1,0</pre>
<p>Now <code>(find-tag "foo")</code> (<kbd>M</kbd>-<kbd>.</kbd> upon <code>foo</code>) jumps to <code>A.py</code>, rather than to <code>x.py</code>. The problem manifests when the filename of the module using a symbol precedes alphabetically the filename of the module where the symbol is defined, but in the first place the problem is that import lines are indexed as symbol definitions.
<h4>Solution</h4>
<p>Initially I thought it is a bug in <code>etags</code>, but now I think that it is a designed behaviour, though not very user-friendly.
<p>If we see help for <code>find-tag</code> function, we may notice, that it has an optional <code>next-p</code> parameter:
<pre>(find-tag tagname &optional next-p regexp-p)
Find tag (in current tags table) whose name contains tagname.
Select the buffer containing the tag's definition, and move point there.
The default for tagname is the expression in the buffer around or before point.
If second arg next-p is t (interactively, with prefix arg), search for
another tag that matches the last tagname or regexp used. When there are
multiple matches for a tag, more exact matches are found first. If next-p
is the atom `-' (interactively, with prefix arg that is a negative number
or just M--), pop back to the previous tag gone to.</pre>
<p>So, it is possible to cycle through all the positions where the tag was found by pressing <kbd>M</kbd>-<kbd>1</kbd> <kbd>M</kbd>-<kbd>.</kbd> or <kbd>Ctrl</kbd>-<kbd>u</kbd> <kbd>M</kbd>-<kbd>.</kbd>. To cycle back, <kbd>M</kbd>-<kbd>-</kbd> <kbd>M</kbd>-<kbd>1</kbd> <kbd>M</kbd>-<kbd>.</kbd>.
<p>I don't think it is the best solution usability-wise. While there is some value in having import positions indexed too, I'd prefer to jump always to the definition first, rather than to a random import statement. Also, the default keyboard bindings are not very ergonomic, but at least they allow to find the definition and don't require to install anything special.
<p>P.S. Vim and <code>ctags</code> are affected too. To jump to the second entry of the tag, use <kbd>2</kbd> <kbd>Ctrl</kbd>-<kbd>]</kbd> rather than <kbd>Ctrl</kbd>-<kbd>]</kbd>.
<p><a class="FlattrButton" style="display:none;" href="http://nix-tips.blogspot.com/2011/06/jumping-to-correct-python-tags-in-emacs.html"></a>
<noscript><a href="http://flattr.com/thing/310375/Jumping-to-correct-Python-tags-in-Emacs-and-in-Vim" target="_blank">
<img src="http://api.flattr.com/button/flattr-badge-large.png" alt="Flattr this" title="Flattr this" border="0" /></a></noscript>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3091594052708753760.post-36864282605270320222011-03-09T13:59:00.012+01:002011-04-27T15:50:11.141+02:00How to choose a Haskell array library<p>Choosing an array type in Haskell is a difficult task. For one-dimensional random access data structure the vector library seems to be the optimal choice most of the time. Things are more complicated if you happen to need two- or multi-dimensional arrays (matrices), access their blocks and slices as first-class structures (like in Python), enjoy destructive updates, use some linear algebra, interoperate with C and run your code in parallel...
<p>I've reviewed what array libraries are available on Hackage, and compiled <a href="https://spreadsheets.google.com/pub?hl=en&hl=en&key=0AsGYigQZXDPKdEl1YUNpREQ2aDVkVkhsaUVidVA2ZVE&single=true&gid=0&range=A1:G34&output=html">this feature matrix</a>.
<p><iframe width='700' height='400' frameborder='0' src='https://spreadsheets.google.com/pub?hl=en&hl=en&key=0AsGYigQZXDPKdEl1YUNpREQ2aDVkVkhsaUVidVA2ZVE&single=true&gid=0&range=A1%3AG34&output=html&widget=true'></iframe>
<p>It is not complete and not finished, please let me know if I am mistaken about some of the libraries.
<p><code>Data.Array</code> and its variants from the <a href="http://hackage.haskell.org/package/array">array</a> library seem to be the standard choice of multi-dimensional arrays for a Haskeller. They are not good anymore when you need to write array-to-array operations, access their blocks and slices, or need some linear algebra in general. Choosing the right variant is another question.
<p><code>Data.Vector</code> from the <a href="http://hackage.haskell.org/package/vector">vector</a> library is fast and has very nice API. It is going to become part of the Haskell Platform. Unfortunately, it is not usable for multi-dimensional arrays, and it doesn't support slices (strides). Only boxed variant is parallelizable.
<p><code>Data.Packed.Vector</code> and <code>Data.Packed.Matrix</code> from the <a href="http://hackage.haskell.org/package/hmatrix">hmatrix</a> library provide a very nice API, and can do almost anything, if all you need is at most two-dimensional array (a matrix). A big warning sign: hmatrix is GPL. Not a sensible LGPL, but the poisonous GPL library. Also, <code>Data.Packed.Vector</code> and <code>Data.Packed.Matrix</code> are not parallelizable as far as I can see.
<p><code>Data.Vector.*</code>, <code>Data.Matrix.*</code> and <code>Data.Tensor.*</code> from the <a href="http://hackage.haskell.org/package/blas">blas</a> library. They can do all the standard linear algebra which the BLAS level 3 can offer. Their API is designed with BLAS API in mind. I didn't check if they are interoperable with C arrays, or if the elements are unboxed, or if the operations are parallelizable. The last release was in January 2009, it is not buildable on new GHC 7 yet.
<p>Finally, there is new <code>Data.Array.Repa</code> arrays from the <a href="http://hackage.haskell.org/package/repa">repa</a> library. Nice thing about them: they are designed for parallelization. Not so nice thing about them: they don't give performance advantages on GHC 6<strike>, and are not yet buildable on GHC 7</strike>. Some more important limitations of the repa: <strike>no access to strides or array blocks,</strike> no built-in linear algebra, no interoperability with C.
<p>I didn't include in the table <a href="http://hackage.haskell.org/package/Vec">Vec</a> and <a href="http://hackage.haskell.org/package/vect">vect</a> libraries, which provide special case solutions in the low-dimension arrays and low-rank matrices. They are mostly tailored towards computer graphics.
<p><a class="FlattrButton" style="display:none;"
href="http://nix-tips.blogspot.com/2011/03/how-to-choose-haskell-array-library.html"></a>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3091594052708753760.post-72241092321652798812011-01-24T09:23:00.004+01:002011-01-24T09:35:52.845+01:00Pro Git book in ePub<p>I've followed <a href="http://johnmacfarlane.net/pandoc/epub.html">Pandoc instructions</a>, and built a e-book version of <a href="http://progit.org/book/">Pro Git</a> book by Scott Chacon. If you do not want to repeat these steps yourself, you can download the result:
<p><a href="http://ubuntuone.com/p/Zm0/" title="Scott Chacon. Pro Git. Unofficial EPUB version">Scott Chacon. Pro Git. [EPUB]</a>
<p><a href="http://dl.dropbox.com/u/14110648/progit.epub" title="Scott Chacon. Pro Git. Unofficial EPUB version">Scott Chacon. Pro Git. [EPUB]</a> (alternative link)
<p>P.S. The author <a href="http://progit.org/support.html">kindly asks</a> to purchase a paper version of the book to encourage more authors and publishers to use Creative Commons licenses.Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3091594052708753760.post-55192529449040330172010-11-25T23:30:00.008+01:002010-12-03T16:24:45.188+01:00Unplot.py: from plots to tabular data<p>Recently, when doing backups, I noticed a script on my hard drive, which I think may be useful to someone else. It takes an image with a line plot and generates a data file for that plot. So it does an operation inverse to plotting, i.e. <em>un</em>plotting.
<p>Download: <a href="http://bitbucket.org/jetxee/unplot.py/raw/tip/unplot.py" title="Utility to extract tabular data from line plots.">unplot.py</a>.
<p>The script is available on <a href="http://bitbucket.org/jetxee/unplot.py/src/tip/unplot.py">bitbucket</a>. Feel free to improve.
<p>To run the script, you first need to decide which part of the plot image you want to scan, and what values the pixels correspond to. I prefer to use Gimp or Geeqie to find pixel coordinates.
<p>If there are many lines on the same plot, You may also decide to colourize the line you are interested in with some distinct colour. Use Gimp if necessary. Write down the colour's HTML code.
<p>This plot is a good point to start:
<p><a href="http://img-fotki.yandex.ru/get/5500/s-astanin.18/0_49325_311ae7dd_orig.jpg" target="_blank"><img src="http://img-fotki.yandex.ru/get/5500/s-astanin.18/0_49325_311ae7dd_L.jpg" width="500" height="265" title="colourized plot" alt="colourized plot" class="whitebg"/></a>
<p>Then you can run the script against it. In the directory with the script, run:
<pre class="sh_sh">./unplot.py "#00ff00" 0 151 0 475 5.0 824 0.09 85 /path/to/plot.png > /path/to/data.txt</pre>
<p>The first parameter is the HTML colour code of the line to select. Then there is the values and the pixel coordinate of the bottom left corner of the plot: X value, X pixel coordinate, Y value, Y pixel coordinate. Then the same for the top right angle. And finally the name of the file with the plot. The output is redirected to the text file. Please note that the origin of digital images is usually the top left corner.
<p>After running the script try plotting the data once again to make sure you selected the right colour and region. I went too far to the right in this example and the green letters were mistaken for the part of the plot.
<p><a href="http://img-fotki.yandex.ru/get/4400/s-astanin.18/0_49326_814f290c_orig.jpg" target="_blank"><img src="http://img-fotki.yandex.ru/get/4400/s-astanin.18/0_49326_814f290c_L.jpg" width="500" height="375" title="" alt="" class="whitebg"/></a>
<p><a class="FlattrButton" style="display:none;"
href="http://nix-tips.blogspot.com/2010/11/unplot-data.html"></a>
<p><em>In Russian: <a href="http://sovety.blogspot.com/2010/11/unplotpy-from-plots-to-tabular-data.html">unplot.py: извлекаем табличные данные из графиков</a>.</em>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3091594052708753760.post-20458791566856661532010-11-11T18:00:00.004+01:002011-03-17T11:57:25.610+01:00How to select a region by intensity range in Gimp?<p>I wrote this as an answer to <a href="http://photo.stackexchange.com/questions/4839/gimp-select-by-histogram-range">Photo.StackExchange question</a>, how to select by histogram range in Gimp.</p>
<h3>Step-by-step</h3>
<ol>
<li>Make a copy of the layer (Layer → Duplicate Layer)</li>
<li>Select the duplicate layer, apply threshold (Colors → Threshold) to select the range of intensities.</li>
<li>In Layer → Mask → Add Layer Mask (or right click in the list of layers). Select “Grayscale copy of layer” and “Invert mask”.</li>
<li>idem: Mask to selection.</li>
<li>Hide or remove the layer with mask.</li>
</ol>
<h3>An example:</h3>
<p>An original image. I want to select the circle:</p>
<p><img src="http://i.imgur.com/FkyGI.png" alt="Original image"></p>
<p>Make a copy of the layer:</p>
<p><img src="http://i.imgur.com/2Ak9z.png" alt="Duplicate layer"></p>
<p>Apply threshold. Note that the area to be selected is black:</p>
<p><img src="http://i.imgur.com/7Lxy8.png" alt="Threshold"></p>
<p>Add Layer Mask using the grayscale value of the image:</p>
<p><img src="http://i.imgur.com/WSTdJ.png" alt="Add Layer Mask"></p>
<p>Now you've obtained an image with the mask. Everything except the black circle is transparent (we can see the bottom layer through it):</p>
<p><img src="http://i.imgur.com/oO6VS.png" alt="Image with the mask"></p>
<p>Convert the mask to selection. Switch to the original layer. The circle is selected.</p>
<p><img src="http://i.imgur.com/BogLR.png" alt="Mask to selection"></p>
<p><a class="FlattrButton" style="display:none;"
href="http://nix-tips.blogspot.com/2010/11/how-to-select-region-by-intensity-range.html"></a>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3091594052708753760.post-41289066721142632522010-09-14T17:21:00.020+02:002011-11-14T16:22:32.830+01:00JavaScript highlighter for Haskell code<p>In this and in my <a href="http://sovety.blogspot.com">other Blogspot blog</a> I use <a href="http://shjs.sourceforge.net/">SHJS</a> to highlight syntax in Haskell snippets. SHJS is a JavaScript highlighter which uses language definitions from GNU Source-highlight.
<p>To use it, you need to put online:
<ul><li>SHJS main script: <a href="http://shjs.sourceforge.net/sh_main.min.js">sh_main.min.js</a>,
<li>One or more language definitions files. Haskell one: <a href="http://bitbucket.org/jetxee/src-highlight-haskell/raw/tip/sh_haskell.min.js">sh_haskell.min.js</a>,
<li>One of the <a href="http://shjs.sourceforge.net/css/">stylesheets</a>. (You can preview them on <a href="http://shjs.sourceforge.net/">the main SHJS page</a>)
</ul>
<p>You may wish to download more than one language definition and concatenate all JavaScript files together. It's OK</p>
<p>Then put this somewhere in the <code><head></code>:
<pre class="sh_html"><head>
...
<script type="text/javascript" src="http://example.com/path/to/sh_main.min.js"></script>
<script type="text/javascript" src="http://example.com/path/to/sh_haskell.min.js"></script>
<link type="text/css" rel="stylesheet" href="http://example.com/path/to/stylesheet.css">
...
</head></pre>
<p>And add a callback to documents' <code>body</code> element:
<pre class="sh_html"><body onload="sh_highlightDocument();"></pre>
<p>Finally, wrap your Haskell snippets with <code><pre class="sh_haskell"></pre></code>. This is how it looks:
<pre class="sh_haskell">-- Fibonacci numbers
fibs :: [Int]
fibs = 0 : 1 : zipWith (+) fibs (tail fibs)</pre>
<p>As far as I know, Haskell language definition has already found its way to GNU Source-highlight, but the corresponding file is not yet shipped with SHJS. So you can download it either from my <a href="http://bitbucket.org/jetxee/src-highlight-haskell/src">source repository</a>, or generate from the language definition file (<code>haskell.lang</code>) yourself.
<p>Haskell is a quirky language to highlight, but this language definition handles it <a href="http://arboreus.com/sovety/haskell-shjs-test/">reasonably well</a>, better than the current <a href="http://img-fotki.yandex.ru/get/4505/s-astanin.13/0_3e358_c9ec3cd3_XL.jpg" title="google-code-prettify Haskell example">google-code-prettify</a>, and much better than the current <a href="http://img-fotki.yandex.ru/get/4805/s-astanin.13/0_3e248_eb05d782_orig.jpg" title="GeSHi Haskell example">GeSHi highlighter</a>.
<p><i>Update, 2011-01-25:</i> Suprisingly, this highlighter is still better than more recent <a href="http://img-fotki.yandex.ru/get/5801/s-astanin.1d/0_4fff9_86c592f9_orig.jpg">highlight.js</a> and an alternative language definition for SHJS by <a href="http://img-fotki.yandex.ru/get/4402/s-astanin.1d/0_4fffe_59e00e97_orig.jpg">Nicolas Wu (zenzike)</a>.
<p>So this is the complete list of JavaScript highlighters for Haskell which I know about:
<ul>
<li><a href="https://bitbucket.org/jetxee/src-highlight-haskell/" title="JavaScript highlighter for Haskell">this highlighter (SHJS + Haskell language definition)</a> (<a href="http://img-fotki.yandex.ru/get/4903/s-astanin.16/0_434c3_1fe66f60_orig">test screenshot</a>)
<li><a href="http://softwaremaniacs.org/soft/highlight/en/">highlight.js</a> (Haskell is supported since January 2010) (<a href="http://img-fotki.yandex.ru/get/5801/s-astanin.1d/0_4fff9_86c592f9_orig.jpg">test screenshot</a>)
<li><a href="http://code.google.com/p/google-code-prettify/">google-code-prettify</a> (<a href="http://img-fotki.yandex.ru/get/4505/s-astanin.13/0_3e358_c9ec3cd3_XL.jpg" title="google-code-prettify Haskell example">test screenshot</a>)
<li><a href="http://qbnz.com/highlighter/">GeSHi</a> (<a href="http://img-fotki.yandex.ru/get/4805/s-astanin.13/0_3e248_eb05d782_orig.jpg" title="GeSHi Haskell example">test screenshot</a>)
<li>An alternative Haskell language definition for SHJS by <a href="https://github.com/zenzike/src-highlite-haskell">Nicolas Wu</a> (<a href="http://img-fotki.yandex.ru/get/4402/s-astanin.1d/0_4fffe_59e00e97_orig.jpg" title="Haskell highlighted by SHJS with language definition by Nicolas Wu (zenzike)">test screenshot</a>)
</ul></p>
<p><a class="FlattrButton" style="display:none;"
href="http://nix-tips.blogspot.com/2010/09/javascript-highlighter-for-haskell-code.html"></a>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3091594052708753760.post-80192245142221013142010-09-14T17:06:00.004+02:002011-06-14T15:57:17.349+02:00Typography keyboard layout in Linux<p>From a Russian blog <a href="http://www.slovomania.ru/dnevnik/2010/09/11/typolayout-in-linux/">Slovomania</a> I learned that in the new versions of X11 there is a typography layout option, similar to <a href="http://ilyabirman.ru/english/typography-layout/">Ilya Birman layout</a>. This is its Linux counterpart:
<p><img src="http://img-fotki.yandex.ru/get/4603/s-astanin.13/0_3d9e0_c478a614_XL.jpg" width="758" height="258" title="Typography layout in Linux" alt="Typography layout in Linux" border="0"/>
<p>To enable it in GNOME: <em>System</em> → <em>Preferences</em> → <em>Keyboard</em>. Then <em>Layouts</em> tab, <em>Options...</em>. Enable <em>Key to choose 3rd level</em> (right <kbd>Alt</kbd> will do) and <em>Enable extra typographic characters</em> under <em>Miscellaneous compatibility options</em>.
<p><em>Also in Russian: <a href="http://sovety.blogspot.com/2010/09/typography-keyboard-layout-in-linux.html" title="Типографская раскладка в Линуксе">Типографская раскладка в Линуксе</a>.</em>
<p><a class="FlattrButton" style="display:none;" href="http://nix-tips.blogspot.com/2010/09/typography-keyboard-layout-in-linux.html"></a>
<noscript><a href="http://flattr.com/thing/310389/Typography-keyboard-layout-in-Linux" target="_blank">
<img src="http://api.flattr.com/button/flattr-badge-large.png" alt="Flattr this" title="Flattr this" border="0" /></a></noscript>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3091594052708753760.post-70775874613425201862010-09-04T16:32:00.017+02:002011-11-14T16:21:54.930+01:00LibZip 0.1: read and write zip archives from Haskell<p>I am happy to present a major release of my Haskell bindings to <a href="http://nih.at/libzip/index.html"><code>libzip</code></a> library, to manipulate zip archives. It took me longer than I initially expected, but finally I like the result. Essential links first:
<p><a href="http://hackage.haskell.org/package/LibZip">Hackage page</a>
<a href="http://bitbucket.org/jetxee/hs-libzip/">Source repository</a>
<a href="http://arboreus.com/sovety/doc/LibZip/Codec-Archive-LibZip.html">Documentation</a>
<a href="http://arboreus.com/sovety/doc/LibZip-0.1.0-hpc/">Test coverage report</a>
<a href="http://bitbucket.org/jetxee/hs-libzip/src/tip/examples/hzip.hs">An example</a>
<h4>What's new</h4>
<p>LibZip 0.1 is a complete remake. Under the hood it was made with <a href="http://bitbucket.org/mauricio/bindings-dsl/wiki/Home">bindings-DSL</a> instead of C2Hs as before. The new LibZip offers a lot:
<ul>
<li>Support of almost all features of <code>libzip</code>: creating, reading, updating, renaming, and deleting files in zip archives, reading and writing file and archive comments. (LibZip 0.0 was read-only)
<li>Support of various data sources: supply contents of a file from a list, from a file on disk, from a file in another archive, or even from a Haskell function.
<li>A new monadic interface. It takes care of managing handlers and pointers behind the scenes. Less to type, less space for a user error.
<li>Unit tests, better documentation and examples.
</ul>
<p>LibZip the only non-GPL library for Haskell to deal with zip archives. It is also fast to deal with large on-disk archives (more about it below).
<p>Users of LibZip 0.0 (all 1.5 of them) may still use the old API by importing <code>Codec.Archive.LibZip.LegacyZeroZero</code> instead of <code>Codec.Archive.LibZip</code>. However, the old API is deprecated and will not be supported in the future.
<h4>LibZip vs zip-archive</h4>
<p>There is another Haskell library to deal with zip archives, namely <a href="http://hackage.haskell.org/package/zip-archive">zip-archive</a>. And here are the differences:
<table>
<tr style="bolder"><td></td><td>LibZip 0.1/libzip</td><td>Zip-Archive 0.1</td></tr>
<tr><td>License</td><td style="color:green">BSD</td><td>GPL v.2</td></tr>
<tr><td>Pure?</td><td>No</td><td style="color:green">Yes</td></tr>
<tr><td>Large on-disk archives<td style="color:green">Fast</td><td>Slow</td></tr>
</table>
<p>Few notes about the last line. This is the actual reason why LibZip exists. Zip-archive was unacceptably memory-hungry and slow when dealing with large archives. So I started working on LibZip. I suppose that the problem with zip-archive is that it works with lazy bytestrings, not files. Bytestrings are sequential and don't have fseek. There is no reliable way to implement random access.
<p>To get an idea what's the problem, get some moderately sized zip-archive off the web (for example, <a href="https://sourceforge.net/projects/simutrans/files/pak128/pak128%20for%20102-2-2/pak128-1.4.6--102.2.zip/download">this one</a>, 22 MiB), and print the list of files using both libraries.
<p>With zip-archive, I used this code:
<pre class="sh_haskell">import Codec.Archive.Zip
import Control.Monad (liftM)
import System.Environment (getArgs)
import qualified Data.ByteString.Lazy as BS
main = mapM_ list =<< getArgs
list file = do
a <- toArchive `liftM` BS.readFile file
mapM_ print $ filesInArchive a</pre>
<p>On my laptop, it takes 3.5 seconds to run against the downloaded archive:
<pre>$ time ./zip-archive-ls pak128-1.4.6--102.2.zip > /dev/null
real 0m3.499s
user 0m3.430s
sys 0m0.050s</pre>
<p>And this is the code using LibZip:
<pre class="sh_haskell">import Codec.Archive.LibZip
import System.Environment (getArgs)
main = mapM_ list =<< getArgs
list file =
withArchive [] file $ do
names <- fileNames []
lift $ mapM_ print names</pre>
<p>It takes 0.05  seconds on the same file:
<pre>$ time ./libzip-ls pak128-1.4.6--102.2.zip > /dev/null
real 0m0.051s
user 0m0.040s
sys 0m0.010s</pre>
<p>The difference gets more dramatic as the size of the archive increases. So, in my opinion, license issues aside, LibZip is a better choice when dealing with large archives on disk. Zip-archive may be more suitable choice to generate small archives in memory, without even hitting the disk.
<h4>Some implementation notes</h4>
<p>I switched from C2Hs to bindings-DSL to implement the FFI bindings. And actually I liked bindings-DSL more. It is simple and makes fewer assumptions about the semantics of the C code. As a result, I had working low-level bindings very early. The rest was just to wrap them with a higher-level API to my liking. C2Hs experience was less smooth: in particular, when a C function is not designed as C2Hs expects it, I had to write wrappers manually anyway (for example, if a function returns a value <em>and</em> writes something to memory). Bindings-DSL seems to be better supported right now.
<p>I changed the order of file names and file access flags in all API functions. File name being the last seems to be more useful for partial function application. An example of such order of arguments is:
<pre class="sh_haskell">fileSize :: [FileFlag] -> FilePath -> Archive Int</pre>
<p>I ditched ByteString support from the new API. ByteStrings are my Haskell nightmare: there are too many flavours of them to support, and they are not interchangeable. With LibZip 0.0 I had to support two versions of otherwise identical code, all to discover some time later that I need to <code>pack . unpack</code> bytestrings in the application code (impendance mismatch with another library, which chose to use a different flavour of bytestrings).
<p>In this version I chose to use lists as input and output buffers to some functions (<code>sourceBuffer</code>, <code>sourcePure</code>, <code>readBytes</code>, <code>readContents</code>). I suspect this may have negative performance impact, but it needs to be studied. Marshalling of the byte buffers is another question. I suppose that <code>sourceFile</code> and <code>sourcePure</code> may help to workaround this problem if it actually arises. User feedback is required.
<p>Some of the library functions (most notably <code>sourceBuffer</code>) accept <code>String</code>s as data buffers. This is convenient for testing, but those Strings should not contains code points above 255. The library doesn't handle text encodings. The user is responsible of providing a correctly encoded <em>byte stream</em> to the library.
<p><code>Libzip</code> can use C callbacks as data source. LibZip bindings can wrap a pure Haskell function and make the C library call it when necessary (see <code>sourcePure</code>). It is not as convenient as the usual lazy evaluation in Haskell, but, hopefully, may somewhat compensate for impurity of the library. I consider adding also <code>sourceIO</code>.
<h4>Thanks?</h4>
<p>LibZip is under BSD3 licencse. So it is Free. If you want to say “thanks”, consider using this Flattr button:
<p><a href="http://flattr.com/thing/1850/Haskell-bindings-to-libzip-library" target="_blank">
<img src="http://api.flattr.com/button/button-static-50x60.png" alt="Flattr this" title="Flattr this" border="0" /></a>
<p>I think Flattr is a great idea and I'll be glad if more people start using it.
<p>If you use the library, please let me know. It will make me happier, and will motivate me more to support and improve the library.Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3091594052708753760.post-52462142897661800192010-08-05T19:08:00.016+02:002011-06-14T16:00:14.585+02:00Cleaning sensor dust with Gimp<p>Recently I found an easy cure for dust spots on the photos. I am talking about sensor dust here. Any owner of a camera with interchangeable lenses (and more than one lens) sees out-of-focus dark spots sooner or later. It is <a href="http://www.luminous-landscape.com/essays/sensor-cleaning.shtml">sensor dust</a>. For example:
<p><a href="http://img-fotki.yandex.ru/get/5103/s-astanin.12/0_38f37_4415d4f7_orig.jpg"><img src="http://img-fotki.yandex.ru/get/5103/s-astanin.12/0_38f37_4415d4f7_L.jpg" width="500" height="357" title="Sensor dust on the shot" alt="Sensor dust on the shot" border="0"/></a>
<p>It is easily recognised by always appearing on the place. The dust spots are most noticeable when shooting a uniform bright object (e.g. sky) with small aperture (f/22 or smaller). And cleaning the lens will not get rid of the spots. The right way to address it is to clean the sensor. <a href="http://www.google.com/search?ie=UTF-8&q=how+to+clean+sensor+dust">Someone else wrote about it</a>.
<p>Let's talk about how to remove these spots from the photos already taken. For this purpose I use Gimp and its <a href="http://www.logarithmic.net/pfh/resynthesizer">Resynthesizer</a> plugin. Clone brush is OK too, but cleaning more than a photo or two is tiresome. Think of Resynthesizer as an automatic clone brush. Ubuntu users may install it with <code>gimp-resynthesizer</code> package.
<p>To clean dust spots, first find and select them. I usually use a free selection tool <img src="http://docs.gimp.org/en/images/toolbox/stock-tool-free-select-22.png" alt="(Lasso)">. Press <kbd>Shift</kbd> to add more than one spot to selection. Select also a handful of “clean” pixels around. Which pixels are on the border of the selection matters.
<p>Run <em>Filters -> Map -> Resynthesize</em> filter. Default parameters should be OK, tiling options are not necessary for our scope.
<p><a href="http://img-fotki.yandex.ru/get/5402/s-astanin.12/0_38f36_8cf0a8c_orig.jpg"><img src="http://img-fotki.yandex.ru/get/5402/s-astanin.12/0_38f36_8cf0a8c_XL.jpg" width="529" height="473" title="" alt="" border="0"/></a>
<p>Now you have to wait a few seconds. Resynthesizer takes some time to redraw the selection. Anyway, it is faster than manual clone brush.
<p>Finally, inspect the result. Make sure that the plugin didn't draw anything strange. Usually it is OK from the first attempt:
<p><a href="http://img-fotki.yandex.ru/get/5400/s-astanin.12/0_38f38_998a1810_orig.jpg"><img src="http://img-fotki.yandex.ru/get/5400/s-astanin.12/0_38f38_998a1810_L.jpg" width="500" height="357" title="Sensor dust removed with Resynthesizer" alt="Sensor dust removed with Resynthesizer" border="0"/></a>
<p>Enjoy!
<p>And if useful, then <a href="http://flattr.com/thing/44848/Cleaning-sensor-dust-with-Gimp" target="_blank"><img src="http://api.flattr.com/button/button-compact-static-100x17.png" alt="Flattr this" title="Flattr this" border="0" /></a>
<p><a href="http://www.flickr.com/photos/arboreus/4767655865/" title="Riviera by arboreus, on Flickr"><img src="http://farm5.static.flickr.com/4082/4767655865_b214a47488_z.jpg" width="640" height="428" alt="Riviera" /></a>
<p><em>Also in Russian: <a href="http://sovety.blogspot.com/2010/08/clean-sensor-dust-with-gimp.html">Удаление пыли на матрице в Gimp</a>.</em>
<p><a class="FlattrButton" style="display:none;" href="http://nix-tips.blogspot.com/2010/08/cleaning-sensor-dust-with-gimp.html"></a>
<noscript><a href="http://flattr.com/thing/44848/Cleaning-sensor-dust-with-Gimp" target="_blank">
<img src="http://api.flattr.com/button/flattr-badge-large.png" alt="Flattr this" title="Flattr this" border="0" /></a></noscript>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3091594052708753760.post-24360615500119081882010-07-21T17:29:00.008+02:002010-07-21T18:13:06.239+02:00.emacs of a Vim userI've been using Emacs for few weeks now, and now I touch <code>.emacs</code> less and less often. Mostly I added alternatives to some Vim commands and defined more ergonomic keybindings. Here it is:
<a name='more'></a>
<script src="http://gist.github.com/484686.js"></script>
<a href="http://gist.github.com/484686">.emacs</a>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3091594052708753760.post-84223379066328895742010-07-07T16:57:00.006+02:002011-06-14T16:02:20.124+02:00The hierarchy of numeric typeclasses in Haskell<p>Some time ago I posted this in my <a href="http://sovety.blogspot.com/2009/04/class-diagram-of-haskell-numerical.html">Russian blog</a>. Re-posting it here, for it's inconvenient to not be able to find it when googling in English. I use it for reference :-)
<p><a href="http://lh4.ggpht.com/_PiUWFeprZSw/Sd72lQjUr3I/AAAAAAAAKfA/4PLE8uFQqUk/hs-nums.png"><img src="http://lh4.ggpht.com/_PiUWFeprZSw/Sd72lQjUr3I/AAAAAAAAKfA/4PLE8uFQqUk/s720/hs-nums.png" alt="Haskell Numeric typeclasses"/></a>
<p>Non-abstract types are gathered together in gray frames. Polymorphic types and type classes have rounded boxes. Their possible type parameters are indicated with inversed rhombus arrows.
<p><a class="FlattrButton" style="display:none;" href="http://nix-tips.blogspot.com/2010/07/hierarchy-of-numeric-typeclasses-in.html"></a>
<noscript><a href="http://flattr.com/thing/310397/The-hierarchy-of-numeric-typeclasses-in-Haskell" target="_blank">
<img src="http://api.flattr.com/button/flattr-badge-large.png" alt="Flattr this" title="Flattr this" border="0" /></a></noscript>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3091594052708753760.post-23927183289590593572010-06-28T21:44:00.006+02:002010-06-29T12:14:25.787+02:00Working AppEngine environment on Ubuntu Lucid<p>Ubuntu Lucid ships Python 2.6. Python 2.5 has been completely removed. Google AppEngine requires Python 2.5 to work. So if you want to develop AppEngine apps on a Lucid machine, you need to setup your working environment manually. This post tells how to do it.
<p>(I assume you install Python to <code>/opt/python2.5</code> and the user can write there; I assume you create a virtual environment, a directory where Python packages are installed, in <code>$HOME/.py25</code>. Choose different locations if you like and adjust instructions accordingly)
<p>1. Get the latest Python 2.5 release and build it from source.
<p>Make sure that you have necessary development libraries installed. In particular, you probably want to install <code>libsqlite3-dev</code> before building Python. Otherwise you'll have a Python without SQLite3 support, and GAE will not work with it.
<p>Go to a directory where you build software and do from the command line:
<pre class="sh_code">wget <a href="http://www.python.org/ftp/python/2.5.5/Python-2.5.5.tar.bz2">http://www.python.org/ftp/python/2.5.5/Python-2.5.5.tar.bz2</a> -O - | tar jx
cd Python-2.5.5
./configure --prefix=/opt/python2.5
make -j 2
make install
cd ..</pre>
<p>If there are configure errors, likely missing packages or header files for C libraries, install them and repeat.
<p>2. Get <code>virtualenv</code> and setup a new Python environment
<p>You will use a separate Python environment for AppEngine. So you will not mess with system packages. Fetch and unpack virtualenv:
<pre>wget <a href="http://bitbucket.org/ianb/virtualenv/get/tip.gz">http://bitbucket.org/ianb/virtualenv/get/tip.gz</a> -O - | tar zx</pre>
<p>Run it to get a new environment in <code>~/.py25</code> (I use full path to the newly installed Python 2.5 here):
<pre>/opt/python2.5/bin/python virtualenv/virtualenv.py ~/.py25</pre>
<p>Now to enable Python 2.5 you can source activate script from this environment, and to disable deactivate it:
<pre class="sh_code">$ which python ; python --version
/usr/bin/python
Python 2.6.5
$ source ~/.py25/bin/activate
(.py25)$ which python ; python --version
/home/sergey/.py25/bin/python
Python 2.5.5
(.py25)$ deactivate</pre>
<p>To install packages for Python 2.5 you can use pip. For example, to install Python Imaging Library (PIL), which is used by AppEngine, run:
<pre>(.py25)$ pip install PIL</pre>
<p>Or you can use <code>-E ~/.py25</code> option of <code>pip</code> without activating the environment.
<p>3. Run the development server
<p>Put your GAE SDK in the PATH, get the source of the application, and run the server. For example:
<pre class="sh_code">(.py25)$ git clone http://github.com/anotherjesse/webpy-appengine-helloworld.git gae-hello
...
(.py25)$ cd gae-hello
...
(.py25)$ dev_appserver.py .
INFO 2010-06-28 16:35:04,755 appengine_rpc.py:159] Server: appengine.google.com
INFO 2010-06-28 16:35:04,761 appcfg.py:357] Checking for updates to the SDK.
INFO 2010-06-28 16:35:05,068 appcfg.py:371] The SDK is up to date.
INFO 2010-06-28 16:35:05,106 dev_appserver_main.py:431] Running application hello-webpy on port 8080: http://localhost:8080</pre>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3091594052708753760.post-2709849476745068472010-06-15T14:29:00.003+02:002010-07-05T19:00:02.998+02:00Network Management disabled<p>Sometimes (I suspect it happens after complete battery discharge and incorrect shutdown), my Ubuntu Lucid machine boots with networking disabled. This is a nasty bug, because it doesn't allow to google for solution. <p /> The solution is <p /> </p> <div class="CodeRay"> <div class="code"><pre>sudo sed -i 's/\(NetworkingEnabled=\).*/\1true/' /var/lib/NetworkManager/NetworkManager.state</pre></div> </div> <p>and then</p> <div class="CodeRay"> <div class="code"><pre>sudo service network-manager restart</pre></div> </div> <p><p /> <a href="https://bugs.launchpad.net/ubuntu/+bug/555571">https://bugs.launchpad.net/ubuntu/+bug/555571</a></p>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3091594052708753760.post-24435100266272745292010-06-15T03:01:00.004+02:002010-07-06T17:51:09.395+02:00OpenMP in the Multicore Era<p>Nice slides by Christian Terboven on “OpenMP in the Multicore Era”. Enough to get started.</p> <p><a href="http://www.autodiff.org/ad08/talks/ad08_terboven.pdf">http://www.autodiff.org/ad08/talks/ad08_terboven.pdf</a></p> <p>By the way, to build an OpenMP program with modern GCC it’s enough to</p> <pre class="sh_sh">gcc -fopenmp -o eval42 eval42.c</pre> <p>It should work out of the box. GCC 4.1 supports OpenMP 2.5, and GCC 4.4 supports OpenMP 3.0.</p>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3091594052708753760.post-90532362572224378472010-06-14T17:16:00.008+02:002011-01-18T17:58:37.618+01:00Trying to do my daily tasks with Emacs<p>Lots of frustration, but I slowly remember some habits from 10 years ago. The thing I miss the most is ability to move quickly around and to delete/change an object of text, such as a word, a sentence, a paragraph, part of the line, a tag, etc. All those <b>d</b><i>something</i> from vim.<p /> Some command correspondence (beyond the crash course <a href="http://karl-voit.at/vim-emacs-cheatsheet_of_freezing_hell.txt">available elsewhere</a>):
<iframe width='780' height='400' frameborder='0' src='http://spreadsheets.google.com/pub?key=0AsGYigQZXDPKdHFDYndRS3V0RDVJMzdBaXVqaFZrcGc&hl=en&single=true&gid=0&range=A1%3AD24&output=html&widget=true'></iframe>
(<a href="http://spreadsheets.google.com/pub?key=0AsGYigQZXDPKdHFDYndRS3V0RDVJMzdBaXVqaFZrcGc&hl=en&single=true&gid=0&range=A1:D24">open table in a separate page</a>)</p>
<p /> It seems it's time to learn Emacs Lisp. I've bound <b>toggle-viper-mode</b> to C-<escape>. It's my preferred mode to move the cursor and delete stuff.Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3091594052708753760.post-15884620059074607882009-09-04T16:51:00.004+02:002009-09-04T17:05:12.367+02:00Tag installed packages to delete later<p>One of the lesser known but very useful features of <code>aptitude</code> is user tags.
<p>
For example, someone wants to install a set of packages temporarily just to build some program from source. Later, when the -dev packages are not needed anymore, it is not easy to remember which packages were installed.
<p>
Fortunately, one may tag the packages during install (I use <code>builddeps</code> tag in the example below):
<pre>$ sudo aptitude install <b>--add-user-tag builddeps</b> \
libsomething-dev libsomething-else-dev ...</pre>
<p>Now when these packages are to be deleted, we just do:
<pre>$ sudo aptitude purge <b>'?user-tag(builddeps)'</b></pre>
<p>The search pattern <code>?user-tag(yourtag)</code> can be used with any other <a href="http://algebraicthunk.net/~dburrows/projects/aptitude/doc/en/ch02s03.html">aptitude search patterns</a>. Also, you may tag packages during many other commands, not only <code>install</code>.
<p>
Tested in Debian Lenny (<code>aptitude-0.4.11</code>).
<p><em><a href="http://sovety.blogspot.com/2009/09/aptitude-tag-packages-to-remove-later.html" title="Как пометить пакеты в Aptitude, чтобы потом удалить">Прочесть по-русски</a></em>.Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3091594052708753760.post-48912320462543326162009-08-26T01:11:00.003+02:002009-09-04T17:06:32.150+02:003 ways to re-indent XML<p>There is a lot of data in XML formats, but often it's hardly readable: written by programs for programs, everything in one line. Indenting XML automatically helps to read such files.
<h5>1. Using XSLT</h5>I have a file with an XSL transformation:<pre class="sh_xml"><xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml"/>
<xsl:param name="indent-increment" select="' '" />
<xsl:template match="*">
<xsl:param name="indent" select="'&#xA;'"/>
<xsl:value-of select="$indent"/>
<xsl:copy>
<xsl:copy-of select="@*" />
<xsl:apply-templates>
<xsl:with-param name="indent"
select="concat($indent, $indent-increment)"/>
</xsl:apply-templates>
<xsl:value-of select="$indent"/>
</xsl:copy>
</xsl:template>
<xsl:template match="comment()|processing-instruction()">
<xsl:copy />
</xsl:template>
<!-- WARNING: this is dangerous. Handle with care -->
<xsl:template match="text()[normalize-space(.)='']"/>
</xsl:stylesheet></pre>
I found it <a href="http://www.dpawson.co.uk/xsl/sect2/pretty.html" title="Pretty Printing XML">here</a>. There are also some other variants.
<p>In addition to XSLT file, I have a one-line script which actually runs this transformation. I use <code><a href="http://xmlstar.sourceforge.net/">xmlstarlet</a></code>, which is a nice CLI utility to deal with XML.<pre class="sh_sh">#!/bin/sh
xmlstarlet tr ~/bin/indent-xml.xsl</pre>
Run this script as:<pre class="sh_sh">$ xmlindent < original.xml</pre>
Along with <code>xmlstarlet</code> you can use other XSL processors.
For example, <code>xsltproc</code> should work too.
<h5>2. Using <code>xmllint</code></h5>Inside <code>libxml2-utils</code> package (Debian/Ubuntu) there is an XML validator tool <code>xmllint</code>. It can also reformat (indent) XML:
<pre class="sh_sh">$ xmllint --format original.xml</pre>
This must be even easier.
<h5>3. <code>xmlindent</code></h5><code>xmlindent</code> is a pure C utility with almost no dependencies. It is intended to do just what it is named: indent XML. I didn't try it.
<p><em><a href="http://sovety.blogspot.com/2009/08/autoindent-xml.html" title="Автоматические отступы в XML">Прочесть по-русски</a></em>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3091594052708753760.post-44608648155049636152009-08-26T00:49:00.003+02:002009-09-04T17:07:23.177+02:00Monitor file changes in a shell script<p><strong>Problem:</strong> monitor file changes from a shell script and execute some commands when necessary. For example, rebuild LaTeX document or compile program every time when one of its source files is changed.
<p><strong>Solution:</strong> <a href="http://inotify-tools.sourceforge.net/">inotify-tools</a> help to monitor file changes. There are two utilities. The first one, <code>inotifywait</code>, blocks and waits for changes, then returns. If the event it was waiting for happened, its return code is <code>0</code> (success). See an example of using <code>inotifywait</code> below. The second utility is <code>inotifywatch</code>, it monitors files' changes, <em>collects</em> information and prints a nice table on exit. Please visit <a href="http://inotify-tools.sourceforge.net/">inotify-tools</a>' site to see examples of its use.
<p><strong>Example:</strong> <code>inotifywait</code> monitors all *.tex and *.bib files in the current directory, and when any of them changes, it runs pdfLaTeX and BibTeX to rebuild document:
<pre class="sh_sh">while true ; do \
inotifywait *.tex *.bib \
&& ( pdflatex -interaction=nonstopmode mypaper && \
bibtex mypaper && \
pdflatex -interaction=nonstopmode mypaper ) \
done</pre>
<p>P.S. Please note that when we run LaTeX with <code>-interaction=nonstopmode</code>, it does not ask questions on errors but we can still see those errors.
<p>P.P.S. <code>inotify-tools</code> run only on Linux. You may need to use <a href="http://mark.heily.com/pnotify" title="inotify alternative for BSD">pnotify</a> or <a href="http://wiki.netbsd.se/kqueue_tutorial">kqueue</a> on *BSD.
<p><em><a href="http://sovety.blogspot.com/2009/08/monitor-file-changes-in-shell-script.html" title="Как отслеживать изменения файлов в скриптах">Прочесть по-русски</a></em></p>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3091594052708753760.post-65073588560560353572009-06-15T10:00:00.009+02:002009-09-18T23:46:35.966+02:00Haskell horrors<p>As a beginner Haskell programmer I still remember its horrors. Time to write them down. It is my first functional language, so I have to learn new concepts and new language in parallel, but it is fun. I hope these notes may be useful to other beginners. I'll give also some examples in Python, to show that Haskell in fact is not so scary for beginners.
<h3>1. Lambdas</h3>
<div style="padding-left: 50%; font-style: italic;">We call it lembas or waybread, and it is more strengthening than any food by men, by all accounts…</div>
<p>This is why I learned about Haskell. I was just curious what those lambdas are. A paper <a href="http://portal.acm.org/citation.cfm?id=72554">Conception, evolution, and application of functional programming languages</a> by Paul Hudak was really helpful to get started (It's kind of old, and, probably, there are better introductions, but I read this one first).
<p>Lambda expression is an essence of ‘function’: it's an expression of form
<blockquote><img style="border: 0pt none ; padding: 1ex 2ex;" alt="\lambda x \;.\; \text{expression with $x$}" src="http://mathurl.com/krqmc7.png"/></blockquote>
<p>The <em>value</em> of this expression is a <em>function</em>, which takes one argument (<em>x</em>) and calculates something with it (an expression on the right hand side of the dot). There is much mathematical theory about lambdas, but sometimes I just think about <img alt="\lambda" src="http://mathurl.com/c8c2x3.png"/> as a <em>keyword</em> to define functions. In fact, I was already used to lambdas in Python, where they look like:
<pre class="sh_python">lambda x: expression with x</pre>
<p>They were very useful in <code>filter()</code> and <code>reduce()</code>. They may be used almost anywhere where a function name is required, but lambdas don't have names. Hence, they are called <a href="http://en.wikipedia.org/wiki/Anonymous_function">Anonymous functions</a>.
<p>Sometimes in Python I give them names to define new functions on-the-fly
<pre class="sh_python">add_42 = lambda x: x + 42</pre>
<p>Name <code>add_42</code> now refers to a function. This is almost the same as defining functions in a more usual way:
<pre class="sh_python">def add_42(x):
return x+42</pre>
<p>Now, what about Haskell? It's pretty much the same. <code>\</code> symbol stays for <img alt="\lambda" src="http://mathurl.com/c8c2x3.png"/>, <code>-></code> stays for dot:
<pre class="sh_haskell">\x -> expression with x</pre>
<p>We can even give them names to define reusable functions:
<pre class="sh_haskell">add_42 = \x -> x + 42</pre>
<p>Very similar, isn't it?
<p>There is a subtle point. As soon as I started reading about Haskell, I saw lambda-expressions which looked a little bit strange at first:
<pre class="sh_haskell">\x -> \y -> an expression with x and y</pre>
<p>What are those multiple ‘arrows’ I asked? The answer was really simple and very helpful to understand Haskell code.
<p>All Haskell functions are functions of one argument. This may sound like a restriction initially, but then it turns out to be a very convenient concept. A function of <em>n</em> arguments may be represented as a function of one argument producing another function of <em>n–1</em> arguments. This is called <a href="http://haskell.org/haskellwiki/Currying">currying</a>.
<p>Once we know this, we can read any expression with many arrows:
<pre class="sh_haskell">\x -> (\y -> an expression with x and y)</pre>
<p>The value of the expression is a function which takes an argument and produces another function which takes yet another argument. This expression as a whole behaves as if it were a function of two arguments. For example, we may apply such a function to two arguments in <code>ghci</code> (Haskell interpreter):
<pre>ghci> (\x -> \y -> x + y ) 3 4
7</pre>
Of course, there is a shorthand form where we can write it as a function of two arguments (please note also that we don't need parentheses for function arguments):
<pre>ghci> (\x y -> x + y) 3 4
7</pre>
<p>But it is very useful to know that internally every function of two or more arguments is actually a function of only one argument. This also helps reading function types later. For example, a type of <code>map</code> function reads like
<pre class="sh_haskell">map :: (a -> b) -> [a] -> [b]</pre>
<p>I usually read it like this: ‘function <code>map</code> takes two arguments, a function from <code>a</code> to <code>b</code> and a list of <code>a</code> and produces a list of <code>b</code>’. But sometimes it is more natural to read as written:
<pre class="sh_haskell">map :: (a -> b) -> ([a] -> [b])</pre>
<p>‘A function which takes a function from <code>a</code> to <code>b</code> and produces a function which converts a list of <code>a</code> to a list of <code>b</code>’.
<p>These simple ideas about lambda functions were enough to get me started with Haskell and to understand most of the examples and tutorials.
<h3>2. Equals sign</h3>
<div style="padding-left: 50%; font-style: italic;">‘If there's no meaning in it,’ said the King, ‘that saves a world of trouble, you know, as we needn't try to find any. And yet I don't know,’</div>
<p>As for me, equals sign <code>=</code> was probably the most important Haskell symbol to understand the language. I think its semantics is underemphasized in tutorials. For example, it's the only ‘keyword’ which is missing on a <a href="http://www.haskell.org/haskellwiki/Keywords">Haskell keywords</a>' wikipage.
<p>Unlike in most imperative languages where <code>=</code> means <a href="http://en.wikipedia.org/wiki/Assignment_(computer_science)">an assignemt</a> (i.e. <em>an action</em>), in Haskell it means that its left hand side <em>equals</em> its right hand side.
<p>‘Equals’ is not ‘becomes’. It means that something is equal to something else. Always. Just like in mathematics. <code>a = b</code> in Haskell means that <code>a</code> equals <code>b</code> <a href="http://mathworld.wolfram.com/Defined.html">by definition</a>, <code>a</code> is equivalent to <code>b</code>.
<p>So, <code>=</code> in Haskell serves to write definitions. It defines all sorts of things, but it defines them statically. It doesn't depend on the order of execution. It is something we may rely upon.
<p>This may sound too evident, but this is a major difference for anyone with an imperative background. Now we may give names to our anonymous functions:
<pre class="sh_haskell">add = \x -> \y -> x + y</pre>
To tell the truth, it's not very readable, so most of the time functions are defined like this:
<pre class="sh_haskell">add x y = x + y</pre>
But this one is still a <em>definition</em> of <code>add</code>.
<h3>3. Type classes</h3>
<div style="padding-left: 50%; font-style: italic;">Significant benefits arise from sharing a common type system, a common toolset, and so forth. These technical advantages translate into important practical benefits such as enabling groups with moderately differing needs to share a language rather than having to apply a number of specialized languages. — Bjarne Stroustrup</div>
Type system in Haskell is beatiful. It feels very natural to reason about. And probably type classes is the least alien concept for people coming to Haskell from procedural/OO world. At least they are for me. But type classes are not the same thing as classes in C++ or Java. They are much more like abstract template classes in C++, because they
<ul>
<li>define only an abstract interface (don't provide a default
implementation)</li>
<li>allow independent implementations (that is any class may
become an instance of the type class if it implements
class methods)</li>
<li>are polymorphic by their nature and support inheritance</li>
<li>don't have state variables</li>
</ul>
Once it's accepted that type classes are not C++ classes, but abstract interfaces, and class instances are not “objects”, but implementations of the abstract interfaces, Haskell becomes friendly and easy.
I suggest to read an excellent a wiki article <a href="http://www.haskell.org/haskellwiki/OOP_vs_type_classes">OOP vs type classes</a> which covers this topic in much more detail.
<h3>4. Monads</h3>
<div style="padding-left: 50%; font-style: italic;">And as every present state of a simple substance is naturally a consequence of its preceding state, so its present is pregnant with its future.</div>
<p>Now matter how gentle your introduction to Haskell is, sooner or later you stumble into the wall of <a href="http://en.wikipedia.org/wiki/Monads_in_functional_programming">Monads</a>. Yes, there is some serious abstract mathematics behind them.
<p>But I learned that it is not necessary to understand abstract mathematics to use monads, and they are indeed a very nice programming technique. They looked a little bit strange to me initially, but understanding monads is easier than memorizing countless OO <a href="http://en.wikipedia.org/wiki/Design_pattern_(computer_science)">design-patterns</a> and using them right.
<p>There are plenty of tutorials on monads, so I wont repeat them and expect that you've already read <a href="http://www.haskell.org/haskellwiki/Tutorials#Using_monads">one or two</a>. So, what's wrong with monads? For my imperatively prepared mind spoiled with years of object oriented thinking, monads looked strange. They may look like an abstract container class with a mystic <code>>>=</code> method:
<pre class="sh_haskell">class Monad m where
return :: a -> m a
(>>=) :: m a -> (a -> m b) -> m b
fail :: String -> m a</pre>Well, if <code>return</code> is a constructor, then why such a strange name? If this is a container class, how can I take values out? And what's the point of applying a function inside monad (<code>>>=</code> method, also known as <em>bind</em>) if we cannot take the result out of it?
<p>Let's answer the last question first. <b>What is the purpose of bind (<code>>>=</code>)?</b> Monads are and are not containers. They are wrappers around <em>computations</em> rather than values (<code>return</code>). But they wrap computations not to store them conveniently in monadic boxes, but to link them together. Think standard bricks rather than boxes, or Adapter pattern. Each monad defines a standard interface to pass the result from one computation to the other (<code>>>=</code>). No matter what happens, the result is still in the same monad (even <code>fail</code>).
<p>The most simple programming analogy I found is pipes in unix shell. Monads provide a unidirectional pipeline for computations. The same unix pipes do. For example:
<pre>$ seq 1 5 | awk '{print $1 " " $1*$1*$1 ;}'
1 1
2 8
3 27
4 64
5 125</pre>
<p><code>seq</code> produced a list of integers. <code>awk</code> calculated cubes for all of them. What's cool about this? We have two loosely coupled programs which work together. Text stream flows from left to write, each subsequent program in a pipe is capable to read this stream, do something with it, and output another text stream. Text stream is a common computation result, <code>|</code> binds computations together.
<p>Monads are similar. <code>>>=</code> takes the inner computation from the monad on the left and puts it into the computation on the right, which always produces the same monad. You probably already know that lists and Maybe type in Haskell are monads. For example, if we have a simple computation which returns a pair of a number and its cube into the monad:
<pre class="sh_haskell">\x -> return (x, x^3)</pre>
<p>then we can ‘pipe’ the list into this computation:
<pre class="sh_haskell">ghci> [1,2,3,4,5] >>= \x -> return (x,x^3)
[(1,1),(2,8),(3,27),(4,64),(5,125)]</pre>
<p>Please note that we received a list of pairs. It's the same monad (the list). But if we ‘pipe’ a Maybe value into the same computation, we get the same monad as input (Maybe):
<pre class="sh_haskell">ghci> Just 5 >>= \x -> return (x,x^3)
Just (5,125)</pre>
<p>You see, we can construct a pipeline of two computations, and the behaviour of this pipeline depends on the context (i.e. on the Monad instance), not on the computations themselves. Unlike unix pipes, Monads are type-safe, type system takes care that the output of one monadic computation is compatible with the input of the other. And unlike unix pipes, we can define our own binding rules (<code>>>=</code>). Like ‘don't join more than 42 computations in a row’ or ‘look at the input, and do one thing or the other’. Monads encapsulate rules how to bind computations together.
<p>Now, I hope, you understand monads at least as well as I do (not necessarily perfectly). I'd like to discuss some mnemonics. <b>Why <code>return</code> is named ‘return’?</b>
<p>In most languages, <code>return</code> returns result from a function. In Haskell, it acts as constructor for monads. Weird. Well, let's look at how <code>>>=</code> works, we see that it <em>extracts</em> a value from the monad on the left, and then binds it to the argument of the function on the right. This function should <em>return</em> the result back into monad, so that the next <code>>>=</code> can work. This is the first mnemonics.
<p>The second mnemonics. On the top level any Haskell program is executed in <code>IO</code> monad (<code>main</code>'s type is <code>IO ()</code>), which can do input/output (and sequential operations in general). Thus, monadic code is executed on the top level and it calls any pure code when necessary, not otherwise. So any pure value if not discarded is eventually <em>returned</em> into the monad of the caller.
<p>With these two explanations the name <code>return</code> does not sound so weird for me as before. I don't pretend these explanations are technically correct.
<p>The next question, <b>how to take a value of the computation out of the monad?</b> Well, it's not always possible <em>by design</em>. For example, you cannot take a pure value outside of the <code>IO</code> monad. If we have such a one-way monad, the only thing we can do, is pipeline the value to the next computation. Fortunately, Haskell has a nice <code>do</code>-notation, which makes such pipelining look almost identical to imperative programming languages. These two programs describe the same thing, with do-notation:
<pre class="sh_haskell">main :: IO ()
main = do
name <- getLine
putStrLn ("Hi, " ++ name)</pre>
and with explicit <code>>>=</code>:
<pre class="sh_haskell">main :: IO ()
main = getLine >>= \name -> putStrLn ("Hi, " ++ name)</pre>
An equivalent Python program:
<pre class="sh_python">from sys import stdin, stdout
if __name__ == "__main__":
name = stdin.readline()
stdout.write("Hi, " + name)</pre>
However, sometimes it is possible to take the pure value out of the monadic computation. This is not possible with the standard monad interface, so the developer of the monad should provide means to extract values. For example, it is possible to extract values from Maybe monad using <code>fromMaybe</code> function:
<pre class="sh_haskell">ghci> fromMaybe 0 $ Just 3
3
ghci> fromMaybe 0 $ Nothing
0</pre>
<p><b>Résumé on monads</b>
<p>So, bind (<code>>>=</code>) allows to put various monadic computations together and combine them. Anywhere, where we have a chain of computation, monads fit naturally. Particular monad implementation may encapsulate custom rules of combining two computations together. Name <code>return</code> may be confusing for beginners, but we have to remember that it returns a value into the monad, not from the monad. Understanding this helped me a lot.
<h3>5. Scary words</h3>
<div style="padding-left: 50%; font-style: italic;"> I know nothing except the fact of my ignorance.</div>
Finally, I have to acknowledge, after months of studying Haskell and being able to write useful programs in it, there is still a lot of concepts in the Haskell world which I don't know anything about or have only a vague idea. I call these kind of concepts “scary words”. But still I see people creating and using new libraries which implement those concepts.
Let's face it: Haskell remains a test bed for research. It's both good and bad. It's good, because it feels like a bleeding edge is really close, and you may benefits from the new approaches if you like. And it's bad, because if you want to use some cool new library, you may find that it heavily relies on a concept you are not very comfortable with.
For example, there is a modern XML library, HXT. It uses <em>arrows</em> heavily. Arrows provide more general combinators than monads, but it took me much more than a day to understand them. Strictly speaking, Arrows are not part of the language, but they are an actively used concept in practice. There are many other examples like this.
I think that it's important not to be afraid of the “scary words”. Fortunately, all the concepts are well documented. There are papers which explain the ideas. Personally I decided to learn such concepts as need arise. At least it promises to be manageable and entertaining.
<h3>Conclusion</h3>
I mentioned 5 simple ideas which helped me to get used to Haskell: lambdas are just a way to write functions, a function of two or more arguments is actually a function of only one argument which returns another function, type classes correspond to abstract polymorphic interfaces in the object oriented wold, monads are just a tool to merge computations together, and scary words are just scary words.
I hope this helps someone else.
<em>This post is also available <a href="http://sovety.blogspot.com/2009/09/haskell-horrors.html" title="(Новичковые) ужасы Хаскеля">in Russian</a>.</emUnknownnoreply@blogger.comtag:blogger.com,1999:blog-3091594052708753760.post-63322382480985379162009-06-09T13:03:00.012+02:002009-07-20T21:54:02.092+02:00Debian Lenny on Samsung X22<h3>The laptop</h3>
<p>My new laptop is <a href="http://www.samsung.com/uk/business/b2b/products/notebooks/business_class/x22.htm">Samsung X22</a>. I like its serious, almost ascetic look. The screen is matte, not glossy. The weight is just within my requirements: 2.18 kg (unfortunately, an AC adapter is bulky). What's more, the laptop came <a href="http://badvista.fsf.org/">without Vista</a>. I don't mind having WinXP Pro around just in case I need to do test some software on Windows (or to play games). The keyboard is big and very similar to the one of my previous laptop (only <kbd>Ctrl</kbd> and <kbd>Fn</kbd> are swapped). Nice things to have: 1.3 MP webcam and SD/MS/xD card reader (I have some xD cards, but many embedded card readers don't support them). On the flip side: the laptop is rather big (14"), it is not water-shock-whatever-proof, the screen is only 1280×800 and its viewing angle is relatively narrow, and the battery life is short. Bonus: Bluetooth, ExpressCard slot, HDMI, optical drive.
<p>This is the first time I bought a laptop with discrete graphics (Ati Radeon HD2400 aka RV610). The open source driver does support 3D only in <a href="http://xorg.freedesktop.org/wiki/radeon">experimental branches</a>, but since I don't need 3D right now and AMD/Ati was kind enough <a href="http://www.phoronix.com/scan.php?page=article&item=amd_r600_700_guide&num=1">to release the specs, some code and programming guide</a>, I hope this card will have good open source drivers in the future. Anyway, with Intel I could not use 3D neither :)
<p>Hardware: <a href="http://pastebin.com/f46f717d8">lspci</a>, <a href="http://pastebin.com/f799ef72f">lsusb</a> and <a href="http://pastebin.com/f20bb803a">dmidecode</a>.
<h3>Installation</h3>
<p>I used <code>netinst</code> image of Debian Lenny (5.0.1) and installed <a href="http://www.debian.org/releases/stable/i386/ch04s03.html.en">from a USB pen drive</a>. This was the first time I tried Debian graphic installer. Well, I liked it. Probably, partition editor was not very GUI-sh, but it's OK. I installed <code>i386</code> kernel.</p>
<p>The installer asked me for <code>iwlwifi</code> proprietary firmware (I didn't have one nor didn't put it on the installation pendrive). So, I installed from ethernet. Installation was smooth.</p>
<h3>Post-installation</h3>
<h4>Wireless</h4>
<p>I installed <a href="http://packages.debian.org/lenny/all/firmware-iwlwifi/download">firmware-iwlwifi</a> package, and the wireless now works perfectly. For convenience, I replaced <code>network-manager</code> with <code><a href="http://wicd.sourceforge.net/">wicd</a></code>. I like it more.</p>
<h4>Keyboard</h4>
<p>Some Fn-combos don't work. Most notably: <kbd>Fn</kbd>+(<kbd>F2</kbd>|<kbd>F4</kbd>|<kbd>F5</kbd>|<kbd>F8</kbd>|<kbd>F9</kbd>|<kbd>F10</kbd>|<kbd>Up</kbd>|<kbd>Down</kbd>). These keys don't produce any <code>xev</code> keycode. Among features which are not available are manual brightness (<kbd>Fn</kbd>+<kbd>Up</kbd>/<kbd>Fn</kbd>+<kbd>Down</kbd>) and wireless switch (<kbd>Fn</kbd>+<kbd>F9</kbd>).
<p>I found a kernel bug (almost resolved) with another Samsung model: <a href="http://bugzilla.kernel.org/show_bug.cgi?id=12021" title="Fn + brightness up/down keys doesn't work if ACPI enabled - Samsung NC10">#12021</a>, and another similar Debian bug <a href="http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=475851" title="keycode info for similar Samsung Q45 laptop">#475851</a> with Samsung Q45. After reading through it, I managed to enable brightness buttons with
<pre>$ sudo setkeycodes e008 225 e009 224</pre>
<p>Finally, after <a href="http://pastebin.com/f51dc8175">a small patch</a> to hal-info, all the keys work (at least, generate <code>xev</code> codes).
<p>Good news: sleep button, mute and volume buttons work out of the box.
<h4>Video</h4>
<p><code>radeon</code> driver in Debian Lenny is working OK (2D), but XVideo is not enabled by default, so probably I have to try a newer kernel or <a href="http://wiki.x.org/wiki/radeon%3Ar6xx_r7xx_branch">rebiuld the driver</a>. Up-to-date version claims to support XVideo.</p>
<p>Proprietary Ati driver (<code>fglrx</code>) works too; (hint: enable non-free repository and install <code>fglrx-driver</code>, run <code>aticonfig</code>).</p>
<p>I tried also newer <code>radeon</code> driver from the unstable (since 6.12 <code>radeon</code> supports video and 2D acceleration on R6xx). In fact, HD video played even better than with a proprietary driver. Though I couldn't get proprietary <code>fglrx</code> working with X.org from unstable.</p>
<h4>Webcam</h4>
<p>Webcam just works. I tried it in Cheese and in Skype. Both work. Picture quality is good. It seems to be the same Vimicro webcam (USB ID <a href="http://linux-uvc.berlios.de/">0ac8:c302</a>) used in Samsung Q45 notebook.</p>
<h4>Sound</h4>
<p>When I installed Lenny, sound was playing without problems, but I could not get microphone working. As usual with Lenny and Intel-HDA hardware, I had to build newer ALSA (as I did <a href="http://sovety.blogspot.com/2008/12/eeepc-901-debian.html" title="Debian Lenny на eeePC 901">for eeePC 901</a>, link points to a post in Russian). To make mic working on X22 I also had to add in <code>/etc/modprobe.d/alsa-base</code> this line:
<pre>options snd-hda-intel model=ultra</pre>
This <a href="https://help.ubuntu.com/community/HdaIntelSoundHowto">HDA Intel Sound HOWTO</a> was very helpful. I found, that Samsung X22 uses Realtek ALC262 codec. Then, I tried some <a href="http://www.mjmwired.net/kernel/Documentation/sound/alsa/HD-Audio-Models.txt">ALC262 models</a>, and <code>model=utlra</code> worked.
<h4>Suspend</h4>
<p><kbd>Fn</kbd>+<kbd>Esc</kbd> did not restore screen/backlight on wake up. Blind typing upon wake up was useless.
<p>I found, that either
<pre class="sh_sh">s2ram -f -a 2</pre>
or
<pre class="sh_sh">pm-suspend --quirk-s3-mode</pre>
suspend and restore the session normally. In X they restore the screen perfectly, but in console they don't turn on backlight. However, <kbd>Ctrl</kbd>+<kbd>Alt</kbd>+<kbd>F7</kbd> switches to X and the backlight is restored. Minor glitch: when going to suspend, the screen is filled with blinking ’ñ’. Not a big deal, but I don't think it's normal.
<p>I edited <code>/usr/share/hal/fdi/information/10freedesktop/20-video-quirk-pm-samsung.fdi</code>, and the snippet corresponding to X22 now looks like this:
<pre> <!-- this does not work on my SX22S! -->
<match key="system.hardware.product" string_outof="R40/R41;CoronaR">
<merge key="power_management.quirk.vbestate_restore" type="bool">true</merge>
</match>
<!-- I use this one: -->
<match key="system.hardware.product" string="SX22S">
<merge key="power_management.quirk.s3_mode" type="bool">true</merge>
</match></pre>
<p>Now <kbd>Fn</kbd>+<kbd>Esc</kbd> works as intended.
<h4>Optical drive</h4>
<p>It seems to support most of the CD and DVD media, but I didn't use it much yet. It is reading CDs without problem.
<h4>Card reader</h4>
<p>Works with my xD and SD cards as intended. I don't have MemoryStick cards to test.
<h3>Things not tested</h3>
<p>I didn't try HDMI, ExpressCard slot, Bluetooth at all.
<h3>Links</h3>
<ul>
<li><a href="http://sovety.blogspot.com/2009/06/debian-lenny-samsung-x22.html" title="Debian Lenny на Samsung X22">This report in Russian (abbreviated)</a>.
</ul>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3091594052708753760.post-5287409161538792052009-05-21T15:29:00.005+02:002011-04-07T13:29:09.768+02:00Animated gif to avi/flvUsing <a href="http://www.lcdf.org/gifsicle/">gifsicle</a> to unoptimize the animated GIF and split it into frames, and <a href="http://www.ffmpeg.org/">ffmpeg</a> to build a video:
<pre>gifsicle -U --explode "input.gif"
for f in *.gif.* ; do mv "$f" "$f.gif" ; done
ffmpeg -r 25 -i "input.gif.%03d.gif" -sameq -s 320x240 output.flv</pre>
If we need to do some padding with black:
<pre>ffmpeg -i input.file -s 320x180 -padtop 30 -padbottom 30 output.file</pre>
UPDATE: With newer ffmpeg (for example, ffmpeg 0.6.90), this becomes:
<pre>ffmpeg -i input.file -vf "scale=320:180,pad=320:240:0:30" output.file</pre>
I don't use <code>convert</code> (ImageMagick) to split frames, because <code>gifsicle</code> appears to work faster and require less memory.
<p><em><a href="http://sovety.blogspot.com/2009/09/animated-gif-to-avi-or-flv.html" title="Как сделать видеофайл из анимированного GIF-а">(читать по-русски)</a></em></p>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3091594052708753760.post-14786674445532744862009-05-21T14:52:00.010+02:002009-07-05T22:21:35.804+02:00Random Python snippets<h3>Lists and dictionaries</h3>
Given a flat list, like <code>[key1, value1, key2, value2]</code> convert it to an alist or dictionary:
<pre class="sh_python">>>> toalist = lambda kvs: zip(kvs[0::2], kvs[1::2])
>>> toalist(range(4))
[(0, 1), (2, 3)]
>>> dict(toalist(range(4)))
{0: 1, 2: 3}</pre>
Convert a dictionary to a flat list:
<pre class="sh_python">>>> # dict to alist
... al = list({1:2,3:4}.iteritems())
>>> al
[(1, 2), (3, 4)]
>>> # alist to flat list
... reduce(lambda acc,t: acc + list(t), al, [])
[1, 2, 3, 4]</pre>
To tranpose a list of lists/tuples unpack as a list of function arguments and zip <code>zip(*mylist)</code>:
<pre class="sh_python">>>> l = list(enumerate("abcdef"))
>>> l
[(0, 'a'), (1, 'b'), (2, 'c'), (3, 'd'), (4, 'e'), (5, 'f')]
>>> # transpose list of lists/tuples
... zip(*l)
[(0, 1, 2, 3, 4, 5), ('a', 'b', 'c', 'd', 'e', 'f')]
>>> # once again
... zip(*_)
[(0, 'a'), (1, 'b'), (2, 'c'), (3, 'd'), (4, 'e'), (5, 'f')]</pre>
Flatten a list of lists:
<pre class="sh_python">>>> lofl = [[1,2], [3], [4,5]]
>>> import operator
>>> reduce(operator.add, lofl)
[1, 2, 3, 4, 5]</pre>
An alternative approach is to use <code>chain</code> from <code>itertools</code> (this works also on huge lists if used wisely!):
<pre class="sh_python">>>> list(itertools.chain(*lofl))
[1, 2, 3, 4]</pre>
Apply a function to <em>either</em> an iterable (list, tuple) <em>or</em> a scalar:
<pre class="sh_python">>> def fmap(f,xs):
... try: return map(f,xs)
... except TypeError: return f(xs)
...
>>> fmap(lambda x:x*x, range(5))
[0, 1, 4, 9, 16]
>>> fmap(lambda x:x*x, 5)
25</pre>
<h3>Strings and Unicode</h3>
Unicode stuff is <a href="http://docs.python.org/3.0/howto/unicode.html#unicode-howto">changing in 3.0</a>. For earlier versions, it is important to distinguish strings (<code>"abc"</code>) and unicode strings (<code>u"abc"</code>). The former can be converted to the latter with <code>unicode()</code>:
<pre class="sh_python">>>> "абв"
'\xd0\xb0\xd0\xb1\xd0\xb2'
>>> u"абв"
u'\u0430\u0431\u0432'
>>> unicode("абв","utf8")
u'\u0430\u0431\u0432'</pre>
Please note there are 3 unicode symbols in the original literal and there are three values in the unicode string. This is how the strings are to be represented internally.
Any communication with an external world usually requires that unicode data is <em>encoded</em>. There are various encodings, "UTF-8" is one of the most common. Any encoded input should be <em>decoded</em> to be processed:
<pre class="sh_python">>>> "абв".decode("utf8")
u'\u0430\u0431\u0432'
>>> u"абв".encode("utf8")
'\xd0\xb0\xd0\xb1\xd0\xb2'</pre>To live a long and happy life it is important to understand if you are working with an encoded data (practically binary data) or decoded unicode text.
To test if an object is a string (either ascii string or unicode), test if it is an instance of <code>basestring</code>:
<pre class="sh_python">>>> isinstance("abc",basestring)
True
>>> isinstance(u"abc",basestring)
True
>>> isinstance(42,basestring)
False</pre>
To convert to a string and from string (depends on type):
<pre class="sh_python">>>> str(42)
'42'
>>> unicode(42)
u'42'
>>> int("42")
42
>>> float("42")
42.0</pre>
<h3>Backporting to Python 2.4</h3>
With Python 2.5, 2.6 and even 3.0 around, I still need to make some scripts run with Python 2.4. Just two tricks, to make sqlite3 code work:
<pre class="sh_python">try:
import sqlite3
except:
from pysqlite2 import dbapi2 as sqlite3 # cheating with py2.4</pre>
and to make ElementTree work:
<pre class="sh_python">try:
import xml.etree.ElementTree as ET
except:
import cElementTree as ET # not xml.etree in py2.4, use celementtree</pre>Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3091594052708753760.post-75727898500376789422009-02-23T23:15:00.007+01:002014-05-23T15:05:29.964+02:00epi2fox: import Epiphany bookmarks into Firefox 3<p>I used Epiphany as my main browser for a long time because I find its bookmarks system much better than anything else. However, as new Firefox 3 permits tagged bookmarks too, I decided to give it a try once again. But I wanted all my bookmarks from Epiphany available in Firefox too. With the same tags.
<p>I didn't find any ready solution, so I wrote a script, <a href="http://bitbucket.org/astanin/epi2fox/src/tip/epi2fox.py" title="convert bookmarks from Epiphany to Firefox 3">epi2fox.py</a>. Assuming, you have an almost empty Firefox profile, run this script like this:<pre>$ epi2fox.py ~/.mozilla/firefox/yourprofile/places.sqlite</pre>The script is not perfect, but it did the job. One of its major shortcomings: while Epiphany permits multiple bookmarks for the same URL, Firefox does not. Probably, such bookmarks should be merged on importing, but the script just throws away duplicates (and prints error messages).
<p>Links:<ul>
<li><a href="http://bitbucket.org/astanin/epi2fox/">epi2fox, script repository</a>
<li><a href="https://wiki.mozilla.org/images/d/d5/Places.sqlite.schema3.pdf">Firefox bookmarks DB schema</a> (PDF)
<li><a href="http://sovety.blogspot.com/2009/02/epi2fox-epiphany-firefox-3.html">epi2fox: конвертация закладок Epiphany в Firefox 3</a> (this announcement in Russian)</ul>
<p>PS. Please backup your <code>places.sqlite</code> before running the script.Unknownnoreply@blogger.comtag:blogger.com,1999:blog-3091594052708753760.post-41414719491181803712009-01-28T21:18:00.008+01:002009-05-22T15:06:58.533+02:00rss2xmpp, a script to crosspost any feed to JabberUsage: <pre>$ rss2xmpp.py feed-URL your-jabber-id</pre>On the first run the script will complain that you have to put jabber settings in <code>~/.rss2xmpp</code>. It writes an example for GoogleTalk there. Either RSS or Atom feeds should work.
<p>Requirements: <a href="http://www.feedparser.org/">FeedParser</a>, <a href="http://www.aaronsw.com/2002/html2text/">html2text</a>, and <a href="http://xmpppy.sf.net/">xmpppy</a>, and Python, of course.
<p>The script itself is in the BitBucket: <a href="http://bitbucket.org/jetxee/rss2xmpp/src/tip/rss2xmpp.py" title="rss2xmpp.py">rss2xmpp.py</a>.
<p>BTW, I discovered, that BitBucket not only provides free OpenSource hosting for mercurial repositories, but has free SSH access and allows one private repository per account.
<p>See also:
<p>☙ <a href="http://sovety.blogspot.com/2009/01/rss2xmpp-jabber.html">Скрипт rss2xmpp, кросспост чего угодно в Jabber (this post in Russian)</a>Unknownnoreply@blogger.com