A faster YAML loader

sam. 20 août 2016 by Rémi Duraffort

That's the second issue I had when playing with the LAVA log viewer.

In the new versions of LAVA, the logs are formatted in YAML:

- {"dt": "2016-08-18T14:24:01.096308", "lvl": "info", "msg": "start: 1 tftp-deploy (max 300s)"}
- {"dt": "2016-08-18T14:24:01.099413", "lvl": "debug", "msg": "start: 1.1 download_retry (max 300s)"}
- {"dt": "2016-08-18T14:24:01.100674", "lvl": "debug", "msg": "start: 1.1.1 file_download (max 300s)"}

That's really convenient, but when a job is generating a lot of logs, loading this YAML files is becoming longer and longer:

% time python -c "import yaml; y=yaml.load(open('output.yaml'));"
18,25s user 0,23s system 100% cpu 18,475 total
% wc -l output.yaml
36817 output.yaml

But 18s to load 36817 lines of text sounds unreasonable. I looked for some explanation and found that by default, the Python YAML parser, is using the Pure-Python loader instead of the (way) faster C implementation.

So, in order to use the faster C implementation, you should use:

% time python -c "import yaml; y=yaml.load(open('output.yaml'), Loader=yaml.CLoader);"
2,28s user 0,06s system 99% cpu 2,346 total

That's still really slow, but we can live with that for the moment.

Playing with CSS rules

ven. 12 août 2016 by Rémi Duraffort

While working on improving the log viewer in LAVA I ran into an issue.

The issue is really simple: how to add display: block or display: none to a list of nodes while another JS script is adding more of theses nodes to the page.

The classical approach is to ...

read more

Logging for ser2net

mer. 22 juin 2016 by Rémi Duraffort

Today, I was debugging an strange issue in LAVA while trying to boot a board.

I was under the impression that the board was not receiving all the u-boot commands that LAVA was sending.

The serial connection is accessible on the network by ser2net.

I discovered that ser2net can log ...

read more

Static analysis for Python

ven. 13 mai 2016 by Rémi Duraffort

I've always been a big fan of static analysis. While I was working with C and C++, I was scanning my code with:

For a longer list of static analyzers for each language, look at wikipedia

As I'm now mostly ...

read more

type() vs isinstance()

mar. 03 mai 2016 by Rémi Duraffort

When running pylint on your Python source code, you might encounter this message:

Using type() instead of isinstance() for a typecheck. (unidiomatic-typecheck)

This message will be raised for this kind of code:

d = dict()
if type(d) == dict:
    print("d is a dict")

The code is valid but pylint does ...

read more

Migrating from Python2 to Python3

mar. 19 avril 2016 by Rémi Duraffort

Porting code from Python 2 to Python 3 is made easier by using 2to3. This application will find patterns that should be changed to keep the same behavior in Python 2 and 3.

However, 2to3 is sometime too conservative, trying to keep the exact same semantic.

For instance, dictionaries functions ...

read more


mer. 08 janvier 2014 by Rémi Duraffort

We will be at FOSDEM to talk about PRoot and some other tools based on it.

Cedric will do a Lightning talk Saturday about syscall instrumentation. He will present some tools, based on syscall instrumentation, that we develop and use at STMicroelectronics:

  • PRoot, emulate chroot, bind mount and binfmt_misc for ...
read more

Pretending to be root inside PRoot

jeu. 21 novembre 2013 by Rémi Duraffort

While working on PRoot to improve the extension that fake the root user, I (re)discovered an old article that I never published before. So let's focus on the fake_id0 feature and on how it allows to use package manager directly inside the root file systems.

Some root causes ...

read more

PRoot now in Debian

mar. 20 août 2013 by Rémi Duraffort

A really short message to inform you that PRoot is now part of Debian Sid.

You can now install PRoot with apt-get install proot.

Enjoy !

read more

Scaling the time

lun. 24 septembre 2012 by Rémi Duraffort

Using valgrind on time-dependent softwares like VLC media player is often a nightmare and leads to poor results. Let's present a simple and elegant way to workaround this issue

Valgrind and realtime softwares

Valgrind is a really useful tool that checks for memory leaks, wrong memory access and many ...

read more