LinuxMeerkat

I swear! Meerkats can do Linux


Leave a comment

Why you should use super in Python

Although I used Python a long time and OOP, I never really dwelled into the reasons that someone would use super instead of other ways in Python. Usually I would use other ways in an effort do avoid confusing words and those ugly underscores. Sometimes however it is worth making something a bit less readable and such a case is super.

Why should you learn to use super though? For a single reason.. super equals less headaches.

Calling a parent class’ initializer

Remember that with initializer I merely mean the __init__ method of a class. So let’s take for example the class A below.

class A(object):
  def __init__(self):
    print('This is A')
  def hello(self):
    print('Hello, this is a method from A')

When we instantiate this class, the initializer runs and thus we get printed ‘This is A’. Now we want B to inherit A:

class B(A):
  def __init__(self):
    print('This is B')

The result is a hybrid class – half A, half B. The problem is that both classes have an initializer and in such cases the hybrid’s methods, variables, etc. are preferred. So in practice B has the method hello that it inherited from A but will only run its own initializer.

In real life we tend to run the initializer for every parent class. This is simply because of how program designs tend to be. In our simple example a way to solve this is to explicitly call the initializer of A:

class B(A):
  def __init__(self):
    A.__init__(self)
    print('This is B')

The self always makes me dizzy so I will explain a bit on it. Namely why can’t we just have A.__init__()? The reason is that A is not an instance but a class. self however is an instance and that’s why we use it. As you might have noticed though, it is an instance of B and still we pass it to A as if it was an instance of A. So why the hell does it work?

The reason it works is that as we said B is a hybrid – half A, half B. This is very similar to having a double citizenship. A half Greek, half Norwegian can be recognized in both Greece and Norway. In the same way A and B can be recognized as either A or B. Logical, aye?

The bless of not knowing

The above example works fine. But what if one changes the name of A into G? For a simple example like ours, we could just change every occurence of A into G. However if you are dealing with medium to large projects you might have many classes that inherit from A and way many files. Furthermore if you have tests, you probably have all sort of test classes that inherit as well.

The point is that in such cases a little change somewhere can invoke havoc. The programmer will need to keep track of every single place where we inherit class A which just is not practical. That’s where super comes into play.

With super we can call A’s initializer without ever typing the name of the class:

class B(A):
  def __init__(self):
    super(B, self).__init__() # notice we type B, not A
    print('This is B')

Now, no matter if you rename A to G or V, you won’t have to make any changes to classes that inherit from that class!

The bless of caring even less

So you saw how super takes away the problem of having to keep track of class names we inherit from. I think all this makes much more sense when we inherit from multiple classes.

Say we have classes X and Y:

class X(object):
  def __init__(self):
    print('This is X')

class Y(object):
  def __init__(self):
    print('This is Y')

Now if B inherit from everyone else, in the no-super way it will look like this:

class B(A, X, Y):
  def __init__(self):
    A.__init__(self)
    X.__init__(self)
    Y.__init__(self)
    print('This is B')

With super we can minimize it to:

class B(A):
  def __init__(self):
    super(B, self).__init__(self)
    print('This is B')

At first glance this looks like we merely minimize the code to a single line. The real benefit however is that if we did not use super, now our class B would be much more prone to mistakes since either A, X, or Y might change name somewhere (more classes – higher probability of a rename).

I hope all this makes it very apparent that in big OOP projects where you have a lot of interaction between objects, classes, etc. Using super is just a simple trick that adds a huge gain for the programmer. So whenever you need to call an initializer (or any other method) from a parent class, please save yourself some trouble and use super!

Python 2 issues

You might have noticed that I use object in every parent class in the examples above. In Python 3 you don’t have to do this.

class A(object):
  ..

This merely makes a class inherit from object. The problem in Python 2 you see is that not everything is an object. As such we have to explicitly state it. In Python 3 all classes inherit from object be default so the code becomes much cleaner. Notice that even super is much cleaner in Python 3:

class A:
  ..

class B(A):
  def __init__(self):
    super().__init__() # no self pollution
    ..


2 Comments

Converting hex to binary in the brain

This is a tutorial on hex which is very useful if you are ever going to read low-level code or program low-level things like network protocols or microcontrollers. I use a real project that I worked on to showcase all this, namely a matrix of 9 LEDs.

led_fisheye_300

You should be able and understand why people put hex in the code instead of raw binary (if it exists for that programming language). There are very specific reasons for doing this and since converting from hex to binary is so damn easy, there is no excuse for you to not be able and do it in your brain.

Binary and LED patterns

I was building a trivial LED matrix the other day for an MBED microcontroller (think Arduino-like). The usual problem is that my brain is faulty so I do all sorts of things in the wrong way. I take this blog as the opportunity to make up for what I learn just to make sure that I won’t forget (and ofcourse to teach others if they are interested).

So my task was to achieve some patterns with 9 LEDs. Notice that it doesn’t matter how the microcontroller was connected etc. since here I am only dealing with how bits and bytes fit into low level programming. My first pattern was a rotating star that you can see below:

rotating-star

This star is made out of two LED patterns: a star and a cross.

text3997

The 1s and 0s simply mean if the LED at a position should be turned on or off. Now, when we’re dealing with low level things the minimum unit of information that can be sent is 1 byte (= 8 bits). In our case we have 9 LEDs so we need however a minimum of 2 bytes (= 16 bits). The above examples become the below binaries:

Star:  0000000101010101
Cross: 0000000010111010

Now, the problem is that when we deal with low level programming, most low level languages (C, C++ etc.) don’t let you write numbers as binary in your code. You can’t write printf(0b101) for example. You need separate libraries if you want to do that and that would be fine for our case. But imagine if there was a matrix of 100 LEDs. Someone reading printf(001001010101101010101010101010101110101001011100101) would just get lost in the 0s and 1s. That’s one of the big reasons hex is used – it’s super minimal.

Binaries as integers

At first when I wanted to create a star, I simply converted each binary to an integer and just put it in my code. Below you can see a simplified version of my code.

..
#define STAR  341
#define CROSS 186

int main() {
    while (1) {
        leds.write(CROSS)
        sleep(1)
        leds.write(STAR)
        sleep(1)
    }
}

The way I found those integers was by simply using Python. It is a rather trivial procedure:

>>> 0b101010101
341
>>> 0b10111010
186

Notice that I omit the extra 0s since they don’t add any value just like 000150 is always going to be 150 no matter how many zeros you add at front.

Binaries as hex

The code I used, worked fine. The problem with this solution is that it’s impossible to have a clue what an integer is in binary – and when we deal with low-level programming that matters most of the times. In our case for example each single 1 and 0 controls one LED. Being able to figure out fast the binary of a number in this case is very important.

For example say you find the code snippet below somewhere:

#define STAR1 341
#define STAR2 186

How can you tell if it’s the STAR1 or STAR2 that looks like an ‘X’? It’s just impossible. And what if there were many more stars or if the LED matrix was huge? Then it would be a nightmare to understand the code. That’s where hex comes in handy.

The good thing with hex is that someone can see the hex and decode it to binary in his brain. So say we had the above example but instead of integers had hex values:

#define STAR1 0x155
#define STAR2 0xba

A skilled programmer would directly see 0001 0101 0101 and 0000 1011 1010 with no effort. And he wouldn’t either need to decode the whole number to find out. Watching just the last hex digit of each STAR would give him (or us) a hint about which STAR is which.

It’s time we become that skilled programmer, don’t you think?

Hex to binary in da brain

Fortunately it is very simple to convert hex to binary in the brain. You simply have to understand that each hex number is made out of 4 bits since we need a max of 4 bits to represent the largest number in base 16 (which is the character ‘F’). So 0xF is 0b1111. (Notice that putting 0x in front denotes that the number is in hexadecimal represation and putting 0b denotes the binary representation accordingly.)

The procedure of binarizing a hex is simple:

  1. Find the binary of each hex character
  2. Place 0s in front of each binary (from above) so we always have 4 digits
  3. Squeeze them all together as if they were strings

So for example:

F   is       1111
5   is       0110
FF  is  1111 1111
55  is  0110 0110
5F  is  0110 1111
F5  is  1111 0110

Hopefully you get the hang of it. The question is.. what happens if we have 0x102? This might seem tricky since we get three very simple binaries: 1, 0 and 10. But as I said, if you add the 0s in front before you squeeze them together, you should get the correct value – in this case 1 0000 0010!

Also you need to memorise a few things to be able and do all this. I have written the bare minimum below:

Binary     Decimal
   1    =    1
  10    =    2
 100    =    4
1000    =    8
1010    =    A
1111    =    F   

Then it’s quite easy to find in brain all the rest. For example to find the binary of B we can simply think that A is 1010, and that since B is just one step ahead, we add 1 to it and thus get 1011. Or to find 5 we simply have to add 1 to 4 which becomes 100+1=101. And so on.

This should also make it clear what the command chmod 777 in Linux does.

Big fat hex stars

The below is more like an exercise to test what we’ve learned. It should be rather simple to find the hex of the star below.

star_7x7

It might seem overwhelming, but the only thing you need to do is go in groups of 4s and write down each hex value.

Grouping in 4bit groups:
star_7x7_marked

Decoding the above becomes 8388A08388A0 which is WRONG.

Row 1: 8 3..
..
Row 7: ..A 0 (and a remaining 1?!)

This was actually a trap to teach you the hard way that we should always start from the last digit. In this case in the end we are in a situation where we have an orphan digit 1. We can’t work with that since we need 4 digits to make a hex number.

The right way is to always start from the end. This is for all numbers no matter if they are represented in binary, octal, hex, decimal or whatever – as long as they are numbers, always start from the last digit and you’ll do fine. The reason is that when you finally get to the last number you can add as many zeros as you like (or need) without altering the value of the whole thing.

So the correct grouping is this (starting from bottom-right):
star_7x7_marked_right

And then we just start from the bottom and get 1051141051141! Notice that in the end we again have a single 1 (at the top left this time), but this time we can add as many zeroes as we want since adding zeros in front of a number doesn’t change its vallue.

We can also validate what we got with Python:

>>> bin(0x1051141051141)
'0b1000001010001000101000001000001010001000101000001'


18 Comments

Running a GUI application in a Docker container

This guide will show you how to run a GUI application headless in a Docker container and even more specific scenarios involving running Firefox and Chrome. If you are not interested about those then you can just stop in the middle of this tutorial.

What the hell is X?

X is a program that sits on a Linux machine with a monitor (so servers usually don’t use X). X’s job is to talk to the Linux kernel in behalf of GUI programs. So if you are playing a game for example, the game (that is, the application) is constantly sending drawing commands to the X server like “draw me a rectangle here”. X forwards all this to the Kernel which will further forward the information to the GPU to render it on the monitor.

X can even receive commands from the keyboard or mouse. When you click to shoot on your game for example, the command “click at 466,333” is sent from your mouse to the kernel, from the kernel to the X and from X to the game. That way the game can have a clue on what is happening!

You will often hear X being called a server and the reason for that is simply because the way the applications send commands to it is through sockets. For that reason the applications are also referred to as clients many times.

If you are reading this then the X is running on your PC. Let’s prove it:

> ps aux | grep X
root      1436  3.2  0.7 687868 94444 tty7     Ssl+ 09:47   2:50 /usr/bin/X -core :0 -seat seat0 -auth /var/run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch

We can see that X is running as root and has PID 1436. An other important thing is to notice the :0 which is called display in X jargon. A display is essentially:

  • A monitor
  • A mouse
  • A keyboard

And this is the bigger picture of how it all looks together:

Now there is a variable in Linux that is used whenever we run a GUI program. That variable is surprisingly called DISPLAY. The syntax of the DISPLAY variable is

<hostname>:<display>.<monitor>

. Let’s check the DISPLAY on our computer:

> echo $DISPLAY
:0

I get :0, which means we use display 0. Notice however that this says nothing about which monitor we use. This makes sense since if you are running 2 or more monitors on your Linux you still have the same environment variables in both of them. It wouldn’t make sense that an environment variable changes just because you echo it from a different screen, would it? For that reason we get the display and not the monitor so that we get the same output on both. As about the hostname, since there is no info about it, the local host is assumed.

On a notice, if you have multiple monitors you can still specify which monitor to run an application by simply typing the full display variable you want. So if you have a monitor 0 and a monitor 1 on the current display, I can run firefox on monitor 1 with:

DISPLAY=:0.1 firefox

Creating a virtual monitor

Instead of running X, we can run a different version of it that can create virtual displays. Xvfb (virtual framebuffer – whatever the hell that means) will create a virtual monitor for us.

So let’s make a new monitor (I assume you have installed xvfb):

Xvfb :1 -screen 0 1024x768x16

This will start the Xvfb server with a display 1 and a virtual screen(monitor) 0. We can access this by simply typing DISPLAY=:1.0 before running our graphical program. In this case the program will start in the virtual screen instead of our monitor.

Let’s make sure that the screen is still running:

> ps aux | grep X
root      1436  3.1  0.7 684580 91144 tty7     Ssl+ 09:47   3:31 /usr/bin/X -core :0 -seat seat0 -auth /var/run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch
manos    22018  0.0  0.1 164960 20756 pts/27   Sl+  11:37   0:00 Xvfb :1 -screen 0 1024x768x16

We see we have the normal display 0. (A way to tell it is the default screen is to see that it runs as root.) We can also see the second display :1 and screen 0 with resolution 1024×768. So what if we want to use it?

Open a new terminal and type:

> DISPLAY=:1.0 firefox
..

This will start firefox at the given display. The reason I use the DISPLAY at the same line is to make sure that the subprocess inherits the variable DISPLAY. An other way to do this is to type:

> DISPLAY=:1.0
> export DISPLAY
> firefox
..

Run a GUI program in a Docker container

We will now create a virtual screen inside a docker container.

> docker run -it ubuntu bash
root@660ddd5cc806:/# apt-get update
root@660ddd5cc806:/# apt-get install xvfb
root@660ddd5cc806:/# Xvfb :1 -screen 0 1024x768x16 &> xvfb.log  &
root@660ddd5cc806:/# ps aux | grep X
root        11  0.0  0.1 169356 20676 ?        Sl   10:49   0:00 Xvfb :1 -screen 0 1024x768x16

So now we are sure that we are running the virtual screen. Let’s access it and run something graphical on it. In this case I will run Firefox and Python+Selenium just as a proof of concept of what is happening.

First I put my display variable and use export to assure that any sub-shells or sub-processes use the same display (with export, they inherit the variable DISPLAY!):

root@660ddd5cc806:/# DISPLAY=:1.0
root@660ddd5cc806:/# export DISPLAY

Now we can simply run a browser

root@660ddd5cc806:/# firefox
(process:14967): GLib-CRITICAL **: g_slice_set_config: assertion 'sys_page_size == 0' failed
Xlib:  extension "RANDR" missing on display ":99.0".
(firefox:14967): GConf-WARNING **: Client failed to connect to the D-BUS daemon:
//bin/dbus-launch terminated abnormally without any error message
..

The errors don’t mean anything. But we can’t be sure, can we? I mean, since we can’t see what’s happening it’s really hard to tell. There are two things we can do, either use ImageMagick to take a snapshot and send it to our host via a socket or we can simply use Selenium. I will do that since most people probably want to achieve all this for testing purposes anyway.

root@0e395f0ef30a:/# apt-get install python-pip
root@0e395f0ef30a:/# pip install selenium
root@0e395f0ef30a:/# python
>>> from selenium import webdriver
>>> browser=webdriver.Firefox()
>>> browser.get("http://www.google.com")
>>> browser.page_source

If you get a bunch of HTML, then we have succeeded!

The Chrome issue

If you try and run Chrome in a Docker container, it won’t work even if you have setup everything correctly. The reason is that Chrome uses something called sandboxing. Reading this I could not let but notice the word jail. Apparently it seems that Chrome uses Linux containers (the same that Docker uses). For this reason you have to put a bit of extra effort to solve this issue since because of technical difficulties it’s not possible to run containers in containers.

There are two workarounds:

  1. Use my docker-enter
  2. Use –privileged when running the container

The second solution is probably the best one. However while testing things, there’s nothing wrong with the first one.

So to make things work (notice I run everything from the start):

> docker run -it --privileged ubuntu bash
root@7dd2c07cb8cb:/# apt-get update
root@7dd2c07cb8cb:/# apt-get install wget python-pip xvfb
root@7dd2c07cb8cb:/# wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
OK
root@7dd2c07cb8cb:/# echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list
root@7dd2c07cb8cb:/# apt-get update
root@7dd2c07cb8cb:/# apt-get install google-chrome-stable
root@7dd2c07cb8cb:/# pip install selenium

I have now installed Selenium, Chrome and Xvfb. Now I am going to make make a virtual monitor and run Chrome:

root@7dd2c07cb8cb:/# Xvfb :99 -screen 0 1024x768x16 &> xvfb.log &
[1] 6729
root@7dd2c07cb8cb:/# DISPLAY=:99.0
root@7dd2c07cb8cb:/# export DISPLAY
root@7dd2c07cb8cb:/# google-chrome
Xlib:  extension "RANDR" missing on display ":99.0".
Xlib:  extension "RANDR" missing on display ":99.0".
[6736:6736:1017/143449:ERROR:desktop_window_tree_host_x11.cc(802)] Not implemented reached in virtual void views::DesktopWindowTreeHostX11::InitModalType(ui::ModalType)
ATTENTION: default value of option force_s3tc_enable overridden by environment.
failed to create drawable
[6775:6775:1017/143449:ERROR:gl_surface_glx.cc(633)] glXCreatePbuffer failed.
[6775:6775:1017/143449:ERROR:gpu_info_collector.cc(27)] gfx::GLContext::CreateOffscreenGLSurface failed
[6775:6775:1017/143449:ERROR:gpu_info_collector.cc(89)] Could not create surface for info collection.
[6775:6775:1017/143449:ERROR:gpu_main.cc(402)] gpu::CollectGraphicsInfo failed (fatal).
[6775:6775:1017/143449:ERROR:sandbox_linux.cc(305)] InitializeSandbox() called with multiple threads in process gpu-process
[6775:6775:1017/143449:ERROR:gpu_child_thread.cc(143)] Exiting GPU process due to errors during initialization
[6736:6736:1017/143449:ERROR:gpu_process_transport_factory.cc(418)] Failed to establish GPU channel.

It seems that it works. It’s normal that we get the gpu errors since we don’t have a gpu! However I don’t like gambling so we will take it a step further to check that the browser actually works. However for this I will need to download the webdriver for Google Chrome..

root@7dd2c07cb8cb:/# apt-get install curl unzip
root@7dd2c07cb8cb:/# cpu_arch=$(lscpu | grep Architecture | sed "s/^.*_//")
root@7dd2c07cb8cb:/# version=$(curl 'http://chromedriver.storage.googleapis.com/LATEST_RELEASE' 2> /dev/null)
root@7dd2c07cb8cb:/# url_file="chromedriver_linux${cpu_arch}.zip"
root@7dd2c07cb8cb:/# url_base="http://chromedriver.storage.googleapis.com"
root@7dd2c07cb8cb:/# url="${url_base}/${version}/${url_file}"
root@7dd2c07cb8cb:/# wget "$url"
root@7dd2c07cb8cb:/# unzip chromedriver_*.zip -d tmp
root@7dd2c07cb8cb:/# mv tmp/chromedriver usr/bin/

Now (FINALLY!) we can test with Selenium:

root@7dd2c07cb8cb:/# python
>>> from selenium import webdriver
>>> browser=webdriver.Chrome()
>>> browser.get("http://en.wikipedia.org/wiki/Open_source")
>>> browser.page_source

You should get a bunch of HTML code. So there we go!

Common errors

.. Gtk: cannot open display:

DISPLAY has wrong value or you forgot to export it!

References

http://www.x.org/wiki/Development/Documentation/HowVideoCardsWork/
http://www.x.org/archive/X11R7.7/doc/man/man1/Xvfb.1.xhtml
http://blog.mecheye.net/2012/06/the-linux-graphics-stack/
http://people.freedesktop.org/~marcheu/linuxgraphicsdrivers.pdf
http://www.tldp.org/HOWTO/Framebuffer-HOWTO/
http://en.wikipedia.org/wiki/X_Window_System
http://en.wikipedia.org/wiki/Framebuffer
http://en.wikipedia.org/wiki/X.Org_Server
http://en.wikipedia.org/wiki/Display_server
http://www.google.com/googlebooks/chrome/med_26.html
http://linux.die.net/man/1/xvfb
https://www.freebsd.org/doc/handbook/x-understanding.html