LinuxMeerkat

I swear! Meerkats can do Linux


Leave a comment

Python detection of USB storage device with DBUS

I spent quite some time lately trying to do all of sort of things with USB and Python. I will put out what I’ve learned through this course. Ofcourse I could use Pyusb or some other library but where’s the learning in that? As the Brits say: give a man a fish and you feed him for a day; teach a man to fish and you feed him for a lifetime.

This tutorial starts with a brief overview and then goes into general usage of Dbus and lastly using Dbus with UDisks2 to get notified if a USB storage device has been inserted.

Update:
I initially wrote this article with the title “Python detection of USB storage device”. After some time of working with UDisks2 and Dbus however I found out that the code tends to be very ugly and unexpected behaviours occur (not sure if it’s because of Dbus or UDisks2). So if you want to control an input/output device like a USB device or get events, I recommmend you use pyudev straight away! I will probably write an article on how to use that too. If you want to use DBus for educational purposes or want to use it maybe with some other software, please continue reading.

The daemons checkin’ out your USB

There are two main daemons running on Ubuntu (and probably all major distros) that deal with connecting/disconnecting devices.

The one is udisks which deals with storage devices like USB sticks and the like. The second is udev, a daemon that deals with all kind of devices from PCI boards to the keyboard and mouse (including everything that udisks deals with).

Let’s prove that they exist and that they are running on our system:

manos@box:~$ ps ax | egrep 'udev|udisks'
  352 ?        S      0:00 upstart-udev-bridge --daemon
  357 ?        Ss     0:00 /lib/systemd/systemd-udevd --daemon
 2844 ?        Sl     0:00 /usr/lib/gvfs/gvfs-udisks2-volume-monitor
 2850 ?        Sl     0:00 /usr/lib/udisks2/udisksd --no-debug
..

Now, depending on how old your Linux distribution is you might have udisks or udisks2 (fourth process) and then you have the udev daemon (second process). But what do these daemons do? Well let’s check for ourselves, shall we?

Let’s monitor our USB

Open two terminals. In the first type:

udisksctl monitor

And on the second one type:

udevadm monitor

These are front-ends for the daemons udisks and udev respectively.

Now, while looking at the two terminals we do the below:

  • First remove or put in a non-storage device like a mouse or a keyboard (USB mouse/keyboard works fine).
  • Now put in or remove a storage device (USB stick, external hard disk, etc).

You will notice that for the first action, udev prints a bunch of stuff while udisks does nothing. For the second case where we mingle with a storage device, both services print out stuff.

For example when I insert a USB stick I get this on the udisk terminal:

manos@box:~$ udisksctl monitor
Monitoring the udisks daemon. Press Ctrl+C to exit.
11:31:00.793: The udisks-daemon is running (name-owner :1.76).
11:31:05.850: Added /org/freedesktop/UDisks2/drives/Samsung_1040d8f1f9b92b56e6d1af1d7a0dd4c82781
  org.freedesktop.UDisks2.Drive:
    CanPowerOff:                true
    Configuration:              {}
    ConnectionBus:              usb
    Ejectable:                  true
..

and this on the udev terminal:

manos@box:~$ udevadm monitor
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent

KERNEL[2569.327243] add      /devices/pci0000:00/0000:00:14.0/usb2/2-3/2-3.4/2-3.4.2 (usb)
KERNEL[2569.328960] add      /devices/pci0000:00/0000:00:14.0/usb2/2-3/2-3.4/2-3.4.2/2-3.4.2:1.0 (usb)
KERNEL[2569.329026] add      /devices/pci0000:00/0000:00:14.0/usb2/2-3/2-3.4/2-3.4.2/2-3.4.2:1.0/host7 (scsi)
..

This makes it clear that udisks records only changes to storage devices like USB sticks and hard disks. udev on the other hand monitors any device that can connect/disconnect to your PC.

On a sidenote (in case you’re wondering) the KERNEL[blah blah] messages that appeared on the udev terminal, are Netlink messages sent from the kernel to udev.

The bigger picture

Now, we know that udisks is just like udev but with the difference that udisks plays with storage devices.

It turns out that udisks actually uses udev itself:

DEVICE INFORMATION
udisks relies on recent versions of udev(7) and the Linux kernel.

So what does udev do then? Well udev’s main work is to populate the /dev folder on your root directory. So it’s kind of a discovering daemon that tells the kernel what devices are connected and where. That’s all!

Below you can see the bigger picture of how everything works together.

Kernel udev udisks dbus communication

udev, udisks and dbus, they are all daemons running in the background from the time you start your computer.

Netlink and Dbus
Netlink and Dbus are two different protocols used for processes to talk with each other. In the first case the kernel communicates with udev and in the second case udisks with the dbus daemon. Many applications use the dbus protocol like Unity, NetworkManager, Skype, and pretty much all Gnome apps.

Notice that the dbus-daemon that is running all the time is part of the dbus protocol. I talk more about how dbus works, later so you get an understanding as to why a daemon is needed.

What’s udev?

The kernel uses the Netlink protocol (essentially UNIX sockets) to send messages to udev daemon. These messages are the ones we saw earlier when we typed udevadm monitor in the terminal. The way this works is by udev setting up some specified rules that the kernel reads at bootup. Then whenener a device is connected to your computer, a message to udev is sent from the kernel.

udisks uses the udev library and thus has access to all this. If you can use udev then you can pretty much access the kernel messages sent to udev.

And what about that dbus thingy?
The dbus daemon is not specific to udisks only. dbus daemon runs always and is a generic solution for processes to exchange information. NetworkManager for example uses this, so someone could create a program that automatically gets notified if an ethernet card is inserted by simply using dbus. All programs that use dbus, have access to each other. Luckily for us, udisks uses dbus 🙂

Choosing between udev, dbus and direct kernel messaging

From all the above it should become apparent that there are three main ways we can get notified about a USB insertion:

  1. Talk directly with the kernel with Netlink messages. In this case we essentially place ourselves in udev’s position.
  2. Use the udev library like udisks does. In python this is done with using pyudev.
  3. Use the Dbus protocol to talk with udisks directly.

There are pros and cons with all these three approaches. I will explain a bit on all three starting from the lowest level to the highest level.

Kernel
The kernel way is probably the best but is probably a hell to get working since it’s so low-level. You need to deal with raw sockets since that’s how Netlink communication is done. The good thing with this approach is that you are not dependent on anything more than the kernel itself. So it’s pretty hard that something will break and you can always be sure that your solution will work for any distribution.

udev
udev is probably the best practical solution. It handles the messages sent from the kernel so we don’t need to handle them. We only use a udev library and that’s all. From what I am aware all major Linux distributions use udev so you’re not going wrong with this solution. The only bad thing (and this is my own opinion) is that the udev library bindings for Python don’t come with Python so you need to install them on your system on your own.

Dbus
Dbus uses itself udev as we mentioned earlier so it has already a dependency. However as we sayied, pretty much all distros use udev so that shouldn’t be a problem. The main good things with Dbus is that the dbus library comes with Python (both 2 and 3) so you can start coding directly without having to install a bunch of stuff. An extra good thing is that maaaaaaany programs use the Dbus protocol so once you learn it, you can potentially access a lot of things. The bad thing with dbus is its ugly design (my opinion again).

In this tutorial I will talk about dbus since it’s one of the things that you learn once and can use in many different situations in the future.

How dbus works

Choosing Dbus means we will have to learn a bit on how Dbus protocol works and then try and find out how udisks uses dbus. I will try to make this as easy as possible even if it’s a quite hard task considering how sparse information is and how ugly some things in dbus are.

So what exactly is dbus? How does it connect things together? I think the easiest way to explain, is by showing you a diagram.

dbus diagram

The dbus-daemon keeps a sort of a freeway which all programs can access. In our diagram, udisks, NetworkManager and our python program can share information between them. The dotted yellow rectangles in the diagram are representing the messages exchanged between the programs.

There are actually two freeways (buses)

Dbus uses two different freeways like the one I mentioned above. That also means there are two dbus-daemon running. One is a system-wide freeway and the other is a session-wide freeway. In this context, session is the duration of time that a user is logged in.

These dbus daemons are running:

ps ax | grep dbus-daemon
  844 ?        Ss     0:00 dbus-daemon --system --fork
 2451 ?        Ss     0:00 dbus-daemon --fork --session --address=unix:abstract=/tmp/dbus-IP8uQnHdYR
 2545 ?        S      0:00 /bin/dbus-daemon --config-file=/etc/at-spi2/accessibility.conf --nofork --print-address 3

As you see there are two dbus-daemon running, or three in this case. Omit the third one (no idea what that is). The two main daemons are the first two. Notice the --system in the first one and the --session in the second. These are exactly what you might guess they are.. The first one keeps the system-wide freeway and the second keeps the session-wide freeway. I think it’s time we drop the ‘freeway’ word and use bus instead.

Now each freeway has its own rules. Furthermore, each program using dbus, can setup its own rules. All these rules are of the type “root can send this type of message” and “any user can access this type of message but not send this type”, etc.

An important thing to notice is that surprisingly there is a dbus-daemon running on your system. This daemon is used to put all programs using dbus communication together. So when we will try to connect to program A via dbus, we will actually connect to dbus-daemon which acts as a mediator.

So how do we use this module? I give below two examples. First I try to access NetworkManager and then udisks. I do this to show you how the workflow is and as a proof that you can use dbus for more than merely USB insertion scenarios.

Listing programs connected to dbus

The simplest way to find which programs are connected to dbus is from Python.::

>>> import dbus
>>> bus = dbus.SystemBus()
>>> bus.list_names()
dbus.Array([dbus.UTF8String('org.freedesktop.DBus'), dbus.UTF8String(':1.7'), dbus.UTF8String(':1.8'), dbus.UTF8String(':1.9'), dbus.UTF8String('org.freedesktop.ModemManager1'), dbus.UTF8String('org.freedesktop.NetworkManager'), dbus.UTF8String('com.ubuntu.Upstart'), dbus.UTF8String('org.freedesktop.Accounts'), dbus.UTF8String('org.freedesktop.RealtimeKit1'), dbus.UTF8String(':1.60'), dbus.UTF8String(':1.61'), dbus.UTF8String(':1.62'), dbus.UTF8String(':1.40'), dbus.UTF8String('com.canonical.NMOfono'), dbus.UTF8String(':1.85'), dbus.UTF8String(':1.63'), dbus.UTF8String(':1.86'), dbus.UTF8String('org.freedesktop.PolicyKit1')
..

That will list all programs connected to the system bus. In practice, this means that we can connect to any of these programs (as long as we connect to the right bus ofcourse)!

In the same fashion we can list programs connected to the session bus (these are probably much more):

>>> import dbus
>>> bus = dbus.SessionBus()
>>> bus.list_names()
dbus.Array([dbus.UTF8String('org.freedesktop.DBus'), dbus.UTF8String(':1.128'), dbus.UTF8String('com.canonical.Unity.Launcher'), dbus.UTF8String('org.freedesktop.Notifications'), dbus.UTF8String(':1.7'), dbus.UTF8String('com.canonical.indicator.datetime'), dbus.UTF8String(':1.8'), dbus.UTF8String(':1.9'), dbus.UTF8String('org.gtk.Private.AfcVolumeMonitor'), dbus.UTF8String('com.canonical.indicator.sound'), dbus.UTF8String('org.gtk.vfs.Daemon'), dbus.UTF8String('org.pulseaudio.Server'), dbus.UTF8String('com.canonical.indicator.application'), dbus.UTF8String('com.canonical.Unity.Webapps.Service'), dbus.UTF8String('com.ubuntu.Upstart'), dbus.UTF8String(':1.80'), dbus.UTF8String('org.gnome.SessionManager'), dbus.UTF8String(':1.81'), dbus.UTF8String('org.gnome.evolution.dataserver.Sources2'), dbus.UTF8String('com.canonical.indicator.session'), dbus.UTF8String('com.canonical.hud'), dbus.UTF8String(':1.82'), dbus.UTF8String(':1.83'),
..

Notice that in some cases we get domain looking strings and in some cases we get weird numbers like ‘:1.83’ and ‘:1.128’. These are pretty much the same thing as domains and IP on the internet. In this case however instead of the domain/IP pointing to a host, they point to a program on our computer. Ofcourse the number “:1.128” is quite cryptic and there’s actually no way to know which program it resolves to. But that is fine since the programs that are meant to be accessed, are given domain-like names just so that we can find them easily.


The right tools

Now that we know which program we want to connect to, we have to somehow find out how and what we can access from it. For this I will use a tool called d-tree. This tool gives you an overview of all programs connected to dbus and also give you a list of things that you can access.

On Ubuntu you can install this program directly with:

sudo apt-get install d-feet

On a side-note, you might feel tempted to access all this via Python or by using the terminal. Do your self a favour and don’t. I already took that path and there are just too many issues you might run into that you’ll never be able and truly understand how things run without getting in the low-level source code.

Running a program’s method

Whenever you want to access a program’s method or property, you need exactly four things:

  1. The program’s domain name
  2. Object path
  3. Interface
  4. The method or property (signals are a special case so will talk about them separately)

All these things are pretty random and make little sense. However that’s the API that we have for DBus so if you want it you have to bare with me. You just have to learn how to find each one of them and then how to use them in your Python program to access the property or method you want. You will be using d-feet to locate all these four things!

As an example I will take NetworkManager. Just look below..

d-feet NetworkManager

The method I want to run in this case is GetDevices(). However in order to run it I have to locate all the other things: the program, the interface and the object path. Just looking at the snapshot it’s easy to see where all these are located. As said, you just have to get used to finding them for any program.

Translating all this to code looks like this:

import dbus
bus = dbus.SystemBus()
obj = bus.get_object('org.freedesktop.NetworkManager', '/org/freedesktop/NetworkManager')
obj.GetDevices()

Where ‘org.freedesktop.NetworkManager’ is the program’s domain and ‘/org/freedesktop/NetworkManager’ is the so called object path. There is no real logic as to what an object path is. It’s just some bad implementation in the core, where someone tried to bring object oriented coding into Dbus which is written in C. What came out of this effort is this monster of illogical terms that just don’t fit together and just complicate things. However we have to learn all this if we are to use Dbus.

Output from the above code:

dbus.Array(
  [
    dbus.ObjectPath('/org/freedesktop/NetworkManager/Devices/0'),
    dbus.ObjectPath('/org/freedesktop/NetworkManager/Devices/1')
  ],
    signature=dbus.Signature('o')
)

Now if you are a person that spots things, you probably noticed that we didn’t enter the Interface (which happens to look exactly like the program name) anywhere in our code. So how come the code works when I just stated that we always need 4 things? Well.. it’s one of those bad designs. The reason it works is because the dbus library tries to “guess” which interface you want to use. In this case it was right. However many times it doesn’t work and you will run into problems that are really hard to debug. So a friendly advice is to ALWAYS give the 4 things mentioned. FOUR IS THE NEW FIVE.

In our case the fail-safe code would be:

import dbus
bus = dbus.SystemBus()
obj = bus.get_object('org.freedesktop.NetworkManager', '/org/freedesktop/NetworkManager')
iface = dbus.Interface(obj, 'org.freedesktop.NetworkManager')
iface.GetDevices()

Same output as before. But use this convention and you will never run into debugging nightmares. After all, explicit is better than implicit and that is especially true in this case.

Listing devices with udisks

Building on top of what we did with NetworkManager we continue but this time with UDisks2.

udisks2 in d-feet

As you might notice, udisks doesn’t have a straighforward method that we can call to list all devices. So someone has to dig through the things, read UDisks2 dbus API or simply google. Luckily for you, I did the digging.

udisks2 in d-feet

From the d-feet snapshot we can see directly the four things we need: the program domain (org.freedesktop.UDisks2), the interface (org.freedesktop.DBus.ObjectManager), the object path (/org/freedesktop/UDisks2) and lastly the method (GetManagedObjects()).

And the appropriate code is:

import dbus
bus = dbus.SystemBus()
obj = bus.get_object('org.freedesktop.UDisks2', '/org/freedesktop/UDisks2')
iface = dbus.Interface(obj, 'org.freedesktop.DBus.ObjectManager')
iface.GetManagedObjects()

You should get a blob of stuff including any mounted drives. Indeed you still have to dig through things but at least you shouldn’t be totally lost by now.

Getting properties/attributes

Building on the above, we will try to access some properties of a device. But first have a look at d-feet and pay attention to the fact that if you scroll down you will see the storage devices connected to your computer.

bunch of devices

Then we can also see that the device I choose has a bunch of properties.

bunch of properties

Now the logical thing to access these properties would be to do as we did earlier with the methods. Instead of methods we would just call the properties. And that my friend.. just won’t work! (Ugly Dbus, I hate you.)

So in order to access any property of any object we need to use a very specific interface: ‘org.freedesktp.DBus.Properties’.

So to not make this too tedious I will give the whole code again:

import dbus
bus = dbus.SystemBus()
obj = bus.get_object('org.freedesktop.UDisks2', '/org/freedesktop/UDisks2/drives/MBED_microcontrolleur_10105a42e87da33c103dccfb6bc235360a97')
iface = dbus.Interface(obj, 'org.freedesktop.DBus.Properties') # Here we use this 'magic' interface
iface.GetAll('org.freedesktop.UDisks2.Drive')

Notice that the object path ‘/org/freedesktop/UDisks2/drives/MBED_microcontrolleur_10105a42e87da33c103dccfb6bc235360a97’ has to be replaced with one that actually exists on your computer.

Then we open the correct interface and call GetAll(). The argument to GetAll() is the interface name that you want to get the properties of. As I told many times.. Dbus is so damn confusing! Anyway, the good thing is that the ‘org.freedesktp.DBus.Properties’ interface has only three methods we need to learn: Get(), Set(), GetAll(). So things can’t get more complicated than this.

If we want to access a very specific property we just use Get() instead of GetAll() and pass the interface name followed by the name of the property we want:

import dbus
bus = dbus.SystemBus()
obj = bus.get_object('org.freedesktop.UDisks2', '/org/freedesktop/UDisks2/drives/MBED_microcontrolleur_10105a42e87da33c103dccfb6bc235360a97')
iface = dbus.Interface(obj, 'org.freedesktp.DBus.Properties')
iface.Get('org.freedesktop.UDisks2.Drive', 'Id')     # Only difference

How about that USB insertion notification? (signals)

We saw how we can call methods but what about the interesting things like getting notified about an event like USB insertion? For this we can use signals.

All a signal is, is an incoming message from UDisks2 telling us that a mountable device has been inserted. From our part this signal will run a function in our code, a so called callback function. If you check in d-feet you will see many types of signals. The one that interests us is InterfacesAdded() from org.freedesktop.DBus.ObjectManager.

So how do we do this? I will get you straight the code for this one and do the explaining afterwards.

import dbus
from dbus.mainloop.glib import DBusGMainLoop
DBusGMainLoop(set_as_default=True)

bus = dbus.SystemBus()

# Function which will run when signal is received
def callback_function(*args):
    print('Received something .. ', args)

# Which signal to have an eye for
iface  = 'org.freedesktop.DBus.ObjectManager'
signal = 'InterfacesAdded'
bus.add_signal_receiver(callback_function, signal, iface)

# Let's start the loop
import gobject
loop = gobject.MainLoop()
loop.run()

If you run the above code, you will get printed things for every disconnection or connection of a storage device. Try plugging in your USB stick and see for yourself.

There are two things worth noticing in the code with signals:

  1. We import and use a bunch of weird things like gobject and DBusGMainLoop
  2. We don’t need the four things like earlier. We only need to know the interface and the signal (oh and both are just strings)

For our use the only thing we need from these libraries are the so called loops. The loops are needed since someone has to be looping somewhere waiting for the signal to arrive. We could probably create our own loop but I am not so sure how easy it would be. You could take that route if you want, may Zoidberg Jesus guide you the way. Below I mention a bit on what excactly Glib and Gobject is, just in case you’re curious and bored to google.

Glib
Glib is a library that originally was developed to be used with Gnome’s GTK. However later it split from it so that it could be used on any Linux machine. Many programs use the Glib library since it’s it provides so many things that is hard to find (in C) in a general-purpose library. Some things offered are data-structures like hash tables and linked lists (just keep in mind next time you will need any of this).

Gobject
Gobject is built on top of Glib. What it does is merely allow object oriented programming with the Glib. Since Glib is a C library, it doesn’t have object-oriented things lik objects, inheritance, classes etc. For that reason Gobject was developed. In our case, the reason we use it is to easily be able and change things on the loop without having to dig inside the C code.

Unblocking the loop

Now the loop above seems to work fine. There is a problem though. Your whole program blocks in this loop. Even if you add 1000 threads, they will all block for some weird reason and only the loop will run.

In order to stop the loop you have to:

  1. Add the code with the signal on a Python thread
  2. Initialize internally a thread for gobject (no clue what that does but won’t work if you don’t do it)

So in practice we just put the whole code above in a separate thread and then we make sure that gobject is calling threads.init() before we run() the loop.

And here is the code with the loop running on its own thread:

import threading, time

def start_listening():
    import dbus
    from dbus.mainloop.glib import DBusGMainLoop
    DBusGMainLoop(set_as_default=True)

    bus = dbus.SystemBus()

    # Function which will run when signal is received
    def callback_function(*args):
        print('Received something .. ', args)

    # Which signal to have an eye for
    iface  = 'org.freedesktop.DBus.ObjectManager'
    signal = 'InterfacesAdded'
    bus.add_signal_receiver(callback_function, signal, iface)

    # Let's start the loop
    import gobject
    gobject.threads_init()      # Without this, we will be stuck in the glib loop
    loop = gobject.MainLoop()
    loop.run()

# Our thread will run start_listening
thread=threading.Thread(target=start_listening)
thread.daemon=True              # This makes sure that CTRL+C works
thread.start()

# And our program will continue in this pointless loop
while True:
    time.sleep(1)
    print("tralala")

This is a good starting point for your program. Feel free to copy/paste it 🙂

Digging further

All this should give you an idea on how to get started. Now the main thing is probably that you have to filter out some signal events since InterfacesAdded() might run several times for a single unplug or plug-in of a device. Or depending on your case maybe you want to wait for a different signal. Using d-feet this should be rather easy to figure out.

The easiest way to do things from my experience is to use d-free and the Python interpreter directly to test things. However if you feel comfortable with ugly complicated interfaces then the UDisks2 dbus API might be of benefit for you.

References

http://dbus.freedesktop.org/doc/dbus-specification.html
https://www.kernel.org/doc/ols/2003/ols2003-pages-249-257.pdf
http://www.freedesktop.org/wiki/Software/dbus/
http://udisks.freedesktop.org/docs/latest/
http://en.wikipedia.org/wiki/Udev
http://en.wikipedia.org/wiki/D-Bus
http://dbus.freedesktop.org/doc/dbus-python/api/
http://doc.opensuse.org/products/draft/SLES/SLES-admin_sd_draft/cha.udev.html
http://free-electrons.com/doc/udev.pdf
http://dbus.freedesktop.org/doc/dbus-python/doc/tutorial.html


Leave a comment

Docker talking to docker

The problem

If you are mingling with docker and containers on servers, you’ll at some point want to have two containers communicate with each other.

For example if you have an Apache server, Django+Python and a MySQL server and you want to do tests on different configurations/versions of these, you should probably go the container-way: putting each of them on their own container. However since all these services are interchanged in some way (database has to communicate with web server, apache server has to communicate with django), the containers have to talk with each other somehow.

Possible solutions

There are essentially four ways to accomplish container-to-container communication:

  1. Having containers inside containers
  2. Using volumes (shared folders)
  3. Mapping ports (communication over TCP/IP)
  4. Linking containers (communication over.. TCP/IP?)

I will completely omit the first way since it’s a brainfuck both conceptually and technically (yes, I am aware of Dind). Been there, done that, and it wasn’t pretty.

The second solution is about sharing folders between host and container or between containers by using docker’s --volume flag. This is practical when you want to share files (source code, configuration files etc.) on a local machine. If you want containers on different hosts to communicate however this won’t suffice.

The two remaining solutions is the meat I will try to break down for you (and minimize your flatulence from the learning curve). I did my research and I will do my best to convey what I came down to. But enough talk, let’s start!

Terminal to terminal speaking

I will use netcat to show how all these things work. To remind you, netcat is a Unix tool similar to cat but supposed to work on the network (thus the name net-cat). Practically it’s used when you have to copy/paste from host to an other host, chat with someone fast, or just test communication between two hosts like we do here.

You can test it on your home computer. Open two terminals and type in one of them

netcat -l 8000

and on the other

netcat 127.0.0.1 8000

Whatever you type on one terminal now, will appear on the other one.

Terminal to container speaking

Let’s take this a step further, by containerizing the listening netcat. In a new terminal type:

docker run -i -n -p 9000:2000 ubuntu netcat -l 2000

that creates a new container with an internal port 2000. The port 9000 is the port on our system. Whatever arrives to that will automatically be forwarded to our container’s port 2000. This is somehow similar to port forwarding on home routers. (I explain more on how this actually works later.)

The flag -n is a deprecated flag for networking. In practice without the -n flag, everything will work as before but for some weird reason there will be an initial lag for the first netcat message. So if you don’t use it, just wait a bit longer.

text3782

We should now have a container running a “listening” netcat. Let’s open a terminal (on the same host) to try and talk with it:

netcat 127.0.0.1 9000

Or from a different host:

netcat <computer IP> 9000

Keep in mind that different containers on the same machine can have the exact same internal IP but not the same system one. So you can create a bunch of containers as long as you change the system’s IP:

docker run -d -p 9000:2000 ubuntu netcat -l 2000
docker run -d -p 9001:2000 ubuntu netcat -l 2000
docker run -d -p 9002:2000 ubuntu netcat -l 2000

It’s all legit baby!

Container to container speaking

We take this a step further by having the listening container as before but instead of opening a terminal to connect to it, we will create a new container on a different host.

text3783

So create a new container on the first host (same command as before)

docker run -i -p 9000:2000 ubuntu netcat -l 2000

Then we create a new container on the second machine (or same):

docker run -i -t ubuntu bash
root@879ad5d4251a:/#

On the new prompt that we get we can netcat directly to the first host:

root@879ad5d4251a:/# netcat 10.2.202.156 9000

This should work as a charm as long as you use your LAN IP.

There are some weird networking behaviours. For example if you netcat by using the IP of the container (eth0 inside container) there is an initial lag but after some seconds the message arrives. The same behaviour occurs if you omit the -n flag but use the LAN IP (which otherwise works). If I start a bash session in the container and listen to 9000 with netcat there, then it works flawlessly. Weird stuff indeed.

A whole network under the city

I omitted quite some information in an effort to jumpstart you. This information is however crucial if you want to use port forwarding and container communication on different hosts. There’s a concept that will save yourself some headaches:

There is a whole virtualized network when we use Docker.

Let me prove that to you. I create two containers and then I check the IPs of them and the IPs of the host. I do that by running ifconfig on each terminal.

docker_IPs_internal

Putting it all down sums up to the below picture.

docker_virtual_network

You see, Docker creates a virtual IP for each container we create. Furthermore on our host we have a virtual interface called docker0. That can be considered the router. Now on the same host we are able to communicate with any container if we know their IP.

Now, if a container is listening on a specific port we are able to communicate with it by knocking on its door with a pair of IP address and port number.

Let’s use this new knowledge on the previous example with netcat. Let’s assume that container 1 is listening on internal port 2000 and system port 9000 (like before):

docker run -i -p 9000:2000 ubuntu netcat -l 2000

We can connect to it in all these ways:

netcat 172.17.0.2 2000
netcat 127.0.0.1 9000
netcat localhost 9000
netcat 10.2.202.156 9000

The significant information here is that we can directly speak to the container if we know its IP. Pay attention also to that we need to use the correct port in that case.

Container to container speaking via links

So what are those so called links? Nothing special actually. It’s just some variables passed from one container to an other. In most cases these variables will hold things like IPs, ports, etc. Thus it’s just a step of simplifying things for us – in the end we will probably just use these variables to connect as before.

Let’s see how it works. Start by creating a first container that will act as a server:

docker run -i -t -p 8000:8000 --name myserver ubuntu bash

and then a second container that will act as a client:

docker run -i -t --name myclient --link myserver:myserver ubuntu bash

Nothing special huh? Well the magic happens if you inside the client (second container) run:

root@3b00d48e549a:/# ping $MYSERVER_PORT_8000_TCP_ADDR

You will successfully ping the other container. $MYSERVER_PORT_8000_TCP_ADDR is simply a variable set by docker on the newly created container, the one with the --link flag. You can see all variables with the command env. Depending on what flags you passed to the container, you will see appropriate things. In practice, if you don’t use the -p flag, you won’t see anything interesting (useful).

The --name flag is needed when we use linking as we need an identifier for the container. When we do the linking we need pass the name of the other container, the one that we want to link to and an alias for it. The alias is used merely as a prefix for the variable names. For example if we used --link myserver:dingdong instead of --link myserver:myserver then the variable above would be $DINGDONG_PORT_8000_TCP_ADDR.

That’s pretty much all to linking! By using these variables you can get the IPs and ports of other containers on the same host. That’s an important annotation. You can only link to a container running on the same hosts. For communicating to containers on distant hosts you’ll need to use port mapping as explained earlier.

Useful commands

Show which ports are opened by docker

sudo netstat -tulpn | grep docker


Leave a comment

Setting up a mail server

On this tutorial I will explain how to install and configure Postfix to be able and receive emails from the Internet and LAN. Running Postfix at home allows you to have an email that looks as you wish. If you have the website mywebsite.com then you can have emails like support@mywebsite.com, admin@mywebsite.com etc.

I set-up Postfix in a gradual way:

  1. First I set up a LAN mail server
  2. Then I configure the mail server to accept emails from the Internet
  3. Lastly I configure the mail server to allow outgoing emails to the Internet

Starting with a LAN mail server

I start by setting up a mail server that works for the LAN. My LAN network looks as below.

lan_postifx

ubuntux and laptop are the hostnames (names) of the computers that can be used to identify them in the LAN. For example if you want to ping ubuntux you don’t have to know its IP. You can just ping the computer by providing its hostname. Keep in mind that hostnames cannot be seen outside the LAN.

If you don’t know the hostname of your computer you can run in the terminal the command hostname. Throughout the tutorial I will be using ubuntux as my main computer where I install Postfix.

The router has an external IP 20.20.20.20 and an internal 192.168.1.1. Simply put, that means that from the internet our network is seen as 20.20.20.20 while the computers in the LAN sees the router having the IP 192.168.1.1.

Let’s install Postfix:

sudo apt-get install postfix

The install will ask you some questions. Just use the default values. After install, the daemon is launched automatically so we can start sending emails.

Sending the first email in the LAN

If you haven’t read my article on sending emails via telnet, you could check it out. In that article I describe how you can use telnet to send emails via a gmail mail server.

To send our emails we will be using telnet like in that article. The only difference is that while we had to authenticate with the gmail server with a login/password, here we won’t. The reason is that our mail server is in the LAN and by default configuration users in LAN are considered trusted.

The reason I use telnet and not an email client like Outlook Express is that I want you to see what is actually happening low-level and to get an in insight into the SMTP protocol.

First I send an email from ubuntux (192.168.1.2):

>telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 ubuntux.localdomain ESMTP Postfix (Ubuntu)
HELO whatever
250 ubuntux.localdomain
MAIL FROM: whatever
250 2.1.0 Ok
RCPT TO: animal@localhost
250 2.1.5 Ok
data
354 End data with .
This is a test
.
250 2.0.0 Ok: queued as 22AEA133C91
quit
221 2.0.0 Bye

In the telnet session I sent an email to animal@localhost. We assume that animal is the user I am logged in as at the moment. Then I can just type mail in the terminal and I will see the email arrived. Typing the number of the email I can read it. Typing ‘q’ will quit the application. All emails can be found in their raw format at /var/spool/mail.

Let’s try and send an email from the second machine (192.168.1.10):

>telnet 192.168.1.2 25
Trying 192.168.1.2...
Connected to 192.168.1.2.
220 ubuntux.localdomain ESMTP Postfix (Ubuntu)
HELO whatever
250 ubuntux.localdomain
MAIL FROM: whatever
250 2.1.0 Ok
RCPT TO: animal@ubuntux
250 2.1.5 Ok
data 
354 End data with .
Sending from my laptop.
.
250 2.0.0 Ok: queued as 68DF4133C91
quit
221 2.0.0 Bye

The only difference here is that we connected to ubuntux instead of localhost. Accordingly we had to change the recipient from animal@localhost to animal@ubuntux.

The recipient after “RCPT TO:” can essentially be any user (existing or not) as long as the domain used in the email is the hostname of our mail server (ubuntux in this case). An email with a different domain could be for example animal@randomdomain. The email with such a recipient would get totally discarded from the server and lost.

Taking it a step further – receiving emails from the Internet

At this state, emails can be delivered to the mail server from users in the LAN. We will try and make the mail server to be able and receive emails from the internet as well.

WAN_TO_LAN

In the illustration above an email is being sent from the gmail server to our own mail server. We don’t care how the email got to the gmail mail server. What we care about is how the email will reach our mail server. For the correct routing of the email we have to configure three things:

  1. Our DNS server
  2. Our router
  3. Our mail server

Configuring DNS

Every time we visit a website, a DNS request has to be made to find out the IP of that website. In the same fashion a DNS request is made by the mail server when an email is to be sent. This request is somehow different, in that it looks for a very specific record; an MX (mail exchange) record.

WAN_TO_LAN_dns

(1) As seen in the illustration the mail server looks at the domain after the ‘@’ symbol in the recipient of the email.

(2) It then sends a DNS request for the mail server in that domain (linuxmeerkat.com). To get an idea of what the answer of such a request can be you can use the debugging tool dig and test it with dig MX linuxmeerkat.com.

(3) A reply arrives from the DNS server with the IP of the mail server.

(4) The email is forwarded to that IP, and further to the mail server.

Now ideally the MX record in the DNS server would look like this:

linuxmeerkat.com.	IN	MX	10	20.20.20.20

That is however breaking the standard and therefore it is unacceptable. By defition an MX record should bind a domain to a hostname (or more). Therefore we need to replace the IP in the above example with a hostname:

linuxmeerkat.com.	IN	MX	10	mail.linuxmeerkat.com.
mail	IN	A	20.20.20.20

Since the mail.linuxmeerkat.com hostname (or FQDN to be exact) doesn’t exist, we have to create it with an A record. An A record binds a (made up) hostname to an IP by definition. Since our server is behind the router I use the IP of the router instead (the IP should never be a LAN one). For hostname I use mail but you could use whatever you wish. Just keep in mind that if you change it, the hostname must also be reflected in the MX record – you should also change the FQDN.

Keep in mind that if you do changes to the DNS records, it might take some time for the effects to take place. So continue a bit later, or after a day or two depending on how fast the changes are updated (essentially it depends on the TTL values).

Is your network reachable from outside?

Time to test if the DNS configuration works.

When we ping mail.linuxmeerkat.com we want that it pings 20.20.20.20. So let’s test it. Go to http://ping.eu/ping/ and ping your mail server’s FQDN (in this case mail.linuxmeerkat.com). It’s important that you ping from outside your LAN so that no local cache is used.

If you don’t get a response, try pinging directly your WAN IP (20.20.20.20). If you still don’t get a response, then probably your router is blocking ping from outside. If you are getting a response by pinging the IP but not when you ping the hostname, then you haven’t configured your DNS records correctly or they haven’t taken effect yet.

If you run your own DNS server you can check for warnings and errors in /var/log/syslog.

Configure your router

At this point we know that an email can arrive as far as to the router. We only need to configure the router so it forwards the email to the correct machine in the LAN.

Since we know that mail servers use port 25 and if we haven’t changed that, then our mail server also uses port 25, we just have to configure the NAT of our router.

Forward all TCP at port 25 to your mail server (in my case 192.168.1.2). My router’s port forwarding looks as below:

router

Sending an email to Postfix from gmail

If we can ping the mail server from the internet, then we can assume that our network is reachable. Sending an email to animal@mail.linuxmeerkat.com will not work still though as the router won’t know to which computer in the LAN to forward the email to. Setting the NAT of the router to forward all incoming TCP connections on port 25 to the mail server fixes this.

We can now test it by sending an email to animal@mail.linuxmeerkat.com from a gmail account. The mail server will reply with an error:

Delivery to the following recipient failed permanently:

     animal@mail.linuxmeerkat.com

Technical details of permanent failure:
Google tried to deliver your message, but it was rejected by the server for the recipient domain mail.linuxmeerkat.com by mail.linuxmeerkat.com. [20.20.20.20].

The reason the email gets rejected lies in the Postfix configuration. The configuration file /etc/postfix/main.cf has this (or a similar) line:


mydestination = ubuntux, localhost

mydestination is the variable that defines what FQDN/domains in emails are accepted. In essence all emails arrive at the server, it’s just the depending on this configuration some of these emailes will be discarded and some will be accepted by the server.

With the current configuration we can see that only emails of the format username@megistanas and username@localhost are accepted. Trying to send an email to animal@ubuntux from gmail won’t work as ubuntux just can’t be seen from outside our LAN; it doesn’t resolve to anything. For this reason we want to add one ore more domains that point towards our network. That way, when the an email arrives it doesn’t get rejected.

In my case I add linuxmeerkat.com so that I accept all emails with the format username@linuxmeerkat.com:

mydestination = linuxmeerkat.com, ubuntux, localhost

Restart Postfix and sending an email to animal@linuxmeerkat.com should now work.

You could essentially also add mail.linuxmeerkat.com as that would take the same route as well. We can add pretty much any domain that points to our router.

Sending an email to gmail or hotmail from Postfix

LAN_TO_WAN

There are essentially two entries in the configuration file that we have to change to make sending emails to the Internet possible:

smtpd_sender_restrictions = permit_mynetworks
mynetworks = 192.168.0.0/16 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128

smtpd_sender_restrictions can have more options than the one shown. However the only interesting option for us at the moment is to permit anyone from mynetworks to be able and send emails. So make sure permit_mynetworks is amongst the options.

mynetworks specifies local networks where users are considered trusted to use our mail server. For that we have added the 192.168.0.0/16 which includes all computers in the network. Be careful, you don’t want to have a WAN ip here as that would make our mail server an open relay (I talk more about that below).

This configuration will let anyone from our computer to send an email of the format username@linuxmeerkat.com. If we want users of the LAN to be able to use our mail server from the Internet, some sort of authentication has to take place (login or key), but I am will not go further into that.

Let’s test and see if we can send an email to a gmail account:

>telnet mail.linuxmeerkat.com 25
220 mail.linuxmeerkat.com ESMTP Postfix (Ubuntu)
HELO dickhead
250 mail.linuxmeerkat.com
MAIL FROM: animal@mail.linuxmeerkat.com
250 2.1.0 Ok
RCPT TO: animal@gmail.com
250 2.1.5 Ok
DATA
354 End data with <CR><LF>.<CR><LF>
This is a test.
.
250 2.0.0 Ok: queued as 3D31311F765
quit
221 2.0.0 Bye

If everything went as planned, you should have received an email in your gmail account (animal@gmail.com in the example).

Make sure your mail server is not an open relay

You always want to avoid having an open relay. An open relay constitutes that your mail server can be used by anyone on the internet including spammers.

open_relay

Once spammers find out that your server is an open relay, they will start using it. At some point, an other mail server will find out that you have an open relay and will blacklist your server.

So how do you ensure that you don’t have an open relay? Very simply. In your configuration file pay attention to these two lines:

smtpd_sender_restrictions = permit_mynetworks
mynetworks = 192.168.0.0/16 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128

As long as the networks in mynetworks are LAN IPs and not WAN IPs, your mail server is not an open relay as no-one outside the LAN can use your mail server to send emails.

You can test to see if your mail server is an open relay at http://www.mailradar.com/openrelay/.

Use a relay host if you can

Our mail server works fine at the moment but it’s very minimal. It doesn’t even filter incoming spam emails. Configuring filters etc. can be time consuming and will need further maintenance all the time. An easy solution to this is to forward our emails to a second mail server, a relay server (keep in mind: not same as an open relay, as a relay server is not open to everyone). So if we have to deliver an email to user11@gmail.com, instead of delivering to the gmail server directly, we can use a second mail server which we have access to. That server can then do the filtering etc. for us and deliver the email to its final destination.

My ISP is called Bahnhof and by googling around I found that it offers a relay at mail2.bahnhof.se. Most ISPs do offer mail servers that can be used as relays. Luckily for me, I don’t even need to enter a password and username as I am authenticated by my IP or MAC (depends on how you’re connected to the ISP). In your case you might need to authenticate first before using the relay. I won’t go into how you do that. I will however tell you how you can test to see if you can use the relay as a relay server.

If you can send an email directly from the relay server then you can use it as your relay. See the telnet session below:

telnet mail2.bahnhof.se 25
Connected to mail2.bahnhof.se.
Escape character is '^]'.
220 mxf3.bahnhof.se ESMTP
HELO dickhead
250 mxf3.bahnhof.se
MAIL FROM: animal@linuxmeerkat.com
250 Ok
RCPT TO: animal@gmail.com  
250 Ok
DATA
354 End data with <CR><LF>.<CR><LF>
This is a test.
.
250 Ok: queued as 0C67C53CEB7
QUIT
221 Bye

I just connected to my ISP’s mail server and sent an email to a gmail account. Since I got the email delivered and wasn’t asked for a login/password, that means that I can directly use the server as a relay without authentication.

If your ISP’s mail server requires a username and password you’ll have to do some extra work in configuring that. Also keep in mind that in the field “MAIL FROM:” you have to provide an email with a hostname that points to your network. The server will probably check to see if whatever@hostname actually resolves to your network. If not, you will get an error like below:

450 <spamotron@spammer.spam>: Sender address rejected: Domain not found

Now that we know we can use the server as a relay, we simply have to edit /etc/postfix/main.cf and make sure the relayhost is set correctly:

relayhost = mail2.bahnhof.se:25

Restart Postfix and you should be able to use your mail server as before with the difference that now all outgoing emails go first through the relay server and the relay server forwards them further to their destinations.


13 Comments

Emailing from a gmail acount via Telnet

Telnet is a small amazing tool that can be used for pretty much everything concerning networks. Telnet sends ASCII strings directly to a host. So as long as a protocol supports ASCII messages, we can communicate with a host with any protocol we want. Some examples are SMTP and HTTP.

In practise this method is being used to test things. If you are developing a web-server for example it would be easy if you could test your web-server directly without having to launch a browser and type a command. It’s a blessing for automation.

Here we will try to send an email by using the SMTP protocol. We will use telnet for that, by sending and receiving raw messages with the SMTP server.

Preparation

First we need to install telnet with SSL support. All SMTP servers use SSL encryption nowadays so using simply telnet will not work. The telnet shipping with Ubuntu does not support SSL so we need to upgrade it:
sudo apt-get install telnet-ssl

To run the new telnet-ssl we just issue telnet in the terminal as before. The difference is that telnet now supports some new flags, namely -z, which is used for SSL.

We are also going to need http://www.base64encode.org to encode our username and password to base 64 format. Keep in mind that base 64 does not encrypt a message. It just changes the way the information is stored. In practise we use base 64 to be able and send any kind of data as simple characters. The real encryption is under the hood however. Check Extras for more info.

Steps

  1. Start telnet
  2. telnet -z ssl smtp.gmail.com 465
    The flag -z ssl tells telnet to use SSL over the connection. smtp.gmail.com is Gmail’s server domain and 465 is the port used by the server.

  3. Handshake with server
  4. HELO yo
    The text after the HELO is supposedly your domain or literal address. Essentially you can enter whatever you want. Pressing enter, the exact message arrives at the mail server, which in response should reply with
    250 mx.google.com at your service

  5. Login on gmail
  6. AUTH LOGIN
    This message is self-explanatory. After the server receives this message, it’s going to ask for our username. The response looks like this
    334 VXNlcm5hbWU6 and indicates that the username should be entered.

  7. Username
  8. Go to http://www.base64encode.org and encode your username (username@gmail.com) to 64 base. Copy and paste the encoded string in your telnet session and hit enter. If the username is accepted, you get back the message bellow.
    334 UGFzc3dvcmQ6
    This message indicates that the password should be entered.

  9. Password
  10. Do the same as above but instead of your username, encode your password. Then paste the password and hit enter. If the username/password combination is correct, you are greeted with the message
    235 2.7.0 Accepted

  11. Email sender
  12. First we are going to add the sender of the email:
    MAIL FROM: <username@gmail.com>
    For gmail it doesn’t really matter who you set as sender. We logged in our gmail account when we sent the AUTH LOGIN message to the server. Gmail has as a mechanism to set the sender automatically to your email address.
    The response back is
    250 2.1.0 OK js17sm40481494lab.5 - gsmtp

  13. Email recipient
  14. Now we will add the recipent of the email. Here we need to provide a valid email or else the email will not go any further. If the email doesn’t exist, we will probably get it back into our inbox which is the default behaviour for undelivered emails.
    RCPT TO: <username@gmail.com>
    On success we get
    250 2.1.5 OK js17sm40481494lab.5 - gsmtp

  15. Email body
  16. Now we want to start typing the email. We send the following command to the server so that the server knows that what follows is the actual email.
    DATA
    The response from the server is
    354 Go ahead js17sm40481494lab.5 - gsmtp

    Now we need to set the subject(if we want).
    Subject: test
    and everything else following is considered the text of the email

    This is a line in the email.
    This is a second line in the email.
    .

    The dot on its own line in the end is to signal the end of the email. After that, a response from the server should come:
    250 2.0.0 OK 1381416452 js17sm40481494lab.5 - gsmtp
    Now the email has been sent.

  17. Close telnet
  18. QUIT
    This is an actual command to the telnet client and not a message sent to the mail server.

Extras

EHLO vs HELO

EHLO can be used instead of HELO. The difference is that EHLO lists also all the commands that the server supports. EHLO is newer than HELO and is suggested to be used instead of EHLO. However HELO is always going to be supported, so in reality there is no big difference as long as the server supports the message.

SSL and encryption

Something important, as mentioned in the beginning of this page, is that encoding data to base 64 is not encryption but rather a different representation of the same data. It’s SSL which adds the encryption to our communication and you can be assured that everything is encrypted between the client and the server.

I captured the whole procedure of sending an email, with Wireshark to prove my point.
Screenshot from 2013-10-10 17:33:52_crop

The packets are in the order they were sent and received. We can see that telnet first contacts the DNS server to get the domain name of the mail server. 192.168.1.1 is my router and 192.168.1.2 is my computer from where I use telnet.

After we get the domain we do a first handshake with the mail server. It’s those three TCP messages with SYN/ACK in them. After that we initiate a TLS session. TLS and SSL are pretty much the same thing. They are just different versions for encrypted communication, with TLS being the newer one.

Everything we type and read from the time we start telnet is encrypted. The reason we see the messages clearly in the terminal is that they get decrypted by telnet on their arrival. When messages leave telnet, they get encrypted to travel on the wire. That’s why with Wireshark we only see the decrypted data, because we are looking at what is passing through the wire. I circled the TLS messages to make it more apparent that the whole session is encrypted by using TLS. The few TCP packets seen here and there, are packets with encrypted data. If you could check what these packets hold inside(with Wireshark), you would only see gibberish.

If instead of Gmail we were using an SMTP server that doesn’t use SSL, we would be able to intercept all the packets with their raw data, and thus the actual messages, passwords, usernames, etc.