LinuxMeerkat

I swear! Meerkats can do Linux


18 Comments

Running a GUI application in a Docker container

This guide will show you how to run a GUI application headless in a Docker container and even more specific scenarios involving running Firefox and Chrome. If you are not interested about those then you can just stop in the middle of this tutorial.

What the hell is X?

X is a program that sits on a Linux machine with a monitor (so servers usually don’t use X). X’s job is to talk to the Linux kernel in behalf of GUI programs. So if you are playing a game for example, the game (that is, the application) is constantly sending drawing commands to the X server like “draw me a rectangle here”. X forwards all this to the Kernel which will further forward the information to the GPU to render it on the monitor.

X can even receive commands from the keyboard or mouse. When you click to shoot on your game for example, the command “click at 466,333” is sent from your mouse to the kernel, from the kernel to the X and from X to the game. That way the game can have a clue on what is happening!

You will often hear X being called a server and the reason for that is simply because the way the applications send commands to it is through sockets. For that reason the applications are also referred to as clients many times.

If you are reading this then the X is running on your PC. Let’s prove it:

> ps aux | grep X
root      1436  3.2  0.7 687868 94444 tty7     Ssl+ 09:47   2:50 /usr/bin/X -core :0 -seat seat0 -auth /var/run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch

We can see that X is running as root and has PID 1436. An other important thing is to notice the :0 which is called display in X jargon. A display is essentially:

  • A monitor
  • A mouse
  • A keyboard

And this is the bigger picture of how it all looks together:

Now there is a variable in Linux that is used whenever we run a GUI program. That variable is surprisingly called DISPLAY. The syntax of the DISPLAY variable is

<hostname>:<display>.<monitor>

. Let’s check the DISPLAY on our computer:

> echo $DISPLAY
:0

I get :0, which means we use display 0. Notice however that this says nothing about which monitor we use. This makes sense since if you are running 2 or more monitors on your Linux you still have the same environment variables in both of them. It wouldn’t make sense that an environment variable changes just because you echo it from a different screen, would it? For that reason we get the display and not the monitor so that we get the same output on both. As about the hostname, since there is no info about it, the local host is assumed.

On a notice, if you have multiple monitors you can still specify which monitor to run an application by simply typing the full display variable you want. So if you have a monitor 0 and a monitor 1 on the current display, I can run firefox on monitor 1 with:

DISPLAY=:0.1 firefox

Creating a virtual monitor

Instead of running X, we can run a different version of it that can create virtual displays. Xvfb (virtual framebuffer – whatever the hell that means) will create a virtual monitor for us.

So let’s make a new monitor (I assume you have installed xvfb):

Xvfb :1 -screen 0 1024x768x16

This will start the Xvfb server with a display 1 and a virtual screen(monitor) 0. We can access this by simply typing DISPLAY=:1.0 before running our graphical program. In this case the program will start in the virtual screen instead of our monitor.

Let’s make sure that the screen is still running:

> ps aux | grep X
root      1436  3.1  0.7 684580 91144 tty7     Ssl+ 09:47   3:31 /usr/bin/X -core :0 -seat seat0 -auth /var/run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch
manos    22018  0.0  0.1 164960 20756 pts/27   Sl+  11:37   0:00 Xvfb :1 -screen 0 1024x768x16

We see we have the normal display 0. (A way to tell it is the default screen is to see that it runs as root.) We can also see the second display :1 and screen 0 with resolution 1024×768. So what if we want to use it?

Open a new terminal and type:

> DISPLAY=:1.0 firefox
..

This will start firefox at the given display. The reason I use the DISPLAY at the same line is to make sure that the subprocess inherits the variable DISPLAY. An other way to do this is to type:

> DISPLAY=:1.0
> export DISPLAY
> firefox
..

Run a GUI program in a Docker container

We will now create a virtual screen inside a docker container.

> docker run -it ubuntu bash
root@660ddd5cc806:/# apt-get update
root@660ddd5cc806:/# apt-get install xvfb
root@660ddd5cc806:/# Xvfb :1 -screen 0 1024x768x16 &> xvfb.log  &
root@660ddd5cc806:/# ps aux | grep X
root        11  0.0  0.1 169356 20676 ?        Sl   10:49   0:00 Xvfb :1 -screen 0 1024x768x16

So now we are sure that we are running the virtual screen. Let’s access it and run something graphical on it. In this case I will run Firefox and Python+Selenium just as a proof of concept of what is happening.

First I put my display variable and use export to assure that any sub-shells or sub-processes use the same display (with export, they inherit the variable DISPLAY!):

root@660ddd5cc806:/# DISPLAY=:1.0
root@660ddd5cc806:/# export DISPLAY

Now we can simply run a browser

root@660ddd5cc806:/# firefox
(process:14967): GLib-CRITICAL **: g_slice_set_config: assertion 'sys_page_size == 0' failed
Xlib:  extension "RANDR" missing on display ":99.0".
(firefox:14967): GConf-WARNING **: Client failed to connect to the D-BUS daemon:
//bin/dbus-launch terminated abnormally without any error message
..

The errors don’t mean anything. But we can’t be sure, can we? I mean, since we can’t see what’s happening it’s really hard to tell. There are two things we can do, either use ImageMagick to take a snapshot and send it to our host via a socket or we can simply use Selenium. I will do that since most people probably want to achieve all this for testing purposes anyway.

root@0e395f0ef30a:/# apt-get install python-pip
root@0e395f0ef30a:/# pip install selenium
root@0e395f0ef30a:/# python
>>> from selenium import webdriver
>>> browser=webdriver.Firefox()
>>> browser.get("http://www.google.com")
>>> browser.page_source

If you get a bunch of HTML, then we have succeeded!

The Chrome issue

If you try and run Chrome in a Docker container, it won’t work even if you have setup everything correctly. The reason is that Chrome uses something called sandboxing. Reading this I could not let but notice the word jail. Apparently it seems that Chrome uses Linux containers (the same that Docker uses). For this reason you have to put a bit of extra effort to solve this issue since because of technical difficulties it’s not possible to run containers in containers.

There are two workarounds:

  1. Use my docker-enter
  2. Use –privileged when running the container

The second solution is probably the best one. However while testing things, there’s nothing wrong with the first one.

So to make things work (notice I run everything from the start):

> docker run -it --privileged ubuntu bash
root@7dd2c07cb8cb:/# apt-get update
root@7dd2c07cb8cb:/# apt-get install wget python-pip xvfb
root@7dd2c07cb8cb:/# wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
OK
root@7dd2c07cb8cb:/# echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list
root@7dd2c07cb8cb:/# apt-get update
root@7dd2c07cb8cb:/# apt-get install google-chrome-stable
root@7dd2c07cb8cb:/# pip install selenium

I have now installed Selenium, Chrome and Xvfb. Now I am going to make make a virtual monitor and run Chrome:

root@7dd2c07cb8cb:/# Xvfb :99 -screen 0 1024x768x16 &> xvfb.log &
[1] 6729
root@7dd2c07cb8cb:/# DISPLAY=:99.0
root@7dd2c07cb8cb:/# export DISPLAY
root@7dd2c07cb8cb:/# google-chrome
Xlib:  extension "RANDR" missing on display ":99.0".
Xlib:  extension "RANDR" missing on display ":99.0".
[6736:6736:1017/143449:ERROR:desktop_window_tree_host_x11.cc(802)] Not implemented reached in virtual void views::DesktopWindowTreeHostX11::InitModalType(ui::ModalType)
ATTENTION: default value of option force_s3tc_enable overridden by environment.
failed to create drawable
[6775:6775:1017/143449:ERROR:gl_surface_glx.cc(633)] glXCreatePbuffer failed.
[6775:6775:1017/143449:ERROR:gpu_info_collector.cc(27)] gfx::GLContext::CreateOffscreenGLSurface failed
[6775:6775:1017/143449:ERROR:gpu_info_collector.cc(89)] Could not create surface for info collection.
[6775:6775:1017/143449:ERROR:gpu_main.cc(402)] gpu::CollectGraphicsInfo failed (fatal).
[6775:6775:1017/143449:ERROR:sandbox_linux.cc(305)] InitializeSandbox() called with multiple threads in process gpu-process
[6775:6775:1017/143449:ERROR:gpu_child_thread.cc(143)] Exiting GPU process due to errors during initialization
[6736:6736:1017/143449:ERROR:gpu_process_transport_factory.cc(418)] Failed to establish GPU channel.

It seems that it works. It’s normal that we get the gpu errors since we don’t have a gpu! However I don’t like gambling so we will take it a step further to check that the browser actually works. However for this I will need to download the webdriver for Google Chrome..

root@7dd2c07cb8cb:/# apt-get install curl unzip
root@7dd2c07cb8cb:/# cpu_arch=$(lscpu | grep Architecture | sed "s/^.*_//")
root@7dd2c07cb8cb:/# version=$(curl 'http://chromedriver.storage.googleapis.com/LATEST_RELEASE' 2> /dev/null)
root@7dd2c07cb8cb:/# url_file="chromedriver_linux${cpu_arch}.zip"
root@7dd2c07cb8cb:/# url_base="http://chromedriver.storage.googleapis.com"
root@7dd2c07cb8cb:/# url="${url_base}/${version}/${url_file}"
root@7dd2c07cb8cb:/# wget "$url"
root@7dd2c07cb8cb:/# unzip chromedriver_*.zip -d tmp
root@7dd2c07cb8cb:/# mv tmp/chromedriver usr/bin/

Now (FINALLY!) we can test with Selenium:

root@7dd2c07cb8cb:/# python
>>> from selenium import webdriver
>>> browser=webdriver.Chrome()
>>> browser.get("http://en.wikipedia.org/wiki/Open_source")
>>> browser.page_source

You should get a bunch of HTML code. So there we go!

Common errors

.. Gtk: cannot open display:

DISPLAY has wrong value or you forgot to export it!

References

http://www.x.org/wiki/Development/Documentation/HowVideoCardsWork/
http://www.x.org/archive/X11R7.7/doc/man/man1/Xvfb.1.xhtml
http://blog.mecheye.net/2012/06/the-linux-graphics-stack/
http://people.freedesktop.org/~marcheu/linuxgraphicsdrivers.pdf
http://www.tldp.org/HOWTO/Framebuffer-HOWTO/
http://en.wikipedia.org/wiki/X_Window_System
http://en.wikipedia.org/wiki/Framebuffer
http://en.wikipedia.org/wiki/X.Org_Server
http://en.wikipedia.org/wiki/Display_server
http://www.google.com/googlebooks/chrome/med_26.html
http://linux.die.net/man/1/xvfb
https://www.freebsd.org/doc/handbook/x-understanding.html


Leave a comment

Docker talking to docker

The problem

If you are mingling with docker and containers on servers, you’ll at some point want to have two containers communicate with each other.

For example if you have an Apache server, Django+Python and a MySQL server and you want to do tests on different configurations/versions of these, you should probably go the container-way: putting each of them on their own container. However since all these services are interchanged in some way (database has to communicate with web server, apache server has to communicate with django), the containers have to talk with each other somehow.

Possible solutions

There are essentially four ways to accomplish container-to-container communication:

  1. Having containers inside containers
  2. Using volumes (shared folders)
  3. Mapping ports (communication over TCP/IP)
  4. Linking containers (communication over.. TCP/IP?)

I will completely omit the first way since it’s a brainfuck both conceptually and technically (yes, I am aware of Dind). Been there, done that, and it wasn’t pretty.

The second solution is about sharing folders between host and container or between containers by using docker’s --volume flag. This is practical when you want to share files (source code, configuration files etc.) on a local machine. If you want containers on different hosts to communicate however this won’t suffice.

The two remaining solutions is the meat I will try to break down for you (and minimize your flatulence from the learning curve). I did my research and I will do my best to convey what I came down to. But enough talk, let’s start!

Terminal to terminal speaking

I will use netcat to show how all these things work. To remind you, netcat is a Unix tool similar to cat but supposed to work on the network (thus the name net-cat). Practically it’s used when you have to copy/paste from host to an other host, chat with someone fast, or just test communication between two hosts like we do here.

You can test it on your home computer. Open two terminals and type in one of them

netcat -l 8000

and on the other

netcat 127.0.0.1 8000

Whatever you type on one terminal now, will appear on the other one.

Terminal to container speaking

Let’s take this a step further, by containerizing the listening netcat. In a new terminal type:

docker run -i -n -p 9000:2000 ubuntu netcat -l 2000

that creates a new container with an internal port 2000. The port 9000 is the port on our system. Whatever arrives to that will automatically be forwarded to our container’s port 2000. This is somehow similar to port forwarding on home routers. (I explain more on how this actually works later.)

The flag -n is a deprecated flag for networking. In practice without the -n flag, everything will work as before but for some weird reason there will be an initial lag for the first netcat message. So if you don’t use it, just wait a bit longer.

text3782

We should now have a container running a “listening” netcat. Let’s open a terminal (on the same host) to try and talk with it:

netcat 127.0.0.1 9000

Or from a different host:

netcat <computer IP> 9000

Keep in mind that different containers on the same machine can have the exact same internal IP but not the same system one. So you can create a bunch of containers as long as you change the system’s IP:

docker run -d -p 9000:2000 ubuntu netcat -l 2000
docker run -d -p 9001:2000 ubuntu netcat -l 2000
docker run -d -p 9002:2000 ubuntu netcat -l 2000

It’s all legit baby!

Container to container speaking

We take this a step further by having the listening container as before but instead of opening a terminal to connect to it, we will create a new container on a different host.

text3783

So create a new container on the first host (same command as before)

docker run -i -p 9000:2000 ubuntu netcat -l 2000

Then we create a new container on the second machine (or same):

docker run -i -t ubuntu bash
root@879ad5d4251a:/#

On the new prompt that we get we can netcat directly to the first host:

root@879ad5d4251a:/# netcat 10.2.202.156 9000

This should work as a charm as long as you use your LAN IP.

There are some weird networking behaviours. For example if you netcat by using the IP of the container (eth0 inside container) there is an initial lag but after some seconds the message arrives. The same behaviour occurs if you omit the -n flag but use the LAN IP (which otherwise works). If I start a bash session in the container and listen to 9000 with netcat there, then it works flawlessly. Weird stuff indeed.

A whole network under the city

I omitted quite some information in an effort to jumpstart you. This information is however crucial if you want to use port forwarding and container communication on different hosts. There’s a concept that will save yourself some headaches:

There is a whole virtualized network when we use Docker.

Let me prove that to you. I create two containers and then I check the IPs of them and the IPs of the host. I do that by running ifconfig on each terminal.

docker_IPs_internal

Putting it all down sums up to the below picture.

docker_virtual_network

You see, Docker creates a virtual IP for each container we create. Furthermore on our host we have a virtual interface called docker0. That can be considered the router. Now on the same host we are able to communicate with any container if we know their IP.

Now, if a container is listening on a specific port we are able to communicate with it by knocking on its door with a pair of IP address and port number.

Let’s use this new knowledge on the previous example with netcat. Let’s assume that container 1 is listening on internal port 2000 and system port 9000 (like before):

docker run -i -p 9000:2000 ubuntu netcat -l 2000

We can connect to it in all these ways:

netcat 172.17.0.2 2000
netcat 127.0.0.1 9000
netcat localhost 9000
netcat 10.2.202.156 9000

The significant information here is that we can directly speak to the container if we know its IP. Pay attention also to that we need to use the correct port in that case.

Container to container speaking via links

So what are those so called links? Nothing special actually. It’s just some variables passed from one container to an other. In most cases these variables will hold things like IPs, ports, etc. Thus it’s just a step of simplifying things for us – in the end we will probably just use these variables to connect as before.

Let’s see how it works. Start by creating a first container that will act as a server:

docker run -i -t -p 8000:8000 --name myserver ubuntu bash

and then a second container that will act as a client:

docker run -i -t --name myclient --link myserver:myserver ubuntu bash

Nothing special huh? Well the magic happens if you inside the client (second container) run:

root@3b00d48e549a:/# ping $MYSERVER_PORT_8000_TCP_ADDR

You will successfully ping the other container. $MYSERVER_PORT_8000_TCP_ADDR is simply a variable set by docker on the newly created container, the one with the --link flag. You can see all variables with the command env. Depending on what flags you passed to the container, you will see appropriate things. In practice, if you don’t use the -p flag, you won’t see anything interesting (useful).

The --name flag is needed when we use linking as we need an identifier for the container. When we do the linking we need pass the name of the other container, the one that we want to link to and an alias for it. The alias is used merely as a prefix for the variable names. For example if we used --link myserver:dingdong instead of --link myserver:myserver then the variable above would be $DINGDONG_PORT_8000_TCP_ADDR.

That’s pretty much all to linking! By using these variables you can get the IPs and ports of other containers on the same host. That’s an important annotation. You can only link to a container running on the same hosts. For communicating to containers on distant hosts you’ll need to use port mapping as explained earlier.

Useful commands

Show which ports are opened by docker

sudo netstat -tulpn | grep docker


1 Comment

Docker in a development enviroment

Intro

A few days ago I wrote a tutorial on how to setup docker and use it between different machines. Now that was a nice first insight on how to jump-start using docker. It was also a nice way to showcase the possibilities and limitations of Docker.

In this post I will give some practical information on how to use docker as a developer.

Setup

To use Docker for development of software we want mainly three things:

  • Have our source code on the host machine. That way we can use GUI editors and whatever tools we want from outside the container.
  • Be able to have multiple terminals to the same container. This is good for debugging
  • Setup a docker image which we will use for running our program. I will use Django and Python for that.

And for the visual brains out there:
container_host_communication
As you see in the pic, I am using Ubuntu as my host machine. At the same machine I have a folder with the source code and two terminals. Then I run a container with OpenSUSE. The folder and terminals reside ont he host machine but they communicate directly with the container. I will describe below how to achieve all this.

Multiple terminals

The easiest way to have multiple terminas is to use a small tool called nsenter. The guide can be found at https://github.com/jpetazzo/nsenter but it sums up to running this one-liner from any folder:

> docker run --rm -v /usr/local/bin:/target jpetazzo/nsenter

That installs nsenter on the host machine. After that, we can use it directly. So let’s try it. Open bash in a container with ubuntu as our basic image:

> docker run -t -i ubuntu /bin/bash
root@04fe75de21d4:/# touch TESTFILE
root@04fe75de21d4:/# ls
TESTFILE  boot	etc   lib    media  opt   root	sbin  sys  usr
bin	  dev	home  lib64  mnt    proc  run	srv   tmp  var

In the terminal above, I created a file called TESTFILE. We will try to open a second terminal and check to see the file from it.

To use xsenter we need the process ID of the container. Unfortunately we can’t use ps aux but rather have to use docker’s inspect command. I open a new terminal and type the below

> PID=$(docker inspect --format {{.State.Pid}} 04fe75de21d4)
> sudo nsenter --target $PID --mount --uts --ipc --net --pid

The string 04fe75de21d4 I gave is the ID of my container. If everything went ok, your terminal prompt will change to the same ID:

root@04fe75de21d4:/# ls
TESTFILE  bin  boot  dev  etc  home  lib  lib64  media	mnt  opt  proc	root  run  sbin  srv  sys  tmp	usr  var

See the TESTFILE there? Congrats! Now we have a second terminal to the exact same container!

Share a folder between host and container

Now I want to have a folder on my host computer and be able and access it through a container.

Luckily for us there is a built-in way to do that. We just have to specify a flag -v to docker. First let’s make a folder though that will be mounted:

> mkdir /home/manos/myproject

Let’s now mount it into the container:

> sudo docker run -i -t -v /home/manos/myproject:/home/myproject ubuntu /bin/bash
> root@7fe33a71ac2f:/#

If I now create a file inside /home/manos/myproject the change will be reflected from inside the container and vice versa. Play a bit with it by creating and deleting files from either the host or from inside the container to see for yourself.

Create a user in the container

It is wise to have a normal user in your image. If you don’t then you should create one and save the image. That way the source files can be opened from a normal user on your host – you won’t need to launch your IDE with root privileges.

> adduser manos
..

Follow the instructions and then commit your image. That way whenever you load the image again, you will be having user manos. To change to user manos just type

> su manos

All files you create now, will be accesible by a normal user at the host machine. Something else you could do is to somehow

Real life scenario: Python, Django and virtualenv

I wanted to learn Django. Installing Django is commonly made with the package manager pip, but pip has a bad history of breaking up things since it doesn’t communicate with Debian’s apt. So at some point if you installed/uninstalled python stuff with apt, pip wouldn’t know about it and vice versa. In the end you would end up with a broken python environment. That’s why a tool called virtualenv is being used – a tool that provides isolation. Since we have docker though which also provides isolation we can simply use that.

So what I really want:

  1. Have the source code on my host.
  2. Run django and python inside a container.
  3. Debug from at least two terminals.

Visually my setup looks as something like this:

docker_dev_setup_labels_900x500

I assume you have an image with django and python installed. Let’s call the image py3django.

Firstly create the folder where you want your project source code to be. This is the folder that we will mount. My project resides in /home/manos/django_projects/myblog for example.

Once it’s created I just run bash on the image py3django. This will be my primary terminal (terminal 1):

> sudo docker run -i -t -p 8000:8000 -v /home/manos/django_projects/myblog:/home/myblog py3django /bin/bash
root@2fe3611c1ec2:/home#

The flag -p makes sure that docker doesn’t choose a random port for us. Since we run Django we will want to run a web server with a fixed port (on 8000). The flag -v mounts our host folder /home/manos/django_projects/myblog to the container’s folder /home/myblog. py3django is the image I have.

Now we have a folder where we can put our source code and a working terminal to play with. I want though a second terminal (terminal 2) to run my python webserver. So I open a second terminal and type:

> sudo nsenter --target $(docker inspect --format {{.State.Pid}} 2fe3611c1ec2) --mount --uts --ipc --net --pid
> root@2fe3611c1ec2:/#

Mind that I had to put the appropriate container ID in the command above.

Now all this is very nice but admittedly it’s very complex and it will be impossible to remember all these commands and boring to type them each single day. Therefore I suggest you create a BASH script that initiates the whole thing.

For me it took a whole day to come up with the script below:

#! /bin/bash

django_project_path="/home/manos/django_projects/netmag" # Path to project on host
image="pithikos/py3django_netmag_rmved"                  # Image to run containers on

echo "-------------------------------------------------"
echo "Project:  $django_project_path"
echo "Image  :  $image"


# 1. Start the container in a second terminal
proj_name=`basename $django_project_path`
old_container=`docker ps -n=1 -q`
export docker_line="docker run -i -t -p 8000:8000 -v $django_project_path:/home/$proj_name $image /bin/bash"
export return_code_file="$proj_name"_temp
rm -f "$return_code_file"
gnome-terminal -x bash -c '$docker_line; echo $? > $return_code_file'
sleep 1
if [ -f "$return_code_file" ] && [ 0 != "$(cat $return_code_file)" ]
then
	echo
	echo "--> ERROR: Could not load new container."
	echo "    Stop any other instances of this container"
	echo "    if they are running and try again."
	echo 
	echo "    To reproduce the error, run the below:"
	echo "    $docker_line"
	echo
	rm -f "$return_code_file"
	exit 1
fi
rm -f "$return_code_file"


# 2. Connect to the new container
while [ "$old_container" == "`docker ps -n=1 -q`" ]; do
	sleep 0.2
done
container_ID=`docker ps -n=1 -q`
sudo nsenter --target $(docker inspect --format {{.State.Pid}} $container_ID) --mount --uts --ipc --net --pid

This script starts a container on a second terminal and then connects to the container from the current terminal. If starting the container fails, an appropriate message is given. django_project_path is the full path to the folder on the host with the source code. The variable image holds the name of the image to be used.

You can combine this with devilspie, an other nice tool that automates the position and size of windows when they’re launched.

In case you wonder about the top window with all the containers, that’s simply a watch command, a tool that updates regularly a command. In my case I use watch with docker ps. Simple stuff:

> watch docker ps

I use this because I personally like having an overview on the running containers. That way I don’t end up with trillions of forgotten containers that eat up my system.

Now that you have everything setup you can also run django server from one of the two terminals or whatever else you might want.


1 Comment

Docker tutorial

So as an intern in a big company I was given the task to get comfortable with docker. The problem is that docker is quite fresh so there isn’t really that much of good tutorials out there. After reading a bunch of articles and sparse tutorials (even taken the official tutorial at https://www.docker.com/tryit), I still straggled to get a firm grip on what docker even is supposed to be used for. Therefore I decided to make this tutorial for a total beginner like me.

Docker vs VirtualBox

There are many different explanations on the internet about what docker is and when to use it. Most of them however tend to complicate things more than giving some practical information for a total beginner.

Docker simply put is a replacement for virtual machines. I will use virtual box as a comparison example since it’s very easy for anyone to download and try to see the differences themselves.

The application VirtualBox is essentially a virtual machine manager.
text3806

Each and every of the OSes you see in the picture above, is an installed virtual machines. Each such machine has it’s own installed OS, kernel, virtual devices like hard disks, network cards etc. All this takes a considerate amount of memory and needs extra processing power. All virtual machines (VM) like VMware, Parallels, behave the same.

text4037

Now imagine that we want to use nmap from an OpenSUSE machine but we are on an Ubuntu. Using VirtualBox we would have to install the whole OS and then run it as a virtual machine. The memory consumption is humongous for such a trivial task.

In contrast to VirtualBox or any other virtual machine, Docker doesn’t install the whole OS. Instead it uses the kernel and hardware of our primary computer (the host). This makes Docker to virtualize super fast and consume only a fraction of the memory we would else need. See the benefits? Imagine if we wanted to run 4 different programs on 4 different OSes. That would take at a minimum 2GB of RAM.

But why would you want to run nmap on openSUSE instead of the host computer? Well this was just a silly example. There are other examples that prove the importance of a tool like Docker. Imagine that you’re a developer and you want to test your program on 10 different distributions for example. Or maybe you are the server administrator on a company and just updated your web server but the update broke something. No problem, you can run your web server virtualized on the older system version. Or maybe you want to run a web service in a quarantine for security reasons. As you see there are loads of different uses.

One question might rise though: how do we separate each “virtual machine” from the rest of the stuff on our computer? Docker solves this with different kernel (and non-kernel) mechanisms. We don’t have to bother about them though, since Docker takes hands of everything for us. That’s the beauty of it afterall: simplicity.

Install docker

Docker is in the ubuntu repositories (Ubuntu 14.04 here) so it’s as straightforward as:
sudo apt-get install docker

Once installed, a daemon (service) of the docker will be running. You can check that with

sudo service docker.io status

The daemon is called docker.io as you might have noticed. The client that we willuse is simply called docker. Pay attention to this tiny but significant detail.

Configuration

Do these two things before using docker to avoid any annoying warnings and problems.

Firstly we need to add ourselves to the docker group. This will let us to use docker without having to use sudo every time:

sudo adduser <your username here> docker

Log out and then in.

Secondly we will edit the daemon configuration to ensure that it doesn’t use any local DNS servers (like 127.0.0.1). Use your favourite editor to edit the /etc/default/docker.io file. Uncomment the line with DOCKER_OPTS. The result file looks like this for me:

# Docker Upstart and SysVinit configuration file

# Customize location of Docker binary (especially for development testing).
#DOCKER="/usr/local/bin/docker"

# Use DOCKER_OPTS to modify the daemon startup options.
DOCKER_OPTS="-dns 8.8.8.8 -dns 8.8.4.4"

# If you need Docker to use an HTTP proxy, it can also be specified here.
#export http_proxy="http://127.0.0.1:3128/"

# This is also a handy place to tweak where Docker's temporary files go.
#export TMPDIR="/mnt/bigdrive/docker-tmp"

We need to restart the daemon for the change to take effect:

sudo service docker.io restart

Get an image to start with

In our scenario we want to virtualize an Arch machine. On VirtualBox, we would download the Arch .iso file and go through the installation process. In Docker we download a “fixed” image from a central server. There are thousands of different such image files. You can even upload your own image as you will see later.

> docker pull base/arch
Pulling repository base/arch
a64697d71089: Download complete 
511136ea3c5a: Download complete 
4bbfef585917: Download complete

This will download a default image for Arch Linux. “base/arch” is the identifier for the Arch Linux image.

To see a list of all the images locally stored on your computer type

> docker images
REPOSITORY            TAG                   IMAGE ID            CREATED             VIRTUAL SIZE
base/arch             2014.04.01            a64697d71089        12 weeks ago        277.1 MB
base/arch             latest                a64697d71089        12 weeks ago        277.1 MB

Starting processes with docker

Once we have an image, we can start doing things in it as if it was a virtual machine. The most common thing is to run bash in it:

> docker run -i -t base/arch bash
[root@8109626c57f5 /]#

See how the command prompt changed? Now we are inside the image (virtual machine) running a bash instance. In docker jargon we are actually inside a container. The string 8109626c57f5 is the ID of the container. You don’t need to know much about that now. Just pay attention to how we acquired that ID, you will need it.

Let’s do some changes. Firstly I want to install nmap. Since pacman is the default package manager in Arch, I will use that:

[root@8109626c57f5 /]# pacman -S nmap
resolving dependencies...
looking for inter-conflicts...
..

Let’s run nmap to see if it works:

> nmap www.google.com
Starting Nmap 6.46 ( http://nmap.org ) at 2014-07-18 13:33 UTC
Nmap scan report for www.google.com (173.194.34.116)
Host is up (0.00097s latency).
..

It seems we installed it successfully! Let’s also create a file:

[root@8109626c57f5 /]# touch TESTFILE

So now we have installed nmap and created a file in this image. Let’s exit the bash

[root@8109626c57f5 /]# exit
exit
> 

In VirtualBox you can save the state of the virtual machine at any time and load it later. The same is possible with docker. For this I will need the ID of the container that I was using. In our case that is 8109626c57f5 (it was written in the terminal prompt all the time). In case you don’t remember the ID or you have many different containers, you can list all the containers:

> docker ps -a
CONTAINER ID     IMAGE                    COMMAND      CREATED            STATUS
8109626c57f5     base/arch:2014.04.01     bash         25 minutes ago     Exit 0

Let’s save the current state to a new image called mynewimage:

> docker commit -m "Installed nmap and created a file" 8109626c57f5 mynewimage
6bf56047833bd41c43c9fc3073424f37bfbc96993b65b868cb8d6a336ac28b0b

Now we have the saved image locally on our computer. We can load it anytime we want to come back to this state. And the demonstration..

> docker run -i -t mynewimage bash
[root@55c343f1643a /]# ls
TESTFILE  bin  boot  dev  etc  home  lib  lib64  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
[root@55c343f1643a /]# whereis nmap
nmap: /usr/bin/nmap /usr/share/nmap /usr/share/man/man1/nmap.1.gz

Loading the image on an other computer

We now have two images, the initial Arch image we started with and the new image that we saved:

> docker images
REPOSITORY            TAG                   IMAGE ID            CREATED             VIRTUAL SIZE
mynewimage            latest                6bf56047833b        2 hours ago         305.8 MB
base/arch             2014.04.01            a64697d71089        12 weeks ago        277.1 MB
base/arch             latest                a64697d71089        12 weeks ago        277.1 MB

It’s time to load the new image on a totally different computer. First I need to save the image on a server though. Luckily for a docker user, this is very simple. First you need to make an account at https://hub.docker.com/

Once that is done we need to upload the image to the hub. However we have to save the image in a specific format, namely username/whatever.

Let's save the image following that rule:
> docker commit -m "Installed nmap and created a file" 8109626c57f5 pithikos/mynewimage
12079e0719ce517ec7687b4bf225381b99b880510cda3bc1e587ba1da067bd3b

First we need to login to the server:

> docker login
Username (pithikos): 
Login Succeeded

Then I upload the image to the server:

> docker push pithikos/mynewimage
The push refers to a repository [pithikos/mynewimage] (len: 1)
Sending image list
..

Once everything is uploaded, we can pull it from anywhere just as we did when we first pulled the Arch image.

I will do that from inside an OpenSUSE install on a totally different machine. First I try to run nmap

Screenshot from 2014-07-18 17:17:05

As you see it’s not installed on the computer. Let’s load our Arch image that we installed nmap on
Screenshot from 2014-07-18 17:22:34

Once the download of the image is complete we will run a bash on it just to look around:

Screenshot from 2014-07-18 17:27:19_

As you see, the TESTFILE is in there and we can run nmap. We are running the same Arch I ran earlier on Ubuntu, on a totally new machine with a totally different OS, but still running it as an Arch.

A bit on containers

Now you probably got a good idea on what images are. Images are simply states of a “virtual machine”.

When we use docker run whatever we are running is put inside a container. A container is pretty much a Linux concept that arose recently with the recent addition of Linux containers to the kernel. In practise container is running a process (or group of processes) in isolation from the rest of the system. This makes the process in the container to not being able to have access to other processes or devices.

Every time we run a process with Docker, we are creating a new container.

> docker run ubuntu ping www.google.com
PING www.google.com (64.15.115.20) 56(84) bytes of data.
64 bytes from cache.google.com (64.15.115.20): icmp_seq=1 ttl=49 time=11.4 ms
64 bytes from cache.google.com (64.15.115.20): icmp_seq=2 ttl=49 time=11.3 ms
^C
> docker run ubuntu ping www.yahoo.com
PING ds-any-fp3-real.wa1.b.yahoo.com (46.228.47.114) 56(84) bytes of data.
64 bytes from ir2.fp.vip.ir2.yahoo.com (46.228.47.114): icmp_seq=1 ttl=45 time=46.5 ms
64 bytes from ir2.fp.vip.ir2.yahoo.com (46.228.47.114): icmp_seq=2 ttl=45 time=46.1 ms
^C

Here I ran two instances of the ping command. First I pinged http://www.google.com and then http://www.yahoo.com. I had to stop them both with CTRL-Z to get back to the terminal.

> docker ps -a | head
CONTAINER ID      IMAGE             COMMAND                CREATED              STATUS
7c44887b2b1c      ubuntu:14.04      ping www.yahoo.com     About a minute ago   Exit 0
72d1ca1b42c9      ubuntu:14.04      ping www.google.com    7 minutes ago        Exit 0

As you see, each command got its own container ID. We can further analyse the two containers with the inspect command. Below I compare the two different ping commands I ran to make it more apparent how the differentiate in the two containers:

> docker inspect 7c4 > yahoo
> docker inspect 72d > google
> diff yahoo google 
2,3c2,3
<     "ID": "7c44887b2b1c9f0f7c10ef5e23e2643026c99029fce8f1575816a23e56e0c2d0",
<     "Created": "2014-07-21T10:05:45.106057381Z",
---
>     "ID": "72d1ca1b42c995973086a5fb3c5e256d5cb2c5055e8f9040037bb6bb915c8187",
>     "Created": "2014-07-21T10:00:08.653219667Z",
6c6
<         "www.yahoo.com"
---
>         "www.google.com"
9c9
<         "Hostname": "7c44887b2b1c",
---
>         "Hostname": "72d1ca1b42c9",
29c29
<             "www.yahoo.com"
---
>             "www.google.com"
44,45c44,45
..

Notice that I don’t have to write the whole string. For example instead of 7c44887b2b1c, I just type the first three letters 7c4. In most cases this will suffice.


Leave a comment

Setting up a mail server

On this tutorial I will explain how to install and configure Postfix to be able and receive emails from the Internet and LAN. Running Postfix at home allows you to have an email that looks as you wish. If you have the website mywebsite.com then you can have emails like support@mywebsite.com, admin@mywebsite.com etc.

I set-up Postfix in a gradual way:

  1. First I set up a LAN mail server
  2. Then I configure the mail server to accept emails from the Internet
  3. Lastly I configure the mail server to allow outgoing emails to the Internet

Starting with a LAN mail server

I start by setting up a mail server that works for the LAN. My LAN network looks as below.

lan_postifx

ubuntux and laptop are the hostnames (names) of the computers that can be used to identify them in the LAN. For example if you want to ping ubuntux you don’t have to know its IP. You can just ping the computer by providing its hostname. Keep in mind that hostnames cannot be seen outside the LAN.

If you don’t know the hostname of your computer you can run in the terminal the command hostname. Throughout the tutorial I will be using ubuntux as my main computer where I install Postfix.

The router has an external IP 20.20.20.20 and an internal 192.168.1.1. Simply put, that means that from the internet our network is seen as 20.20.20.20 while the computers in the LAN sees the router having the IP 192.168.1.1.

Let’s install Postfix:

sudo apt-get install postfix

The install will ask you some questions. Just use the default values. After install, the daemon is launched automatically so we can start sending emails.

Sending the first email in the LAN

If you haven’t read my article on sending emails via telnet, you could check it out. In that article I describe how you can use telnet to send emails via a gmail mail server.

To send our emails we will be using telnet like in that article. The only difference is that while we had to authenticate with the gmail server with a login/password, here we won’t. The reason is that our mail server is in the LAN and by default configuration users in LAN are considered trusted.

The reason I use telnet and not an email client like Outlook Express is that I want you to see what is actually happening low-level and to get an in insight into the SMTP protocol.

First I send an email from ubuntux (192.168.1.2):

>telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 ubuntux.localdomain ESMTP Postfix (Ubuntu)
HELO whatever
250 ubuntux.localdomain
MAIL FROM: whatever
250 2.1.0 Ok
RCPT TO: animal@localhost
250 2.1.5 Ok
data
354 End data with .
This is a test
.
250 2.0.0 Ok: queued as 22AEA133C91
quit
221 2.0.0 Bye

In the telnet session I sent an email to animal@localhost. We assume that animal is the user I am logged in as at the moment. Then I can just type mail in the terminal and I will see the email arrived. Typing the number of the email I can read it. Typing ‘q’ will quit the application. All emails can be found in their raw format at /var/spool/mail.

Let’s try and send an email from the second machine (192.168.1.10):

>telnet 192.168.1.2 25
Trying 192.168.1.2...
Connected to 192.168.1.2.
220 ubuntux.localdomain ESMTP Postfix (Ubuntu)
HELO whatever
250 ubuntux.localdomain
MAIL FROM: whatever
250 2.1.0 Ok
RCPT TO: animal@ubuntux
250 2.1.5 Ok
data 
354 End data with .
Sending from my laptop.
.
250 2.0.0 Ok: queued as 68DF4133C91
quit
221 2.0.0 Bye

The only difference here is that we connected to ubuntux instead of localhost. Accordingly we had to change the recipient from animal@localhost to animal@ubuntux.

The recipient after “RCPT TO:” can essentially be any user (existing or not) as long as the domain used in the email is the hostname of our mail server (ubuntux in this case). An email with a different domain could be for example animal@randomdomain. The email with such a recipient would get totally discarded from the server and lost.

Taking it a step further – receiving emails from the Internet

At this state, emails can be delivered to the mail server from users in the LAN. We will try and make the mail server to be able and receive emails from the internet as well.

WAN_TO_LAN

In the illustration above an email is being sent from the gmail server to our own mail server. We don’t care how the email got to the gmail mail server. What we care about is how the email will reach our mail server. For the correct routing of the email we have to configure three things:

  1. Our DNS server
  2. Our router
  3. Our mail server

Configuring DNS

Every time we visit a website, a DNS request has to be made to find out the IP of that website. In the same fashion a DNS request is made by the mail server when an email is to be sent. This request is somehow different, in that it looks for a very specific record; an MX (mail exchange) record.

WAN_TO_LAN_dns

(1) As seen in the illustration the mail server looks at the domain after the ‘@’ symbol in the recipient of the email.

(2) It then sends a DNS request for the mail server in that domain (linuxmeerkat.com). To get an idea of what the answer of such a request can be you can use the debugging tool dig and test it with dig MX linuxmeerkat.com.

(3) A reply arrives from the DNS server with the IP of the mail server.

(4) The email is forwarded to that IP, and further to the mail server.

Now ideally the MX record in the DNS server would look like this:

linuxmeerkat.com.	IN	MX	10	20.20.20.20

That is however breaking the standard and therefore it is unacceptable. By defition an MX record should bind a domain to a hostname (or more). Therefore we need to replace the IP in the above example with a hostname:

linuxmeerkat.com.	IN	MX	10	mail.linuxmeerkat.com.
mail	IN	A	20.20.20.20

Since the mail.linuxmeerkat.com hostname (or FQDN to be exact) doesn’t exist, we have to create it with an A record. An A record binds a (made up) hostname to an IP by definition. Since our server is behind the router I use the IP of the router instead (the IP should never be a LAN one). For hostname I use mail but you could use whatever you wish. Just keep in mind that if you change it, the hostname must also be reflected in the MX record – you should also change the FQDN.

Keep in mind that if you do changes to the DNS records, it might take some time for the effects to take place. So continue a bit later, or after a day or two depending on how fast the changes are updated (essentially it depends on the TTL values).

Is your network reachable from outside?

Time to test if the DNS configuration works.

When we ping mail.linuxmeerkat.com we want that it pings 20.20.20.20. So let’s test it. Go to http://ping.eu/ping/ and ping your mail server’s FQDN (in this case mail.linuxmeerkat.com). It’s important that you ping from outside your LAN so that no local cache is used.

If you don’t get a response, try pinging directly your WAN IP (20.20.20.20). If you still don’t get a response, then probably your router is blocking ping from outside. If you are getting a response by pinging the IP but not when you ping the hostname, then you haven’t configured your DNS records correctly or they haven’t taken effect yet.

If you run your own DNS server you can check for warnings and errors in /var/log/syslog.

Configure your router

At this point we know that an email can arrive as far as to the router. We only need to configure the router so it forwards the email to the correct machine in the LAN.

Since we know that mail servers use port 25 and if we haven’t changed that, then our mail server also uses port 25, we just have to configure the NAT of our router.

Forward all TCP at port 25 to your mail server (in my case 192.168.1.2). My router’s port forwarding looks as below:

router

Sending an email to Postfix from gmail

If we can ping the mail server from the internet, then we can assume that our network is reachable. Sending an email to animal@mail.linuxmeerkat.com will not work still though as the router won’t know to which computer in the LAN to forward the email to. Setting the NAT of the router to forward all incoming TCP connections on port 25 to the mail server fixes this.

We can now test it by sending an email to animal@mail.linuxmeerkat.com from a gmail account. The mail server will reply with an error:

Delivery to the following recipient failed permanently:

     animal@mail.linuxmeerkat.com

Technical details of permanent failure:
Google tried to deliver your message, but it was rejected by the server for the recipient domain mail.linuxmeerkat.com by mail.linuxmeerkat.com. [20.20.20.20].

The reason the email gets rejected lies in the Postfix configuration. The configuration file /etc/postfix/main.cf has this (or a similar) line:


mydestination = ubuntux, localhost

mydestination is the variable that defines what FQDN/domains in emails are accepted. In essence all emails arrive at the server, it’s just the depending on this configuration some of these emailes will be discarded and some will be accepted by the server.

With the current configuration we can see that only emails of the format username@megistanas and username@localhost are accepted. Trying to send an email to animal@ubuntux from gmail won’t work as ubuntux just can’t be seen from outside our LAN; it doesn’t resolve to anything. For this reason we want to add one ore more domains that point towards our network. That way, when the an email arrives it doesn’t get rejected.

In my case I add linuxmeerkat.com so that I accept all emails with the format username@linuxmeerkat.com:

mydestination = linuxmeerkat.com, ubuntux, localhost

Restart Postfix and sending an email to animal@linuxmeerkat.com should now work.

You could essentially also add mail.linuxmeerkat.com as that would take the same route as well. We can add pretty much any domain that points to our router.

Sending an email to gmail or hotmail from Postfix

LAN_TO_WAN

There are essentially two entries in the configuration file that we have to change to make sending emails to the Internet possible:

smtpd_sender_restrictions = permit_mynetworks
mynetworks = 192.168.0.0/16 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128

smtpd_sender_restrictions can have more options than the one shown. However the only interesting option for us at the moment is to permit anyone from mynetworks to be able and send emails. So make sure permit_mynetworks is amongst the options.

mynetworks specifies local networks where users are considered trusted to use our mail server. For that we have added the 192.168.0.0/16 which includes all computers in the network. Be careful, you don’t want to have a WAN ip here as that would make our mail server an open relay (I talk more about that below).

This configuration will let anyone from our computer to send an email of the format username@linuxmeerkat.com. If we want users of the LAN to be able to use our mail server from the Internet, some sort of authentication has to take place (login or key), but I am will not go further into that.

Let’s test and see if we can send an email to a gmail account:

>telnet mail.linuxmeerkat.com 25
220 mail.linuxmeerkat.com ESMTP Postfix (Ubuntu)
HELO dickhead
250 mail.linuxmeerkat.com
MAIL FROM: animal@mail.linuxmeerkat.com
250 2.1.0 Ok
RCPT TO: animal@gmail.com
250 2.1.5 Ok
DATA
354 End data with <CR><LF>.<CR><LF>
This is a test.
.
250 2.0.0 Ok: queued as 3D31311F765
quit
221 2.0.0 Bye

If everything went as planned, you should have received an email in your gmail account (animal@gmail.com in the example).

Make sure your mail server is not an open relay

You always want to avoid having an open relay. An open relay constitutes that your mail server can be used by anyone on the internet including spammers.

open_relay

Once spammers find out that your server is an open relay, they will start using it. At some point, an other mail server will find out that you have an open relay and will blacklist your server.

So how do you ensure that you don’t have an open relay? Very simply. In your configuration file pay attention to these two lines:

smtpd_sender_restrictions = permit_mynetworks
mynetworks = 192.168.0.0/16 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128

As long as the networks in mynetworks are LAN IPs and not WAN IPs, your mail server is not an open relay as no-one outside the LAN can use your mail server to send emails.

You can test to see if your mail server is an open relay at http://www.mailradar.com/openrelay/.

Use a relay host if you can

Our mail server works fine at the moment but it’s very minimal. It doesn’t even filter incoming spam emails. Configuring filters etc. can be time consuming and will need further maintenance all the time. An easy solution to this is to forward our emails to a second mail server, a relay server (keep in mind: not same as an open relay, as a relay server is not open to everyone). So if we have to deliver an email to user11@gmail.com, instead of delivering to the gmail server directly, we can use a second mail server which we have access to. That server can then do the filtering etc. for us and deliver the email to its final destination.

My ISP is called Bahnhof and by googling around I found that it offers a relay at mail2.bahnhof.se. Most ISPs do offer mail servers that can be used as relays. Luckily for me, I don’t even need to enter a password and username as I am authenticated by my IP or MAC (depends on how you’re connected to the ISP). In your case you might need to authenticate first before using the relay. I won’t go into how you do that. I will however tell you how you can test to see if you can use the relay as a relay server.

If you can send an email directly from the relay server then you can use it as your relay. See the telnet session below:

telnet mail2.bahnhof.se 25
Connected to mail2.bahnhof.se.
Escape character is '^]'.
220 mxf3.bahnhof.se ESMTP
HELO dickhead
250 mxf3.bahnhof.se
MAIL FROM: animal@linuxmeerkat.com
250 Ok
RCPT TO: animal@gmail.com  
250 Ok
DATA
354 End data with <CR><LF>.<CR><LF>
This is a test.
.
250 Ok: queued as 0C67C53CEB7
QUIT
221 Bye

I just connected to my ISP’s mail server and sent an email to a gmail account. Since I got the email delivered and wasn’t asked for a login/password, that means that I can directly use the server as a relay without authentication.

If your ISP’s mail server requires a username and password you’ll have to do some extra work in configuring that. Also keep in mind that in the field “MAIL FROM:” you have to provide an email with a hostname that points to your network. The server will probably check to see if whatever@hostname actually resolves to your network. If not, you will get an error like below:

450 <spamotron@spammer.spam>: Sender address rejected: Domain not found

Now that we know we can use the server as a relay, we simply have to edit /etc/postfix/main.cf and make sure the relayhost is set correctly:

relayhost = mail2.bahnhof.se:25

Restart Postfix and you should be able to use your mail server as before with the difference that now all outgoing emails go first through the relay server and the relay server forwards them further to their destinations.


Leave a comment

Mastering sed

sed
sed is a very famous tool to the UNIX* community but which is very often misused. Most people try to use it in cases that it’s not the right tool or they tend to use it in the wrong way. I try in this post to show the things that are worth knowing when it comes to sed. I thus skip things like buffer holders, labels etc. which just make scripts totally unreadable for no benefit. I also talk a bit about the inner-workings of sed so the user has a grasp of why sometimes things don’t work as expected.

Why even learn sed?

For two reasons:

  1. You can automate any boring mechanical work you would do on a normal text editor
  2. If you know the syntax of sed, then you are a better vi/vim user

The second point, becomes obvious when you realise that the two programs have similar if not the exact syntax. For example, substituting the word “cow” with the word “horse” in the third line of the document in vim is :3s/cow/horse/ while in sed it’s 3s/cow/horse/. See the magic?!

A bit of history

Sed, awk and grep are the offsprings of a line editor called ed. ed pretty much let the user to edit one line at a time. That is also the reason that sed, awk and grep work on lines. All three programs have inherited the syntax of ed to some extend. In ed, to search & replace the word “cow” with the word “milk” in a text document, someone would type s/milk/beer/. That is exactly the same command used in sed – an indication of ancestry.
The tool grep actually takes its name from the command g/re/p, a command used in ed to only show lines that contain a specific regular expression. The ‘p’ in the end means to print on the screen while the ‘g’ in front means to go through all the lines.

When to use sed

While all three of these tools (grep, sed and awk) work on lines, sed and awk are very similar to each other while grep is more of a loner. Grep is used merely to filter out (or in) a line based on a regular expression. Sed and awk offer much more. What differs sed from awk is the data that they were built to edit.
Awk should be used when every line in the file has a specific structure. In other words that includes files where each line has a specific number of fields with every field separated with a delimiter (in most cases a tab). Such files can be CVS files, tables, the output of ls in linux, and more.
For everything else, use sed. Common examples are raw texts, this post, a C file, a script, an HTML file, etc. What all these files have in common is that lines don’t have a specific structure: the first line can have one word, while the second can have 100 words etc. Sed can still edit files that awk edits, but the opposite is most times impossible. If you are trying to just do that then you are most probably using the wrong tool.

Syntax

Sed in the terminal

The common syntax for sed in the terminal is:

sed SCRIPT INPUTFILE

The meat of sed is the SCRIPT and that is pretty much what I cover in this post. It’s a good convention to put quotation marks around it in case there is a space inside it (the shell might interpret it as multiple commands then).
sed can have multiple SCRIPTS or it can use a file with commands.

Multiple script lines:

sed 'SCRIPT1; SCRIPT2; SCRIPT3;' INPUTFILE

Using a script file:

sed -f SCRIPTFILE INPUTFILE

The SCRIPTFILE should have a command on each line. So SCRIPT1 should be on separate line than SCRIPT2 and SCRIPT3 on a separate etc.
Thoughout

Sed’s script syntax

Sed uses three things to accomplish tasks:

  • line specifiers (address)
  • commands
  • flags

The syntax of a single SCRIPT line is:

<line><command><flag>

A line or line specifier is a way to specify which lines you want the command to affect (parse). If the line specifier is missing, then the command affects all lines, which is the default behaviour.

A command is denoted by single character. For example to replace (substitute) a word with an other word we use the command ‘s’:

s/word1/word2/

A flag is used to modify the <command> or the <line>’s behaviour a bit and is placed after the whole command or line. Using the flag g on the example above, we get:

s/word1/word2/g

g stands for global and is used to replace all word1 in the line. Without it, only the first occurrence of word1 in the line is replaced.

A flag goes hand in hand with regular expressions so they can only be used if <command> or <line> have a regular expression in them. If the <line> is specified with a number for example, it’s illegal to use a flag:

#Illegal
sed -n '6g'

The reason is that flags are made for text strings (patterns) so it doesn’t make sense to sed when you are telling sed to use the flag ‘g’ on a line (which is something else than a string) and not a pattern.

The minimum thing needed on a SCRIPT line is a command or a line specifier. Someone can have both, one of them – with a flag or without. These combinations (or permutations to be exact) are allowed:

<line>
<line><flag>
<command>
<command><flag>
<line><command>
<line><command><flag>
<line><flag><command><flag>
<line><flag><command>

A flag applies to the command or line before it and assume that the previous has a regular expression in it.

How sed works internally

Imagine we have the file list.txt with the lines:

Today I will drink my milk
and afterwards I will eat a cow.
The cow will taste like cow.

Sed works with lines. As stated earlier we can have multiple script lines seperated with question marks:

sed 's/Today/Tomorrow/g; s/Today/Next Friday/g' list.txt

sed has a working buffer for each line (called pattern space). sed will initially load the first line of list.txt in the buffer. Then it will go through all script lines one by one altering everything in-place (in the buffer). In our example the buffer for the first line initially has:

Today I will drink my milk

After the first script command executes, it becomes:

Tomorrow I will drink my milk

The second script line doesn’t alter anything as sed can’t find an occurrence of the word ‘Today’ since it just got altered.
Once the first line is done, sed will load the second line in the buffer and go through the same procedure until all lines in list.txt have been parsed.

Something important to notice is that each SCRIPT line will run regardless if the previous SCRIPT line succeeded or not. With success we mean that the command did what it is meant to do. If substitution is used, then we define success as the alteration of a line. If we just specify a line, then success is if the line exists etc.

Addressing specific lines

<line><command><flag>
By default, sed goes through all the lines. However, one can address a specific line or a range of lines. That can be done by specifying lines by their number in the file (1st line, 2nd line, 50th line etc.) or by a line’s content.

To run a command on the 10th line we do:

10<command>

To run a command on each line that contains the word “cow” we do:

/cow/<command>

The latter makes use of regular expressions. When we use regular expressions we need to add slashes to the start and end of the regular expression.

For a range of lines we use a comma (awkward, I know). To specify all lines between the 10th and 25th line (including those) we would write:

10,25<command>

If we want to specify a range of lines by using regular expressions we still have to encapsulate the regular expressions in slashes. To run a command on all the lines between the first line found with the word “cow” and the first line found with the word “grass”, we would issue:

/cow/,/grass/<command>

Bellow you can see all the ways for specifying lines.

<line1>,<line2>
Lines between <line1> and <line2> (including those)
<line1>~N
Every Nth line after <line1>
<line1>!
All lines except line1
/<regex>/
Lines matching the regular expression
$
Last line
<line1>,+N
All N number of lines after <line1>

Some more practical examples can be seen bellow. The command p is used to print the line that is specified. (Notice that sed needs the parameter -n for the command p to work.)

sed -n '2p'             -> print line 2
sed -n '2,4p'           -> print line 2 to 4
sed -n '$p'             -> print last line
sed -n '2!p'            -> print all lines but line 2
sed -n '/red/p'         -> print every line that contains the word red
sed -n '/red/,/green/p' -> print all lines between the <strong>first occurrences</strong> of the words 'red' and 'green'

Of course for all these examples to work you need to either feed sed with a stream from a file or a pipe.

Using commands

<line><command><flag>
Now we are to the meat of all meats.I have explained the substitution command a bit but bellow you can see all the commands with their syntax (if they have one).

s/<regex>/<subst>/ substitute Replace a match (using regular expression) with a string.
p print Prints a specific line or a range of lines. The sed flag -n should be used for this command to work.
= line number Shows the number of the line
d delete Deletes (omits) the matched line
y/<char1>/<char2>/ transform Substitutes char1 with char2. Works even with a sequence of characters. For example y/abc/ABC/ will replace a with A, b with B and c with C.

Substitution

Substitution is the most commonly used command. It’s syntax is as follows:

s<d><regex><d><subst><d>

where <d> is a delimiter which should be a single character. Most commonly the delimiter is a slash / but it can essentially be any character. All lines bellow are equivalent.

s/cow/horse/
s_cow_horse_
sDcowDhorseD

In the example above the first occurrence of the word cow on each line will be replaced with the word horse, which is the default behaviour. If you want all occurrences of the word cow in a line to be substituted, the flag g (global) has to be appended to the line:

s/cow/horse/g

The substitution field <subst> can take some special variables like the ampersand symbol “&” which holds the matched string from the regular expression. There are also a few macros to automate conversion of capitals to lower-case and vice versa.
All these are shown in the table bellow.

& Holds the matched string.
\1 Holds a part of the match specified in the regular expression with parentheses.
\U<string> Converts all letters in <string> to capitals.
\u<string> Converts the first letter in <string> to capital.
\L<string> Converts all letters in <string> to lower-case.
\l<string> Converts the first letter in <string> to lower-case.
<string>\E Ends the conversion at specific point. Should be used in conjunction with \L and \U.

The match holders \1 \2 \3 etc. have to be specified in the regular expression with parentheses. The parentheses themselves have to be escaped or else sed will be looking for parenthesis characters in the line. So to replace each word “cow” with “supercow” we can do:

s/\(cow\)/super\1/

or

s/cow/super&/

The latter is more elegant of course. However there are two cases where the specific holder has to be used:

  1. When we want only a portion of the matched string and not the whole string (&).
  2. When there are many different strings you want to grab from a match.

For example this can’t be solved by merely using the & symbol:

s/\(\w*\) cow \(\w*\)/\2 cow \1/

This script will look for each occurrence where “cow” has a word before it and a word after it and it will change their order. The \w matches any character while the asterisk * tells the pattern that the word can be arbitrarily long. We use the parentheses around the first word and the second word to denote which matched parts of the regular expression should be given to \1 and \2. In <subst> we just reverse their order by putting the second word first and the first word second.

Flags

<line><command><flag>
A flag can be used on a <command> or a <line> or both.

g global For all occurences in the line (default is to stop at first occurence)
I Ignore case Not case-sensitive
p Print Output only this line (not everything as default). The sed flag -n should be used for this flag to work.

Find line in lines of lines in lines..

An important concept to comprehend in sed is nesting. I try to leave out all the advanced things sed offers, like holder buffer (horror to read), labels etc. but nesting is worth learning as it gives a lot of extra power for a little learning curve. Nesting is similar to the IF..THEN conditional.

We have this text file:
Today I will drink my milk
and afterwards I will eat a cow.
The cow will taste like cow.
Today is not afterwards if I am a cow. Right?

Imagine that we want to check if the last line of the text contains the word cow. Someone might think that putting $p together with /cow/p would work:

sed -n '$p; /cow/p'

Admittedly the output seems strange:
and afterwards I will eat a cow.
The cow will taste like cow.
Today is not afterwards if I am a cow. Right?
Today is not afterwards if I am a cow. Right?

The reason this doesn’t work as expected is that the second script runs regardless if the first one succeeded or not. Thus in the first line none of the scripts print anything. On the second line, the first script fails but the second script finds the word cow so it prints the line. The same happens at the third line. At the forth line, the first script succeeds as the line is the last line of the file, so that line gets printed. Then the second script runs and that succeeds. Thus we get the line printed again (a second time).

A way to solve this is is to nest the second script line in the first somehow so that it runs only if the first script line has succeeded. This is similar to the pseudocode:

if <line1>
  then SCRIPT

Where <line1> is a line specified by a number, range or pattern (regular expression).
The syntax for nesting script lines in sed is

<line>{<line><command><flag>}

For our example:

sed -n '${/cow/p}'

This translates to: if this is the last line ($) then do whatever is in the brackets. So everything in the brackets will be checked only if the current line being parsed is the last one.

We can even nest inside a nested script:

sed -n '2,4{/cow/{/Today/p}}'

This script will go through lines 2 to 4. It will check first if a line contains the word cow. If the line contains the word cow, then it will check if it contains the word Today. If it does, it will print it. Ofcourse there is no practical reason to have second braces in the above example. I just wrote it to show that it’s feasible. In some cases you need braces to accomplish tasks, especially in cases where we avoid using too advanced things.

Examples

Printing a specific line

sed -n '2p'

The p in this case is the p command (and not flag). Remember that flags apply only to regular expressions. When we use sed’s -n parameter, only lines specified with the p command or flag will be printed on screen.

Printing a range of lines

sed -n '1,3p'

The comma is used to specify a range. In this example we specify line 1 to line 3 and then we print each such line.

Hiding a specific line

sed -n '2!p'

This is similar to:

sed '2d'

Show the last line

sed -n '$p'

In regular expressions the dollar sign $ denotes the end of a string. In sed when it’s being used with substitution it denotes the end of a line.
However when used with the command p, it denotes the last line of the input. In the same way the regex symbol ^ denotes the first line.

Converting a specific word to uppercase

sed 's/Today/\U&/' list.txt

This will match any word ‘Today’ and replace it with ‘TODAY’. In the replacement the escaped U (\U) tells sed to convert to uppercase everything following in the replacement string. In this example we use an ampersand which in sed represents the matched string. If we wanted to stop the conversion we just need to add \E where we want it to end.

Converting all text to uppercase

sed 's/.*/\U&/'

For this we use the substituion command s.
.* is a regular expression which fits any sequence of characters. As sed is working on lines, .* matches a whole line each time.

Converting all text to lowercase

sed 's/.*/\L&/'

Similar to the previous example with only difference that we use \L instead of \U.

Grabbing all content between the body tags in HTML

Say we have the ugly HTML code bellow and we want to grab all the content between the body tags.

<html><head></head>
<h1>h1 outside of body</h1><body><h1>h1 stuck to body</h1>
<img src="images/soon.png"/>
<p>a paragraph</p></body></html>

Most sed experts would go about using some very advanced commands to accomplish this. The readability of that becomes horrific. As my opinion is that you can do the same things without knowing all the advanced commands I am going to just do that.
Notice that I use pipes instead of using advanced commands.

Solution:

sed -n '/<body>/,/<\/body>/p' | sed 's_.*<body>\(.*\)_\1_; s_\(.*\)</body>.*_\1_'

It might seem like a mess but I can assure you that it’s much more elegant than a pure single sed SCRIPT solution. I will break it down so you can see how it works.

/body/,/body/p

matches all lines between the body tags, including the body tags. In this way I have minimized my problem to:

<h1>h1 outside of body</h1><body><h1>h1 stuck to body</h1>
<img src="images/soon.png"/>
<p>a paragraph</p></body></html>

After that, we are sure that the first line has the start of the body tag and the last line has the closing of the body tag. So we start by filtering out things we don’t need from the first line: the tag itself and everything preceding it.

s_.*<body>\(.*\)_\1_

I use _ as a delimeter instead of slashes to make the substitution code a bit more readable. The regular expression .* matches 0 to infinite number of arbitrary characters. So I use it around the body tag in case there is something before and after it. I put the second .* in parentheses to grab the text that might be there as I want to keep that. Using \1 in the substitution field I accomplish substituting the whole line with the text after body.

s_\(.*\)</body>.*_\1_

We do something similar to the line with </body>. The only difference is that now the portion of the match that we are interested in is the one before the body tag so we move the parentheses there.

Keep in mind that this solution will not work if the body tag includes some attributes like style. Someone might think that just using the bellow regular expression in the substitution would work.

.*<body.*>\(.*\)

Notice that the only difference is that we added the regular expression .* between body and its closing arrow > to point out that there can be nothing in between or there could be some arbitrary things (in our case attributes).

That regular expression doesn’t work however as it matches the last > in the line. The reason is that in sed .* is greedy and there is no way to make it non-greedy. With greedy we mean that the pattern will try to match as much as it can in the line. So if you want to match the first > it’s not possible. Or.. actually it is possible but the code as you will see in the next example starts looking like a monster.

Grabbing all content between tags in HTML

You are probably better off learning Perl or Python if you need to do these kind of “advanced things”. I will however show that you can achieve things like this without using the more advanced commands or other programs/languages. This solution is a continuation of the previous example on catching the content between the body tags. The mere difference is that in this solution we allow even attributes to a tag and are a bit more permissive towards whitespaces. This makes it a more general solution that can be used for other tags than the body tag.

Solution:

sed -n '/<body>/,/<\/body>/p' | sed '1s_.*<body[a-zA-Z0-9="\x27_ ]*> *<\(.*\)_<\1_; 1{/<body>/d}; s_\(.*\)</body>.*_\1_'

Essentially the only thing that got changed from the previous example is the alteration of the commands in the first line:

's_.*<body>\(.*\)_\1_;'

to two commands:

'1s_.*<body[a-zA-Z0-9="\x27_ ]*> *<\(.*\)_<\1_; 1/{/<body>/d};'

The first script line (everything before the first ;) looks for a second tag after body. We don’t really need to specify line 1 but it is a good convention as it makes the code easier to understand and the processing faster. I am looking for the body tag followed by an attribute or not. TO define an attribute in HTML, only the characters in the brackets are allowed (says the HTML protocol, not me). \x27 is the code for a single quote mark . The reason I use its code instead of the mark itself (I use the double quote after all), is that I use the single quote marks around the whole command so if I insert it in the expression, then it will break it. After that I use ” *<" (notice the space) to denote that there might be an arbitrary number of spaces or none before the opening of the new tag. I replace everything with the opening of the tag after body with the rest of the line.

If the first command didn’t succeed then it means that there’s not a second tag after the body. Thus it’s safe to delete the whole line in that case. First we check if there is a body tag in the first line. If there is a body tag (/<body>/), then we delete it with the command d.


13 Comments

Emailing from a gmail acount via Telnet

Telnet is a small amazing tool that can be used for pretty much everything concerning networks. Telnet sends ASCII strings directly to a host. So as long as a protocol supports ASCII messages, we can communicate with a host with any protocol we want. Some examples are SMTP and HTTP.

In practise this method is being used to test things. If you are developing a web-server for example it would be easy if you could test your web-server directly without having to launch a browser and type a command. It’s a blessing for automation.

Here we will try to send an email by using the SMTP protocol. We will use telnet for that, by sending and receiving raw messages with the SMTP server.

Preparation

First we need to install telnet with SSL support. All SMTP servers use SSL encryption nowadays so using simply telnet will not work. The telnet shipping with Ubuntu does not support SSL so we need to upgrade it:
sudo apt-get install telnet-ssl

To run the new telnet-ssl we just issue telnet in the terminal as before. The difference is that telnet now supports some new flags, namely -z, which is used for SSL.

We are also going to need http://www.base64encode.org to encode our username and password to base 64 format. Keep in mind that base 64 does not encrypt a message. It just changes the way the information is stored. In practise we use base 64 to be able and send any kind of data as simple characters. The real encryption is under the hood however. Check Extras for more info.

Steps

  1. Start telnet
  2. telnet -z ssl smtp.gmail.com 465
    The flag -z ssl tells telnet to use SSL over the connection. smtp.gmail.com is Gmail’s server domain and 465 is the port used by the server.

  3. Handshake with server
  4. HELO yo
    The text after the HELO is supposedly your domain or literal address. Essentially you can enter whatever you want. Pressing enter, the exact message arrives at the mail server, which in response should reply with
    250 mx.google.com at your service

  5. Login on gmail
  6. AUTH LOGIN
    This message is self-explanatory. After the server receives this message, it’s going to ask for our username. The response looks like this
    334 VXNlcm5hbWU6 and indicates that the username should be entered.

  7. Username
  8. Go to http://www.base64encode.org and encode your username (username@gmail.com) to 64 base. Copy and paste the encoded string in your telnet session and hit enter. If the username is accepted, you get back the message bellow.
    334 UGFzc3dvcmQ6
    This message indicates that the password should be entered.

  9. Password
  10. Do the same as above but instead of your username, encode your password. Then paste the password and hit enter. If the username/password combination is correct, you are greeted with the message
    235 2.7.0 Accepted

  11. Email sender
  12. First we are going to add the sender of the email:
    MAIL FROM: <username@gmail.com>
    For gmail it doesn’t really matter who you set as sender. We logged in our gmail account when we sent the AUTH LOGIN message to the server. Gmail has as a mechanism to set the sender automatically to your email address.
    The response back is
    250 2.1.0 OK js17sm40481494lab.5 - gsmtp

  13. Email recipient
  14. Now we will add the recipent of the email. Here we need to provide a valid email or else the email will not go any further. If the email doesn’t exist, we will probably get it back into our inbox which is the default behaviour for undelivered emails.
    RCPT TO: <username@gmail.com>
    On success we get
    250 2.1.5 OK js17sm40481494lab.5 - gsmtp

  15. Email body
  16. Now we want to start typing the email. We send the following command to the server so that the server knows that what follows is the actual email.
    DATA
    The response from the server is
    354 Go ahead js17sm40481494lab.5 - gsmtp

    Now we need to set the subject(if we want).
    Subject: test
    and everything else following is considered the text of the email

    This is a line in the email.
    This is a second line in the email.
    .

    The dot on its own line in the end is to signal the end of the email. After that, a response from the server should come:
    250 2.0.0 OK 1381416452 js17sm40481494lab.5 - gsmtp
    Now the email has been sent.

  17. Close telnet
  18. QUIT
    This is an actual command to the telnet client and not a message sent to the mail server.

Extras

EHLO vs HELO

EHLO can be used instead of HELO. The difference is that EHLO lists also all the commands that the server supports. EHLO is newer than HELO and is suggested to be used instead of EHLO. However HELO is always going to be supported, so in reality there is no big difference as long as the server supports the message.

SSL and encryption

Something important, as mentioned in the beginning of this page, is that encoding data to base 64 is not encryption but rather a different representation of the same data. It’s SSL which adds the encryption to our communication and you can be assured that everything is encrypted between the client and the server.

I captured the whole procedure of sending an email, with Wireshark to prove my point.
Screenshot from 2013-10-10 17:33:52_crop

The packets are in the order they were sent and received. We can see that telnet first contacts the DNS server to get the domain name of the mail server. 192.168.1.1 is my router and 192.168.1.2 is my computer from where I use telnet.

After we get the domain we do a first handshake with the mail server. It’s those three TCP messages with SYN/ACK in them. After that we initiate a TLS session. TLS and SSL are pretty much the same thing. They are just different versions for encrypted communication, with TLS being the newer one.

Everything we type and read from the time we start telnet is encrypted. The reason we see the messages clearly in the terminal is that they get decrypted by telnet on their arrival. When messages leave telnet, they get encrypted to travel on the wire. That’s why with Wireshark we only see the decrypted data, because we are looking at what is passing through the wire. I circled the TLS messages to make it more apparent that the whole session is encrypted by using TLS. The few TCP packets seen here and there, are packets with encrypted data. If you could check what these packets hold inside(with Wireshark), you would only see gibberish.

If instead of Gmail we were using an SMTP server that doesn’t use SSL, we would be able to intercept all the packets with their raw data, and thus the actual messages, passwords, usernames, etc.