I swear! Meerkats can do Linux

1 Comment

Docker in a development enviroment


A few days ago I wrote a tutorial on how to setup docker and use it between different machines. Now that was a nice first insight on how to jump-start using docker. It was also a nice way to showcase the possibilities and limitations of Docker.

In this post I will give some practical information on how to use docker as a developer.


To use Docker for development of software we want mainly three things:

  • Have our source code on the host machine. That way we can use GUI editors and whatever tools we want from outside the container.
  • Be able to have multiple terminals to the same container. This is good for debugging
  • Setup a docker image which we will use for running our program. I will use Django and Python for that.

And for the visual brains out there:
As you see in the pic, I am using Ubuntu as my host machine. At the same machine I have a folder with the source code and two terminals. Then I run a container with OpenSUSE. The folder and terminals reside ont he host machine but they communicate directly with the container. I will describe below how to achieve all this.

Multiple terminals

The easiest way to have multiple terminas is to use a small tool called nsenter. The guide can be found at but it sums up to running this one-liner from any folder:

> docker run --rm -v /usr/local/bin:/target jpetazzo/nsenter

That installs nsenter on the host machine. After that, we can use it directly. So let’s try it. Open bash in a container with ubuntu as our basic image:

> docker run -t -i ubuntu /bin/bash
root@04fe75de21d4:/# touch TESTFILE
root@04fe75de21d4:/# ls
TESTFILE  boot	etc   lib    media  opt   root	sbin  sys  usr
bin	  dev	home  lib64  mnt    proc  run	srv   tmp  var

In the terminal above, I created a file called TESTFILE. We will try to open a second terminal and check to see the file from it.

To use xsenter we need the process ID of the container. Unfortunately we can’t use ps aux but rather have to use docker’s inspect command. I open a new terminal and type the below

> PID=$(docker inspect --format {{.State.Pid}} 04fe75de21d4)
> sudo nsenter --target $PID --mount --uts --ipc --net --pid

The string 04fe75de21d4 I gave is the ID of my container. If everything went ok, your terminal prompt will change to the same ID:

root@04fe75de21d4:/# ls
TESTFILE  bin  boot  dev  etc  home  lib  lib64  media	mnt  opt  proc	root  run  sbin  srv  sys  tmp	usr  var

See the TESTFILE there? Congrats! Now we have a second terminal to the exact same container!

Share a folder between host and container

Now I want to have a folder on my host computer and be able and access it through a container.

Luckily for us there is a built-in way to do that. We just have to specify a flag -v to docker. First let’s make a folder though that will be mounted:

> mkdir /home/manos/myproject

Let’s now mount it into the container:

> sudo docker run -i -t -v /home/manos/myproject:/home/myproject ubuntu /bin/bash
> root@7fe33a71ac2f:/#

If I now create a file inside /home/manos/myproject the change will be reflected from inside the container and vice versa. Play a bit with it by creating and deleting files from either the host or from inside the container to see for yourself.

Create a user in the container

It is wise to have a normal user in your image. If you don’t then you should create one and save the image. That way the source files can be opened from a normal user on your host – you won’t need to launch your IDE with root privileges.

> adduser manos

Follow the instructions and then commit your image. That way whenever you load the image again, you will be having user manos. To change to user manos just type

> su manos

All files you create now, will be accesible by a normal user at the host machine. Something else you could do is to somehow

Real life scenario: Python, Django and virtualenv

I wanted to learn Django. Installing Django is commonly made with the package manager pip, but pip has a bad history of breaking up things since it doesn’t communicate with Debian’s apt. So at some point if you installed/uninstalled python stuff with apt, pip wouldn’t know about it and vice versa. In the end you would end up with a broken python environment. That’s why a tool called virtualenv is being used – a tool that provides isolation. Since we have docker though which also provides isolation we can simply use that.

So what I really want:

  1. Have the source code on my host.
  2. Run django and python inside a container.
  3. Debug from at least two terminals.

Visually my setup looks as something like this:


I assume you have an image with django and python installed. Let’s call the image py3django.

Firstly create the folder where you want your project source code to be. This is the folder that we will mount. My project resides in /home/manos/django_projects/myblog for example.

Once it’s created I just run bash on the image py3django. This will be my primary terminal (terminal 1):

> sudo docker run -i -t -p 8000:8000 -v /home/manos/django_projects/myblog:/home/myblog py3django /bin/bash

The flag -p makes sure that docker doesn’t choose a random port for us. Since we run Django we will want to run a web server with a fixed port (on 8000). The flag -v mounts our host folder /home/manos/django_projects/myblog to the container’s folder /home/myblog. py3django is the image I have.

Now we have a folder where we can put our source code and a working terminal to play with. I want though a second terminal (terminal 2) to run my python webserver. So I open a second terminal and type:

> sudo nsenter --target $(docker inspect --format {{.State.Pid}} 2fe3611c1ec2) --mount --uts --ipc --net --pid
> root@2fe3611c1ec2:/#

Mind that I had to put the appropriate container ID in the command above.

Now all this is very nice but admittedly it’s very complex and it will be impossible to remember all these commands and boring to type them each single day. Therefore I suggest you create a BASH script that initiates the whole thing.

For me it took a whole day to come up with the script below:

#! /bin/bash

django_project_path="/home/manos/django_projects/netmag" # Path to project on host
image="pithikos/py3django_netmag_rmved"                  # Image to run containers on

echo "-------------------------------------------------"
echo "Project:  $django_project_path"
echo "Image  :  $image"

# 1. Start the container in a second terminal
proj_name=`basename $django_project_path`
old_container=`docker ps -n=1 -q`
export docker_line="docker run -i -t -p 8000:8000 -v $django_project_path:/home/$proj_name $image /bin/bash"
export return_code_file="$proj_name"_temp
rm -f "$return_code_file"
gnome-terminal -x bash -c '$docker_line; echo $? > $return_code_file'
sleep 1
if [ -f "$return_code_file" ] && [ 0 != "$(cat $return_code_file)" ]
	echo "--> ERROR: Could not load new container."
	echo "    Stop any other instances of this container"
	echo "    if they are running and try again."
	echo "    To reproduce the error, run the below:"
	echo "    $docker_line"
	rm -f "$return_code_file"
	exit 1
rm -f "$return_code_file"

# 2. Connect to the new container
while [ "$old_container" == "`docker ps -n=1 -q`" ]; do
	sleep 0.2
container_ID=`docker ps -n=1 -q`
sudo nsenter --target $(docker inspect --format {{.State.Pid}} $container_ID) --mount --uts --ipc --net --pid

This script starts a container on a second terminal and then connects to the container from the current terminal. If starting the container fails, an appropriate message is given. django_project_path is the full path to the folder on the host with the source code. The variable image holds the name of the image to be used.

You can combine this with devilspie, an other nice tool that automates the position and size of windows when they’re launched.

In case you wonder about the top window with all the containers, that’s simply a watch command, a tool that updates regularly a command. In my case I use watch with docker ps. Simple stuff:

> watch docker ps

I use this because I personally like having an overview on the running containers. That way I don’t end up with trillions of forgotten containers that eat up my system.

Now that you have everything setup you can also run django server from one of the two terminals or whatever else you might want.

1 Comment

Docker tutorial

So as an intern in a big company I was given the task to get comfortable with docker. The problem is that docker is quite fresh so there isn’t really that much of good tutorials out there. After reading a bunch of articles and sparse tutorials (even taken the official tutorial at, I still straggled to get a firm grip on what docker even is supposed to be used for. Therefore I decided to make this tutorial for a total beginner like me.

Docker vs VirtualBox

There are many different explanations on the internet about what docker is and when to use it. Most of them however tend to complicate things more than giving some practical information for a total beginner.

Docker simply put is a replacement for virtual machines. I will use virtual box as a comparison example since it’s very easy for anyone to download and try to see the differences themselves.

The application VirtualBox is essentially a virtual machine manager.

Each and every of the OSes you see in the picture above, is an installed virtual machines. Each such machine has it’s own installed OS, kernel, virtual devices like hard disks, network cards etc. All this takes a considerate amount of memory and needs extra processing power. All virtual machines (VM) like VMware, Parallels, behave the same.


Now imagine that we want to use nmap from an OpenSUSE machine but we are on an Ubuntu. Using VirtualBox we would have to install the whole OS and then run it as a virtual machine. The memory consumption is humongous for such a trivial task.

In contrast to VirtualBox or any other virtual machine, Docker doesn’t install the whole OS. Instead it uses the kernel and hardware of our primary computer (the host). This makes Docker to virtualize super fast and consume only a fraction of the memory we would else need. See the benefits? Imagine if we wanted to run 4 different programs on 4 different OSes. That would take at a minimum 2GB of RAM.

But why would you want to run nmap on openSUSE instead of the host computer? Well this was just a silly example. There are other examples that prove the importance of a tool like Docker. Imagine that you’re a developer and you want to test your program on 10 different distributions for example. Or maybe you are the server administrator on a company and just updated your web server but the update broke something. No problem, you can run your web server virtualized on the older system version. Or maybe you want to run a web service in a quarantine for security reasons. As you see there are loads of different uses.

One question might rise though: how do we separate each “virtual machine” from the rest of the stuff on our computer? Docker solves this with different kernel (and non-kernel) mechanisms. We don’t have to bother about them though, since Docker takes hands of everything for us. That’s the beauty of it afterall: simplicity.

Install docker

Docker is in the ubuntu repositories (Ubuntu 14.04 here) so it’s as straightforward as:
sudo apt-get install docker

Once installed, a daemon (service) of the docker will be running. You can check that with

sudo service status

The daemon is called as you might have noticed. The client that we willuse is simply called docker. Pay attention to this tiny but significant detail.


Do these two things before using docker to avoid any annoying warnings and problems.

Firstly we need to add ourselves to the docker group. This will let us to use docker without having to use sudo every time:

sudo adduser <your username here> docker

Log out and then in.

Secondly we will edit the daemon configuration to ensure that it doesn’t use any local DNS servers (like Use your favourite editor to edit the /etc/default/ file. Uncomment the line with DOCKER_OPTS. The result file looks like this for me:

# Docker Upstart and SysVinit configuration file

# Customize location of Docker binary (especially for development testing).

# Use DOCKER_OPTS to modify the daemon startup options.
DOCKER_OPTS="-dns -dns"

# If you need Docker to use an HTTP proxy, it can also be specified here.
#export http_proxy=""

# This is also a handy place to tweak where Docker's temporary files go.
#export TMPDIR="/mnt/bigdrive/docker-tmp"

We need to restart the daemon for the change to take effect:

sudo service restart

Get an image to start with

In our scenario we want to virtualize an Arch machine. On VirtualBox, we would download the Arch .iso file and go through the installation process. In Docker we download a “fixed” image from a central server. There are thousands of different such image files. You can even upload your own image as you will see later.

> docker pull base/arch
Pulling repository base/arch
a64697d71089: Download complete 
511136ea3c5a: Download complete 
4bbfef585917: Download complete

This will download a default image for Arch Linux. “base/arch” is the identifier for the Arch Linux image.

To see a list of all the images locally stored on your computer type

> docker images
REPOSITORY            TAG                   IMAGE ID            CREATED             VIRTUAL SIZE
base/arch             2014.04.01            a64697d71089        12 weeks ago        277.1 MB
base/arch             latest                a64697d71089        12 weeks ago        277.1 MB

Starting processes with docker

Once we have an image, we can start doing things in it as if it was a virtual machine. The most common thing is to run bash in it:

> docker run -i -t base/arch bash
[root@8109626c57f5 /]#

See how the command prompt changed? Now we are inside the image (virtual machine) running a bash instance. In docker jargon we are actually inside a container. The string 8109626c57f5 is the ID of the container. You don’t need to know much about that now. Just pay attention to how we acquired that ID, you will need it.

Let’s do some changes. Firstly I want to install nmap. Since pacman is the default package manager in Arch, I will use that:

[root@8109626c57f5 /]# pacman -S nmap
resolving dependencies...
looking for inter-conflicts...

Let’s run nmap to see if it works:

> nmap
Starting Nmap 6.46 ( ) at 2014-07-18 13:33 UTC
Nmap scan report for (
Host is up (0.00097s latency).

It seems we installed it successfully! Let’s also create a file:

[root@8109626c57f5 /]# touch TESTFILE

So now we have installed nmap and created a file in this image. Let’s exit the bash

[root@8109626c57f5 /]# exit

In VirtualBox you can save the state of the virtual machine at any time and load it later. The same is possible with docker. For this I will need the ID of the container that I was using. In our case that is 8109626c57f5 (it was written in the terminal prompt all the time). In case you don’t remember the ID or you have many different containers, you can list all the containers:

> docker ps -a
CONTAINER ID     IMAGE                    COMMAND      CREATED            STATUS
8109626c57f5     base/arch:2014.04.01     bash         25 minutes ago     Exit 0

Let’s save the current state to a new image called mynewimage:

> docker commit -m "Installed nmap and created a file" 8109626c57f5 mynewimage

Now we have the saved image locally on our computer. We can load it anytime we want to come back to this state. And the demonstration..

> docker run -i -t mynewimage bash
[root@55c343f1643a /]# ls
TESTFILE  bin  boot  dev  etc  home  lib  lib64  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
[root@55c343f1643a /]# whereis nmap
nmap: /usr/bin/nmap /usr/share/nmap /usr/share/man/man1/nmap.1.gz

Loading the image on an other computer

We now have two images, the initial Arch image we started with and the new image that we saved:

> docker images
REPOSITORY            TAG                   IMAGE ID            CREATED             VIRTUAL SIZE
mynewimage            latest                6bf56047833b        2 hours ago         305.8 MB
base/arch             2014.04.01            a64697d71089        12 weeks ago        277.1 MB
base/arch             latest                a64697d71089        12 weeks ago        277.1 MB

It’s time to load the new image on a totally different computer. First I need to save the image on a server though. Luckily for a docker user, this is very simple. First you need to make an account at

Once that is done we need to upload the image to the hub. However we have to save the image in a specific format, namely username/whatever.

Let's save the image following that rule:
> docker commit -m "Installed nmap and created a file" 8109626c57f5 pithikos/mynewimage

First we need to login to the server:

> docker login
Username (pithikos): 
Login Succeeded

Then I upload the image to the server:

> docker push pithikos/mynewimage
The push refers to a repository [pithikos/mynewimage] (len: 1)
Sending image list

Once everything is uploaded, we can pull it from anywhere just as we did when we first pulled the Arch image.

I will do that from inside an OpenSUSE install on a totally different machine. First I try to run nmap

Screenshot from 2014-07-18 17:17:05

As you see it’s not installed on the computer. Let’s load our Arch image that we installed nmap on
Screenshot from 2014-07-18 17:22:34

Once the download of the image is complete we will run a bash on it just to look around:

Screenshot from 2014-07-18 17:27:19_

As you see, the TESTFILE is in there and we can run nmap. We are running the same Arch I ran earlier on Ubuntu, on a totally new machine with a totally different OS, but still running it as an Arch.

A bit on containers

Now you probably got a good idea on what images are. Images are simply states of a “virtual machine”.

When we use docker run whatever we are running is put inside a container. A container is pretty much a Linux concept that arose recently with the recent addition of Linux containers to the kernel. In practise container is running a process (or group of processes) in isolation from the rest of the system. This makes the process in the container to not being able to have access to other processes or devices.

Every time we run a process with Docker, we are creating a new container.

> docker run ubuntu ping
PING ( 56(84) bytes of data.
64 bytes from ( icmp_seq=1 ttl=49 time=11.4 ms
64 bytes from ( icmp_seq=2 ttl=49 time=11.3 ms
> docker run ubuntu ping
PING ( 56(84) bytes of data.
64 bytes from ( icmp_seq=1 ttl=45 time=46.5 ms
64 bytes from ( icmp_seq=2 ttl=45 time=46.1 ms

Here I ran two instances of the ping command. First I pinged and then I had to stop them both with CTRL-Z to get back to the terminal.

> docker ps -a | head
CONTAINER ID      IMAGE             COMMAND                CREATED              STATUS
7c44887b2b1c      ubuntu:14.04      ping     About a minute ago   Exit 0
72d1ca1b42c9      ubuntu:14.04      ping    7 minutes ago        Exit 0

As you see, each command got its own container ID. We can further analyse the two containers with the inspect command. Below I compare the two different ping commands I ran to make it more apparent how the differentiate in the two containers:

> docker inspect 7c4 > yahoo
> docker inspect 72d > google
> diff yahoo google 
<     "ID": "7c44887b2b1c9f0f7c10ef5e23e2643026c99029fce8f1575816a23e56e0c2d0",
<     "Created": "2014-07-21T10:05:45.106057381Z",
>     "ID": "72d1ca1b42c995973086a5fb3c5e256d5cb2c5055e8f9040037bb6bb915c8187",
>     "Created": "2014-07-21T10:00:08.653219667Z",
<         ""
>         ""
<         "Hostname": "7c44887b2b1c",
>         "Hostname": "72d1ca1b42c9",
<             ""
>             ""

Notice that I don’t have to write the whole string. For example instead of 7c44887b2b1c, I just type the first three letters 7c4. In most cases this will suffice.

Leave a comment

Setting up a mail server

On this tutorial I will explain how to install and configure Postfix to be able and receive emails from the Internet and LAN. Running Postfix at home allows you to have an email that looks as you wish. If you have the website then you can have emails like, etc.

I set-up Postfix in a gradual way:

  1. First I set up a LAN mail server
  2. Then I configure the mail server to accept emails from the Internet
  3. Lastly I configure the mail server to allow outgoing emails to the Internet

Starting with a LAN mail server

I start by setting up a mail server that works for the LAN. My LAN network looks as below.


ubuntux and laptop are the hostnames (names) of the computers that can be used to identify them in the LAN. For example if you want to ping ubuntux you don’t have to know its IP. You can just ping the computer by providing its hostname. Keep in mind that hostnames cannot be seen outside the LAN.

If you don’t know the hostname of your computer you can run in the terminal the command hostname. Throughout the tutorial I will be using ubuntux as my main computer where I install Postfix.

The router has an external IP and an internal Simply put, that means that from the internet our network is seen as while the computers in the LAN sees the router having the IP

Let’s install Postfix:

sudo apt-get install postfix

The install will ask you some questions. Just use the default values. After install, the daemon is launched automatically so we can start sending emails.

Sending the first email in the LAN

If you haven’t read my article on sending emails via telnet, you could check it out. In that article I describe how you can use telnet to send emails via a gmail mail server.

To send our emails we will be using telnet like in that article. The only difference is that while we had to authenticate with the gmail server with a login/password, here we won’t. The reason is that our mail server is in the LAN and by default configuration users in LAN are considered trusted.

The reason I use telnet and not an email client like Outlook Express is that I want you to see what is actually happening low-level and to get an in insight into the SMTP protocol.

First I send an email from ubuntux (

>telnet localhost 25
Connected to localhost.
Escape character is '^]'.
220 ubuntux.localdomain ESMTP Postfix (Ubuntu)
HELO whatever
250 ubuntux.localdomain
MAIL FROM: whatever
250 2.1.0 Ok
RCPT TO: animal@localhost
250 2.1.5 Ok
354 End data with .
This is a test
250 2.0.0 Ok: queued as 22AEA133C91
221 2.0.0 Bye

In the telnet session I sent an email to animal@localhost. We assume that animal is the user I am logged in as at the moment. Then I can just type mail in the terminal and I will see the email arrived. Typing the number of the email I can read it. Typing ‘q’ will quit the application. All emails can be found in their raw format at /var/spool/mail.

Let’s try and send an email from the second machine (

>telnet 25
Connected to
220 ubuntux.localdomain ESMTP Postfix (Ubuntu)
HELO whatever
250 ubuntux.localdomain
MAIL FROM: whatever
250 2.1.0 Ok
RCPT TO: animal@ubuntux
250 2.1.5 Ok
354 End data with .
Sending from my laptop.
250 2.0.0 Ok: queued as 68DF4133C91
221 2.0.0 Bye

The only difference here is that we connected to ubuntux instead of localhost. Accordingly we had to change the recipient from animal@localhost to animal@ubuntux.

The recipient after “RCPT TO:” can essentially be any user (existing or not) as long as the domain used in the email is the hostname of our mail server (ubuntux in this case). An email with a different domain could be for example animal@randomdomain. The email with such a recipient would get totally discarded from the server and lost.

Taking it a step further – receiving emails from the Internet

At this state, emails can be delivered to the mail server from users in the LAN. We will try and make the mail server to be able and receive emails from the internet as well.


In the illustration above an email is being sent from the gmail server to our own mail server. We don’t care how the email got to the gmail mail server. What we care about is how the email will reach our mail server. For the correct routing of the email we have to configure three things:

  1. Our DNS server
  2. Our router
  3. Our mail server

Configuring DNS

Every time we visit a website, a DNS request has to be made to find out the IP of that website. In the same fashion a DNS request is made by the mail server when an email is to be sent. This request is somehow different, in that it looks for a very specific record; an MX (mail exchange) record.


(1) As seen in the illustration the mail server looks at the domain after the ‘@’ symbol in the recipient of the email.

(2) It then sends a DNS request for the mail server in that domain ( To get an idea of what the answer of such a request can be you can use the debugging tool dig and test it with dig MX

(3) A reply arrives from the DNS server with the IP of the mail server.

(4) The email is forwarded to that IP, and further to the mail server.

Now ideally the MX record in the DNS server would look like this:	IN	MX	10

That is however breaking the standard and therefore it is unacceptable. By defition an MX record should bind a domain to a hostname (or more). Therefore we need to replace the IP in the above example with a hostname:	IN	MX	10
mail	IN	A

Since the hostname (or FQDN to be exact) doesn’t exist, we have to create it with an A record. An A record binds a (made up) hostname to an IP by definition. Since our server is behind the router I use the IP of the router instead (the IP should never be a LAN one). For hostname I use mail but you could use whatever you wish. Just keep in mind that if you change it, the hostname must also be reflected in the MX record – you should also change the FQDN.

Keep in mind that if you do changes to the DNS records, it might take some time for the effects to take place. So continue a bit later, or after a day or two depending on how fast the changes are updated (essentially it depends on the TTL values).

Is your network reachable from outside?

Time to test if the DNS configuration works.

When we ping we want that it pings So let’s test it. Go to and ping your mail server’s FQDN (in this case It’s important that you ping from outside your LAN so that no local cache is used.

If you don’t get a response, try pinging directly your WAN IP ( If you still don’t get a response, then probably your router is blocking ping from outside. If you are getting a response by pinging the IP but not when you ping the hostname, then you haven’t configured your DNS records correctly or they haven’t taken effect yet.

If you run your own DNS server you can check for warnings and errors in /var/log/syslog.

Configure your router

At this point we know that an email can arrive as far as to the router. We only need to configure the router so it forwards the email to the correct machine in the LAN.

Since we know that mail servers use port 25 and if we haven’t changed that, then our mail server also uses port 25, we just have to configure the NAT of our router.

Forward all TCP at port 25 to your mail server (in my case My router’s port forwarding looks as below:


Sending an email to Postfix from gmail

If we can ping the mail server from the internet, then we can assume that our network is reachable. Sending an email to will not work still though as the router won’t know to which computer in the LAN to forward the email to. Setting the NAT of the router to forward all incoming TCP connections on port 25 to the mail server fixes this.

We can now test it by sending an email to from a gmail account. The mail server will reply with an error:

Delivery to the following recipient failed permanently:

Technical details of permanent failure:
Google tried to deliver your message, but it was rejected by the server for the recipient domain by [].

The reason the email gets rejected lies in the Postfix configuration. The configuration file /etc/postfix/ has this (or a similar) line:

mydestination = ubuntux, localhost

mydestination is the variable that defines what FQDN/domains in emails are accepted. In essence all emails arrive at the server, it’s just the depending on this configuration some of these emailes will be discarded and some will be accepted by the server.

With the current configuration we can see that only emails of the format username@megistanas and username@localhost are accepted. Trying to send an email to animal@ubuntux from gmail won’t work as ubuntux just can’t be seen from outside our LAN; it doesn’t resolve to anything. For this reason we want to add one ore more domains that point towards our network. That way, when the an email arrives it doesn’t get rejected.

In my case I add so that I accept all emails with the format

mydestination =, ubuntux, localhost

Restart Postfix and sending an email to should now work.

You could essentially also add as that would take the same route as well. We can add pretty much any domain that points to our router.

Sending an email to gmail or hotmail from Postfix


There are essentially two entries in the configuration file that we have to change to make sending emails to the Internet possible:

smtpd_sender_restrictions = permit_mynetworks
mynetworks = [::ffff:]/104 [::1]/128

smtpd_sender_restrictions can have more options than the one shown. However the only interesting option for us at the moment is to permit anyone from mynetworks to be able and send emails. So make sure permit_mynetworks is amongst the options.

mynetworks specifies local networks where users are considered trusted to use our mail server. For that we have added the which includes all computers in the network. Be careful, you don’t want to have a WAN ip here as that would make our mail server an open relay (I talk more about that below).

This configuration will let anyone from our computer to send an email of the format If we want users of the LAN to be able to use our mail server from the Internet, some sort of authentication has to take place (login or key), but I am will not go further into that.

Let’s test and see if we can send an email to a gmail account:

>telnet 25
220 ESMTP Postfix (Ubuntu)
HELO dickhead
250 2.1.0 Ok
250 2.1.5 Ok
354 End data with <CR><LF>.<CR><LF>
This is a test.
250 2.0.0 Ok: queued as 3D31311F765
221 2.0.0 Bye

If everything went as planned, you should have received an email in your gmail account ( in the example).

Make sure your mail server is not an open relay

You always want to avoid having an open relay. An open relay constitutes that your mail server can be used by anyone on the internet including spammers.


Once spammers find out that your server is an open relay, they will start using it. At some point, an other mail server will find out that you have an open relay and will blacklist your server.

So how do you ensure that you don’t have an open relay? Very simply. In your configuration file pay attention to these two lines:

smtpd_sender_restrictions = permit_mynetworks
mynetworks = [::ffff:]/104 [::1]/128

As long as the networks in mynetworks are LAN IPs and not WAN IPs, your mail server is not an open relay as no-one outside the LAN can use your mail server to send emails.

You can test to see if your mail server is an open relay at

Use a relay host if you can

Our mail server works fine at the moment but it’s very minimal. It doesn’t even filter incoming spam emails. Configuring filters etc. can be time consuming and will need further maintenance all the time. An easy solution to this is to forward our emails to a second mail server, a relay server (keep in mind: not same as an open relay, as a relay server is not open to everyone). So if we have to deliver an email to, instead of delivering to the gmail server directly, we can use a second mail server which we have access to. That server can then do the filtering etc. for us and deliver the email to its final destination.

My ISP is called Bahnhof and by googling around I found that it offers a relay at Most ISPs do offer mail servers that can be used as relays. Luckily for me, I don’t even need to enter a password and username as I am authenticated by my IP or MAC (depends on how you’re connected to the ISP). In your case you might need to authenticate first before using the relay. I won’t go into how you do that. I will however tell you how you can test to see if you can use the relay as a relay server.

If you can send an email directly from the relay server then you can use it as your relay. See the telnet session below:

telnet 25
Connected to
Escape character is '^]'.
HELO dickhead
250 Ok
250 Ok
354 End data with <CR><LF>.<CR><LF>
This is a test.
250 Ok: queued as 0C67C53CEB7
221 Bye

I just connected to my ISP’s mail server and sent an email to a gmail account. Since I got the email delivered and wasn’t asked for a login/password, that means that I can directly use the server as a relay without authentication.

If your ISP’s mail server requires a username and password you’ll have to do some extra work in configuring that. Also keep in mind that in the field “MAIL FROM:” you have to provide an email with a hostname that points to your network. The server will probably check to see if whatever@hostname actually resolves to your network. If not, you will get an error like below:

450 <spamotron@spammer.spam>: Sender address rejected: Domain not found

Now that we know we can use the server as a relay, we simply have to edit /etc/postfix/ and make sure the relayhost is set correctly:

relayhost =

Restart Postfix and you should be able to use your mail server as before with the difference that now all outgoing emails go first through the relay server and the relay server forwards them further to their destinations.

Leave a comment

Mastering sed

sed is a very famous tool to the UNIX* community but which is very often misused. Most people try to use it in cases that it’s not the right tool or they tend to use it in the wrong way. I try in this post to show the things that are worth knowing when it comes to sed. I thus skip things like buffer holders, labels etc. which just make scripts totally unreadable for no benefit. I also talk a bit about the inner-workings of sed so the user has a grasp of why sometimes things don’t work as expected.

Why even learn sed?

For two reasons:

  1. You can automate any boring mechanical work you would do on a normal text editor
  2. If you know the syntax of sed, then you are a better vi/vim user

The second point, becomes obvious when you realise that the two programs have similar if not the exact syntax. For example, substituting the word “cow” with the word “horse” in the third line of the document in vim is :3s/cow/horse/ while in sed it’s 3s/cow/horse/. See the magic?!

A bit of history

Sed, awk and grep are the offsprings of a line editor called ed. ed pretty much let the user to edit one line at a time. That is also the reason that sed, awk and grep work on lines. All three programs have inherited the syntax of ed to some extend. In ed, to search & replace the word “cow” with the word “milk” in a text document, someone would type s/milk/beer/. That is exactly the same command used in sed – an indication of ancestry.
The tool grep actually takes its name from the command g/re/p, a command used in ed to only show lines that contain a specific regular expression. The ‘p’ in the end means to print on the screen while the ‘g’ in front means to go through all the lines.

When to use sed

While all three of these tools (grep, sed and awk) work on lines, sed and awk are very similar to each other while grep is more of a loner. Grep is used merely to filter out (or in) a line based on a regular expression. Sed and awk offer much more. What differs sed from awk is the data that they were built to edit.
Awk should be used when every line in the file has a specific structure. In other words that includes files where each line has a specific number of fields with every field separated with a delimiter (in most cases a tab). Such files can be CVS files, tables, the output of ls in linux, and more.
For everything else, use sed. Common examples are raw texts, this post, a C file, a script, an HTML file, etc. What all these files have in common is that lines don’t have a specific structure: the first line can have one word, while the second can have 100 words etc. Sed can still edit files that awk edits, but the opposite is most times impossible. If you are trying to just do that then you are most probably using the wrong tool.


Sed in the terminal

The common syntax for sed in the terminal is:


The meat of sed is the SCRIPT and that is pretty much what I cover in this post. It’s a good convention to put quotation marks around it in case there is a space inside it (the shell might interpret it as multiple commands then).
sed can have multiple SCRIPTS or it can use a file with commands.

Multiple script lines:


Using a script file:


The SCRIPTFILE should have a command on each line. So SCRIPT1 should be on separate line than SCRIPT2 and SCRIPT3 on a separate etc.

Sed’s script syntax

Sed uses three things to accomplish tasks:

  • line specifiers (address)
  • commands
  • flags

The syntax of a single SCRIPT line is:


A line or line specifier is a way to specify which lines you want the command to affect (parse). If the line specifier is missing, then the command affects all lines, which is the default behaviour.

A command is denoted by single character. For example to replace (substitute) a word with an other word we use the command ‘s’:


A flag is used to modify the <command> or the <line>’s behaviour a bit and is placed after the whole command or line. Using the flag g on the example above, we get:


g stands for global and is used to replace all word1 in the line. Without it, only the first occurrence of word1 in the line is replaced.

A flag goes hand in hand with regular expressions so they can only be used if <command> or <line> have a regular expression in them. If the <line> is specified with a number for example, it’s illegal to use a flag:

sed -n '6g'

The reason is that flags are made for text strings (patterns) so it doesn’t make sense to sed when you are telling sed to use the flag ‘g’ on a line (which is something else than a string) and not a pattern.

The minimum thing needed on a SCRIPT line is a command or a line specifier. Someone can have both, one of them – with a flag or without. These combinations (or permutations to be exact) are allowed:


A flag applies to the command or line before it and assume that the previous has a regular expression in it.

How sed works internally

Imagine we have the file list.txt with the lines:

Today I will drink my milk
and afterwards I will eat a cow.
The cow will taste like cow.

Sed works with lines. As stated earlier we can have multiple script lines seperated with question marks:

sed 's/Today/Tomorrow/g; s/Today/Next Friday/g' list.txt

sed has a working buffer for each line (called pattern space). sed will initially load the first line of list.txt in the buffer. Then it will go through all script lines one by one altering everything in-place (in the buffer). In our example the buffer for the first line initially has:

Today I will drink my milk

After the first script command executes, it becomes:

Tomorrow I will drink my milk

The second script line doesn’t alter anything as sed can’t find an occurrence of the word ‘Today’ since it just got altered.
Once the first line is done, sed will load the second line in the buffer and go through the same procedure until all lines in list.txt have been parsed.

Something important to notice is that each SCRIPT line will run regardless if the previous SCRIPT line succeeded or not. With success we mean that the command did what it is meant to do. If substitution is used, then we define success as the alteration of a line. If we just specify a line, then success is if the line exists etc.

Addressing specific lines

By default, sed goes through all the lines. However, one can address a specific line or a range of lines. That can be done by specifying lines by their number in the file (1st line, 2nd line, 50th line etc.) or by a line’s content.

To run a command on the 10th line we do:


To run a command on each line that contains the word “cow” we do:


The latter makes use of regular expressions. When we use regular expressions we need to add slashes to the start and end of the regular expression.

For a range of lines we use a comma (awkward, I know). To specify all lines between the 10th and 25th line (including those) we would write:


If we want to specify a range of lines by using regular expressions we still have to encapsulate the regular expressions in slashes. To run a command on all the lines between the first line found with the word “cow” and the first line found with the word “grass”, we would issue:


Bellow you can see all the ways for specifying lines.

Lines between <line1> and <line2> (including those)
Every Nth line after <line1>
All lines except line1
Lines matching the regular expression
Last line
All N number of lines after <line1>

Some more practical examples can be seen bellow. The command p is used to print the line that is specified. (Notice that sed needs the parameter -n for the command p to work.)

sed -n '2p'             -> print line 2
sed -n '2,4p'           -> print line 2 to 4
sed -n '$p'             -> print last line
sed -n '2!p'            -> print all lines but line 2
sed -n '/red/p'         -> print every line that contains the word red
sed -n '/red/,/green/p' -> print all lines between the <strong>first occurrences</strong> of the words 'red' and 'green'

Of course for all these examples to work you need to either feed sed with a stream from a file or a pipe.

Using commands

Now we are to the meat of all meats.I have explained the substitution command a bit but bellow you can see all the commands with their syntax (if they have one).

s/<regex>/<subst>/ substitute Replace a match (using regular expression) with a string.
p print Prints a specific line or a range of lines. The sed flag -n should be used for this command to work.
= line number Shows the number of the line
d delete Deletes (omits) the matched line
y/<char1>/<char2>/ transform Substitutes char1 with char2. Works even with a sequence of characters. For example y/abc/ABC/ will replace a with A, b with B and c with C.


Substitution is the most commonly used command. It’s syntax is as follows:


where <d> is a delimiter which should be a single character. Most commonly the delimiter is a slash / but it can essentially be any character. All lines bellow are equivalent.


In the example above the first occurrence of the word cow on each line will be replaced with the word horse, which is the default behaviour. If you want all occurrences of the word cow in a line to be substituted, the flag g (global) has to be appended to the line:


The substitution field <subst> can take some special variables like the ampersand symbol “&” which holds the matched string from the regular expression. There are also a few macros to automate conversion of capitals to lower-case and vice versa.
All these are shown in the table bellow.

& Holds the matched string.
\1 Holds a part of the match specified in the regular expression with parentheses.
\U<string> Converts all letters in <string> to capitals.
\u<string> Converts the first letter in <string> to capital.
\L<string> Converts all letters in <string> to lower-case.
\l<string> Converts the first letter in <string> to lower-case.
<string>\E Ends the conversion at specific point. Should be used in conjunction with \L and \U.

The match holders \1 \2 \3 etc. have to be specified in the regular expression with parentheses. The parentheses themselves have to be escaped or else sed will be looking for parenthesis characters in the line. So to replace each word “cow” with “supercow” we can do:




The latter is more elegant of course. However there are two cases where the specific holder has to be used:

  1. When we want only a portion of the matched string and not the whole string (&).
  2. When there are many different strings you want to grab from a match.

For example this can’t be solved by merely using the & symbol:

s/\(\w*\) cow \(\w*\)/\2 cow \1/

This script will look for each occurrence where “cow” has a word before it and a word after it and it will change their order. The \w matches any character while the asterisk * tells the pattern that the word can be arbitrarily long. We use the parentheses around the first word and the second word to denote which matched parts of the regular expression should be given to \1 and \2. In <subst> we just reverse their order by putting the second word first and the first word second.


A flag can be used on a <command> or a <line> or both.

g global For all occurences in the line (default is to stop at first occurence)
I Ignore case Not case-sensitive
p Print Output only this line (not everything as default). The sed flag -n should be used for this flag to work.

Find line in lines of lines in lines..

An important concept to comprehend in sed is nesting. I try to leave out all the advanced things sed offers, like holder buffer (horror to read), labels etc. but nesting is worth learning as it gives a lot of extra power for a little learning curve. Nesting is similar to the IF..THEN conditional.

We have this text file:
Today I will drink my milk
and afterwards I will eat a cow.
The cow will taste like cow.
Today is not afterwards if I am a cow. Right?

Imagine that we want to check if the last line of the text contains the word cow. Someone might think that putting $p together with /cow/p would work:

sed -n '$p; /cow/p'

Admittedly the output seems strange:
and afterwards I will eat a cow.
The cow will taste like cow.
Today is not afterwards if I am a cow. Right?
Today is not afterwards if I am a cow. Right?

The reason this doesn’t work as expected is that the second script runs regardless if the first one succeeded or not. Thus in the first line none of the scripts print anything. On the second line, the first script fails but the second script finds the word cow so it prints the line. The same happens at the third line. At the forth line, the first script succeeds as the line is the last line of the file, so that line gets printed. Then the second script runs and that succeeds. Thus we get the line printed again (a second time).

A way to solve this is is to nest the second script line in the first somehow so that it runs only if the first script line has succeeded. This is similar to the pseudocode:

if <line1>
  then SCRIPT

Where <line1> is a line specified by a number, range or pattern (regular expression).
The syntax for nesting script lines in sed is


For our example:

sed -n '${/cow/p}'

This translates to: if this is the last line ($) then do whatever is in the brackets. So everything in the brackets will be checked only if the current line being parsed is the last one.

We can even nest inside a nested script:

sed -n '2,4{/cow/{/Today/p}}'

This script will go through lines 2 to 4. It will check first if a line contains the word cow. If the line contains the word cow, then it will check if it contains the word Today. If it does, it will print it. Ofcourse there is no practical reason to have second braces in the above example. I just wrote it to show that it’s feasible. In some cases you need braces to accomplish tasks, especially in cases where we avoid using too advanced things.


Printing a specific line

sed -n '2p'

The p in this case is the p command (and not flag). Remember that flags apply only to regular expressions. When we use sed’s -n parameter, only lines specified with the p command or flag will be printed on screen.

Printing a range of lines

sed -n '1,3p'

The comma is used to specify a range. In this example we specify line 1 to line 3 and then we print each such line.

Hiding a specific line

sed -n '2!p'

This is similar to:

sed '2d'

Show the last line

sed -n '$p'

In regular expressions the dollar sign $ denotes the end of a string. In sed when it’s being used with substitution it denotes the end of a line.
However when used with the command p, it denotes the last line of the input. In the same way the regex symbol ^ denotes the first line.

Converting a specific word to uppercase

sed 's/Today/\U&/' list.txt

This will match any word ‘Today’ and replace it with ‘TODAY’. In the replacement the escaped U (\U) tells sed to convert to uppercase everything following in the replacement string. In this example we use an ampersand which in sed represents the matched string. If we wanted to stop the conversion we just need to add \E where we want it to end.

Converting all text to uppercase

sed 's/.*/\U&/'

For this we use the substituion command s.
.* is a regular expression which fits any sequence of characters. As sed is working on lines, .* matches a whole line each time.

Converting all text to lowercase

sed 's/.*/\L&/'

Similar to the previous example with only difference that we use \L instead of \U.

Grabbing all content between the body tags in HTML

Say we have the ugly HTML code bellow and we want to grab all the content between the body tags.

<h1>h1 outside of body</h1><body><h1>h1 stuck to body</h1>
<img src="images/soon.png"/>
<p>a paragraph</p></body></html>

Most sed experts would go about using some very advanced commands to accomplish this. The readability of that becomes horrific. As my opinion is that you can do the same things without knowing all the advanced commands I am going to just do that.
Notice that I use pipes instead of using advanced commands.


sed -n '/<body>/,/<\/body>/p' | sed 's_.*<body>\(.*\)_\1_; s_\(.*\)</body>.*_\1_'

It might seem like a mess but I can assure you that it’s much more elegant than a pure single sed SCRIPT solution. I will break it down so you can see how it works.


matches all lines between the body tags, including the body tags. In this way I have minimized my problem to:

<h1>h1 outside of body</h1><body><h1>h1 stuck to body</h1>
<img src="images/soon.png"/>
<p>a paragraph</p></body></html>

After that, we are sure that the first line has the start of the body tag and the last line has the closing of the body tag. So we start by filtering out things we don’t need from the first line: the tag itself and everything preceding it.


I use _ as a delimeter instead of slashes to make the substitution code a bit more readable. The regular expression .* matches 0 to infinite number of arbitrary characters. So I use it around the body tag in case there is something before and after it. I put the second .* in parentheses to grab the text that might be there as I want to keep that. Using \1 in the substitution field I accomplish substituting the whole line with the text after body.


We do something similar to the line with </body>. The only difference is that now the portion of the match that we are interested in is the one before the body tag so we move the parentheses there.

Keep in mind that this solution will not work if the body tag includes some attributes like style. Someone might think that just using the bellow regular expression in the substitution would work.


Notice that the only difference is that we added the regular expression .* between body and its closing arrow > to point out that there can be nothing in between or there could be some arbitrary things (in our case attributes).

That regular expression doesn’t work however as it matches the last > in the line. The reason is that in sed .* is greedy and there is no way to make it non-greedy. With greedy we mean that the pattern will try to match as much as it can in the line. So if you want to match the first > it’s not possible. Or.. actually it is possible but the code as you will see in the next example starts looking like a monster.

Grabbing all content between tags in HTML

You are probably better off learning Perl or Python if you need to do these kind of “advanced things”. I will however show that you can achieve things like this without using the more advanced commands or other programs/languages. This solution is a continuation of the previous example on catching the content between the body tags. The mere difference is that in this solution we allow even attributes to a tag and are a bit more permissive towards whitespaces. This makes it a more general solution that can be used for other tags than the body tag.


sed -n '/<body>/,/<\/body>/p' | sed '1s_.*<body[a-zA-Z0-9="\x27_ ]*> *<\(.*\)_<\1_; 1{/<body>/d}; s_\(.*\)</body>.*_\1_'

Essentially the only thing that got changed from the previous example is the alteration of the commands in the first line:


to two commands:

'1s_.*<body[a-zA-Z0-9="\x27_ ]*> *<\(.*\)_<\1_; 1/{/<body>/d};'

The first script line (everything before the first ;) looks for a second tag after body. We don’t really need to specify line 1 but it is a good convention as it makes the code easier to understand and the processing faster. I am looking for the body tag followed by an attribute or not. TO define an attribute in HTML, only the characters in the brackets are allowed (says the HTML protocol, not me). \x27 is the code for a single quote mark . The reason I use its code instead of the mark itself (I use the double quote after all), is that I use the single quote marks around the whole command so if I insert it in the expression, then it will break it. After that I use ” *<" (notice the space) to denote that there might be an arbitrary number of spaces or none before the opening of the new tag. I replace everything with the opening of the tag after body with the rest of the line.

If the first command didn’t succeed then it means that there’s not a second tag after the body. Thus it’s safe to delete the whole line in that case. First we check if there is a body tag in the first line. If there is a body tag (/<body>/), then we delete it with the command d.

Leave a comment

Converting a URL string into Json

I was making a website the other day and I wanted to somehow pass variables that can be read with javascript. So if the user browsed to, the variables height and width should be read from javascript. Note that this method of passing the variables in the URL is used most notably for CGI, aka server-side scripting(PHP anyone?).


So I was a bit puzzled while looking around at stackoverflow as many people do think that there is something magical about Json. Json is nothing more than a standard on how things are stored. The standard pretty much sums up to this:

  1. Variables are stored as varName = value
  2. Arrays are stored as arrayName = [value1, value2, value3 .. ]
  3. A value can be: a number, a string, true, false, none
  4. The whole thing is encapsulated in wavy braces

For my example, the Json structure(after parsing the URL string) should look like this:

   "height": "100px"
   "width" : "50px"

In this case the values are strings. Keep in mind however that according to the standard, they could be anything between a number, a string, true, false and none.

My parser

So to get this structure from the URL, I needed some kind of parsing. All the solutions I found either used regular expressions, or they wanted a whole library to be imported, or just didn’t support arrays. So I made up my own ugly solution.

The function url2json() uses pure JavaScript code which doesn’t use regular expressions and accepts both arrays and variables:

function url2json(url) {
   var obj={};

   function arr_vals(arr){
      if (arr.indexOf(',') > 1){
         var vals = arr.slice(1, -1).split(',');
         var arr = [];
         for (var i = 0; i < vals.length; i++)
         return arr;
         return arr.slice(1, -1);

   function eval_var(avar){
      if (avar[1].indexOf('[') == 0)
         obj[avar[0]] = arr_vals(avar[1]);
         obj[avar[0]] = avar[1];

   if (url.indexOf('?') > -1){
      var params = url.split('?')[1];
      if(params.indexOf('&') > 2){
         var vars = params.split('&');
         for (var i in vars)

   return obj;

To keep things clean, all values are parsed into strings. As the input of the function is a string it just makes sense to give back strings so no extra processing takes place if it’s not needed(checking if every value is of a certain type). It’s up to the user to convert the strings into numbers or whatever they want, if they really have to.

Parsing variables

To use the function with the example above, I would just run

obj = url2json(";width=50px

Launching the console in the browser(CTRL+SHIFT+K), we get the results:


Parsing arrays

Arrays are parsed like this

obj = url2json("[100,45,88,90]&mixed=[red,56,blue,20]");

The object logged in the console looks like this:

   "numbers" : ["100", "45", "88", "90"]
   "mixed" : ["red", "56", "blue", "20"]


Emailing from a gmail acount via Telnet

Telnet is a small amazing tool that can be used for pretty much everything concerning networks. Telnet sends ASCII strings directly to a host. So as long as a protocol supports ASCII messages, we can communicate with a host with any protocol we want. Some examples are SMTP and HTTP.

In practise this method is being used to test things. If you are developing a web-server for example it would be easy if you could test your web-server directly without having to launch a browser and type a command. It’s a blessing for automation.

Here we will try to send an email by using the SMTP protocol. We will use telnet for that, by sending and receiving raw messages with the SMTP server.


First we need to install telnet with SSL support. All SMTP servers use SSL encryption nowadays so using simply telnet will not work. The telnet shipping with Ubuntu does not support SSL so we need to upgrade it:
sudo apt-get install telnet-ssl

To run the new telnet-ssl we just issue telnet in the terminal as before. The difference is that telnet now supports some new flags, namely -z, which is used for SSL.

We are also going to need to encode our username and password to base 64 format. Keep in mind that base 64 does not encrypt a message. It just changes the way the information is stored. In practise we use base 64 to be able and send any kind of data as simple characters. The real encryption is under the hood however. Check Extras for more info.


  1. Start telnet
  2. telnet -z ssl 465
    The flag -z ssl tells telnet to use SSL over the connection. is Gmail’s server domain and 465 is the port used by the server.

  3. Handshake with server
  4. HELO yo
    The text after the HELO is supposedly your domain or literal address. Essentially you can enter whatever you want. Pressing enter, the exact message arrives at the mail server, which in response should reply with
    250 at your service

  5. Login on gmail
    This message is self-explanatory. After the server receives this message, it’s going to ask for our username. The response looks like this
    334 VXNlcm5hbWU6 and indicates that the username should be entered.

  7. Username
  8. Go to and encode your username ( to 64 base. Copy and paste the encoded string in your telnet session and hit enter. If the username is accepted, you get back the message bellow.
    334 UGFzc3dvcmQ6
    This message indicates that the password should be entered.

  9. Password
  10. Do the same as above but instead of your username, encode your password. Then paste the password and hit enter. If the username/password combination is correct, you are greeted with the message
    235 2.7.0 Accepted

  11. Email sender
  12. First we are going to add the sender of the email:
    MAIL FROM: <>
    For gmail it doesn’t really matter who you set as sender. We logged in our gmail account when we sent the AUTH LOGIN message to the server. Gmail has as a mechanism to set the sender automatically to your email address.
    The response back is
    250 2.1.0 OK js17sm40481494lab.5 - gsmtp

  13. Email recipient
  14. Now we will add the recipent of the email. Here we need to provide a valid email or else the email will not go any further. If the email doesn’t exist, we will probably get it back into our inbox which is the default behaviour for undelivered emails.
    RCPT TO: <>
    On success we get
    250 2.1.5 OK js17sm40481494lab.5 - gsmtp

  15. Email body
  16. Now we want to start typing the email. We send the following command to the server so that the server knows that what follows is the actual email.
    The response from the server is
    354 Go ahead js17sm40481494lab.5 - gsmtp

    Now we need to set the subject(if we want).
    Subject: test
    and everything else following is considered the text of the email

    This is a line in the email.
    This is a second line in the email.

    The dot on its own line in the end is to signal the end of the email. After that, a response from the server should come:
    250 2.0.0 OK 1381416452 js17sm40481494lab.5 - gsmtp
    Now the email has been sent.

  17. Close telnet
  18. QUIT
    This is an actual command to the telnet client and not a message sent to the mail server.



EHLO can be used instead of HELO. The difference is that EHLO lists also all the commands that the server supports. EHLO is newer than HELO and is suggested to be used instead of EHLO. However HELO is always going to be supported, so in reality there is no big difference as long as the server supports the message.

SSL and encryption

Something important, as mentioned in the beginning of this page, is that encoding data to base 64 is not encryption but rather a different representation of the same data. It’s SSL which adds the encryption to our communication and you can be assured that everything is encrypted between the client and the server.

I captured the whole procedure of sending an email, with Wireshark to prove my point.
Screenshot from 2013-10-10 17:33:52_crop

The packets are in the order they were sent and received. We can see that telnet first contacts the DNS server to get the domain name of the mail server. is my router and is my computer from where I use telnet.

After we get the domain we do a first handshake with the mail server. It’s those three TCP messages with SYN/ACK in them. After that we initiate a TLS session. TLS and SSL are pretty much the same thing. They are just different versions for encrypted communication, with TLS being the newer one.

Everything we type and read from the time we start telnet is encrypted. The reason we see the messages clearly in the terminal is that they get decrypted by telnet on their arrival. When messages leave telnet, they get encrypted to travel on the wire. That’s why with Wireshark we only see the decrypted data, because we are looking at what is passing through the wire. I circled the TLS messages to make it more apparent that the whole session is encrypted by using TLS. The few TCP packets seen here and there, are packets with encrypted data. If you could check what these packets hold inside(with Wireshark), you would only see gibberish.

If instead of Gmail we were using an SMTP server that doesn’t use SSL, we would be able to intercept all the packets with their raw data, and thus the actual messages, passwords, usernames, etc.

Leave a comment

Traversing binary trees

It can be hard to remember how to traverse trees by name. It can also be hard to understand the difference between traversing in different ways(postorder, preorder, inorder, level order, etc). I try here to make it easier for the reader to understand the different ways of traversing a binary tree. Intuition is the key to remembering(and visualization ofcourse).

I use the words parents and children for the elements in the tree instead of nodes, root, branches and leafs. I think those words are easier to describe what is going on without getting anyone too confused.

Recursive traversal

I want to firstly point out that this method is commonly called “depth/height traversal”. However I use the word recursion as it seems more appropriate to me.
There are three ways to traverse a tree recursively. The only real difference between them is the order we visit the parent node.

  • Pre order ——- here we visit the parent in the beginning
  • In order ——— here we visit the parent second
  • Post order —— here we visit the parent lastly

Because recursion is hard to grasp and even harder to have a visual insight of the goings, we will use a simple example to begin with. We start by examining, with all three methods, this binary tree:

Pre order

Here the path we follow to traverse is:

  1. Parent
  2. Left child
  3. Right child


Applied to our example tree:

So if we use pre-order to traverse the example binary tree, then we would get the values(by order we visited them):

4, 2, 7

In order

Here the path we follow to traverse is:

  1. Left child
  2. Parent
  3. Right child


Applied to our example tree:

So if we use in-order to traverse the example binary tree, then we would get the values(by order we visited them):

2, 4, 7

Post order

Here the path we follow to traverse is:

  1. Left child
  2. Right child
  3. Parent


Applied to our example tree:

So if we use in-order to traverse the example binary tree, then we would get the values(by order we visited them):

2, 7, 4

The bigger picture

All this seems simple for a tree with only three nodes. But what happens when we have a huge tree with multiple nodes? Which node would we visit first?

The answers are rather simple but there are not good resources actually to give a good visualization of how a recursive traversing can look like.

So let’s start by taking this big tree:

The trick is to start by seeing see the whole structure as a tree with three nodes. In all three traversing methods we start with the node on the top of the tree: the root node. We thus see that as the parent in the beginning of the traversing. All the rest of the nodes on the left is considered the left child and everything on the right of it is considered the right child.
The bigger complex tree has been compressed to a three-node tree. We can now start traversing it with whatever -order traversal method we want.

I will demonstrate how I would go with in-order traversal. Read this line by line.

  1. First I visit the left child: node b
    1. I now get into this subtree and visit the left child: 5
    2. Then I go to the parent node: 2
    3. Then I go to the right child: 9
  2. Then I visit the parent: node a
    1. Here there is no parent or children. The only thing I can do is to return the value of the single node: 4
  3. Then I visit the right child: node c
    1. I now get into this subtree and visit the left child: 6
    2. Then I go to the parent node: 7
    3. Then I go to the right child: 1

Note that 1, 2, 3 are traversal of the a, b, c nodes. All nested traversals are traversals for the nodes inside a, b and c.

An even more complicated tree could look like bellow. However exactly the same principles are valid.
This looks a lot like a fractal, doesn’t it? That’s because fractals are actually based on recursion. The only difference with this example is that the recursion is not endless but instead is made of three different levels: the outer level where we see three cyan circles, the level where we see three green circles and the level where we see three nodes(inside each green circle).

Traversal by Breadth

All methods discussed earlier are actually what is called traversal by height or depth. I am not so sure as to why we call it that instead of traversal by recursion as the real difference between depth traversal and breadth traversal is:

  • Height traversal uses recursion.
  • Breadth traversal is linear, going from one node to the next as you see the whole tree.

Traverse by level

Traversing by breadth or by level(which is more intuitive) is the simplest method to traverse a tree. You can see the nodes of an arbitrary tree grouped into different levels depending on how close to the root they are. In the bellow tree I have numbered the different level of nodes:
Node with value 4(the root) belongs to level 1. Nodes 2 and 7 belong to level 2. Lastly, nodes 5, 6, 9 and 1 belong to level 3.

To traverse the tree with traversal by level we just go from left to right on each level jumping from node to node like this:



Thus we traverse the nodes in this order:

4, 2, 7, 5, 9, 6, 1

If the tree would be more complicated or not complete(missing nodes at some level) then we just jump over the missing nodes and get to the next one. An example follows bellow.




The nodes visited in order:

4, 2, 7, 5, 9, 6, 1, 26, 13, 10, 8, 3