This guide will show you how to run a GUI application headless in a Docker container and even more specific scenarios involving running Firefox and Chrome. If you are not interested about those then you can just stop in the middle of this tutorial.
What the hell is X?
X is a program that sits on a Linux machine with a monitor (so servers usually don’t use X). X’s job is to talk to the Linux kernel in behalf of GUI programs. So if you are playing a game for example, the game (that is, the application) is constantly sending drawing commands to the X server like “draw me a rectangle here”. X forwards all this to the Kernel which will further forward the information to the GPU to render it on the monitor.
X can even receive commands from the keyboard or mouse. When you click to shoot on your game for example, the command “click at 466,333” is sent from your mouse to the kernel, from the kernel to the X and from X to the game. That way the game can have a clue on what is happening!
You will often hear X being called a server and the reason for that is simply because the way the applications send commands to it is through sockets. For that reason the applications are also referred to as clients many times.
If you are reading this then the X is running on your PC. Let’s prove it:
We can see that X is running as root and has PID 1436. An other important thing is to notice the :0 which is called display in X jargon. A display is essentially:
A monitor
A mouse
A keyboard
And this is the bigger picture of how it all looks together:
Now there is a variable in Linux that is used whenever we run a GUI program. That variable is surprisingly called DISPLAY. The syntax of the DISPLAY variable is
<hostname>:<display>.<monitor>
. Let’s check the DISPLAY on our computer:
> echo $DISPLAY
:0
I get :0, which means we use display 0. Notice however that this says nothing about which monitor we use. This makes sense since if you are running 2 or more monitors on your Linux you still have the same environment variables in both of them. It wouldn’t make sense that an environment variable changes just because you echo it from a different screen, would it? For that reason we get the display and not the monitor so that we get the same output on both. As about the hostname, since there is no info about it, the local host is assumed.
On a notice, if you have multiple monitors you can still specify which monitor to run an application by simply typing the full display variable you want. So if you have a monitor 0 and a monitor 1 on the current display, I can run firefox on monitor 1 with:
DISPLAY=:0.1 firefox
Creating a virtual monitor
Instead of running X, we can run a different version of it that can create virtual displays. Xvfb (virtual framebuffer – whatever the hell that means) will create a virtual monitor for us.
So let’s make a new monitor (I assume you have installed xvfb):
Xvfb :1 -screen 0 1024x768x16
This will start the Xvfb server with a display 1 and a virtual screen(monitor) 0. We can access this by simply typing DISPLAY=:1.0 before running our graphical program. In this case the program will start in the virtual screen instead of our monitor.
We see we have the normal display 0. (A way to tell it is the default screen is to see that it runs as root.) We can also see the second display :1 and screen 0 with resolution 1024×768. So what if we want to use it?
Open a new terminal and type:
> DISPLAY=:1.0 firefox
..
This will start firefox at the given display. The reason I use the DISPLAY at the same line is to make sure that the subprocess inherits the variable DISPLAY. An other way to do this is to type:
> DISPLAY=:1.0
> export DISPLAY
> firefox
..
Run a GUI program in a Docker container
We will now create a virtual screen inside a docker container.
So now we are sure that we are running the virtual screen. Let’s access it and run something graphical on it. In this case I will run Firefox and Python+Selenium just as a proof of concept of what is happening.
First I put my display variable and use export to assure that any sub-shells or sub-processes use the same display (with export, they inherit the variable DISPLAY!):
root@660ddd5cc806:/# firefox
(process:14967): GLib-CRITICAL **: g_slice_set_config: assertion 'sys_page_size == 0' failed
Xlib: extension "RANDR" missing on display ":99.0".
(firefox:14967): GConf-WARNING **: Client failed to connect to the D-BUS daemon:
//bin/dbus-launch terminated abnormally without any error message
..
The errors don’t mean anything. But we can’t be sure, can we? I mean, since we can’t see what’s happening it’s really hard to tell. There are two things we can do, either use ImageMagick to take a snapshot and send it to our host via a socket or we can simply use Selenium. I will do that since most people probably want to achieve all this for testing purposes anyway.
If you get a bunch of HTML, then we have succeeded!
The Chrome issue
If you try and run Chrome in a Docker container, it won’t work even if you have setup everything correctly. The reason is that Chrome uses something called sandboxing. Reading this I could not let but notice the word jail. Apparently it seems that Chrome uses Linux containers (the same that Docker uses). For this reason you have to put a bit of extra effort to solve this issue since because of technical difficulties it’s not possible to run containers in containers.
I have now installed Selenium, Chrome and Xvfb. Now I am going to make make a virtual monitor and run Chrome:
root@7dd2c07cb8cb:/# Xvfb :99 -screen 0 1024x768x16 &> xvfb.log &
[1] 6729
root@7dd2c07cb8cb:/# DISPLAY=:99.0
root@7dd2c07cb8cb:/# export DISPLAY
root@7dd2c07cb8cb:/# google-chrome
Xlib: extension "RANDR" missing on display ":99.0".
Xlib: extension "RANDR" missing on display ":99.0".
[6736:6736:1017/143449:ERROR:desktop_window_tree_host_x11.cc(802)] Not implemented reached in virtual void views::DesktopWindowTreeHostX11::InitModalType(ui::ModalType)
ATTENTION: default value of option force_s3tc_enable overridden by environment.
failed to create drawable
[6775:6775:1017/143449:ERROR:gl_surface_glx.cc(633)] glXCreatePbuffer failed.
[6775:6775:1017/143449:ERROR:gpu_info_collector.cc(27)] gfx::GLContext::CreateOffscreenGLSurface failed
[6775:6775:1017/143449:ERROR:gpu_info_collector.cc(89)] Could not create surface for info collection.
[6775:6775:1017/143449:ERROR:gpu_main.cc(402)] gpu::CollectGraphicsInfo failed (fatal).
[6775:6775:1017/143449:ERROR:sandbox_linux.cc(305)] InitializeSandbox() called with multiple threads in process gpu-process
[6775:6775:1017/143449:ERROR:gpu_child_thread.cc(143)] Exiting GPU process due to errors during initialization
[6736:6736:1017/143449:ERROR:gpu_process_transport_factory.cc(418)] Failed to establish GPU channel.
It seems that it works. It’s normal that we get the gpu errors since we don’t have a gpu! However I don’t like gambling so we will take it a step further to check that the browser actually works. However for this I will need to download the webdriver for Google Chrome..
If you are mingling with docker and containers on servers, you’ll at some point want to have two containers communicate with each other.
For example if you have an Apache server, Django+Python and a MySQL server and you want to do tests on different configurations/versions of these, you should probably go the container-way: putting each of them on their own container. However since all these services are interchanged in some way (database has to communicate with web server, apache server has to communicate with django), the containers have to talk with each other somehow.
Possible solutions
There are essentially four ways to accomplish container-to-container communication:
Having containers inside containers
Using volumes (shared folders)
Mapping ports (communication over TCP/IP)
Linking containers (communication over.. TCP/IP?)
I will completely omit the first way since it’s a brainfuck both conceptually and technically (yes, I am aware of Dind). Been there, done that, and it wasn’t pretty.
The second solution is about sharing folders between host and container or between containers by using docker’s --volume flag. This is practical when you want to share files (source code, configuration files etc.) on a local machine. If you want containers on different hosts to communicate however this won’t suffice.
The two remaining solutions is the meat I will try to break down for you (and minimize your flatulence from the learning curve). I did my research and I will do my best to convey what I came down to. But enough talk, let’s start!
Terminal to terminal speaking
I will use netcat to show how all these things work. To remind you, netcat is a Unix tool similar to cat but supposed to work on the network (thus the name net-cat). Practically it’s used when you have to copy/paste from host to an other host, chat with someone fast, or just test communication between two hosts like we do here.
You can test it on your home computer. Open two terminals and type in one of them
netcat -l 8000
and on the other
netcat 127.0.0.1 8000
Whatever you type on one terminal now, will appear on the other one.
Terminal to container speaking
Let’s take this a step further, by containerizing the listening netcat. In a new terminal type:
docker run -i -n -p 9000:2000 ubuntu netcat -l 2000
that creates a new container with an internal port 2000. The port 9000 is the port on our system. Whatever arrives to that will automatically be forwarded to our container’s port 2000. This is somehow similar to port forwarding on home routers. (I explain more on how this actually works later.)
The flag -n is a deprecated flag for networking. In practice without the -n flag, everything will work as before but for some weird reason there will be an initial lag for the first netcat message. So if you don’t use it, just wait a bit longer.
We should now have a container running a “listening” netcat. Let’s open a terminal (on the same host) to try and talk with it:
netcat 127.0.0.1 9000
Or from a different host:
netcat <computer IP> 9000
Keep in mind that different containers on the same machine can have the exact same internal IP but not the same system one. So you can create a bunch of containers as long as you change the system’s IP:
docker run -d -p 9000:2000 ubuntu netcat -l 2000
docker run -d -p 9001:2000 ubuntu netcat -l 2000
docker run -d -p 9002:2000 ubuntu netcat -l 2000
It’s all legit baby!
Container to container speaking
We take this a step further by having the listening container as before but instead of opening a terminal to connect to it, we will create a new container on a different host.
So create a new container on the first host (same command as before)
docker run -i -p 9000:2000 ubuntu netcat -l 2000
Then we create a new container on the second machine (or same):
docker run -i -t ubuntu bash
root@879ad5d4251a:/#
On the new prompt that we get we can netcat directly to the first host:
root@879ad5d4251a:/# netcat 10.2.202.156 9000
This should work as a charm as long as you use your LAN IP.
There are some weird networking behaviours. For example if you netcat by using the IP of the container (eth0 inside container) there is an initial lag but after some seconds the message arrives. The same behaviour occurs if you omit the -n flag but use the LAN IP (which otherwise works). If I start a bash session in the container and listen to 9000 with netcat there, then it works flawlessly. Weird stuff indeed.
A whole network under the city
I omitted quite some information in an effort to jumpstart you. This information is however crucial if you want to use port forwarding and container communication on different hosts. There’s a concept that will save yourself some headaches:
There is a whole virtualized network when we use Docker.
Let me prove that to you. I create two containers and then I check the IPs of them and the IPs of the host. I do that by running ifconfig on each terminal.
Putting it all down sums up to the below picture.
You see, Docker creates a virtual IP for each container we create. Furthermore on our host we have a virtual interface called docker0. That can be considered the router. Now on the same host we are able to communicate with any container if we know their IP.
Now, if a container is listening on a specific port we are able to communicate with it by knocking on its door with a pair of IP address and port number.
Let’s use this new knowledge on the previous example with netcat. Let’s assume that container 1 is listening on internal port 2000 and system port 9000 (like before):
The significant information here is that we can directly speak to the container if we know its IP. Pay attention also to that we need to use the correct port in that case.
Container to container speaking via links
So what are those so called links? Nothing special actually. It’s just some variables passed from one container to an other. In most cases these variables will hold things like IPs, ports, etc. Thus it’s just a step of simplifying things for us – in the end we will probably just use these variables to connect as before.
Let’s see how it works. Start by creating a first container that will act as a server:
docker run -i -t -p 8000:8000 --name myserver ubuntu bash
and then a second container that will act as a client:
docker run -i -t --name myclient --link myserver:myserver ubuntu bash
Nothing special huh? Well the magic happens if you inside the client (second container) run:
You will successfully ping the other container. $MYSERVER_PORT_8000_TCP_ADDR is simply a variable set by docker on the newly created container, the one with the --link flag. You can see all variables with the command env. Depending on what flags you passed to the container, you will see appropriate things. In practice, if you don’t use the -p flag, you won’t see anything interesting (useful).
The --name flag is needed when we use linking as we need an identifier for the container. When we do the linking we need pass the name of the other container, the one that we want to link to and an alias for it. The alias is used merely as a prefix for the variable names. For example if we used --link myserver:dingdong instead of --link myserver:myserver then the variable above would be $DINGDONG_PORT_8000_TCP_ADDR.
That’s pretty much all to linking! By using these variables you can get the IPs and ports of other containers on the same host. That’s an important annotation. You can only link to a container running on the same hosts. For communicating to containers on distant hosts you’ll need to use port mapping as explained earlier.
A few days ago I wrote a tutorial on how to setup docker and use it between different machines. Now that was a nice first insight on how to jump-start using docker. It was also a nice way to showcase the possibilities and limitations of Docker.
In this post I will give some practical information on how to use docker as a developer.
Setup
To use Docker for development of software we want mainly three things:
Have our source code on the host machine. That way we can use GUI editors and whatever tools we want from outside the container.
Be able to have multiple terminals to the same container. This is good for debugging
Setup a docker image which we will use for running our program. I will use Django and Python for that.
And for the visual brains out there:
As you see in the pic, I am using Ubuntu as my host machine. At the same machine I have a folder with the source code and two terminals. Then I run a container with OpenSUSE. The folder and terminals reside ont he host machine but they communicate directly with the container. I will describe below how to achieve all this.
Multiple terminals
The easiest way to have multiple terminas is to use a small tool called nsenter. The guide can be found at https://github.com/jpetazzo/nsenter but it sums up to running this one-liner from any folder:
> docker run --rm -v /usr/local/bin:/target jpetazzo/nsenter
That installs nsenter on the host machine. After that, we can use it directly. So let’s try it. Open bash in a container with ubuntu as our basic image:
> docker run -t -i ubuntu /bin/bash
root@04fe75de21d4:/# touch TESTFILE
root@04fe75de21d4:/# ls
TESTFILE boot etc lib media opt root sbin sys usr
bin dev home lib64 mnt proc run srv tmp var
In the terminal above, I created a file called TESTFILE. We will try to open a second terminal and check to see the file from it.
To use xsenter we need the process ID of the container. Unfortunately we can’t use ps aux but rather have to use docker’s inspect command. I open a new terminal and type the below
If I now create a file inside /home/manos/myproject the change will be reflected from inside the container and vice versa. Play a bit with it by creating and deleting files from either the host or from inside the container to see for yourself.
Create a user in the container
It is wise to have a normal user in your image. If you don’t then you should create one and save the image. That way the source files can be opened from a normal user on your host – you won’t need to launch your IDE with root privileges.
> adduser manos
..
Follow the instructions and then commit your image. That way whenever you load the image again, you will be having user manos. To change to user manos just type
> su manos
All files you create now, will be accesible by a normal user at the host machine. Something else you could do is to somehow
Real life scenario: Python, Django and virtualenv
I wanted to learn Django. Installing Django is commonly made with the package manager pip, but pip has a bad history of breaking up things since it doesn’t communicate with Debian’s apt. So at some point if you installed/uninstalled python stuff with apt, pip wouldn’t know about it and vice versa. In the end you would end up with a broken python environment. That’s why a tool called virtualenv is being used – a tool that provides isolation. Since we have docker though which also provides isolation we can simply use that.
So what I really want:
Have the source code on my host.
Run django and python inside a container.
Debug from at least two terminals.
Visually my setup looks as something like this:
I assume you have an image with django and python installed. Let’s call the image py3django.
Firstly create the folder where you want your project source code to be. This is the folder that we will mount. My project resides in /home/manos/django_projects/myblog for example.
Once it’s created I just run bash on the image py3django. This will be my primary terminal (terminal 1):
The flag -p makes sure that docker doesn’t choose a random port for us. Since we run Django we will want to run a web server with a fixed port (on 8000). The flag -v mounts our host folder /home/manos/django_projects/myblog to the container’s folder /home/myblog. py3django is the image I have.
Now we have a folder where we can put our source code and a working terminal to play with. I want though a second terminal (terminal 2) to run my python webserver. So I open a second terminal and type:
Mind that I had to put the appropriate container ID in the command above.
Now all this is very nice but admittedly it’s very complex and it will be impossible to remember all these commands and boring to type them each single day. Therefore I suggest you create a BASH script that initiates the whole thing.
For me it took a whole day to come up with the script below:
#! /bin/bash
django_project_path="/home/manos/django_projects/netmag" # Path to project on host
image="pithikos/py3django_netmag_rmved" # Image to run containers on
echo "-------------------------------------------------"
echo "Project: $django_project_path"
echo "Image : $image"
# 1. Start the container in a second terminal
proj_name=`basename $django_project_path`
old_container=`docker ps -n=1 -q`
export docker_line="docker run -i -t -p 8000:8000 -v $django_project_path:/home/$proj_name $image /bin/bash"
export return_code_file="$proj_name"_temp
rm -f "$return_code_file"
gnome-terminal -x bash -c '$docker_line; echo $? > $return_code_file'
sleep 1
if [ -f "$return_code_file" ] && [ 0 != "$(cat $return_code_file)" ]
then
echo
echo "--> ERROR: Could not load new container."
echo " Stop any other instances of this container"
echo " if they are running and try again."
echo
echo " To reproduce the error, run the below:"
echo " $docker_line"
echo
rm -f "$return_code_file"
exit 1
fi
rm -f "$return_code_file"
# 2. Connect to the new container
while [ "$old_container" == "`docker ps -n=1 -q`" ]; do
sleep 0.2
done
container_ID=`docker ps -n=1 -q`
sudo nsenter --target $(docker inspect --format {{.State.Pid}} $container_ID) --mount --uts --ipc --net --pid
This script starts a container on a second terminal and then connects to the container from the current terminal. If starting the container fails, an appropriate message is given. django_project_path is the full path to the folder on the host with the source code. The variable image holds the name of the image to be used.
You can combine this with devilspie, an other nice tool that automates the position and size of windows when they’re launched.
In case you wonder about the top window with all the containers, that’s simply a watch command, a tool that updates regularly a command. In my case I use watch with docker ps. Simple stuff:
> watch docker ps
I use this because I personally like having an overview on the running containers. That way I don’t end up with trillions of forgotten containers that eat up my system.
Now that you have everything setup you can also run django server from one of the two terminals or whatever else you might want.
So as an intern in a big company I was given the task to get comfortable with docker. The problem is that docker is quite fresh so there isn’t really that much of good tutorials out there. After reading a bunch of articles and sparse tutorials (even taken the official tutorial at https://www.docker.com/tryit), I still straggled to get a firm grip on what docker even is supposed to be used for. Therefore I decided to make this tutorial for a total beginner like me.
Docker vs VirtualBox
There are many different explanations on the internet about what docker is and when to use it. Most of them however tend to complicate things more than giving some practical information for a total beginner.
Docker simply put is a replacement for virtual machines. I will use virtual box as a comparison example since it’s very easy for anyone to download and try to see the differences themselves.
The application VirtualBox is essentially a virtual machine manager.
Each and every of the OSes you see in the picture above, is an installed virtual machines. Each such machine has it’s own installed OS, kernel, virtual devices like hard disks, network cards etc. All this takes a considerate amount of memory and needs extra processing power. All virtual machines (VM) like VMware, Parallels, behave the same.
Now imagine that we want to use nmap from an OpenSUSE machine but we are on an Ubuntu. Using VirtualBox we would have to install the whole OS and then run it as a virtual machine. The memory consumption is humongous for such a trivial task.
In contrast to VirtualBox or any other virtual machine, Docker doesn’t install the whole OS. Instead it uses the kernel and hardware of our primary computer (the host). This makes Docker to virtualize super fast and consume only a fraction of the memory we would else need. See the benefits? Imagine if we wanted to run 4 different programs on 4 different OSes. That would take at a minimum 2GB of RAM.
But why would you want to run nmap on openSUSE instead of the host computer? Well this was just a silly example. There are other examples that prove the importance of a tool like Docker. Imagine that you’re a developer and you want to test your program on 10 different distributions for example. Or maybe you are the server administrator on a company and just updated your web server but the update broke something. No problem, you can run your web server virtualized on the older system version. Or maybe you want to run a web service in a quarantine for security reasons. As you see there are loads of different uses.
One question might rise though: how do we separate each “virtual machine” from the rest of the stuff on our computer? Docker solves this with different kernel (and non-kernel) mechanisms. We don’t have to bother about them though, since Docker takes hands of everything for us. That’s the beauty of it afterall: simplicity.
Install docker
Docker is in the ubuntu repositories (Ubuntu 14.04 here) so it’s as straightforward as: sudo apt-get install docker
Once installed, a daemon (service) of the docker will be running. You can check that with
sudo service docker.io status
The daemon is called docker.io as you might have noticed. The client that we willuse is simply called docker. Pay attention to this tiny but significant detail.
Configuration
Do these two things before using docker to avoid any annoying warnings and problems.
Firstly we need to add ourselves to the docker group. This will let us to use docker without having to use sudo every time:
sudo adduser <your username here> docker
Log out and then in.
Secondly we will edit the daemon configuration to ensure that it doesn’t use any local DNS servers (like 127.0.0.1). Use your favourite editor to edit the /etc/default/docker.io file. Uncomment the line with DOCKER_OPTS. The result file looks like this for me:
# Docker Upstart and SysVinit configuration file
# Customize location of Docker binary (especially for development testing).
#DOCKER="/usr/local/bin/docker"
# Use DOCKER_OPTS to modify the daemon startup options.
DOCKER_OPTS="-dns 8.8.8.8 -dns 8.8.4.4"
# If you need Docker to use an HTTP proxy, it can also be specified here.
#export http_proxy="http://127.0.0.1:3128/"
# This is also a handy place to tweak where Docker's temporary files go.
#export TMPDIR="/mnt/bigdrive/docker-tmp"
We need to restart the daemon for the change to take effect:
sudo service docker.io restart
Get an image to start with
In our scenario we want to virtualize an Arch machine. On VirtualBox, we would download the Arch .iso file and go through the installation process. In Docker we download a “fixed” image from a central server. There are thousands of different such image files. You can even upload your own image as you will see later.
This will download a default image for Arch Linux. “base/arch” is the identifier for the Arch Linux image.
To see a list of all the images locally stored on your computer type
> docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
base/arch 2014.04.01 a64697d71089 12 weeks ago 277.1 MB
base/arch latest a64697d71089 12 weeks ago 277.1 MB
Starting processes with docker
Once we have an image, we can start doing things in it as if it was a virtual machine. The most common thing is to run bash in it:
> docker run -i -t base/arch bash
[root@8109626c57f5 /]#
See how the command prompt changed? Now we are inside the image (virtual machine) running a bash instance. In docker jargon we are actually inside a container. The string 8109626c57f5 is the ID of the container. You don’t need to know much about that now. Just pay attention to how we acquired that ID, you will need it.
Let’s do some changes. Firstly I want to install nmap. Since pacman is the default package manager in Arch, I will use that:
> nmap www.google.com
Starting Nmap 6.46 ( http://nmap.org ) at 2014-07-18 13:33 UTC
Nmap scan report for www.google.com (173.194.34.116)
Host is up (0.00097s latency).
..
It seems we installed it successfully! Let’s also create a file:
[root@8109626c57f5 /]# touch TESTFILE
So now we have installed nmap and created a file in this image. Let’s exit the bash
[root@8109626c57f5 /]# exit
exit
>
In VirtualBox you can save the state of the virtual machine at any time and load it later. The same is possible with docker. For this I will need the ID of the container that I was using. In our case that is 8109626c57f5 (it was written in the terminal prompt all the time). In case you don’t remember the ID or you have many different containers, you can list all the containers:
> docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS
8109626c57f5 base/arch:2014.04.01 bash 25 minutes ago Exit 0
Let’s save the current state to a new image called mynewimage:
> docker commit -m "Installed nmap and created a file" 8109626c57f5 mynewimage
6bf56047833bd41c43c9fc3073424f37bfbc96993b65b868cb8d6a336ac28b0b
Now we have the saved image locally on our computer. We can load it anytime we want to come back to this state. And the demonstration..
> docker run -i -t mynewimage bash
[root@55c343f1643a /]# ls
TESTFILE bin boot dev etc home lib lib64 mnt opt proc root run sbin srv sys tmp usr var
[root@55c343f1643a /]# whereis nmap
nmap: /usr/bin/nmap /usr/share/nmap /usr/share/man/man1/nmap.1.gz
Loading the image on an other computer
We now have two images, the initial Arch image we started with and the new image that we saved:
> docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
mynewimage latest 6bf56047833b 2 hours ago 305.8 MB
base/arch 2014.04.01 a64697d71089 12 weeks ago 277.1 MB
base/arch latest a64697d71089 12 weeks ago 277.1 MB
It’s time to load the new image on a totally different computer. First I need to save the image on a server though. Luckily for a docker user, this is very simple. First you need to make an account at https://hub.docker.com/
Once that is done we need to upload the image to the hub. However we have to save the image in a specific format, namely username/whatever.
Let's save the image following that rule:
> docker commit -m "Installed nmap and created a file" 8109626c57f5 pithikos/mynewimage
12079e0719ce517ec7687b4bf225381b99b880510cda3bc1e587ba1da067bd3b
> docker push pithikos/mynewimage
The push refers to a repository [pithikos/mynewimage] (len: 1)
Sending image list
..
Once everything is uploaded, we can pull it from anywhere just as we did when we first pulled the Arch image.
I will do that from inside an OpenSUSE install on a totally different machine. First I try to run nmap
As you see it’s not installed on the computer. Let’s load our Arch image that we installed nmap on
Once the download of the image is complete we will run a bash on it just to look around:
As you see, the TESTFILE is in there and we can run nmap. We are running the same Arch I ran earlier on Ubuntu, on a totally new machine with a totally different OS, but still running it as an Arch.
A bit on containers
Now you probably got a good idea on what images are. Images are simply states of a “virtual machine”.
When we use docker run whatever we are running is put inside a container. A container is pretty much a Linux concept that arose recently with the recent addition of Linux containers to the kernel. In practise container is running a process (or group of processes) in isolation from the rest of the system. This makes the process in the container to not being able to have access to other processes or devices.
Every time we run a process with Docker, we are creating a new container.
> docker run ubuntu ping www.google.com
PING www.google.com (64.15.115.20) 56(84) bytes of data.
64 bytes from cache.google.com (64.15.115.20): icmp_seq=1 ttl=49 time=11.4 ms
64 bytes from cache.google.com (64.15.115.20): icmp_seq=2 ttl=49 time=11.3 ms
^C
> docker run ubuntu ping www.yahoo.com
PING ds-any-fp3-real.wa1.b.yahoo.com (46.228.47.114) 56(84) bytes of data.
64 bytes from ir2.fp.vip.ir2.yahoo.com (46.228.47.114): icmp_seq=1 ttl=45 time=46.5 ms
64 bytes from ir2.fp.vip.ir2.yahoo.com (46.228.47.114): icmp_seq=2 ttl=45 time=46.1 ms
^C
Here I ran two instances of the ping command. First I pinged http://www.google.com and then http://www.yahoo.com. I had to stop them both with CTRL-Z to get back to the terminal.
> docker ps -a | head
CONTAINER ID IMAGE COMMAND CREATED STATUS
7c44887b2b1c ubuntu:14.04 ping www.yahoo.com About a minute ago Exit 0
72d1ca1b42c9 ubuntu:14.04 ping www.google.com 7 minutes ago Exit 0
As you see, each command got its own container ID. We can further analyse the two containers with the inspect command. Below I compare the two different ping commands I ran to make it more apparent how the differentiate in the two containers:
Notice that I don’t have to write the whole string. For example instead of 7c44887b2b1c, I just type the first three letters 7c4. In most cases this will suffice.
On this tutorial I will explain how to install and configure Postfix to be able and receive emails from the Internet and LAN. Running Postfix at home allows you to have an email that looks as you wish. If you have the website mywebsite.com then you can have emails like support@mywebsite.com, admin@mywebsite.com etc.
I set-up Postfix in a gradual way:
First I set up a LAN mail server
Then I configure the mail server to accept emails from the Internet
Lastly I configure the mail server to allow outgoing emails to the Internet
Starting with a LAN mail server
I start by setting up a mail server that works for the LAN. My LAN network looks as below.
ubuntux and laptop are the hostnames (names) of the computers that can be used to identify them in the LAN. For example if you want to ping ubuntux you don’t have to know its IP. You can just ping the computer by providing its hostname. Keep in mind that hostnames cannot be seen outside the LAN.
If you don’t know the hostname of your computer you can run in the terminal the command hostname. Throughout the tutorial I will be using ubuntux as my main computer where I install Postfix.
The router has an external IP 20.20.20.20 and an internal 192.168.1.1. Simply put, that means that from the internet our network is seen as 20.20.20.20 while the computers in the LAN sees the router having the IP 192.168.1.1.
Let’s install Postfix:
sudo apt-get install postfix
The install will ask you some questions. Just use the default values. After install, the daemon is launched automatically so we can start sending emails.
Sending the first email in the LAN
If you haven’t read my article on sending emails via telnet, you could check it out. In that article I describe how you can use telnet to send emails via a gmail mail server.
To send our emails we will be using telnet like in that article. The only difference is that while we had to authenticate with the gmail server with a login/password, here we won’t. The reason is that our mail server is in the LAN and by default configuration users in LAN are considered trusted.
The reason I use telnet and not an email client like Outlook Express is that I want you to see what is actually happening low-level and to get an in insight into the SMTP protocol.
First I send an email from ubuntux (192.168.1.2):
>telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 ubuntux.localdomain ESMTP Postfix (Ubuntu)
HELO whatever
250 ubuntux.localdomain
MAIL FROM: whatever
250 2.1.0 Ok
RCPT TO: animal@localhost
250 2.1.5 Ok
data
354 End data with .
This is a test
.
250 2.0.0 Ok: queued as 22AEA133C91
quit
221 2.0.0 Bye
In the telnet session I sent an email to animal@localhost. We assume that animal is the user I am logged in as at the moment. Then I can just type mail in the terminal and I will see the email arrived. Typing the number of the email I can read it. Typing ‘q’ will quit the application. All emails can be found in their raw format at /var/spool/mail.
Let’s try and send an email from the second machine (192.168.1.10):
>telnet 192.168.1.2 25
Trying 192.168.1.2...
Connected to 192.168.1.2.
220 ubuntux.localdomain ESMTP Postfix (Ubuntu)
HELO whatever
250 ubuntux.localdomain
MAIL FROM: whatever
250 2.1.0 Ok
RCPT TO: animal@ubuntux
250 2.1.5 Ok
data
354 End data with .
Sending from my laptop.
.
250 2.0.0 Ok: queued as 68DF4133C91
quit
221 2.0.0 Bye
The only difference here is that we connected to ubuntux instead of localhost. Accordingly we had to change the recipient from animal@localhost to animal@ubuntux.
The recipient after “RCPT TO:” can essentially be any user (existing or not) as long as the domain used in the email is the hostname of our mail server (ubuntux in this case). An email with a different domain could be for example animal@randomdomain. The email with such a recipient would get totally discarded from the server and lost.
Taking it a step further – receiving emails from the Internet
At this state, emails can be delivered to the mail server from users in the LAN. We will try and make the mail server to be able and receive emails from the internet as well.
In the illustration above an email is being sent from the gmail server to our own mail server. We don’t care how the email got to the gmail mail server. What we care about is how the email will reach our mail server. For the correct routing of the email we have to configure three things:
Our DNS server
Our router
Our mail server
Configuring DNS
Every time we visit a website, a DNS request has to be made to find out the IP of that website. In the same fashion a DNS request is made by the mail server when an email is to be sent. This request is somehow different, in that it looks for a very specific record; an MX (mail exchange) record.
(1) As seen in the illustration the mail server looks at the domain after the ‘@’ symbol in the recipient of the email.
(2) It then sends a DNS request for the mail server in that domain (linuxmeerkat.com). To get an idea of what the answer of such a request can be you can use the debugging tool dig and test it with dig MX linuxmeerkat.com.
(3) A reply arrives from the DNS server with the IP of the mail server.
(4) The email is forwarded to that IP, and further to the mail server.
Now ideally the MX record in the DNS server would look like this:
linuxmeerkat.com. IN MX 10 20.20.20.20
That is however breaking the standard and therefore it is unacceptable. By defition an MX record should bind a domain to a hostname (or more). Therefore we need to replace the IP in the above example with a hostname:
linuxmeerkat.com. IN MX 10 mail.linuxmeerkat.com.
mail IN A 20.20.20.20
Since the mail.linuxmeerkat.com hostname (or FQDN to be exact) doesn’t exist, we have to create it with an A record. An A record binds a (made up) hostname to an IP by definition. Since our server is behind the router I use the IP of the router instead (the IP should never be a LAN one). For hostname I use mail but you could use whatever you wish. Just keep in mind that if you change it, the hostname must also be reflected in the MX record – you should also change the FQDN.
Keep in mind that if you do changes to the DNS records, it might take some time for the effects to take place. So continue a bit later, or after a day or two depending on how fast the changes are updated (essentially it depends on the TTL values).
Is your network reachable from outside?
Time to test if the DNS configuration works.
When we ping mail.linuxmeerkat.com we want that it pings 20.20.20.20. So let’s test it. Go to http://ping.eu/ping/ and ping your mail server’s FQDN (in this case mail.linuxmeerkat.com). It’s important that you ping from outside your LAN so that no local cache is used.
If you don’t get a response, try pinging directly your WAN IP (20.20.20.20). If you still don’t get a response, then probably your router is blocking ping from outside. If you are getting a response by pinging the IP but not when you ping the hostname, then you haven’t configured your DNS records correctly or they haven’t taken effect yet.
If you run your own DNS server you can check for warnings and errors in /var/log/syslog.
Configure your router
At this point we know that an email can arrive as far as to the router. We only need to configure the router so it forwards the email to the correct machine in the LAN.
Since we know that mail servers use port 25 and if we haven’t changed that, then our mail server also uses port 25, we just have to configure the NAT of our router.
Forward all TCP at port 25 to your mail server (in my case 192.168.1.2). My router’s port forwarding looks as below:
Sending an email to Postfix from gmail
If we can ping the mail server from the internet, then we can assume that our network is reachable. Sending an email to animal@mail.linuxmeerkat.com will not work still though as the router won’t know to which computer in the LAN to forward the email to. Setting the NAT of the router to forward all incoming TCP connections on port 25 to the mail server fixes this.
We can now test it by sending an email to animal@mail.linuxmeerkat.com from a gmail account. The mail server will reply with an error:
Delivery to the following recipient failed permanently:
animal@mail.linuxmeerkat.com
Technical details of permanent failure:
Google tried to deliver your message, but it was rejected by the server for the recipient domain mail.linuxmeerkat.com by mail.linuxmeerkat.com. [20.20.20.20].
The reason the email gets rejected lies in the Postfix configuration. The configuration file /etc/postfix/main.cf has this (or a similar) line:
mydestination = ubuntux, localhost
mydestination is the variable that defines what FQDN/domains in emails are accepted. In essence all emails arrive at the server, it’s just the depending on this configuration some of these emailes will be discarded and some will be accepted by the server.
With the current configuration we can see that only emails of the format username@megistanas and username@localhost are accepted. Trying to send an email to animal@ubuntux from gmail won’t work as ubuntux just can’t be seen from outside our LAN; it doesn’t resolve to anything. For this reason we want to add one ore more domains that point towards our network. That way, when the an email arrives it doesn’t get rejected.
In my case I add linuxmeerkat.com so that I accept all emails with the format username@linuxmeerkat.com:
mydestination = linuxmeerkat.com, ubuntux, localhost
Restart Postfix and sending an email to animal@linuxmeerkat.com should now work.
You could essentially also add mail.linuxmeerkat.com as that would take the same route as well. We can add pretty much any domain that points to our router.
Sending an email to gmail or hotmail from Postfix
There are essentially two entries in the configuration file that we have to change to make sending emails to the Internet possible:
smtpd_sender_restrictions can have more options than the one shown. However the only interesting option for us at the moment is to permit anyone from mynetworks to be able and send emails. So make sure permit_mynetworks is amongst the options.
mynetworks specifies local networks where users are considered trusted to use our mail server. For that we have added the 192.168.0.0/16 which includes all computers in the network. Be careful, you don’t want to have a WAN ip here as that would make our mail server an open relay (I talk more about that below).
This configuration will let anyone from our computer to send an email of the format username@linuxmeerkat.com. If we want users of the LAN to be able to use our mail server from the Internet, some sort of authentication has to take place (login or key), but I am will not go further into that.
Let’s test and see if we can send an email to a gmail account:
>telnet mail.linuxmeerkat.com 25
220 mail.linuxmeerkat.com ESMTP Postfix (Ubuntu)
HELO dickhead
250 mail.linuxmeerkat.com
MAIL FROM: animal@mail.linuxmeerkat.com
250 2.1.0 Ok
RCPT TO: animal@gmail.com
250 2.1.5 Ok
DATA
354 End data with <CR><LF>.<CR><LF>
This is a test.
.
250 2.0.0 Ok: queued as 3D31311F765
quit
221 2.0.0 Bye
If everything went as planned, you should have received an email in your gmail account (animal@gmail.com in the example).
Make sure your mail server is not an open relay
You always want to avoid having an open relay. An open relay constitutes that your mail server can be used by anyone on the internet including spammers.
Once spammers find out that your server is an open relay, they will start using it. At some point, an other mail server will find out that you have an open relay and will blacklist your server.
So how do you ensure that you don’t have an open relay? Very simply. In your configuration file pay attention to these two lines:
As long as the networks in mynetworks are LAN IPs and not WAN IPs, your mail server is not an open relay as no-one outside the LAN can use your mail server to send emails.
Our mail server works fine at the moment but it’s very minimal. It doesn’t even filter incoming spam emails. Configuring filters etc. can be time consuming and will need further maintenance all the time. An easy solution to this is to forward our emails to a second mail server, a relay server (keep in mind: not same as an open relay, as a relay server is not open to everyone). So if we have to deliver an email to user11@gmail.com, instead of delivering to the gmail server directly, we can use a second mail server which we have access to. That server can then do the filtering etc. for us and deliver the email to its final destination.
My ISP is called Bahnhof and by googling around I found that it offers a relay at mail2.bahnhof.se. Most ISPs do offer mail servers that can be used as relays. Luckily for me, I don’t even need to enter a password and username as I am authenticated by my IP or MAC (depends on how you’re connected to the ISP). In your case you might need to authenticate first before using the relay. I won’t go into how you do that. I will however tell you how you can test to see if you can use the relay as a relay server.
If you can send an email directly from the relay server then you can use it as your relay. See the telnet session below:
telnet mail2.bahnhof.se 25
Connected to mail2.bahnhof.se.
Escape character is '^]'.
220 mxf3.bahnhof.se ESMTP
HELO dickhead
250 mxf3.bahnhof.se
MAIL FROM: animal@linuxmeerkat.com
250 Ok
RCPT TO: animal@gmail.com
250 Ok
DATA
354 End data with <CR><LF>.<CR><LF>
This is a test.
.
250 Ok: queued as 0C67C53CEB7
QUIT
221 Bye
I just connected to my ISP’s mail server and sent an email to a gmail account. Since I got the email delivered and wasn’t asked for a login/password, that means that I can directly use the server as a relay without authentication.
If your ISP’s mail server requires a username and password you’ll have to do some extra work in configuring that. Also keep in mind that in the field “MAIL FROM:” you have to provide an email with a hostname that points to your network. The server will probably check to see if whatever@hostname actually resolves to your network. If not, you will get an error like below:
450 <spamotron@spammer.spam>: Sender address rejected: Domain not found
Now that we know we can use the server as a relay, we simply have to edit /etc/postfix/main.cf and make sure the relayhost is set correctly:
relayhost = mail2.bahnhof.se:25
Restart Postfix and you should be able to use your mail server as before with the difference that now all outgoing emails go first through the relay server and the relay server forwards them further to their destinations.
sed is a very famous tool to the UNIX* community but which is very often misused. Most people try to use it in cases that it’s not the right tool or they tend to use it in the wrong way. I try in this post to show the things that are worth knowing when it comes to sed. I thus skip things like buffer holders, labels etc. which just make scripts totally unreadable for no benefit. I also talk a bit about the inner-workings of sed so the user has a grasp of why sometimes things don’t work as expected.
Why even learn sed?
For two reasons:
You can automate any boring mechanical work you would do on a normal text editor
If you know the syntax of sed, then you are a better vi/vim user
The second point, becomes obvious when you realise that the two programs have similar if not the exact syntax. For example, substituting the word “cow” with the word “horse” in the third line of the document in vim is :3s/cow/horse/ while in sed it’s 3s/cow/horse/. See the magic?!
A bit of history
Sed, awk and grep are the offsprings of a line editor called ed. ed pretty much let the user to edit one line at a time. That is also the reason that sed, awk and grep work on lines. All three programs have inherited the syntax of ed to some extend. In ed, to search & replace the word “cow” with the word “milk” in a text document, someone would type s/milk/beer/. That is exactly the same command used in sed – an indication of ancestry.
The tool grep actually takes its name from the command g/re/p, a command used in ed to only show lines that contain a specific regular expression. The ‘p’ in the end means to print on the screen while the ‘g’ in front means to go through all the lines.
When to use sed
While all three of these tools (grep, sed and awk) work on lines, sed and awk are very similar to each other while grep is more of a loner. Grep is used merely to filter out (or in) a line based on a regular expression. Sed and awk offer much more. What differs sed from awk is the data that they were built to edit.
Awk should be used when every line in the file has a specific structure. In other words that includes files where each line has a specific number of fields with every field separated with a delimiter (in most cases a tab). Such files can be CVS files, tables, the output of ls in linux, and more.
For everything else, use sed. Common examples are raw texts, this post, a C file, a script, an HTML file, etc. What all these files have in common is that lines don’t have a specific structure: the first line can have one word, while the second can have 100 words etc. Sed can still edit files that awk edits, but the opposite is most times impossible. If you are trying to just do that then you are most probably using the wrong tool.
Syntax
Sed in the terminal
The common syntax for sed in the terminal is:
sed SCRIPT INPUTFILE
The meat of sed is the SCRIPT and that is pretty much what I cover in this post. It’s a good convention to put quotation marks around it in case there is a space inside it (the shell might interpret it as multiple commands then).
sed can have multiple SCRIPTS or it can use a file with commands.
Multiple script lines:
sed 'SCRIPT1; SCRIPT2; SCRIPT3;' INPUTFILE
Using a script file:
sed -f SCRIPTFILE INPUTFILE
The SCRIPTFILE should have a command on each line. So SCRIPT1 should be on separate line than SCRIPT2 and SCRIPT3 on a separate etc.
Thoughout
Sed’s script syntax
Sed uses three things to accomplish tasks:
line specifiers (address)
commands
flags
The syntax of a single SCRIPT line is:
<line><command><flag>
A line or line specifier is a way to specify which lines you want the command to affect (parse). If the line specifier is missing, then the command affects all lines, which is the default behaviour.
A command is denoted by single character. For example to replace (substitute) a word with an other word we use the command ‘s’:
s/word1/word2/
A flag is used to modify the <command> or the <line>’s behaviour a bit and is placed after the whole command or line. Using the flag g on the example above, we get:
s/word1/word2/g
g stands for global and is used to replace all word1 in the line. Without it, only the first occurrence of word1 in the line is replaced.
A flag goes hand in hand with regular expressions so they can only be used if <command> or <line> have a regular expression in them. If the <line> is specified with a number for example, it’s illegal to use a flag:
#Illegal
sed -n '6g'
The reason is that flags are made for text strings (patterns) so it doesn’t make sense to sed when you are telling sed to use the flag ‘g’ on a line (which is something else than a string) and not a pattern.
The minimum thing needed on a SCRIPT line is a command or a line specifier. Someone can have both, one of them – with a flag or without. These combinations (or permutations to be exact) are allowed:
A flag applies to the command or line before it and assume that the previous has a regular expression in it.
How sed works internally
Imagine we have the file list.txt with the lines:
Today I will drink my milk
and afterwards I will eat a cow.
The cow will taste like cow.
Sed works with lines. As stated earlier we can have multiple script lines seperated with question marks:
sed 's/Today/Tomorrow/g; s/Today/Next Friday/g' list.txt
sed has a working buffer for each line (called pattern space). sed will initially load the first line of list.txt in the buffer. Then it will go through all script lines one by one altering everything in-place (in the buffer). In our example the buffer for the first line initially has:
Today I will drink my milk
After the first script command executes, it becomes:
Tomorrow I will drink my milk
The second script line doesn’t alter anything as sed can’t find an occurrence of the word ‘Today’ since it just got altered.
Once the first line is done, sed will load the second line in the buffer and go through the same procedure until all lines in list.txt have been parsed.
Something important to notice is that each SCRIPT line will run regardless if the previous SCRIPT line succeeded or not. With success we mean that the command did what it is meant to do. If substitution is used, then we define success as the alteration of a line. If we just specify a line, then success is if the line exists etc.
Addressing specific lines
<line><command><flag>
By default, sed goes through all the lines. However, one can address a specific line or a range of lines. That can be done by specifying lines by their number in the file (1st line, 2nd line, 50th line etc.) or by a line’s content.
To run a command on the 10th line we do:
10<command>
To run a command on each line that contains the word “cow” we do:
/cow/<command>
The latter makes use of regular expressions. When we use regular expressions we need to add slashes to the start and end of the regular expression.
For a range of lines we use a comma (awkward, I know). To specify all lines between the 10th and 25th line (including those) we would write:
10,25<command>
If we want to specify a range of lines by using regular expressions we still have to encapsulate the regular expressions in slashes. To run a command on all the lines between the first line found with the word “cow” and the first line found with the word “grass”, we would issue:
/cow/,/grass/<command>
Bellow you can see all the ways for specifying lines.
<line1>,<line2>
Lines between <line1> and <line2> (including those)
<line1>~N
Every Nth line after <line1>
<line1>!
All lines except line1
/<regex>/
Lines matching the regular expression
$
Last line
<line1>,+N
All N number of lines after <line1>
Some more practical examples can be seen bellow. The command p is used to print the line that is specified. (Notice that sed needs the parameter -n for the command p to work.)
sed -n '2p' -> print line 2
sed -n '2,4p' -> print line 2 to 4
sed -n '$p' -> print last line
sed -n '2!p' -> print all lines but line 2
sed -n '/red/p' -> print every line that contains the word red
sed -n '/red/,/green/p' -> print all lines between the <strong>first occurrences</strong> of the words 'red' and 'green'
Of course for all these examples to work you need to either feed sed with a stream from a file or a pipe.
Using commands
<line><command><flag>
Now we are to the meat of all meats.I have explained the substitution command a bit but bellow you can see all the commands with their syntax (if they have one).
s/<regex>/<subst>/
substitute
Replace a match (using regular expression) with a string.
p
print
Prints a specific line or a range of lines. The sed flag -n should be used for this command to work.
=
line number
Shows the number of the line
d
delete
Deletes (omits) the matched line
y/<char1>/<char2>/
transform
Substitutes char1 with char2. Works even with a sequence of characters. For example y/abc/ABC/ will replace a with A, b with B and c with C.
Substitution
Substitution is the most commonly used command. It’s syntax is as follows:
s<d><regex><d><subst><d>
where <d> is a delimiter which should be a single character. Most commonly the delimiter is a slash / but it can essentially be any character. All lines bellow are equivalent.
s/cow/horse/
s_cow_horse_
sDcowDhorseD
In the example above the first occurrence of the word cow on each line will be replaced with the word horse, which is the default behaviour. If you want all occurrences of the word cow in a line to be substituted, the flag g (global) has to be appended to the line:
s/cow/horse/g
The substitution field <subst> can take some special variables like the ampersand symbol “&” which holds the matched string from the regular expression. There are also a few macros to automate conversion of capitals to lower-case and vice versa.
All these are shown in the table bellow.
&
Holds the matched string.
\1
Holds a part of the match specified in the regular expression with parentheses.
\U<string>
Converts all letters in <string> to capitals.
\u<string>
Converts the first letter in <string> to capital.
\L<string>
Converts all letters in <string> to lower-case.
\l<string>
Converts the first letter in <string> to lower-case.
<string>\E
Ends the conversion at specific point. Should be used in conjunction with \L and \U.
The match holders \1\2\3 etc. have to be specified in the regular expression with parentheses. The parentheses themselves have to be escaped or else sed will be looking for parenthesis characters in the line. So to replace each word “cow” with “supercow” we can do:
s/\(cow\)/super\1/
or
s/cow/super&/
The latter is more elegant of course. However there are two cases where the specific holder has to be used:
When we want only a portion of the matched string and not the whole string (&).
When there are many different strings you want to grab from a match.
For example this can’t be solved by merely using the & symbol:
s/\(\w*\) cow \(\w*\)/\2 cow \1/
This script will look for each occurrence where “cow” has a word before it and a word after it and it will change their order. The \w matches any character while the asterisk * tells the pattern that the word can be arbitrarily long. We use the parentheses around the first word and the second word to denote which matched parts of the regular expression should be given to \1 and \2. In <subst> we just reverse their order by putting the second word first and the first word second.
Flags
<line><command><flag>
A flag can be used on a <command> or a <line> or both.
g
global
For all occurences in the line (default is to stop at first occurence)
I
Ignore case
Not case-sensitive
p
Print
Output only this line (not everything as default). The sed flag -n should be used for this flag to work.
Find line in lines of lines in lines..
An important concept to comprehend in sed is nesting. I try to leave out all the advanced things sed offers, like holder buffer (horror to read), labels etc. but nesting is worth learning as it gives a lot of extra power for a little learning curve. Nesting is similar to the IF..THEN conditional.
We have this text file: Today I will drink my milk
and afterwards I will eat a cow.
The cow will taste like cow.
Today is not afterwards if I am a cow. Right?
Imagine that we want to check if the last line of the text contains the word cow. Someone might think that putting $p together with /cow/p would work:
sed -n '$p; /cow/p'
Admittedly the output seems strange: and afterwards I will eat a cow.
The cow will taste like cow.
Today is not afterwards if I am a cow. Right?
Today is not afterwards if I am a cow. Right?
The reason this doesn’t work as expected is that the second script runs regardless if the first one succeeded or not. Thus in the first line none of the scripts print anything. On the second line, the first script fails but the second script finds the word cow so it prints the line. The same happens at the third line. At the forth line, the first script succeeds as the line is the last line of the file, so that line gets printed. Then the second script runs and that succeeds. Thus we get the line printed again (a second time).
A way to solve this is is to nest the second script line in the first somehow so that it runs only if the first script line has succeeded. This is similar to the pseudocode:
if <line1>
then SCRIPT
Where <line1> is a line specified by a number, range or pattern (regular expression).
The syntax for nesting script lines in sed is
<line>{<line><command><flag>}
For our example:
sed -n '${/cow/p}'
This translates to: if this is the last line ($) then do whatever is in the brackets. So everything in the brackets will be checked only if the current line being parsed is the last one.
We can even nest inside a nested script:
sed -n '2,4{/cow/{/Today/p}}'
This script will go through lines 2 to 4. It will check first if a line contains the word cow. If the line contains the word cow, then it will check if it contains the word Today. If it does, it will print it. Ofcourse there is no practical reason to have second braces in the above example. I just wrote it to show that it’s feasible. In some cases you need braces to accomplish tasks, especially in cases where we avoid using too advanced things.
Examples
Printing a specific line
sed -n '2p'
The p in this case is the p command (and not flag). Remember that flags apply only to regular expressions. When we use sed’s -n parameter, only lines specified with the p command or flag will be printed on screen.
Printing a range of lines
sed -n '1,3p'
The comma is used to specify a range. In this example we specify line 1 to line 3 and then we print each such line.
Hiding a specific line
sed -n '2!p'
This is similar to:
sed '2d'
Show the last line
sed -n '$p'
In regular expressions the dollar sign $ denotes the end of a string. In sed when it’s being used with substitution it denotes the end of a line.
However when used with the command p, it denotes the last line of the input. In the same way the regex symbol ^ denotes the first line.
Converting a specific word to uppercase
sed 's/Today/\U&/' list.txt
This will match any word ‘Today’ and replace it with ‘TODAY’. In the replacement the escaped U (\U) tells sed to convert to uppercase everything following in the replacement string. In this example we use an ampersand which in sed represents the matched string. If we wanted to stop the conversion we just need to add \E where we want it to end.
Converting all text to uppercase
sed 's/.*/\U&/'
For this we use the substituion command s. .* is a regular expression which fits any sequence of characters. As sed is working on lines, .* matches a whole line each time.
Converting all text to lowercase
sed 's/.*/\L&/'
Similar to the previous example with only difference that we use \L instead of \U.
Grabbing all content between the body tags in HTML
Say we have the ugly HTML code bellow and we want to grab all the content between the body tags.
<html><head></head>
<h1>h1 outside of body</h1><body><h1>h1 stuck to body</h1>
<img src="images/soon.png"/>
<p>a paragraph</p></body></html>
Most sed experts would go about using some very advanced commands to accomplish this. The readability of that becomes horrific. As my opinion is that you can do the same things without knowing all the advanced commands I am going to just do that.
Notice that I use pipes instead of using advanced commands.
Solution:
sed -n '/<body>/,/<\/body>/p' | sed 's_.*<body>\(.*\)_\1_; s_\(.*\)</body>.*_\1_'
It might seem like a mess but I can assure you that it’s much more elegant than a pure single sed SCRIPT solution. I will break it down so you can see how it works.
/body/,/body/p
matches all lines between the body tags, including the body tags. In this way I have minimized my problem to:
<h1>h1 outside of body</h1><body><h1>h1 stuck to body</h1>
<img src="images/soon.png"/>
<p>a paragraph</p></body></html>
After that, we are sure that the first line has the start of the body tag and the last line has the closing of the body tag. So we start by filtering out things we don’t need from the first line: the tag itself and everything preceding it.
s_.*<body>\(.*\)_\1_
I use _ as a delimeter instead of slashes to make the substitution code a bit more readable. The regular expression .* matches 0 to infinite number of arbitrary characters. So I use it around the body tag in case there is something before and after it. I put the second .* in parentheses to grab the text that might be there as I want to keep that. Using \1 in the substitution field I accomplish substituting the whole line with the text after body.
s_\(.*\)</body>.*_\1_
We do something similar to the line with </body>. The only difference is that now the portion of the match that we are interested in is the one before the body tag so we move the parentheses there.
Keep in mind that this solution will not work if the body tag includes some attributes like style. Someone might think that just using the bellow regular expression in the substitution would work.
.*<body.*>\(.*\)
Notice that the only difference is that we added the regular expression .* between body and its closing arrow > to point out that there can be nothing in between or there could be some arbitrary things (in our case attributes).
That regular expression doesn’t work however as it matches the last > in the line. The reason is that in sed .* is greedy and there is no way to make it non-greedy. With greedy we mean that the pattern will try to match as much as it can in the line. So if you want to match the first > it’s not possible. Or.. actually it is possible but the code as you will see in the next example starts looking like a monster.
Grabbing all content between tags in HTML
You are probably better off learning Perl or Python if you need to do these kind of “advanced things”. I will however show that you can achieve things like this without using the more advanced commands or other programs/languages. This solution is a continuation of the previous example on catching the content between the body tags. The mere difference is that in this solution we allow even attributes to a tag and are a bit more permissive towards whitespaces. This makes it a more general solution that can be used for other tags than the body tag.
Solution:
sed -n '/<body>/,/<\/body>/p' | sed '1s_.*<body[a-zA-Z0-9="\x27_ ]*> *<\(.*\)_<\1_; 1{/<body>/d}; s_\(.*\)</body>.*_\1_'
Essentially the only thing that got changed from the previous example is the alteration of the commands in the first line:
The first script line (everything before the first ;) looks for a second tag after body. We don’t really need to specify line 1 but it is a good convention as it makes the code easier to understand and the processing faster. I am looking for the body tag followed by an attribute or not. TO define an attribute in HTML, only the characters in the brackets are allowed (says the HTML protocol, not me). \x27 is the code for a single quote mark ‘. The reason I use its code instead of the mark itself (I use the double quote after all), is that I use the single quote marks around the whole command so if I insert it in the expression, then it will break it. After that I use ” *<" (notice the space) to denote that there might be an arbitrary number of spaces or none before the opening of the new tag. I replace everything with the opening of the tag after body with the rest of the line.
If the first command didn’t succeed then it means that there’s not a second tag after the body. Thus it’s safe to delete the whole line in that case. First we check if there is a body tag in the first line. If there is a body tag (/<body>/), then we delete it with the command d.
Telnet is a small amazing tool that can be used for pretty much everything concerning networks. Telnet sends ASCII strings directly to a host. So as long as a protocol supports ASCII messages, we can communicate with a host with any protocol we want. Some examples are SMTP and HTTP.
In practise this method is being used to test things. If you are developing a web-server for example it would be easy if you could test your web-server directly without having to launch a browser and type a command. It’s a blessing for automation.
Here we will try to send an email by using the SMTP protocol. We will use telnet for that, by sending and receiving raw messages with the SMTP server.
Preparation
First we need to install telnet with SSL support. All SMTP servers use SSL encryption nowadays so using simply telnet will not work. The telnet shipping with Ubuntu does not support SSL so we need to upgrade it: sudo apt-get install telnet-ssl
To run the new telnet-ssl we just issue telnet in the terminal as before. The difference is that telnet now supports some new flags, namely -z, which is used for SSL.
We are also going to need http://www.base64encode.org to encode our username and password to base 64 format. Keep in mind that base 64 does not encrypt a message. It just changes the way the information is stored. In practise we use base 64 to be able and send any kind of data as simple characters. The real encryption is under the hood however. Check Extras for more info.
Steps
Start telnet
telnet -z ssl smtp.gmail.com 465
The flag -z ssl tells telnet to use SSL over the connection. smtp.gmail.com is Gmail’s server domain and 465 is the port used by the server.
Handshake with server
HELO yo
The text after the HELO is supposedly your domain or literal address. Essentially you can enter whatever you want. Pressing enter, the exact message arrives at the mail server, which in response should reply with 250 mx.google.com at your service
Login on gmail
AUTH LOGIN
This message is self-explanatory. After the server receives this message, it’s going to ask for our username. The response looks like this 334 VXNlcm5hbWU6 and indicates that the username should be entered.
Username
Go to http://www.base64encode.org and encode your username (username@gmail.com) to 64 base. Copy and paste the encoded string in your telnet session and hit enter. If the username is accepted, you get back the message bellow. 334 UGFzc3dvcmQ6
This message indicates that the password should be entered.
Password
Do the same as above but instead of your username, encode your password. Then paste the password and hit enter. If the username/password combination is correct, you are greeted with the message 235 2.7.0 Accepted
Email sender
First we are going to add the sender of the email: MAIL FROM: <username@gmail.com>
For gmail it doesn’t really matter who you set as sender. We logged in our gmail account when we sent the AUTH LOGIN message to the server. Gmail has as a mechanism to set the sender automatically to your email address.
The response back is 250 2.1.0 OK js17sm40481494lab.5 - gsmtp
Email recipient
Now we will add the recipent of the email. Here we need to provide a valid email or else the email will not go any further. If the email doesn’t exist, we will probably get it back into our inbox which is the default behaviour for undelivered emails. RCPT TO: <username@gmail.com>
On success we get 250 2.1.5 OK js17sm40481494lab.5 - gsmtp
Email body
Now we want to start typing the email. We send the following command to the server so that the server knows that what follows is the actual email. DATA
The response from the server is 354 Go ahead js17sm40481494lab.5 - gsmtp
Now we need to set the subject(if we want). Subject: test
and everything else following is considered the text of the email
This is a line in the email.
This is a second line in the email.
.
The dot on its own line in the end is to signal the end of the email. After that, a response from the server should come: 250 2.0.0 OK 1381416452 js17sm40481494lab.5 - gsmtp
Now the email has been sent.
Close telnet
QUIT
This is an actual command to the telnet client and not a message sent to the mail server.
Extras
EHLO vs HELO
EHLO can be used instead of HELO. The difference is that EHLO lists also all the commands that the server supports. EHLO is newer than HELO and is suggested to be used instead of EHLO. However HELO is always going to be supported, so in reality there is no big difference as long as the server supports the message.
SSL and encryption
Something important, as mentioned in the beginning of this page, is that encoding data to base 64 is not encryption but rather a different representation of the same data. It’s SSL which adds the encryption to our communication and you can be assured that everything is encrypted between the client and the server.
I captured the whole procedure of sending an email, with Wireshark to prove my point.
The packets are in the order they were sent and received. We can see that telnet first contacts the DNS server to get the domain name of the mail server. 192.168.1.1 is my router and 192.168.1.2 is my computer from where I use telnet.
After we get the domain we do a first handshake with the mail server. It’s those three TCP messages with SYN/ACK in them. After that we initiate a TLS session. TLS and SSL are pretty much the same thing. They are just different versions for encrypted communication, with TLS being the newer one.
Everything we type and read from the time we start telnet is encrypted. The reason we see the messages clearly in the terminal is that they get decrypted by telnet on their arrival. When messages leave telnet, they get encrypted to travel on the wire. That’s why with Wireshark we only see the decrypted data, because we are looking at what is passing through the wire. I circled the TLS messages to make it more apparent that the whole session is encrypted by using TLS. The few TCP packets seen here and there, are packets with encrypted data. If you could check what these packets hold inside(with Wireshark), you would only see gibberish.
If instead of Gmail we were using an SMTP server that doesn’t use SSL, we would be able to intercept all the packets with their raw data, and thus the actual messages, passwords, usernames, etc.
LinuxMeerkat 