If you go to any serious company you will find that they have automated tests running for every new update in the codebase of the programs they develop. This assures that every random thing some developer fixes, doesn’t break something else in the program. In most cases you will find Jenkins or some of the commercial solutions like Travis and CircleCI for this.
As a metro-sexual hipster developer however you want something that is fast to setup, easy to use and minimal. To be fair there is a huge amount of open-source free continuous integration (CI) programs that do this. On my recent hunt however I found one that shines amongst them: StriderCD.
The big advantages of Strider:
Stable
Beautiful intuitive interface
Integration with Github
Small enough to hack (they even have a guide on how to hack it!)
Plugin API
Helpful active community!
In this tutorial I will setup Strider so that it works even if you sit behind 9.000 firewalls in your company. I will also set it up so that you can move it around and reuse it in the future without the hustle of setting up things from the beginning. For this I will use Docker ofcourse, the hipster’s French wrench. I will also make it so that you can run other containers inside the container where Strider runs. This is helpful if you want to run tests inside their own container for example.
Building Strider
To be sure that you have the latest Strider it is wise that we build everything from scratch. I have created a Dockerfile that will make an image with the latest Strider and setup everything up so that you can use it with Docker.
In case you don’t have Docker:
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
sudo sh -c "echo deb https://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list"
sudo apt-get update
sudo apt-get install -y lxc-docker
preprocess is a simple shell script I made that will edit the Dockerfile so that in the next step where we build it, it installs the same Docker version as the one outside of the container. This is critical since if the Docker inside the container is different from the one outside of it, we won’t be able to run Docker inside the Strider container.
Running Strider
At this point we have a Docker image named strider:barebones which has a minimal but powerful setup for Strider. It’s time we run it:
ID=$(nohup docker run -d --privileged -v $PWD/data/db:/data/db -p 3000:3000 strider:barebones)
This will run the container with the database stored at $PWD/data/db. That way if for whatever reason the container stops running, we have all the user accounts, projects, etc. intact. Notice also that I use the --privileged flag. This is merely so that we can run containers in containers.
To verify that everything works open your browser at http://localhost:3000 and you should see the login page. Notice that you won’t be able to login to Strider at this point since my Dockerfile recipy keeps things minimal – that means no user accounts by default. Adding accounts is super-easy and you don’t even have to restart anything so that won’t be a problem.
Adding an account
In order to add an administrator account to Strider you just have to access the running container and create a user from within. If you use an older Docker you can use my tool docker-enter. If you run Docker 1.3+ then you can use docker exec.
docker-enter way:
sudo docker-enter $ID
strider addUser
..
exit
docker exec way:
docker exec -it $ID strider addUser
Follow the instructions to make a new account. Then just exit the running image and just refresh the webpage http://localhost:3000 (no need to restart the running image). You should be able and login in to Strider with the credentials you just gave!
Saving for the future!
Now, notice that all configurations, projects, user accounts, etc. are all stored in the folder db/data. In order to move the whole thing somewhere you simply move the folder to the computer you want and run strider:barebones as we did before. Ofcourse you will need to rebuild the whole thing as before as well or you can either save it to the cloud:
Remember to replace username with your own username at the Docker registry. Once all this is done, you can run directly strider from any computer. The image will be downloaded automatically.
docker run -d --privileged -v $PWD/data/db:/data/db -p 3000:3000 username/strider:barebones
If data/db is not at the directory where you try to run Strider then a fresh database instance will be used.
This guide will show you how to run a GUI application headless in a Docker container and even more specific scenarios involving running Firefox and Chrome. If you are not interested about those then you can just stop in the middle of this tutorial.
What the hell is X?
X is a program that sits on a Linux machine with a monitor (so servers usually don’t use X). X’s job is to talk to the Linux kernel in behalf of GUI programs. So if you are playing a game for example, the game (that is, the application) is constantly sending drawing commands to the X server like “draw me a rectangle here”. X forwards all this to the Kernel which will further forward the information to the GPU to render it on the monitor.
X can even receive commands from the keyboard or mouse. When you click to shoot on your game for example, the command “click at 466,333” is sent from your mouse to the kernel, from the kernel to the X and from X to the game. That way the game can have a clue on what is happening!
You will often hear X being called a server and the reason for that is simply because the way the applications send commands to it is through sockets. For that reason the applications are also referred to as clients many times.
If you are reading this then the X is running on your PC. Let’s prove it:
We can see that X is running as root and has PID 1436. An other important thing is to notice the :0 which is called display in X jargon. A display is essentially:
A monitor
A mouse
A keyboard
And this is the bigger picture of how it all looks together:
Now there is a variable in Linux that is used whenever we run a GUI program. That variable is surprisingly called DISPLAY. The syntax of the DISPLAY variable is
<hostname>:<display>.<monitor>
. Let’s check the DISPLAY on our computer:
> echo $DISPLAY
:0
I get :0, which means we use display 0. Notice however that this says nothing about which monitor we use. This makes sense since if you are running 2 or more monitors on your Linux you still have the same environment variables in both of them. It wouldn’t make sense that an environment variable changes just because you echo it from a different screen, would it? For that reason we get the display and not the monitor so that we get the same output on both. As about the hostname, since there is no info about it, the local host is assumed.
On a notice, if you have multiple monitors you can still specify which monitor to run an application by simply typing the full display variable you want. So if you have a monitor 0 and a monitor 1 on the current display, I can run firefox on monitor 1 with:
DISPLAY=:0.1 firefox
Creating a virtual monitor
Instead of running X, we can run a different version of it that can create virtual displays. Xvfb (virtual framebuffer – whatever the hell that means) will create a virtual monitor for us.
So let’s make a new monitor (I assume you have installed xvfb):
Xvfb :1 -screen 0 1024x768x16
This will start the Xvfb server with a display 1 and a virtual screen(monitor) 0. We can access this by simply typing DISPLAY=:1.0 before running our graphical program. In this case the program will start in the virtual screen instead of our monitor.
We see we have the normal display 0. (A way to tell it is the default screen is to see that it runs as root.) We can also see the second display :1 and screen 0 with resolution 1024×768. So what if we want to use it?
Open a new terminal and type:
> DISPLAY=:1.0 firefox
..
This will start firefox at the given display. The reason I use the DISPLAY at the same line is to make sure that the subprocess inherits the variable DISPLAY. An other way to do this is to type:
> DISPLAY=:1.0
> export DISPLAY
> firefox
..
Run a GUI program in a Docker container
We will now create a virtual screen inside a docker container.
So now we are sure that we are running the virtual screen. Let’s access it and run something graphical on it. In this case I will run Firefox and Python+Selenium just as a proof of concept of what is happening.
First I put my display variable and use export to assure that any sub-shells or sub-processes use the same display (with export, they inherit the variable DISPLAY!):
root@660ddd5cc806:/# firefox
(process:14967): GLib-CRITICAL **: g_slice_set_config: assertion 'sys_page_size == 0' failed
Xlib: extension "RANDR" missing on display ":99.0".
(firefox:14967): GConf-WARNING **: Client failed to connect to the D-BUS daemon:
//bin/dbus-launch terminated abnormally without any error message
..
The errors don’t mean anything. But we can’t be sure, can we? I mean, since we can’t see what’s happening it’s really hard to tell. There are two things we can do, either use ImageMagick to take a snapshot and send it to our host via a socket or we can simply use Selenium. I will do that since most people probably want to achieve all this for testing purposes anyway.
If you get a bunch of HTML, then we have succeeded!
The Chrome issue
If you try and run Chrome in a Docker container, it won’t work even if you have setup everything correctly. The reason is that Chrome uses something called sandboxing. Reading this I could not let but notice the word jail. Apparently it seems that Chrome uses Linux containers (the same that Docker uses). For this reason you have to put a bit of extra effort to solve this issue since because of technical difficulties it’s not possible to run containers in containers.
I have now installed Selenium, Chrome and Xvfb. Now I am going to make make a virtual monitor and run Chrome:
root@7dd2c07cb8cb:/# Xvfb :99 -screen 0 1024x768x16 &> xvfb.log &
[1] 6729
root@7dd2c07cb8cb:/# DISPLAY=:99.0
root@7dd2c07cb8cb:/# export DISPLAY
root@7dd2c07cb8cb:/# google-chrome
Xlib: extension "RANDR" missing on display ":99.0".
Xlib: extension "RANDR" missing on display ":99.0".
[6736:6736:1017/143449:ERROR:desktop_window_tree_host_x11.cc(802)] Not implemented reached in virtual void views::DesktopWindowTreeHostX11::InitModalType(ui::ModalType)
ATTENTION: default value of option force_s3tc_enable overridden by environment.
failed to create drawable
[6775:6775:1017/143449:ERROR:gl_surface_glx.cc(633)] glXCreatePbuffer failed.
[6775:6775:1017/143449:ERROR:gpu_info_collector.cc(27)] gfx::GLContext::CreateOffscreenGLSurface failed
[6775:6775:1017/143449:ERROR:gpu_info_collector.cc(89)] Could not create surface for info collection.
[6775:6775:1017/143449:ERROR:gpu_main.cc(402)] gpu::CollectGraphicsInfo failed (fatal).
[6775:6775:1017/143449:ERROR:sandbox_linux.cc(305)] InitializeSandbox() called with multiple threads in process gpu-process
[6775:6775:1017/143449:ERROR:gpu_child_thread.cc(143)] Exiting GPU process due to errors during initialization
[6736:6736:1017/143449:ERROR:gpu_process_transport_factory.cc(418)] Failed to establish GPU channel.
It seems that it works. It’s normal that we get the gpu errors since we don’t have a gpu! However I don’t like gambling so we will take it a step further to check that the browser actually works. However for this I will need to download the webdriver for Google Chrome..
A few days ago I wrote a tutorial on how to setup docker and use it between different machines. Now that was a nice first insight on how to jump-start using docker. It was also a nice way to showcase the possibilities and limitations of Docker.
In this post I will give some practical information on how to use docker as a developer.
Setup
To use Docker for development of software we want mainly three things:
Have our source code on the host machine. That way we can use GUI editors and whatever tools we want from outside the container.
Be able to have multiple terminals to the same container. This is good for debugging
Setup a docker image which we will use for running our program. I will use Django and Python for that.
And for the visual brains out there:
As you see in the pic, I am using Ubuntu as my host machine. At the same machine I have a folder with the source code and two terminals. Then I run a container with OpenSUSE. The folder and terminals reside ont he host machine but they communicate directly with the container. I will describe below how to achieve all this.
Multiple terminals
The easiest way to have multiple terminas is to use a small tool called nsenter. The guide can be found at https://github.com/jpetazzo/nsenter but it sums up to running this one-liner from any folder:
> docker run --rm -v /usr/local/bin:/target jpetazzo/nsenter
That installs nsenter on the host machine. After that, we can use it directly. So let’s try it. Open bash in a container with ubuntu as our basic image:
> docker run -t -i ubuntu /bin/bash
root@04fe75de21d4:/# touch TESTFILE
root@04fe75de21d4:/# ls
TESTFILE boot etc lib media opt root sbin sys usr
bin dev home lib64 mnt proc run srv tmp var
In the terminal above, I created a file called TESTFILE. We will try to open a second terminal and check to see the file from it.
To use xsenter we need the process ID of the container. Unfortunately we can’t use ps aux but rather have to use docker’s inspect command. I open a new terminal and type the below
If I now create a file inside /home/manos/myproject the change will be reflected from inside the container and vice versa. Play a bit with it by creating and deleting files from either the host or from inside the container to see for yourself.
Create a user in the container
It is wise to have a normal user in your image. If you don’t then you should create one and save the image. That way the source files can be opened from a normal user on your host – you won’t need to launch your IDE with root privileges.
> adduser manos
..
Follow the instructions and then commit your image. That way whenever you load the image again, you will be having user manos. To change to user manos just type
> su manos
All files you create now, will be accesible by a normal user at the host machine. Something else you could do is to somehow
Real life scenario: Python, Django and virtualenv
I wanted to learn Django. Installing Django is commonly made with the package manager pip, but pip has a bad history of breaking up things since it doesn’t communicate with Debian’s apt. So at some point if you installed/uninstalled python stuff with apt, pip wouldn’t know about it and vice versa. In the end you would end up with a broken python environment. That’s why a tool called virtualenv is being used – a tool that provides isolation. Since we have docker though which also provides isolation we can simply use that.
So what I really want:
Have the source code on my host.
Run django and python inside a container.
Debug from at least two terminals.
Visually my setup looks as something like this:
I assume you have an image with django and python installed. Let’s call the image py3django.
Firstly create the folder where you want your project source code to be. This is the folder that we will mount. My project resides in /home/manos/django_projects/myblog for example.
Once it’s created I just run bash on the image py3django. This will be my primary terminal (terminal 1):
The flag -p makes sure that docker doesn’t choose a random port for us. Since we run Django we will want to run a web server with a fixed port (on 8000). The flag -v mounts our host folder /home/manos/django_projects/myblog to the container’s folder /home/myblog. py3django is the image I have.
Now we have a folder where we can put our source code and a working terminal to play with. I want though a second terminal (terminal 2) to run my python webserver. So I open a second terminal and type:
Mind that I had to put the appropriate container ID in the command above.
Now all this is very nice but admittedly it’s very complex and it will be impossible to remember all these commands and boring to type them each single day. Therefore I suggest you create a BASH script that initiates the whole thing.
For me it took a whole day to come up with the script below:
#! /bin/bash
django_project_path="/home/manos/django_projects/netmag" # Path to project on host
image="pithikos/py3django_netmag_rmved" # Image to run containers on
echo "-------------------------------------------------"
echo "Project: $django_project_path"
echo "Image : $image"
# 1. Start the container in a second terminal
proj_name=`basename $django_project_path`
old_container=`docker ps -n=1 -q`
export docker_line="docker run -i -t -p 8000:8000 -v $django_project_path:/home/$proj_name $image /bin/bash"
export return_code_file="$proj_name"_temp
rm -f "$return_code_file"
gnome-terminal -x bash -c '$docker_line; echo $? > $return_code_file'
sleep 1
if [ -f "$return_code_file" ] && [ 0 != "$(cat $return_code_file)" ]
then
echo
echo "--> ERROR: Could not load new container."
echo " Stop any other instances of this container"
echo " if they are running and try again."
echo
echo " To reproduce the error, run the below:"
echo " $docker_line"
echo
rm -f "$return_code_file"
exit 1
fi
rm -f "$return_code_file"
# 2. Connect to the new container
while [ "$old_container" == "`docker ps -n=1 -q`" ]; do
sleep 0.2
done
container_ID=`docker ps -n=1 -q`
sudo nsenter --target $(docker inspect --format {{.State.Pid}} $container_ID) --mount --uts --ipc --net --pid
This script starts a container on a second terminal and then connects to the container from the current terminal. If starting the container fails, an appropriate message is given. django_project_path is the full path to the folder on the host with the source code. The variable image holds the name of the image to be used.
You can combine this with devilspie, an other nice tool that automates the position and size of windows when they’re launched.
In case you wonder about the top window with all the containers, that’s simply a watch command, a tool that updates regularly a command. In my case I use watch with docker ps. Simple stuff:
> watch docker ps
I use this because I personally like having an overview on the running containers. That way I don’t end up with trillions of forgotten containers that eat up my system.
Now that you have everything setup you can also run django server from one of the two terminals or whatever else you might want.
So as an intern in a big company I was given the task to get comfortable with docker. The problem is that docker is quite fresh so there isn’t really that much of good tutorials out there. After reading a bunch of articles and sparse tutorials (even taken the official tutorial at https://www.docker.com/tryit), I still straggled to get a firm grip on what docker even is supposed to be used for. Therefore I decided to make this tutorial for a total beginner like me.
Docker vs VirtualBox
There are many different explanations on the internet about what docker is and when to use it. Most of them however tend to complicate things more than giving some practical information for a total beginner.
Docker simply put is a replacement for virtual machines. I will use virtual box as a comparison example since it’s very easy for anyone to download and try to see the differences themselves.
The application VirtualBox is essentially a virtual machine manager.
Each and every of the OSes you see in the picture above, is an installed virtual machines. Each such machine has it’s own installed OS, kernel, virtual devices like hard disks, network cards etc. All this takes a considerate amount of memory and needs extra processing power. All virtual machines (VM) like VMware, Parallels, behave the same.
Now imagine that we want to use nmap from an OpenSUSE machine but we are on an Ubuntu. Using VirtualBox we would have to install the whole OS and then run it as a virtual machine. The memory consumption is humongous for such a trivial task.
In contrast to VirtualBox or any other virtual machine, Docker doesn’t install the whole OS. Instead it uses the kernel and hardware of our primary computer (the host). This makes Docker to virtualize super fast and consume only a fraction of the memory we would else need. See the benefits? Imagine if we wanted to run 4 different programs on 4 different OSes. That would take at a minimum 2GB of RAM.
But why would you want to run nmap on openSUSE instead of the host computer? Well this was just a silly example. There are other examples that prove the importance of a tool like Docker. Imagine that you’re a developer and you want to test your program on 10 different distributions for example. Or maybe you are the server administrator on a company and just updated your web server but the update broke something. No problem, you can run your web server virtualized on the older system version. Or maybe you want to run a web service in a quarantine for security reasons. As you see there are loads of different uses.
One question might rise though: how do we separate each “virtual machine” from the rest of the stuff on our computer? Docker solves this with different kernel (and non-kernel) mechanisms. We don’t have to bother about them though, since Docker takes hands of everything for us. That’s the beauty of it afterall: simplicity.
Install docker
Docker is in the ubuntu repositories (Ubuntu 14.04 here) so it’s as straightforward as: sudo apt-get install docker
Once installed, a daemon (service) of the docker will be running. You can check that with
sudo service docker.io status
The daemon is called docker.io as you might have noticed. The client that we willuse is simply called docker. Pay attention to this tiny but significant detail.
Configuration
Do these two things before using docker to avoid any annoying warnings and problems.
Firstly we need to add ourselves to the docker group. This will let us to use docker without having to use sudo every time:
sudo adduser <your username here> docker
Log out and then in.
Secondly we will edit the daemon configuration to ensure that it doesn’t use any local DNS servers (like 127.0.0.1). Use your favourite editor to edit the /etc/default/docker.io file. Uncomment the line with DOCKER_OPTS. The result file looks like this for me:
# Docker Upstart and SysVinit configuration file
# Customize location of Docker binary (especially for development testing).
#DOCKER="/usr/local/bin/docker"
# Use DOCKER_OPTS to modify the daemon startup options.
DOCKER_OPTS="-dns 8.8.8.8 -dns 8.8.4.4"
# If you need Docker to use an HTTP proxy, it can also be specified here.
#export http_proxy="http://127.0.0.1:3128/"
# This is also a handy place to tweak where Docker's temporary files go.
#export TMPDIR="/mnt/bigdrive/docker-tmp"
We need to restart the daemon for the change to take effect:
sudo service docker.io restart
Get an image to start with
In our scenario we want to virtualize an Arch machine. On VirtualBox, we would download the Arch .iso file and go through the installation process. In Docker we download a “fixed” image from a central server. There are thousands of different such image files. You can even upload your own image as you will see later.
This will download a default image for Arch Linux. “base/arch” is the identifier for the Arch Linux image.
To see a list of all the images locally stored on your computer type
> docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
base/arch 2014.04.01 a64697d71089 12 weeks ago 277.1 MB
base/arch latest a64697d71089 12 weeks ago 277.1 MB
Starting processes with docker
Once we have an image, we can start doing things in it as if it was a virtual machine. The most common thing is to run bash in it:
> docker run -i -t base/arch bash
[root@8109626c57f5 /]#
See how the command prompt changed? Now we are inside the image (virtual machine) running a bash instance. In docker jargon we are actually inside a container. The string 8109626c57f5 is the ID of the container. You don’t need to know much about that now. Just pay attention to how we acquired that ID, you will need it.
Let’s do some changes. Firstly I want to install nmap. Since pacman is the default package manager in Arch, I will use that:
> nmap www.google.com
Starting Nmap 6.46 ( http://nmap.org ) at 2014-07-18 13:33 UTC
Nmap scan report for www.google.com (173.194.34.116)
Host is up (0.00097s latency).
..
It seems we installed it successfully! Let’s also create a file:
[root@8109626c57f5 /]# touch TESTFILE
So now we have installed nmap and created a file in this image. Let’s exit the bash
[root@8109626c57f5 /]# exit
exit
>
In VirtualBox you can save the state of the virtual machine at any time and load it later. The same is possible with docker. For this I will need the ID of the container that I was using. In our case that is 8109626c57f5 (it was written in the terminal prompt all the time). In case you don’t remember the ID or you have many different containers, you can list all the containers:
> docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS
8109626c57f5 base/arch:2014.04.01 bash 25 minutes ago Exit 0
Let’s save the current state to a new image called mynewimage:
> docker commit -m "Installed nmap and created a file" 8109626c57f5 mynewimage
6bf56047833bd41c43c9fc3073424f37bfbc96993b65b868cb8d6a336ac28b0b
Now we have the saved image locally on our computer. We can load it anytime we want to come back to this state. And the demonstration..
> docker run -i -t mynewimage bash
[root@55c343f1643a /]# ls
TESTFILE bin boot dev etc home lib lib64 mnt opt proc root run sbin srv sys tmp usr var
[root@55c343f1643a /]# whereis nmap
nmap: /usr/bin/nmap /usr/share/nmap /usr/share/man/man1/nmap.1.gz
Loading the image on an other computer
We now have two images, the initial Arch image we started with and the new image that we saved:
> docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
mynewimage latest 6bf56047833b 2 hours ago 305.8 MB
base/arch 2014.04.01 a64697d71089 12 weeks ago 277.1 MB
base/arch latest a64697d71089 12 weeks ago 277.1 MB
It’s time to load the new image on a totally different computer. First I need to save the image on a server though. Luckily for a docker user, this is very simple. First you need to make an account at https://hub.docker.com/
Once that is done we need to upload the image to the hub. However we have to save the image in a specific format, namely username/whatever.
Let's save the image following that rule:
> docker commit -m "Installed nmap and created a file" 8109626c57f5 pithikos/mynewimage
12079e0719ce517ec7687b4bf225381b99b880510cda3bc1e587ba1da067bd3b
> docker push pithikos/mynewimage
The push refers to a repository [pithikos/mynewimage] (len: 1)
Sending image list
..
Once everything is uploaded, we can pull it from anywhere just as we did when we first pulled the Arch image.
I will do that from inside an OpenSUSE install on a totally different machine. First I try to run nmap
As you see it’s not installed on the computer. Let’s load our Arch image that we installed nmap on
Once the download of the image is complete we will run a bash on it just to look around:
As you see, the TESTFILE is in there and we can run nmap. We are running the same Arch I ran earlier on Ubuntu, on a totally new machine with a totally different OS, but still running it as an Arch.
A bit on containers
Now you probably got a good idea on what images are. Images are simply states of a “virtual machine”.
When we use docker run whatever we are running is put inside a container. A container is pretty much a Linux concept that arose recently with the recent addition of Linux containers to the kernel. In practise container is running a process (or group of processes) in isolation from the rest of the system. This makes the process in the container to not being able to have access to other processes or devices.
Every time we run a process with Docker, we are creating a new container.
> docker run ubuntu ping www.google.com
PING www.google.com (64.15.115.20) 56(84) bytes of data.
64 bytes from cache.google.com (64.15.115.20): icmp_seq=1 ttl=49 time=11.4 ms
64 bytes from cache.google.com (64.15.115.20): icmp_seq=2 ttl=49 time=11.3 ms
^C
> docker run ubuntu ping www.yahoo.com
PING ds-any-fp3-real.wa1.b.yahoo.com (46.228.47.114) 56(84) bytes of data.
64 bytes from ir2.fp.vip.ir2.yahoo.com (46.228.47.114): icmp_seq=1 ttl=45 time=46.5 ms
64 bytes from ir2.fp.vip.ir2.yahoo.com (46.228.47.114): icmp_seq=2 ttl=45 time=46.1 ms
^C
Here I ran two instances of the ping command. First I pinged http://www.google.com and then http://www.yahoo.com. I had to stop them both with CTRL-Z to get back to the terminal.
> docker ps -a | head
CONTAINER ID IMAGE COMMAND CREATED STATUS
7c44887b2b1c ubuntu:14.04 ping www.yahoo.com About a minute ago Exit 0
72d1ca1b42c9 ubuntu:14.04 ping www.google.com 7 minutes ago Exit 0
As you see, each command got its own container ID. We can further analyse the two containers with the inspect command. Below I compare the two different ping commands I ran to make it more apparent how the differentiate in the two containers:
Notice that I don’t have to write the whole string. For example instead of 7c44887b2b1c, I just type the first three letters 7c4. In most cases this will suffice.
sed is a very famous tool to the UNIX* community but which is very often misused. Most people try to use it in cases that it’s not the right tool or they tend to use it in the wrong way. I try in this post to show the things that are worth knowing when it comes to sed. I thus skip things like buffer holders, labels etc. which just make scripts totally unreadable for no benefit. I also talk a bit about the inner-workings of sed so the user has a grasp of why sometimes things don’t work as expected.
Why even learn sed?
For two reasons:
You can automate any boring mechanical work you would do on a normal text editor
If you know the syntax of sed, then you are a better vi/vim user
The second point, becomes obvious when you realise that the two programs have similar if not the exact syntax. For example, substituting the word “cow” with the word “horse” in the third line of the document in vim is :3s/cow/horse/ while in sed it’s 3s/cow/horse/. See the magic?!
A bit of history
Sed, awk and grep are the offsprings of a line editor called ed. ed pretty much let the user to edit one line at a time. That is also the reason that sed, awk and grep work on lines. All three programs have inherited the syntax of ed to some extend. In ed, to search & replace the word “cow” with the word “milk” in a text document, someone would type s/milk/beer/. That is exactly the same command used in sed – an indication of ancestry.
The tool grep actually takes its name from the command g/re/p, a command used in ed to only show lines that contain a specific regular expression. The ‘p’ in the end means to print on the screen while the ‘g’ in front means to go through all the lines.
When to use sed
While all three of these tools (grep, sed and awk) work on lines, sed and awk are very similar to each other while grep is more of a loner. Grep is used merely to filter out (or in) a line based on a regular expression. Sed and awk offer much more. What differs sed from awk is the data that they were built to edit.
Awk should be used when every line in the file has a specific structure. In other words that includes files where each line has a specific number of fields with every field separated with a delimiter (in most cases a tab). Such files can be CVS files, tables, the output of ls in linux, and more.
For everything else, use sed. Common examples are raw texts, this post, a C file, a script, an HTML file, etc. What all these files have in common is that lines don’t have a specific structure: the first line can have one word, while the second can have 100 words etc. Sed can still edit files that awk edits, but the opposite is most times impossible. If you are trying to just do that then you are most probably using the wrong tool.
Syntax
Sed in the terminal
The common syntax for sed in the terminal is:
sed SCRIPT INPUTFILE
The meat of sed is the SCRIPT and that is pretty much what I cover in this post. It’s a good convention to put quotation marks around it in case there is a space inside it (the shell might interpret it as multiple commands then).
sed can have multiple SCRIPTS or it can use a file with commands.
Multiple script lines:
sed 'SCRIPT1; SCRIPT2; SCRIPT3;' INPUTFILE
Using a script file:
sed -f SCRIPTFILE INPUTFILE
The SCRIPTFILE should have a command on each line. So SCRIPT1 should be on separate line than SCRIPT2 and SCRIPT3 on a separate etc.
Thoughout
Sed’s script syntax
Sed uses three things to accomplish tasks:
line specifiers (address)
commands
flags
The syntax of a single SCRIPT line is:
<line><command><flag>
A line or line specifier is a way to specify which lines you want the command to affect (parse). If the line specifier is missing, then the command affects all lines, which is the default behaviour.
A command is denoted by single character. For example to replace (substitute) a word with an other word we use the command ‘s’:
s/word1/word2/
A flag is used to modify the <command> or the <line>’s behaviour a bit and is placed after the whole command or line. Using the flag g on the example above, we get:
s/word1/word2/g
g stands for global and is used to replace all word1 in the line. Without it, only the first occurrence of word1 in the line is replaced.
A flag goes hand in hand with regular expressions so they can only be used if <command> or <line> have a regular expression in them. If the <line> is specified with a number for example, it’s illegal to use a flag:
#Illegal
sed -n '6g'
The reason is that flags are made for text strings (patterns) so it doesn’t make sense to sed when you are telling sed to use the flag ‘g’ on a line (which is something else than a string) and not a pattern.
The minimum thing needed on a SCRIPT line is a command or a line specifier. Someone can have both, one of them – with a flag or without. These combinations (or permutations to be exact) are allowed:
A flag applies to the command or line before it and assume that the previous has a regular expression in it.
How sed works internally
Imagine we have the file list.txt with the lines:
Today I will drink my milk
and afterwards I will eat a cow.
The cow will taste like cow.
Sed works with lines. As stated earlier we can have multiple script lines seperated with question marks:
sed 's/Today/Tomorrow/g; s/Today/Next Friday/g' list.txt
sed has a working buffer for each line (called pattern space). sed will initially load the first line of list.txt in the buffer. Then it will go through all script lines one by one altering everything in-place (in the buffer). In our example the buffer for the first line initially has:
Today I will drink my milk
After the first script command executes, it becomes:
Tomorrow I will drink my milk
The second script line doesn’t alter anything as sed can’t find an occurrence of the word ‘Today’ since it just got altered.
Once the first line is done, sed will load the second line in the buffer and go through the same procedure until all lines in list.txt have been parsed.
Something important to notice is that each SCRIPT line will run regardless if the previous SCRIPT line succeeded or not. With success we mean that the command did what it is meant to do. If substitution is used, then we define success as the alteration of a line. If we just specify a line, then success is if the line exists etc.
Addressing specific lines
<line><command><flag>
By default, sed goes through all the lines. However, one can address a specific line or a range of lines. That can be done by specifying lines by their number in the file (1st line, 2nd line, 50th line etc.) or by a line’s content.
To run a command on the 10th line we do:
10<command>
To run a command on each line that contains the word “cow” we do:
/cow/<command>
The latter makes use of regular expressions. When we use regular expressions we need to add slashes to the start and end of the regular expression.
For a range of lines we use a comma (awkward, I know). To specify all lines between the 10th and 25th line (including those) we would write:
10,25<command>
If we want to specify a range of lines by using regular expressions we still have to encapsulate the regular expressions in slashes. To run a command on all the lines between the first line found with the word “cow” and the first line found with the word “grass”, we would issue:
/cow/,/grass/<command>
Bellow you can see all the ways for specifying lines.
<line1>,<line2>
Lines between <line1> and <line2> (including those)
<line1>~N
Every Nth line after <line1>
<line1>!
All lines except line1
/<regex>/
Lines matching the regular expression
$
Last line
<line1>,+N
All N number of lines after <line1>
Some more practical examples can be seen bellow. The command p is used to print the line that is specified. (Notice that sed needs the parameter -n for the command p to work.)
sed -n '2p' -> print line 2
sed -n '2,4p' -> print line 2 to 4
sed -n '$p' -> print last line
sed -n '2!p' -> print all lines but line 2
sed -n '/red/p' -> print every line that contains the word red
sed -n '/red/,/green/p' -> print all lines between the <strong>first occurrences</strong> of the words 'red' and 'green'
Of course for all these examples to work you need to either feed sed with a stream from a file or a pipe.
Using commands
<line><command><flag>
Now we are to the meat of all meats.I have explained the substitution command a bit but bellow you can see all the commands with their syntax (if they have one).
s/<regex>/<subst>/
substitute
Replace a match (using regular expression) with a string.
p
print
Prints a specific line or a range of lines. The sed flag -n should be used for this command to work.
=
line number
Shows the number of the line
d
delete
Deletes (omits) the matched line
y/<char1>/<char2>/
transform
Substitutes char1 with char2. Works even with a sequence of characters. For example y/abc/ABC/ will replace a with A, b with B and c with C.
Substitution
Substitution is the most commonly used command. It’s syntax is as follows:
s<d><regex><d><subst><d>
where <d> is a delimiter which should be a single character. Most commonly the delimiter is a slash / but it can essentially be any character. All lines bellow are equivalent.
s/cow/horse/
s_cow_horse_
sDcowDhorseD
In the example above the first occurrence of the word cow on each line will be replaced with the word horse, which is the default behaviour. If you want all occurrences of the word cow in a line to be substituted, the flag g (global) has to be appended to the line:
s/cow/horse/g
The substitution field <subst> can take some special variables like the ampersand symbol “&” which holds the matched string from the regular expression. There are also a few macros to automate conversion of capitals to lower-case and vice versa.
All these are shown in the table bellow.
&
Holds the matched string.
\1
Holds a part of the match specified in the regular expression with parentheses.
\U<string>
Converts all letters in <string> to capitals.
\u<string>
Converts the first letter in <string> to capital.
\L<string>
Converts all letters in <string> to lower-case.
\l<string>
Converts the first letter in <string> to lower-case.
<string>\E
Ends the conversion at specific point. Should be used in conjunction with \L and \U.
The match holders \1\2\3 etc. have to be specified in the regular expression with parentheses. The parentheses themselves have to be escaped or else sed will be looking for parenthesis characters in the line. So to replace each word “cow” with “supercow” we can do:
s/\(cow\)/super\1/
or
s/cow/super&/
The latter is more elegant of course. However there are two cases where the specific holder has to be used:
When we want only a portion of the matched string and not the whole string (&).
When there are many different strings you want to grab from a match.
For example this can’t be solved by merely using the & symbol:
s/\(\w*\) cow \(\w*\)/\2 cow \1/
This script will look for each occurrence where “cow” has a word before it and a word after it and it will change their order. The \w matches any character while the asterisk * tells the pattern that the word can be arbitrarily long. We use the parentheses around the first word and the second word to denote which matched parts of the regular expression should be given to \1 and \2. In <subst> we just reverse their order by putting the second word first and the first word second.
Flags
<line><command><flag>
A flag can be used on a <command> or a <line> or both.
g
global
For all occurences in the line (default is to stop at first occurence)
I
Ignore case
Not case-sensitive
p
Print
Output only this line (not everything as default). The sed flag -n should be used for this flag to work.
Find line in lines of lines in lines..
An important concept to comprehend in sed is nesting. I try to leave out all the advanced things sed offers, like holder buffer (horror to read), labels etc. but nesting is worth learning as it gives a lot of extra power for a little learning curve. Nesting is similar to the IF..THEN conditional.
We have this text file: Today I will drink my milk
and afterwards I will eat a cow.
The cow will taste like cow.
Today is not afterwards if I am a cow. Right?
Imagine that we want to check if the last line of the text contains the word cow. Someone might think that putting $p together with /cow/p would work:
sed -n '$p; /cow/p'
Admittedly the output seems strange: and afterwards I will eat a cow.
The cow will taste like cow.
Today is not afterwards if I am a cow. Right?
Today is not afterwards if I am a cow. Right?
The reason this doesn’t work as expected is that the second script runs regardless if the first one succeeded or not. Thus in the first line none of the scripts print anything. On the second line, the first script fails but the second script finds the word cow so it prints the line. The same happens at the third line. At the forth line, the first script succeeds as the line is the last line of the file, so that line gets printed. Then the second script runs and that succeeds. Thus we get the line printed again (a second time).
A way to solve this is is to nest the second script line in the first somehow so that it runs only if the first script line has succeeded. This is similar to the pseudocode:
if <line1>
then SCRIPT
Where <line1> is a line specified by a number, range or pattern (regular expression).
The syntax for nesting script lines in sed is
<line>{<line><command><flag>}
For our example:
sed -n '${/cow/p}'
This translates to: if this is the last line ($) then do whatever is in the brackets. So everything in the brackets will be checked only if the current line being parsed is the last one.
We can even nest inside a nested script:
sed -n '2,4{/cow/{/Today/p}}'
This script will go through lines 2 to 4. It will check first if a line contains the word cow. If the line contains the word cow, then it will check if it contains the word Today. If it does, it will print it. Ofcourse there is no practical reason to have second braces in the above example. I just wrote it to show that it’s feasible. In some cases you need braces to accomplish tasks, especially in cases where we avoid using too advanced things.
Examples
Printing a specific line
sed -n '2p'
The p in this case is the p command (and not flag). Remember that flags apply only to regular expressions. When we use sed’s -n parameter, only lines specified with the p command or flag will be printed on screen.
Printing a range of lines
sed -n '1,3p'
The comma is used to specify a range. In this example we specify line 1 to line 3 and then we print each such line.
Hiding a specific line
sed -n '2!p'
This is similar to:
sed '2d'
Show the last line
sed -n '$p'
In regular expressions the dollar sign $ denotes the end of a string. In sed when it’s being used with substitution it denotes the end of a line.
However when used with the command p, it denotes the last line of the input. In the same way the regex symbol ^ denotes the first line.
Converting a specific word to uppercase
sed 's/Today/\U&/' list.txt
This will match any word ‘Today’ and replace it with ‘TODAY’. In the replacement the escaped U (\U) tells sed to convert to uppercase everything following in the replacement string. In this example we use an ampersand which in sed represents the matched string. If we wanted to stop the conversion we just need to add \E where we want it to end.
Converting all text to uppercase
sed 's/.*/\U&/'
For this we use the substituion command s. .* is a regular expression which fits any sequence of characters. As sed is working on lines, .* matches a whole line each time.
Converting all text to lowercase
sed 's/.*/\L&/'
Similar to the previous example with only difference that we use \L instead of \U.
Grabbing all content between the body tags in HTML
Say we have the ugly HTML code bellow and we want to grab all the content between the body tags.
<html><head></head>
<h1>h1 outside of body</h1><body><h1>h1 stuck to body</h1>
<img src="images/soon.png"/>
<p>a paragraph</p></body></html>
Most sed experts would go about using some very advanced commands to accomplish this. The readability of that becomes horrific. As my opinion is that you can do the same things without knowing all the advanced commands I am going to just do that.
Notice that I use pipes instead of using advanced commands.
Solution:
sed -n '/<body>/,/<\/body>/p' | sed 's_.*<body>\(.*\)_\1_; s_\(.*\)</body>.*_\1_'
It might seem like a mess but I can assure you that it’s much more elegant than a pure single sed SCRIPT solution. I will break it down so you can see how it works.
/body/,/body/p
matches all lines between the body tags, including the body tags. In this way I have minimized my problem to:
<h1>h1 outside of body</h1><body><h1>h1 stuck to body</h1>
<img src="images/soon.png"/>
<p>a paragraph</p></body></html>
After that, we are sure that the first line has the start of the body tag and the last line has the closing of the body tag. So we start by filtering out things we don’t need from the first line: the tag itself and everything preceding it.
s_.*<body>\(.*\)_\1_
I use _ as a delimeter instead of slashes to make the substitution code a bit more readable. The regular expression .* matches 0 to infinite number of arbitrary characters. So I use it around the body tag in case there is something before and after it. I put the second .* in parentheses to grab the text that might be there as I want to keep that. Using \1 in the substitution field I accomplish substituting the whole line with the text after body.
s_\(.*\)</body>.*_\1_
We do something similar to the line with </body>. The only difference is that now the portion of the match that we are interested in is the one before the body tag so we move the parentheses there.
Keep in mind that this solution will not work if the body tag includes some attributes like style. Someone might think that just using the bellow regular expression in the substitution would work.
.*<body.*>\(.*\)
Notice that the only difference is that we added the regular expression .* between body and its closing arrow > to point out that there can be nothing in between or there could be some arbitrary things (in our case attributes).
That regular expression doesn’t work however as it matches the last > in the line. The reason is that in sed .* is greedy and there is no way to make it non-greedy. With greedy we mean that the pattern will try to match as much as it can in the line. So if you want to match the first > it’s not possible. Or.. actually it is possible but the code as you will see in the next example starts looking like a monster.
Grabbing all content between tags in HTML
You are probably better off learning Perl or Python if you need to do these kind of “advanced things”. I will however show that you can achieve things like this without using the more advanced commands or other programs/languages. This solution is a continuation of the previous example on catching the content between the body tags. The mere difference is that in this solution we allow even attributes to a tag and are a bit more permissive towards whitespaces. This makes it a more general solution that can be used for other tags than the body tag.
Solution:
sed -n '/<body>/,/<\/body>/p' | sed '1s_.*<body[a-zA-Z0-9="\x27_ ]*> *<\(.*\)_<\1_; 1{/<body>/d}; s_\(.*\)</body>.*_\1_'
Essentially the only thing that got changed from the previous example is the alteration of the commands in the first line:
The first script line (everything before the first ;) looks for a second tag after body. We don’t really need to specify line 1 but it is a good convention as it makes the code easier to understand and the processing faster. I am looking for the body tag followed by an attribute or not. TO define an attribute in HTML, only the characters in the brackets are allowed (says the HTML protocol, not me). \x27 is the code for a single quote mark ‘. The reason I use its code instead of the mark itself (I use the double quote after all), is that I use the single quote marks around the whole command so if I insert it in the expression, then it will break it. After that I use ” *<" (notice the space) to denote that there might be an arbitrary number of spaces or none before the opening of the new tag. I replace everything with the opening of the tag after body with the rest of the line.
If the first command didn’t succeed then it means that there’s not a second tag after the body. Thus it’s safe to delete the whole line in that case. First we check if there is a body tag in the first line. If there is a body tag (/<body>/), then we delete it with the command d.
I was making a website the other day and I wanted to somehow pass variables that can be read with javascript. So if the user browsed to http://www.example.com?height=100px&width=50px, the variables height and width should be read from javascript. Note that this method of passing the variables in the URL is used most notably for CGI, aka server-side scripting(PHP anyone?).
JSON
So I was a bit puzzled while looking around at stackoverflow as many people do think that there is something magical about Json. Json is nothing more than a standard on how things are stored. The standard pretty much sums up to this:
Variables are stored as varName = value
Arrays are stored as arrayName = [value1, value2, value3 .. ]
A value can be: a number, a string, true, false, none
The whole thing is encapsulated in wavy braces
For my example, the Json structure(after parsing the URL string) should look like this:
{
"height": "100px"
"width" : "50px"
}
In this case the values are strings. Keep in mind however that according to the standard, they could be anything between a number, a string, true, false and none.
My parser
So to get this structure from the URL, I needed some kind of parsing. All the solutions I found either used regular expressions, or they wanted a whole library to be imported, or just didn’t support arrays. So I made up my own ugly solution.
The function url2json() uses pure JavaScript code which doesn’t use regular expressions and accepts both arrays and variables:
function url2json(url) {
var obj={};
function arr_vals(arr){
if (arr.indexOf(',') > 1){
var vals = arr.slice(1, -1).split(',');
var arr = [];
for (var i = 0; i < vals.length; i++)
arr[i]=vals[i];
return arr;
}
else
return arr.slice(1, -1);
}
function eval_var(avar){
if (avar[1].indexOf('[') == 0)
obj[avar[0]] = arr_vals(avar[1]);
else
obj[avar[0]] = avar[1];
}
if (url.indexOf('?') > -1){
var params = url.split('?')[1];
if(params.indexOf('&') > 2){
var vars = params.split('&');
for (var i in vars)
eval_var(vars[i].split('='));
}
else
eval_var(params.split('='));
}
return obj;
}
To keep things clean, all values are parsed into strings. As the input of the function is a string it just makes sense to give back strings so no extra processing takes place if it’s not needed(checking if every value is of a certain type). It’s up to the user to convert the strings into numbers or whatever they want, if they really have to.
Parsing variables
To use the function with the example above, I would just run
I am working on a greasemonkey script where I want to be able and play mp3 files in the background. The <audio> tag would be a good and easy solution but unfortunately Firefox doesn’t support mp3 playback because of license issues.
With headless we mean gui-less. So I was looking for a dummy mp3 player that would just do the playback while all the control would be javascript driven.
I found in the end this simple .swf file here that supports the follow:
Play
Pause
Stop
Loop on/off
The problem I faced, as usual, was zero documentation. Although the source code of the mp3 player is available, it was hard for me to decode how to use the player, especially when I am totally unfamiliar with actionscript.
In the end I managed and in fact I made two minimal templates to help out future users.
index.html uses inline javascript inside the html code to control the mp3 player. In particular it makes normal <a> tags that once clicked send a query to the mp3player.swf:
This is the fast dirty way to get things work without using an external javascript file.
Scenario 2
Index.html2 loads an external javascript file which in this case is controlplayer.js. This is much more flexible and lets you add advanced behaviour from inside the javascript file.
does work as good as the previous code without breaking any functionality.
NOTICE: If controlling the playback doesn’t work once you unzipped, make sure that the files are on your website folder. Launching the .html files directly from a local folder(for example “desktop”) will not work. That is due to security reasons when it comes to flash.
LinuxMeerkat 