Docker Socket File for IPC
While setting up a Consul cluster, I decided to dig a bit deeper into the whole /var/run/docker.sock phenomenon. While it is fairly common that a lot of Service Meshes like Consul, and System Monitoring Services like Newrelic and DataDog ask to mount /var/run/docker.sock, I must admit I’ve always been curious about this particular socket (vs. taking the mount step for granted). These are my notes from calling Docker Server by curl-ing the socket.
Before we get to the socket file, here is a quick refresher on Unix Sockets and typical Docker setup for context.
Unix Sockets
The term Sockets commonly refers to IP Sockets. These are the ones that are bound to a port (and address), we send TCP requests to, and get responses from.
Another type of Socket is a Unix Socket, which I never really ran into as much after school, except in recent few years when dealing with Docker runtime. As a refresher, these sockets are used for IPC (Interprocess Communication). They’re also called Unix Domain Sockets (UDS). Unix Sockets use local filesystem for communication, while IP Sockets use the network. This makes Unix Sockets faster - but then again, they are confined to local communication only. Kind of like what containers would use to talk to the docker host on the same machine.
A good way to look at them is to list the /var/run folder, here is my Mac -
bash-3.2$ ls -al /var/run | grep sock
srw-rw-rw- 1 root daemon 0 Mar 29 11:46 com.docker.vmnetd.sock
lrwxr-xr-x 1 mpandit 1498914130 68 Mar 29 11:48 docker.sock -> /Users/mpandit/Library/Containers/com.docker.docker/Data/docker.sock
srwxrwxrwx 1 root daemon 0 Mar 29 11:46 portmap.socket
srw-rw-rw- 1 root daemon 0 Mar 29 11:46 systemkeychaincheck.socket
srw------- 1 root daemon 0 Mar 29 11:46 vpncontrol.sock
Docker on MacOS
Docker installation on the Mac via Docker Desktop has three components to it -
- The Docker Daemon (aka Docker Server), which runs atop HyperKit Hypervisor as a LinuxKit VM.
- The Docker CLI (
docker,docker-compose, anddocker-machine). - The Docker REST API, which is explosed via a Unix Socket at
/var/run/docker.sockby default.
Things are a little different when using Docker Toolbox (legacy), which uses docker-machine to launch the LinuxKit VM in VirtualBox.
In Docker Desktop, /var/run/docker.sock replaces docker-machine.
Docker Server uses this socket to listen to the REST API, and the clients use the socket to send API requests to the server. The CLI is one such client.
Using curl
curl is the swiss army knife for any communication. We can use curl to act as a client and use the Docker REST API. It’s a good idea to familiarize yourself with curl and all the powers this little command holds. Trust me on this.
curl can talk to a Unix Socket via the --unix-socket flag. Since Docker Server API is exposed as REST, we’d need to send commands over HTTP. Also, as this server is local (remeember, the file system), we can pass any hostname in the URL (or stick to localhost, that will work fine too!). The server does not care about the hostname, just the path.
Lets go!
We can start fresh, or if you already have a bunch of containers running, that is fine too. Your output may look a little different, but the commands will work just fine.
For this post, I’ll start with no images, or containers on the system.
Do not run system prune unless you’ve used it before, or are sure about wiping your Docker setup clean. In doubt? Skip to the next step.
bash-3.2$ docker system prune -a
WARNING! This will remove:
- all stopped containers
- all networks not used by at least one container
- all images without at least one container associated to them
- all build cache
Are you sure you want to continue? [y/N] y
Deleted Containers:
1a5e227d2251507e71fff25dfca60e9aea1f9a61d4a7cde078b8b2b357d70a71
f9dc7950dcd9b29a85197030579487f40cebbb2556f56180b6ca4929d3f61c9d
Deleted Images:
untagged: nginx:latest
untagged: nginx@sha256:c8a861b8a1eeef6d48955a6c6d5dff8e2580f13ff4d0f549e082e7c82a8617a2
deleted: sha256:2bcb04bdb83f7c5dc30f0edaca1609a716bda1c7d2244d4f5fbbdfef33da366c
deleted: sha256:dfce9ec5eeabad339cf90fce93b20f179926d5819359141e49e0006a52c066ca
deleted: sha256:166d13b0f0cb542034a2aef1c034ee2271e1d6aaee4490f749e72d1c04449c5b
Total reclaimed space: 54.01MB
bash-3.2$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
bash-3.2$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
bash-3.2$
Again, you may have images in the cache, and containers running. That is perfectly fine.
Lets start a redis image, which we will use for the rest of the commands.
bash-3.2$ docker run -d -p 6379:6379 redis:latest
Unable to find image 'redis:latest' locally
latest: Pulling from library/redis
Digest: sha256:000339fb57e0ddf2d48d72f3341e47a8ca3b1beae9bdcb25a96323095b72a79b
Status: Downloaded newer image for redis:latest
c025ae0e698eae82e8d783089693c5ada75b9c10ff01e3884255ed019b12c569
We just started a redis container, with ID c025ae0e698eae82e8d783089693c5ada75b9c10ff01e3884255ed019b12c569, which is listening on port 6379.
We can validate it via redis-cli or telnet. Type INFO and then QUIT to exit the session.
bash-3.2$ telnet localhost 6379
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
INFO
$3304
# Server
redis_version:5.0.3
redis_git_sha1:00000000
.....
# Keyspace
db0:keys=2,expires=0,avg_ttl=0
QUIT
+OK
Connection closed by foreign host.
Get images
We can use the images API to get the images. Note that we can use any hostname. We are connecting to the socket, and calling an API /images/json to get the images.
I use jq to format the output, among other things. Check it out - brew install jq
bash-3.2$ curl --unix-socket /var/run/docker.sock http://foo/images/json | jq
[
{
"Containers": -1,
"Created": 1553647740,
"Id": "sha256:a55fbf438dfd878424c402e365ef3d80c634f07d0f5832193880ee1b95626e4e",
"Labels": null,
"ParentId": "",
"RepoDigests": [
"redis@sha256:000339fb57e0ddf2d48d72f3341e47a8ca3b1beae9bdcb25a96323095b72a79b"
],
"RepoTags": [
"redis:latest"
],
"SharedSize": -1,
"Size": 95000962,
"VirtualSize": 95000962
}
]
Compare this to another API client - the CLI
bash-3.2$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
redis latest a55fbf438dfd 11 days ago 95MB
Notice the CLI renders the same output in a much more readable manner.
Inspect an Image
The API is /images/{id}/json. The output is huge, so I’ll truncate it.
bash-3.2$ curl --unix-socket /var/run/docker.sock http://foo/images/a55fbf438dfd/json | jq
{
"Id": "sha256:a55fbf438dfd878424c402e365ef3d80c634f07d0f5832193880ee1b95626e4e",
"RepoTags": [
"redis:latest"
],
"RepoDigests": [
"redis@sha256:000339fb57e0ddf2d48d72f3341e47a8ca3b1beae9bdcb25a96323095b72a79b"
],
"Parent": "",
"Comment": "",
"Created": "2019-03-27T00:49:00.123917534Z",
"Container": "6113cb2d4e6d2d4f375b72c8f5d5c953e3e7f4c22bfad42d5dd0aabf163b79bd",
"ContainerConfig": {
"Hostname": "6113cb2d4e6d",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"6379/tcp": {}
}
}
}
Again, the CLI command to do so would be docker inspect <imageId> (response truncated). Note the similarity between the two responses.
bash-3.2$ docker inspect a55fbf438dfd | jq
[
{
"Id": "sha256:a55fbf438dfd878424c402e365ef3d80c634f07d0f5832193880ee1b95626e4e",
"RepoTags": [
"redis:latest"
],
"RepoDigests": [
"redis@sha256:000339fb57e0ddf2d48d72f3341e47a8ca3b1beae9bdcb25a96323095b72a79b"
],
"Parent": "",
"Comment": "",
"Created": "2019-03-27T00:49:00.123917534Z",
"Container": "6113cb2d4e6d2d4f375b72c8f5d5c953e3e7f4c22bfad42d5dd0aabf163b79bd",
"ContainerConfig": {
"Hostname": "6113cb2d4e6d",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"6379/tcp": {}
}
}
]
Tag an image
Adding this example to perform a write operation. Let us tag the redis image with a tag, foo. We use the POST operation /images/{id}/tag and pass repo and tag as query params. I found this a bit odd, as POST with query string params is not something we run into often (or ever?). But here it is -
bash-3.2$ curl -i -X POST --unix-socket /var/run/docker.sock "http://foo/images/a55fbf438dfd/tag?repo=redis&tag=foo"
HTTP/1.1 201 Created
Api-Version: 1.39
Content-Length: 0
Date: Sun, 07 Apr 2019 08:47:38 GMT
Docker-Experimental: true
Ostype: linux
Server: Docker/18.09.2 (linux)
We can verify this further (beyond the 201 response)
bash-3.2$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
redis foo a55fbf438dfd 11 days ago 95MB
redis latest a55fbf438dfd 11 days ago 95MB
Get Containers
Here we will use the /containers/json just like /images/json -
bash-3.2$ curl --unix-socket /var/run/docker.sock "http://foo/containers/json" | jq
[
{
"Id": "c025ae0e698eae82e8d783089693c5ada75b9c10ff01e3884255ed019b12c569",
"Names": [
"/infallible_roentgen"
],
"Image": "redis:latest",
"ImageID": "sha256:a55fbf438dfd878424c402e365ef3d80c634f07d0f5832193880ee1b95626e4e",
"Command": "docker-entrypoint.sh redis-server",
"Created": 1554625562,
"Ports": [
{
"IP": "0.0.0.0",
"PrivatePort": 6379,
"PublicPort": 6379,
"Type": "tcp"
}
],
"Labels": {},
"State": "running",
"Status": "Up 24 minutes",
"HostConfig": {
"NetworkMode": "default"
}
]
}
]
The CLI, you ask?
bash-3.2$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c025ae0e698e redis:latest "docker-entrypoint.s…" 25 minutes ago Up 25 minutes 0.0.0.0:6379->6379/tcp infallible_roentgen
Stream Events
I’ll end this sequence of APIs with my favorite - events.
The /events API allows streaming of events on the Docker Server. This can be used to build rich interfaces, or pipelines among many possiblities.
Note that we’re adding --no-buffer to curl to print output as the events occur, flushing the output right away.
bash-3.2$ curl --no-buffer --unix-socket /var/run/docker.sock http://foo/events
In a new terminal window, we start another container, nginx. We will also verify that it works.
bash-3.2$ docker run -d -p 80:80 nginx:latest
Unable to find image 'nginx:latest' locally
latest: Pulling from library/nginx
Status: Downloaded newer image for nginx:latest
a9cc0132647be42cdcf3d2d4d5ea7ba1e980d5895978b97c1bf3735ed4408042
bash-3.2$ curl --head http://localhost:80
HTTP/1.1 200 OK
Server: nginx/1.15.10
Date: Sun, 07 Apr 2019 09:00:22 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 26 Mar 2019 14:04:38 GMT
Connection: keep-alive
ETag: "5c9a3176-264"
Accept-Ranges: bytes
bash-3.2$
If we go back to the terminal which was running the /events call, we see the following output -
{"status":"pull","id":"nginx:latest","Type":"image","Action":"pull","Actor":{"ID":"nginx:latest","Attributes":{"maintainer":"NGINX Docker Maintainers \u003cdocker-maint@nginx.com\u003e","name":"nginx"}},"scope":"local","time":1554627574,"timeNano":1554627574220666900}
{"status":"create","id":"a9cc0132647be42cdcf3d2d4d5ea7ba1e980d5895978b97c1bf3735ed4408042","from":"nginx:latest","Type":"container","Action":"create","Actor":{"ID":"a9cc0132647be42cdcf3d2d4d5ea7ba1e980d5895978b97c1bf3735ed4408042","Attributes":{"image":"nginx:latest","maintainer":"NGINX Docker Maintainers \u003cdocker-maint@nginx.com\u003e","name":"focused_hodgkin"}},"scope":"local","time":1554627574,"timeNano":1554627574386805100}
{"Type":"network","Action":"connect","Actor":{"ID":"11b3729299eba1d7ea896f3c41ad7f1797ed5730831352a7cc13e9aed7e9bf27","Attributes":{"container":"a9cc0132647be42cdcf3d2d4d5ea7ba1e980d5895978b97c1bf3735ed4408042","name":"bridge","type":"bridge"}},"scope":"local","time":1554627574,"timeNano":1554627574567980600}
{"status":"start","id":"a9cc0132647be42cdcf3d2d4d5ea7ba1e980d5895978b97c1bf3735ed4408042","from":"nginx:latest","Type":"container","Action":"start","Actor":{"ID":"a9cc0132647be42cdcf3d2d4d5ea7ba1e980d5895978b97c1bf3735ed4408042","Attributes":{"image":"nginx:latest","maintainer":"NGINX Docker Maintainers \u003cdocker-maint@nginx.com\u003e","name":"focused_hodgkin"}},"scope":"local","time":1554627575,"timeNano":1554627575041391200}
Notice that we’re seeing streaming events, as they occur on the Docker Server. We can stop the redis container, and see that show up in the event stream as well.
Also we can see the full sequence for docker run, which is pull_image->create_container->network_connect->start_container.
bash-3.2$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c025ae0e698e redis:latest "docker-entrypoint.s…" 37 minutes ago Up 37 minutes 0.0.0.0:6379->6379/tcp infallible_roentgen
bash-3.2$ docker stop c025ae0e698e
c025ae0e698e
And we have the events!
{"status":"kill","id":"c025ae0e698eae82e8d783089693c5ada75b9c10ff01e3884255ed019b12c569","from":"redis:latest","Type":"container","Action":"kill","Actor":{"ID":"c025ae0e698eae82e8d783089693c5ada75b9c10ff01e3884255ed019b12c569","Attributes":{"image":"redis:latest","name":"infallible_roentgen","signal":"15"}},"scope":"local","time":1554627858,"timeNano":1554627858776466300}
{"status":"die","id":"c025ae0e698eae82e8d783089693c5ada75b9c10ff01e3884255ed019b12c569","from":"redis:latest","Type":"container","Action":"die","Actor":{"ID":"c025ae0e698eae82e8d783089693c5ada75b9c10ff01e3884255ed019b12c569","Attributes":{"exitCode":"0","image":"redis:latest","name":"infallible_roentgen"}},"scope":"local","time":1554627859,"timeNano":1554627859011940300}
{"Type":"network","Action":"disconnect","Actor":{"ID":"11b3729299eba1d7ea896f3c41ad7f1797ed5730831352a7cc13e9aed7e9bf27","Attributes":{"container":"c025ae0e698eae82e8d783089693c5ada75b9c10ff01e3884255ed019b12c569","name":"bridge","type":"bridge"}},"scope":"local","time":1554627859,"timeNano":1554627859413159900}
{"Type":"volume","Action":"unmount","Actor":{"ID":"14e0049f6dd0cb3493d39b6d6ce709a95620d15f8d1f50bdfe6d63372896be62","Attributes":{"container":"c025ae0e698eae82e8d783089693c5ada75b9c10ff01e3884255ed019b12c569","driver":"local"}},"scope":"local","time":1554627859,"timeNano":1554627859500455300}
{"status":"stop","id":"c025ae0e698eae82e8d783089693c5ada75b9c10ff01e3884255ed019b12c569","from":"redis:latest","Type":"container","Action":"stop","Actor":{"ID":"c025ae0e698eae82e8d783089693c5ada75b9c10ff01e3884255ed019b12c569","Attributes":{"image":"redis:latest","name":"infallible_roentgen"}},"scope":"local","time":1554627859,"timeNano":1554627859510757400}
Notice the kill->die->network_disconnect->volume_unmount->stop event sequence triggered by the docker stop command.
Almost all the APIs are accessible via Docker CLI. This post should help connect the dots when certain containers ask to mount /var/run/docker.sock. This file provides a very rich (and fast!) IPC channel with the Docker Server, which can then be used for various functions like cluster management, monitoring, discovery, routing, etc.
I’d appreciate your comments and feedback below.