docker rootless模式和基于kind的k8s测试环境

发布于：2024-10-15 ⋅ 阅读:(57) ⋅ 点赞:(0)

什么是容器rootless（AI生成）

 
        容器rootless模式是一种在不依赖于root权限的情况下运行容器的方式。传统的容器技术通常需要root权限来创建和管理容器，这可能存在一些安全风险。而容器rootless模式通过使用用户命名空间技术和其他安全特性，使得非特权用户也能够创建和管理容器。

        容器rootless模式的核心思想是将容器的运行环境隔离到一个普通用户的命名空间中。在这个命名空间中，容器可以拥有自己的进程树、文件系统、网络栈和其他资源，但是它的权限受限于普通用户而不是root用户。

具体来说，容器rootless模式需要使用一些技术来实现：

1. 用户命名空间（User Namespace）：通过创建一个新的用户命名空间，容器可以拥有独立的用户和用户组，从而实现权限的隔离和控制。

2. 文件系统镜像（Filesystem Image）：使用文件系统镜像来创建容器的文件系统，这样每个容器都可以拥有自己独立的文件系统，并且文件系统中的文件和目录权限也受到用户命名空间的限制。

3. 网络隔离（Network Isolation）：通过使用网络命名空间和虚拟网络设备，容器可以拥有独立的网络栈，从而实现网络的隔离。

4. 容器运行时（Container Runtime）：容器rootless模式需要使用特殊的容器运行时，该运行时可以在非特权用户的环境中创建和管理容器。

        容器rootless模式的优点是可以提供更加安全的容器环境，减少了对root权限的依赖，从而减少了潜在的安全风险。同时，容器rootless模式也可以使得非特权用户可以更加灵活地使用容器技术，提高了容器的可用性和可扩展性。但是容器rootless模式也可能存在一些限制和性能损失，因此需要根据具体的使用场景进行评估和选择。

测试环境情况

1. Ubuntu Server 22.04.5

2. 按照官方说明[Optional] cgroup v2 | Rootless Containers进行cgroup v2相关配置

3. 配置内核参数net.ipv4.ip_forward=1

4. apt安装uidmap包

5. 创建一个测试用户（后续所有操作请使用测试用户）

 

docker rootless安装

1. 下载docker-ce二进制文件和rootless扩展，解压后将内部文件拷贝到用户目录bin（没有就新建）下，下面为国内下载链接：

  • https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/static/stable/x86_64/docker-27.3.1.tgz
  • https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/static/stable/x86_64/docker-rootless-extras-27.3.1.tgz 

2. 执行bin/dockerd-rootless-setuptool.sh install命令进行安装：

longtds@ubuntu:~$ bin/dockerd-rootless-setuptool.sh install
[INFO] Creating /home/longtds/.config/systemd/user/docker.service
[INFO] starting systemd service docker.service
+ systemctl --user start docker.service
+ sleep 3
+ systemctl --user --no-pager --full status docker.service
● docker.service - Docker Application Container Engine (Rootless)
     Loaded: loaded (/home/longtds/.config/systemd/user/docker.service; disabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-10-14 06:07:01 UTC; 3s ago
       Docs: https://docs.docker.com/go/rootless/
   Main PID: 1781 (rootlesskit)
      Tasks: 42
     Memory: 41.2M
        CPU: 1.253s
     CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/docker.service
             ├─1781 rootlesskit --state-dir=/run/user/1000/dockerd-rootless --net=vpnkit --mtu=1500 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --propagation=rslave /home/longtds/bin/dockerd-rootless.sh
             ├─1788 /proc/self/exe --state-dir=/run/user/1000/dockerd-rootless --net=vpnkit --mtu=1500 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --propagation=rslave /home/longtds/bin/dockerd-rootless.sh
             ├─1799 vpnkit --ethernet /run/user/1000/dockerd-rootless/vpnkit-ethernet.sock --mtu 1500 --host-ip 0.0.0.0
             ├─1812 dockerd
             └─1831 containerd --config /run/user/1000/docker/containerd/containerd.toml

Oct 14 06:06:57 ubuntu dockerd-rootless.sh[1831]: time="2024-10-14T06:06:57.607918697Z" level=info msg="containerd successfully booted in 0.115929s"
Oct 14 06:06:58 ubuntu dockerd-rootless.sh[1812]: time="2024-10-14T06:06:58.781696851Z" level=info msg="Loading containers: start."
Oct 14 06:06:58 ubuntu dockerd-rootless.sh[1812]: time="2024-10-14T06:06:58.785101611Z" level=info msg="skipping firewalld management for rootless mode"
Oct 14 06:07:01 ubuntu dockerd-rootless.sh[1812]: time="2024-10-14T06:07:01.003045680Z" level=info msg="Loading containers: done."
Oct 14 06:07:01 ubuntu dockerd-rootless.sh[1812]: time="2024-10-14T06:07:01.077865606Z" level=warning msg="WARNING: bridge-nf-call-iptables is disabled"
Oct 14 06:07:01 ubuntu dockerd-rootless.sh[1812]: time="2024-10-14T06:07:01.077930926Z" level=warning msg="WARNING: bridge-nf-call-ip6tables is disabled"
Oct 14 06:07:01 ubuntu dockerd-rootless.sh[1812]: time="2024-10-14T06:07:01.077960334Z" level=info msg="Docker daemon" commit=41ca978 containerd-snapshotter=false storage-driver=overlay2 version=27.3.1
Oct 14 06:07:01 ubuntu dockerd-rootless.sh[1812]: time="2024-10-14T06:07:01.078138011Z" level=info msg="Daemon has completed initialization"
Oct 14 06:07:01 ubuntu dockerd-rootless.sh[1812]: time="2024-10-14T06:07:01.202323068Z" level=info msg="API listen on /run/user/1000/docker.sock"
Oct 14 06:07:01 ubuntu systemd[1206]: Started Docker Application Container Engine (Rootless).
+ DOCKER_HOST=unix:///run/user/1000/docker.sock /home/longtds/bin/docker version
Client:
 Version:           27.3.1
 API version:       1.47
 Go version:        go1.22.7
 Git commit:        ce12230
 Built:             Fri Sep 20 11:39:44 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.7
  Git commit:       41ca978
  Built:            Fri Sep 20 11:41:02 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.22
  GitCommit:        7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc:
  Version:          1.1.14
  GitCommit:        v1.1.14-0-g2c9f560
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          2.3.1
  ApiVersion:       1.1.1
  NetworkDriver:    vpnkit
  PortDriver:       builtin
  StateDir:         /run/user/1000/dockerd-rootless
 vpnkit:
  Version:          7f0eff0dd99b576c5474de53b4454a157c642834
+ systemctl --user enable docker.service
Created symlink /home/longtds/.config/systemd/user/default.target.wants/docker.service → /home/longtds/.config/systemd/user/docker.service.
[INFO] Installed docker.service successfully.
[INFO] To control docker.service, run: `systemctl --user (start|stop|restart) docker.service`
[INFO] To run docker.service on system startup, run: `sudo loginctl enable-linger longtds`

[INFO] Creating CLI context "rootless"
Successfully created context "rootless"
[INFO] Using CLI context "rootless"
Current context is now "rootless"

[INFO] Make sure the following environment variable(s) are set (or add them to ~/.bashrc):
export PATH=/home/longtds/bin:$PATH

[INFO] Some applications may require the following environment variable too:
export DOCKER_HOST=unix:///run/user/1000/docker.sock

3. 查看docker info：

longtds@ubuntu:~$ docker info
Client:
 Version:    27.3.1
 Context:    rootless
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 27.3.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc version: v1.1.14-0-g2c9f560
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 6.8.0-45-generic
 Operating System: Ubuntu 22.04.5 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 7.71GiB
 Name: ubuntu
 ID: 3a35f3ef-8949-48bb-a866-ebc3c93fb1a5
 Docker Root Dir: /home/longtds/.local/share/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

 

通过kind创建k8s集群

1. kind二进制文件获取（Releases · kubernetes-sigs/kind (github.com)）

2. 创建kind集群配置文件

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  image: kindest/node:v1.31.0
  extraPortMappings:
  - containerPort: 30080
    hostPort: 30080
    listenAddress: "0.0.0.0"
- role: worker
  image: kindest/node:v1.31.0
- role: worker
  image: kindest/node:v1.31.0

3. 创建kind集群

longtds@ubuntu:~$ kind create cluster --config kind.yaml
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.31.0) 🖼
 ✓ Preparing nodes 📦 📦 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
 ✓ Joining worker nodes 🚜
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

4. 查看集群信息

longtds@ubuntu:~$ kubectl cluster-info
Kubernetes control plane is running at https://127.0.0.1:34571
CoreDNS is running at https://127.0.0.1:34571/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
longtds@ubuntu:~$ kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE   ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0               Healthy   ok
longtds@ubuntu:~$ kubectl get node
NAME                 STATUS   ROLES           AGE   VERSION
kind-control-plane   Ready    control-plane   86s   v1.31.0
kind-worker          Ready    <none>          66s   v1.31.0
kind-worker2         Ready    <none>          65s   v1.31.0
longtds@ubuntu:~$ kubectl get po -A
NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
kube-system          coredns-6f6b679f8f-dwt4z                     1/1     Running   0          84s
kube-system          coredns-6f6b679f8f-m5mgb                     1/1     Running   0          84s
kube-system          etcd-kind-control-plane                      1/1     Running   0          91s
kube-system          kindnet-2h7ml                                1/1     Running   0          75s
kube-system          kindnet-f9hrf                                1/1     Running   0          84s
kube-system          kindnet-xkgt8                                1/1     Running   0          74s
kube-system          kube-apiserver-kind-control-plane            1/1     Running   0          90s
kube-system          kube-controller-manager-kind-control-plane   1/1     Running   0          87s
kube-system          kube-proxy-74gd4                             1/1     Running   0          74s
kube-system          kube-proxy-ktqq5                             1/1     Running   0          84s
kube-system          kube-proxy-zrrxp                             1/1     Running   0          75s
kube-system          kube-scheduler-kind-control-plane            1/1     Running   0          87s
local-path-storage   local-path-provisioner-57c5987fd4-ht4mf      1/1     Running   0          80s

5. 创建测试服务

# 创建nginx测试pod
longtds@ubuntu:~$  kubectl run nginx --image=nginx --port=80
pod/nginx created
# 创建service
longtds@ubuntu:~$  kubectl expose pod nginx --port=80 --name=nginx --type=NodePort
service/nginx exposed
longtds@ubuntu:~$ kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP        20m
nginx        NodePort    10.96.101.224   <none>        80:31207/TCP   40s
# 修改nodeport为30080
longtds@ubuntu:~$ kubectl edit svc nginx
service/nginx edited
longtds@ubuntu:~$ kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP        21m
nginx        NodePort    10.96.101.224   <none>        80:30080/TCP   61s
longtds@ubuntu:~$ kubectl get pod
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          74s

6. 访问测试服务

longtds@ubuntu:~$ curl localhost:30080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
longtds@ubuntu:~$ curl 192.168.121.136:30080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

总结

1. docker-rootless服务通过systemctl --user管理，服务定义位置~/.config/systemd/user/docker.service

2. docker配置文件~/.config/docker/daemon.json（需要自己创建配置）

3. docker数据目录~/.local/share/docker 

4. docker-rootless模式可以直接兼容kind创建k8s集群

5. 支持基本的pod运行和服务访问（复杂的待测试）

网站公告

今日签到

点亮在社区的每一天
去签到

热门文章

最新发布