lxc & lxc-templates
1.项目信息
| 名称 | 描述 |
|---|---|
| 名称 | lxc |
| 版本 | 5.0.3 |
| 官方项目地址 | https://github.com/lxc/lxc |
| 名称 | 描述 |
|---|---|
| 名称 | lxc-templates |
| 版本 | 3.0.3 |
| 官方项目地址 | https://github.com/lxc/lxc-templates |
2.环境信息
| 名称 | 描述 |
|---|---|
| 系统 | loongnix alpine |
3.软件包安装
4. 环境准备
4.1 开启cgroup服务
需要启动cgroup服务，确保/proc/1/cgroup 文件不为空：
/home/lxc-alpine-package/lxc-templates-legacy # rc-service cgroups start
* Caching service dependencies ... [ ok ]
/home/lxc-alpine-package/lxc-templates-legacy # rc-update add cgroups
* service cgroups added to runlevel default
5.使用lxc-alpine模板创建容器与启动
5.1 容器创建
lxc-create -n test -t alpine //-n后面跟的是要创建的容器名称，-t后面跟的是模板文件
......
==> Container's rootfs and config have been created
Edit the config file /var/lib/lxc/test/config to check/enable networking setup.
The installed system is preconfigured for a loopback and single network
interface configured via DHCP.
To start the container, run "lxc-start -n test".
The root password is not set; to enter the container run "lxc-attach -n test".
此时容器test已经创建：
5.2 容器启动
此时直接启动容器会报错，如下：
/home/alpine/alpine/lxc-templates-legacy # lxc-start -n test --logfile=aaa
lxc-start: test: ../src/lxc/lxccontainer.c: wait_on_daemonized_start: 878 Received container state "ABORTING" instead of "RUNNING"
lxc-start: test: ../src/lxc/tools/lxc_start.c: main: 306 The container failed to start
lxc-start: test: ../src/lxc/tools/lxc_start.c: main: 309 To get more details, run the container in foreground mode
lxc-start: test: ../src/lxc/tools/lxc_start.c: main: 311 Additional information can be obtained by setting the --logfile and --logpriority options
/home/alpine/alpine/lxc-templates-legacy # cat aaa
lxc-start test 20240312083640.836 ERROR network - ../src/lxc/network.c:netdev_configure_server_veth:711 - No such file or directory - Failed to attach "veth6KwNcs" to bridge "lxclxcbr0", bridge interface doesn't exist
lxc-start test 20240312083640.924 ERROR network - ../src/lxc/network.c:lxc_create_network_priv:3427 - No such file or directory - Failed to create network device
lxc-start test 20240312083640.924 ERROR start - ../src/lxc/start.c:lxc_spawn:1840 - Failed to create the network
lxc-start test 20240312083640.924 ERROR lxccontainer - ../src/lxc/lxccontainer.c:wait_on_daemonized_start:878 - Received container state "ABORTING" instead of "RUNNING"
lxc-start test 20240312083640.924 ERROR lxc_start - ../src/lxc/tools/lxc_start.c:main:306 - The container failed to start
lxc-start test 20240312083640.924 ERROR lxc_start - ../src/lxc/tools/lxc_start.c:main:309 - To get more details, run the container in foreground mode
lxc-start test 20240312083640.924 ERROR lxc_start - ../src/lxc/tools/lxc_start.c:main:311 - Additional information can be obtained by setting the --logfile and --logpriority options
lxc-start test 20240312083640.924 ERROR start - ../src/lxc/start.c:__lxc_start:2107 - Failed to spawn container "test"
10 lxc.net.0.type = veth
11 lxc.net.0.link = lxcbr0
12 lxc.net.0.flags = up
13 lxc.net.0.hwaddr = 00:16:3e:82:db:3a
14 lxc.rootfs.path = dir:/var/lib/lxc/test/rootfs
若想要启动容器，可将文件/var/lib/lxc/test/config的第11行代码暂时注释:
10 lxc.net.0.type = veth
11 #lxc.net.0.link = lxcbr0
12 lxc.net.0.flags = up
13 lxc.net.0.hwaddr = 00:16:3e:82:db:3a
14 lxc.rootfs.path = dir:/var/lib/lxc/test/rootfs
/home/alpine/alpine/lxc-templates-legacy # lxc-start test
/home/alpine/alpine/lxc-templates-legacy # lxc-attach test
/ # pwd
/
/ # ls
bin dev etc home lib media mnt opt proc root run sbin srv sys tmp usr var
/ # cd /home
/home #
/home # ping baidu.com
^C
6.网络设置
网络设置有两种方法，一种是直接使用alpine自带的软件包dnsmasq，另外一种通过手动设置。
6.1 使用dnsmasq
6.1.1 安装dnsmasq
apk add dnsmasq
ln -s /etc/init.d/dnsmasq /etc/init.d/dnsmasq.lxcbr0
/etc/init.d/dnsmasq.lxcbr0 start
6.1.2 设置容器通过网桥lxcbr0通信
将容器配置文件/var/lib/test/config中网络设置为通过lxcbr0进行通信：
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:e0:d6:08
lxc.rootfs.path = dir:/var/lib/lxc/test/rootfs
6.1.3 重新启动容器
此时进入容器可以看到，容器已经获取到了ip地址，并添加了路由，也可以ping通百度# lxc-attach -n test
/ # ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1000
link/sit 0.0.0.0 brd 0.0.0.0
3: eth0@if71: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP qlen 1000
link/ether 00:16:3e:13:79:4c brd ff:ff:ff:ff:ff:ff
inet 10.0.3.7/24 brd 10.0.3.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:fe13:794c/64 scope link
valid_lft forever preferred_lft forever
/ # route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.3.1 0.0.0.0 UG 203 0 0 eth0
10.0.3.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
/ # ping baidu.com -c 3
PING baidu.com (110.242.68.66): 56 data bytes
64 bytes from 110.242.68.66: seq=0 ttl=53 time=20.565 ms
64 bytes from 110.242.68.66: seq=1 ttl=53 time=20.479 ms
64 bytes from 110.242.68.66: seq=2 ttl=53 time=20.454 ms
--- baidu.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 20.454/20.499/20.565 ms
/ # ping 10.130.0.184 -c 3
PING 10.130.0.184 (10.130.0.184): 56 data bytes
64 bytes from 10.130.0.184: seq=0 ttl=63 time=0.506 ms
64 bytes from 10.130.0.184: seq=1 ttl=63 time=0.469 ms
64 bytes from 10.130.0.184: seq=2 ttl=63 time=0.491 ms
--- 10.130.0.184 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.469/0.488/0.506 ms
6.2 手动设置
这里通过网桥和路由的方式来配置容器网络，将要创建的网络名称是lxcbr0。
6.2.1 开启ip转发功能
只有保证ip转发功能开启，才可以保证配置的路由可以生效
0表示未开启ip转发功能，或者也可以直接查看文件/proc/sys/net/ipv4/ip_forward，使用下命令开启ip转发功能sudo sysctl -w net.ipv4.ip_forward=1
以上只是临时设置，若要长久设置，可在文件/etc/sysctl.conf中写入：
vim /etc/sysctl.conf
net.ipv4.ip_forward=1
6.2.2 修改容器网络通过网桥lxcbr0进行通信
首先将容器配置文件/var/lib/test/config中网络设置为通过lxcbr0进行通信：
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:e0:d6:08
lxc.rootfs.path = dir:/var/lib/lxc/test/rootfs
6.2.3 在宿主机上创建网桥lxcbr0
/home/alpine # brctl addbr lxcbr0
/home/alpine # brctl show
bridge name bridge id STP enabled interfaces
lxcbr0 8000.000000000000 no
/home/alpine # ip add
11: lxcbr0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether fe:c8:e8:4c:79:15 brd ff:ff:ff:ff:ff:ff
/home/alpine # lxc-start test
/home/alpine # brctl show
bridge name bridge id STP enabled interfaces
lxcbr0 8000.fec8e84c7915 no vethc7wLS1
6.2.4 给网桥lxcbr0分配ip
为了实现网桥所在的主机和网桥所桥接的容器进行通信，需要给网桥和容器分配同一网段的ip地址。 主机上有两个物理网卡，eth2,eth3，网段分别为10.130.0.xxx和192.168.0.xxx，为了防止ip冲突，故给lxcbr0设置在192.168.200.xxx的网段
5: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:26:f8:90:10:4a brd ff:ff:ff:ff:ff:ff
inet 10.130.0.143/24 brd 10.130.0.255 scope global eth2
valid_lft forever preferred_lft forever
inet6 fe80::226:f8ff:fe90:104a/64 scope link
valid_lft forever preferred_lft forever
6: sit0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN qlen 1000
link/sit 0.0.0.0 brd 0.0.0.0
inet6 ::c0a8:314/96 scope global
valid_lft forever preferred_lft forever
inet6 ::a82:8f/96 scope global
valid_lft forever preferred_lft forever
inet6 ::7f00:1/96 scope host
valid_lft forever preferred_lft forever
7: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether 48:72:65:65:6e:01 brd ff:ff:ff:ff:ff:ff
inet 192.168.3.20/24 scope global eth3
valid_lft forever preferred_lft forever
inet6 fe80::4a72:65ff:fe65:6e01/64 scope link
valid_lft forever preferred_lft forever
给lxcbr0设置ip地址：
/home/alpine # ip addr add 192.168.200.11/24 dev lxcbr0
/home/alpine # ip add
......
11: lxcbr0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether fe:c8:e8:4c:79:15 brd ff:ff:ff:ff:ff:ff
inet 192.168.200.11/24 scope global lxcbr0
valid_lft forever preferred_lft forever
.......
/home/alpine/alpine/lxc-templates-legacy # route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.130.0.1 0.0.0.0 UG 205 0 0 eth2
10.130.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2
10.130.0.0 0.0.0.0 255.255.255.0 U 0 0 0 lxcbr0
192.168.3.0 0.0.0.0 255.255.255.0 U 0 0 0 eth3
启动lxcbr0:
/home/alpine # ifconfig lxcbr0 up
/home/alpine # ifconfig
......
lxcbr0 Link encap:Ethernet HWaddr FE:C8:E8:4C:79:15
inet addr:192.168.200.11 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::fcc8:e8ff:fe4c:7915/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:844 (844.0 B)
......
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.130.0.1 0.0.0.0 UG 205 0 0 eth2
10.130.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2
192.168.3.0 0.0.0.0 255.255.255.0 U 0 0 0 eth3
192.168.200.0 0.0.0.0 255.255.255.0 U 0 0 0 lxcbr0
6.2.5 给容器分配ip地址
这里同样使用固定分配ip的方式 查看容器内的虚拟网卡，如下，共有2个虚拟网卡。其中，lo是回环网卡通常用于测试数据包和软件配置是否正常。故这里使用eth0：
/ # ifconfig
eth0 Link encap:Ethernet HWaddr 00:16:3E:E0:D6:08
inet6 addr: fe80::216:3eff:fee0:d608/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:12 errors:0 dropped:0 overruns:0 frame:0
TX packets:25 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1016 (1016.0 B) TX bytes:5926 (5.7 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
给网卡eth0设置ip地址，网段需要与lxcbr0一致：
/ # ip addr add 192.168.200.12/24 dev eth0
/ # ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1000
link/sit 0.0.0.0 brd 0.0.0.0
3: eth0@if14: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP qlen 1000
link/ether 00:16:3e:e0:d6:08 brd ff:ff:ff:ff:ff:ff
inet 192.168.200.12/24 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:fee0:d608/64 scope link
valid_lft forever preferred_lft forever
/ # ping 192.168.200.11
PING 192.168.200.11 (192.168.200.11): 56 data bytes
64 bytes from 192.168.200.11: seq=0 ttl=64 time=0.213 ms
64 bytes from 192.168.200.11: seq=1 ttl=64 time=0.023 ms
^C
/ # ping 10.130.0.143
PING 10.130.0.143 (10.130.0.143): 56 data bytes
ping: sendto: Network unreachable
6.2.6 在容器内添加到主机的路由
下面的命令表示容器(192.168.200.12)发送给10.130.0.0网段的包，将通过192.168.200.11转发出去。
/ # route add -net 10.130.0.0/24 gw 192.168.200.11 dev eth0
/ # route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.130.0.0 192.168.200.11 0.0.0.0 UG 0 0 0 eth0
192.168.200.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
/ # ping 10.130.0.143
PING 10.130.0.143 (10.130.0.143): 56 data bytes
64 bytes from 10.130.0.143: seq=0 ttl=64 time=0.100 ms
64 bytes from 10.130.0.143: seq=1 ttl=64 time=0.032 ms
^X^C
--- 10.130.0.143 ping statistics ---
/ # ping 10.130.0.20
PING 10.130.0.20 (10.130.0.20): 56 data bytes
^C^C
--- 10.130.0.20 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
/ # ping baidu.com
^C
6.2.6 添加nat规则
nat规则只需要在启动容器的主机上设置 查看当前主机上的nat规则，为空：
/home/alpine # iptables -nvL --line-number //查看nat规则
/home/alpine # iptables -nvL -t nat --line-numbers
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 320 packets, 23842 bytes)
num pkts bytes target prot opt in out source destination
在主机上设置nat规则：
这条规则的作用是将容器192.168.200.12发送的包中的ip地址全部改成ip 10.130.0.143，故对于其他机器而言识别到的是主机10.130.0.143发送的包（实际是容器192.168.200.12发送的包）。 故此时主机所拥有的网络通信，容器192.168.200.12也全部拥有。从而实现与10.130.0.xxx网段其他ip之间的通信，以及ping通百度(前提是主机10.130.0.143可以ping通百度)。/home/alpine/alpine/lxc-templates-legacy # iptables -nvL -t nat --line-numbers
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 1 packets, 136 bytes)
num pkts bytes target prot opt in out source destination
1 0 0 SNAT 0 -- * * 192.168.200.12 0.0.0.0/0 to:10.130.0.143
/home/alpine/alpine/lxc-templates-legacy # iptables -t nat -D POSTROUTING 1 //删除nat规则，这里的1指向的是上面的num 1
6.2.7 设置FORWARD 链
容器与主机之间的通信是通过FORWARD链进行转发的，需要确认iptables中的FORWARD链是否打开，使用命令“iptables -nvL”查看FORWARD链是否打开：
alpine-3:~# iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) //ACCEPT表示打开
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
若FORWARD的状态是DROP，则表示关闭，若不添加规则，则意味着防火墙将拒绝所有经过它的转发数据包：
alpine-3:~# iptables -nvL
......
Chain FORWARD (policy DROP 109 packets, 9156 bytes) //DROP表示拒绝所有包，后面的数字表示一共拒绝了109个包，一共有9156个字节
pkts bytes target prot opt in out source destination
# iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy DROP 987 packets, 81732 bytes)
pkts bytes target prot opt in out source destination
8 672 ACCEPT 0 -- lxcbr0 * 0.0.0.0/0 0.0.0.0/0
8 672 ACCEPT 0 -- * lxcbr0 0.0.0.0/0 0.0.0.0/0
/ # ping baidu.com -c 3
PING baidu.com (110.242.68.66): 56 data bytes
64 bytes from 110.242.68.66: seq=0 ttl=53 time=20.565 ms
64 bytes from 110.242.68.66: seq=1 ttl=53 time=20.479 ms
64 bytes from 110.242.68.66: seq=2 ttl=53 time=20.454 ms
--- baidu.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 20.454/20.499/20.565 ms
/ # ping 10.130.0.184 -c 3
PING 10.130.0.184 (10.130.0.184): 56 data bytes
64 bytes from 10.130.0.184: seq=0 ttl=63 time=0.506 ms
64 bytes from 10.130.0.184: seq=1 ttl=63 time=0.469 ms
64 bytes from 10.130.0.184: seq=2 ttl=63 time=0.491 ms
--- 10.130.0.184 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.469/0.488/0.506 ms
7. 从源码构建lxc & lxc-templates
7.1 lxc
源码修改：参考https://github.com/Loongson-Cloud-Community/lxc/tree/loongarch64-lxc-5.0.2的补丁 依赖包安装：
apk add build-base docbook2x libapparmor-dev libcap-dev libcap-static libseccomp-dev linux-headers linux-pam-dev meson
apk add automake autoconf libmagic file libstdc++-dev musl-dev g++ fortify-headers build-base linux-headers binutils
abuild-meson -Db_lto= -Ddistrosysconfdir=/etc/default -Dpam-cgroup=true -Dtests=true -Dinit-script=[] . output //在构建时会在当前目录创建一个output目录，用来存储构建结果
meson compile -C output
meson install --no-rebuild -C output //若这里没有-C output则默认安装在/usr/local/bin或者/usr/bin目录下
7.2 lxc-templates
源码修改：参考https://github.com/Loongson-Cloud-Community/lxc-templates/commits/loongarch64-lxc-templates-3.0.4/的git log信息“Modify alpine to support loongarch64” 构建命令：