Thursday, February 21, 2019

Linux container LXC,Docker的史祖

Dockers并不是第一个Linux container solution。

Refer to this tutorial working-linux-containers-lxc and links in the web page:
Dockers V.S. LXC


As Docker vs LXC mentioned: Docker, previously called dotCloud, was started as a side project and only open-sourced in 2013. It is really an extension of LXC’s capabilities. This it achieves using a high-level API that provides a lightweight virtualization solution to run processes in isolation. Docker is developed in the Go language and utilizes LXC, cgroups, and the Linux kernel itself. Since it’s based on LXC, a Docker container does not include a separate operating system; instead it relies on the operating system’s own functionality as provided by the underlying infrastructure. So Docker acts as a portable container engine, packaging the application and all its dependencies in a virtual container that can run on any Linux server.


Commands lxc-attach/lxc-console for enter a container. Others(-t = --template, -n = --name, -d = --daemon, u1 is container name):
[sudo] lxc-create -t download -n u1 -- -d ubuntu -r xenial -a amd64 => no need of sudo for creating unprivileged container
sudo lxc-ls --fancy
sudo lxc-start --name u1 --daemon
sudo lxc-info --name u1
sudo lxc-stop --name u1
sudo lxc-destroy --name u1

Unprivileged containers allow users to create and administer containers without having any root privilege. The feature underpinning this is called user namespaces. User namespaces are hierarchical, with privileged tasks in a parent namespace being able to map its ids into child namespaces. By default every task on the host runs in the initial user namespace, where the full range of ids is mapped onto the full range. This can be seen by looking at /proc/self/uid_map and /proc/self/gid_map, which both will show "0 0 4294967295" when read from the initial user namespace. As of Ubuntu 14.04, when new users are created they are by default offered a range of userids. The list of assigned ids can be seen in the files /etc/subuid and /etc/subgid See their respective manpages for more information. Subuids and subgids are by convention started at id 100000 to avoid conflicting with system users.

For Unprivileged container, if not using sudo, may need refer to this and this to grant permission for accessing .local folder, like this:
chmod a+rx ~/.local ~/.local/share or: cd $HOME && setfacl -m u:100000:x .local .local/share, "100000" here is the uid you get from /etc/subuid. Basically need to add root-of-the-unpriv-container (usually 100000 for the first normal user) "x" access to your home and .local directory. With the 2nd way, when run getfacl against .local, will see a new line: “mask::--x”.

After installed openssh-server on the container, will need to run useradd and passwd to create user for ssh access.

Global configuration


The following configuration files are consulted by LXC. For privileged use, they are found under /etc/lxc, while for unprivileged use they are under ~/.config/lxc.
lxc.conf may optionally specify alternate values for several lxc settings, including the lxcpath, the default configuration, cgroups to use, a cgroup creation pattern, and storage backend settings for lvm and zfs.
default.conf specifies configuration which every newly created container should contain. This usually contains at least a network section, and, for unprivileged users, an id mapping section
lxc-usernet.conf specifies how unprivileged users may connect their containers to the host-owned network.
lxc.conf and default.conf are both under /etc/lxc and $HOME/.config/lxc, while lxc-usernet.conf is only host-wide.

By default, containers are located under /var/lib/lxc for the root user, and $HOME/.local/share/lxc otherwise. The location can be specified for all lxc commands using the "-P|--lxcpath" argument. => unprivileged container will be stored at /var/lib/lxc if run lxc-create with sudo.

Practice

lxc-clone is deprecated in favor of lxc-copy. Refer to wikipedia for LVM. For running multiple same container, to save space, creating the 2nd container C2 from origin container C1, use snapshot option: lxc-stop -n C1 && lxc-copy -n C1 -s -N C2
Almost no space cost for the 2nd container.
Both container can see the host’s kernel log by running dmesg, and can access sysfs of host.

Saturday, February 16, 2019

未来,数字化的世界

2013年初写了创意无限-创新无止境,现在是2019年,8年前康宁的A Day Made of Glass和7年前的A Day Made of Glass 2还在指引未来的数字化生活。片中的场景到底只是幻想,还是不远的未来?康宁在2012年就给出了一些解释:A Day Made of Glass 2: Unpacked. The Story Behind Corning's Vision. (2012)


Top 10 future transportation that will blow your mind

Tuesday, February 12, 2019

学习舵轮(kubernetes)和埠(docker) - Kubernetes装载docker image

有了dockers image,用Kubernetes来自动装载管理很简单。

Creating docker image and deploying Kubernetes pod/service

Refer to kubernetes containers images/. Note, with snap, Docker stores keys for private registries in $HOME/snap/docker/.docker/config.json but not $HOME/.dockercfg or $HOME/.docker/config.json. But for non private repository, no need of the credentials and k8s uses docker under the hood. 我在Docker Hub 创建了ifun public repository. Then run:
 docker tag ifun:v0.2 quyq/ifun:v0.2
and
 docker push quyq/ifun:v0.2
‘push'命令把local build的image 上传到Docker Hub, which is 83M in size (my local shows the image is 206M?). To start the image as kubernetes pod: kubectl run ifun --image=quyq/ifun:v0.2 --port=3000 (If get: ErrImagePull=>trying and failing to pull image, ImagePullBackOff=>image can't be pulled). To create service run: kubectl expose deployment/ifun --type="NodePort" --port 3000 => get: service/ifun exposed. Then run kubectl get services will get like:
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)     AGE
ifun         NodePort    10.106.182.186  <none>      3000:32574/TCP    1m
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP          16d

Use this url to access: $(minikube ip):32574, where minikube ip will return 192.168.99.100, which is the VM’s IP. Run minikube dashboard for detail.

To use a yaml file for the container with host volume, create the yaml file like this(try to avoid tab in yaml):

apiVersion: v1
kind: Pod
metadata:
 name: node-mongo
 labels:
   purpose: test-node_mongo
spec:
 containers:
 - name: ifun
   image: quyq/ifun:v0.2
   ports:
   - containerPort: 3000
   volumeMounts:
   - mountPath: /data
     name: db-volume
 volumes:
 - name: db-volume
   hostPath:
     path: c:/Temp # directory location on host(the node/VM, not host PC running the VM)
     type: Directory # this field is optional
Run: kubectl create -f my.yaml => run kubectl get pods will show: node-mongo   0/1     ContainerCreating   0  … 
Run kubectl logs node-mongo get:
Error from server (BadRequest): container "ifun" in pod "node-mongo" is waiting to start: ContainerCreating. kubectl describe pods gives detail of problem: SetUp failed for volume "db-volume" : hostPath type check failed: c:/Temp is not a directory. 我是在Windows PC上运行上述命令,如上述comment所述,mount的volume必须是那个运行docker的Linux VM而不是host Windows PC. Mount failure 导致pod failed to start. Change c:/Temp to /tmp and re-run, it can create the pod now. 

But run kubectl get deployments show nothing? Refer to this SO: set --restart=Always. When use yaml, have to create a deploy yaml like:
apiVersion: extensions/v1beta1 #why cannot be just v1?
kind: Deployment
metadata:
  name: node-mongo
spec:
  replicas: 2    #optional, default is one pod
  template:
     Pod’ metadata and spec here ...
This will create two pods and db would be stored in the VM’s /tmp folder. Since /tmp might get cleaned when restart VM, so should create a dedicated folder, such as /mongo_db on the VM, and change the volumes/hostPath/path to /mongo_db.现在可以运行Kubernetes supported NodeJS + MongoDB,database文件会存在虚拟机的/tmp目录下

Updating image

Kubernetes Docker VM contains docker, and it can access shared folder of host PC, such as c:\Users folder. So for running Kubernetes on Windows, after updating docker file, can push the update from the VM with exist tag as this: docker login; docker push quyq/ifun:v0.2; need a way to force image pull
To create the image with a new tag:

 docker image build -t ifun:v0.3 . (same as docker build --tag=ifun:v0.3 . ? image is optional);
 docker tag ifun:v0.3 quyq/ifun:v0.3; 
 docker push quyq/ifun:v0.3;
 then run kubectl edit deployment/node-mongo, and wait the deployment is done.


Monday, February 11, 2019

Running Linux on Windows

讲了不少虚拟机,container,模拟器等,大部分都是运行在Linux上。我想微软认识到了Linux支持的重要性,特别是在开发应用中,比如Android开发,Linux的开发者用户群很大,所以一年多前微软就通过Windows 10的Insider/Creators program (WSL, Windows Subsystem for Linux )引人Ubuntu的支持,可以在Windows console下运行Ubuntu命令:如 bash, ssh, git, apt等等。现在这一feature已经正式出现在Windows App Store: https://www.microsoft.com/en-us/p/ubuntu/9nblggh4msv6,甚至可以在旧版Windows 10 (如1709)上运行。从微软网站摘录的:
Note that Windows 10 S does not support running this app. Before installing Ubuntu on Windows or before the first run please open the Control Panel, visit Programs and Features' submenu Turn Windows features on or off and select Windows Subsystem for Linux. Click OK, reboot, and then your system is ready to run this app. Instead of the above steps You can set up your system by executing the following command in the Administrator PowerShell prompt: Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux When your system is ready simply start "ubuntu" in cmd.exe or launch Ubuntu from the start menu.

Linux上的VM



Linux有很好的VM support,如Kernel Virtual Machine (KVM), 由kernel module提供支持。QEMU (https://www.qemu.org, 一个开源的machine emulator and virtualizer)包含KVM的userspace component。如果你做过Android开发,用过Android的虚拟机AVD,那就是用QEMU实现的。KVM和QEMU是基于Linux内核的,所有有很小的footprint,效率却很高。When used as a virtualizer, QEMU achieves near native performance by executing the guest code directly on the host CPU. QEMU supports virtualization when executing under the Xen hypervisor or using the KVM kernel module in Linux. When using KVM, QEMU can virtualize x86, server and embedded PowerPC, 64-bit POWER, S390, 32-bit and 64-bit ARM, and MIPS guests. 

KVM vs VirtualBox

Both are opensource. According to this: KVM is better integrated with Linux, it's smaller and faster, and while you can use it with other guests besides Linux. Vbox supports suspends, and may be better for other guest OS such as BSD and Solaris.

Running Windows program on Linux and MacOS: Wine

因为很多原因我工作之外更喜欢使用Linux/Ubuntu操作系统。我在家的常用计算机是双操作系统,Windows10 + Ubuntu。Boot来boot去的到也很快,但还是不很痛快。比如,Linux可以access Windows的硬盘分区,可是Windows不支持Linux的ext4分区。又如Google Drive App只有Windows和Mac版,没有Linux版(很奇怪)。有时我会玩老游戏如微软的帝国时代和红警,感谢开源Wine (Wine Is Not Emulator) enables Linux, Mac, FreeBSD, and Solaris users to run Windows applications without a copy of Microsoft Windows. Wine is free software under constant development. Other platforms may benefit as well. 
https://wiki.winehq.org/Ubuntu 提供了源程序和binary,很容易安装。wine FAQ 提及At present there are some significant bugs that prevent many 32 bit applications from working in a 64 bit wineprefix。很多Windows程序都是32位的,所以需要32位Linux support:

If your system is 64 bit, enable 32 bit architecture (if you haven't already):
sudo dpkg --add-architecture i386
为了用apt安装WINE,需要把WINE加到Ubuntu的软件list里,download and add the repository key:

sudo apt-key add winehq.key

Ubuntu 18.04: sudo apt-add-repository 'deb https://dl.winehq.org/wine-builds/ubuntu/ bionic main'

sudo apt update && sudo apt install --install-recommends winehq-stable


安装完后,在home folder的.wine目录下,可以找到drive_c,就是mapped的C drive。运行的程序并不必须copy到这个目录下。在这个.wine目录下运行winecfg会自动安装.net等package。之后可以运行wine notepad来确认wine设置成功。

因为我的游戏是中文汉化版,为了中文支持,首先需要在Host OS上加中午支持。
User guide: For Chinese font support, refer to test language: need to install language package for linux first, such as:
sudo apt-get install language-pack-zh-hans
then run like: LANG=zh_CN.UTF-8 wine empires2.exe to start Empire 2.
注意怎样在Ubuntu上添加拼音输入支持,先要安装如上的pack,然后在setting/Language and Region input source里加入拼音输入,注意,一定要加pinyin 而不是Chinese,不然,就没法用拼音。

帝国时代游戏里有选项可以改变游戏分辨率。取决于winecfg的Graphics设置,可能会有不同的效果和问题。如右图所示, Graphics有4个checkbox,最重要的是第3项和第4项:
Allow the window manager to control the windows
Emulate a virtual desktop 和 Desktop size
当这两项都没有选取时,游戏显示是正常的,在游戏中改变分辨率,游戏的显示还是正常,但是游戏的快捷键可能不起作用。如果只选第3项,键盘工作正常,但切换游戏分辨率后,一部分屏幕可能不能正常刷新。我相信应该有办法tweak wine解决这个问题,不过最简单的办法就是把3和4都选上,显示是virtual的,而不是全屏的,看起来画面更细致。
 




WINE也可以安装到Mac OS上。

On MacOS, refer to https://wiki.winehq.org/MacOS
如上述链接所说的,需要先确保Xquartz已安装。Xquartz也是一个开源软件: https://www.xquartz.org/, an open-source effort to develop a version of the X.Org X Window System that runs on OS X. 在MacOS上32bit wine是基本安装, 64bit 是可选项。设置运行和Linux /Ubuntu基本一样。
总的来说,WINE的solution与VM和container完全不同。 VM是在指令一层simulate,container则必须运行在同种OS,而WINE是把Windows API calls 转换成POSIX API calls on-the-fly,各有千秋。 

Update in 2020: 参见winetricks: A useful tool for using common workarounds to current deficiencies in Wine,很方便用来安装runtime support。