Setup from KubeEdge Installer

Keadm is used to install the cloud and edge components of KubeEdge. It is not responsible for installing K8s and runtime, so users must install a k8s master on cloud and runtime on edge first. Or use an existing cluster.

Please refer kubernetes-compatibility to get Kubernetes compatibility and determine what version of Kubernetes would be installed.

KubeEdge interacts with the standard K8s API, so the K8s cluster can be installed with any tools, such as:

Limitation

  • Currently support of keadm is available for Ubuntu and CentOS OS. RaspberryPi supports is in-progress.

Getting KubeEdge Installer

There are currently two ways to get keadm

  • Download from KubeEdge Release

    1. Go to KubeEdge Release page and download keadm-$VERSION-$OS-$ARCH.tar.gz..
    2. Untar it at desired location, by executing tar -xvzf keadm-$VERSION-$OS-$ARCH.tar.gz.
    3. kubeedge folder is created after execution the command.
  • Building from source

    1. Download the source code.

      git clone https://github.com/kubeedge/kubeedge.git $GOPATH/src/github.com/kubeedge/kubeedge
      cd $GOPATH/src/github.com/kubeedge/kubeedge
      make all WHAT=keadm
      

      or

      go get github.com/kubeedge/kubeedge/keadm/cmd/keadm
      
    2. If you used go get, the keadm binary is available in $GOPATH/bin/

      If you compiled from source, the keadm binary is in $GOPATH/src/github.com/kubeedge/kubeedge/_output/local/bin/

Setup Cloud Side (KubeEdge Master Node)

By default ports ‘10000’ and ‘10002’ in your cloudcore needs to be accessible for your edge nodes.

Note: ‘10002’ only needed since 1.3 release

keadm init will install cloudcore, generate the certs and install the CRDs. It also provides a flag by which a specific version can be set.

  1. Execute keadm init: keadm needs super user rights (or root rights) to run successfully.

    Command flags

    The optional flags with this command are mentioned below

    "keadm init" command install KubeEdge's master node (on the cloud) component.
    It checks if the Kubernetes Master are installed already,
    If not installed, please install the Kubernetes first.
    
    Usage:
      keadm init [flags]
    
    Examples:
    
    keadm init
    
    - This command will download and install the default version of KubeEdge cloud component
    
    keadm init --kubeedge-version=1.2.0  --kube-config=/root/.kube/config
    
      - kube-config is the absolute path of kubeconfig which used to secure connectivity between cloudcore and kube-apiserver
    
    Flags:
          --advertise-address string            Use this key to set SANs in certificate of cloudcore. eg: 10.10.102.78,10.10.102.79
      -h, --help                                help for init
          --kube-config string                  Use this key to set kube-config path, eg: $HOME/.kube/config (default "/root/.kube/config")
          --kubeedge-version string[="1.2.0"]   Use this key to download and use the required KubeEdge version (default "1.2.0")
          --master string                       Use this key to set K8s master address, eg: http://127.0.0.1:8080
    

IMPORTANT NOTE:

  1. At least one of kubeconfig or master must be configured correctly, so that it can be used to verify the version and other info of the k8s cluster.
  2. --advertise-address(only needed since 1.3 release) is the address exposed by the cloud side (will be added to the SANs of the CloudCore certificate), the default value is the local IP

Examples:

 keadm init --advertise-address=`THE-EXPOSED-IP`(only needed since 1.3 release)

Sample execution output:

Kubernetes version verification passed, KubeEdge installation will start...
...
KubeEdge cloudcore is running, For logs visit:  /var/log/kubeedge/cloudcore.log

(Only Needed in Pre 1.3 Release) Manually copy certs.tgz from cloud host to edge host(s)

Note: Since release 1.3, feature EdgeNode auto TLS Bootstrapping has been added and there is no need to manually copy certificate.

Now users still need to copy the certs to the edge nodes. In the future, it will support the use of tokens for authentication.

On edge host

mkdir -p /etc/kubeedge

On cloud host

cd /etc/kubeedge/
scp -r certs.tgz username@ipEdgevm:/etc/kubeedge

On edge host untar the certs.tgz file

cd /etc/kubeedge
tar -xvzf certs.tgz

Setup Edge Side (KubeEdge Worker Node)

Get Token From Cloud Side

Run keadm gettoken in cloud side will return the token, which will be used when joining edge nodes.

# from cloud side
keadm gettoken
27a37ef16159f7d3be8fae95d588b79b3adaaf92727b72659eb89758c66ffda2.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTAyMTYwNzd9.JBj8LLYWXwbbvHKffJBpPd5CyxqapRQYDIXtFZErgYE

Join Edge Node

keadm join will install edgecore and mqtt. It also provides a flag by which a specific version can be set.

Execute keadm join <flags>

Command flags

The optional flags with this command are shown in below shell

"keadm join" command bootstraps KubeEdge's worker node (at the edge) component.
It will also connect with cloud component to receive
further instructions and forward telemetry data from
devices to cloud

Usage:
  keadm join [flags]

Examples:

keadm join --cloudcore-ipport=<ip:port address> --edgenode-name=<unique string as edge identifier>

  - For this command --cloudcore-ipport flag is a required option
  - This command will download and install the default version of pre-requisites and KubeEdge

keadm join --cloudcore-ipport=10.20.30.40:10000 --edgenode-name=testing123 --kubeedge-version=1.2.0

Flags:
      --certPath string                     The certPath used by edgecore, the default value is /etc/kubeedge/certs (default "/etc/kubeedge/certs")
  -s, --certport string                     The port where to apply for the edge certificate
  -e, --cloudcore-ipport string             IP:Port address of KubeEdge CloudCore
  -i, --edgenode-name string                KubeEdge Node unique identification string, If flag not used then the command will generate a unique id on its own
  -h, --help                                help for join
      --interfacename string                KubeEdge Node interface name string, the default value is eth0
      --kubeedge-version string[="1.2.0"]   Use this key to download and use the required KubeEdge version (default "1.2.0")
  -r, --runtimetype string                  Container runtime type
  -t, --token string                        Used for edge to apply for the certificate

IMPORTANT NOTE:

  1. For this command --cloudcore-ipport flag is a mandatory flag.
  2. If you want to apply certificate for edge node automatically, --token is needed.
  3. The kubeEdge version used in cloud and edge side should be same.

Examples:

 keadm join --cloudcore-ipport=192.168.20.50:10000 --token=27a37ef16159f7d3be8fae95d588b79b3adaaf92727b72659eb89758c66ffda2.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTAyMTYwNzd9.JBj8LLYWXwbbvHKffJBpPd5CyxqapRQYDIXtFZErgYE

Sample execution output:

Host has mosquit+ already installed and running. Hence skipping the installation steps !!!
...
KubeEdge edgecore is running, For logs visit:  /var/log/kubeedge/edgecore.log

Setup Edge Side with VM provisioning (KubeEdge Worker Node)

Requirement, restrictions

  1. Current phase supports only one runtime, either container or VM runtime in one cluster.
  2. Make sure no libvirt is running on the worker nodes.

Steps

  1. Install CNI plugin:

    • Download CNI plugin release and extract it:
    $ wget https://github.com/containernetworking/plugins/releases/download/v0.8.2/cni-plugins-linux-amd64-v0.8.2.tgz
    
    # Extract the tarball
    $ mkdir cni
    $ tar -zxvf v0.2.0.tar.gz -C cni
    
    $ mkdir -p /opt/cni/bin
    $ cp ./cni/* /opt/cni/bin/
    
    • Configure cni plugin
    $ mkdir -p /etc/cni/net.d/
    
    $ cat >/etc/cni/net.d/bridge.conf <<EOF
    {
      "cniVersion": "0.3.1",
      "name": "containerd-net",
      "type": "bridge",
      "bridge": "cni0",
      "isGateway": true,
      "ipMasq": true,
      "ipam": {
        "type": "host-local",
        "subnet": "10.88.0.0/16",
        "routes": [
          { "dst": "0.0.0.0/0" }
        ]
      }
    }
    EOF
    
  2. Setup VM runtime: Use script hack/setup-vmruntime.sh to set up VM runtime. It makes use of Arktos Runtime release to start three containers:

     vmruntime_vms
     vmruntime_libvirt
     vmruntime_virtlet
    
  3. Start edgecore service and join the cluster: The step is similare to provision containers with specify remote-runtime-endpoint.

Examples:

 keadm join --cloudcore-ipport=192.168.20.50:10000 -r remote --remote-runtime-endpoint=unix:///run/virtlet.sock
  1. Test create a VM workload: (optional) On the master node, create a sample yaml file vm.yaml as:
apiVersion: v1
kind: Pod
metadata:
 name: testvm
spec:
 containers:
 - name: testvm
   image: download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86_64-disk.img
   imagePullPolicy: Always
   resources:
     limits:
       cpu: "3"
       memory: "200Mi"
     requests:
       cpu: "3"
       memory: "200Mi"

Then use kubectl create -f vm.yaml to create VM pod on the edge node. You should see the workload on master:

On master:

# kubectl get pods -o wide
NAME     READY   STATUS    RESTARTS   AGE   IP           NODE          NOMINATED NODE   READINESS GATES
testvm   1/1     Running   0          38s   10.88.0.18   testnodevm3   <none>           <none>

On the edge worker node: either ssh into the VM instance or virsh list can verify the VM is created and running:

Id    Name                           State
----------------------------------------------------
1     virtlet-10628888-2584-testvm   running

Reset KubeEdge Master and Worker nodes

keadm reset will stop KubeEdge components. It doesn’t uninstall/remove any of the pre-requisites.

Execute keadm reset

Command flags

keadm reset --help

keadm reset command can be executed in both cloud and edge node
In cloud node it shuts down the cloud processes of KubeEdge
In edge node it shuts down the edge processes of KubeEdge

Usage:
  keadm reset [flags]

Examples:

For cloud node:
keadm reset

For edge node:
keadm reset

Flags:
  -h, --help   help for reset

Errata

  1. Error in CloudCore

    If you are getting the below error in Cloudcore.log

    E1231 04:37:27.397431   19607 reflector.go:125] github.com/kubeedge/kubeedge/cloud/pkg/devicecontroller/manager/device.go:40: Failed to list *v1alpha1.Device: the server could not find the requested resource (get devices.devices.kubeedge.io)
    E1231 04:37:27.398273   19607 reflector.go:125] github.com/kubeedge/kubeedge/cloud/pkg/devicecontroller/manager/devicemodel.go:40: Failed to list *v1alpha1.DeviceModel: the server could not find the requested resource (get devicemodels.devices.kubeedge.io)
    

    browse to the

    cd $GOPATH/src/github.com/kubeedge/kubeedge/build/crds/devices
    

    and apply the below

      kubectl create -f devices_v1alpha1_devicemodel.yaml
      kubectl create -f devices_v1alpha1_device.yaml
    

    or

     kubectl create -f https://raw.githubusercontent.com/kubeedge/kubeedge/<kubeEdge Version>/build/crds/devices/devices_v1alpha1_device.yaml
     kubectl create -f https://raw.githubusercontent.com/kubeedge/kubeedge/<kubeEdge Version>/build/crds/devices/devices_v1alpha1_devicemodel.yaml
    

    Also, create ClusterObjectSync and ObjectSync CRDs which are used in reliable message delivery.

     cd $GOPATH/src/github.com/kubeedge/kubeedge/build/crds/reliablesyncs
     kubectl create -f cluster_objectsync_v1alpha1.yaml
     kubectl create -f objectsync_v1alpha1.yaml