Setting Up CSC Virtual Machines for WC Production Platform V1

Version 1.6

This document is combination of great WIMMA Lab forces for several years :)

Mysticons 2021 (Sini Karvonen, Timo Lehosvuo, Heikki Pekkarinen, Matti Saarelma, Samuli Ylönen)
Mysticons 2022 (Toni Peltola, Tero Räsänen, Kristian Kantola, Eero Järvinen)
Mysticons 2023 (Aarne Salmi, Petteri Stjerna, Karri Päivärinta )
NarsuMan 2024

STEP 1: Ubuntu running on CSC-service. SSH-key for SSH connection.

STEP 2: SSH connection to VM. Run script to install Microk8s Kubernetes with needed add-ons and fetch needed Certificate and Tokens.

STEP 3: Integrate GitLab project with Microk8s Kubernetes

STEP 4: Get access to Dashboard API

4.1 Step 1

At this first step we define the Instance which works over Cloud based server

4.1.1 CSC

In this project we used the cloud environment provided by CSC.

Login to the website https://my.csc.fi/ with your HAKA credentials.

Fill your own details for CSC.

Under "My Projects" you need to create a new project.

After creating, we need to apply resources for it, -> scroll down and you find "Apply for resources"

Add description and keywords, check your field of science and sub science areas.

After approved resource application you need to add service for it. Pick cPouta from list on right and tick "I have read and accepted the cPouta Terms and Conditions". -> Add service

You also need to fill more your details in "My Profile" to get access to your projects in CSC if you didn't give them right in the beginning.

Way to go, now you have a project in CSC.

4.1.2 cPouta

To access to your cPouta project login to the site https://pouta.csc.fi.

Again, use your HAKA credentials.

If your new project does immediately appear in cPouta, just wait for a while its creation takes time. When it's ready, your new project can be found at the top in the dropdown menu.

In this overview view you can see your resources for this project.

4.1.3 Set up SSH keys

Helpful step at this point is to create key pairs for your upcoming virtual machine. Go to the Key pairs section. Please do not use tool Create key pair what pouta.csc offers ( In our cases It does not seem work properly )

We recommend you use PuTTYGen for generating the key pair

https://www.puttygen.com/download.php?val=49

( Other option is create keys in command prompt like shown in chapter: For OSX users below ).

In PuTTY Key generator go in Key -> Parameters for saving key files

Change PPK file version from 3 to 2.

Generate a public/private key pair:

Copy Public key into your clipboard
(You may add a passphrase if you want)
And save the private key for upcoming SSH connections

Next push import public key in CSC key pairs section.

Paste public key from your clipboard to public key zone and give recognizable name for your key pair.

4.1.4 VM

To create VM in your cPouta project go in Compute/Images.

We prefer to use Ubuntu 20.04 -> Launch.

Kuva, joka sisältää kohteen pöytä Kuvaus luotu
automaattisesti

In next sections you need to:

Give a name for your instance.

Select the flavor. It depends on what your upcoming system needs are.

(You may add 2 instances with equal power over cPouta with standard.large flavor)

Notice to run Kubernetes on CSC virtual machine minimum requirements are:

4 Core
8gb RAM

Check key pair section that the key that you created is righty allocated.

Next launch instance.

Created instance can be found from Compute/Instances.

Next give floating IP address for this new instance.

Because there is no floating IP address to pick, we need to create one by clicking the plus sing.

Press allocate IP.

Then select allocated for Instance and associate.

Now you see that your Instance has Floating IP.

4.1.5 Firewall rules for projection

To access via SSH to your Instance you need to setup Security group rule in your cPouta.

Go to the: Security Groups and there manage rules.

There you may find all default rules. Add new rule.

From the bottom of the list, you should find an SSH rule. Choose it and click add.

Now there is new Ingress (inbound) rule for port 22 and your Instance is ready for SSH connection.

4.2 Step 2

At this second step we:

take SSH connection to our VM.
Install Kubernetes
Install and enable Kubernetes tools by script
Get certificate and tokens for next steps

4.2.1 SSH connection with PuTTY

For SSH connection we prefer to use PuTTY.

From PuTTY choose SSH/Auth and choose the key that you created earlier

Go back to session section and give an IP address (Floating IP from Cpouta Instance)
Give the session a name for upcoming connections
Save these setting for that session
Open connection

To login use admin: ubuntu

{

4.2.2 Install Microk8s, two options

To Install Kubernetes platform, you have two options:

You may use ansible-playbook (follow 4.2.3 ) : https://gitlab.labranet.jamk.fi/wimmalab2021/mysticons/scripts/-/blob/master/microk8s_install_ansible_v003.yml
Or use script (follow 4.2.5 ) :

https://gitlab.labranet.jamk.fi/wimmalab2021/mysticons/scripts/-/blob/master/microk8s_install_bash_v003.sh

4.2.3 Ansible

To run ansible run:

sudo apt update
sudo apt install ansible

do you want to continue y[es]{.ul}

Go root

sudo su ( This is important, It does not work without root)

Create ansible file ( or alternatively SFTP copy ansible-playbook file into VM).

Source: https://gitlab.labranet.jamk.fi/wimmalab2021/mysticons/scripts/-/blob/master/microk8s_install_ansible_v003.yml

nano ansible.yml
copy paste script to ansible.yml

Run ansible

ansible-playbook ansible.yml

Sit tight some steps took its time

Copy CA certificate to your txt file

Be precise when copying, start from lines ----- and end the copy after the lines -----

Copy the token for GitLab certificate to your txt file

Be precise when copying, avoid empty spaces and " mark from the end.

Copy the second token for dashboard certificate to your txt file

Be precise when copying, avoid empty spaces and " mark from the end.

Copy the GitLab Kubernetes API URL to your txt file too

Now you are ready for step 3 GitLab sync.

4.2.4 I want to get certificate and tokens

If you face problems copying certificates or tokens or want to get them again please run the script below:

Source: https://gitlab.labranet.jamk.fi/wimmalab2021/mysticons/scripts/-/blob/master/microk8s_certs_tokens_bash_v003.sh

That gives you desired information again without reinstalling Microk8s.

4.2.5 Script

Same as above but in bash script mode.

To run script:

Create script file

Source: https://gitlab.labranet.jamk.fi/wimmalab2021/mysticons/scripts/-/blob/master/microk8s_install_bash_v003.sh

nano script.sh
copy paste script to script.sh

bash script.sh (Use bash, the script does not work with "./script.sh")

Copy CA certificate to your txt file

Be precise when copying, start from lines ----- and end the copy after the lines -----

Copy the token for GitLab certificate to your txt file

Copy the second token for dashboard certificate to your txt file

Copy the GitLab Kubernetes API URL to your txt file too

Kuva, joka sisältää kohteen teksti, sisä, näyttökuva Kuvaus luotu
automaattisesti

Now you are ready for step 3 GitLab sync.

4.2.6 I want to get certificate and tokens

If you face problems copying certificates or tokens or want to get them again, please run the script below:

Source: https://gitlab.labranet.jamk.fi/wimmalab2021/mysticons/scripts/-/blob/master/microk8s_certs_tokens_bash_v003.sh

That gives you desired information again without reinstalling Microk8s.

4.3 Step 3 Gitlab Sync

In this step we are going to connect a Microk8s platform with existing Gitlab repository.

If you don't already have a repository in GitLab which you want to connect with Kubernetes you need to create one:

Give the project a name
Choose visibility (for WIMMA Lab projects Public is recommended)
Add a tick in the "Initialize repository with a README" box
Create the project

4.3.1 Open port for GitLab API

To manage Kubernetes working with GitLab you must open port from CSC in same way you did with SSH port earlier.

In CSC project go Security Groups/Manage Rules.

Add Rule.

Port: 8443

Now you have new rule created and GitLab can access to cPouta.

Kuva, joka sisältää kohteen pöytä Kuvaus luotu
automaattisesti

4.3.2 Kubernetes in repository

To connect Kubernetes with your repository, go to Operations/Kubernetes.

Integrate with a cluster certificate.

Connect existing cluster

Give name to Kubernetes cluster
Environment scope

* When you are adding more than one Kubernetes cluster to your project, you need to differentiate them with an environment scope.

API URL

Paste your API URL here, check that port is 8443

CA Certificate

Paste your CA Certificate here

Service token

Paste Service token here

Add Kubernetes cluster

4.3.3 Kubernetes applications

After successful Kubernetes connection install following applications.

Go in applications.

Install Prometheus and GitLab Runner.

Kubernetes cluster is now connected.

4.4 Step 4

"Kubernetes Dashboard is a web-based Kubernetes user Interface". (Kubernetes IO)

You can use the Kubernetes dashboard to deploy package applications to the Kubernetes cluster, troubleshoot the container application, and manage cluster resources. Use the dashboard to get an overview of the applications running in the cluster and to create or edit individual Kubernetes resources.

4.4.1 Access to dashboard

To access dashboard, we need to create new rule in cPouta with following steps.

Go in your project in cPouta https://pouta.csc.fi/

Add rule

Kuva, joka sisältää kohteen teksti, näyttökuva, sisä Kuvaus luotu
automaattisesti

Use Custom TCP RULE

Add port: 10443

Now you have new port rule to access dashboard.

4.4.2 Access to dashboard

Go back in PUTTY.

Run command sudo screen microk8s dashboard-proxy

after that push ctrl +a +d that you may use dashboard-proxy in background.

In following steps do not copy token what dashboard-proxy gives to you.

Next open your browser.

Use your API URL and port 10443 to access Kubernetes Dashboard

example: https://vmxxxx.kaj.pouta.csc.fi:10443

Kuva, joka sisältää kohteen teksti, näyttökuva, näyttö Kuvaus luotu
automaattisesti

Now you have access to Dashboard API.

Solution in cases where your dashboard won't work properly:

check that you have dashboard-proxy running
Your token for dashboard copies correctly

5 General commands to make your working easier

To run Kubernetes commands easier and shorter in command-line interface on PuTTY

Create the following aliases:

alias kubectl=\'sudo microk8s kubectl\'

alias microk8s=\'sudo microk8s\'

6 Setting DevSecOps Pipeline

To set up a pipeline you need to have.gitlab-ci.yml file in your repository root. The file describes all the stages and jobs that the pipeline runs:

In the example below our pipeline scans our GitLab repository (secret detection), builds the container, scans it, deploys it content and finally scans the published website. These jobs are spread across five stages: secret, build, test, deploy and DAST:

In order to implement the security scans in your pipeline you need to include the templates of the scans you want in your .gitlab-ci.yml-file. All the scans are run at the test stage unless configured otherwise and some of them need variables to work properly. It is important to think about the order of the stages and the scans you are running. Running a container scan before building the container or running a DAST scan before deployment makes no sense. You can see your pipelines by going to CI/CD -> Pipelines. After the pipeline has finishes you can see the result of the by clicking on the pipeline:

Results of the security scans can be found under the security tab:

6.1 gitlab-ci.yml

Gitlab-ci.yml file is used to configure your GitLab pipelines and it must be located under your repository root. In that file you define what scripts you want to run and what other configuration files and templates you want to include.

6.2 Stages

At the start of Gitlab-ci.yml file specified stages what needed in project. Stages determine to run listed contain in defined order.

6.3 Secret

Secret scan. It scans the content of the repository to find API keys and other information that should not be there.

6.4 Test

All other scans except secret.

6.5 SAST

In this static analysis, vulnerabilities are searched in the application binary or code.

6.6 Gemnasium-dependency

Automatically find vulnerabilities in your dependencies as you develop and test your applications.

6.7 License scanning

License scan job should mark license compliance as specified.

6.8 Node scan-sast

Is static code scanner which is used to find security flaws specially in node.js applications

6.9 Retire-js-dependency

Helps you to identify dependencies with known vulnerabilities in your application javascript libraries.

6.10 Semgrep-sast

Semgrep is default SAST analyzer for Javascript, Python and Typescript languages.

6.11 Build

Build the container and puts it to GitLab container registry

6.12 Deploy

Deploys the container to kubernetes
In deploy stage we use Bitnami stacks run scripts like kubectl --f \ to Virtual machine.
Use image bitnami/kubectl:latest

6.13 Robot Framework

Holds the Robot framework test
Use docker container to run.
Export report from test in html format.

6.14 DAST

Website scanner
Can be applied to test in a runtime environment

7 Yamls

All configuration files in Kubernetes are written in YAML.

In Gitlab-ci.yml deploy stage script runs service.yaml and deployment.yaml files.

microk8s kubectl apply -f \

7.1 Deployment

Deployment is a configuration file where you describe the desired state of your deployment. In a deployment file you configure e.g. the image for your Pods and the number of Pods.

View from deployments in Kubernetes Dashboard-API.

7.2 Service

Services are meant to give your Pods a static IP address, DNS name and can load-balance across them.

View from services in Kubernetes Dashboard-API.

8 Gitlab runner

Gitlab runner runs the command you assing in .gitlab-ci.yml file inside your Kubernetes cluster:

Use command when: manual if you want to unautomated specific stage for runner.

Information about installation can be found under "3.2 kubernetes applications"

9 Policy

We used Kyverno to test different policies in our local Kubernetes cluster due to its ease-of-use and policy template library. However, we decided to not implement these policies to production since it would hinder the deployment of WIMMA Lab virtual companies' products. It is important to implement these policies in the future since it greatly increases the security of the Kubernetes cluster and prevents misconfigurations. Due to time constraints, we only tested handful of the policies that Kyverno has to offer.

Step 1

Connect to your VM and install Kyverno using their latest release manifest (https://kyverno.io/docs/introduction/#quick-start):

kubectl create -f https://raw.githubusercontent.com/kyverno/kyverno/main/definitions/release/install.yaml

Step 2

After installing Kyverno, you can start to implement their policies, which can be found from their website (https://kyverno.io/policies/):

Step 3

Choose a policy to implement e.g disallow default namespace and copy its YAML file content:

Step 4

Paste the content into a YAML file and change the validationFailureAction from audit to enforce. After this apply the file with "sudo microk8s kubectl apply --f \<filename>.yaml":

Step 5

Try to create a deployment in the default namespace and you should see an error:

List of all the policies we tested can be found below:

Disallow Latest Tag

<!-- -->

- Disallow Default Namespace

Disallow NodePort
Disallow Service Type LoadBalancer
Require Limits and Requests
Add Network Policy

10 Dashboard-API

Dashboard-API owns security risk. It should be noted that Dashboard-API should be active only when it is on use. It is recommended to use SSH connection and cmd instead.

11 Prometheus

After installing Prometheus via GitLab you may find Prometheus monitoring window under

GitLab/ Kubernetes/ Health

Prometheus health view.

Kuva, joka sisältää kohteen pöytä Kuvaus luotu
automaattisesti

12 Summary

During summer 2021 we managed to create working Kubernetes platform. With earlier presented steps it's easy and quite simple to create / install. There are two different ways to install the Microk8s. For this project assignment and further development, it was useful to create those both.

In our project we work mainly with Microk8s Kubernetes distribution. We didn't have time to greatly compare and do major tests what was different when using other Kubernetes distributions like k9s or k3s. Right at the beginning of the project we tested some steps and exercises using a Minicube, but there is no basis for wider comparison either.

All pipelines in our projects were created in GitLab, we did not use any external CI-plugins.

GitLab provided great tools for CI/CD pipeline. Of course, it would be useful to learn more in future about how external pipeline are connected to work with Kubernetes and GitLab and what potential benefits it would bring to.

When working with Kubernetes yml.file syntax becomes familiar. In advanced use of Kubernetes platform, it's useful to expand your yml file knowledge with Helm charts. With helm charts you can create versions from your configuration. It provides a vehicle and way for developers to package their applications and share them with their community.

13 Things that were not achieved

Due to the project schedule and the completion times of other teams' projects, certain milestones were missed.

For our part, we wished that there was more time for the project. So that we could go even further in our development.

13.1 Monitoring

We did not have time for proper monitoring.

It would have been good to test more widely the user load of other teams completed projects.

In our tests the Kubernetes platform is stable, and pods are update as we wanted when they were moved or crashed.

13.2 Testing

No time for to do targeted and specified robot framework tests.

13.3 Proxy & port control

Adding proxy could add security and ease port management and port control.

13.4 Hardening

Security with Kubernetes is challenging. Although containers that runs on Kubernetes enable greater ability, speed, and portability, to take advantage of microservices architectures, they can also create security blind spots and increase your attack surface. , therefore, hardening of the cluster is an absolute must if you are to run Kubernetes in production.

A deep understanding of how Kubernetes works and how to follow best practices should be achieved in order to run a secure, highly accessible, production-ready Kubernetes cluster

Kubernetes is a robust container orchestration platform, the sheer level of complexity with multiple moving parts overwhelms all administrators.

Microk8s have a huge number and possibility of configurations, and while you can configure a few things correctly, chances are you could easily misconfigure a few things.

14 Things to consider, fix and implement

The following topics for further development and to consider. Depending on the project what runs on Kubernetes the next topics should be planned.

14.1 Multiple nodes

If your cluster has multiple nodes e.g. backend and frontend

you should think and design the level of integration with GitLab

(is integrating the cluster with frontend repo the best option since the cluster holds both front and backend?).

14.2 Runners:

Which kind of GitLab runner you are going to use?

14.3 Probes:

Network policy ® isolating every namespace and opening up only the connections that are needed (pods can connect to pods in every namespace by default which is a big security risk)

Cluster should have proxy VM. Metallb loadbalancer helps but a proxy machine is better. CSC has limited resources so this can be tricky if the CSC project has lot of machines e.g. DB, Back/Frontend.

Http(s) traffic ® proxy ® metallb/ingress ® kube cluster ® pod

TLS should be implemented to both local(LAN) and public traffic

Container security ® hack a container=hack the cluster