Git & GitLab: Feature Branch

Contents

  1. Tutorial
  2. Quick Reference
    • Getting started:
      • New Project
      • Existing Project
    • Creating a working branch
    • Submitting a Merge Request
    • Post merge clean up
  3. Situational
    • Did not select "Delete source branch" before merging
    • Did not selct "Squash commits" before merging
    • Self resolve a conflict
    • Switching your upstream branch when rebasing
  4. Good to Know
    • .gitignore
  5. GitLab CICD
  6. Data Automation

Tutorial

This tutorial is written for Linux distributions, Please adapt the commands accordingly for your platform

Setup

Install Git

Go to your home directory:

cd

Check if Git has already been installed, Git is usually preinstalled on Mac and Linux. Type the following command and then press enter:

git --version

You should receive a message that will tell you which Git version you have on your computer. If you don’t receive a "Git version" message, it means that you need to download Git.

After you are finished installing, open a new shell and type git --version again to verify that it was correctly installed.

Add your Git username and set your email

It is important to configure your Git username and email address as every Git commit will use this information to identify you as the author.

On your shell, type the following command to add your username:

git config --global user.name "Firstname Lastname"

Then verify that you have the correct username:

git config --global user.name

To set your email address, type the following command:

git config --global user.email "email@telkom.co.za"

To verify that you entered your email correctly, type:

git config --global user.email

To view the information that you entered, type:

git config --global --list

You'll need to do this only once as you are using the --global option. It tells Git to always use this information for anything you do on that system. If you want to override this with a different username or email address for specific projects, you can run the command without the --global option when you’re in that project.

Setup SSH

You now need to create an SSH private and public key pair for your account. This will allow you to clone and push without having to type in your username and password everytime. Follow these instructions to do that.

If you are working in a shared or unsecure environemnt, it's better to clone via HTTPS so that your SSH key is not exposed.

Create a Working Branch

Clone a Repo

We can now clone existing reposotories in GitLab to our machine.

Create a workspace directory

mkdir workspace

Navigate to your workspace direcotry

cd workspace

Clone the repository you would like to work on

git clone git@gitlab.com:bcx-insights/test

Navigate to the cloned repository

cd test

You should be on the master branch. You can check this with "git status"

git status

Outputs:

On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean

Create a new local branch

git checkout -b script_update

Push your new branch to the remote (on gitlab.com) so it can be tracked:

git push origin script_update

If you don't want to specify the branch every time you push, can tell tell git to always push to a branch by setting the upstream:

git branch --set-upstream-to=origin/script_update
  • note: always use the same name locally and on gitlab.com for branches
  • however: it is possible to push to another branch instead

You can then push to the remote branch on gitlab using:

git push

You can also auto create or open merge request for feature branch, e.g script_update

git push --set-upstream origin script_update -o merge_request.create -o merge_request.target=target_branch_name -o merge_request.title="Merge request title"

We have now pushed our new branch to GitLab. This won't overwrite master, but rather create a copy of master where we can safly do our work

Make some changes to the repo such as adding a new file:

touch test.sh

Check the status of the repo

git status

Outputs:

On branch script_update
Untracked files:
  (use "git add <file>..." to include in what will be committed)

        test.sh

nothing added to commit but untracked files present (use "git add" to track)

Add the changes using git add. You can use the wildcard * here to add all files. Files in the .gitignore will be ignored.

git add test.sh

Commit the changes adding a meesage that describes the chnges on a high level

git commit -m "Added the test.sh script"

Now push the local changes to your remote branch

git push origin script_update

Now you can safely start editing code. Good luck!

Submitting a Merge Request

Now that we have finished work on our branch, we need to incorperate the work back into master by submitting a merge request.

Pull updates from master

First make sure all your current work is commited

git add *
git commit -m "updates add to allow for use on Windows"

Then we need to check if any updatdes have been pushed to master

git fetch origin master

If there are no updates, your output looks like this and you can push (see below)

From gitlab.com:strategicinsights/environments
 * branch            master     -> FETCH_HEAD
 ```

If there are updates, the output looks more like this
```shell
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (1/1), done.
From gitlab.com:strategicinsights/environments
 * branch            master     -> FETCH_HEAD
   1d4286c..76fce30  master     -> origin/master

We will need to pull them into our branch before we can merge into master

git pull origin master --rebase

We should see something along the lines of

From gitlab.com:strategicinsights/environments
 * branch            master     -> FETCH_HEAD
First, rewinding head to replay your work on top of it...
Applying: someone_elses_merge

If you see "CONFLICT" here, you need to call a senior and start a fight. If you win then your work may be merged. Just kidding, it means someone else changed the same thing you did and a senior needs to decide whose is better, so they will sit with you and get it sorted.

Sometimes, at this point you may need to commit your changes again, otherwise you should be able to push

Push and Create Merge Request
git push origin script_update

This might also fail if your branch's history was changed due to the rebase, in that case, just add the -F flag

git push -F origin script_update

You should see an output similar to this

Counting objects: 2, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 262 bytes | 0 bytes/s, done.
Total 2 (delta 1), reused 0 (delta 0)
remote:
remote: To create a merge request for test, visit:
remote:   https://gitlab.com/strategicinsights/environments/-/merge_requests/new?merge_request%5Bsource_branch%5D=test

As the shell kindly told us above, we can visit the repo on GitLab to create a Merge Request

Login to GitLab and navigate to the repo

GitLab Login

Once we are on the page of the repo, we can create a new merge in a number of diffrent ways, if you recently pushed, you can click the blue button as done below. Alternativce you can use the side bar to go to "Repository → Branches" and click "Merge request" next to the branch you would like to merge.

GitLab Login

As you can see, we typed a lousey message in our demo merge request, still giving a description of the work done. Markdown is supported so your's should be better, here is a nice template:

##### This merge completes two tickets:
- [x] [<Jira Ticket #>](<Jira Ticket URL>) <Jira Ticket Decription>
- [x] [<Jira Ticket #>](<Jira Ticket URL>) <Jira Ticket Decription>

## <Subject of work>
* A thing was done
* A thing was improved
* Also the silly broken thing is fixed! Rejoice!

## Cloud Functions:
* Renamed and re-ordered some function
* Updated the links for the APIs
* Added the new recruits as users

## Other Updates
* I made a small change over here

## Package Changes
##### 
* __Updated!__ D3 Format{- 1.3.0 -} → {+ 1.3.2 +}
* __Updated!__ Firebase {- 5.1.0 -} → {+ 5.4.1 +}

Make sure to let your reporter/team know your code is ready for review by droping a message in Teams. You should continue with other work while you wait for your code to be reviewed and your merge request approved. Good luck!

Post Merge Clean Up

Merge

Congratulations! Someone believes in your code enough to allow you to merge it into Master.

The first thing to do is merge the code if it has not been done by your reviewer. Make sure it has been 👍'ed. Please be sure to check "Delete source branch" and "Squash commits". This will save us from doing a lot of cleaning up later

GitLab Merge

Clean Up

Congratulations! Your code is now part of history. And thanks to the "Delete source branch" and "Squash commits" the remote is clean and ready for everyone to rebase off of. We still need to clean up our local repo though. Head back to your terminal and switch to the master branch

git checkout master

You should have an output similar to:

Switched to branch 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
  (use "git pull" to update your local branch)

Let's pull the changes we just made

git pull origin master

Outputs:

remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (1/1), done.
From https://gitlab.com/strategicinsights/openserve-gis
 * branch            master     -> FETCH_HEAD
   751fd56..ad28358  master     -> origin/master
Updating 50196c4..ad28358
Fast-forward
 .gitignore | 19 +++++++++++++++++++
 README.md  | 30 +++++++++++++++++++++++++++++-
 2 files changed, 48 insertions(+), 1 deletion(-)
 create mode 100644 .gitignore

Now we delete our local working branch that we just merged:

git branch -D branch_name

You will get a

Deleted branch branch_name (was e931d47).

If instead you get a

error: The branch 'initial' is not fully merged.
If you are sure you want to delete it, run 'git branch -D initial'.

First make sure that:

  • the merge was succsesful
  • you are on master
  • have pulled the latest changes

After that, running another git status should give you:

On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean

And you're done! You probably want to start working on the next feature, create a new working branch to do that:

git checkout -b new_feature 
git push origin new_feature

You're now ready to work on your shiny new feature. Good luck!

Rename branch local and remote

  1. git checkout <old_branch_name>
  2. git branch -m <new_branch_name>
    • At this point, you have renamed the local branch.
    • If you’ve already pushed the branch to the remote repository , perform the next steps to rename the remote branch.
  3. git push origin -u <new_branch_name>
  4. git push origin --delete <old_branch_name>

Quick Reference

Terminology

  • Repo → Short for repository, the place we used to store code in the cloud. We usually have one per proect/tool
  • Remote → The nickname for any repo in the cloud
  • Clone → Copy the repo from the cloud to your machine
  • Branch → The key mechinisim that allows multiple people to work on the same thing. After cloning you create your own "branch", and this allows you to isolate your work from everyone elses
  • Feature Branch Methadology → The practice of creating a new branch for every feature you create. This keeps things clean and makes them easier to manage
  • Master → The main or "production" branch, its usually protected and all code needs to be reviewed before being added to master
  • Commit → A snapshot of all the changes made on your branch. After editing, we "commit" our work and this acts as a checkpoint we can revert back to anytime. Commit as often as possible!
  • Push → After commiting our work, it will be saved locally, but if we want to save our commit to the remote as well, we need to "push" it there. Pushing is key to making sure your work is backed up, you should push everytime you commit
  • Merge → The act of bringing two branches together
  • Merge Request → A request required to be opened to mere your branch into another
  • Pull → Downloading all updates from the remote is a manual process and is triggered with the pull command
Advanced
  • Squish → Taking any number of commits on a branch and turning them all into a single commit
  • Rebase → Rewinding all the changes you have made so that updates for your current branch can be pulled from the remote, and then your commits are added back again
  • Conflict → What happens when the same line of code has been modified by two seperate branches. Needs to be manually resolved, find a senior to help!

Getting Started

New Project
  1. Go to GitLab's home page and click "Create Project" in the top right
  2. Give the project a descriptive name in "Project name"
  3. Click the dropdown under "Project URL" and make sure to select te correct namespace for the project, ideally your team's Group
  4. The "Project slug" will be the part of the URL that comes after the namespace slected under "Project URL"
  5. Make sure "Visibility Level" is set to "Private"
  6. Tick "Initialize repository with a README"
  7. Click "Create project"
  8. Clone your newly created project as outlined below
Existing Project
  1. Navigate to the page of the repo you wish to clone
  2. Click on the "Clone" dropdown
  3. If you are cloning to a machine setup with SSH (see here), copy the SSH URL, otherwise copy the HTTPS URL (requires you to authenticate everytime you access the remote repo)
  4. Open your terminal and type git clone <url> where "" is the URL you copied
  5. Press enter

By default the repo will be cloned into a new directory of the same name. You can also choose your own name: git clone <url> <dir_name> where "" is your name for the directory

Create a working branch

Submitting a Merge Request

Post merge clean up

Commands

GitLab CICD

About

GitLab CI/CD is a tool built into GitLab for software development through continuous methodologies like Continuous Integration (CI), Continuous Delivery (CD), Continuous Deployment (CD). Continuous Integration works by pushing code changes to your application's codebase hosted in a Git repository, and, to every push, run a pipeline of scripts to build, test, and validate the code changes before merging them into the main branch. Continuous Delivery and Deployment consist of a step further CI, deploying your application to production at every push to the default branch of the repository.

These methodologies allow you to catch bugs and errors early in the development cycle, ensuring that all the code deployed to production complies with the code standards you established for your app/project. GitLab CI/CD is configured by a file called .gitlab-ci.yml placed at the repository's root. The scripts set in this file are executed by the GitLab Runner.

alt text

Continous Integration (CI)

Continous Integration allows developers to merge their changes back to the main branch as often as possible. The developers changes are being validated by creating a build running automated tests against that build and when done correctly you are able to validate more frequently that an application or project has not broken as commits are integrated into the main branch, ensuring the introduced changes pass all tests, guidelines, and code compliance standards you established for your app/project.

Continous Delivery (CD)

Continous Delivery is an augmentation of continuous integration to confirm you're able to release new changes more often in an acceptible way, and in addition to automating your testing, the automation of your release process gives you the ability to deploy your application continuously, though the deployments are triggered manually. This method ensures the code is checked automatically but requires human intervention to manually and strategically trigger the deployment of the changes.

Continous Deployment (CD)

Continuous Deployment is also a further step beyond continuous integration, similar to continuous delivery. The difference is that instead of deploying your application manually, you set it to be deployed automatically. It does not require human intervention at all to have your application deployed.

GitLab CI/CD Components

  • Pipeline: are the top-level component of continuous integration, delivery, and deployment. Pipelines comprise of jobs that define what to run. For example, code compilation or test runs and stages that define when and how to run.

alt text

  • .gitlab-ci.yml: GitLab CI/CD pipelines are configured using a YAML file called .gitlab-ci.yml within each project. The .gitlab-ci.yml file defines the structure and order of the pipelines and determines what to execute using GitLab Runner and what decisions to make when specific conditions are encountered.

alt text

A simple pipeline commonly has three stages:

* build
* test
* deploy

You do not need to use all three stages; stages with no jobs are ignored.

  • GitLab Runner: GitLab Runner is the open source project that is used to run your jobs and send the results back to GitLab. It is used in conjunction with GitLab CI, the open-source continuous integration service included with GitLab that coordinates the jobs.
  • GitLab Package Registry: GitLab Packages allows organizations to utilize GitLab as a private repository for a variety of common package managers. Users are able to build and publish packages, which can be easily consumed as a dependency in downstream projects.

How GitLab CI/CD Works?

To use GitLab CI/CD, all you need is an application codebase hosted in a Git repository, and for your build, test, and deployment scripts to be specified in a file called .gitlab-ci.yml, located in the root of your repository. In this file, you can define the scripts you want to run, define include and cache dependencies, choose commands you want to run in sequence and those you want to run in parallel, define where you want to deploy your app/project, and specify whether you will want to run the scripts automatically or trigger any of them manually. Once you’re familiar with GitLab CI/CD you can add more advanced steps into the configuration file.

To add scripts to that file, you'll need to organize them in a sequence that suits your application and are in accordance with the tests you wish to perform. To visualize the process, imagine that all the scripts you add to the configuration file are the same as the commands you run on a terminal on your computer. Once you've added your .gitlab-ci.yml configuration file to your repository, GitLab will detect it and run your scripts with the tool called GitLab Runner, which works similarly to your terminal. The scripts are grouped into jobs, and together they compose a pipeline.

GitLab Runners

About

GitLab Runner is the open source project that is used to run your jobs and send the results back to GitLab. It is used in conjunction with GitLab CI, the open-source continuous integration service included with GitLab that coordinates the jobs.

Requirements

  • GitLab Runner is written in Go and can be run as a single binary, no language specific requirements are needed.
  • It is designed to run on the GNU/Linux, MacOS, and Windows operating systems. Other operating systems will probably work as long as you can compile a Go binary on them.
  • If you want to use Docker for Docker Executor and Docker images, install the latest version. GitLab Runner requires a minimum of Docker v1.13.0

In this documentation, I'm focusing on installing gitlab runner in Linux, with Docker Engine.

Install Docker Engine

To install Docker Engine, you need the 64-bit version of one of these Ubuntu versions. Docker Engine is supported on x86_64 (or amd64), armhf, and arm64 architectures.

Setup the repository

I'm installing using the repository, for ease of installation and upgrade tasks. This is the recommended approach.

  1. Update the apt package index and install packages to allow apt to use a repository over HTTPS:
sudo apt-get update
sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common
  1. Add Docker's official GPG key:
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Verify that you now have the key with the fingerprint 9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88, by searching for the last 8 characters of the fingerprint.

sudo apt-key fingerprint 0EBFCD88
  1. Set up the stable repository
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

Install Docker

Update the apt package index, and install the latest version of Docker Engine and containerd.

sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io

Allow your user to access the Docker CLI without needing root access.

sudo usermod -aG docker $USER

To check if docker is installed successfully, run the following comming:

docker --version

Setting up proxy for Docker in Ubuntu

The Docker services will not use environment variables to get proxy information. Instead, you will have to configure the service to use a proxy.

Systemd Docker Service

In order to set the proxy for Docker, you will need to create a configuration file for the Docker service. No configuration files exist by default, so one will have to be created.

All Systemd service configuration are stored under /etc/systemd/system. In order to keep things organized, we can create a child directory docker.service.d, rather than creating a file called /etc/systemd/system/docker.service.

Within this configuration file, we can set our HTTP and HTTPS proxy. We can also set our NO_PROXY value too.

Creating Proxy Configuration

  1. Create a new directory for our Docker service configurations.
sudo mkdir -p /etc/systemd/system/docker.service.d
  1. Create a file called proxy.conf in our configuration directory
sudo nano /etc/systemd/system/docker.service.d/proxy.conf
  1. Add the following content, changing the values to match your environment.
[Service]
Environment="HTTP_PROXY=http://our_proxy:8080"
Environment="HTTPS_PROXY=https://our_proxy:8080/"
Environment="NO_PROXY="localhost,127.0.0.1,::1"
  1. Save changes and exit the text editor
  2. Reload the daemon configuration
sudo systemctl daemon-reload
  1. Restart Docker to apply our changes
sudo systemctl restart docker.service

NB: After the service is restarted Docker should be able to pull images from external repositories. You can test this by attempting to pull down an image. If the download completes and does not timeout, your proxy settings have been applied.

sudo docker run hello-world

Install GitLab Runner

Installing gitlab runner using binary file.

  1. Simply download one of the binaries for your system (I'm using one for Linux)
# Linux x86-64
sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64
  1. Give it permissions to execute
sudo chmod +x /usr/local/bin/gitlab-runner
  1. Create a GitLab CI user
sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash
  1. Install and run as service
sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
sudo gitlab-runner start

  1. Registering Runners Registering a Runner is the process that binds the Runner with a GitLab instance.

    • Register the runner: sudo gitlab-runner register
    • Enter your GitLab instance URL You can find it in repository > settings > CI/CD > expand Runners tab.

    alt text Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com ) https://gitlab.com * Enter the token you obtained to register the Runner You can find it in repository > settings > CI/CD > expand Runners tab, below gitlab instance url. Please enter the gitlab-ci token for this runner xxx * Enter a description for the Runner, you can change this later in GitLab's UI Please enter the gitlab-ci description for this runner [hostname] TSI Runner * Enter the tags associated with the Runner (Optional), you can change this later in GitLab's UI Please enter the gitlab-ci tags for this runner (comma separated): my-tag,another-tag or you can leave empty * Enter the Runner executor Please enter the executor: ssh, docker+machine, docker-ssh+machine, kubernetes, docker, parallels, virtualbox, docker-ssh, shell: shell If you chose Docker as your executor, you'll be asked for the default image to be used for projects that do not define one in .gitlab-ci.yml

Please enter the Docker image (eg. ruby:2.6):
alpine:latest
  1. Open the nano editor on Ubuntu
sudo nano /etc/sudoers

then add the following line to the bottom of the file

gitlab-runner ALL=(ALL) NOPASSWD: ALL

Running GitLab Runner behind proxy

The proxy variables need to also be added to the Runner's configuration, so that it can get builds assigned from GitLab behind the proxy. 1. Create a systemd drop-in directory for the gitlab-runner service

mkdir /etc/systemd/system/gitlab-runner.service.d
  1. Create a file called /etc/systemd/system/gitlab-runner.service.d/http-proxy.conf that adds the HTTP_PROXY environment variable(s)
sudo nano /etc/systemd/system/gitlab-runner.service.d/http-proxy.conf
  1. Add the following content
[Service]
Environment="HTTP_PROXY=http://our_proxy:8080/"
Environment="HTTPS_PROXY=http://our_proxy:8080/"
  1. Save the file and flush changes
systemctl daemon-reload
  1. Restart GitLab Runner
sudo systemctl restart gitlab-runner
  1. Verify that the configuration has been loaded
systemctl show --property=Environment gitlab-runner

You should see the following:

Environment=HTTP_PROXY=http://our_proxy:8080/ HTTPS_PROXY=http://our_proxy:8080/

Adding the proxy to the Docker containers

After you registered your Runner, you might want to propagate your proxy settings to the Docker containers (for git clone and other stuff).

To do that, you need to edit /etc/gitlab-runner/config.toml and add the following to the [[runners]]section:

pre_clone_script = "git config --global http.proxy $HTTP_PROXY; git config --global https.proxy $HTTPS_PROXY"
environment = ["https_proxy=http://our_proxy:8080", "http_proxy=http://our_proxy:8080", "HTTPS_PROXY=our_proxy:8080", "HTTP_PROXY=our_proxy:8080"]

Using SSH Public Key Authentication

For us to be able to deploy to the dev or production environment, we need passwordless connection to the server. For us to this, we need to generate the SSH keys. 1. Switch to gitlab-runner user

sudo su - gitlab-runner
  1. Run the following command
ssh-keygen

You’ll see a response similar to:

Generating public/private rsa key pair.
Enter file in which to save the key (/home/gitlab-runner/.ssh/id_rsa):

Press ENTER button to leave it as default settings, then you will see the following (leave it empty too by pressing ENTER):

Enter passphrase (empty for no passphrase):
Enter same passphrase again:

If successful, you’ll see confirmation of where the ssh-keygen command saved your identification and private key.

Copy private key to GitLab variable

  1. Navigate to repository > settings > CI/CD > Variables tab, then create new variable. We'll save our key as $SSH_PRIVATE_KEY variable. This is the key we use in our .gitlab-ci.yml file using ssh-agent.
  2. Save the output of the following command to variable $SSH_KNOWN_HOSTS
ssh-keyscan -H host
  1. Copy public key to authorized_keys to make it available in the server, this is for login in the server passwordless.
cat /tmp/id_rsa.pub >> ~/.ssh/authorized_keys

Then you should be able to connect to the remote server without issues!

Advanced Configuration

GitLab Runner configuration uses the TOML format.

The file to be edited can be found in: /etc/gitlab-runner/config.toml, where all of the gitlab-runner information (name, registered token, executor type, shell etc...) are listed in config.toml file.

listen_address = "0.0.0.0:0000"
check_interval = 0

[[runners]]
  name = "data-automation runner 1"
  url = "https://gitlab.com/"
  token = "*******token*******"
  executor = "shell"

There are many configurations available on GitLab docs but in this section, I will just focus on concurrency pipeline settings.

In our case, three important options are: concurrent, limit, and request-concurrency.

  • concurrent (global section): limits how many jobs globally can be run concurrently...,That means, it applies to all the runners independently regardless of the executor [docker, ssh, kubernetes etc...]. The most upper limit of jobs using all defined runners. 0 does not mean unlimited.
  • limit (runner section): limit how many jobs can be handled concurrently by this token. Suppose that we have 2 runners registered by 2 different tokens, then its limit could be adjusted separately: runner-one limit = 2, runner-two limit = 3, etc...! 0 (default) simply means don’t limit.
  • request_concurrency (runner section): limit number of concurrent requests for new jobs from GitLab (default = 1)

Below code snippet will help us understand clearly about three option outline above:

root@hubul-cov-tsi01:/home/gitlab-runner# nano /etc/gitlab-runner/config.toml
listen_address = "0.0.0.0:0000"
concurrent = 4
check_interval = 0

[[runners]]
  limit = 3
  request_concurrency = 3
  name = "data-automation runner 1"
  url = "https://gitlab.com/"
  token = "**********token**********"
  executor = "shell"

[[runners]]
  limit = 3
  request_concurrency = 3
  name = "data-automation runner 2"
  url = "https://gitlab.com/"
  token = "**********token**********"
  executor = "shell"

  [[runners]]
  limit = 3
  request_concurrency = 3
  name = "FFP runner 1"
  url = "https://gitlab.com/"
  token = "**********token**********"
  executor = "shell"

  [[runners]]
  limit = 3
  request_concurrency = 3
  name = "FFP runner 2"
  url = "https://gitlab.com/"
  token = "**********token**********"
  executor = "shell"

With above specs, one gitlab-runner hosts now could run in parallel up to 4 jobs concurrently (4 runners, and each runner have 3 sub-processes).

alt text

Additional Resources

Data Automation Merge Guidelines

Step by step guide on how to merge a feature branch into master:

This is the step by step guide for data automation flow from a feature_branch all the way to master. The changes (from dev or feature_branches) go through several steps (branches) before they gets to master (prod).

The Flow:

alt text

Importat Notes:

Before changes are merged into any target branch, it must be checked or reviewed by two code base knowledgeable people.

Steps:

  1. Merge feature_branch into dev branch.
    • When merging a feature_branch into dev, we delete a feature_branch and squash the commits as well.
  2. Once all changes from a feature branch are merged in dev, then we merge dev branch into master_dev.
    • When merging dev into master_dev, we DON'T DELETE the dev branch and we also DON'T SQUASH the commits to avoid branch diverge.
  3. This is an automated step run by gitlab ci pipeline.
    • What happens is master_staging branch get created off master_dev and then cherry pick to_prod changes into master_staging. This process converts all _dev stuff into prod.

    • In the process of cherry picking, we might get conflicts that needs to be resolved manual. The pipeline job will still show succeed, the best way to see if this step was successful is to check the graph to see if to_prod commit was applied into master_staging branch - refer to screenshot below.

      alt text

      otherwise master_dev and master_staging will appear in the same line to_dev commit. - To resolve the conflicts manual, run the commands below:

      First make sure you have updated `master_dev` and `master_staging` branches locally.
```
# For master_dev
git checkout master_dev
git pull origin master_dev

# For master_staging
git checkout master_staging
git pull origin master_staging
```
Once you have them updated, then run the following:
```
git checkout master_dev
git branch --set-upstream-to=origin/master_dev
git pull
git checkout master_staging
git cherry-pick SHA commit for to_prod branch
git push origin master_staging -f
```
After this, you should have the same output as in the screenshot above.
      1. Merge master_staging into master.
        • When merging master_staging into master, we delete master_staging and squash the commits.
        • Update merge description to include descriptive commit message(s) for all commits made.
      2. Make sure the CI pipeline runs for master to rebase dev and to_dev branches with the latest changes on master, rebase to_prod with to_dev. The pipeline also creates master_dev off dev branch and master_staging off master_dev branch then apply to_prod commit into master_staging.

Troubleshooting

When we running the Gitlab CI/CD pipeline, we might encounter errors in our jobs along the way.

The error we most likey to encounter is CONFLICTS when running REBASE on feature_branches, dev, to_dev, and to_prod branches and when running CHERRY-PICK on master staging.

Solution to conflicts is to resolve the conflicts manual in your local machine and push back the changes.

For each of the branches mentioned about if they have conflict issue, this is how you resolve them:

In feature branches:

git checkout <feature branch name>
git pull origin dev --rebase
git push origin <feature branch name> -f

In Dev:

git checkout dev
git pull origin dev
git branch --set-upstream-to=origin/master
git pull origin master --rebase
git push origin dev -f

In to_dev:

git checkout to_dev
git pull origin to_dev
git branch --set-upstream-to=origin/master
git pull origin master --rebase
git push origin to_dev -f

In to_prod:

git checkout to_prod
git pull origin to_prod
git branch --set-upstream-to=origin/to_dev
git pull origin to_dev --rebase
git push origin to_prod -f

In master_staging:

Refer to step: 3 above. It's the same process.

Once done pushing the changes you can retry running each job, respectively. It is not compulsory as changes are already updated with your push.