Table of Contents


I recently switched almost all of my personal projects away from Github to Gitlab. Gitlab offers a ton of cool free features, including a built-in CI tool. I’ve set up and used various CI tools before and currently run a Jenkins server on my Raspberry Pi. This setup is used to update my wife’s weekly “crime and comedy” webcomic site shamseecomic.com. All 150ish pages of the second book are done and we post every Wednesday.

Up till now I’ve been manually copying over the new image and post every Wednesday, building locally, then using the S3_website plugin to deploy the site to S3 and CloudFront. But I was at the O’Reilly Velocity Conference in NYC last week for work and the new page didn’t go up as expected so I started looking for a better solution.

Enter GitlabCi.

Much like Github’s ability to build and host your Jekyll site, GitlabCI offers the same functionality with a WHOLE lot more configuration and customization options.

One of the key differences being I’m using GitlabCI to build and deploy my site to Amazon S3 instead of hosting it directly on Gitlab Pages. The reasons why I’m using S3 are for another time, for now, lets look at the how.

Enabling CI

Since I’m starting with an existing Gitlab project CI was disabled. I’m not sure if I accidentally did that when I first migrated over from Github or what, but if you don’t see the CI/CD settings (CI/CD Pipelines, Triggers, Runners, etc) like the screenshot, here’s how to re-enable it:
CI/CD Settings

Open your project and choose Edit Project
Open your project and choose Edit Project

Scroll down to Feature Visibility and if Builds is disabled then enable it.
Enable Builds

Add Secure Variables

Remember when I said that Gitlab offered some cool features that Github doesn’t? Secure Variables are one of those features. In short, you can create variables via the Gitlab website and then use them during your GitlabCI stages. This is great for doing things like supplying passwords or AWS credentials and prevents you from accidentally commiting them to a public repo.

The S3_website tool (which we’ll set up in the next step) requires four pieces of information from us in order to work. We’ll provide that information here and import it back in later.
Find this information:

Let’s create our Variables. Go to your project, and click on the settings gear:
Click on Settings

Enter the key value pairs for each of the four variables you copied in the previous step:
Click on Settings

  • AWS S3_ID
  • AWS S3_SECRET
  • AWS S3_BUCKET
  • CLOUDFRONT_DISTRIBUTION_ID

Add S3 Website to Your Gemfile

The easiest way to install and use S3_Website is to install it as a Gem. Make sure to add the following to the Gemfile in the root of your project.
gem "s3_website".

You don’t need to worry about installing it for now since that step will be taken care of by our GitlabCI config.

Create s3 Website Config

As I’ve said, the s3_website tool requires a little bit of configuration from us. What I’ve included below works for anyone hosting their static site on S3 with Amazon’s CloudFront CDN. The config file pulls in the four Secure Variables you created in the step before. The file is below.

# These vars are passed from GitlabCI Secure Variables
s3_id: <%= ENV['S3_ID'] %>
s3_secret: <%= ENV['S3_SECRET'] %>
s3_bucket: <%= ENV['S3_BUCKET'] %>

site: public/
max_age:
  "css/*": 604800
  "images/*": 604800
  "*": 300

gzip: true

cloudfront_distribution_id: <%= ENV['CLOUDFRONT_DISTRIBUTION_ID'] %>
cloudfront_invalidate_root: true

Create gitlab-ci Config

Now that we’ve enabled builds we need to create our .gitlab-ci.yaml file. This file tells GitlabCI exactly what to do. Below is my initial working script. It’s ugly and not optimized but it’s functional. Eventually I’ll create a custom Container that contains all dependencies. But for now, here’s the functional file. I’ll go through it chunk by chunk, but in short I’ve defined the following:

  • Run a jekyll build every time you push to master
  • Create a “trigger” / url endpoint that when visited will trigger
    • a jekyll build
    • an s3_website deploy

This division allows us to constantly push to Master and see wether a build passed or failed before we actually deploy the site via visiting the trigger.

image: ruby:2.3

cache:
  paths: 
    - vendor/

before_script:
  - bundle install --path vendor

build:
  stage: build
  script:
    - JEKYLL_ENV=production bundle exec jekyll build -d public/
  artifacts:
    paths:
      - public

vars:
  type: test
  script:
    - 'export S3_ID="$(echo "$S3_ID")" >> s3_deploy_vars'
    - 'export S3_SECRET="$(echo "$S3_SECRET")" > s3_deploy_vars'
    - 'export S3_BUCKET="$(echo "$S3_BUCKET")" > s3_deploy_vars'
    - 'export CLOUDFRONT_DISTRIBUTION_ID="$(echo "$CLOUDFRONT_DISTRIBUTION_ID")" > s3_deploy_vars'
  artifacts:
    paths:
      - s3_deploy_vars
  only:
    - triggers

deploy:
  stage: deploy
  script:
    - echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" >> /etc/apt/sources.list.d/java-8-debian.list
    - echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" >> /etc/apt/sources.list.d/java-8-debian.list
    - apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886
    - apt-get update
    - echo "oracle-java8-installer shared/accepted-oracle-license-v1-1 select true" | debconf-set-selections
    - apt-get install -y oracle-java8-installer
    - apt-get install -y oracle-java8-set-default
    - source s3_deploy_vars
    - bundle exec s3_website push
  artifacts:
    paths:
      - public
  only:
    - triggers

Disecting the .gitlab-ci.yaml File

image: ruby:2.3
This says to use the Docker image ruby version 2.3 from DockerHub.

cache:
  paths: 
    - vendor/

This says that when running these scripts cache and reuse the contents of the vendor folder. This folder is created by a later step.

before_script:
  - bundle install --path vendor

This is telling GitlabCI to run the command bundle install --path vendor before any script is run for a task. This ensures that all required Gems have been installed.

build:
  stage: build
  script:
    - JEKYLL_ENV=production bundle exec jekyll build -d public/
  artifacts:
    paths:
      - public

Above is our first task which I’ve cleverly titled build.

  • STAGE: GitlabCI executes 3 stages for us to hook into, build, test, and deploy.
    • We’ve said it belongs to the stage build
  • SCRIPT: This is the command we want to be executed.
    • in this case we’re telling it to run jekyll build and save the site to /public instead of the default _site. This is simply because GitlabCI looks for the contents of the /public folder by default when running the deploy stage.
  • ARTIFACT: This is the directory we want attached to the build after success. In this case it’s our compiled website.
vars:
  stage: test
  script:
    - 'export S3_ID="$(echo "$S3_ID")" >> s3_deploy_vars'
    - 'export S3_SECRET="$(echo "$S3_SECRET")" > s3_deploy_vars'
    - 'export S3_BUCKET="$(echo "$S3_BUCKET")" > s3_deploy_vars'
    - 'export CLOUDFRONT_DISTRIBUTION_ID="$(echo "$CLOUDFRONT_DISTRIBUTION_ID")" > s3_deploy_vars'
  artifacts:
    paths:
      - s3_deploy_vars
  only:
    - triggers

Above is our second task which I’ve titled vars. We use it to make the ‘Secure Variables’ we set up earlier available for the build.

  • STAGE: This runs during the test stage which won’t begin until the ‘Build’ stage has completed successfully.
  • SCRIPT We’re echoing our Gitlab Secret Variables to a file called s3_deploy_vars, making them available for any tasks that follow.
  • ARTIFACTS We want to keep the s3_deploy_vars file around even after the build finishes.
  • ONLY This is where we add a conditional to when GitlabCI should trigger a build.
    • This task only builds if it was triggered and won’t build just because we pushed changes to master.
deploy:
  stage: deploy
  script:
    - echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" >> /etc/apt/sources.list.d/java-8-debian.list
    - echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" >> /etc/apt/sources.list.d/java-8-debian.list
    - apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886
    - apt-get update
    - echo "oracle-java8-installer shared/accepted-oracle-license-v1-1 select true" | debconf-set-selections
    - apt-get install -y oracle-java8-installer
    - apt-get install -y oracle-java8-set-default
    - source s3_deploy_vars
    - bundle exec s3_website push
  artifacts:
    paths:
      - public
  only:
    - triggers

Above is our third task, titled deploy. It’s purpose is to use the s3_website tool to push our site to the Amazon S3 bucket it’s served from. If you’re not using AWS or S3 to host your site, this part isn’t for you.

  • STAGE: This runs during the deploy stage which won’t begin until the ‘test’ stage has completed successfully.
    • In our case, that’s our ‘vars’ task, which is why I included it in the ‘test’ stage to begin with. We want our $vars successfully exported before running our deploy task or else the deploy task will fail when it doesn’t have the required information.
  • SCRIPT: This is the ‘severely unoptimized’ aspect I was talking about. s3_website requires Java. Our ruby container doesn’t have Java installed.
    • the first 7 actions are just to install Java.
    • then we source s3_deploy_vars which reads the contents of our vars file and makes them available as environmental variables.
    • then we run the bundle exec s3_website push command to push our site
  • ARTIFACTS: We’re using the public directory again, which holds our compiled site
  • ONLY: And we’re only building if it was triggered again, not on pushes to master.

Create a Trigger

Now that everything is in place, let’s create our trigger. Go to your project, click on the settings gear and then click on Triggers:
Click on Settings

Click on add trigger
(the token in the screenshot was revoked before this was posted)
Click on Settings

That’s it, now test it out. Replace TOKEN with your actual trigger token and PROJECTID with your Gitlab project’s id and then Copy and paste this curl command to your trigger and watch your site build and deploy.

curl -X POST \
     -F token=TOKEN \
     -F ref=master \
     https://gitlab.com/api/v3/projects/PROJECTID/trigger/builds

Using Lambda and GitlabCI to schedule Jekyll posts

Like I said, this project was instigated by my wife’s webcomic site. We complete the updates well ahead of schedule so we always missed having a simple, reliable way to schedule updates. Now we have one - AWS Lambda.

In my next post I’ll show you how I use Lambda and GitlabCI to have fully automated scheduled posts with Jekyll.