Table of Contents
- Enabling CI
- Secure Variables
- Create a Trigger
- Using Lambda and GitlabCI to schedule Jekyll posts
I recently switched almost all of my personal projects away from Github to Gitlab. Gitlab offers a ton of cool free features, including a built-in CI tool. I’ve set up and used various CI tools before and currently run a Jenkins server on my Raspberry Pi. This setup is used to update my wife’s weekly “crime and comedy” webcomic site shamseecomic.com. All 150ish pages of the second book are done and we post every Wednesday.
Up till now I’ve been manually copying over the new image and post every Wednesday, building locally, then using the S3_website plugin to deploy the site to S3 and CloudFront. But I was at the O’Reilly Velocity Conference in NYC last week for work and the new page didn’t go up as expected so I started looking for a better solution.
Much like Github’s ability to build and host your Jekyll site, GitlabCI offers the same functionality with a WHOLE lot more configuration and customization options.
One of the key differences being I’m using GitlabCI to build and deploy my site to Amazon S3 instead of hosting it directly on Gitlab Pages. The reasons why I’m using S3 are for another time, for now, lets look at the how.
Since I’m starting with an existing Gitlab project CI was disabled. I’m not sure if I accidentally did that when I first migrated over from Github or what, but if you don’t see the CI/CD settings (CI/CD Pipelines, Triggers, Runners, etc) like the screenshot, here’s how to re-enable it:
Open your project and choose Edit Project
Scroll down to Feature Visibility and if Builds is disabled then enable it.
Add Secure Variables
Remember when I said that Gitlab offered some cool features that Github doesn’t? Secure Variables are one of those features. In short, you can create variables via the Gitlab website and then use them during your GitlabCI stages. This is great for doing things like supplying passwords or AWS credentials and prevents you from accidentally commiting them to a public repo.
The S3_website tool (which we’ll set up in the next step) requires four pieces of information from us in order to work. We’ll provide that information here and import it back in later. Find this information:
- AWS S3_ID: the Access Key Id of the AWS IAM user with permission to access the S3 bucket that hosts your site.
- You can find this at: https://console.aws.amazon.com/iam/home
- AWS S3_SECRET: the AWS Secret Key of the AWS IAM user with permission to access the S3 bucket that hosts your site.
- You can find this at: https://console.aws.amazon.com/iam/home
- AWS S3_BUCKET: the name of your bucket, most likely www.yoursite.com
- CLOUDFRONT_DISTRIBUTION_ID: the distribution id that you’ll be pushing to
- You can find this by logging in to your AWS dashboard and going to https://console.aws.amazon.com/cloudfront/home.
Let’s create our Variables. Go to your project, and click on the settings gear:
Enter the key value pairs for each of the four variables you copied in the previous step:
- AWS S3_ID
- AWS S3_SECRET
- AWS S3_BUCKET
Add S3 Website to Your Gemfile
The easiest way to install and use S3_Website is to install it as a Gem. Make sure to add the following to the Gemfile in the root of your project.
You don’t need to worry about installing it for now since that step will be taken care of by our GitlabCI config.
Create s3 Website Config
As I’ve said, the s3_website tool requires a little bit of configuration from us. What I’ve included below works for anyone hosting their static site on S3 with Amazon’s CloudFront CDN. The config file pulls in the four Secure Variables you created in the step before. The file is below.
# These vars are passed from GitlabCI Secure Variables s3_id: <%= ENV['S3_ID'] %> s3_secret: <%= ENV['S3_SECRET'] %> s3_bucket: <%= ENV['S3_BUCKET'] %> site: public/ max_age: "css/*": 604800 "images/*": 604800 "*": 300 gzip: true cloudfront_distribution_id: <%= ENV['CLOUDFRONT_DISTRIBUTION_ID'] %> cloudfront_invalidate_root: true
Create gitlab-ci Config
Now that we’ve enabled builds we need to create our .gitlab-ci.yaml file. This file tells GitlabCI exactly what to do. Below is my initial working script. It’s ugly and not optimized but it’s functional. Eventually I’ll create a custom Container that contains all dependencies. But for now, here’s the functional file. I’ll go through it chunk by chunk, but in short I’ve defined the following:
- Run a jekyll build every time you push to master
- Create a “trigger” / url endpoint that when visited will trigger
- a jekyll build
- an s3_website deploy
This division allows us to constantly push to Master and see wether a build passed or failed before we actually deploy the site via visiting the trigger.
image: ruby:2.3 cache: paths: - vendor/ before_script: - bundle install --path vendor build: stage: build script: - JEKYLL_ENV=production bundle exec jekyll build -d public/ artifacts: paths: - public vars: type: test script: - 'export S3_ID="$(echo "$S3_ID")" >> s3_deploy_vars' - 'export S3_SECRET="$(echo "$S3_SECRET")" > s3_deploy_vars' - 'export S3_BUCKET="$(echo "$S3_BUCKET")" > s3_deploy_vars' - 'export CLOUDFRONT_DISTRIBUTION_ID="$(echo "$CLOUDFRONT_DISTRIBUTION_ID")" > s3_deploy_vars' artifacts: paths: - s3_deploy_vars only: - triggers deploy: stage: deploy script: - echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" >> /etc/apt/sources.list.d/java-8-debian.list - echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" >> /etc/apt/sources.list.d/java-8-debian.list - apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886 - apt-get update - echo "oracle-java8-installer shared/accepted-oracle-license-v1-1 select true" | debconf-set-selections - apt-get install -y oracle-java8-installer - apt-get install -y oracle-java8-set-default - source s3_deploy_vars - bundle exec s3_website push artifacts: paths: - public only: - triggers
Disecting the .gitlab-ci.yaml File
This says to use the Docker image ruby version 2.3 from DockerHub.
cache: paths: - vendor/
This says that when running these scripts cache and reuse the contents of the vendor folder. This folder is created by a later step.
before_script: - bundle install --path vendor
This is telling GitlabCI to run the command
bundle install --path vendor before any script is run for a task. This ensures that all required Gems have been installed.
build: stage: build script: - JEKYLL_ENV=production bundle exec jekyll build -d public/ artifacts: paths: - public
Above is our first task which I’ve cleverly titled build.
- STAGE: GitlabCI executes 3 stages for us to hook into, build, test, and deploy.
- We’ve said it belongs to the stage build
- SCRIPT: This is the command we want to be executed.
- in this case we’re telling it to run
jekyll buildand save the site to /public instead of the default _site. This is simply because GitlabCI looks for the contents of the /public folder by default when running the deploy stage.
- in this case we’re telling it to run
- ARTIFACT: This is the directory we want attached to the build after success. In this case it’s our compiled website.
vars: stage: test script: - 'export S3_ID="$(echo "$S3_ID")" >> s3_deploy_vars' - 'export S3_SECRET="$(echo "$S3_SECRET")" > s3_deploy_vars' - 'export S3_BUCKET="$(echo "$S3_BUCKET")" > s3_deploy_vars' - 'export CLOUDFRONT_DISTRIBUTION_ID="$(echo "$CLOUDFRONT_DISTRIBUTION_ID")" > s3_deploy_vars' artifacts: paths: - s3_deploy_vars only: - triggers
Above is our second task which I’ve titled vars. We use it to make the ‘Secure Variables’ we set up earlier available for the build.
- STAGE: This runs during the test stage which won’t begin until the ‘Build’ stage has completed successfully.
- SCRIPT We’re echoing our Gitlab Secret Variables to a file called s3_deploy_vars, making them available for any tasks that follow.
- ARTIFACTS We want to keep the s3_deploy_vars file around even after the build finishes.
- ONLY This is where we add a conditional to when GitlabCI should trigger a build.
- This task only builds if it was triggered and won’t build just because we pushed changes to master.
deploy: stage: deploy script: - echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" >> /etc/apt/sources.list.d/java-8-debian.list - echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" >> /etc/apt/sources.list.d/java-8-debian.list - apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886 - apt-get update - echo "oracle-java8-installer shared/accepted-oracle-license-v1-1 select true" | debconf-set-selections - apt-get install -y oracle-java8-installer - apt-get install -y oracle-java8-set-default - source s3_deploy_vars - bundle exec s3_website push artifacts: paths: - public only: - triggers
Above is our third task, titled deploy. It’s purpose is to use the s3_website tool to push our site to the Amazon S3 bucket it’s served from. If you’re not using AWS or S3 to host your site, this part isn’t for you.
- STAGE: This runs during the deploy stage which won’t begin until the ‘test’ stage has completed successfully.
- In our case, that’s our ‘vars’ task, which is why I included it in the ‘test’ stage to begin with. We want our $vars successfully exported before running our deploy task or else the deploy task will fail when it doesn’t have the required information.
- SCRIPT: This is the ‘severely unoptimized’ aspect I was talking about. s3_website requires Java. Our ruby container doesn’t have Java installed.
- the first 7 actions are just to install Java.
- then we
source s3_deploy_varswhich reads the contents of our vars file and makes them available as environmental variables.
- then we run the
bundle exec s3_website pushcommand to push our site
- ARTIFACTS: We’re using the public directory again, which holds our compiled site
- ONLY: And we’re only building if it was triggered again, not on pushes to master.
Create a Trigger
Now that everything is in place, let’s create our trigger. Go to your project, click on the settings gear and then click on Triggers:
Click on add trigger (the token in the screenshot was revoked before this was posted)
That’s it, now test it out. Replace TOKEN with your actual trigger token and PROJECTID with your Gitlab project’s id and then Copy and paste this curl command to your trigger and watch your site build and deploy.
curl -X POST \ -F token=TOKEN \ -F ref=master \ https://gitlab.com/api/v3/projects/PROJECTID/trigger/builds
Using Lambda and GitlabCI to schedule Jekyll posts
Like I said, this project was instigated by my wife’s webcomic site. We complete the updates well ahead of schedule so we always missed having a simple, reliable way to schedule updates. Now we have one - AWS Lambda.
In my next post I’ll show you how I use Lambda and GitlabCI to have fully automated scheduled posts with Jekyll.