Lepo

Website previews for custom Netlify deployments using GitLab CI

As a developer of various websites, I find Netlify to be a fantastic platform for hosting static websites these days. It’s straightforward to use and it provides a lot of nice integrations to development tools. For example, Netlify can attached to Git hosting platforms such as GitHub and GitLab, where a lot of web development is coordinated these days.

One of the key features the Netlify integration provides is the ability to preview websites from pull/merge requests. When a developer provides website changes for review as a pull/merge request, Netlify will automatically prepare the site and make it available for preview. This way, the people reviewing the changes can easily do so from a temporary website instead of having to manually pull the changes to their computer.

Unfortunately, the website preview feature seems to be only available when Netlify takes over the CI/CD pipeline for your website. In other words, Netlify has to build your website to offer it for preview through the integration. This can be a dealbreaker when you’re already invested in a CI/CD platform such as GitHub Actions or GitLab CI. For example, I use GitLab CI a lot because of its flexibility (powered by Docker), and I’d prefer to keep all my pipelines there when possible.

Inspired by Netlify, I set out to marry the best of both worlds to provide website previews from GitLab CI pipelines to GitLab merge requests. In this post, I’ll demonstrate how you can set up your own GitLab CI pipeline to build a static website and publish both preview and production versions of it.

The setup

First, let’s have a brief look at the overall setup we would like to have for our website hosting and development. In the setup, we have the following components:

The setup has the following actors involved:

The components, the actors, and their relationships are visible in the architecture diagram below.

Architecture diagram

From the website development perspective, there are three main flows: The creation of the draft website when a developer uploads changes, the approval process the reviewer goes through, and the publishing of the changes. Here’s how they work in this setup.

When a developer uploads changes, they’ll make the changes available for review as a merge request (MR). Once the merge request has been created, a GitLab CI pipeline tests and builds the website and publishes it as a draft website in Netlify. Finally, the pipeline posts the draft website link to the merge request as a comment.

When a new merge request is created, the reviewer is asked to review the changes. From the merge request, the reviewer can inspect all the code and asset changes. From the draft link, the reviewer can inspect what the website would look like after applying the changes. The approval is done by either adding an approval mark to the merge request or by merging the merge request directly. The reviewer can also reject the changes and ask for the developer to add further changes on top of the merge request.

Once the merge request has been merged, the GitLab CI pipeline will again build the website and publish the changes to Netlify. This time, instead of creating a draft website, the changes will be made visible to the end-users.

All of these flows are visualized in the flow diagram below.

Architecture sequence diagram

In this guide, we’ll be using a very simplified setup for building the website. Normally, you would use a full-blown static website generator to build the website. The test phase could check that the content is laid out in proper format, the client-side JavaScript does what it’s supposed to, and that the links point to proper locations. Instead of using these, we’ll be using a single Markdown file that we convert to one HTML page, and we will not have any tests at all.

With the setup made clear, we can begin implementing it.

Accounts

To get started, we need the following accounts:

Bot access token

To post comments from the bot account, a personal access token needs to be generated. Generate a new access token with the api scope and no expiration date. Make sure you generate the token using the bot account.

Technically, the GitLab user account could be used for posting comments to the merge requests, but I wouldn’t recommend doing so. This is because the api scope grants wide read and write access to the entire account. If the token were to accidentally leak to unwanted users, they could abuse the account access in various ways: e.g. they could delete all user GitLab projects. Therefore, the bot account is used for limiting the blast radius in case such a leak happens.

Unfortunately, this is the only known way to arrange merge request comment access in GitLab for now. Hopefully, GitLab will add the capability to post comments to the merge requests using the pipeline token in the future.

Website space on Netlify

Next, we’ll need a space in Netlify to host our website. We can create one from the Netlify web UI.

Instead of creating a new site from a Git repository, we need to manually upload a directory first. This is so that Netlify doesn’t try to integrate its hooks into the project in GitLab, and that we can control the website deployment process ourselves. Here’s how we can do it:

  1. Create a new directory in your file browser.
  2. In the directory, create a new empty file named index.html.
  3. In the Netlify web UI, navigate to the “Sites” tab.
  4. Drag the new directory you created to the area that reads “drag and drop your site output folder here”.

Your website space should now be created. You can continue tuning the site settings to your liking (e.g. change the site name).

API access to Netlify

To access our Netlify website space from GitLab CI, we’ll need the following information:

  1. Site ID: A unique identifier of our Netlify website. This is also called the API ID. It can be found from the site details section in the site settings page in the Netlify web UI. It’s listed as the API ID.
  2. Personal access token: Access token to modify the Netlify website. It can be generated in the user application settings page in the Netlify web UI by following the wizard that appears when the “new access token” button is clicked.

Similar to the personal access tokens in GitLab, the token in Netlify has wide access as well, so you need to be careful how you handle it. Unintended access could result in, for example, your website being defaced.

GitLab project

Next, we’ll need a project in GitLab to host the source code for our website. We can create one by following the official guide using the GitLab user account. If you made your repository private, you’ll need to add the bot user as a guest member as well.

We can now start uploading our source code to the GitLab project. For now, we’ll use a simple Markdown page that we’ll convert to HTML. Upload the following Markdown file in path index.md to the master branch.

# Hello world!

We’ll later use Pandoc’s containers to convert our index.md Markdown file to the index.html HTML file in the GitLab CI pipeline. We can also test it locally with Docker as follows:

$ docker run --rm -v $(pwd):/data \
    pandoc/ubuntu \
    --from markdown --to html --output index.html index.md

If you wish, you can also try out a full-blown static website generator to generate a more complex static website.

GitLab CI pipeline variables

Before we can create the pipeline, we’ll need to add a couple of variables to the CI pipeline configuration. The variables can be accessed from the GitLab project page. For all the variables we’ll add, we’re going to use these settings:

Let’s add the following variable values.

GitLab CI pipeline

GitLab CI is used for wiring up all the components we’ve set up earlier. It’ll build our website from the Markdown file, upload the contents to Netlify, and send a message with a draft link to the appropriate merge request. With the variables added, we can begin constructing our GitLab CI pipeline using a YAML file. Let’s create a new file named .gitlab-ci.yml and add the following contents to it.

stages:
- build
- publish

workflow:
  rules:
  - if: $CI_MERGE_REQUEST_ID
  - if: $CI_COMMIT_BRANCH == "master"

variables:
  NETLIFY_DIRECTORY: public

Here’s what each of the sections does:

Next, let’s add a pipeline step for building the website. We can do this by adding the following contents to the end of the .gitlab-ci.yml YAML file.

build-site:
  stage: build
  image:
    name: pandoc/ubuntu
    entrypoint: [""]
  before_script:
  - mkdir -p $NETLIFY_DIRECTORY
  script:
  - pandoc --from markdown --to html --output "$NETLIFY_DIRECTORY/index.html" index.md
  artifacts:
    paths:
    - $NETLIFY_DIRECTORY

Here’s what each section above does:

The above pipeline step just builds the website and stores it in the pipeline, and we still need to upload the contents to Netlify. For that, we need to add another pipeline step. Let’s add the following contents to the end of the .gitlab-ci.yml YAML file.

publish-to-netlify:
  stage: publish
  image: registry.gitlab.com/lepovirta/netlify-deployer:ci
  script:
  - gitlab-deploy-site

In the above YAML snippet, we’ll add a pipeline step to the publish stage and execute a container image I created for publishing sites to Netlify. The step will execute one command, gitlab-deploy-site, that’s pre-built to the container image. Pretty much all of the magic for publishing to Netlify happen within the container image. We’ll take a look at it in a later section.

With the above contents added to the .gitlab-ci.yml file, we can upload it to the root directory of the GitLab project. After the upload, a new pipeline should and execute the steps we created. After the pipeline has finished, you should see a “Hello world” message in the Netlify site you created earlier.

Test run

We should now have everything set up, so let’s complete this guide with a test round where we create a merge request to preview website changes.

  1. Create a new branch in your GitLab project
  2. In the new branch, change the contents of the index.md file.
  3. Push the changes to the branch in GitLab.
  4. Create a new merge request for the branch.
  5. Wait for the pipeline to complete.

After completing these steps, you should see a comment from your GitLab bot account pop up in the merge request with a link to a preview site. Clicking the link will lead you to a temporary site on Netlify that displays the changes you made. If you managed to get the preview site up, congratulations for completing this guide! To get the changes visible in the actual site, merge the merge request and wait for the pipeline to complete.

netlify-deployer

Most of the heavy lifting in arranging the site preview on GitLab CI pipelines was done using a small project I created called netlify-deployer. What it does is not that advanced but it manages to combine most of the details in a neat little package (a container image to be more precise). Here’s how it works.

The site deploy functionality is implemented using a small Go program. It performs the deployment using the official Netlify Open API client library from files on a directory. The directory, the site ID, the authentication token, and the rest of the details are configured using environment variables that are picked up using Kelsey Hightower’s envconfig library. The program creates a preview site when the environment variable NETLIFY_DRAFT is set.

The site deployment could have also been done using Netlify’s command-line interface, but I felt it didn’t fit the purpose. The first issue is that it was pretty difficult to capture the draft link from the tool output so that I could send it forward. Second, the tool depends on a high amount of libraries and tools which made it hard to keep the container image small in size. For the CLI, it makes sense to have all the dependencies it has because it does more than just deployments, but in my case, they were just in the way.

In the GitLab CI pipeline, the Go program is wrapped with a Bash script that figures out all the configuration details from the CI environment variables and passes them to the program. Additionally, it takes care of posting the draft link as a comment to the merge request using curl.

The netlify-deployer tool is available as a Docker container image in the GitLab container registry in two variants. One of the variants just includes the Go program in a minimal Docker image that’s intended to be used from the command-line or automation scripts. The second variant includes the wrapper script and other useful tools that make it easy to use it in a GitLab CI pipeline.

Conclusions

In this article, I presented how you can integrate Netlify website previews in your GitLab CI pipeline for your static website without handing over the CI duties to Netlify. For this task, I used my small program called netlify-deployer to glue GitLab and Netlify together.

If you’re interested in seeing a real-world example of the setup in action, check out my website’s GitLab project. Thanks for reading!

posted
tags
gitlab, netlify
more
previous, all