How to Manage Project Dependencies with GitLab – Part 2.

In the previous post I’ve talked about creating and installing own Python packages using GitLab’s tools. Today I’d like to focus a bit more on the topic and show how you can make this process even better. We’re going to use GitLab’s CI to build the PyPi packages automatically.

Problems

Let’s imagine an average Cathy, who learns to program and has just finished her first Python tutorial. She’d like to contribute to some open source projects to gain experience. Knowing mymath module from her first projects she came up with an idea on how to improve it.

What is more of a problem is that she does not have is the knowledge of how to test the code (the tutorial didn’t offer much hands-on experience). Moreover, she has no idea of building packages, coding style conventions and so on.

Automation for help

That’s where I as a person responsible for the project must come in. What I’d like to achieve is to have an automated process that would:

  1. automatically run linter,
  2. check type hints,
  3. run automatically unit tests,
  4. build package and set a new version, but only when master is merged.

This way Cathy would always be sure that she didn’t break anything, be able to adopt a coherent coding style and learn how packages are built. Let’s help her.

Local tests

First we need to write proper tests locally before we pull them into our CI process.

Let’s begin with requirements. I want black for linting, pytest for unit tests and mypy for type checking. The requirements.txt file would look the following way:

black==20.8b1
mypy==0.790
pytest==6.1.2

Next, I need to update the .gitignore so that virtual environments don’t get commited to the repo.

*.pyc
__pycache__/

build/
dist/
mymath.egg-info/

venv/
.vscode/

.mypy_cache/
.pytest_cache/

Now I can write tests. I want to keep them next to the source code in my_math/tests.py module.

import pytest

from my_math import (
    add,
    div,
    mul,
    sub,
)


@pytest.mark.parametrize("a,b,expected", [(1, 2, 3), (4, -1, 3)])
def test_add(a: int, b: int, expected: int):
    assert add(a, b) == expected


@pytest.mark.parametrize("a,b,expected", [(3, 2, 1), (4, -1, 5)])
def test_sub(a: int, b: int, expected: int):
    assert sub(a, b) == expected


@pytest.mark.parametrize("a,b,expected", [(1, 2, 2), (4, -1, -4), (2, 0, 0)])
def test_mul(a: int, b: int, expected: int):
    assert mul(a, b) == expected


@pytest.mark.parametrize("a,b,expected", [(6, 4, 1.5), (2, 1, 2)])
def test_div(a: int, b: int, expected: float):
    assert div(a, b) == expected


def test_div_by_zero():
    with pytest.raises(ZeroDivisionError):
        div(2, 0)

I’d like to stop here for a moment. I normally choose unittest, but here pytest with its functional approach seems like a more natural choice. In order to cover more potential edge cases (like operating on negative numbers or zero multiplication) I used the parametrization which allows me to avoid code duplication. Now we can run them with pytest my_math/tests/py command.

$ pytest my_math/tests.py 
======================================= test session starts ========================================
platform linux -- Python 3.8.5, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
rootdir: /home/w.gonczaronek/Projects/blog/my-math
collected 10 items                                                                                 

my_math/tests.py ..........                                                                  [100%]

======================================== 10 passed in 0.02s =======================================

OK. Now let’s move on to black and mypy. For black I also want to loosen the requirement of max line length to 90 characters.

 $ mypy my_math/
Success: no issues found in 3 source files
 $ black --check --config pyproject.toml my_math/
All done! ✨ 🍰 ✨
3 files would be left unchanged.

The pyproject.toml file you see has simple contents:

[tool.black]
line-length = 90

It seems that at least locally everything is fine and we can start the integration with GitLab’s CI.

Automation

At this stage we want to move all of those steps to CI to automate tasks, help Cathy (and us as well). For this we need proper CI config file.

.before: &before_script
  before_script:
    - pip install -r requirements.txt

stages:
  - test

default:
  image: "python:3.8"

pytest:
  stage: test
  script:
    - pytest my_math/*
  <<: *before_script

black:
  stage: test
  script:
    - black --config pyproject.toml --check my_math/
  <<: *before_script

mypy:
  stage: test
  script:
    - mypy my_math/
  <<: *before_script

Aaand it passed.

This means we can try to support different python versions:

.before: &before_script
  before_script:
    - pip install -r requirements.txt

stages:
  - test

default:
  image: "python:3.8"

.pytest:
  stage: test
  script:
    - pytest my_math/*
  <<: *before_script

.black:
  stage: test
  script:
    - black --config pyproject.toml --check my_math/
  <<: *before_script

.mypy:
  stage: test
  script:
    - mypy my_math/
  <<: *before_script


pytest-3.7:
  extends: ".pytest"
  image: "python:3.7"
black-3.7:
  extends: ".black"
  image: "python:3.7"
mypy-3.7:
  extends: ".mypy"
  image: "python:3.7"

pytest-3.8:
  extends: ".pytest"
  image: "python:3.8"
black-3.8:
  extends: ".black"
  image: "python:3.8"
mypy-3.8:
  extends: ".mypy"
  image: "python:3.8"

pytest-3.9:
  extends: ".pytest"
  image: "python:3.9"
black-3.9:
  extends: ".black"
  image: "python:3.9"
mypy-3.9:
  extends: ".mypy"
  image: "python:3.9"

Since that works, we can now move on to publishing our packages. This nearly boils down to passing proper commands to next execution stage. What is cool, though is that you don’t need to pass any tokens or project ids as previously, because GitLab’s runner can take it from its environment variables and use CI_JOB_TOKEN to authenticate to the registry.

stages:
  - test
  - build

# ...

build_package:
  stage: build
  image: "python:3.9-slim"
  before_script:
    - pip install twine
  script:
    - python3 setup.py sdist bdist_wheel
    - TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=gitlab-ci-token python3 -m twine upload --verbose --repository-url https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/packages/pypi dist/*
  only:
    refs:
      - master

Back to Cathy

Finally we have full infrastructure prepared for our collaborators. Cathy wants to help and introduce a new function: modulo operations. What does she have to do?

First, she writes her function:

def mod(a: int, b: int) -> str:
    return a % b

There are several problems here: it doesn’t have test coverage, it does not meet PEP8 requirements (no newline at the end, too little space between previous function) and it returns incorrect type. She pushes her changes anyway.

The job has failed miserably (notice that build process wasn’t even planned – it’s only meat to be run on master).

Afterwards she may want to change return type to correct one:

def mod(a: int, b: int) -> int:
    return a % b

and run mypy:

mypy my_math/
Success: no issues found in 3 source files

Hurray! Let’s commit those changes and move on to fix code style.

black --config pyproject.toml my_math/
reformatted /home/gonczor/Projects/my-math/my_math/my_math.py
All done! ✨ 🍰 ✨
1 file reformatted, 2 files left unchanged.

Now we can commit again.

The last thing is writing tests. We want Cathy to provide proper coverage for the new function. Luckily, she can make use of the previous code with which she comes up with the following solution:

@pytest.mark.parametrize("a,b,expected", [(6, 4, 2), (2, 2, 0)])
def test_mod(a: int, b: int, expected: int):
    assert mod(a, b) == expected

Since tests have passed:

we might now bump version number and deploy our package.

Summary

To sum up, we’ve created a decent infrastructure that would help our future collaborators work on our project. We’ve automated build process so that tests for different python versions are triggered automatically. In addition to this we’ve ensured proper code quality by including linter and type checks in our CI process.

I strongly encourage you to share your thoughts and ideas for further improvements in the comments section below.

See more