Launching tests#

pytest#

Since we use pytest to run the tests, you can also use it on its own. We do recommend using the tox command though:

pytest pylint -k test_functional

You can use pytest directly. If you want to run tests on a specific portion of the code with pytest and your local python version:

python3 -m pytest

Everything in tests/message with coverage for the relevant code (require pytest-cov):

python3 -m pytest tests/message/ --cov=pylint.message
coverage html

Only the functional test "missing_kwoa_py3":

python3 -m pytest "tests/test_functional.py::test_functional[missing_kwoa_py3]"

tox#

You can also optionally install tox and run our tests using the tox package, as in:

python -m tox
python -m tox -epy38 # for Python 3.8 suite only
python -m tox -epylint # for running Pylint over Pylint's codebase
python -m tox -eformatting # for running formatting checks over Pylint's codebase

It's usually a good idea to run tox with --recreate. This flag tells tox to re-download all dependencies before running the tests. This can be important when a new version of astroid or any of the other dependencies has been published:

python -m tox --recreate # The entire tox environment will be recreated
python -m tox --recreate -e py310 # The python 3.10 tox environment will be recreated

To run only a specific test suite, use a pattern for the test filename (without the .py extension), as in:

python -m tox -e py310 -- -k test_functional
python -m tox -e py310 -- -k  \*func\*
python -m tox --recreate -e py310 -- -k test_functional # With recreation of the environment

Primer tests#

Pylint also uses what we refer to as primer tests. These are tests that are run automatically in our Continuous Integration and check whether any changes in Pylint lead to crashes or fatal errors on the stdlib and a selection of external repositories.

To run the primer tests you can add either --primer-stdlib or --primer-external to the pytest command. If you want to only run the primer you can add either of their marks, for example:

pytest -m primer_stdlib --primer-stdlib

The external primer can be run with:

pytest -m primer_external_batch_one --primer-external # Runs batch one

The list of repositories is created on the basis of three criteria: 1) projects need to use a diverse range of language features, 2) projects need to be well maintained and 3) projects should not have a codebase that is too repetitive. This guarantees a good balance between speed of our CI and finding potential bugs.

You can find the latest list of repositories and any relevant code for these tests in the tests/primer directory.